[
https://issues.apache.org/jira/browse/SPARK-27367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-27367:
----------------------------------
Affects Version/s: (was: 3.0.0)
3.1.0
> Faster RoaringBitmap Serialization with v0.8.0
> ----------------------------------------------
>
> Key: SPARK-27367
> URL: https://issues.apache.org/jira/browse/SPARK-27367
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.1.0
> Reporter: Imran Rashid
> Priority: Major
>
> RoaringBitmap 0.8.0 adds faster serde, but also requires us to change how we
> call the serde routines slightly to take advantage of it. This is probably a
> worthwhile optimization as the every shuffle map task with a large # of
> partitions generates these bitmaps, and the driver especially has to
> deserialize many of these messages.
> See
> * https://github.com/apache/spark/pull/24264#issuecomment-479675572
> * https://github.com/RoaringBitmap/RoaringBitmap/pull/325
> * https://github.com/RoaringBitmap/RoaringBitmap/issues/319
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
