Peter Torok created SPARK-5949: ---------------------------------- Summary: Driver program has to register roaring bitmap classes used by spark with Kryo when number of partitions is greater than 2000 Key: SPARK-5949 URL: https://issues.apache.org/jira/browse/SPARK-5949 Project: Spark Issue Type: Bug Affects Versions: 1.2.0 Reporter: Peter Torok
When more than 2000 partitions are being used with Kryo, the following classes need to be registered by driver program: - org.apache.spark.scheduler.HighlyCompressedMapStatus - org.roaringbitmap.RoaringBitmap - org.roaringbitmap.RoaringArray - org.roaringbitmap.ArrayContainer - org.roaringbitmap.RoaringArray$Element - org.roaringbitmap.RoaringArray$Element[] Our project doesn't have dependency on roaring bitmap and HighlyCompressedMapStatus is intended for internal spark usage. Spark should take care of this registration when Kryo is used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org