Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4877#discussion_r25724861
--- Diff:
core/src/test/scala/org/apache/spark/serializer/KryoSerializerSuite.scala ---
@@ -242,6 +244,24 @@ class KryoSerializerSuite extends FunSuite with
SharedSparkContext {
ser.newInstance().deserialize[ClassLoaderTestingObject](bytes)
}
}
+
+ test("registration of HighlyCompressedMapStatus") {
+ val conf = new SparkConf(false)
+ conf.set("spark.kryo.registrationRequired", "true")
+
+ // these cases require knowing the internals of RoaringBitmap a
little. Blocks span 2^16
+ // values, and they use a bitmap (dense) if they have more than 4096
values, and an
+ // array (sparse) if they use less. So we just create two cases, one
sparse and one dense.
+ // and we use a roaring bitmap for the empty blocks, so we trigger the
dense case w/ mostly
+ // empty blocks
+
+ val ser = new KryoSerializer(conf).newInstance()
+ val denseBlockSizes = new Array[Long](5000)
+ val sparseBlockSizes = Array[Long](0L, 1L, 0L, 2L)
+ Seq(denseBlockSizes, sparseBlockSizes).foreach{ blockSizes =>
--- End diff --
good find on the dense vs sparse thing.
can you add a space before { ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
