java.util.NoSuchElementException when serializing Map with default value

[Maciej Szymkiewicz]

Hi everyone,

I suspect there is no point in submitting a JIRA to fix this (not a
Spark issue?) but I would like to know if this problem is documented
anywhere. Somehow Kryo is loosing default value during serialization:

    scala> import org.apache.spark.{SparkContext, SparkConf}
    import org.apache.spark.{SparkContext, SparkConf}

    scala> val aMap = Map[String, Long]().withDefaultValue(0L)
    aMap: scala.collection.immutable.Map[String,Long] = Map()

    scala> aMap("a")
    res6: Long = 0

    scala> val sc = new SparkContext(new
    SparkConf().setAppName("bar").set("spark.serializer",
    "org.apache.spark.serializer.KryoSerializer"))

    scala> sc.parallelize(Seq(aMap)).map(_("a")).first
    16/09/28 09:13:47 ERROR Executor: Exception in task 2.0 in stage 2.0
    (TID 7)
    java.util.NoSuchElementException: key not found: a

while Java serializer works just fine:

    scala> val sc = new SparkContext(new
    SparkConf().setAppName("bar").set("spark.serializer",
    "org.apache.spark.serializer.JavaSerializer"))

    scala> sc.parallelize(Seq(aMap)).map(_("a")).first
    res9: Long = 0

-- 
Best regards,
Maciej