Re: Map with state keys serialization

That fixed it!. I still had the serializer registered as a workaround for SPARK-12591. Thanks so much for your help Ryan! -Joey On Wed, Oct 12, 2016 at 2:16 PM, Shixiong(Ryan) Zhu wrote: > Oh, OpenHashMapBasedStateMap is serialized using Kryo's >

Re: Map with state keys serialization

Oh, OpenHashMapBasedStateMap is serialized using Kryo's "com.esotericsoftware.kryo.serializers.JavaSerializer". Did you set it for OpenHashMapBasedStateMap? You don't need to set anything for Spark's classes in 1.6.2. On Wed, Oct 12, 2016 at 7:11 AM, Joey Echeverria wrote: > I

Re: Map with state keys serialization

I tried with 1.6.2 and saw the same behavior. -Joey On Tue, Oct 11, 2016 at 5:18 PM, Shixiong(Ryan) Zhu wrote: > There are some known issues in 1.6.0, e.g., > https://issues.apache.org/jira/browse/SPARK-12591 > > Could you try 1.6.1? > > On Tue, Oct 11, 2016 at 9:55 AM,

Re: Map with state keys serialization

There are some known issues in 1.6.0, e.g., https://issues.apache.org/jira/browse/SPARK-12591 Could you try 1.6.1? On Tue, Oct 11, 2016 at 9:55 AM, Joey Echeverria wrote: > I tried wrapping my Tuple class (which is generated by Avro) in a > class that implements Serializable,

Re: Map with state keys serialization

I tried wrapping my Tuple class (which is generated by Avro) in a class that implements Serializable, but now I'm getting a ClassNotFoundException in my Spark application. The exception is thrown while trying to deserialize checkpoint state:

Re: Map with state keys serialization

That's enough. Did you see any error? On Mon, Oct 10, 2016 at 5:08 AM, Joey Echeverria wrote: > Hi Ryan! > > Do you know where I need to configure Kryo for this? I already have > spark.serializer=org.apache.spark.serializer.KryoSerializer in my > SparkConf and I registered the