Looking at the source code for StateMap[1], which is used by
JavaPairDStream#mapWithState(), it looks like state keys are
serialized using an ObjectOutputStream. I couldn't find a reference to
this restriction in the documentation. Did I miss that?
Unless I'm mistaken, I'm guessing there isn't a w
, Shixiong(Ryan) Zhu
wrote:
> You can use Kryo. It also implements KryoSerializable which is supported by
> Kryo.
>
> On Fri, Oct 7, 2016 at 11:39 AM, Joey Echeverria wrote:
>>
>> Looking at the source code for StateMap[1], which is used by
>> JavaPairDStream#mapWi
, 2016 at 9:54 AM, Shixiong(Ryan) Zhu
wrote:
> That's enough. Did you see any error?
>
> On Mon, Oct 10, 2016 at 5:08 AM, Joey Echeverria wrote:
>>
>> Hi Ryan!
>>
>> Do you know where I need to configure Kryo for this? I already have
>> spark.serializer=o
On Mon, Oct 10, 2016 at 11:28 AM, Joey Echeverria wrote:
> I do, I get the stack trace in this gist:
>
> https://gist.github.com/joey/d3bf040af31e854b3be374e2c016d7e1
>
> The class it references, com.rocana.data.Tuple, is registered with
> Kryo. Also, this is with 1.6.0 so if t
I tried with 1.6.2 and saw the same behavior.
-Joey
On Tue, Oct 11, 2016 at 5:18 PM, Shixiong(Ryan) Zhu
wrote:
> There are some known issues in 1.6.0, e.g.,
> https://issues.apache.org/jira/browse/SPARK-12591
>
> Could you try 1.6.1?
>
> On Tue, Oct 11, 2016 at 9:55 AM, Joey
o's
> "com.esotericsoftware.kryo.serializers.JavaSerializer". Did you set it for
> OpenHashMapBasedStateMap? You don't need to set anything for Spark's classes
> in 1.6.2.
>
>
> On Wed, Oct 12, 2016 at 7:11 AM, Joey Echeverria wrote:
>>
>> I tried with 1.6.2 and saw the same behavior.
>&g
