[
https://issues.apache.org/jira/browse/SPARK-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14088768#comment-14088768
]
Soren Macbeth commented on SPARK-2878:
--------------------------------------
My particular case _feels_ a bit different, although I suspect the root
case is likely the same. In my case, intermittently, some of the scala
collection classes that are part of chill and part of the default kryo
serializer wouldn't get registered in my mesos slave tasks. I fixed it my
explicitly registering them in my custom serializer to ensure the the class
in question, scala's `Some`, always had a consistent id in kryo.
I haven't commented out my hack in >= 1.0.0 to see if it's still an issue
or not.
On Wed, Aug 6, 2014 at 7:23 PM, Graham Dennis (JIRA) <[email protected]>
--
http://about.me/soren
> Inconsistent Kryo serialisation with custom Kryo Registrator
> ------------------------------------------------------------
>
> Key: SPARK-2878
> URL: https://issues.apache.org/jira/browse/SPARK-2878
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.0.0, 1.0.2
> Environment: Linux RedHat EL 6, 4-node Spark cluster.
> Reporter: Graham Dennis
>
> The custom Kryo Registrator (a class with the
> org.apache.spark.serializer.KryoRegistrator trait) is not used with every
> Kryo instance created, and this causes inconsistent serialisation and
> deserialisation.
> The Kryo Registrator is sometimes not used because of a ClassNotFound
> exception that only occurs if it *isn't* the Worker thread (of an Executor)
> that tries to create the KryoRegistrator.
> A complete description of the problem and a project reproducing the problem
> can be found at https://github.com/GrahamDennis/spark-kryo-serialisation
> I have currently only tested this with Spark 1.0.0, but will try to test
> against 1.0.2.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
