[
https://issues.apache.org/jira/browse/SPARK-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14088539#comment-14088539
]
Graham Dennis commented on SPARK-2878:
--------------------------------------
This issue is possibly the root cause of a number of similar problems on the
web:
https://spark-project.atlassian.net/browse/SPARK-755
https://mail-archives.apache.org/mod_mbox/spark-user/201310.mbox/%3ccabedajgp5czvryvqvq8xvqcmipnxg1bai+dj3towreeq6op...@mail.gmail.com%3E
https://groups.google.com/forum/#!topic/spark-users/gJMjhm8cby0
http://stackoverflow.com/questions/19629360/spark-with-cassandra-failed-to-register-spark-kryo-registrator
http://apache-spark-developers-list.1001551.n3.nabble.com/Kryo-Issue-on-Spark-1-0-1-Mesos-0-18-2-td7498.html
http://apache-spark-user-list.1001560.n3.nabble.com/Crazy-Kryo-Exception-td5257.html
> Inconsistent Kryo serialisation with custom Kryo Registrator
> ------------------------------------------------------------
>
> Key: SPARK-2878
> URL: https://issues.apache.org/jira/browse/SPARK-2878
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.0.0, 1.0.2
> Environment: Linux RedHat EL 6, 4-node Spark cluster.
> Reporter: Graham Dennis
>
> The custom Kryo Registrator (a class with the
> org.apache.spark.serializer.KryoRegistrator trait) is not used with every
> Kryo instance created, and this causes inconsistent serialisation and
> deserialisation.
> The Kryo Registrator is sometimes not used because of a ClassNotFound
> exception that only occurs if it *isn't* the Worker thread (of an Executor)
> that tries to create the KryoRegistrator.
> A complete description of the problem and a project reproducing the problem
> can be found at https://github.com/GrahamDennis/spark-kryo-serialisation
> I have currently only tested this with Spark 1.0.0, but will try to test
> against 1.0.2.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
