[
https://issues.apache.org/jira/browse/SPARK-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen updated SPARK-3166:
-----------------------------
Target Version/s: (was: 1.2.0)
> Custom serialisers can't be shipped in application jars
> -------------------------------------------------------
>
> Key: SPARK-3166
> URL: https://issues.apache.org/jira/browse/SPARK-3166
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.0.2
> Reporter: Graham Dennis
>
> Spark cannot currently use a custom serialiser that is shipped with the
> application jar. Trying to do this causes a java.lang.ClassNotFoundException
> when trying to instantiate the custom serialiser in the Executor processes.
> This occurs because Spark attempts to instantiate the custom serialiser
> before the application jar has been shipped to the Executor process. A
> reproduction of the problem is available here:
> https://github.com/GrahamDennis/spark-custom-serialiser
> I've verified this problem in Spark 1.0.2, and Spark master and 1.1 branches
> as of August 21, 2014. This issue is related to SPARK-2878, and my fix for
> that issue (https://github.com/apache/spark/pull/1890) also solves this. My
> pull request was not merged because it adds the user jar to the Executor
> processes' class path at launch time. Such a significant change was thought
> by [~rxin] to require more QA, and should be considered for inclusion in 1.2
> at the earliest.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
