[
https://issues.apache.org/jira/browse/SPARK-19675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15877219#comment-15877219
]
Kohki Nishio commented on SPARK-19675:
--------------------------------------
First of all, sbt (0.13) is built with Scala2.10 and it can handle 2.11
application
When launching an application from "sbt run", it sets up all of necessary class
loader tree without having System class loader in the tree but within the same
JVM. This is why it doesn't cause any problem inside the application for most
of time.
However in Spark, it creates ExecutorClassLoader and its parent class:
ClassLoader's default constructor is to grab SystemClassLoader and set it as
its parent which is the problem here. That's why that class has different path
to load a class.
Since the application uses the class loader which is setup by SBT, it uses
classes from there and serializing objects, however deserializing in the worker
thread will try to load a class form System Class Loader.
I believe ExecutorClassLoader should always try to load classes from given
class loader, otherwise there's a chance to see java.io.InvalidClassException
> ExecutorClassLoader loads classes from SystemClassLoader
> --------------------------------------------------------
>
> Key: SPARK-19675
> URL: https://issues.apache.org/jira/browse/SPARK-19675
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.1.0, 2.2.0
> Environment: sbt / Play Framework
> Reporter: Kohki Nishio
> Priority: Minor
>
> Spark Executor loads classes from SystemClassLoader which contains
> sbt-launch.jar and it contains Scala2.10 binary, however Spark itself is
> built on Scala2.11, thus it's throwing InvalidClassException
> java.io.InvalidClassException: scala.Option; local class incompatible: stream
> classdesc serialVersionUID = -114498752079829388, local class
> serialVersionUID = 5081326844987135632
> at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616)
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1630)
> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1630)
> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> ExecutorClassLoader's desired class loder (parentLoader) actually contains
> the correct path (scala-library-2.11.8.jar) but it is not being used.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
