[
https://issues.apache.org/jira/browse/SPARK-1863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
koert kuipers updated SPARK-1863:
---------------------------------
Description:
See here:
http://apache-spark-user-list.1001560.n3.nabble.com/java-serialization-errors-with-spark-files-userClassPathFirst-true-td5832.html
The issue seems to be that within ChildExecutorURLClassLoader userClassLoader
has no visibility on classes managed by parentClassLoader because their is no
parent/child relationship. What this means that if a class is loaded by
userClassLoader and it refers to a class loaded by parentClassLoader you get a
NoClassDefFoundError.
When i addresses this by creating a new version of ChildExecutorURLClassLoader
that does have the proper parent-child relationship and reverses the loading
order inside loadClass the class loading seemed to work fine but now classes
like SparkEnv were loaded by ChildExecutorURLClassLoader leading to NPEs on
SparkEnv.get()
To verify that the issue was that SparkEnv was now loaded by
ChildExecutorURLClassLoader i forced SparkEnv to be loaded by the parent
classloader. That didnt help. Then i forced all spark classes to be loaded by
parent classloader and that did help. But it causes even bigger problems:
java.lang.LinkageError: loader constraint violation: when resolving overridden
method
"myclass.MyRDD.compute(Lorg/apache/spark/Partition;Lorg/apache/spark/TaskContext;)Lscala/collection/Iterator;"
the class loader (instance of
org/apache/spark/executor/ChildExecutorURLClassLoader) of the current class,
myclass/MyRDD, and its superclass loader (instance of
sun/misc/Launcher$AppClassLoader), have different Class objects for the type
TaskContext;)Lscala/collection/Iterator; used in the signature
was:
See here:
http://apache-spark-user-list.1001560.n3.nabble.com/java-serialization-errors-with-spark-files-userClassPathFirst-true-td5832.html
The issue seems to be that within ChildExecutorURLClassLoader userClassLoader
has no visibility on classes managed by parentClassLoader because their is no
parent/child relationship. What this means that if a class is loaded by
userClassLoader and it refers to a class loaded by parentClassLoader you get a
NoClassDefFoundError.
When i addresses this by creating a new version of ChildExecutorURLClassLoader
that does have the proper parent-child relationship and reverses the loading
order inside loadClass the class loading seemed to work fine but now classes
like SparkEnv were loaded by ChildExecutorURLClassLoader leading to NPEs on
SparkEnv.get
> Allowing user jars to take precedence over Spark jars does not work as
> expected
> -------------------------------------------------------------------------------
>
> Key: SPARK-1863
> URL: https://issues.apache.org/jira/browse/SPARK-1863
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.0.0
> Reporter: koert kuipers
> Priority: Minor
>
> See here:
> http://apache-spark-user-list.1001560.n3.nabble.com/java-serialization-errors-with-spark-files-userClassPathFirst-true-td5832.html
> The issue seems to be that within ChildExecutorURLClassLoader userClassLoader
> has no visibility on classes managed by parentClassLoader because their is no
> parent/child relationship. What this means that if a class is loaded by
> userClassLoader and it refers to a class loaded by parentClassLoader you get
> a NoClassDefFoundError.
> When i addresses this by creating a new version of
> ChildExecutorURLClassLoader that does have the proper parent-child
> relationship and reverses the loading order inside loadClass the class
> loading seemed to work fine but now classes like SparkEnv were loaded by
> ChildExecutorURLClassLoader leading to NPEs on SparkEnv.get()
> To verify that the issue was that SparkEnv was now loaded by
> ChildExecutorURLClassLoader i forced SparkEnv to be loaded by the parent
> classloader. That didnt help. Then i forced all spark classes to be loaded by
> parent classloader and that did help. But it causes even bigger problems:
> java.lang.LinkageError: loader constraint violation: when resolving
> overridden method
> "myclass.MyRDD.compute(Lorg/apache/spark/Partition;Lorg/apache/spark/TaskContext;)Lscala/collection/Iterator;"
> the class loader (instance of
> org/apache/spark/executor/ChildExecutorURLClassLoader) of the current class,
> myclass/MyRDD, and its superclass loader (instance of
> sun/misc/Launcher$AppClassLoader), have different Class objects for the type
> TaskContext;)Lscala/collection/Iterator; used in the signature
--
This message was sent by Atlassian JIRA
(v6.2#6252)
