[
https://issues.apache.org/jira/browse/SPARK-1863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
koert kuipers updated SPARK-1863:
---------------------------------
Description:
See here:
http://apache-spark-user-list.1001560.n3.nabble.com/java-serialization-errors-with-spark-files-userClassPathFirst-true-td5832.html
The issue seems to be that within ChildExecutorURLClassLoader userClassLoader
has no visibility on classes managed by parentClassLoader because their is no
parent/child relationship. What this means that if a class is loaded by
userClassLoader and it refers to a class loaded by parentClassLoader you get a
NoClassDefFoundError.
When i addresses this by creating a new version of ChildExecutorURLClassLoader
that does have the proper parent-child relationship and reverses the loading
order inside loadClass the class loading seemed to work fine but now classes
like SparkEnv were loaded by ChildExecutorURLClassLoader leading to NPEs on
SparkEnv.get
was:
See here:
http://apache-spark-user-list.1001560.n3.nabble.com/java-serialization-errors-with-spark-files-userClassPathFirst-true-td5832.html
The issue seems to be that within ChildExecutorURLClassLoader userClassLoader
has no visibility on classes managed by parentClassLoader because their is no
parent/child relationship. What this means that if a class is loaded by
userClassLoader and it refers to a class loaded by parentClassLoader you get a
NoClassDefFoundError.
> Allowing user jars to take precedence over Spark jars does not work as
> expected
> -------------------------------------------------------------------------------
>
> Key: SPARK-1863
> URL: https://issues.apache.org/jira/browse/SPARK-1863
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.0.0
> Reporter: koert kuipers
> Priority: Minor
>
> See here:
> http://apache-spark-user-list.1001560.n3.nabble.com/java-serialization-errors-with-spark-files-userClassPathFirst-true-td5832.html
> The issue seems to be that within ChildExecutorURLClassLoader userClassLoader
> has no visibility on classes managed by parentClassLoader because their is no
> parent/child relationship. What this means that if a class is loaded by
> userClassLoader and it refers to a class loaded by parentClassLoader you get
> a NoClassDefFoundError.
> When i addresses this by creating a new version of
> ChildExecutorURLClassLoader that does have the proper parent-child
> relationship and reverses the loading order inside loadClass the class
> loading seemed to work fine but now classes like SparkEnv were loaded by
> ChildExecutorURLClassLoader leading to NPEs on SparkEnv.get
--
This message was sent by Atlassian JIRA
(v6.2#6252)
