Matthew Whelan created SPARK-5358:
-------------------------------------
Summary: spark.files.userClassPathFirst doesn't work correctly
Key: SPARK-5358
URL: https://issues.apache.org/jira/browse/SPARK-5358
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.2.0, 1.1.0
Reporter: Matthew Whelan
org.apache.spark.executor.ChildExecutorURLClassLoader#findClass delegates to
two different classloaders: a parent-less URL classloader, and the parent
classloader, as a fallback. The delegation is broken such that calling
loadClass twice in succession with the same parameters will fail the second
time.
The delegation to the userClassLoader calls findClass directly, which bypasses
the classloader's cache. So userClassLoader will attempt to define the same
class multiple times, throwing LinkageErrors after the first time.
The canonical way to change the default delegation scheme is to override
loadClass, rather than just findClass. It also might be sufficient to have
userClassLoader.findClass call super.loadClass.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
