Min Shen created SPARK-18646:
--------------------------------
Summary: ExecutorClassLoader for spark-shell does not honor
spark.executor.userClassPathFirst
Key: SPARK-18646
URL: https://issues.apache.org/jira/browse/SPARK-18646
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.6.2
Reporter: Min Shen
When submitting a spark-shell application, the executor side classloader is set
to be {{ExecutorClassLoader}}.
However, it appears that when {{ExecutorClassLoader}} is used, parameter
{{spark.executor.userClassPathFirst}} is not honored.
It turns out that, since {{ExecutorClassLoader}} class is defined as
{noformat}
class ExecutorClassLoader(conf: SparkConf, classUri: String, parent:
ClassLoader,
userClassPathFirst: Boolean) extends ClassLoader with Logging
{noformat}
its parent classloader is actually the system default classloader (due to
{{ClassLoader}} class's default constructor) rather than the "parent"
classloader specified in {{ExecutorClassLoader}}'s constructor.
As a result, when {{spark.executor.userClassPathFirst}} is set to true, even
though the "parent" classloader is {{ChildFirstURLClassLoader}},
{{ExecutorClassLoader.getParent()}} will return the system default classloader.
Thus, when {{ExecutorClassLoader}} tries to load a class, it will first attempt
to load it through the system default classloader, and this will break the
{{spark.executor.userClassPathFirst}} behavior.
A simple fix would be to define {{ExecutorClassLoader}} as:
{noformat}
class ExecutorClassLoader(conf: SparkConf, classUri: String, parent:
ClassLoader,
userClassPathFirst: Boolean) extends ClassLoader(parent) with Logging
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
