[
https://issues.apache.org/jira/browse/SPARK-18646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan resolved SPARK-18646.
---------------------------------
Resolution: Fixed
Assignee: Min Shen
Fix Version/s: 2.3.0
> ExecutorClassLoader for spark-shell does not honor
> spark.executor.userClassPathFirst
> ------------------------------------------------------------------------------------
>
> Key: SPARK-18646
> URL: https://issues.apache.org/jira/browse/SPARK-18646
> Project: Spark
> Issue Type: Bug
> Components: Spark Shell
> Affects Versions: 1.6.2
> Reporter: Min Shen
> Assignee: Min Shen
> Fix For: 2.3.0
>
>
> When submitting a spark-shell application, the executor side classloader is
> set to be {{ExecutorClassLoader}}.
> However, it appears that when {{ExecutorClassLoader}} is used, parameter
> {{spark.executor.userClassPathFirst}} is not honored.
> It turns out that, since {{ExecutorClassLoader}} class is defined as
> {noformat}
> class ExecutorClassLoader(conf: SparkConf, classUri: String, parent:
> ClassLoader,
> userClassPathFirst: Boolean) extends ClassLoader with Logging
> {noformat}
> its parent classloader is actually the system default classloader (due to
> {{ClassLoader}} class's default constructor) rather than the "parent"
> classloader specified in {{ExecutorClassLoader}}'s constructor.
> As a result, when {{spark.executor.userClassPathFirst}} is set to true, even
> though the "parent" classloader is {{ChildFirstURLClassLoader}},
> {{ExecutorClassLoader.getParent()}} will return the system default
> classloader.
> Thus, when {{ExecutorClassLoader}} tries to load a class, it will first
> attempt to load it through the system default classloader, and this will
> break the {{spark.executor.userClassPathFirst}} behavior.
> A simple fix would be to define {{ExecutorClassLoader}} as:
> {noformat}
> class ExecutorClassLoader(conf: SparkConf, classUri: String, parent:
> ClassLoader,
> userClassPathFirst: Boolean) extends ClassLoader(parent) with Logging
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
