[ https://issues.apache.org/jira/browse/SPARK-18646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Min Shen updated SPARK-18646: ----------------------------- Component/s: (was: Spark Core) Spark Shell > ExecutorClassLoader for spark-shell does not honor > spark.executor.userClassPathFirst > ------------------------------------------------------------------------------------ > > Key: SPARK-18646 > URL: https://issues.apache.org/jira/browse/SPARK-18646 > Project: Spark > Issue Type: Bug > Components: Spark Shell > Affects Versions: 1.6.2 > Reporter: Min Shen > > When submitting a spark-shell application, the executor side classloader is > set to be {{ExecutorClassLoader}}. > However, it appears that when {{ExecutorClassLoader}} is used, parameter > {{spark.executor.userClassPathFirst}} is not honored. > It turns out that, since {{ExecutorClassLoader}} class is defined as > {noformat} > class ExecutorClassLoader(conf: SparkConf, classUri: String, parent: > ClassLoader, > userClassPathFirst: Boolean) extends ClassLoader with Logging > {noformat} > its parent classloader is actually the system default classloader (due to > {{ClassLoader}} class's default constructor) rather than the "parent" > classloader specified in {{ExecutorClassLoader}}'s constructor. > As a result, when {{spark.executor.userClassPathFirst}} is set to true, even > though the "parent" classloader is {{ChildFirstURLClassLoader}}, > {{ExecutorClassLoader.getParent()}} will return the system default > classloader. > Thus, when {{ExecutorClassLoader}} tries to load a class, it will first > attempt to load it through the system default classloader, and this will > break the {{spark.executor.userClassPathFirst}} behavior. > A simple fix would be to define {{ExecutorClassLoader}} as: > {noformat} > class ExecutorClassLoader(conf: SparkConf, classUri: String, parent: > ClassLoader, > userClassPathFirst: Boolean) extends ClassLoader(parent) with Logging > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org