Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/119#discussion_r10506499
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -130,6 +130,16 @@ class SparkContext(
val isLocal = (master == "local" || master.startsWith("local["))
+ // Create a classLoader for use by the driver so that jars added via
addJar are available to the
+ // driver. Do this before all other initialization so that any thread
pools created for this
+ // SparkContext uses the class loader.
+ // Note that this is config-enabled as classloaders can introduce subtle
side effects
+ private[spark] val classLoader = if
(conf.getBoolean("spark.driver.loadAddedJars", false)) {
+ val loader = new SparkURLClassLoader(Array.empty[URL],
this.getClass.getClassLoader)
+ Thread.currentThread.setContextClassLoader(loader)
--- End diff --
This will capture a pointer to the classlaoder in which the SC was created.
So addJars can be called from anywhere and it will always augment this class
loader.
I think this means that the class will be visible to (a) the thread that
created the sc and (b) any threads created by that thread. Though it would be
good to verify that the context class loader is passed on to child threads or
they delegate to that of the parent.
This does mean that a thread entirely outside of the SparkContext-creating
thread and its children won't have the class loaded. I think that's actually
desirable given that you may have a case where mutliple SparkContext's are
created in the same JVM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
