GitHub user velvia opened a pull request:
https://github.com/apache/spark/pull/351
Spark-1230: Enable SparkContext.addJars() to load jars absent from CLASSPATH
This is an up-merge of @pwendell's PR,
https://github.com/apache/spark/pull/119, which was itself an up-merge of my
original PR, https://github.com/apache/incubator-spark/pull/299.
The basic idea is to add a class loader to SparkContext so that jars added
via `addJars()` can be made available to the SparkContext driver and its
threads as well. Presently `addJars()` only adds the jars to the executor's
classloader.
I have merged in changes and fixed the tests so they should all pass. I
will address comments in subsequent changes.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/velvia/spark
SPARK-1230-SparkContext-addJars-classloader
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/351.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #351
----
commit 2082de8bbffaba841bfa98f912ccfb6bbebb3670
Author: Evan Chan <[email protected]>
Date: 2013-12-13T17:22:56Z
Print out the class that was not found
commit 55a91ce31d2f8b0c68c5721fb8dfe815f355c713
Author: Evan Chan <[email protected]>
Date: 2013-12-13T23:40:14Z
Fix bug where classes in jars added via addJar() are not available to the
driver
Adds a driver-specific class loader that knows about the jars added via
addJar(), and
initialize it first thing in SparkEnv, so that all other
ThreadPool/ActorSystem
initialization will take advantage of the classloader, and user-added jars
are
made available to all Spark subsystems.
commit 8c8b15b3e0de8669b14c4eb9e12c797e4abbc7b7
Author: Evan Chan <[email protected]>
Date: 2013-12-29T08:15:17Z
Move classLoader initialization to SparkContext
SparkEnv is used by Executor as well; we want this change to affect driver
only.
commit da83523331ba1439bb7819ed2a7f4ea78e8e5bc9
Author: Evan Chan <[email protected]>
Date: 2013-12-29T08:25:53Z
Pass the current class loader when creating Akka threadpool
commit 2b9a61a8f4d5705d63eba043a04ae2c9ff5f4e4e
Author: Evan Chan <[email protected]>
Date: 2014-01-12T15:59:24Z
CR from pwendell
commit 9d62e3dd10976508502efc0e53d4f11e982af89b
Author: Evan Chan <[email protected]>
Date: 2014-01-17T06:19:27Z
CR: Rename ExecutorURLClassLoader -> SparkURLClassLoader
commit bb29207238a3955002fdfa63f59f9963d2e78a61
Author: Evan Chan <[email protected]>
Date: 2014-01-17T06:34:34Z
Use SparkURLClassLoader for driver only if spark.driver.add-dynamic-jars is
set
commit 77e865e6a2a615de322f009fb77b92efa20824e7
Author: Evan Chan <[email protected]>
Date: 2014-01-17T07:54:58Z
Document new config option and effect on addJar
commit ae4058d8559df7659a6649cced9d25dc806dafca
Author: Patrick Wendell <[email protected]>
Date: 2014-03-11T04:56:49Z
Minor changes
commit f12efb70aaffbed5e22c1ed62bde92cfd23881e1
Author: Patrick Wendell <[email protected]>
Date: 2014-03-11T17:55:32Z
Adding unit tests.
commit a3093cbaca00434075a5c8d04f7fbc5e88d15cfb
Author: Patrick Wendell <[email protected]>
Date: 2014-03-12T17:00:56Z
Review feedback
commit 60d2410cbe74f87ae5bdd09e9b9e6eddc4df4663
Author: Patrick Wendell <[email protected]>
Date: 2014-03-12T17:51:45Z
Style fixes
commit 6938fcd66e968fcda56b6bbc1e000bb8d422caf1
Author: Evan Chan <[email protected]>
Date: 2014-04-07T19:17:11Z
A few fixes to get Spark to compile
commit 5cd73817a94de656325d4d7680fb95f1e6256e65
Author: Evan Chan <[email protected]>
Date: 2014-04-08T00:25:11Z
Reset context classloader after each test so test passes
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
