GitHub user pwendell opened a pull request:
https://github.com/apache/spark/pull/398
SPARK-1480: Clean up use of classloaders
The Spark codebase is a bit fast-and-loose when accessing classloaders and
this has caused a few bugs to surface in master.
This patch defines some utility methods for accessing classloaders. This
makes the intention when accessing a classloader much more explicit in the code
and fixes a few cases where the wrong one was chosen.
case (a) -> We want the classloader that loaded Spark
case (b) -> We want the context class loader, or if not present, we want (a)
This patch provides a better fix for SPARK-1403
(https://issues.apache.org/jira/browse/SPARK-1403) than the current work
around, which it reverts. It also fixes a previously unreported bug that the
`./spark-submit` script did not work for running with `local` master. It didn't
work because the executor classloader did not properly delegate to the context
class loader (if it is defined) and in local mode the context class loader is
set by the `./spark-submit` script. A unit test is added for that case.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pwendell/spark class-loaders
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/398.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #398
----
commit b5d44c76aea8c758a9d5ff221b25e03adc530d09
Author: Patrick Wendell <[email protected]>
Date: 2014-04-13T05:52:12Z
SPARK-1480: Clean up use of classloaders
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
