GitHub user markgrover opened a pull request:
https://github.com/apache/spark/pull/7739
[WIP] SPARK-9384: Easier setting of executor and driver classpath
Adding new property spark.common.extraClassPath for adding to both the
driver and executor classpath.
More testing needs to be done.
Currently one can set extra jars in the driver classpath using
spark.driver.extraClassPath property. Similarly, the executor's extra classpath
can be set using spark.executor.extraClassPath property.
However, many users (myself) included have often suffered from the dogma of
setting one and missing the other. Causing a lot of wasted time. And, this JIRA
is an effort to see if the status quo can be made better.
--jars sets the classpath set by the classloader but there is no such way
of setting the system classpath of the driver and the executor. This change
would allow users to override libraries shipped with Spark, or add jars like
some JDBC driver, that don't work unless they are adding to the system
classpath.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/markgrover/spark classpath2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/7739.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7739
----
commit 5173f9910b12e5446f76d6c6fe96a12ddc201254
Author: Mark Grover <[email protected]>
Date: 2015-07-27T17:46:33Z
Adding new property spark.common.extraClassPath for adding to both the
driver and executor classpath
Limited testing done so far
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
