GitHub user pwendell opened a pull request:
https://github.com/apache/spark/pull/299
[WIP] Clean up and simplify Spark configuration
Over time as we've added more deployment modes, this have gotten a bit
unwieldy with user-facing configuration options in Spark. Going forward we'll
advise all users to run `spark-submit` to launch applications. This is a WIP
patch but it makes the following improvements:
1. Improved `spark-env.sh.template` which was missing a lot of things users
now set in that file.
2. Removes the shipping of SPARK_CLASSPATH, SPARK_JAVA_OPTS, and
SPARK_LIBRARY_PATH to the executors on the cluster. This was an ugly hack.
Instead it introduces config variables spark.executor.extraJavaOpts,
spark.executor.extraLibraryPath, and spark.executor.extraClassPath.
3. Adds ability to set these same variables for the driver using
`spark-submit`.
4. Allows you to load system properties from a `spark-defaults.conf` file
when running `spark-submit`. This will allow setting both SparkConf options and
other system properties utilized by `spark-submit`.
5. Made `SPARK_LOCAL_IP` an environment variable rather than a SparkConf
property. This is more consistent with it being set on each node.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pwendell/spark config-cleanup
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/299.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #299
----
commit 7be5d8a08aecd94df0d36501781535c5d1aaa1c1
Author: Patrick Wendell <[email protected]>
Date: 2014-03-31T20:16:46Z
Change spark.local.dir -> SPARK_LOCAL_DIRS
commit 8804e39e2fc0af20bc4e1603b15af0dd0ddf5985
Author: Patrick Wendell <[email protected]>
Date: 2014-04-01T07:13:52Z
Stash of adding config options in submit script and YARN
commit 5e9485f9b58de132d6734e995f7e002a9413a518
Author: Patrick Wendell <[email protected]>
Date: 2014-04-01T22:00:42Z
executorJavaOpts
commit f3598014134df4c36f12221a8ee7c6dbb9ad9d7e
Author: Patrick Wendell <[email protected]>
Date: 2014-04-01T22:10:31Z
Remove SPARK_LIBRARY_PATH
commit 581391a0ea273689c3e74611f283df168da0565d
Author: Patrick Wendell <[email protected]>
Date: 2014-04-01T22:18:42Z
SPARK_JAVA_OPTS --> SPARK_MASTER_OPTS for master settings
commit a2c2bf091899bdef6511fb977cab42cedfab4995
Author: Patrick Wendell <[email protected]>
Date: 2014-04-01T23:31:11Z
Small clean-up
commit 82191a6d0f3a7dc79640c6a1dc1597bf22287dfe
Author: Patrick Wendell <[email protected]>
Date: 2014-04-02T00:35:13Z
Don't ship executor envs
commit a57c4f87f9026dcf0a37f41d3ed1785182bae82e
Author: Patrick Wendell <[email protected]>
Date: 2014-04-02T01:02:43Z
Clean up terminology inside of spark-env script
commit 3ffc125683ceeed1ea07dd28c89c85b3ae31aa26
Author: Patrick Wendell <[email protected]>
Date: 2014-04-02T05:45:59Z
Library path and classpath for drivers
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
