GitHub user brennonyork opened a pull request:
https://github.com/apache/spark/pull/3559
[SPARK-4616][Core] - SPARK_CONF_DIR is not effective in spark-submit
By default the `SPARK_CONF_DIR` is not capable of being set from the
`spark-submit` script, but a spark properties file is. After diving through the
code it turned out that the `SPARK_CONF_DIR` is actually a cyclic reference as
it is referenced within the `load-spark-env.sh` script and can also then be
reset within the loaded `spark-env.sh` file. What's worse is that, if the
`spark-env.sh` defined a `SPARK_CONF_DIR` it wouldn't be picked up and used to
grab the default `spark-defaults.conf` if no `--properties-file` flag was
present. As such, it seemed best to provide a `--environment-file` flag that
can be used to present an arbitrary bash file to the system which would preload
any necessary environment configuration options (including `SPARK_CONF_DIR`).
This then solves the original problem that the `SPARK_CONF_DIR` wasn't
effective within `spark-submit`, but also removes the cyclic dependency on
where and when the `SPARK_CONF_DIR` is loaded.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/brennonyork/spark SPARK-4616
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3559.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3559
----
commit d857bf713a431d02eb74f0852a0d3264d599e583
Author: Brennon York <[email protected]>
Date: 2014-12-01T20:12:43Z
fixes SPARK-4616 by adding an environment file that can be passed in
through the command line options
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
