GitHub user lvsoft opened a pull request:
https://github.com/apache/spark/pull/3490
spark-submit with accept multiple properties-files and merge the values
Current ```spark-submit``` accepts only one properties-file, and use
```spark-defaults.conf``` if unspecified.
A more nature approach is patching the properties-files sequentially
against ```spark-defaults.conf```.
This PR affairs:
1. spark-submit script: join multiple ```--properties-file``` with comma
and stored as ```SPARK_SUBMIT_PROPERTIES_FILES``` environment variable. Peek
each properties-file to set ```SPARK_SUBMIT_BOOTSTRAP_DRIVER``` flag.
2. SparkSubmitArguments.scala: similar with 1.
3. SparkSubmitDriverBootstrapper.scala: accept
```SPARK_SUBMIT_PROPERTIES_FILES``` and call ```getPropertiesFromFiles``` for
parsing.
4. Utils.scala: add ```getPropertiesFromFiles``` for the parsing of
multiple properties-files.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lvsoft/spark
spark_submit_with_multi_properties
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3490.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3490
----
commit c18a266a1fa0c20331faed1193c168c1021edcf1
Author: Lv, Qi <[email protected]>
Date: 2014-11-25T08:48:03Z
Spark submit accept multiple properties files
commit 752a0581fde0692ee05213b51d0fc0368d8fd205
Author: Lv, Qi <[email protected]>
Date: 2014-11-26T08:56:29Z
test pass
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
