GitHub user piaozhexiu opened a pull request:
https://github.com/apache/spark/pull/5617
[SPARK-7037] [CORE] Inconsistent behavior for non-spark config properties
in spark-shell and spark-submit
When specifying non-spark properties (i.e. names don't start with spark.)
in the command line and config file, spark-submit and spark-shell behave
differently, causing confusion to users.
Here is the summary-
* spark-submit
* --conf k=v => silently ignored
* spark-defaults.conf => applied
* spark-shell
* --conf k=v => show a warning message and ignored
* => show a warning message and ignored
I assume that ignoring non-spark properties is intentional. If so, it
should always be ignored with a warning message in all cases.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/piaozhexiu/spark SPARK-7037
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5617.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5617
----
commit bf991f18bd9bdd08f24f57f1d373da86cca4e669
Author: Cheolsoo Park <[email protected]>
Date: 2015-04-21T19:12:22Z
Ignore non-spark properties with a warning message in all cases
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
