GitHub user liancheng opened a pull request:
https://github.com/apache/spark/pull/1699
[SPARK-2678][Core] Prevents `spark-submit` from shadowing application
options
JIRA issue: [SPARK-2678](https://issues.apache.org/jira/browse/SPARK-2678)
Currently, `spark-submit` shadows user application options. Namely, options
like `--help` are always captured by `spark-submit` and won't be passed to the
user application. A negative impact of this is that, every time we add a new
option to `spark-submit`, we may potentially break some existing user scripts
if the new option name happens to shadow an existing option in the application.
This PR introduced an **incompatible** change to fix this issue: `--` is
used as a separator between `spark-submit` options and user application
options. All arguments after `--` are passed to the application as is. Thus,
```bash
./bin/spark-submit --class Foo user.jar arg1 -arg2 --arg3 x
# or
./bin/spark-submit user.jar --class Foo arg1 -arg2 --arg3 x
```
must be rewritten to
```bash
./bin/spark-submit --class Foo user.jar -- arg1 -arg2 --arg3 x
# or
./bin/spark-submit user.jar --class Foo -- arg1 -arg2 --arg3 x
```
@pwendell Please help review, thanks. Especially, maybe we need a vote here
to decide whether fixing this issue worth an incompatible change.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/liancheng/spark spark-submit-args
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1699.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1699
----
commit d38978125157ee9337fe6b936b1d84910f8e9290
Author: Cheng Lian <[email protected]>
Date: 2014-07-25T19:05:38Z
Fixed SPARK-2678 by introducing "--" as application option separator
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
