Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/1699#issuecomment-50831029
@liancheng no, I'm suggesting keeping the current behavior, which is
defined in SparkSubmitArguments.scala around line 310, starting with this:
case value :: tail =>
if (inSparkOpts) {
What that code does is, as soon as an unrecognized option is found, it
considers everything else as app options. That means that it doesn't matter
where in the command line the "primary resource" is.
Examples:
spark-submit /my.jar --master foo --unknown-option 10
`--unknown-option 10` becomes argument to main class.
spark-submit --master foo /my.jar --unknown-option 10
Same thing. The issue you point out, and that's true, is that if
`--unknown-option` is added as a SparkSubmit option, it won't be passed to the
app anymore. In that case, and that case alone, the command line must be
re-written as either of:
spark-submit /my.jar --master foo -- --unknown-option 10
spark-submit --master foo /my.jar -- --unknown-option 10
Note that it doesn't require anyone except the user who wants his
application to still get `--unknown-option` as an argument to do anything.
Basically, don't change any of the existing code; just add a case that
handles "--" to stop parsing SparkSubmit options. That should be very, very
simple to do. Could be as simple as adding:
case "--" :: tail =>
childArgs = tail
(Although that's obviously completely untested.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
