Marcelo Vanzin commented on SPARK-17976:

They are not being ignored. They are being passed as arguments to "file.py".

A long time ago it was decided that the "resource" (i.e. the jar file or python 
file) would separate Spark options from application options. This was chosen 
for backwards compatibility; another option would be to use an explicit 
separator (e.g. "\-\-") but that would not be compatible with existing user 

So unless you have suggestion on how to differentiate Spark options from app 
options without the need for an explicit separator, this should probably be 

> Global options to spark-submit should not be position-sensitive
> ---------------------------------------------------------------
>                 Key: SPARK-17976
>                 URL: https://issues.apache.org/jira/browse/SPARK-17976
>             Project: Spark
>          Issue Type: Improvement
>          Components: Deploy
>    Affects Versions: 2.0.0, 2.0.1
>            Reporter: Nicholas Chammas
>            Priority: Minor
> It is maddening that this does what you expect:
> {code}
> spark-submit --packages graphframes:graphframes:0.2.0-spark2.0-s_2.11 \
>     file.py 
> {code}
> whereas this doesn't because {{--packages}} is totally ignored:
> {code}
> spark-submit file.py \
>     --packages graphframes:graphframes:0.2.0-spark2.0-s_2.11
> {code}
> Ideally, global options should be valid no matter where they are specified.
> If that's too much work, then I think at the very least {{spark-submit}} 
> should display a warning that some input is being ignored. (Ideally, it 
> should error out, but that's probably not possible for 
> backwards-compatibility reasons at this point.)

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to