Nicholas Chammas created SPARK-17976:

             Summary: Global options to spark-submit should not be 
                 Key: SPARK-17976
             Project: Spark
          Issue Type: Improvement
          Components: Deploy
    Affects Versions: 2.0.1, 2.0.0
            Reporter: Nicholas Chammas
            Priority: Minor

It is maddening that this does what you expect:

spark-submit --packages graphframes:graphframes:0.2.0-spark2.0-s_2.11 \ 

whereas this doesn't because {{--packages}} is totally ignored:

spark-submit \
    --packages graphframes:graphframes:0.2.0-spark2.0-s_2.11

Ideally, global options should be valid no matter where they are specified.

If that's too much work, then I think at the very least {{spark-submit}} should 
display a warning that some input is being ignored. (Ideally, it should error 
out, but that's probably not possible for backwards-compatibility reasons at 
this point.)

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to