[jira] [Created] (SPARK-1894) The default ec2 set-up ignores --driver-class-path

[Andrew Or (JIRA)]

Andrew Or created SPARK-1894:
--------------------------------

             Summary: The default ec2 set-up ignores --driver-class-path
                 Key: SPARK-1894
                 URL: https://issues.apache.org/jira/browse/SPARK-1894
             Project: Spark
          Issue Type: Bug
    Affects Versions: 1.0.0
            Reporter: Andrew Or
             Fix For: 1.0.1


If the user sets up an ec2 cluster, the scripts automatically add the following 
lines to conf/spark-env.sh

<code>
export SPARK_SUBMIT_LIBRARY_PATH="/root/ephemeral-hdfs/lib/native/"
export SPARK_SUBMIT_CLASSPATH=$SPARK_CLASSPATH:/root/ephemeral-hdfs/conf"
<code>

Unfortunately, these variables are exported after spark-submit parses the 
--driver-* flags, which also set these variables. As a result, all values set 
via the --driver-* flags get overridden.

The simple fix is to append to, instead of overwrite, these variables.



--
This message was sent by Atlassian JIRA
(v6.2#6252)