[ https://issues.apache.org/jira/browse/SPARK-8646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14629409#comment-14629409 ]
Lianhui Wang edited comment on SPARK-8646 at 7/16/15 8:25 AM: -------------------------------------------------------------- yes, when i use this command: ./bin/spark-submit ./pi.py yarn-client 10, yarn' client do not upload pyspark.zip, so that can not be worked. i submit a PR that resolve this problem based on master branch. there is some problems on spark-1.4.0 branch because it finds pyspark libraries in sparkSubmit, not in Client. was (Author: lianhuiwang): yes, when i use this command: ./bin/spark-submit ./pi.py yarn-client 10, yarn' client do not upload pyspark.zip, so that can not be worked. i submit a PR that resolve this problem based on master branch. > PySpark does not run on YARN if master not provided in command line > ------------------------------------------------------------------- > > Key: SPARK-8646 > URL: https://issues.apache.org/jira/browse/SPARK-8646 > Project: Spark > Issue Type: Bug > Components: PySpark, YARN > Affects Versions: 1.4.0 > Environment: SPARK_HOME=local/path/to/spark1.4install/dir > also with > SPARK_HOME=local/path/to/spark1.4install/dir > PYTHONPATH=$SPARK_HOME/python/lib > Spark apps are submitted with the command: > $SPARK_HOME/bin/spark-submit outofstock/data_transform.py > hdfs://foe-dev/DEMO_DATA/FACT_POS hdfs:/user/juliet/ex/ yarn-client > data_transform contains a main method, and the rest of the args are parsed in > my own code. > Reporter: Juliet Hougland > Attachments: executor.log, pi-test.log, > spark1.4-SPARK_HOME-set-PYTHONPATH-set.log, > spark1.4-SPARK_HOME-set-inline-HADOOP_CONF_DIR.log, > spark1.4-SPARK_HOME-set.log, spark1.4-verbose.log, verbose-executor.log > > > Running pyspark jobs result in a "no module named pyspark" when run in > yarn-client mode in spark 1.4. > [I believe this JIRA represents the change that introduced this error.| > https://issues.apache.org/jira/browse/SPARK-6869 ] > This does not represent a binary compatible change to spark. Scripts that > worked on previous spark versions (ie comands the use spark-submit) should > continue to work without modification between minor versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org