[
https://issues.apache.org/jira/browse/SPARK-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-2221.
------------------------------
Resolution: Duplicate
Same issue -- quoting and Windows shell. Resolving in favor of the other since
it has more discussion
> Spark fails on windows YARN (HortonWorks 2.4)
> ---------------------------------------------
>
> Key: SPARK-2221
> URL: https://issues.apache.org/jira/browse/SPARK-2221
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.0.0
> Environment: Windows 2008 R2
> HortonWorks Hadoop cluster 2.4
> Spark 1.0
> Reporter: Vlad Rabenok
>
> Windows based Yarn cluster failed to execute submitted Spark job.
> Can be reproduced by submitting JavaSparkPi on Yarn cluster.
> C:\hdp\spark-1.0.0-bin-hadoop2\bin>spark-submit --class
> org.apache.spark.examples.JavaSparkPi
> ./../lib/spark-examples-1.0.0-hadoop2.2.0.jar --master yarn-cluster
> --deploy-mode cluster
> The origin of the problems is
> org.apache.spark.deploy.yarn.ExecutorRunnableUtil.scala. JavaOpts parameters
> that contain environment variable need to be quoted.
> - '%' should be escaped like "kill %%p" . Still doesn't have any sense on
> Windows. Ideally should be passed through "spark.executor.extraJavaOptions".
> - javaOpts += "-Djava.io.tmpdir=\"" + new Path(Environment.PWD.$(),
> YarnConfiguration.DEFAULT_CONTAINER_TEMP_DIR) + "\""
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
