[
https://issues.apache.org/jira/browse/SPARK-5033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen updated SPARK-5033:
-----------------------------
Component/s: Windows
> Spark 1.1.0/1.1.1/1.2.0 can't run well in HDP on Windows
> --------------------------------------------------------
>
> Key: SPARK-5033
> URL: https://issues.apache.org/jira/browse/SPARK-5033
> Project: Spark
> Issue Type: Bug
> Components: Windows, YARN
> Affects Versions: 1.1.0, 1.2.0
> Environment: HDInsight 3.1 in Azure
> Reporter: Rice
> Labels: easyfix
>
> After installation, when I ran .\bin\spark-shell --master yarn, YarnClient
> will report Error in running commands like the followings:
> %JAVA_HOME%/bin/java -server -cp
> %CLASSPATH%;C:\hdp\spark-1.1.1\lib\spark-assembly-1.1.1-hadoop2.4.0.jar
> -Xmx512m -Djava.io.tmpdir=%PWD%/tmp
> '-Dspark.tachyonStore.folderName=spark-919783cd-bdf7-4e6b-86bf-011244e4a49f'
> '-Dspark.yarn.secondary.jars='
> '-Dspark.repl.class.uri=http://192.168.0.13:12972'
> '-Dspark.driver.host=HOME-HYPERVS' '-Dspark.driver.appUIHistoryAddress='
> '-Dspark.app.name=Spark shell'
> '-Dspark.driver.appUIAddress=HOME-HYPERVS:4040' '-Dspark.jars='
> '-Dspark.fileserver.uri=http://192.168.0.13:12992'
> '-Dspark.master=yarn-client' '-Dspark.driver.port=12988'
> org.apache.spark.deploy.yarn.ExecutorLauncher --class 'notused' --jar null
> --arg 'HOME-HYPERVS:12988' --executor-memory 1024 --executor-cores 1
> --num-executors 2
> It will run error because of single quote instead of double quote. The
> following file will need to be modified in File YarnSparkHadoopUtil.scala:
> def escapeForShell(arg: String): String = {
> if (arg != null) {
> val escaped = new StringBuilder("'")
> for (i <- 0 to arg.length() - 1) {
> arg.charAt(i) match {
> case '$' => escaped.append("\\$")
> case '"' => escaped.append("\\\"")
> case '\'' => escaped.append("'\\''")
> case c => escaped.append(c)
> }
> }
> escaped.append("'").toString()
> } else {
> arg
> }
> }
> After modification from single quote to doulbe quote, the command is OK.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
