Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/4188#issuecomment-73316977
Hi @srowen,
I think this isn't the correct fix. I think the proper fix would be to
change this in `Client.scala`:
sparkConf.getOption("spark.driver.extraJavaOptions")
.orElse(sys.env.get("SPARK_JAVA_OPTS"))
.map(Utils.splitCommandString).getOrElse(Seq.empty)
.foreach(opts => javaOpts += opts)
Into this:
val opts sparkConf.getOption("spark.driver.extraJavaOptions")
.orElse(sys.env.get("SPARK_JAVA_OPTS"))
.map(Utils.splitCommandString)
.foreach(opt => javaOpts += YarnSparkHadoopUtil.escapeForShell(opt))
`ExecutorRunnable.scala` has similar code, but it just treats the command
line as a string instead of a list, so it doesn't need to parse the options
into a list then escape them back when building the command line.
`Utils.splitCommandString` is currently doing what it's supposed to do;
parsing a command line into a list of individual arguments that can be fed to
`Runtime.exec()`, for example. So it has to strip quotes that the shell would
interpret, otherwise you break code that does that (e.g. I believe standalone
mode uses `ProcessBuilder` to run executors).
`YarnSparkHadoopUtil.escapeForShell` escapes individual arguments back into
something that a shell would interpret as a single argument. This is needed
since Yarn launches containers by creating a shell script to do the work,
instead of using `ProcessBuilder`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
