Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3409#discussion_r21125348
--- Diff:
yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala ---
@@ -360,6 +360,10 @@ private[spark] trait ClientBase extends Logging {
}
}
+ // include yarn am specific java options
+ sparkConf.getOption("spark.yarn.am.extraJavaOptions")
+ .foreach(opts => javaOpts += opts)
--- End diff --
Just FYI, this works because of how this is used in Yarn. Yarn doesn't just
do a `System.exec()`, it creates a shell script that then executes this
command. So this string gets added to the command as is, so breaking it up just
means it will be rebuilt the same way again in the shell script later on.
That being said, yeah, it's kinda fishy to depend on Yarn's internal
behavior like that, so it's probably safer to change this (and the other place
Zhan identified).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
