Repository: spark
Updated Branches:
refs/heads/master 3cd516191 -> 3b764699f
[SPARK-4461][YARN] pass extra java options to yarn application master
Currently, there is no way to pass yarn am specific java options. It cause some
potential issues when reading classpath from hadoop configuration file. Hadoop
configuration actually replace variables in its property with the system
property passed in java options. How to specify the value depends on different
hadoop distribution.
The new options are SPARK_YARN_JAVA_OPTS or spark.yarn.extraJavaOptions. I make
it as spark global level, because typically we don't want user to specify this
in their command line each time submitting spark job after it is setup in
spark-defaults.conf.
In addition, with this new extra options enabled to be passed to AM, it
provides more flexibility.
For example int the following valid mapred-site.xml file, we have the class
path which specify values using system property. Hadoop can correctly handle it
because it has java options passed in.
This is the example, currently spark will break due to hadoop.version is not
passed in.
<property>
<name>mapreduce.application.classpath</name>
<value>/etc/hadoop/${hadoop.version}/mapreduce/*</value>
</property>
In the meantime, we cannot relies on mapreduce.admin.map.child.java.opts in
mapred-site.xml, because it has its own extra java options specified, which
does not apply to Spark.
Author: Zhan Zhang <[email protected]>
Closes #3409 from zhzhan/Spark-4461 and squashes the following commits:
daec3d0 [Zhan Zhang] solve review comments
08f44a7 [Zhan Zhang] add warning in driver mode if
spark.yarn.am.extraJavaOptions is configured
5a505d3 [Zhan Zhang] solve review comments
4ed43ad [Zhan Zhang] solve review comments
ad777ed [Zhan Zhang] Merge branch 'master' into Spark-4461
3e9e574 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
e3f9abe [Zhan Zhang] solve review comments
8963552 [Zhan Zhang] rebase
f8f6700 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
dea1692 [Zhan Zhang] change the option key name to client mode specific
90d5dff [Zhan Zhang] rebase
8ac9254 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
092a25f [Zhan Zhang] solve review comments
bc5a9ae [Zhan Zhang] solve review comments
782b014 [Zhan Zhang] add new configuration to docs/running-on-yarn.md and
remove it from spark-defaults.conf.template
6faaa97 [Zhan Zhang] solve review comments
369863f [Zhan Zhang] clean up unnecessary var
733de9c [Zhan Zhang] Merge branch 'master' into Spark-4461
a68e7f0 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
864505a [Zhan Zhang] Add extra java options to be passed to Yarn application
master
15830fc [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
685d911 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
03ebad3 [Zhan Zhang] Merge branch 'master' of https://github.com/zhzhan/spark
46d9e3d [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
ebb213a [Zhan Zhang] revert
b983ef3 [Zhan Zhang] test
c4efb9b [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
779d67b [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
4daae6d [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
12e1be5 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
ce0ca7b [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
93f3081 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
3764505 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
a9d372b [Zhan Zhang] Merge branch 'master' of https://github.com/zhzhan/spark
a00f60f [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
497b0f4 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
4a2e36d [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
a72c0d4 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
301eb4a [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
cedcc6f [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
adf4924 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
d10bf00 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
7e0cc36 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
68deb11 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
3ee3b2b [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
2b0d513 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
1ccd7cc [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
af9feb9 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
e4c1982 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
921e914 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
789ea21 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
cb53a2c [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark
f6a8a40 [Zhan Zhang] revert
ba14f28 [Zhan Zhang] test
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/3b764699
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/3b764699
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/3b764699
Branch: refs/heads/master
Commit: 3b764699ffc9c74b9597c855a0e8c04ac24fa3b7
Parents: 3cd5161
Author: Zhan Zhang <[email protected]>
Authored: Thu Dec 18 10:01:46 2014 -0600
Committer: Thomas Graves <[email protected]>
Committed: Thu Dec 18 10:01:46 2014 -0600
----------------------------------------------------------------------
docs/running-on-yarn.md | 8 ++++++++
.../apache/spark/deploy/yarn/ClientBase.scala | 19 +++++++++++++++++++
2 files changed, 27 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/3b764699/docs/running-on-yarn.md
----------------------------------------------------------------------
diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md
index 62b3171..b5fb077 100644
--- a/docs/running-on-yarn.md
+++ b/docs/running-on-yarn.md
@@ -139,6 +139,14 @@ Most of the configs are the same for Spark on YARN as for
other deployment modes
The maximum number of threads to use in the application master for
launching executor containers.
</td>
</tr>
+<tr>
+ <td><code>spark.yarn.am.extraJavaOptions</code></td>
+ <td>(none)</td>
+ <td>
+ A string of extra JVM options to pass to the Yarn ApplicationMaster in
client mode.
+ In cluster mode, use spark.driver.extraJavaOptions instead.
+ </td>
+</tr>
</table>
# Launching Spark on YARN
http://git-wip-us.apache.org/repos/asf/spark/blob/3b764699/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala
----------------------------------------------------------------------
diff --git a/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala
b/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala
index f95d723..5f0c67f 100644
--- a/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala
+++ b/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala
@@ -352,12 +352,31 @@ private[spark] trait ClientBase extends Logging {
if (isLaunchingDriver) {
sparkConf.getOption("spark.driver.extraJavaOptions")
.orElse(sys.env.get("SPARK_JAVA_OPTS"))
+ .map(Utils.splitCommandString).getOrElse(Seq.empty)
.foreach(opts => javaOpts += opts)
val libraryPaths = Seq(sys.props.get("spark.driver.extraLibraryPath"),
sys.props.get("spark.driver.libraryPath")).flatten
if (libraryPaths.nonEmpty) {
prefixEnv = Some(Utils.libraryPathEnvPrefix(libraryPaths))
}
+ if (sparkConf.getOption("spark.yarn.am.extraJavaOptions").isDefined) {
+ logWarning("spark.yarn.am.extraJavaOptions will not take effect in
cluster mode")
+ }
+ } else {
+ // Validate and include yarn am specific java options in yarn-client
mode.
+ val amOptsKey = "spark.yarn.am.extraJavaOptions"
+ val amOpts = sparkConf.getOption(amOptsKey)
+ amOpts.foreach { opts =>
+ if (opts.contains("-Dspark")) {
+ val msg = s"$amOptsKey is not allowed to set Spark options (was
'$opts'). "
+ throw new SparkException(msg)
+ }
+ if (opts.contains("-Xmx") || opts.contains("-Xms")) {
+ val msg = s"$amOptsKey is not allowed to alter memory settings (was
'$opts')."
+ throw new SparkException(msg)
+ }
+ javaOpts ++= Utils.splitCommandString(opts)
+ }
}
// For log4j configuration to reference
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
