Repository: spark
Updated Branches:
refs/heads/branch-1.1 6c532254a -> 5ac55c841
[SPARK-4253] Ignore spark.driver.host in yarn-cluster and standalone-cluster
modes
In yarn-cluster and standalone-cluster modes, we don't know where driver will
run until it is launched. If the `spark.driver.host` property is set on the
submitting machine and propagated to the driver through SparkConf then this
will lead to errors when the driver launches.
This patch fixes this issue by dropping the `spark.driver.host` property in
SparkSubmit when running in a cluster deploy mode.
Author: WangTaoTheTonic <barneystin...@aliyun.com>
Author: WangTao <barneystin...@aliyun.com>
Closes #3112 from WangTaoTheTonic/SPARK4253 and squashes the following commits:
ed1a25c [WangTaoTheTonic] revert unrelated formatting issue
02c4e49 [WangTao] add comment
32a3f3f [WangTaoTheTonic] ingore it in SparkSubmit instead of SparkContext
667cf24 [WangTaoTheTonic] document fix
ff8d5f7 [WangTaoTheTonic] also ignore it in standalone cluster mode
2286e6b [WangTao] ignore spark.driver.host in yarn-cluster mode
(cherry picked from commit 8106b1e36b2c2b9f5dc5d7252540e48cc3fc96d5)
Signed-off-by: Josh Rosen <joshro...@databricks.com>
Conflicts:
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5ac55c84
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/5ac55c84
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/5ac55c84
Branch: refs/heads/branch-1.1
Commit: 5ac55c8419a9063dce97c4050e5acea11cb5ed51
Parents: 6c53225
Author: WangTaoTheTonic <barneystin...@aliyun.com>
Authored: Thu Dec 4 11:52:47 2014 -0800
Committer: Josh Rosen <joshro...@databricks.com>
Committed: Thu Dec 4 12:08:47 2014 -0800
----------------------------------------------------------------------
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala | 5 +++++
.../apache/spark/deploy/yarn/ApplicationMasterArguments.scala | 2 +-
2 files changed, 6 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/5ac55c84/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
index 68308c4..c941d3a 100644
--- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
@@ -284,6 +284,11 @@ object SparkSubmit {
sysProps.getOrElseUpdate(k, v)
}
+ // Ignore spark.driver.host in cluster modes (see SPARK-4253 for context):
+ if (deployMode == CLUSTER) {
+ sysProps -= "spark.driver.host"
+ }
+
(childArgs, childClasspath, sysProps, childMainClass)
}
http://git-wip-us.apache.org/repos/asf/spark/blob/5ac55c84/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMasterArguments.scala
----------------------------------------------------------------------
diff --git
a/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMasterArguments.scala
b/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMasterArguments.scala
index 424b0fb..ca34f5f 100644
---
a/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMasterArguments.scala
+++
b/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMasterArguments.scala
@@ -35,7 +35,7 @@ class ApplicationMasterArguments(val args: Array[String]) {
var args = inputArgs
- while (! args.isEmpty) {
+ while (!args.isEmpty) {
// --num-workers, --worker-memory, and --worker-cores are deprecated
since 1.0,
// the properties with executor in their names are preferred.
args match {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
