Repository: spark Updated Branches: refs/heads/master 5f6bdf97c -> eacd9d8ed
[SPARK-13360][PYSPARK][YARN] PYSPARK_DRIVER_PYTHON and PYSPARK_PYTHON⦠⦠is not picked up in yarn-cluster mode Author: Jeff Zhang <[email protected]> Closes #11238 from zjffdu/SPARK-13360. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/eacd9d8e Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/eacd9d8e Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/eacd9d8e Branch: refs/heads/master Commit: eacd9d8eda68260bbda7b0cd07410321dffaf428 Parents: 5f6bdf9 Author: Jeff Zhang <[email protected]> Authored: Wed Mar 16 10:21:01 2016 -0700 Committer: Marcelo Vanzin <[email protected]> Committed: Wed Mar 16 10:21:01 2016 -0700 ---------------------------------------------------------------------- yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala | 3 +++ 1 file changed, 3 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/eacd9d8e/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---------------------------------------------------------------------- diff --git a/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala b/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala index 40cf9b6..f864ef1 100644 --- a/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala +++ b/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala @@ -745,6 +745,9 @@ private[spark] class Client( } env("SPARK_JAVA_OPTS") = value } + // propagate PYSPARK_DRIVER_PYTHON and PYSPARK_PYTHON to driver in cluster mode + sys.env.get("PYSPARK_DRIVER_PYTHON").foreach(env("PYSPARK_DRIVER_PYTHON") = _) + sys.env.get("PYSPARK_PYTHON").foreach(env("PYSPARK_PYTHON") = _) } sys.env.get(ENV_DIST_CLASSPATH).foreach { dcp => --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
