GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/20799
[SPARK-23635][YARN] AM env variable should not overwrite same name env
variable set through spark.executorEnv.
## What changes were proposed in this pull request?
In the current Spark on YARN code, AM always will copy and overwrite its
env variables to executors, so we cannot set different values for executors.
To reproduce issue, user could start spark-shell like:
```
./bin/spark-shell --master yarn-client --conf
spark.executorEnv.SPARK_ABC=executor_val --conf
spark.yarn.appMasterEnv.SPARK_ABC=am_val
```
Then check executor env variables by
```
sc.parallelize(1 to 1).flatMap \{ i => sys.env.toSeq
}.collect.foreach(println)
```
We will always get `am_val` instead of `executor_val`. So we should not let
AM to overwrite specifically set executor env variables.
## How was this patch tested?
Added UT and tested in local cluster.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jerryshao/apache-spark SPARK-23635
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20799.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20799
----
commit afd88e1a6585851f307413fd39a72167b9d8224c
Author: jerryshao <sshao@...>
Date: 2018-03-12T03:21:40Z
AM env variable should not overwrite same name env variable set through
spark.executorEnv.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
