cnauroth opened a new pull request, #48216:
URL: https://github.com/apache/spark/pull/48216
…rride in app master
### What changes were proposed in this pull request?
This patch corrects handling of a user-supplied `SPARK_USER` environment
variable in the YARN app master. Currently, the user-supplied value gets
appended to the default, like a classpath entry. The patch fixes it by using
only the user-supplied value.
### Why are the changes needed?
Overriding the `SPARK_USER` environment variable in the YARN app master with
configuration property `spark.yarn.appMasterEnv.SPARK_USER` currently results
in an incorrect value. `Client#setupLaunchEnv` first sets a default in the
environment map using the Hadoop user. After that,
`YarnSparkHadoopUtil.addPathToEnvironment` sees the existing value in the map
and interprets the user-supplied value as needing to be appended like a
classpath entry. The end result is the Hadoop user appended with the classpath
delimiter and user-supplied value, e.g. `cnauroth:overrideuser`.
### Does this PR introduce _any_ user-facing change?
Yes, the app master now uses the user-supplied `SPARK_USER` if specified.
(The default is still the Hadoop user.)
### How was this patch tested?
* Existing unit tests pass.
* Added new unit tests covering default and overridden `SPARK_USER` for the
app master. The override test fails without this patch, and then passes after
the patch is applied.
* Manually tested in a live YARN cluster as shown below.
Manual testing used the `DFSReadWriteTest` job with overrides of
`SPARK_USER`:
```
spark-submit \
--deploy-mode cluster \
--files all-lines.txt \
--class org.apache.spark.examples.DFSReadWriteTest \
--conf spark.yarn.appMasterEnv.SPARK_USER=sparkuser_appMaster \
--conf spark.driverEnv.SPARK_USER=sparkuser_driver \
--conf spark.executorEnv.SPARK_USER=sparkuser_executor \
/usr/lib/spark/examples/jars/spark-examples.jar \
all-lines.txt /tmp/DFSReadWriteTest
```
Before the patch, we can see the app master's `SPARK_USER` mishandled by
looking at the `_SUCCESS` file in HDFS:
```
hdfs dfs -ls -R /tmp/DFSReadWriteTest
drwxr-xr-x - cnauroth:sparkuser_appMaster hadoop 0 2024-09-20
23:35 /tmp/DFSReadWriteTest/dfs_read_write_test
-rw-r--r-- 1 cnauroth:sparkuser_appMaster hadoop 0 2024-09-20
23:35 /tmp/DFSReadWriteTest/dfs_read_write_test/_SUCCESS
-rw-r--r-- 1 sparkuser_executor hadoop 2295080
2024-09-20 23:35 /tmp/DFSReadWriteTest/dfs_read_write_test/part-00000
-rw-r--r-- 1 sparkuser_executor hadoop 2288718
2024-09-20 23:35 /tmp/DFSReadWriteTest/dfs_read_write_test/part-00001
```
After the patch, we can see it working correctly:
```
hdfs dfs -ls -R /tmp/DFSReadWriteTest
drwxr-xr-x - sparkuser_appMaster hadoop 0 2024-09-23 17:13
/tmp/DFSReadWriteTest/dfs_read_write_test
-rw-r--r-- 1 sparkuser_appMaster hadoop 0 2024-09-23 17:13
/tmp/DFSReadWriteTest/dfs_read_write_test/_SUCCESS
-rw-r--r-- 1 sparkuser_executor hadoop 2295080 2024-09-23 17:13
/tmp/DFSReadWriteTest/dfs_read_write_test/part-00000
-rw-r--r-- 1 sparkuser_executor hadoop 2288718 2024-09-23 17:13
/tmp/DFSReadWriteTest/dfs_read_write_test/part-00001
```
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
