advancedxy opened a new pull request, #41196:
URL: https://github.com/apache/spark/pull/41196
### What changes were proposed in this pull request?
1. add a new feature step to handle special environment variables, such as
`PATH=/usr/bin:$PATH`
* pull these special env variables and reorder them in a new config map
* mount the config map in driver and/or executor's container
* export these env variables by leverage sh/bash's `.` capability in the
`entrypoint.sh`
* update original container's env list with special envs excluded
2. support `spark.executor.exraLibraryPath` on K8S by leveraging the new
feature step
3. update `spark-rbac.yaml` to including more permissions, which was found
when add integration tests.
### Why are the changes needed?
1. support `spark.executor.extraLibraryPath` on k8s
2. feature parity with spark on yarn. Spark on yarn handles this environment
variables that need to be substituted correctly.
3. K8S itself cannot substitute variables in the container spec, we have to
do some extra work at the entrypoint.sh
### Does this PR introduce _any_ user-facing change?
Yes. Before this PR, for spark running on K8S,
`spark.executor.extraLibraryPath` doesn't take effect and cannot support
variable substitution.
### How was this patch tested?
1. added new UTs
2. added new integration test
3. [WIP] verifing in the real prod cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
