GitHub user rvesse opened a pull request:
https://github.com/apache/spark/pull/23017
[WIP][SPARK-26015][K8S] Set a default UID for Spark on K8S Images
## What changes were proposed in this pull request?
Adds USER directives to the Dockerfiles which is configurable via build
argument (`spark_uid`) for easy customisation. A `-u` flag is added to
`bin/docker-image-tool.sh` to make it easy to customise this e.g.
```
> bin/docker-image-tool.sh -r rvesse -t uid -u 185 build
> bin/docker-image-tool.sh -r rvesse -t uid push
```
If no UID is explicitly specified it defaults to `185` - this is per
@skonto's suggestion to align with the OpenShift standard reserved UID for Java
apps (
https://lists.openshift.redhat.com/openshift-archives/users/2016-March/msg00283.html)
Notes:
- We have to make the `WORKDIR` writable by the root group or otherwise
jobs will fail with `AccessDeniedException`
- We have to specify `HOME` to set it to the `WORKDIR` to avoid an Ivy
error caused by `user.home` not having a valid value otherwise
To Do:
- [ ] Debug and resolve issue with client mode test
- [ ] Consider whether to always propagate `SPARK_USER_NAME` to environment
of driver and executor pods so `entrypoint.sh` can insert that into
`/etc/passwd` entry
- [ ] Rebase once PR #23013 is merged and update documentation accordingly
## How was this patch tested?
Built the Docker images with the new Dockerfiles that include the `USER`
directives. Ran the Spark on K8S integration tests against the new images.
All pass except client mode which I am currently debugging further.
Also manually dropped myself into the resulting container images via
`docker run` and checked `id -u` output to see that UID is as expected.
Tried customising the UID from the default via the new `-u` argument to
`docker-image-tool.sh` and again checked the resulting image for the correct
runtime UID.
cc @felixcheung @skonto @vanzin
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rvesse/spark SPARK-26015
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/23017.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #23017
----
commit 92e22f76bbe61035dcde54c5fb2990cb7099a98d
Author: Rob Vesse <rvesse@...>
Date: 2018-11-12T13:44:03Z
[SPARK-26015][K8S] Set a default UID for Spark on K8S Images
Adds USER directives to the Dockerfiles which is configurable via build
argument for easy customisation. A -u flag is added to
bin/docker-image-tool.sh to make it easy to customise this.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
