GitHub user skonto opened a pull request:
https://github.com/apache/spark/pull/21317
[SPARK-24232][Kubernetes] Add support for secret env vars
## What changes were proposed in this pull request?
* Allows to refer a secret as an env var.
* Introduces new properties in the form described in the ticket.
* Updates docs.
* Adds required unit tests.
## How was this patch tested?
Manually tested and confirmed that the secrets exist in driver's and
executor's container env.
First created a secret with the following yaml:
```
apiVersion: v1
kind: Secret
metadata:
name: test-secret
data:
username: c3RhdnJvcwo=
password: Mzk1MjgkdmRnN0pi
-------
$ echo -n 'stavros' | base64
c3RhdnJvcw==
$ echo -n '1f2d1e2e67df' | base64
MWYyZDFlMmU2N2Rm
```
Run a job as follows:
```./bin/spark-submit \
--master k8s://http://localhost:9000 \
--deploy-mode cluster \
--name spark-pi \
--class org.apache.spark.examples.SparkPi \
--conf spark.executor.instances=1 \
--conf spark.kubernetes.container.image=skonto/spark:k8envs2 \
--conf spark.kubernetes.driver.secretKeyRef.test-secret=username \
--conf spark.kubernetes.executor.secretKeyRef.test-secret=username \
--conf spark.kubernetes.container.image.pullPolicy=Always \
local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0-SNAPSHOT.jar 10000
```
Secret loaded correctly at the driver container:
Also if I log into the exec container:
```
kubectl exec -it spark-pi-1526265507716-exec-1 bash
bash-4.4# env
SPARK_EXECUTOR_MEMORY=1g
SPARK_EXECUTOR_CORES=1
LANG=C.UTF-8
HOSTNAME=spark-pi-1526265507716-exec-1
SPARK_APPLICATION_ID=spark-application-1526265518652
JAVA_HOME=/usr/lib/jvm/java-1.8-openjdk
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_ADDR=10.100.0.1
**TEST-SECRET.USERNAME=stavros**
...
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/skonto/spark k8s-fix-env-secrets
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21317.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21317
----
commit 24832c55e42cd768c191ba895ba054bd44d5905c
Author: Stavros Kontopoulos <stavros.kontopoulos@...>
Date: 2018-05-14T02:41:20Z
add support for secret env vars
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
