GitHub user abhisrao opened a pull request:
https://github.com/apache/spark/pull/22536
[SPARK-25500][K8s]Specify configmap and secrets in Spark driver and
executor pods in Kubernetes
## What changes were proposed in this pull request?
This PR introduces support for specifying configmap as spark-configuration.
Using PR #22146, the above functionality can be achieved by passing
template file. However, for spark properties (like log4j.properties,
fairscheduler.xml and metrics.properties), we are proposing this approach as
this is native to other configuration options specifications in spark.
The configmaps has to be pre-created before using this as spark
configuration.
(Please fill in changes proposed in this fix)
Add the following parameters while doing spark-submit
spark.kubernetes.driver.volumes.configMap.<volume-name>.mount.path=<mount
location inside pod>
spark.kubernetes.driver.volumes.configMap.<volume-name>.options.name=<configmap
name>
Using these properties, mount the configmap to spark-driver and
spark-executor pods.
Changes are required in the `KubernetesVolumeSpec` and
`MountVolumesFeatureStep` .
## How was this patch tested?
Manual testing on a k8s cluster. Unit tests added to
`KubernetesVolumeUtilsSuite` and `MountVolumesFeatureStepSuite`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/abhisrao/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22536.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22536
----
commit 54fd540680e26a8a40dff31be42f6605f0707687
Author: RAO,ABHISHEK (ABHISHEK) <abhishek.rao@...>
Date: 2018-09-23T16:22:17Z
First Commit
commit 3258ff63a1aaec2107518c73cd207e5400a1eca5
Author: RAO,ABHISHEK (ABHISHEK) <abhishek.rao@...>
Date: 2018-09-23T17:04:49Z
Documentation update
commit 606c13ea08e6c5b341e78ae99cf9ca2dd57eaa75
Author: RAO,ABHISHEK (ABHISHEK) <abhishek.rao@...>
Date: 2018-09-24T09:39:37Z
Removed changes for secret as this is already supported since Spark 2.3
commit 9f48160cd64255c4b90eadfe1db8aacbb7fd7566
Author: RAO,ABHISHEK (ABHISHEK) <abhishek.rao@...>
Date: 2018-09-24T09:39:59Z
Removed changes for secret as this is already supported since Spark 2.3
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
