Xuzhou Yin created SPARK-33761:
----------------------------------
Summary: [k8s] Support fetching driver and executor pod templates
from HCFS
Key: SPARK-33761
URL: https://issues.apache.org/jira/browse/SPARK-33761
Project: Spark
Issue Type: Improvement
Components: Kubernetes
Affects Versions: 3.0.1
Reporter: Xuzhou Yin
Currently Spark 3 on Kubernetes supports loading driver and executor pod
templates from Local file system:
[https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala#L87.]
However, this is not very convenient as user needs to either bake the pod
templates into client pod image or manually mounting the file as configMap. It
would be nice if Spark supports loading pod templates from Hadoop Compatible
File Systems (such as S3A), so that user can directly update the pod template
files in S3 without changing the underline Kubernetes job definition (eg.
Updating Docker image or updating ConfigMap)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
