Xuzhou Yin created SPARK-33761: ---------------------------------- Summary: [k8s] Support fetching driver and executor pod templates from HCFS Key: SPARK-33761 URL: https://issues.apache.org/jira/browse/SPARK-33761 Project: Spark Issue Type: Improvement Components: Kubernetes Affects Versions: 3.0.1 Reporter: Xuzhou Yin
Currently Spark 3 on Kubernetes supports loading driver and executor pod templates from Local file system: [https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala#L87.] However, this is not very convenient as user needs to either bake the pod templates into client pod image or manually mounting the file as configMap. It would be nice if Spark supports loading pod templates from Hadoop Compatible File Systems (such as S3A), so that user can directly update the pod template files in S3 without changing the underline Kubernetes job definition (eg. Updating Docker image or updating ConfigMap) -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org