advancedxy commented on PR #31428: URL: https://github.com/apache/spark/pull/31428#issuecomment-1552531182
Hi, @dongjoon-hyun and @ScrapCodes, When I was developing https://github.com/apache/spark/pull/41196, I initially thought it may be desirable to reuse the `SPARK_CONF_DIR` config map, but it seems a bit of complex and therefore create a new ConfigMap instead. However after some digging and found some limitation exposed by K8S: 1. config map is an object, it may has quota limitation and add burdens to K8S's api server and etcd 2. for access control/security reasons, it may not be possible to create new objects from driver pod(such as the cases in this PR) Therefore I think there should be some rules when developing feature steps for spark on K8S regarding config maps: 1. the config map should be created as less as possible, unless it's unavoidable, such as a new HADOOP_CONF map 2. the config map should only be created on the client(spark-submit) side, not from the driver pod. I'd like to make a proposal here: 1. create SPARK_CONF config map on the client side, records the config map's name, the driver pod should reuse this config map to mount SPARK_CONF on the executor side, rather creating a new ConfigMap. Therefore the driver and executor shares the same SPARK_CONF mount. Also, this makes `spark.kubernetes.executor.disableConfigMap` unnecessary 2. extends KubernetesFeatureConfigStep to add a method `getAdditionalDataFileForSparkConfConfigMap` (an example one), to allow other feature steps to dynamic add data files when building driver pod. By leveraging this method, `PodTemplateConfigMapStep` could eliminate the need to create a new config map. New feature steps could also benefit from this method. WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
