[ https://issues.apache.org/jira/browse/SPARK-30985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prashant Sharma updated SPARK-30985: ------------------------------------ Description: SPARK_CONF_DIR hosts configuration files like, 1) spark-defaults.conf - containing all the spark properties. 2) log4j.properties - Logger configuration. 3) spark-env.sh - Environment variables to be setup at driver and executor. 4) core-site.xml - Hadoop related configuration. 5) fairscheduler.xml - Spark's fair scheduling policy at the job level. 6) metrics.properties - Spark metrics. 7) Any user specific - library or framework specific configuration file. Traditionally, SPARK_CONF_DIR has been the home to all user specific configuration files. So this feature, will let the user specific configuration files be mounted on the driver and executor pods' SPARK_CONF_DIR. Please review the attached design doc, for more details. https://drive.google.com/file/d/1p6gaJyOJdlB1rosJDFner3bj5VekTCJ3/view?usp=sharing was: SPARK_CONF_DIR hosts configuration files like, 1) spark-defaults.conf - containing all the spark properties. 2) log4j.properties - Logger configuration. 3) spark-env.sh - Environment variables to be setup at driver and executor. 4) core-site.xml - Hadoop related configuration. 5) fairscheduler.xml - Spark's fair scheduling policy at the job level. 6) metrics.properties - Spark metrics. 7) Any user specific - library or framework specific configuration file. Traditionally, SPARK_CONF_DIR has been the home to all user specific configuration files. So this feature, will let the user specific configuration files be mounted on the driver and executor pods' SPARK_CONF_DIR. Please review the attached design doc, for more details. > Propagate SPARK_CONF_DIR files to driver and exec pods. > ------------------------------------------------------- > > Key: SPARK-30985 > URL: https://issues.apache.org/jira/browse/SPARK-30985 > Project: Spark > Issue Type: Improvement > Components: Kubernetes, Spark Core > Affects Versions: 3.1.0 > Reporter: Prashant Sharma > Priority: Major > > SPARK_CONF_DIR hosts configuration files like, > 1) spark-defaults.conf - containing all the spark properties. > 2) log4j.properties - Logger configuration. > 3) spark-env.sh - Environment variables to be setup at driver and executor. > 4) core-site.xml - Hadoop related configuration. > 5) fairscheduler.xml - Spark's fair scheduling policy at the job level. > 6) metrics.properties - Spark metrics. > 7) Any user specific - library or framework specific configuration file. > Traditionally, SPARK_CONF_DIR has been the home to all user specific > configuration files. > So this feature, will let the user specific configuration files be mounted on > the driver and executor pods' SPARK_CONF_DIR. > Please review the attached design doc, for more details. > > https://drive.google.com/file/d/1p6gaJyOJdlB1rosJDFner3bj5VekTCJ3/view?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org