[
https://issues.apache.org/jira/browse/SPARK-32223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prashant Sharma updated SPARK-32223:
------------------------------------
Description:
One of the challenge with this is, spark.properties is not user provided and is
calculated based on certain factors. So a user provided config map, cannot be
used as is to mount as SPARK_CONF_DIR, so it will have to be somehow augmented
with the correct spark.properties.
Q, Do we support update to config map properties for an already running job?
Ans: No, since the spark.properties is calculated at the time of job
submission, it cannot be updated on the fly and it is not supported by Spark at
the moment for all the configuration values.
Q. What are the usecases, where supplying SPARK_CONF_DIR via a config map helps?
One of the use case, I can think of is programmatically submitting a `spark on
k8s` job - e.g. spark as a service on a cloud deployment may find this feature
useful.
was:
One of the challenge with this is, spark.properties is not user provided and is
calculated based on certain factors. So a user provided config map, cannot be
used as is to mount as SPARK_CONF_DIR, so it will have to be somehow augmented
with the correct spark.properties.
Q, Do we support update to config map properties for an already running job?
Ans: No, since the spark.properties is calculated at the time of job
submission, it cannot be updated on the fly and it is not supported by Spark at
the moment for all the configuration values.
Q. What are the usecases, where supplying SPARK_CONF_DIR via a config map helps?
One of the use case, I can think of is programmatically submitting a spark on
k8s job - e.g. spark as a service on cloud deployment may find this feature
useful.
> Support adding a user provided config map.
> ------------------------------------------
>
> Key: SPARK-32223
> URL: https://issues.apache.org/jira/browse/SPARK-32223
> Project: Spark
> Issue Type: Sub-task
> Components: Kubernetes
> Affects Versions: 3.1.0
> Reporter: Prashant Sharma
> Priority: Major
>
> One of the challenge with this is, spark.properties is not user provided and
> is calculated based on certain factors. So a user provided config map, cannot
> be used as is to mount as SPARK_CONF_DIR, so it will have to be somehow
> augmented with the correct spark.properties.
> Q, Do we support update to config map properties for an already running job?
> Ans: No, since the spark.properties is calculated at the time of job
> submission, it cannot be updated on the fly and it is not supported by Spark
> at the moment for all the configuration values.
> Q. What are the usecases, where supplying SPARK_CONF_DIR via a config map
> helps?
> One of the use case, I can think of is programmatically submitting a `spark
> on k8s` job - e.g. spark as a service on a cloud deployment may find this
> feature useful.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
