[
https://issues.apache.org/jira/browse/SPARK-43454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17722640#comment-17722640
]
Snoot.io commented on SPARK-43454:
----------------------------------
User 'advancedxy' has created a pull request for this issue:
https://github.com/apache/spark/pull/41168
> SparkConf's get and getAllWithPrefix support variable substitution
> ------------------------------------------------------------------
>
> Key: SPARK-43454
> URL: https://issues.apache.org/jira/browse/SPARK-43454
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes, Spark Core
> Affects Versions: 3.2.4, 3.3.2, 3.4.0
> Reporter: YE
> Priority: Major
>
> After SPARK-16272:
> variable substitution is available for `ConfigEntry`s, this creates some
> confusion for get conf directly vs get conf from ConfigEntry. For example:
> {code:java}
> // suppose spark.a=${env:var_a} is set in sparkConf, and it's defined as a
> ConfigEntry as CONFIG_ENTRY_A
> // this would return ${env:var_a}
> val a = sparkConf.get("spark.a")
> // this would return the value of var_a in the current JVM process
> val configA = sparkConf.get(CONFIG_ENTRY_A)
> {code}
> I'd like to propose that we should support variable substitution for get
> config directly.
>
> I noticed this problem when running spark applications on k8s, and noticed
> some substitutions are not work as expected. For k8s particularly, it uses
> getAllWithPrefix extensively, which also suffers this problem
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
