YE created SPARK-43454:
--------------------------
Summary: SparkConf's get and getAllWithPrefix support variable
substitution
Key: SPARK-43454
URL: https://issues.apache.org/jira/browse/SPARK-43454
Project: Spark
Issue Type: Improvement
Components: Kubernetes, Spark Core
Affects Versions: 3.4.0, 3.3.2, 3.2.4
Reporter: YE
After SPARK-16272:
variable substitution is available for `ConfigEntry`s, this creates some
confusion for get conf directly vs get conf from ConfigEntry. For example:
{code:java}
// suppose spark.a=${env:var_a} is set in sparkConf, and it's defined as a
ConfigEntry as CONFIG_ENTRY_A
// this would return ${env:var_a}
val a = sparkConf.get("spark.a")
// this would return the value of var_a in the current JVM process
val configA = sparkConf.get(CONFIG_ENTRY_A)
{code}
I'd like to propose that we should support variable substitution for get config
directly.
I noticed this problem when running spark applications on k8s, and noticed some
substitutions are not work as expected. For k8s particularly, it uses
getAllWithPrefix extensively, which also suffers this problem
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
