Roland Johann created SPARK-28627:
-------------------------------------
Summary: properties file via SPARK_CONF_DIR and --properties-file
prevents definition of own properties via secrets/own mounts
Key: SPARK-28627
URL: https://issues.apache.org/jira/browse/SPARK-28627
Project: Spark
Issue Type: Bug
Components: Kubernetes
Affects Versions: 2.4.3
Reporter: Roland Johann
K8S resource manager dumps the config map to
/opt/spark/conf/spark-defaults.conf and passes it to spark submit twice:
* via env var SPARK_CONF_DIR
* argument --properties-file
This prevents definition if user defined properties, such as from secrets -
currently it seems to be the only possibility to define spark config properties
from secrets but is locked currently.
Is this implementation intent? If yes, what are the reasons behind this
decision?
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
