Jungtaek Lim created SPARK-27575:
------------------------------------
Summary: Spark overwrites existing value of spark.yarn.dist.*
instead of merging value
Key: SPARK-27575
URL: https://issues.apache.org/jira/browse/SPARK-27575
Project: Spark
Issue Type: Bug
Components: Spark Submit, YARN
Affects Versions: 2.4.0
Reporter: Jungtaek Lim
If we specify `--files` arg when submitting app where configuration has
"spark.yarn.dist.files", SparkSubmit overwrites the new files (files provided
in arg) instead of merging existing value in configuration.
Same issue happens also on "spark.yarn.dist.pyFiles", "spark.yarn.dist.jars",
"spark.yarn.dist.archives".
While I encountered the issue in Spark 2.4.0, I can see the issue from codebase
in master branch and also branch-2.3 as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
