Github user lvsoft commented on the pull request:
https://github.com/apache/spark/pull/3490#issuecomment-66404194
Sorry for late reply. I'll explain the use cases for multiple properties
files.
Currently I'm working on a benchmark utility for spark. It'll be nature to
adjust properties for different workloads.
I'd like to setup the configures with two parts: global confs for common
properties, and private confs for each workloads. Without the support of
multiple properties files, I have to merge the properties as a tmp conf file,
and remove it after spark-submit finished. What's more, consider to submit
multiple workloads for multiple times concurrently, the tmp conf file name need
to be mutually exclusive. And if the benchmark processing was interrupted, the
tmp conf files will be hard to clean.
So I think, a more elegant approach is to add the support of multiple
properties files for spark.
Another reason for this PR: currently spark will use `spark-defaults.conf`
if no properties-file specified, or use the specified properties-file and
*discard* `spark-defaults.conf`. This behavior is also counter-intuitive for
beginners. In most systems, it is a natural assumption that the values in
`xxx-defaults.conf` will take effect if the properties is not overrided in
user's config.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
