John Zhuge created SPARK-54313:
----------------------------------
Summary: Add --extra-properties-file option for configuration
layering in spark-submit
Key: SPARK-54313
URL: https://issues.apache.org/jira/browse/SPARK-54313
Project: Spark
Issue Type: New Feature
Components: Spark Core
Affects Versions: 4.0.1
Reporter: John Zhuge
This proposal adds support for specifying multiple properties files in
spark-submit through a repeatable {{--extra-properties-file}} option. This
enables configuration layering and composition, similar to SPARK-34345 but a
bit more cleaner.
Large-scale Spark deployments often need to compose configurations from
multiple sources, for example:
* Base/shared defaults - Common settings across all applications
* Environment-specific configs - Dev/staging/prod differences
* Cluster-specific settings - Different clusters, regions, or Kubernetes
namespaces
* Application-specific overrides - Per-job tuning
Currently, we must either merge properties files before submission or rely
solely on {{--conf}} flags (command line can get very long).
Proposed command-line syntax:
{code:bash}
spark-submit \
--properties-file /path/to/base.conf \
--load-spark-defaults \
--extra-properties-file /path/to/override1.conf \
--extra-properties-file /path/to/override2.conf \
--conf spark.executor.memory=4g \
--class com.example.MyApp \
myapp.jar
{code}
Property precedence (highest to lowest):
# {{--conf}} flags (command-line overrides)
# {{--extra-properties-file}} files (in order specified, later overrides
earlier)
# {{--properties-file}} (base file)
# {{conf/spark-defaults.conf}} (when {{\-\-properties-file}} is not specified
or if {{--load-spark-defaults}} is used)
Key behaviors:
- {{--extra-properties-file}} can be specified multiple times
- Files are loaded left-to-right; later values override earlier ones
- Missing files cause validation errors
- Works with all deployment modes (client, cluster, local)
- Compatible with {{--load-spark-defaults}} flag
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
