[
https://issues.apache.org/jira/browse/SPARK-3620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dale Richardson updated SPARK-3620:
---
Description:
I'm proposing its time to refactor the configuration argument handling code in
spark-submit. The code has grown organically in a short period of time, handles
a pretty complicated logic flow, and is now pretty fragile. Some issues that
have been identified:
1. Hand-crafted property file readers that do not support the property file
format as specified in
http://docs.oracle.com/javase/6/docs/api/java/util/Properties.html#load(java.io.Reader)
2. ResolveURI not called on paths read from conf/prop files
3. inconsistent means of merging / overriding values from different sources
(Some get overridden by file, others by manual settings of field on object,
Some by properties)
4. Argument validation should be done after combining config files, system
properties and command line arguments,
5. Alternate conf file location not handled in shell scripts
6. Some options can only be passed as command line arguments
7. Defaults for options are hard-coded (and sometimes overridden multiple
times) in many through-out the code e.g. master = local[*]
Initial proposal is to use typesafe conf to read in the config information and
merge the various config sources
was:
I'm proposing its time to refactor the configuration argument handling code in
spark-submit. The code has grown organically in a short period of time, handles
a pretty complicated logic flow, and is now pretty fragile. Some issues that
have been identified:
1. Hand-crafted property file readers that do not support the property file
format as specified in
http://docs.oracle.com/javase/6/docs/api/java/util/Properties.html#load(java.io.Reader)
2. ResolveURI not called on paths read from conf/prop files
3. inconsistent means of merging / overriding values from different sources
(Some get overridden by file, others by manual settings of field on object,
Some by properties)
4. Argument validation should be done after combining config files, system
properties and command line arguments,
5. Alternate conf file location not handled in shell scripts
6. Some options can only be passed as command line arguments
7. Defaults for options are hard-coded (and sometimes overridden multiple
times) in many through-out the code e.g. master = local[*]
Refactor config option handling code for spark-submit
-
Key: SPARK-3620
URL: https://issues.apache.org/jira/browse/SPARK-3620
Project: Spark
Issue Type: Improvement
Components: Deploy
Affects Versions: 1.0.0, 1.1.0
Reporter: Dale Richardson
Assignee: Dale Richardson
Priority: Minor
I'm proposing its time to refactor the configuration argument handling code
in spark-submit. The code has grown organically in a short period of time,
handles a pretty complicated logic flow, and is now pretty fragile. Some
issues that have been identified:
1. Hand-crafted property file readers that do not support the property file
format as specified in
http://docs.oracle.com/javase/6/docs/api/java/util/Properties.html#load(java.io.Reader)
2. ResolveURI not called on paths read from conf/prop files
3. inconsistent means of merging / overriding values from different sources
(Some get overridden by file, others by manual settings of field on object,
Some by properties)
4. Argument validation should be done after combining config files, system
properties and command line arguments,
5. Alternate conf file location not handled in shell scripts
6. Some options can only be passed as command line arguments
7. Defaults for options are hard-coded (and sometimes overridden multiple
times) in many through-out the code e.g. master = local[*]
Initial proposal is to use typesafe conf to read in the config information
and merge the various config sources
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
