Dale Richardson created SPARK-3620:
--------------------------------------
Summary: Refactor parameter handling code for spark-submit
Key: SPARK-3620
URL: https://issues.apache.org/jira/browse/SPARK-3620
Project: Spark
Issue Type: Improvement
Components: Deploy
Affects Versions: 1.1.0, 1.0.0
Reporter: Dale Richardson
Priority: Minor
I'm proposing its time to refactor the configuration argument handling code in
spark-submit. The code has grown organically in a short period of time, handles
a pretty complicated logic flow, and is now pretty fragile. Some issues that
have been identified:
1. Hand-crafted property file readers that do not support the property file
format as specified in
http://docs.oracle.com/javase/6/docs/api/java/util/Properties.html#load(java.io.Reader)
2. ResolveURI not called on paths read from conf/prop files
3. inconsistent means of merging / overriding values from different sources
(Some get overridden by file, others by manual settings of field on object,
Some by properties)
4. Argument validation should be done after combining config files, system
properties and command line arguments,
5. Alternate conf file location not handled in shell scripts
6. Some options can only be passed as command line arguments
7. Defaults for options are hard-coded (and sometimes overridden multiple
times) in many through-out the code e.g. master = local[*]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
