Dale Richardson created SPARK-3620: -------------------------------------- Summary: Refactor parameter handling code for spark-submit Key: SPARK-3620 URL: https://issues.apache.org/jira/browse/SPARK-3620 Project: Spark Issue Type: Improvement Components: Deploy Affects Versions: 1.1.0, 1.0.0 Reporter: Dale Richardson Priority: Minor
I'm proposing its time to refactor the configuration argument handling code in spark-submit. The code has grown organically in a short period of time, handles a pretty complicated logic flow, and is now pretty fragile. Some issues that have been identified: 1. Hand-crafted property file readers that do not support the property file format as specified in http://docs.oracle.com/javase/6/docs/api/java/util/Properties.html#load(java.io.Reader) 2. ResolveURI not called on paths read from conf/prop files 3. inconsistent means of merging / overriding values from different sources (Some get overridden by file, others by manual settings of field on object, Some by properties) 4. Argument validation should be done after combining config files, system properties and command line arguments, 5. Alternate conf file location not handled in shell scripts 6. Some options can only be passed as command line arguments 7. Defaults for options are hard-coded (and sometimes overridden multiple times) in many through-out the code e.g. master = local[*] -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org