GitHub user GregOwen opened a pull request: https://github.com/apache/spark/pull/22174
[SPARK-22779] Fallback config defaults should behave like scalar defaults ## What changes were proposed in this pull request? The value returned by a call to `spark.conf.get(key)` may be specified in one of 3 different ways: 1. By explicitly setting the value of `key` via e.g. `spark.conf.set(key, value)` 1. By the client providing a default value at call-time via e.g. `spark.conf.get(key, clientDefault)` 1. By the ConfigBuilder providing a default value when it is declared via e.g. `SQLConf.buildConf(key).createWithDefault(builderDefault)` Currently, the order of precedence among these different sources of value is: 1. Explicit value if one was set 1. Otherwise, client default if one was provided 1. Otherwise, builder default There is currently one exception to this rule: if the builder happened to be declared with a default that is another ConfigBuilder (via e.g. `SQLConf.buildConf(key).fallbackConf(parentConfigBuilder)`), then we check the builder default first (in this case the parent conf and potentially the parent's parent and so on) and then only use the client default if none of the fallback confs had explicit values defined. From a user perspective, this means that the client default may or may not be used depending on whether the config in question was declared with a scalar default or a fallback conf default, which is usually not obvious. In the case of a fallback conf default, the value returned by `spark.conf.get` may depend on an arbitrary number of parent confs, which seems like it will lead to particularly difficult to diagnose errors. ## How was this patch tested? Unit test to verify the precedence order for confs with both scalar and fallback conf defaults. You can merge this pull request into a Git repository by running: $ git pull https://github.com/GregOwen/apache-spark SPARK-22779 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22174.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22174 ---- commit ca4747981f67162d3f937bc83528bcd82431d8c2 Author: Greg Owen <greg@...> Date: 2018-08-21T18:34:31Z Client-provided defaults take precedence over ConfigBuilder defaults commit 28ea2303982763c328b69a71e03ad7c12fa5eb4c Author: Greg Owen <greg@...> Date: 2018-08-21T18:36:00Z moved comment ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org