GitHub user GregOwen opened a pull request:
https://github.com/apache/spark/pull/22174
[SPARK-22779] Fallback config defaults should behave like scalar defaults
## What changes were proposed in this pull request?
The value returned by a call to `spark.conf.get(key)` may be specified in
one of 3 different ways:
1. By explicitly setting the value of `key` via e.g. `spark.conf.set(key,
value)`
1. By the client providing a default value at call-time via e.g.
`spark.conf.get(key, clientDefault)`
1. By the ConfigBuilder providing a default value when it is declared via
e.g. `SQLConf.buildConf(key).createWithDefault(builderDefault)`
Currently, the order of precedence among these different sources of value
is:
1. Explicit value if one was set
1. Otherwise, client default if one was provided
1. Otherwise, builder default
There is currently one exception to this rule: if the builder happened to
be declared with a default that is another ConfigBuilder (via e.g.
`SQLConf.buildConf(key).fallbackConf(parentConfigBuilder)`), then we check the
builder default first (in this case the parent conf and potentially the
parent's parent and so on) and then only use the client default if none of the
fallback confs had explicit values defined.
From a user perspective, this means that the client default may or may not
be used depending on whether the config in question was declared with a scalar
default or a fallback conf default, which is usually not obvious. In the case
of a fallback conf default, the value returned by `spark.conf.get` may depend
on an arbitrary number of parent confs, which seems like it will lead to
particularly difficult to diagnose errors.
## How was this patch tested?
Unit test to verify the precedence order for confs with both scalar and
fallback conf defaults.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/GregOwen/apache-spark SPARK-22779
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22174.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22174
----
commit ca4747981f67162d3f937bc83528bcd82431d8c2
Author: Greg Owen <greg@...>
Date: 2018-08-21T18:34:31Z
Client-provided defaults take precedence over ConfigBuilder defaults
commit 28ea2303982763c328b69a71e03ad7c12fa5eb4c
Author: Greg Owen <greg@...>
Date: 2018-08-21T18:36:00Z
moved comment
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
