GitHub user MaxGekk opened a pull request:
https://github.com/apache/spark/pull/22474
[SPARK-25425][SQL][BACKPORT-2.4] Extra options should override session
options in DataSource V2
## What changes were proposed in this pull request?
In the PR, I propose overriding session options by extra options in
DataSource V2. Extra options are more specific and set via `.option()`, and
should overwrite more generic session options.
## How was this patch tested?
Added tests for read and write paths.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/MaxGekk/spark-1 session-options-2.4
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22474.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22474
----
commit ac704568591d2c505b396b5c93138ea8132e3cc4
Author: Maxim Gekk <maxim.gekk@...>
Date: 2018-09-16T00:24:11Z
[SPARK-25425][SQL] Extra options should override session options in
DataSource V2
In the PR, I propose overriding session options by extra options in
DataSource V2. Extra options are more specific and set via `.option()`, and
should overwrite more generic session options. Entries from seconds map
overwrites entries with the same key from the first map, for example:
```Scala
scala> Map("option" -> false) ++ Map("option" -> true)
res0: scala.collection.immutable.Map[String,Boolean] = Map(option -> true)
```
Added a test for checking which option is propagated to a data source in
`load()`.
Closes #22413 from MaxGekk/session-options.
Lead-authored-by: Maxim Gekk <[email protected]>
Co-authored-by: Dongjoon Hyun <[email protected]>
Co-authored-by: Maxim Gekk <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
commit bf3a52c1240a14f89949f8b9dcb554090f803250
Author: Maxim Gekk <maxim.gekk@...>
Date: 2018-09-19T19:40:25Z
Adding missing import after merge to 2.4
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
