GitHub user MaxGekk opened a pull request:
https://github.com/apache/spark/pull/22489
[SPARK-25425][SQL][BACKPORT-2.3] Extra options should override session
options in DataSource V2
## What changes were proposed in this pull request?
In the PR, I propose overriding session options by extra options in
DataSource V2. Extra options are more specific and set via `.option()`, and
should overwrite more generic session options.
## How was this patch tested?
Added tests for read and write paths.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/MaxGekk/spark-1 session-options-2.3
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22489.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22489
----
commit 0cc174160bf04bc97de62c438d571b27224c53a0
Author: Maxim Gekk <maxim.gekk@...>
Date: 2018-09-16T00:24:11Z
[SPARK-25425][SQL] Extra options should override session options in
DataSource V2
In the PR, I propose overriding session options by extra options in
DataSource V2. Extra options are more specific and set via `.option()`, and
should overwrite more generic session options. Entries from seconds map
overwrites entries with the same key from the first map, for example:
```Scala
scala> Map("option" -> false) ++ Map("option" -> true)
res0: scala.collection.immutable.Map[String,Boolean] = Map(option -> true)
```
Added a test for checking which option is propagated to a data source in
`load()`.
Closes #22413 from MaxGekk/session-options.
Lead-authored-by: Maxim Gekk <[email protected]>
Co-authored-by: Dongjoon Hyun <[email protected]>
Co-authored-by: Maxim Gekk <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
commit 744c5b8b0c8fbc7caa6d03902ed323df5c73986b
Author: Maxim Gekk <maxim.gekk@...>
Date: 2018-09-19T19:40:25Z
Adding missing import after merge to 2.4
commit c87a6de4a1de82514fad590c68f35fbfaf54d825
Author: Maxim Gekk <maxim.gekk@...>
Date: 2018-09-19T20:20:21Z
Fix merge
commit f8b5aa6b3d1fc9e953bcfef8f23aeb6f3ab7818d
Author: Maxim Gekk <maxim.gekk@...>
Date: 2018-09-20T09:47:57Z
Adding options to Reader
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
