GitHub user MaxGekk opened a pull request:
https://github.com/apache/spark/pull/23257
[SPARK-26310][SQL] Verify applicability of JSON options
## What changes were proposed in this pull request?
In the PR, I propose additional verification of JSON options. In
particular, detection of using `read` options in `write` and vice versa. The
verification can be disabled by new SQL config
`spark.sql.verifyDataSourceOptions` (`true` by default). This changes should
prevent misusing of JSON options, and improve usage experience of built-in
datasource.
## How was this patch tested?
Added new test to `JacksonGeneratorSuite` to check how `in read` option is
handled if it is passed `in write`. Also the test checks new SQL config can
disable the verification.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/MaxGekk/spark-1 sep-options-in-save-load
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/23257.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #23257
----
commit edbe0b7e7f29631f6b92255a7b561c916a6c39cc
Author: Maxim Gekk <maxim.gekk@...>
Date: 2018-12-07T17:00:11Z
Introducing JSONOptionsInWrite
commit af1507093f42a206c2c22f6c40cf4c43290244b8
Author: Maxim Gekk <maxim.gekk@...>
Date: 2018-12-07T20:34:24Z
Test for options verification
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
