GitHub user MaxGekk opened a pull request:
https://github.com/apache/spark/pull/22366
[SPARK-25384][SQL] Removing of spark.sql.fromJsonForceNullableSchema
## What changes were proposed in this pull request?
In the PR, I propose to remove the `spark.sql.fromJsonForceNullableSchema`
flag since disabling it can cause corrupted output. The flag was introduced
only for backward compatibility in minor versions. The PR targets Spark 3.0 in
which the flag can be removed.
## How was this patch tested?
It was tested by `JsonExpressionsSuite`, `JsonFunctionsSuite` and
`JsonSuite`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/MaxGekk/spark-1
json-remove-non-nullable-schema
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22366.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22366
----
commit 0de349bc59143eeb26cd422fdfe945037f8353ac
Author: Maxim Gekk <maxim.gekk@...>
Date: 2018-09-08T19:33:20Z
Removing the spark.sql.fromJsonForceNullableSchema flag
commit f9508458d1963e83c7fc23106dc4cb2f1f491524
Author: Maxim Gekk <maxim.gekk@...>
Date: 2018-09-08T19:46:05Z
Bug fix - missing field must not nullable
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
