GitHub user MaxGekk opened a pull request:

    [SPARK-26303][SQL] Return partial results for bad JSON records

    ## What changes were proposed in this pull request?
    In the PR, I propose to return partial results from JSON datasource and 
JSON functions in the PERMISSIVE mode if some of JSON fields are parsed and 
converted to desired types successfully. The changes are made only for 
`StructType`. Whole bad JSON records are placed into the corrupt column 
specified by the `columnNameOfCorruptRecord` option or SQL config.
    Partial results are not returned for malformed JSON input. 
    ## How was this patch tested?
    Added new UT which checks converting JSON strings with one invalid and one 
valid field at the end of the string.

You can merge this pull request into a Git repository by running:

    $ git pull json-bad-record

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #23253
commit 2d2fed923e9567b6bc22cf4355ee3b57f8699cd4
Author: Maxim Gekk <max.gekk@...>
Date:   2018-12-06T21:26:42Z

    Test for bad records

commit de4885899b2a5e866b6c4a365e91fc901cdc0f86
Author: Maxim Gekk <max.gekk@...>
Date:   2018-12-06T23:11:29Z

    Added a test

commit 101df4e7fcee60c042c21951551bd3376ae325cb
Author: Maxim Gekk <max.gekk@...>
Date:   2018-12-06T23:16:09Z

    Return partial results

commit 542ae74a9240b2905f4ca1f9de62fd53f1562d64
Author: Maxim Gekk <maxim.gekk@...>
Date:   2018-12-07T09:02:37Z

    Updating the migration guide

commit 439f57ac59a7c1f481317126934a55d20b38f579
Author: Maxim Gekk <maxim.gekk@...>
Date:   2018-12-07T09:04:58Z

    Revert "Updating the migration guide"
    This reverts commit 542ae74a9240b2905f4ca1f9de62fd53f1562d64.

commit 77d767016607056809d5f8c4e9653809838b91dd
Author: Maxim Gekk <maxim.gekk@...>
Date:   2018-12-07T09:10:31Z

    Updating the migration guide - separate notes

commit fa431ee293d5ae3a7c3b8b441c4a0b6778ac84e1
Author: Maxim Gekk <maxim.gekk@...>
Date:   2018-12-07T09:12:16Z

    Updating the migration guide - minor

commit e13673de2537fb42bdc2bb8df9849560ae399c89
Author: Maxim Gekk <maxim.gekk@...>
Date:   2018-12-07T10:00:07Z

    Updating the migration guide - minor 2



To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to