GitHub user yhuai opened a pull request:
https://github.com/apache/spark/pull/10288
[SPARK-12057] [SQL] Prevent failure on corrupt JSON records
This PR makes JSON parser and schema inference handle more cases where we
have unparsed records. It is based on #10043. The last commit fixes the failed
test and updates the logic of schema inference.
When merge this PR, please make sure that the author is @simplyianm.
JIRA: https://issues.apache.org/jira/browse/SPARK-12057
Closes #10043
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yhuai/spark handleCorruptJson
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10288.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10288
----
commit 0271f0ceecd24aa97cc813994baa4bf8b7ba5256
Author: Ian Macalinao <[email protected]>
Date: 2015-11-30T18:23:34Z
Prevent failure on corrupt JSON records
Return failed record when a record cannot be parsed. Allows parsing of
files containing corrupt records of any form.
commit e7a032a4617823a4c7f3741516d8115b50ffa464
Author: Ian Macalinao <[email protected]>
Date: 2015-12-01T20:10:33Z
Add regression test for corrupt record JSON parsing
commit 4fba0cacd1b301f1678eada315fbd91f58fd7618
Author: Ian Macalinao <[email protected]>
Date: 2015-12-01T20:53:33Z
Correct schema
commit d8722bb30d347fa97c7d358284eb433edb1be6ec
Author: Yin Huai <[email protected]>
Date: 2015-12-14T04:57:20Z
Handle more cases.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
