GitHub user brkyvz opened a pull request:
https://github.com/apache/spark/pull/20302
[SPARK-23094] Fix invalid character handling in JsonDataSource
## What changes were proposed in this pull request?
There were two related fixes regarding `from_json`, `get_json_object` and
`json_tuple` ([Fix
#1](https://github.com/apache/spark/commit/c8803c06854683c8761fdb3c0e4c55d5a9e22a95),
[Fix
#2](https://github.com/apache/spark/commit/86174ea89b39a300caaba6baffac70f3dc702788)),
but they weren't comprehensive it seems. I wanted to extend those fixes to all
the parsers, and add tests for each case.
## How was this patch tested?
Regression tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/brkyvz/spark json-invfix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20302.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20302
----
commit 231152ab0d615166bdea354916a9f6ca0aaf7e6a
Author: Burak Yavuz <brkyvz@...>
Date: 2018-01-18T01:01:00Z
[ES-4104][WARMFIX] Fix invalid character handling in JsonDataSource
Cherry-pick of https://github.com/databricks/spark/pull/1135
## What changes were proposed in this pull request?
I shall also merge this upstream, but wanted to merge here first since it
was an ES ticket related to Qubole workloads.
There were two related fixes regarding `from_json`, `get_json_object` and
`json_tuple` in OSS ([Fix
#1](https://github.com/apache/spark/commit/c8803c06854683c8761fdb3c0e4c55d5a9e22a95),
[Fix
#2](https://github.com/apache/spark/commit/86174ea89b39a300caaba6baffac70f3dc702788)),
but they weren't comprehensive it seems. I wanted to extend those fixes to all
the parsers, and add tests for each case.
## How was this patch tested?
Regression tests
Author: Burak Yavuz <brkyvzgmail.com>
Closes #1135 from brkyvz/json-32-dbx.
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise,
remove this)
Please review http://spark.apache.org/contributing.html before opening a
pull request.
## **IMPORTANT** Warmfix instructions
If this PR needs to be warmfixed (i.e. merged into the release branch after
the code freeze), please follow steps below.
What type of warmfix is this? Please select **exactly one choice**, or
write description in Other.
- [ ] Regression (e.g. fixing the behavior of a feature that regressed in
the current release cycle)
- [ ] ES ticket fix (e.g. Customer or internally requested update/fix)
- [ ] Other (please describe):
Make the following updates to this PR:
- [ ] Add `[WARMFIX]` in the title of this PR.
- [ ] Label the PR using label(s) corrsponding to the WARMFIX branch(es).
The label name should be in the format of `dbr-branch-a.b` (e.g.
`dbr-branch-3.2`), which matches the release branch name for Runtime release
`a.b`.
- [ ] Ask your team lead to sign off this warmfix and add the
`warmfix-approved` label.
- [ ] When merging the PR using the merge script, make sure to get this PR
merged into the following branches:
- The branch against which your PR is opened, and
- Any extra release branch(es) corresponding to the `dbr-branch-a.b`
label(s) applied to your PR.
Author: Burak Yavuz <[email protected]>
Closes #1616 from brkyvz/inv-json4x.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
