GitHub user clockfly opened a pull request:
https://github.com/apache/spark/pull/14990
[SPARK-17426][SQL] Refactor `TreeNode.toJSON` to avoid OOM when converting
unknown fields to JSON
## What changes were proposed in this pull request?
This PR is a follow up of SPARK-17356. Current implementation of
`TreeNode.toJSON` recursively converts all fields to JSON, even if the field is
of type `Seq` or type Map. This may trigger out of memory exception in cases
like:
1. the Seq or Map can be very big. Converting them to JSON make take huge
memory, which may trigger out of memory error.
2. Some user space input may also be propagated to the Plan. The user space
input can be of arbitrary type, and may also be self-referencing. Trying to
print user space input to JSON may trigger out of memory error or stack
overflow error.
For a real example, please check the Jira description of SPARK-17426.
In this PR, we refactor the `TreeNode.toJSON` so that we only convert a
field to JSON string if the field is a safe type.
## How was this patch tested?
Unit test.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/clockfly/spark json_oom2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14990.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14990
----
commit 284d780c446b3f7cb59f8a2f34c522d90bc43fe1
Author: Sean Zhong <[email protected]>
Date: 2016-09-07T03:49:23Z
json oom
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
