GitHub user zhzhan opened a pull request:
https://github.com/apache/spark/pull/17180
[SPARK-19839][Core]release longArray in BytesToBytesMap
## What changes were proposed in this pull request?
When BytesToBytesMap spills, its longArray should be released. Otherwise,
it may not released until the task complete. This array may take a significant
amount of memory, which cannot be used by later operator, such as
UnsafeShuffleExternalSorter, resulting in more frequent spill in sorter. This
patch release the array as destructive iterator will not use this array anymore.
## How was this patch tested?
Manual test in production
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhzhan/spark memory
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17180.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17180
----
commit 562625621f701c77e7755a82c9d9551688f97684
Author: Zhan Zhang <[email protected]>
Date: 2017-03-06T20:02:01Z
release longArray in BytesToBytesMap
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
