GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/8003
[SPARK-9548][SQL] Add a destructive iterator for BytesToBytesMap
This pull request adds a destructive iterator to BytesToBytesMap. When
used, the iterator frees pages as we traverse them. This is part of the effort
to avoid starving when we have more than one operators that can exhaust memory.
This is based on #7924, but fixes a bug there (Don't use destructive
iterator in UnsafeKVExternalSorter).
Closes #7924.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rxin/spark map-destructive-iterator
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8003.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8003
----
commit 9e9d2a3cb1b21e42793ce58342a2790935039348
Author: Liang-Chi Hsieh <[email protected]>
Date: 2015-08-04T06:55:51Z
Add a destructive iterator for BytesToBytesMap.
commit f0ff783e93109747310afed6cd19d80a39b58534
Author: Liang-Chi Hsieh <[email protected]>
Date: 2015-08-05T00:09:36Z
No need to free last page.
commit 581e9e3f79e98dd4c5f52543a1eb635999bb6e60
Author: Liang-Chi Hsieh <[email protected]>
Date: 2015-08-05T02:39:55Z
Merge remote-tracking branch 'upstream/master' into destructive_iter
commit 4a3e9defdf98b0f5809b0ae547d557aa39dd2b81
Author: Liang-Chi Hsieh <[email protected]>
Date: 2015-08-06T04:06:58Z
Merge remote-tracking branch 'upstream/master' into destructive_iter
commit 76520839f0702a78363631e35502a293802be6ab
Author: Liang-Chi Hsieh <[email protected]>
Date: 2015-08-06T04:49:13Z
For comments: add destructiveIterator(), modify unit test, remove code
block.
commit a7bd8ec8da031aed5f482525513adb8f29d2acf0
Author: Reynold Xin <[email protected]>
Date: 2015-08-06T19:00:48Z
Merge remote-tracking branch 'viirya/destructive_iter' into
map-destructive-iterator
commit 6b618c3232000bc6fdc8e0420661c3cb9cca5358
Author: Reynold Xin <[email protected]>
Date: 2015-08-06T19:02:19Z
Don't use destructive iterator in UnsafeKVExternalSorter.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
