GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/8888
[SPARK-10474] [SQL] Aggregation fails to allocate memory for pointer array
(round 2)
This patch reverts most of the changes in a previous fix #8827.
The real cause of the issue is that in `TungstenAggregate`'s prepare method
we only reserve 1 page, but later when we switch to sort-based aggregation we
try to acquire 1 page AND a pointer array. The longer-term fix should be to
reserve also the pointer array, but for now we will simply not track the
pointer array. (Note that elsewhere we already don't track the pointer array,
e.g.
[here](https://github.com/apache/spark/blob/a18208047f06a4244703c17023bb20cbe1f59d73/sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeKVExternalSorter.java#L88))
Note: This patch reuses the unit test added in #8827 so it doesn't show up
in the diff.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewor14/spark dont-track-pointer-array
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8888.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8888
----
commit a00c737da5a701c172dbb4e5bd54bd4bfa1eba7f
Author: Andrew Or <[email protected]>
Date: 2015-09-23T21:10:51Z
Revert "[SPARK-10474] [SQL] Aggregation fails to allocate memory for
pointer array"
This reverts commit 7ff8d68cc19299e16dedfd819b9e96480fa6cf44.
commit 7890baf0d4e48be972168bc9e42ba8ff8729d072
Author: Andrew Or <[email protected]>
Date: 2015-09-23T21:12:23Z
Add back test
commit 00f3739da328d7834ab30686189ea4a337e0348d
Author: Andrew Or <[email protected]>
Date: 2015-09-23T21:14:05Z
Do not track pointer array...
commit fa16b075ddda3cf4ca97ab6a0f4ec444b2f9ddab
Author: Andrew Or <[email protected]>
Date: 2015-09-23T21:19:48Z
Do not release pointer array memory since we don't track it
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
