Tim Armstrong has uploaded a new patch set (#8). Change subject: IMPALA-1346/1590/2344: fix sorter buffer mgmt when spilling ......................................................................
IMPALA-1346/1590/2344: fix sorter buffer mgmt when spilling The Sorter's memory management logic failed to correctly manage buffers when spilling. It would try to make use of all buffers in the system, neglecting to account for other operators' buffer usage. This patch adjusts the logic so that it handles contention for buffers so long as it can get enough buffers to make progress. Instead of precalculating the number of buffers it thinks it should be able to pin, it just makes a best-effort attempt to pin the initial buffers as many runs as possible, up to a limit. As long as it can pin three runs, it can make progress. Testing: Added an additional test that failed before the patch without OOM. An analytic function test that was meant to fail also started succeeding so I had to adjust the limit there too. Change-Id: Idfe55cc13c7f2b54cba1d05ade44cbcf6bb573c0 --- M be/src/runtime/buffered-block-mgr.h M be/src/runtime/sorter.cc M be/src/runtime/sorter.h M testdata/workloads/functional-query/queries/QueryTest/analytic-fns.test M testdata/workloads/functional-query/queries/QueryTest/spilling.test 5 files changed, 187 insertions(+), 115 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/08/2908/8 -- To view, visit http://gerrit.cloudera.org:8080/2908 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idfe55cc13c7f2b54cba1d05ade44cbcf6bb573c0 Gerrit-PatchSet: 8 Gerrit-Project: Impala Gerrit-Branch: cdh5-trunk Gerrit-Owner: Tim Armstrong <[email protected]> Gerrit-Reviewer: Dan Hecht <[email protected]> Gerrit-Reviewer: Matthew Jacobs <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]>
