Tim Armstrong has submitted this change and it was merged. Change subject: IMPALA-1346/1590/2344: fix sorter buffer mgmt when spilling ......................................................................
IMPALA-1346/1590/2344: fix sorter buffer mgmt when spilling The Sorter's memory management logic failed to correctly manage buffers when spilling. It would try to make use of all buffers in the system, neglecting to account for other operators' buffer usage. This patch adjusts the logic so that it handles contention for buffers so long as it can get enough buffers to make progress. Instead of precalculating the number of buffers it thinks it should be able to pin, it just makes a best-effort attempt to pin the initial buffers as many runs as possible, up to a limit. As long as it can pin three runs, it can make progress. Testing: Added an additional test that failed before the patch without OOM. An analytic function test that was meant to fail also started succeeding so I had to adjust the limit there too. Change-Id: Idfe55cc13c7f2b54cba1d05ade44cbcf6bb573c0 Reviewed-on: http://gerrit.cloudera.org:8080/2908 Reviewed-by: Dan Hecht <[email protected]> Tested-by: Tim Armstrong <[email protected]> --- M be/src/runtime/buffered-block-mgr.h M be/src/runtime/sorter.cc M be/src/runtime/sorter.h M testdata/workloads/functional-query/queries/QueryTest/analytic-fns.test M testdata/workloads/functional-query/queries/QueryTest/spilling.test 5 files changed, 187 insertions(+), 115 deletions(-) Approvals: Tim Armstrong: Verified Dan Hecht: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/2908 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: Idfe55cc13c7f2b54cba1d05ade44cbcf6bb573c0 Gerrit-PatchSet: 10 Gerrit-Project: Impala Gerrit-Branch: cdh5-trunk Gerrit-Owner: Tim Armstrong <[email protected]> Gerrit-Reviewer: Dan Hecht <[email protected]> Gerrit-Reviewer: Matthew Jacobs <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]>
