Impala Public Jenkins has submitted this change and it was merged. Change subject: IMPALA-5713: always reserve memory for preaggs ......................................................................
IMPALA-5713: always reserve memory for preaggs Before this change the preaggregation was frequently disabled when running under some memory pressure, e.g. if the aggregation is at the end of a pipeline of joins and those joins eat up all the memory. This can result in huge performance degradation since all rows must then be sent over the network. This change always reserves 16 * (buffer size + 64kb) bytes per preaggregation so that it is always able to build some hash tables and reduce the input somewhat. This has two parts: * Changing the frontend reservation calculation * Removing dead code in the backend that handled the case when the initial partitions and hash tables could not be allocated. Testing: Passes exhaustive tests. Change-Id: I2b544f9ec1a906719e2bbb074743926b95a3a573 Reviewed-on: http://gerrit.cloudera.org:8080/7739 Reviewed-by: Tim Armstrong <[email protected]> Tested-by: Impala Public Jenkins --- M be/src/exec/partitioned-aggregation-node-ir.cc M be/src/exec/partitioned-aggregation-node.cc M be/src/exec/partitioned-aggregation-node.h M fe/src/main/java/org/apache/impala/planner/AggregationNode.java M testdata/workloads/functional-planner/queries/PlannerTest/max-row-size.test M testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing.test M testdata/workloads/functional-query/queries/QueryTest/spilling-aggs.test 9 files changed, 108 insertions(+), 119 deletions(-) Approvals: Impala Public Jenkins: Verified Tim Armstrong: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/7739 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I2b544f9ec1a906719e2bbb074743926b95a3a573 Gerrit-PatchSet: 7 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Tim Armstrong <[email protected]> Gerrit-Reviewer: Alex Behm <[email protected]> Gerrit-Reviewer: Dan Hecht <[email protected]> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]>
