Huaisi Xu has submitted this change and it was merged. Change subject: IMPALA-2612: Free local allocations once for every row batch when building hash tables. ......................................................................
IMPALA-2612: Free local allocations once for every row batch when building hash tables. When building hash tables for the build side in partitioned hash join or aggreagtion, we will evaluate the build or probe side expressions to compute the hash values for each TupleRow. Evaluation of certain expressions (e.g. CastToChar) requires "local" memory allocation. "Local" memory allocation is supposed to be freed after processing each row batch. However, the calls to free local allocations are missing in PartitionedHashJoinNode::BuildHashTableInternal() and PartitionedAggregationNode::ProcessStream(). This causes all "local" memory allocation to accumulate potentially for the entire duration of the query or until GetNext() is called. This may lead to unnecessary memory allocation failure as memory limit is exceeded. This patch calls ExecNode::FreeLocalAllocations() at least once per row-batch when building hash tables. It also adds the missing checks for the query status in the loop building hash tables. Please note that QueryMaintenance() isn't called due to its overhead in memory limit checks. Change-Id: Idbeab043a45b0aaf6b6a8c560882bd1474a1216d Reviewed-on: http://gerrit.cloudera.org:8080/1448 Reviewed-by: Michael Ho <[email protected]> Tested-by: Internal Jenkins (cherry picked from commit 56a552571e166ead563c8c7459693e00ff2f2889) Reviewed-on: http://gerrit.cloudera.org:8080/2636 Reviewed-by: Huaisi Xu <[email protected]> Tested-by: Huaisi Xu <[email protected]> --- M be/src/exec/exec-node.cc M be/src/exec/exec-node.h M be/src/exec/partitioned-aggregation-node.cc M be/src/exec/partitioned-hash-join-node.cc M be/src/runtime/runtime-state.cc M be/src/runtime/runtime-state.h M be/src/udf/udf.cc M testdata/workloads/functional-query/queries/QueryTest/spilling.test M testdata/workloads/tpch/queries/tpch-outer-joins.test 9 files changed, 49 insertions(+), 8 deletions(-) Approvals: Huaisi Xu: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/2636 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: Idbeab043a45b0aaf6b6a8c560882bd1474a1216d Gerrit-PatchSet: 3 Gerrit-Project: Impala Gerrit-Branch: cdh5-2.2.0_5.4.x Gerrit-Owner: Huaisi Xu <[email protected]> Gerrit-Reviewer: Huaisi Xu <[email protected]> Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Michael Ho <[email protected]>
