Hello Michael Ho, Internal Jenkins,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/2636
to review the following change.
Change subject: IMPALA-2612: Free local allocations once for every row batch
when building hash tables.
......................................................................
IMPALA-2612: Free local allocations once for every row batch when building hash
tables.
When building hash tables for the build side in partitioned
hash join or aggreagtion, we will evaluate the build or probe
side expressions to compute the hash values for each TupleRow.
Evaluation of certain expressions (e.g. CastToChar) requires
"local" memory allocation. "Local" memory allocation is supposed
to be freed after processing each row batch.
However, the calls to free local allocations are missing in
PartitionedHashJoinNode::BuildHashTableInternal() and
PartitionedAggregationNode::ProcessStream(). This causes all
"local" memory allocation to accumulate potentially for the
entire duration of the query or until GetNext() is called.
This may lead to unnecessary memory allocation failure as
memory limit is exceeded.
This patch calls ExecNode::FreeLocalAllocations() at least once
per row-batch when building hash tables. It also adds the missing
checks for the query status in the loop building hash tables.
Please note that QueryMaintenance() isn't called due to its
overhead in memory limit checks.
Change-Id: Idbeab043a45b0aaf6b6a8c560882bd1474a1216d
Reviewed-on: http://gerrit.cloudera.org:8080/1448
Reviewed-by: Michael Ho <[email protected]>
Tested-by: Internal Jenkins
(cherry picked from commit 56a552571e166ead563c8c7459693e00ff2f2889)
---
M be/src/exec/exec-node.cc
M be/src/exec/exec-node.h
M be/src/exec/partitioned-aggregation-node.cc
M be/src/exec/partitioned-hash-join-node.cc
M be/src/runtime/runtime-state.cc
M be/src/runtime/runtime-state.h
M be/src/udf/udf.cc
M testdata/workloads/functional-query/queries/QueryTest/spilling.test
M testdata/workloads/tpch/queries/tpch-outer-joins.test
9 files changed, 50 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/36/2636/1
--
To view, visit http://gerrit.cloudera.org:8080/2636
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Idbeab043a45b0aaf6b6a8c560882bd1474a1216d
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-2.2.0_5.4.x
Gerrit-Owner: Huaisi Xu <[email protected]>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Michael Ho <[email protected]>
