Thomas Marshall has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11852
Change subject: IMPALA-3652: Fix resource transfer in subplans with limits ...................................................................... IMPALA-3652: Fix resource transfer in subplans with limits Impala assumes that when Reset() is called on an ExecNode, all of the memory returned from that node by GetNext() has been attached to the output RowBatch. In a query with a LIMIT on the subplan, such that some nodes don't reach 'eos', this may not be the case. The solution is to have Reset() take a RowBatch that any such memory can be attached to. I examined all ExecNodes for the behavior of: the node already calls TransferResourceOwnership() during GetNext() on a RowBatch that is a member variable of that exec node, and the node supports Reset(). For all such RowBatches, I added a call to transfer that batch's resources in Reset(). This patch is mostly a mechanical change that adds a RowBatch parameter to all instances of Reset(). The interesting changes: - Transfers resources from the relevant RowBatch to the parameter RowBatch in Reset() for BlockingJoinNode, SelectNode, and SubplanNode. - Modifies PartialSortNode::Reset() to return an error in release builds instead of silently doing the wrong thing. I was not able to repro the issue in SelectNode or SubplanNode, and I suspect with our current planner it may not be possible to generate a plan with the necessary shape, but I think it makes sense to handle these nodes anyways in case the planner changes in the future. Testing: - Added e2e tests that repro the issue for hash and nested loop joins. Change-Id: I3968a379fcbb5d30fcec304995d3e44933dbbc77 --- M be/src/exec/aggregation-node-base.cc M be/src/exec/aggregation-node-base.h M be/src/exec/analytic-eval-node.cc M be/src/exec/analytic-eval-node.h M be/src/exec/blocking-join-node.cc M be/src/exec/blocking-join-node.h M be/src/exec/cardinality-check-node.cc M be/src/exec/cardinality-check-node.h M be/src/exec/data-source-scan-node.cc M be/src/exec/data-source-scan-node.h M be/src/exec/exchange-node.cc M be/src/exec/exchange-node.h M be/src/exec/exec-node.cc M be/src/exec/exec-node.h M be/src/exec/hbase-scan-node.cc M be/src/exec/hbase-scan-node.h M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/nested-loop-join-node.cc M be/src/exec/nested-loop-join-node.h M be/src/exec/partial-sort-node.cc M be/src/exec/partial-sort-node.h M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/partitioned-hash-join-node.h M be/src/exec/select-node.cc M be/src/exec/select-node.h M be/src/exec/sort-node.cc M be/src/exec/sort-node.h M be/src/exec/streaming-aggregation-node.cc M be/src/exec/streaming-aggregation-node.h M be/src/exec/subplan-node.cc M be/src/exec/subplan-node.h M be/src/exec/topn-node.cc M be/src/exec/topn-node.h M be/src/exec/union-node.cc M be/src/exec/union-node.h M be/src/exec/unnest-node.cc M be/src/exec/unnest-node.h M testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch.test 39 files changed, 95 insertions(+), 55 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/11852/1 -- To view, visit http://gerrit.cloudera.org:8080/11852 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3968a379fcbb5d30fcec304995d3e44933dbbc77 Gerrit-Change-Number: 11852 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall <[email protected]>
