[
https://issues.apache.org/jira/browse/IMPALA-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16568510#comment-16568510
]
Tim Armstrong commented on IMPALA-3652:
---------------------------------------
The required fix is for Reset() to take a RowBatch as an argument, to which
memory referenced by previously returned rows can be returned. The logic for
Reset() attaching memory needs to be similar to the handling of ReachedLimit(),
e.g. this example in NLJNode::GetNext():
{code}
if (ReachedLimit()) {
output_batch->set_num_rows(
output_batch->num_rows() - (num_rows_returned_ - limit_));
eos_ = true;
}
if (eos_) {
*eos = true;
probe_batch_->TransferResourceOwnership(output_batch);
build_batches_->TransferResourceOwnership(output_batch);
}
{code}
> Fix resource transfer in subplans with limits
> ---------------------------------------------
>
> Key: IMPALA-3652
> URL: https://issues.apache.org/jira/browse/IMPALA-3652
> Project: IMPALA
> Issue Type: Task
> Components: Backend
> Affects Versions: Impala 2.6.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Major
> Labels: resource-management
>
> There is a tricky corner case in our resource transfer model with subplans
> and limits. The problem is that the limit in the subplan may mean that the
> exec node is reset before it has returned its full output. The resource
> transfer logic generally attaches resources to batches at specific points in
> the output, e.g. end of partition, end of block, so it's possible that
> batches returned before the Reset() may reference resources that have not yet
> been transferred. It's unclear if we test this scenario consistently or if
> it's always handled correctly.
> One example is this query, reported in IMPALA-5456:
> {code}
> select c_custkey, c_mktsegment, o_orderkey, o_orderdate
> from customer c,
> (select o1.o_orderkey, o2.o_orderdate
> from c.c_orders o1, c.c_orders o2
> where o1.o_orderkey = o2.o_orderkey limit 10) v limit 500;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
