[
https://issues.apache.org/jira/browse/DRILL-640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13992747#comment-13992747
]
Steven Phillips commented on DRILL-640:
---------------------------------------
This solution seems problematic to me. Won't we end up with queries that
continue to run needlessly after the requested records have been returned? On
large tables, this could result in a lot of disk and network IO.
I think what we really need to do is solve the problem of terminating a query.
Maybe it should happen like this:
Limit reaches the end, return IterOutcome.NONE. It does not call
incoming.next() to drain incoming buffers.
After the result has been sent to the client, the foreman informs the leaf
fragments that they should stop execution if they haven't already. The foreman
will inform all intermediate fragments, and finally close the root fragment.
Part of closing each of these fragment will be ensuring that any batch in the
IncomingBuffers queue get released. And we shouldn't have to worry about any
batches that arrive after the fragment has closed, because there will no longer
be a socket connection to receive them.
> Limit operator leaks memory when used with exchanges
> ----------------------------------------------------
>
> Key: DRILL-640
> URL: https://issues.apache.org/jira/browse/DRILL-640
> Project: Apache Drill
> Issue Type: Bug
> Reporter: Mehant Baid
> Assignee: Mehant Baid
> Attachments: DRILL-640.patch
>
>
> Limit operator currently invokes kill() when it reaches the desired limit of
> records. Current semantic of kill() does not stop upstream operators if they
> are across exchanges. So its the responsibility of the operator to drain out
> all the record batches and clear the memory, otherwise we will hit memory
> leaks.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
