[
https://issues.apache.org/jira/browse/IMPALA-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16766330#comment-16766330
]
Pooja Nilangekar commented on IMPALA-4268:
------------------------------------------
Here is an update about the proposed solution:
There could be a ResultsBuffer (or any better class name) which would be shared
between the Coordinator and the PlanRootSink. Initially, it would be of a fixed
size (~8MB). The PlanRootSink::Send() function would be modified to write into
the ResutlsBuffer. There would be no direct consumer calling into
PlanRootSink::GetNext(). Instead, the fetch calls on the coordinator would now
try to read from the ResultBuffer.
To handle a case of a cleanly updating the query state, since the entire tree
would be able to continually write into the ResultsBuffer, once the last row is
added to the buffer, the fragment instance would call DoneExecuting() and hence
the coordinator would be able to release all admission control resources apart
from the ResutlsBuffer. This would ensure that this transition is not
observable to the client since all the fetch calls would continue to be served
from the ResultsBuffer via the coordinator.
> Rework coordinator buffering to buffer more data
> ------------------------------------------------
>
> Key: IMPALA-4268
> URL: https://issues.apache.org/jira/browse/IMPALA-4268
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Affects Versions: Impala 2.8.0
> Reporter: Henry Robinson
> Assignee: Pooja Nilangekar
> Priority: Major
> Labels: query-lifecycle, resource-management
> Attachments: rows-produced-histogram.png
>
>
> {{PlanRootSink}} executes the producer thread (the coordinator fragment
> execution thread) in a separate thread to the consumer (i.e. the thread
> handling the fetch RPC), which calls {{GetNext()}} to retrieve the rows. The
> implementation was simplified by handing off a single batch at a time from
> the producers to consumer.
> This decision causes some problems:
> * Many context switches for the sender. Adding buffering would allow the
> sender to append to the buffer and continue progress without a context switch.
> * Query execution can't release resources until the client has fetched the
> final batch, because the coordinator fragment thread is still running and
> potentially producing backpressure all the way down the plan tree.
> * The consumer can't fulfil fetch requests greater than Impala's internal
> BATCH_SIZE, because it is only given one batch at a time.
> The tricky part is managing the mismatch between the size of the row batches
> processed in {{Send()}} and the size of the fetch result asked for by the
> client without impacting performance too badly. The sender materializes
> output rows in a {{QueryResultSet}} that is owned by the coordinator. That is
> not, currently, a splittable object - instead it contains the actual RPC
> response struct that will hit the wire when the RPC completes. As
> asynchronous sender does not know the batch size, because it can in theory
> change on every fetch call (although most reasonable clients will not
> randomly change the fetch size).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
