[
https://issues.apache.org/jira/browse/IMPALA-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460323#comment-16460323
]
Michael Ho commented on IMPALA-5485:
------------------------------------
Yes but that may cause smaller row batches to be returned from exchange node.
We can revisit the tradeoff when this JIRA becomes higher priority.
> Exchg / data-stream recv-side buffers should operate under a memory
> reservation
> -------------------------------------------------------------------------------
>
> Key: IMPALA-5485
> URL: https://issues.apache.org/jira/browse/IMPALA-5485
> Project: IMPALA
> Issue Type: Sub-task
> Components: Distributed Exec
> Reporter: Henry Robinson
> Priority: Major
>
> Exchange nodes assign a fixed-size buffer to their datastream receivers
> that's used to smooth out differences in send / consume rates between the
> sender and the receiver.
> These buffers should be tracked by the query memtracker, and with the new
> min-reservation support we should allow them to be larger than the configured
> minimum. Increasing the buffer size decreases the amount of time that a
> sender can be blocked on a receiver, and so increases query-parallelism.
> Queries that shuffle a lot of data can see significant speedups from larger
> buffers.
> The buffers need to be sized based on the #senders and the #rows * #avg row
> size. They can dynamically expand trivially - contraction is possible, but a
> bit harder.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
