[
https://issues.apache.org/jira/browse/IMPALA-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16958008#comment-16958008
]
Tim Armstrong commented on IMPALA-5485:
---------------------------------------
This tends to show up as:
{noformat}
ERROR: Memory limit exceeded: Failed to allocate row batch
EXCHANGE_NODE (id=46) could not allocate 8.00 KB without exceeding limit.
E
{noformat}
> Exchg / data-stream recv-side buffers should operate under a memory
> reservation
> -------------------------------------------------------------------------------
>
> Key: IMPALA-5485
> URL: https://issues.apache.org/jira/browse/IMPALA-5485
> Project: IMPALA
> Issue Type: Sub-task
> Components: Distributed Exec
> Reporter: Henry Robinson
> Priority: Major
>
> Exchange nodes assign a fixed-size buffer to their datastream receivers
> that's used to smooth out differences in send / consume rates between the
> sender and the receiver.
> These buffers should be tracked by the query memtracker, and with the new
> min-reservation support we should allow them to be larger than the configured
> minimum. Increasing the buffer size decreases the amount of time that a
> sender can be blocked on a receiver, and so increases query-parallelism.
> Queries that shuffle a lot of data can see significant speedups from larger
> buffers.
> The buffers need to be sized based on the #senders and the #rows * #avg row
> size. They can dynamically expand trivially - contraction is possible, but a
> bit harder.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
