[
https://issues.apache.org/jira/browse/IMPALA-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16563205#comment-16563205
]
ASF subversion and git services commented on IMPALA-7296:
---------------------------------------------------------
Commit 3f8375d3e642554b5506f3e731f94e6328fcbcf9 in impala's branch
refs/heads/master from [[email protected]]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=3f8375d ]
IMPALA-7296: bytes limit for row batch queue
https://goo.gl/N9LgQt summarises the memory problems I'm trying to solve
here.
Limit the number of enqueued row batches to a number of bytes,
instead of limiting the total number of batches. This helps
avoid pathologically high memory consumption for wide rows where the #
batches limit does not effectively limit the memory consumption.
The bytes limit only lowers the effective capacity of the queue
for wider rows, typically 150 bytes or wider. These are the
cases when we want to reduce the queue's capacity.
E.g. on a system with 10 disks, the previous sizing gave a queue
of 100 batches. If we assume rows with 10x16 byte columns, then
100 batches is ~16MB of data.
Remove RowBatchQueueCapacity counter that is less relevant now
and was not correctly initialised.
Testing:
Added some basic unit tests.
Add regression test that fails reliably before this change.
Ran exhaustive build.
Change-Id: Iaa06d1d8da2a6d101efda08f620c0bf84a71e681
Reviewed-on: http://gerrit.cloudera.org:8080/10977
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Soft limit for memory queue in scan node row batch queue
> --------------------------------------------------------
>
> Key: IMPALA-7296
> URL: https://issues.apache.org/jira/browse/IMPALA-7296
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Major
> Labels: resource-management
>
> I'm splitting this out from IMPALA-7096.
> It would be good to have some kind of soft limit for the amount of memory
> that can be queued in the scan node's row batch queue. This would make it
> easier to reason about the expected memory consumption of a scan.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
