[
https://issues.apache.org/jira/browse/ACCUMULO-580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280556#comment-13280556
]
Keith Turner commented on ACCUMULO-580:
---------------------------------------
The intent of this fixed size buffer is to limit memory usage. I do not think
it does a very good job at that. I am thinking this queue should be of
unlimited size, however each batch scanner thread should be limited to two
active batches. So a batch scanner thread would fetch a batch and throw the
entire batch on the queue. It could then go fetch another batch. The thread
would then block until at least one of the previous two batches was consumed.
This would limit memory usage to 2*BatchSizeByte*NumThreads. The current
default batch size is 1MB, this could be set to 512K to approximate current
memory usage.
Allowing two batches per thread should allow for more parallelism and limit
memory usage.
The current fixed size buffer of 1000 entries could lead to obnoxious memory
usage if large key values are returned. This is the reason I think its not a
suitable solution for managing memory.
> Make size of batch scanner client size buffer configurable
> ----------------------------------------------------------
>
> Key: ACCUMULO-580
> URL: https://issues.apache.org/jira/browse/ACCUMULO-580
> Project: Accumulo
> Issue Type: Improvement
> Components: client
> Reporter: Keith Turner
> Assignee: Billie Rinaldi
>
> The batch scanner has a buffer on the client side where results read from
> tservers are stored. This buffer holds 1000 entries and it not configurable
> by the client. When it fills up, all threads reading from tservers block.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
