[
https://issues.apache.org/jira/browse/SOLR-10524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16000982#comment-16000982
]
Erick Erickson commented on SOLR-10524:
---------------------------------------
[~shalinmangar] Hmmm, maybe a better way to phrase it is that we process the
queue in batches of up to 10,000? I didn't mean to convey that the Zookeeper
read was in batches that size, just that we processed them up to that many at
once.
> Explore in-memory partitioning for processing Overseer queue messages
> ---------------------------------------------------------------------
>
> Key: SOLR-10524
> URL: https://issues.apache.org/jira/browse/SOLR-10524
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Erick Erickson
> Attachments: SOLR-10524-NPE-fix.patch, SOLR-10524.patch,
> SOLR-10524.patch, SOLR-10524.patch, SOLR-10524.patch
>
>
> There are several JIRAs (I'll link in a second) about trying to be more
> efficient about processing overseer messages as the overseer can become a
> bottleneck, especially with very large numbers of replicas in a cluster. One
> of the approaches mentioned near the end of SOLR-5872 (15-Mar) was to "read
> large no:of items say 10000. put them into in memory buckets and feed them
> into overseer....".
> This JIRA is to break out that part of the discussion as it might be an easy
> win whereas "eliminating the Overseer queue" would be quite an undertaking.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
