[
https://issues.apache.org/jira/browse/SOLR-10524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16000535#comment-16000535
]
Shalin Shekhar Mangar commented on SOLR-10524:
----------------------------------------------
Thanks for the summary [~erickerickson].
bq. 3> One thing not captured by the numbers is that the size of the Overseer
queue is much less like to spin out of control due to both <2> and the fact
that we're reading/ordering/processing batches of up to 10,000 messages at once.
We aren't reading them 10k at a time because there is no multi-read in ZK.
However we write the results of the processing once per 10K messages or 2.5
seconds or whenever the queue goes empty whichever happens earlier.
The rest looks okay. SOLR-10619 fixes an even bigger problem with the
distributed queue used by overseer so we'll see even bigger gains after it is
resolved.
> Explore in-memory partitioning for processing Overseer queue messages
> ---------------------------------------------------------------------
>
> Key: SOLR-10524
> URL: https://issues.apache.org/jira/browse/SOLR-10524
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Erick Erickson
> Attachments: SOLR-10524.patch, SOLR-10524.patch, SOLR-10524.patch,
> SOLR-10524.patch
>
>
> There are several JIRAs (I'll link in a second) about trying to be more
> efficient about processing overseer messages as the overseer can become a
> bottleneck, especially with very large numbers of replicas in a cluster. One
> of the approaches mentioned near the end of SOLR-5872 (15-Mar) was to "read
> large no:of items say 10000. put them into in memory buckets and feed them
> into overseer....".
> This JIRA is to break out that part of the discussion as it might be an easy
> win whereas "eliminating the Overseer queue" would be quite an undertaking.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
