[
https://issues.apache.org/jira/browse/SOLR-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15926351#comment-15926351
]
Joshua Humphries commented on SOLR-5872:
----------------------------------------
I identified one issue with slowness in processing the overseer queue: a
'downnode' message can result in far more updates to ZK than necessary --
mainly for clusters with many collections where any given collection only has
shard-replicas on a small minority of the nodes. Our cluster has many thousands
of collections, most of which have only one shard and one replica. So
'downnode' was updating about 40x more collections in ZK than actually
necessary. Furthermore, all of the writes are done synchronously/sequentially
which means we pay the RTT to ZK 40x more than necessary. (Also, writes across
collections, even when state_format > 1, could be batched and pipelined, to
further reduce latency here.)
See SOLR-10277 for more details.
> Eliminate overseer queue
> -------------------------
>
> Key: SOLR-5872
> URL: https://issues.apache.org/jira/browse/SOLR-5872
> Project: Solr
> Issue Type: Improvement
> Components: SolrCloud
> Reporter: Noble Paul
> Assignee: Noble Paul
>
> The overseer queue is one of the busiest points in the entire system. The
> raison d'être of the queue is
> * Provide batching of operations for the main clusterstate,json so that
> state updates are minimized
> * Avoid race conditions and ensure order
> Now , as we move the individual collection states out of the main
> clusterstate.json, the batching is not useful anymore.
> Race conditions can easily be solved by using a compare and set in Zookeeper.
> The proposed solution is , whenever an operation is required to be performed
> on the clusterstate, the same thread (and of course the same JVM)
> # read the fresh state and version of zk node
> # construct the new state
> # perform a compare and set
> # if compare and set fails go to step 1
> This should be limited to all operations performed on external collections
> because batching would be required for others
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
