murblanc opened a new pull request #2285: URL: https://github.com/apache/lucene-solr/pull/2285
SOLR-14928: introduce distributed cluster state updates (i.e. not having to go through a ZK queue and Overseer) The motivation behind this PR is to simplify SolrCloud and be the first step to (eventually) a significant scale increase: handle orders of magnitude more collections than currently and manage cluster state caching in a more flexible way. Changes introduced by this PR: - Support of distributed state updates for collections through Compare and Swap for updating state.json Zookeeper files. - Cluster wide configuration in `solr.xml` to pick the way cluster state updates are handled (default remains Overseer based updates) - Randomization based on test seed of the cluster update strategy (Distributed vs Overseer) in tests using `MiniSolrCloudCluster`, so both execution paths get continuously tested. A good entry point to explore this PR is class `DistributedClusterChangeUpdater`. Performance wise, small (low replica count) collection creation is faster with the distributed strategy than with Overseer, and significantly faster when multiple collections are created in parallel (10 parallel threads continuously creating 4 replicas collections show a 30% reduction in collection creation time on my laptop). High replica count (~40 and more) collection creation is slower. This will be addressed by using Per Replica States that limit contention on access to a single state.json file and/or a future splitting of state.json into finer grain elements. Next step after this one is distributing the execution of the Collection API commands to all nodes. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
