murblanc opened a new pull request #70: URL: https://github.com/apache/solr/pull/70
This PR for [SOLR-15146](https://issues.apache.org/jira/browse/SOLR-15146) is about allowing Collection and Config Set API's to run distributed (on any node) rather than requiring a round trip to Overseer. It is a follow up to a previous change distributing the Cluster State Updater ([SOLR-14928](https://issues.apache.org/jira/browse/SOLR-14928)). Commands are executed directly from the handler receiving the request (`CollectionsHandler` or `ConfigSetsHandler`). There are no Zookeeper queues and the Overseer node is not involved (a previous refactoring prepared for this change, see [SOLR-15157](https://issues.apache.org/jira/browse/SOLR-15157)). In order to support long running (and for Collection API, async) operations, commands are executed on a different thread. This allows implementing a behavior very similar to Overseer based processing (a request can time out, yet the command continues execution). Specific async job tracking was implemented (different from the existing one). In case of crash of the handler node running the command, the command will fail and will not be retried, something the caller must be prepared to handle anyway. An entry point is `DistributedApiAsyncTracker`. Locking is done through Zookeeper, by using a read/write lock (as described in Zookeeper recipes) as the base building block. See `DistributedLock` as an entry point into that on top of which higher level locking abstractions are built for Collection and Config Set commands. Some Config Set processing refactoring was required (equivalent to what [SOLR-15157](https://issues.apache.org/jira/browse/SOLR-15157) did for collections), basically moving code from `OverseerConfigSetMessageHandler` to `ConfigSetCmds`. Distributed Collection and Config Set processing is off by default, so out of the box SolrCloud behavior is unchanged. Configuration can be switched to distributed API execution by setting both `distributedClusterStateUpdates` and `distributedCollectionConfigSetExecution` to `true` in `solr.xml` (or using system properties). Tests pick distributed or Overseer Collection and Config Set API execution as well as Cluster State Updates based on the random test seed (if the API is distributed the cluster state update must be as well). This is done in `SolrCloudTestCase`. Even though performance **was not** the main goal of this change (main goal is code and flow simplification to enable future optimizations of state management), a simple test of Collection creation shows improved speed (tests also run overall faster). On a 2018 MacBook Pro with 32GB of RAM, collection creation by 10 threads creating 10 collections each, each collection being 2 shards x 2 replicas (total 400 replicas), connecting via `http` to a single node of the cluster (cluster is 3 nodes with 10GB heap each). The cluster was restarted fresh before 4 consecutive runs with each configuration. Note that only the Collection API and cluster state updates is different between the Overseer based and the distributed runs, the actual core creation requests and execution by each node are identical. Here are the timings in milliseconds per single collection creation: Overseer based Collection API and Cluster State Updates (i.e. normal SolrCloud): ``` Avg, min, max 12164, 10296, 16459 12905, 9420, 22505 12266, 9200, 19906 11814, 9024, 15237 ``` Distributed Collection API and Cluster State Updates: ``` Avg, min, max 9069, 6260, 12340 7544, 5823, 10114 11324, 7103, 14149 7789, 6428, 10282 ``` I plan to merge these changes a week from now unless of course there are comments, discussion etc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
