Hi Hoss, all, Yes, I think both configurable limits and per-request controls would be useful. I’d be interested in implementing this, and I’d like to clarify the intended scope before starting.
My proposal is to expose two separate limits: - A maximum number of coordinator attempts by CloudSolrClient for a client invocation. - A maximum number of replica attempts for each internal shard subrequest. A SolrRequest could override the client’s coordinator-attempt default and carry a separate shard-attempt limit. The shard limit would be forwarded to the server and applied independently to each spawned shard subrequest, subject to the server’s configured ceiling. A caller could lower that ceiling but could not raise it. I suggest counting *attempts*, including the initial dispatch, so a value of one means no failover. Skipping a known-dead endpoint without dispatching would not consume an attempt; dispatching to a live or zombie endpoint would. Refreshing stale collection state would not reset the client’s coordinator-attempt allowance. For the interaction with internal requests, the proposal is that each shard subrequest gets its own allowance, separately in each query phase. For example, a shard-attempt limit of two would allow each shard subrequest to try up to two replicas. It would not restrict the entire distributed query to two nodes. Coordinator retries could also generate additional shard work. That distinction seems worth settling first: would these separate limits address the initial requirement, or is the desired guarantee a ceiling on the total number of distinct nodes contacted across the entire distributed query? The latter would require additional accounting across shards, phases, and coordinator failures. Assuming the separate limits are a useful starting point, I plan to structure the work as three PRs: 1. *Consistent LB enforcement.* Build on the existing LBSolrClient.Req.numServersToTry mechanism, covering synchronous and asynchronous paths and zombie fallback. Add dispatch-count tests and a recognizable exhaustion reason. 2. *CloudSolrClient controls.* Add a client default and per-request override for collection queries. Preserve the remaining allowance across stale-state retries and prevent exhaustion from triggering another automatic retry. 3. *Shard-request controls.* Add an explicit absolute server ceiling and a per-request shard limit. Define their interaction with the existing minimum-absolute and maximum-fraction settings, cover ordinary and parallel shard handlers, and document propagation and exhaustion behavior. Each PR would include its tests and documentation. I’d initially preserve existing defaults and retry-safety rules, focusing on queries without expanding which update or administrative requests may be replayed. I also propose making explicit attempt-budget exhaustion terminal for automatic retry, including preventing client failover from replaying a distributed query after reported shard-budget exhaustion, while preserving existing shards.tolerant behavior. Does that sound like the right initial scope? I’d particularly appreciate feedback on the independent per-shard allowances, terminal exhaustion behavior, and whether a finite default should be part of this work or discussed separately. AI-generated, human-reviewed text. Thanks, Nick On Mon, Oct 26, 2026 at 4:59 PM Chris Hostetter <[email protected]> wrote: > > Unless I'm missing something, there is currently no mechanism for a > CloudSolrClient and/or HttpShardHandler to be told "retry this collection > specific request a max of N times" in the event that there are more then N > replicas of the collection/shard that LBSolrClient could send this request > to. > > (We *do* have MAX_STALE_RETRIES - but that is only for stale state > situations, and has no interaction on the "try next url" loop inside > LBSolrClient.request(...)) > > So let's suppose the existence of some "query of death" that could cause a > solr thread to crash (consume all the heap, or call System.exit, or some > other catastrophe, etc...) Ideally this doesn't exist, but let's think > defensively. > > If you have a huge cluster of solr nodes, each hosting a replica of > collectionX, your front end application probably uses CloudSolrClient > w/LBSolrClient to talk to this cluster so you have some resilency in case > a particular node/replica is down (and you can take advantage of the > LBSolrClient "zombie tracking"). > > But now you have a situation where if CloudSolrClient can be tricked into > sending this query of death to *every* solr node in sequence -- because > the first node it's sent to will crash, so then LBSolrClient will say > "huh, replica1 closed it's socket (i wonder why?) i guess i should mark it > as a zombie and move on to replica2." ... and this process will just keep > repeating. > > In theory, the ReplicaListTransformer API can not only transform > the "order" (sorted or shuffled or etc.) of the replicas in it's input, > but also "filter" the list replicas (Example: just the first N replicas > after being re-ordered) but in practice there is no actual option to do > this, and the ReplicaListTransformerGenerator isn't actually a "plugin" > that supports customized implementations (in neither the slr server, nor > SolrJ usages) > > > So.... should there be a mechanism to configure CloudSolrClient & > HttpShardHandler to limit the total number of nodes/replicas that any > request should be sent to? and/or a "max retries" type option on > SolrRequests? (but if so, how should that "per-request" limit interact > with internal per-shard requests that may spawn from it?) > > > > -Hoss > http://www.lucidworks.com/ > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
