"JIT" is Just-In-Time; a way of looking at it. Or call it on-demand leader elections.
A property of my proposal that may not be obvious is that REBALANCELEADERS would be needless since preferredLeaders would become leaders automatically on-demand (when a leader is next needed). It's not clear how amenable Curator's election recipe is to preferredLeader and ZkShardTerms requirements; so using Curator for elections might simply not be an option? Here's another thematic concept that is tangentially related; maybe it's worth its own discussion: Solr's view of state from ZK could become stale at any time. Embrace sharing of ZK node versions between client and server to trigger the need for either side to update its state from ZK. Watchers are nice but we shouldn't rely on them for correctness. Put differently, SolrCloud should function (albeit perhaps slowly) if its use of ZK Watchers stopped working. Today, CloudSolrClient passes _stateVer_ and it's parsed on the server by org.apache.solr.servlet.HttpSolrCall#checkStateVersionsAreValid. This should be used more pervasively in most interactions, not merely a subset of uses of CloudSolrClient. I think it might only be used to tell the client to get more up to date but ideally it'd be bidirectional. ~ David Smiley Apache Lucene/Solr Search Developer http://www.linkedin.com/in/davidwsmiley On Tue, Oct 11, 2022 at 6:34 PM David Smiley <dsmi...@apache.org> wrote: > At work, I’ve attempted to troubleshoot issues relating to shard > leadership. It’s quite possible that the root-causes may be related to > customizations in my fork of Solr; who knows. The leadership > code/algorithm is so hard to debug/troubleshoot that it’s hard to say. > It’s no secret that Solr’s code here is a complicated puzzle[1]. Out of > this frustration, I began to ponder a fantasy of how I want leader election > to work, informed by my desire to scale to massive numbers of collections & > shards on a cluster. Using Curator for elections would perhaps address > stability but not this scale. I’d like to get input from you all on this > fantasy. Surely I have overlooked things; please offer your insights! > > Thematic concept: Don’t change/elect leaders until it’s actually > necessary. In most cases where I work, the leader will return before we > truly need a leader. Even when not true, I don’t think doing it lazily > should be a noticeable issue? If so, it’s easy to imagine augmenting this > design with an optional eager leadership election. > > A. Only code paths that truly need a leader will do “leadership checks”, > resulting in a potential leader election. This is principally on indexing > in DistributedZkUpdateProcessor but there are likely more spots. > > B. Leader check: Check if the shard’s leader is (a) known, and (b) > state=ACTIVE, and (c) on a “live” node, and (d) the preferredLeader > condition is satisfied. Otherwise, try to elect a leader in a loop until > this set of conditions is achieved or a timeout is reached. > B.A: The preferredLeader condition means that either the leader is marked > as preferredLeader, or no replica with preferredLeader is eligible for > leadership. > > C. “Try to elect a leader”: (The word “election” might not be the best > word for this algorithm, but whatever). > C.1.: A replica must be eligible to be a leader. It must be live (on a > live node) and have an ACTIVE state. And, very important, eligibility > should be governed by ZkShardTerms which knows which replicas have the most > up-to-date state. > C.1.A: Strict use of ZkShardTerms is designed to ensure that there is no > data loss. That said “forceLeader” remains in the toolbox of Solr admins > (which monkey’s with ZkShardTerms to cheat). We may need a new optional > mechanisms to be closer to what we have today — to basically ignore > ZkShardTerms after a configured period of time? > C.1.B. I am assuming that replicas will become eligible on their own (e.g. > as nodes re-join) instead of this algorithm needing to initiate/tell any to > get into this state somehow. > C.2: If there are no leader-eligible replicas, complain with useful > information to diagnose why no leader was found. Don’t log this if we > already logged this same message in our leadership check loop. Sleep > perhaps 1000ms and try the loop again. If we can wait/monitor on the state > of something convenient then do that to avoid sleeping for too long. > C.3: Of the leader-eligible replicas — pick whichever one as the leader > (e.g. random). Prefer preferredLeader=true replicas, of course. ZK will > solve races if this algorithm runs on more than one node. > > D. Only track leadership in Slice (i.e. within the cluster state) which is > backed by one spot in ZK. Don’t put it in places like CloudDescriptor or > other places in ZK. > > Thoughts? > > > [1] > https://lists.apache.org/list?dev@solr.apache.org:2021-10:MILLER%20leader > “ZkCmdExecutor” thread with Mark Miller, and referencing > https://www.solrdev.io/leader-election-adventure.html which no longer > resolves > > > ~ David Smiley > Apache Lucene/Solr Search Developer > http://www.linkedin.com/in/davidwsmiley >
