I don't know enough how the current leader election mechanism works, yet. I miss a comparison between this proposal and the current mechanism.
B. With this proposal, each time we need the leader, we check it. What is the cost of this check? Do we need to read the cluster state each time? C.1.A "Strict use of ZkShardTerms". Does that mean that we wait for one replica to become up to date? Does that mechanism already exist? In the meantime, if the previous leader comes back, we can choose it. Le jeu. 13 oct. 2022 à 16:24, David Smiley <dsmi...@apache.org> a écrit : > "JIT" is Just-In-Time; a way of looking at it. Or call it on-demand leader > elections. > > A property of my proposal that may not be obvious is that REBALANCELEADERS > would be needless since preferredLeaders would become leaders automatically > on-demand (when a leader is next needed). > > It's not clear how amenable Curator's election recipe is to preferredLeader > and ZkShardTerms requirements; so using Curator for elections might simply > not be an option? > > Here's another thematic concept that is tangentially related; maybe it's > worth its own discussion: Solr's view of state from ZK could become stale > at any time. Embrace sharing of ZK node versions between client and server > to trigger the need for either side to update its state from ZK. Watchers > are nice but we shouldn't rely on them for correctness. Put differently, > SolrCloud should function (albeit perhaps slowly) if its use of ZK Watchers > stopped working. Today, CloudSolrClient passes _stateVer_ and it's parsed > on the server by > org.apache.solr.servlet.HttpSolrCall#checkStateVersionsAreValid. This > should be used more pervasively in most interactions, not merely a subset > of uses of CloudSolrClient. I think it might only be used to tell the > client to get more up to date but ideally it'd be bidirectional. > > ~ David Smiley > Apache Lucene/Solr Search Developer > http://www.linkedin.com/in/davidwsmiley > > > On Tue, Oct 11, 2022 at 6:34 PM David Smiley <dsmi...@apache.org> wrote: > > > At work, I’ve attempted to troubleshoot issues relating to shard > > leadership. It’s quite possible that the root-causes may be related to > > customizations in my fork of Solr; who knows. The leadership > > code/algorithm is so hard to debug/troubleshoot that it’s hard to say. > > It’s no secret that Solr’s code here is a complicated puzzle[1]. Out of > > this frustration, I began to ponder a fantasy of how I want leader > election > > to work, informed by my desire to scale to massive numbers of > collections & > > shards on a cluster. Using Curator for elections would perhaps address > > stability but not this scale. I’d like to get input from you all on this > > fantasy. Surely I have overlooked things; please offer your insights! > > > > Thematic concept: Don’t change/elect leaders until it’s actually > > necessary. In most cases where I work, the leader will return before we > > truly need a leader. Even when not true, I don’t think doing it lazily > > should be a noticeable issue? If so, it’s easy to imagine augmenting > this > > design with an optional eager leadership election. > > > > A. Only code paths that truly need a leader will do “leadership checks”, > > resulting in a potential leader election. This is principally on > indexing > > in DistributedZkUpdateProcessor but there are likely more spots. > > > > B. Leader check: Check if the shard’s leader is (a) known, and (b) > > state=ACTIVE, and (c) on a “live” node, and (d) the preferredLeader > > condition is satisfied. Otherwise, try to elect a leader in a loop until > > this set of conditions is achieved or a timeout is reached. > > B.A: The preferredLeader condition means that either the leader is marked > > as preferredLeader, or no replica with preferredLeader is eligible for > > leadership. > > > > C. “Try to elect a leader”: (The word “election” might not be the best > > word for this algorithm, but whatever). > > C.1.: A replica must be eligible to be a leader. It must be live (on a > > live node) and have an ACTIVE state. And, very important, eligibility > > should be governed by ZkShardTerms which knows which replicas have the > most > > up-to-date state. > > C.1.A: Strict use of ZkShardTerms is designed to ensure that there is no > > data loss. That said “forceLeader” remains in the toolbox of Solr admins > > (which monkey’s with ZkShardTerms to cheat). We may need a new optional > > mechanisms to be closer to what we have today — to basically ignore > > ZkShardTerms after a configured period of time? > > C.1.B. I am assuming that replicas will become eligible on their own > (e.g. > > as nodes re-join) instead of this algorithm needing to initiate/tell any > to > > get into this state somehow. > > C.2: If there are no leader-eligible replicas, complain with useful > > information to diagnose why no leader was found. Don’t log this if we > > already logged this same message in our leadership check loop. Sleep > > perhaps 1000ms and try the loop again. If we can wait/monitor on the > state > > of something convenient then do that to avoid sleeping for too long. > > C.3: Of the leader-eligible replicas — pick whichever one as the leader > > (e.g. random). Prefer preferredLeader=true replicas, of course. ZK will > > solve races if this algorithm runs on more than one node. > > > > D. Only track leadership in Slice (i.e. within the cluster state) which > is > > backed by one spot in ZK. Don’t put it in places like CloudDescriptor or > > other places in ZK. > > > > Thoughts? > > > > > > [1] > > > https://lists.apache.org/list?dev@solr.apache.org:2021-10:MILLER%20leader > > “ZkCmdExecutor” thread with Mark Miller, and referencing > > https://www.solrdev.io/leader-election-adventure.html which no longer > > resolves > > > > > > ~ David Smiley > > Apache Lucene/Solr Search Developer > > http://www.linkedin.com/in/davidwsmiley > > >
