Hey All, My team has run into a pretty nasty issue with overseer election in Solr 10, likely caused by the recent migration to Curator. I've given a pretty detailed account of it here: https://issues.apache.org/jira/browse/SOLR-18301
The crux of the problem is that it's now possible, and even likely, to have an orphaned/zombie overseer leader node which permanently blocks the overseer election from converging until next restart of the stuck overseer. The key difference from 9.X here is that we are running through leader election routine across the same zk session whereas before it would only happen across different sessions. The logs get spammed with election retries until the process finally StackOverflows. I've seen this happen on 4-5 different clouds so am pretty confident we're onto something here. I've also managed to recreate the issue: https://github.com/kotman12/solr/commit/9c0390bf9ac2f6370b3630a2c7c4955fc3da042c Digging through this I noticed several ways Overseer election is a bit different: 1. Overseer cancelElection does not clean up the overseer leader "registration" node. Instead a special function invoked only by the "OverseerExitThread" is tasked with this (checkIfIamStillLeader). In shard leader elections the cancelElection takes care of both the leader node as well as the election node. 2. There seem to be several competing pathways to claim overseer leadership from the same node; the "OverseerExitThread" as well as various callbacks registered with curator which makes this hard to untangle. 3. The version guards in overseer election code are straight up incorrect and worked incidentally before because we'd never invoke them since session expiry would delete the leader node and the OverseerExitThread would have nothing to delete. I'd like some feedback from the community on how to fix this. I can think of some various improvements that would work but also realize I may be missing context because of how tangled this is. Some improvements I can think of: 1. Fix the version checks so that we look at the parent version instead of the leader node version (which is always 0). This seems to be a no-brainer. 2. Once you write an overseer leader node you either must call overseer.start() or, if you cannot, you must delete that node you just created. It should be impossible to claim leadership in zk but then not act on it. One could also argue that zkClient makePath(leaderPath,...) should be pulled *inside* the synchronized block and only invoked *after* the isClosed check although this has other implications. I know from reading thru SOLR-16116 that David had some ideas regarding making isClosed backed by a RW-lock which could maybe obviate the whole synchronized business anyway? (getting side-tracked...) 3. The whole concept of OverseerExitThread is interesting. I traced it back to https://issues.apache.org/jira/browse/SOLR-5859 It seems the main idea is the overseer can be killed from a special cluster operation QUIT and so having this generic clean-up routine that gets invoked on *any* exit (not just an exit triggered by some zk event) was desirable. It gets put on its own dedicated thread so as to not propagate any interrupts from the context that spawned it. I still find it interesting that there are call paths that can delete the leader election node but not the leader registration node and this just seems like bad design. In our case it resulted in election nodes that had sequence numbers in the *millions* (that's how many retries it took to overflow) but the overseer leader node itself was stuck on "7". 4. I am not sure if during the curator migration it was intentional to close election contexts on *any* disconnect from zk but this seems to differ from what was done previously, namely only on session expiry. Given how much churn leader election causes in general I wonder if this is something we can discuss. It could very well be that I am missing something here but it seems like a big decision and I couldn't find any discussion about it. For reference this is how onReconnect was triggered before, notice the "Expired" block: https://github.com/apache/solr/blob/c2091d0258400c9064b8c67fe0d974e367ecccfd/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ConnectionManager.java#L156-L200 Any help/insights/opinions are very welcome. This seems like a big bug to me and I'd like to patch it ASAP so maybe we take an iterative approach to the solution. I'm also surprised no-one else ran into this. Is everyone running with PRS or something? Maybe the real solution is sunset overseer? Thanks, Luke
