I can certainly try using the alternative. That being said IMO it's really bad to have overseer be the default in Solr 10 and have this massive bug (I'd argue not having reliable cluster state is really a massive issue). So it feels like this should be at the very least patched unless we plan on removing it incredibly soon which is hard to see happening given that it is not the default, right?
From: [email protected] At: 07/07/26 13:02:30 UTC-4:00To: [email protected] Cc: Luke Kot-Zaniewski (BLOOMBERG/ 919 3RD A ) Subject: Re: Overseer Election May Not Converge After ZK Disconnect I'd like to remind us that the Overseer has a major complexity problem. Consequently it's hard to diagnose bugs, it's hard to get reviews for them, esp. good ones, it's hard to work productively/helpfully with AI (compared to cleaner / less complex code). Secondly, I want to remind us that the Overseer has been disable-able, in a sense, since 10.0 -- also known internally as "distributed cluster processing". Overseer complexity definitely motivated the design. IMO having worked with SolrCloud internals for years, I think the singular best thing for SolrCloud's health and maintainability is pursuing that and the eventual demise of the Overseer. While it may not be the default, the remaining concerns linked to https://issues.apache.org/jira/browse/SOLR-17293 should not greatly affect most users. Please... enable it for some (all?) of your Solr 10 clouds. It's tempting to look closer at your specific points, but at some point we have to give up on the Overseer and embrace the new thing. I've been there... working on Overseer problems... hoping one day to be done with it. Perhaps you can help that along by at least simply *using* the alternative. On Mon, Jul 6, 2026 at 8:19 PM Luke Kot-Zaniewski (BLOOMBERG/ 919 3RD A) via dev <[email protected]> wrote: Hey All, My team has run into a pretty nasty issue with overseer election in Solr 10, likely caused by the recent migration to Curator. I've given a pretty detailed account of it here: https://issues.apache.org/jira/browse/SOLR-18301 The crux of the problem is that it's now possible, and even likely, to have an orphaned/zombie overseer leader node which permanently blocks the overseer election from converging until next restart of the stuck overseer. The key difference from 9.X here is that we are running through leader election routine across the same zk session whereas before it would only happen across different sessions. The logs get spammed with election retries until the process finally StackOverflows. I've seen this happen on 4-5 different clouds so am pretty confident we're onto something here. I've also managed to recreate the issue: https://github.com/kotman12/solr/commit/9c0390bf9ac2f6370b3630a2c7c4955fc3da042c Digging through this I noticed several ways Overseer election is a bit different: 1. Overseer cancelElection does not clean up the overseer leader "registration" node. Instead a special function invoked only by the "OverseerExitThread" is tasked with this (checkIfIamStillLeader). In shard leader elections the cancelElection takes care of both the leader node as well as the election node. 2. There seem to be several competing pathways to claim overseer leadership from the same node; the "OverseerExitThread" as well as various callbacks registered with curator which makes this hard to untangle. 3. The version guards in overseer election code are straight up incorrect and worked incidentally before because we'd never invoke them since session expiry would delete the leader node and the OverseerExitThread would have nothing to delete. I'd like some feedback from the community on how to fix this. I can think of some various improvements that would work but also realize I may be missing context because of how tangled this is. Some improvements I can think of: 1. Fix the version checks so that we look at the parent version instead of the leader node version (which is always 0). This seems to be a no-brainer. 2. Once you write an overseer leader node you either must call overseer.start() or, if you cannot, you must delete that node you just created. It should be impossible to claim leadership in zk but then not act on it. One could also argue that zkClient makePath(leaderPath,...) should be pulled *inside* the synchronized block and only invoked *after* the isClosed check although this has other implications. I know from reading thru SOLR-16116 that David had some ideas regarding making isClosed backed by a RW-lock which could maybe obviate the whole synchronized business anyway? (getting side-tracked...) 3. The whole concept of OverseerExitThread is interesting. I traced it back to https://issues.apache.org/jira/browse/SOLR-5859 It seems the main idea is the overseer can be killed from a special cluster operation QUIT and so having this generic clean-up routine that gets invoked on *any* exit (not just an exit triggered by some zk event) was desirable. It gets put on its own dedicated thread so as to not propagate any interrupts from the context that spawned it. I still find it interesting that there are call paths that can delete the leader election node but not the leader registration node and this just seems like bad design. In our case it resulted in election nodes that had sequence numbers in the *millions* (that's how many retries it took to overflow) but the overseer leader node itself was stuck on "7". 4. I am not sure if during the curator migration it was intentional to close election contexts on *any* disconnect from zk but this seems to differ from what was done previously, namely only on session expiry. Given how much churn leader election causes in general I wonder if this is something we can discuss. It could very well be that I am missing something here but it seems like a big decision and I couldn't find any discussion about it. For reference this is how onReconnect was triggered before, notice the "Expired" block: https://github.com/apache/solr/blob/c2091d0258400c9064b8c67fe0d974e367ecccfd/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ConnectionManager.java#L156-L200 Any help/insights/opinions are very welcome. This seems like a big bug to me and I'd like to patch it ASAP so maybe we take an iterative approach to the solution. I'm also surprised no-one else ran into this. Is everyone running with PRS or something? Maybe the real solution is sunset overseer? Thanks, Luke
