Thanks for finding this and putting so much work into debugging it Luke! Yeah the Curator change was giant and really hard to battle-test, especially since it landed a while before 10.0 was released. There were always going to be growing pains here. (Also 10 is still on 10.0, so the first release of every major version will have some big growing pains, it's inevitable).
I asked on the JIRA, but let's get a PR going that starts with your test confirming at least one of the issues. That way we can iterate together rather than going back and forth on email. As for your comment David, I agree that the Overseer has a major complexity problem, but I'm not sure I'd agree that the distributed overseer fixes all of that complexity. Also I've yet to hear of a major user of Solr that has turned it on for a major use-case. It's hard to recommend that as an alternative, when it hasn't been tested in the real world. This issue with the overseer wasn't seen with the test coverage we have now, so who knows what issues the disturbed overseer has outside of the tests. Also Luke is right that the overseer will be default for the foreseeable future, so this needs to be fixed ASAP (and before a 10.1 is cut). - Houston On Tue, Jul 7, 2026 at 11:00 AM Luke Kot-Zaniewski (BLOOMBERG/ 919 3RD A) via dev <[email protected]> wrote: > I can certainly try using the alternative. That being said > IMO it's really bad to have overseer be the default in > Solr 10 and have this massive bug (I'd argue not having > reliable cluster state is really a massive issue). So > it feels like this should be at the very least patched > unless we plan on removing it incredibly soon which is hard > to see happening given that it is not the default, right? > > From: [email protected] At: 07/07/26 13:02:30 UTC-4:00To: > [email protected] > Cc: Luke Kot-Zaniewski (BLOOMBERG/ 919 3RD A ) > Subject: Re: Overseer Election May Not Converge After ZK Disconnect > > I'd like to remind us that the Overseer has a major complexity problem. > Consequently it's hard to diagnose bugs, it's hard to get reviews for them, > esp. good ones, it's hard to work productively/helpfully with AI (compared > to cleaner / less complex code). Secondly, I want to remind us that the > Overseer has been disable-able, in a sense, since 10.0 -- also known > internally as "distributed cluster processing". Overseer complexity > definitely motivated the design. IMO having worked with SolrCloud > internals for years, I think the singular best thing for SolrCloud's health > and maintainability is pursuing that and the eventual demise of the > Overseer. While it may not be the default, the remaining concerns linked > to https://issues.apache.org/jira/browse/SOLR-17293 should not greatly > affect most users. > > Please... enable it for some (all?) of your Solr 10 clouds. It's tempting > to look closer at your specific points, but at some point we have to give > up on the Overseer and embrace the new thing. I've been there... working > on Overseer problems... hoping one day to be done with it. Perhaps you can > help that along by at least simply *using* the alternative. > > On Mon, Jul 6, 2026 at 8:19 PM Luke Kot-Zaniewski (BLOOMBERG/ 919 3RD A) > via dev <[email protected]> wrote: > > Hey All, > > My team has run into a pretty nasty issue with overseer > election in Solr 10, likely caused by the recent migration > to Curator. I've given a pretty detailed account of it here: > https://issues.apache.org/jira/browse/SOLR-18301 > > The crux of the problem is that it's now possible, and even > likely, to have an orphaned/zombie overseer leader node > which permanently blocks the overseer election from > converging until next restart of the stuck overseer. > The key difference from 9.X here is that we are running > through leader election routine across the same zk session > whereas before it would only happen across different sessions. > The logs get spammed with election retries until the > process finally StackOverflows. I've seen this happen on > 4-5 different clouds so am pretty confident we're onto > something here. I've also managed to recreate the issue: > > > https://github.com/kotman12/solr/commit/9c0390bf9ac2f6370b3630a2c7c4955fc3da042c > > Digging through this I noticed several ways Overseer > election is a bit different: > > 1. Overseer cancelElection does not clean up the overseer > leader "registration" node. Instead a special function > invoked only by the "OverseerExitThread" is tasked with > this (checkIfIamStillLeader). In shard leader elections > the cancelElection takes care of both the leader node > as well as the election node. > > 2. There seem to be several competing pathways to claim > overseer leadership from the same node; the "OverseerExitThread" > as well as various callbacks registered with curator which makes > this hard to untangle. > > 3. The version guards in overseer election code are straight > up incorrect and worked incidentally before because we'd > never invoke them since session expiry would delete the leader > node and the OverseerExitThread would have nothing to delete. > > I'd like some feedback from the community on how to fix this. > I can think of some various improvements that would work but > also realize I may be missing context because of how tangled > this is. Some improvements I can think of: > > 1. Fix the version checks so that we look at the parent version > instead of the leader node version (which is always 0). This > seems to be a no-brainer. > > 2. Once you write an overseer leader node you either must call > overseer.start() or, if you cannot, you must delete that node > you just created. It should be impossible to claim leadership > in zk but then not act on it. One could also argue that zkClient > makePath(leaderPath,...) should be pulled *inside* the > synchronized block and only invoked *after* the isClosed check > although this has other implications. I know from reading thru > SOLR-16116 that David had some ideas regarding making isClosed > backed by a RW-lock which could maybe obviate the whole > synchronized business anyway? (getting side-tracked...) > > 3. The whole concept of OverseerExitThread is interesting. I > traced it back to https://issues.apache.org/jira/browse/SOLR-5859 > It seems the main idea is the overseer can be killed from > a special cluster operation QUIT and so having this generic > clean-up routine that gets invoked on *any* exit (not just > an exit triggered by some zk event) was desirable. It gets > put on its own dedicated thread so as to not propagate > any interrupts from the context that spawned it. I still find > it interesting that there are call paths that can delete > the leader election node but not the leader registration > node and this just seems like bad design. In our case it > resulted in election nodes that had sequence numbers in the > *millions* (that's how many retries it took to overflow) > but the overseer leader node itself was stuck on "7". > > 4. I am not sure if during the curator migration it was intentional > to close election contexts on *any* disconnect from zk but this > seems to differ from what was done previously, namely only on > session expiry. Given how much churn leader election causes > in general I wonder if this is something we can discuss. It > could very well be that I am missing something here but it > seems like a big decision and I couldn't find any discussion > about it. For reference this is how onReconnect was triggered > before, notice the "Expired" block: > > > https://github.com/apache/solr/blob/c2091d0258400c9064b8c67fe0d974e367ecccfd/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ConnectionManager.java#L156-L200 > > Any help/insights/opinions are very welcome. This seems like a > big bug to me and I'd like to patch it ASAP so maybe we take an > iterative approach to the solution. I'm also surprised no-one > else ran into this. Is everyone running with PRS or something? > Maybe the real solution is sunset overseer? > > Thanks, > Luke > > >
