Would a SIP with title "Scalable cluster state management" be appropriate for this topic? Where one could lay out the problems faced by current Overseer, state.json and enumerate a roadmap for improvement. It would obviously discuss the existing implementations of removing Overseer and the PRS code, their weaknesses and proposals for solutions (whether it is incremental or greenfield). I'm not involved in deployments at this kind of scale myself, so I don't have much to bring to the table.
Jan > 16. okt. 2025 kl. 00:21 skrev David Smiley <[email protected]>: > > SolrCloud's "PRS" (Per Replica State) mode offers a variation on how > replica state is tracked, affecting how state is stored in ZooKeeper and > certainly affecting SolrCloud code. It solves a many-replica scalability > problem of large collections. Coincidentally, that is exactly the achilles > heel of disabling the Overseer, however PRS didn't go far enough in its > approach (IMO) as there is still replica information in state.json. I love > the conceptual idea / direction of PRS but not the implementation, which is > complex -- complexified further by being bolted onto the existing code > instead of green-field. If we are to imagine making any serious > improvements like this to how a collection's state is structured in > ZooKeeper, I can't imagine realistically doing such work with PRS > existing. I think such work would start with removing PRS without > concern/burden of compatibility. > > Thankfully, PRS is rather invisible to our users -- it's not documented in > the ref guide. I think only Full Story is using it. So it's de > facto "experimental" right now and is subject to change... **and removal**. > > FWIW I'd love to help design its replacement, but that's not the subject of > this post/thread. > > ~ David Smiley > Apache Lucene/Solr Search Developer > http://www.linkedin.com/in/davidwsmiley --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
