It’s hard to see how they are not related. You can look at both as trying to solve largely the same problem - that the Overseer is mind boggling inefficient. And it’s hard to see how pursuing one makes sense with the other. If you were to eliminate the Overseer and essentially distribute most of its duties to be doable by any individual node, the idea of efficient state updates is kind of irrelevant in its current form. It’s somewhat opposing directions given one is, at its core, improving the Overseer and the other is removing it. And both are basically in their infancy in terms of what is left on both approaches. It feels to me like Houston has framed the problem set pretty well.
On Wed, Sep 21, 2022 at 12:15 PM David Smiley <dsmi...@apache.org> wrote: > I think we should have separate threads for these two topics. I'll start a > new thread for Distributed State Updates vs Overseer. If they become > related for certain points then they can be raised here? > > > On Wed, Sep 21, 2022 at 11:44 AM Houston Putman <hous...@apache.org> > wrote: > > > Hey everyone, > > > > We've seen some interesting developments over the last 2 years in the way > > that Solr state and distributed logic is handled. Notably we've seen the > > introduction of PerReplicaStates (PRS) and the Distributed State Updates > > (no overseer). > > > > I think for the health of our code and future maintainability, we should > > really look to decide on what implementations we want to use for State > > management and Distributed operations. Basically do we want to adopt or > > abandon PRS/Distributed State Updates. Note that these are separate > > concepts, so the decision on each will be separate. > > > > I bring this up because I see PRS a lot through the code and it feels > like > > the code is too separate from the original way of managing state. There > is > > a lot of "if (prsEnabled)" logic throughout the core, and its very hard > to > > understand how PRS actually works with this logic spread all over the > > place. If we want to move forward with PRS, then we hopefully would be > able > > to consolidate the logic. > > > > I don't see the Distributed State Update logic nearly as much, but I > > imagine our code can only get cleaner with one implementation versus two. > > > > This is just my opinion, let me know what y'all think about making > > decisions or going forward with the status quo. > > > > - Houston > > > -- - MRM
