You actually capture most of the history of cloud there AB. ZK is the heart of the system. It’s a rare chance you get the time or financing to lay that out on something that will be used.
I didn’t get it done, changed jobs, and that mostly closed the window on that. Then you have a poor heart that would take a god amount of time and experience for anyone to really fully understand all the nuts and bolts of, even if you stood it up. And it’s about the equivalent of a poorly written concurrent program. So when you come along and try to put something like autoscaling on it, it’s going to subvert you the whole way. And unless you are going to change auto scaling to discover and rework all the problems in the heart of the system, not a lot you can do about it. And that completely ignores the overseer end of it. It’s a shame, I could setup a great heart to put something like auto scaling on for you now. But the ship has sailed. Very hard to claw that back and the world has adjusted to to getting what they can from what is. But yeah, curator is a huge improvement on a variety of those issues. And I invested enough into to know it’s good. It’s fast. It’s better and more apis and algorithms - documented. Maintained and pushed forward by a separate group dedicated to the task. But I can tell you, it’s by no means some kind of Rubik’s cube, but it is no small lift. Mark On Wed, Sep 29, 2021 at 9:13 AM Mark Miller <[email protected]> wrote: > I very much agree. That code is the root of a very surprising amount of > evil and has been for a surprisingly long time. > > There is a long list of reasons that I won’t iterate of why I don’t see > that as likely happening though - just starting with Ive brought it up to > various people over a couple years and gotten pushback just at the top. > Roughly, it’s on the scale of work and invasiveness, even with some > incremental paths, that I don’t see the path or resources to seriously > consider it myself. You can go back through jira history for quite a while > before you find that kind of item not looking out of place. > > Mark > > On Wed, Sep 29, 2021 at 2:05 AM Andrzej Białecki <[email protected]> wrote: > >> +1 to start working towards using Curator, this is long overdue and >> sooner or later we need to eat this frog - as you dig deeper and deeper it >> turns out that many issues in Solr can be attributed to our home-grown ZK >> code, there are maybe 2 people on the Solr team who understand what’s going >> on there (and I’m certainly not one of them!). And the maintenance cost is >> just too high over time. >> >> — >> >> Andrzej Białecki >> >> On 28 Sep 2021, at 21:31, Mark Miller <[email protected]> wrote: >> >> P.S. this is not actually the zookeeper design I would submit to any >> competition :) >> >> I’ve gone different routes in addressing the zookeeper short fall. This >> one is relatively easy, impactful and isolated for the right developer. >> >> Personally, with fewer scale and isolation limits, by the far the best >> thing I’ve done is remove almost all of our zk recipes and custom stuff and >> use Apache curator and replace our stuff as well as improve and expand on >> things using their large stable of well behaving recipes. I don’t think raw >> zookeeper is good for a project of more than a few people at most. But I >> wouldn’t toss that out there, it’s a much larger undertaking, no one is >> going to bite on that in passing. >> >> Mark >> -- >> - Mark >> >> http://about.me/markrmiller >> >> >> -- > - Mark > > http://about.me/markrmiller > -- - Mark http://about.me/markrmiller
