And to finish off the collection creation scale value, I don’t push to reach there because it’s good for finding holes and races in state updates and collection creation - though it is. It’s totally critical to all the other work.
There is actually a lot more with potential to isolate because of it for one, but the huge thing involves allowing all that effort and time finish in a gold state, in a state that is not still of devilish issues. A lot of core stuff I did - that survive and looks good in tests an led various, there ends remaining all these pretty important issues to straighten out and address and work through. And intense varied afford and testing - completely unknown to me. And stuff totally outside creating collections. But you push there, it pops out easily and reliably at the scale. So no you see these things, it makes a lot of work that in another would be hard to isolate more attainable, and the switch from mostly invisible to easily viable and reproducible makes the assessing easy in a way that perhaps they did ghost up before in various rare areas and times - but that type of thing is a world harder to identify and track down and verify solutions. On Tue, Apr 13, 2021 at 8:40 PM Mark Miller <[email protected]> wrote: > Didn’t really get to the root of the leader election fix yet either ... > > But a couple simple steps. One, even if you solve missing overseer leader, > or shard , it’s a system that can fail too easily, often much too subtly > for any notice. Have spent a lot of time improving hardening. And > previously 5 min getting out of the box great from Curator. And it’s > another thing you can harden like it’s a rock at low scale, that let you > down all over at decent collection scale and load. Not so much curators > impl, although we do still own a critical piece of it. That part I have > been able to harden effective though. If I was stuck without impl, fairly > soon I’d toss for a fresh perspective start. Already I’ve pretty heavily > altered it over time. > > Lots of easy to things that make it much better on our best effort accept > of it beating races and being the legit leader though, lower details and > what not best left to the code. > > Major simple improvements that prevent getting it wrong are mostly late > in anyone addressing due to current whacky impl and code design. But > large steps forward on ensuring correctness in terms of not losing a > leader that you shouldn’t - but IMHO, very, very hard to beat having > something like Curator take over the heavy lift part. Easy to do, faster > and much more solid, maintained and kept up by someone else. > > > > On Tue, Apr 13, 2021 at 6:20 PM Mark Miller <[email protected]> wrote: > >> There is also work to do on test information, just a lot to try to cover, >> not a deep need to delve into all the crevices. But just to kind get on >> relative similar pages, compare goals and expectations and bars. >> >> In the current world, it's very hard to evaluate many things to my >> current satisfaction, just because of the surface of things that are >> affected and the relative low value of the tests. I've worked tremendously >> to tons more value from the tests. And even then, they are of still limited >> value. That's part of why its important that when you clear up all the >> collection and state and core baggage, you move to large scale collections >> stuff. Not just to find small holes and race conditions - but because of >> the surprising stuff that can survive much more solid tests and supler >> looping and hammering at low scale. The stuff you will can often and easily >> be like, woah? How does that not pop until you push this hard? That level >> of off? Yeesh. That's part and parcel of some of the difficulties in making >> changes that users won't eat. You can solidify tests and do lots and lots >> extra - and I'm mind boggled but what gets through. >> >> On Tue, Apr 13, 2021 at 5:34 PM Mark Miller <[email protected]> >> wrote: >> >>> Didn’t mean that issue - couple keyboard jams. But yeah, my last comment >>> is an important one. >>> >>> I mean to and intended to get a little l, and over time much further, >>> because any chunks or isolation is well below the intended goal and point. >>> >>> When I first started down this path, I was making drastic test >>> improvements - moved into drastic production improvements. Feeling great. >>> Just picking one thing after another as you gain momentum. >>> >>> That’s a world I just come back with, start shoving in. Over time and >>> with deeper inspection, that’s where things started to change. I have to >>> kind of run through, get out pretext and storyline, but individual >>> improvements and optimizations and insights or discoveries - that’s simply >>> normal and kills to bring home. It’s much deeper in the journey that I >>> start to realize we have some things to overcome greater than addressing >>> the Overseer ir making things more efficient and faster. I’ve spent months >>> and months before in wonderland of improvement. Great stage. It’s really >>> the whole story I’m after vs any chapter or section. They are all >>> independently lacking in what I want to convey. >>> >>> On Tue, Apr 13, 2021 at 3:29 PM Mark Miller <[email protected]> >>> wrote: >>> >>>> Thats it for a short bit. Pulled in too many directions at the moment >>>> as I try and take advantage of my wife’s absence to escape this current >>>> hellish commitment deficit, def more coming, most usefully everything >>>> already pre promised. >>>> >>>> But I’ll just quickly add - a solid in investment in developer >>>> productivity and introspection and tools - all this change and forward >>>> movement fear - that’s how you address that. >>>> >>>> You need to be able to click buttons and make tons happen. Click >>>> another button and get useful summaries from all the logs that where >>>> dumped. Flip a switch and get those summaries focused on your current >>>> focus. >>>> >>>> Have different metrics for development than the sea that are currently >>>> available for users. Have all that super simple for the devs to setup and >>>> use. So if you are working ok leader election, you can glance over and see >>>> how many cores are made how fast. Thread stats how many cues enter zk >>>> registration and started leader election. Failed leader election. How >>>> many things happened that are part of the chain. How many cores were >>>> created. State updates requested. Written. How many state published done. >>>> How many leaders elected. How many parts of the leader election chain done. >>>> Output to a modern metrics dashboard and shareable with other devs. You >>>> need quick and targeted and summarized feedback and insight, easily >>>> accesible introspection and summarization that comes out as fast and easy >>>> as a human can iterate. You need it. It’s a complex, opaque, wild system. >>>> But if you have an easy and valuable, and directable gaze into it - it’s a >>>> simple system. With loads and loads of high value low hanging fruit that >>>> takes no brilliance to harvest. This is how you move with momentum and >>>> confidence and address the fears. You build the tools you need to see into >>>> darkness and you make them efficient and easy and effective. >>>> >>>> On Tue, Apr 13, 2021 at 12:59 PM Mark Miller <[email protected]> >>>> wrote: >>>> >>>>> Sorry, have to work on my lazy iPhone typos. >>>>> >>>>> But also FYI, everything I’m laying out here, some of it may seem kind >>>>> of out of scope, or rolling on a negative light, but I promise it’s just >>>>> all part of the ramp and build of information that really is the base of >>>>> understanding some of the paths you have to end up at to truly address >>>>> even >>>>> some of the more basic things you want to address - if you want to do it >>>>> for real, in an intense and reliable environment. >>>>> >>>>> It’s not so much you have to solve and get to the bottom of everything >>>>> to solve some of these more basic issues - depending on how well you need >>>>> to solve them - but that the direction and paths are heavily weighted in >>>>> potential and goodness based on all the surrounding context and issues >>>>> that >>>>> they will have to hold up against. >>>>> >>>>> On Tue, Apr 13, 2021 at 12:50 PM Mark Miller <[email protected]> >>>>> wrote: >>>>> >>>>>> So now we start to move into the brighter side of the equation on >>>>>> zookeeper. Again, plenty of low level and code will fill in. >>>>>> >>>>>> But the brighter side. And the kind of pertinent part, where it’s >>>>>> almost impossible to assess the issues you are referring into, without >>>>>> fairly drastic performance and efficiency improvement as well. >>>>>> >>>>>> And really, as I’ll get into, many ways to handle zk and all that, >>>>>> been up and down the road, but a truly fantastic and scalable and >>>>>> beautiful >>>>>> way, is just so easy and impactful and problem eliminating. >>>>>> >>>>>> I would essentially and easier and certainly far better path than >>>>>> what you even get by simply using zookeeper in a more correct and common >>>>>> manner. >>>>>> >>>>>> On Tue, Apr 13, 2021 at 11:33 AM Mark Miller <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> The dumping continues on Zookeeper and resources - I've got a few >>>>>>> more to post, will get into a little on the simple, fast, correct way >>>>>>> things are meant to regarding Zookeeper after a little more of the >>>>>>> trouble >>>>>>> and then jump over to something else. >>>>>>> >>>>>>> On Tue, Apr 13, 2021 at 9:10 AM Mark Miller <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Okay, I've got a few videos to start, first one up, things will >>>>>>>> kind of ramp a bit. Plenty of text and break out and code lined up, >>>>>>>> being organized, edited, etc, but I'll shotgun into some overview >>>>>>>> videos in >>>>>>>> the various parent topics before that all fills in. Short initial one >>>>>>>> in, >>>>>>>> plenty coming on ZooKeeper and related. >>>>>>>> >>>>>>>> https://github.com/markrmiller/solr/issues/2 >>>>>>>> >>>>>>>> On Tue, Apr 13, 2021 at 3:56 AM Mark Miller <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Okay Ilan, you seem like a reasonable guy, smart guy, don't yet >>>>>>>>> have a defensive posture towards the state of the world, >>>>>>>>> preconceived biases against me - let's try to trip the light >>>>>>>>> fantastic and >>>>>>>>> hard crash this thing.The slow roll I'm wrapping up can fill in. >>>>>>>>> >>>>>>>>> I'm gonna link bombs and jet around a bit. >>>>>>>>> >>>>>>>>> Let's kind of root of this at "shards with no leader, replicas not >>>>>>>>> recovering, non functional Overseer etc" >>>>>>>>> >>>>>>>>> You can't really get anywhere without that at least workable. You >>>>>>>>> can't get to scale and performance and understandability and many >>>>>>>>> necessary >>>>>>>>> improvements without it at least semi solid. You can't obtain >>>>>>>>> reliability >>>>>>>>> and greatness without it being ridiculously solid. To test and verify >>>>>>>>> and >>>>>>>>> explore that, you'll have quite a challenge avoiding joining my >>>>>>>>> contest. So >>>>>>>>> it's a good lead. But also, it's an undercurrent through the system. >>>>>>>>> The >>>>>>>>> connections and loop backs and dependencies and cycles - you can try >>>>>>>>> to say >>>>>>>>> what about this or that, but in the end we end up talking about the >>>>>>>>> same >>>>>>>>> thing. The core behavior of the whole thing, that underpins and >>>>>>>>> connects >>>>>>>>> everything. That can be addressed in many different ways, keeping >>>>>>>>> things, >>>>>>>>> removing things, changing things - you could do it many ways. "The >>>>>>>>> overseer >>>>>>>>> is not the problem", it must sound funny to you. But the Overseer as a >>>>>>>>> concept has no problem. Even the poor current design can essentially >>>>>>>>> work. >>>>>>>>> 30 designs can work better, other designs can work too. The root >>>>>>>>> issues >>>>>>>>> around it don't stop at its borders. Anyway, let's do it. I'll dump >>>>>>>>> the >>>>>>>>> path here. Let's try to blast a litte. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>> On Sun, Apr 11, 2021 at 7:16 AM Ilan Ginzburg <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>> Do you consider performance/scale of SolrCloud or corectness >>>>>>>>>>> (shards with no leader, replicas not recovering, non functional >>>>>>>>>>> Overseer >>>>>>>>>>> etc) as the most important areas for improvement if we were to move >>>>>>>>>>> to a >>>>>>>>>>> v2? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Ilan >>>>>>>>>>> >>>>>>>>>>> Le sam. 10 avr. 2021 à 22:37, Mark Miller <[email protected]> >>>>>>>>>>> a écrit : >>>>>>>>>>> >>>>>>>>>>>> Now a I was saying ... SolrCloud2. Well, some people don’t like >>>>>>>>>>>> the name - fair enough - I don’t like purple much and I didn’t >>>>>>>>>>>> invent >>>>>>>>>>>> either... so Solr2. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I have some thoughts, a few things to offer. Yes, the guy >>>>>>>>>>>> behind a lot of the current garbage. They asked me my thoughts on >>>>>>>>>>>> this >>>>>>>>>>>> thing on day one - I told them I don’t have a single one, let me >>>>>>>>>>>> have a go >>>>>>>>>>>> and I’ll get back to you. Done and done. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> So yeah, Solr2. Wait, what’s that ... ? Loud dance floor out >>>>>>>>>>>> here - if I could have the floor for just a moment. I’ve put just >>>>>>>>>>>> a little >>>>>>>>>>>> bit of time and effort in before grabbing the mic. A modicum. My >>>>>>>>>>>> ego, my >>>>>>>>>>>> job, my status, my history, really not part of the equation, so if >>>>>>>>>>>> I could >>>>>>>>>>>> request a few lines before the rebuttal and correction and >>>>>>>>>>>> redirection. >>>>>>>>>>>> No hurry for those things, they are patient characters. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Anyway, give me a moment, and in return I’ll spare you the long >>>>>>>>>>>> winded diversions. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Having considered Solr2 for some time, I see all kinds of >>>>>>>>>>>> roadblocks and restraints and limitations. I see a path fraught >>>>>>>>>>>> with the >>>>>>>>>>>> potential to mimic previous paths. And my goodness, I’m getting >>>>>>>>>>>> old. Fresh, >>>>>>>>>>>> promising paths please. So Ive invested some time and effort to >>>>>>>>>>>> establish >>>>>>>>>>>> an escape route from the safe, conservative hack and slash through >>>>>>>>>>>> the >>>>>>>>>>>> jungle that gets less safe the deeper we dive. I’d simply tell you >>>>>>>>>>>> about >>>>>>>>>>>> it, but jungle stories are all heart and no soul and we will all >>>>>>>>>>>> trade them >>>>>>>>>>>> all day. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> So I’ve got some notes and code and maps and journals and crap >>>>>>>>>>>> instead. Compiled from a couple expeditions. From before >>>>>>>>>>>> spelunking made me >>>>>>>>>>>> an old man. So my machete isn’t solo diving any new trails, single >>>>>>>>>>>> handedly >>>>>>>>>>>> slaying jungle cats any longer. The blade just keeps getting >>>>>>>>>>>> duller. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> But I’ve got these previous materials. I’ll lay them out. Take >>>>>>>>>>>> a look. If we can explore and discuss with just a passing courtesy >>>>>>>>>>>> of >>>>>>>>>>>> respect for our relative time and investment and focus put in >>>>>>>>>>>> before >>>>>>>>>>>> letting loose, I’m sure we can move beyond basic software counter >>>>>>>>>>>> meandering quickly enough to actually enter the jungle. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I’ll try and lay out some evidence to help here. For instance, >>>>>>>>>>>> that zookeeper is not the issue. That the overseer is not the >>>>>>>>>>>> issue. That >>>>>>>>>>>> modules and multitudes of features are not the issue. Previous >>>>>>>>>>>> efforts are >>>>>>>>>>>> not the issue. Object oriented development and agile frameworks >>>>>>>>>>>> are not the >>>>>>>>>>>> issue. And yet all remain pertinent software development >>>>>>>>>>>> conversation. Bike >>>>>>>>>>>> shed rainbows. You can make beautiful multicolored bike sheds with >>>>>>>>>>>> all of >>>>>>>>>>>> them. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> But discuss colors with someone else please. I can’t do it >>>>>>>>>>>> anymore. But I can discuss some jungle hack and slash >>>>>>>>>>>> momentum and trail blazing. Give me a moment. Time is fairly >>>>>>>>>>>> irrelevant on this dance floor. Tomorrow. Next year. 3 years. Solr >>>>>>>>>>>> 42. It’s >>>>>>>>>>>> all the same timeline when the time comes. Hell, my brain feels >>>>>>>>>>>> equally >>>>>>>>>>>> everywhere in all the timelines anyway. So one moment, and I’ll >>>>>>>>>>>> point you >>>>>>>>>>>> to my speech, my journals, my old jungle dune buggy - I did spend >>>>>>>>>>>> a moment >>>>>>>>>>>> or two once or twice preparing the damn thing - just a moment or >>>>>>>>>>>> two, just >>>>>>>>>>>> 1, 2 or 3 times - and then the microphone is all yours. But >>>>>>>>>>>> please, don’t >>>>>>>>>>>> talk to me about rainbows or shades of magenta, I’m a jungle >>>>>>>>>>>> explorer, not >>>>>>>>>>>> a Sherwin Williams employee. Don’t cover me with paint - I’m a roto >>>>>>>>>>>> spinning paint spewing machine. And I’d love to spare you the >>>>>>>>>>>> paint ball >>>>>>>>>>>> war as well. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Links coming. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Mark >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> - Mark >>>>>>>>>>>> >>>>>>>>>>>> http://about.me/markrmiller >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>> - Mark >>>>>>>>>> >>>>>>>>>> http://about.me/markrmiller >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> - Mark >>>>>>>> >>>>>>>> http://about.me/markrmiller >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> - Mark >>>>>>> >>>>>>> http://about.me/markrmiller >>>>>>> >>>>>> -- >>>>>> - Mark >>>>>> >>>>>> http://about.me/markrmiller >>>>>> >>>>> -- >>>>> - Mark >>>>> >>>>> http://about.me/markrmiller >>>>> >>>> -- >>>> - Mark >>>> >>>> http://about.me/markrmiller >>>> >>> -- >>> - Mark >>> >>> http://about.me/markrmiller >>> >> >> >> -- >> - Mark >> >> http://about.me/markrmiller >> > -- > - Mark > > http://about.me/markrmiller > -- - Mark http://about.me/markrmiller
