There is also work to do on test information, just a lot to try to cover, not a deep need to delve into all the crevices. But just to kind get on relative similar pages, compare goals and expectations and bars.
In the current world, it's very hard to evaluate many things to my current satisfaction, just because of the surface of things that are affected and the relative low value of the tests. I've worked tremendously to tons more value from the tests. And even then, they are of still limited value. That's part of why its important that when you clear up all the collection and state and core baggage, you move to large scale collections stuff. Not just to find small holes and race conditions - but because of the surprising stuff that can survive much more solid tests and supler looping and hammering at low scale. The stuff you will can often and easily be like, woah? How does that not pop until you push this hard? That level of off? Yeesh. That's part and parcel of some of the difficulties in making changes that users won't eat. You can solidify tests and do lots and lots extra - and I'm mind boggled but what gets through. On Tue, Apr 13, 2021 at 5:34 PM Mark Miller <[email protected]> wrote: > Didn’t mean that issue - couple keyboard jams. But yeah, my last comment > is an important one. > > I mean to and intended to get a little l, and over time much further, > because any chunks or isolation is well below the intended goal and point. > > When I first started down this path, I was making drastic test > improvements - moved into drastic production improvements. Feeling great. > Just picking one thing after another as you gain momentum. > > That’s a world I just come back with, start shoving in. Over time and > with deeper inspection, that’s where things started to change. I have to > kind of run through, get out pretext and storyline, but individual > improvements and optimizations and insights or discoveries - that’s simply > normal and kills to bring home. It’s much deeper in the journey that I > start to realize we have some things to overcome greater than addressing > the Overseer ir making things more efficient and faster. I’ve spent months > and months before in wonderland of improvement. Great stage. It’s really > the whole story I’m after vs any chapter or section. They are all > independently lacking in what I want to convey. > > On Tue, Apr 13, 2021 at 3:29 PM Mark Miller <[email protected]> wrote: > >> Thats it for a short bit. Pulled in too many directions at the moment as >> I try and take advantage of my wife’s absence to escape this current >> hellish commitment deficit, def more coming, most usefully everything >> already pre promised. >> >> But I’ll just quickly add - a solid in investment in developer >> productivity and introspection and tools - all this change and forward >> movement fear - that’s how you address that. >> >> You need to be able to click buttons and make tons happen. Click another >> button and get useful summaries from all the logs that where dumped. Flip a >> switch and get those summaries focused on your current focus. >> >> Have different metrics for development than the sea that are currently >> available for users. Have all that super simple for the devs to setup and >> use. So if you are working ok leader election, you can glance over and see >> how many cores are made how fast. Thread stats how many cues enter zk >> registration and started leader election. Failed leader election. How >> many things happened that are part of the chain. How many cores were >> created. State updates requested. Written. How many state published done. >> How many leaders elected. How many parts of the leader election chain done. >> Output to a modern metrics dashboard and shareable with other devs. You >> need quick and targeted and summarized feedback and insight, easily >> accesible introspection and summarization that comes out as fast and easy >> as a human can iterate. You need it. It’s a complex, opaque, wild system. >> But if you have an easy and valuable, and directable gaze into it - it’s a >> simple system. With loads and loads of high value low hanging fruit that >> takes no brilliance to harvest. This is how you move with momentum and >> confidence and address the fears. You build the tools you need to see into >> darkness and you make them efficient and easy and effective. >> >> On Tue, Apr 13, 2021 at 12:59 PM Mark Miller <[email protected]> >> wrote: >> >>> Sorry, have to work on my lazy iPhone typos. >>> >>> But also FYI, everything I’m laying out here, some of it may seem kind >>> of out of scope, or rolling on a negative light, but I promise it’s just >>> all part of the ramp and build of information that really is the base of >>> understanding some of the paths you have to end up at to truly address even >>> some of the more basic things you want to address - if you want to do it >>> for real, in an intense and reliable environment. >>> >>> It’s not so much you have to solve and get to the bottom of everything >>> to solve some of these more basic issues - depending on how well you need >>> to solve them - but that the direction and paths are heavily weighted in >>> potential and goodness based on all the surrounding context and issues that >>> they will have to hold up against. >>> >>> On Tue, Apr 13, 2021 at 12:50 PM Mark Miller <[email protected]> >>> wrote: >>> >>>> So now we start to move into the brighter side of the equation on >>>> zookeeper. Again, plenty of low level and code will fill in. >>>> >>>> But the brighter side. And the kind of pertinent part, where it’s >>>> almost impossible to assess the issues you are referring into, without >>>> fairly drastic performance and efficiency improvement as well. >>>> >>>> And really, as I’ll get into, many ways to handle zk and all that, been >>>> up and down the road, but a truly fantastic and scalable and beautiful way, >>>> is just so easy and impactful and problem eliminating. >>>> >>>> I would essentially and easier and certainly far better path than what >>>> you even get by simply using zookeeper in a more correct and common manner. >>>> >>>> On Tue, Apr 13, 2021 at 11:33 AM Mark Miller <[email protected]> >>>> wrote: >>>> >>>>> The dumping continues on Zookeeper and resources - I've got a few more >>>>> to post, will get into a little on the simple, fast, correct way things >>>>> are >>>>> meant to regarding Zookeeper after a little more of the trouble and then >>>>> jump over to something else. >>>>> >>>>> On Tue, Apr 13, 2021 at 9:10 AM Mark Miller <[email protected]> >>>>> wrote: >>>>> >>>>>> Okay, I've got a few videos to start, first one up, things will kind >>>>>> of ramp a bit. Plenty of text and break out and code lined up, >>>>>> being organized, edited, etc, but I'll shotgun into some overview videos >>>>>> in >>>>>> the various parent topics before that all fills in. Short initial one in, >>>>>> plenty coming on ZooKeeper and related. >>>>>> >>>>>> https://github.com/markrmiller/solr/issues/2 >>>>>> >>>>>> On Tue, Apr 13, 2021 at 3:56 AM Mark Miller <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Okay Ilan, you seem like a reasonable guy, smart guy, don't yet have >>>>>>> a defensive posture towards the state of the world, preconceived biases >>>>>>> against me - let's try to trip the light fantastic and hard crash this >>>>>>> thing.The slow roll I'm wrapping up can fill in. >>>>>>> >>>>>>> I'm gonna link bombs and jet around a bit. >>>>>>> >>>>>>> Let's kind of root of this at "shards with no leader, replicas not >>>>>>> recovering, non functional Overseer etc" >>>>>>> >>>>>>> You can't really get anywhere without that at least workable. You >>>>>>> can't get to scale and performance and understandability and many >>>>>>> necessary >>>>>>> improvements without it at least semi solid. You can't obtain >>>>>>> reliability >>>>>>> and greatness without it being ridiculously solid. To test and verify >>>>>>> and >>>>>>> explore that, you'll have quite a challenge avoiding joining my >>>>>>> contest. So >>>>>>> it's a good lead. But also, it's an undercurrent through the system. The >>>>>>> connections and loop backs and dependencies and cycles - you can try to >>>>>>> say >>>>>>> what about this or that, but in the end we end up talking about the same >>>>>>> thing. The core behavior of the whole thing, that underpins and connects >>>>>>> everything. That can be addressed in many different ways, keeping >>>>>>> things, >>>>>>> removing things, changing things - you could do it many ways. "The >>>>>>> overseer >>>>>>> is not the problem", it must sound funny to you. But the Overseer as a >>>>>>> concept has no problem. Even the poor current design can essentially >>>>>>> work. >>>>>>> 30 designs can work better, other designs can work too. The root issues >>>>>>> around it don't stop at its borders. Anyway, let's do it. I'll dump the >>>>>>> path here. Let's try to blast a litte. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>> On Sun, Apr 11, 2021 at 7:16 AM Ilan Ginzburg <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>> Do you consider performance/scale of SolrCloud or corectness (shards >>>>>>>>> with no leader, replicas not recovering, non functional Overseer etc) >>>>>>>>> as >>>>>>>>> the most important areas for improvement if we were to move to a v2? >>>>>>>>> >>>>>>>>> >>>>>>>>> Ilan >>>>>>>>> >>>>>>>>> Le sam. 10 avr. 2021 à 22:37, Mark Miller <[email protected]> >>>>>>>>> a écrit : >>>>>>>>> >>>>>>>>>> Now a I was saying ... SolrCloud2. Well, some people don’t like >>>>>>>>>> the name - fair enough - I don’t like purple much and I didn’t invent >>>>>>>>>> either... so Solr2. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I have some thoughts, a few things to offer. Yes, the guy behind >>>>>>>>>> a lot of the current garbage. They asked me my thoughts on this >>>>>>>>>> thing on >>>>>>>>>> day one - I told them I don’t have a single one, let me have a go >>>>>>>>>> and I’ll >>>>>>>>>> get back to you. Done and done. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> So yeah, Solr2. Wait, what’s that ... ? Loud dance floor out here >>>>>>>>>> - if I could have the floor for just a moment. I’ve put just a >>>>>>>>>> little bit >>>>>>>>>> of time and effort in before grabbing the mic. A modicum. My ego, my >>>>>>>>>> job, >>>>>>>>>> my status, my history, really not part of the equation, so if I could >>>>>>>>>> request a few lines before the rebuttal and correction and >>>>>>>>>> redirection. >>>>>>>>>> No hurry for those things, they are patient characters. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Anyway, give me a moment, and in return I’ll spare you the long >>>>>>>>>> winded diversions. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Having considered Solr2 for some time, I see all kinds of >>>>>>>>>> roadblocks and restraints and limitations. I see a path fraught with >>>>>>>>>> the >>>>>>>>>> potential to mimic previous paths. And my goodness, I’m getting old. >>>>>>>>>> Fresh, >>>>>>>>>> promising paths please. So Ive invested some time and effort to >>>>>>>>>> establish >>>>>>>>>> an escape route from the safe, conservative hack and slash through >>>>>>>>>> the >>>>>>>>>> jungle that gets less safe the deeper we dive. I’d simply tell you >>>>>>>>>> about >>>>>>>>>> it, but jungle stories are all heart and no soul and we will all >>>>>>>>>> trade them >>>>>>>>>> all day. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> So I’ve got some notes and code and maps and journals and crap >>>>>>>>>> instead. Compiled from a couple expeditions. From before spelunking >>>>>>>>>> made me >>>>>>>>>> an old man. So my machete isn’t solo diving any new trails, single >>>>>>>>>> handedly >>>>>>>>>> slaying jungle cats any longer. The blade just keeps getting duller. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> But I’ve got these previous materials. I’ll lay them out. Take a >>>>>>>>>> look. If we can explore and discuss with just a passing courtesy of >>>>>>>>>> respect >>>>>>>>>> for our relative time and investment and focus put in before letting >>>>>>>>>> loose, >>>>>>>>>> I’m sure we can move beyond basic software counter meandering quickly >>>>>>>>>> enough to actually enter the jungle. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I’ll try and lay out some evidence to help here. For instance, >>>>>>>>>> that zookeeper is not the issue. That the overseer is not the issue. >>>>>>>>>> That >>>>>>>>>> modules and multitudes of features are not the issue. Previous >>>>>>>>>> efforts are >>>>>>>>>> not the issue. Object oriented development and agile frameworks are >>>>>>>>>> not the >>>>>>>>>> issue. And yet all remain pertinent software development >>>>>>>>>> conversation. Bike >>>>>>>>>> shed rainbows. You can make beautiful multicolored bike sheds with >>>>>>>>>> all of >>>>>>>>>> them. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> But discuss colors with someone else please. I can’t do it >>>>>>>>>> anymore. But I can discuss some jungle hack and slash momentum >>>>>>>>>> and trail blazing. Give me a moment. Time is fairly irrelevant >>>>>>>>>> on this dance floor. Tomorrow. Next year. 3 years. Solr 42. It’s all >>>>>>>>>> the >>>>>>>>>> same timeline when the time comes. Hell, my brain feels equally >>>>>>>>>> everywhere >>>>>>>>>> in all the timelines anyway. So one moment, and I’ll point you to my >>>>>>>>>> speech, my journals, my old jungle dune buggy - I did spend a moment >>>>>>>>>> or two >>>>>>>>>> once or twice preparing the damn thing - just a moment or two, just >>>>>>>>>> 1, 2 or >>>>>>>>>> 3 times - and then the microphone is all yours. But please, don’t >>>>>>>>>> talk to >>>>>>>>>> me about rainbows or shades of magenta, I’m a jungle explorer, not a >>>>>>>>>> Sherwin Williams employee. Don’t cover me with paint - I’m a roto >>>>>>>>>> spinning >>>>>>>>>> paint spewing machine. And I’d love to spare you the paint ball war >>>>>>>>>> as >>>>>>>>>> well. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Links coming. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Mark >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> - Mark >>>>>>>>>> >>>>>>>>>> http://about.me/markrmiller >>>>>>>>>> >>>>>>>>> -- >>>>>>>> - Mark >>>>>>>> >>>>>>>> http://about.me/markrmiller >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> - Mark >>>>>> >>>>>> http://about.me/markrmiller >>>>>> >>>>> >>>>> >>>>> -- >>>>> - Mark >>>>> >>>>> http://about.me/markrmiller >>>>> >>>> -- >>>> - Mark >>>> >>>> http://about.me/markrmiller >>>> >>> -- >>> - Mark >>> >>> http://about.me/markrmiller >>> >> -- >> - Mark >> >> http://about.me/markrmiller >> > -- > - Mark > > http://about.me/markrmiller > -- - Mark http://about.me/markrmiller
