Thats it for a short bit. Pulled in too many directions at the moment as I try and take advantage of my wife’s absence to escape this current hellish commitment deficit, def more coming, most usefully everything already pre promised.
But I’ll just quickly add - a solid in investment in developer productivity and introspection and tools - all this change and forward movement fear - that’s how you address that. You need to be able to click buttons and make tons happen. Click another button and get useful summaries from all the logs that where dumped. Flip a switch and get those summaries focused on your current focus. Have different metrics for development than the sea that are currently available for users. Have all that super simple for the devs to setup and use. So if you are working ok leader election, you can glance over and see how many cores are made how fast. Thread stats how many cues enter zk registration and started leader election. Failed leader election. How many things happened that are part of the chain. How many cores were created. State updates requested. Written. How many state published done. How many leaders elected. How many parts of the leader election chain done. Output to a modern metrics dashboard and shareable with other devs. You need quick and targeted and summarized feedback and insight, easily accesible introspection and summarization that comes out as fast and easy as a human can iterate. You need it. It’s a complex, opaque, wild system. But if you have an easy and valuable, and directable gaze into it - it’s a simple system. With loads and loads of high value low hanging fruit that takes no brilliance to harvest. This is how you move with momentum and confidence and address the fears. You build the tools you need to see into darkness and you make them efficient and easy and effective. On Tue, Apr 13, 2021 at 12:59 PM Mark Miller <[email protected]> wrote: > Sorry, have to work on my lazy iPhone typos. > > But also FYI, everything I’m laying out here, some of it may seem kind of > out of scope, or rolling on a negative light, but I promise it’s just all > part of the ramp and build of information that really is the base of > understanding some of the paths you have to end up at to truly address even > some of the more basic things you want to address - if you want to do it > for real, in an intense and reliable environment. > > It’s not so much you have to solve and get to the bottom of everything to > solve some of these more basic issues - depending on how well you need to > solve them - but that the direction and paths are heavily weighted in > potential and goodness based on all the surrounding context and issues that > they will have to hold up against. > > On Tue, Apr 13, 2021 at 12:50 PM Mark Miller <[email protected]> > wrote: > >> So now we start to move into the brighter side of the equation on >> zookeeper. Again, plenty of low level and code will fill in. >> >> But the brighter side. And the kind of pertinent part, where it’s almost >> impossible to assess the issues you are referring into, without fairly >> drastic performance and efficiency improvement as well. >> >> And really, as I’ll get into, many ways to handle zk and all that, been >> up and down the road, but a truly fantastic and scalable and beautiful way, >> is just so easy and impactful and problem eliminating. >> >> I would essentially and easier and certainly far better path than what >> you even get by simply using zookeeper in a more correct and common manner. >> >> On Tue, Apr 13, 2021 at 11:33 AM Mark Miller <[email protected]> >> wrote: >> >>> The dumping continues on Zookeeper and resources - I've got a few more >>> to post, will get into a little on the simple, fast, correct way things are >>> meant to regarding Zookeeper after a little more of the trouble and then >>> jump over to something else. >>> >>> On Tue, Apr 13, 2021 at 9:10 AM Mark Miller <[email protected]> >>> wrote: >>> >>>> Okay, I've got a few videos to start, first one up, things will kind of >>>> ramp a bit. Plenty of text and break out and code lined up, >>>> being organized, edited, etc, but I'll shotgun into some overview videos in >>>> the various parent topics before that all fills in. Short initial one in, >>>> plenty coming on ZooKeeper and related. >>>> >>>> https://github.com/markrmiller/solr/issues/2 >>>> >>>> On Tue, Apr 13, 2021 at 3:56 AM Mark Miller <[email protected]> >>>> wrote: >>>> >>>>> Okay Ilan, you seem like a reasonable guy, smart guy, don't yet have a >>>>> defensive posture towards the state of the world, preconceived biases >>>>> against me - let's try to trip the light fantastic and hard crash this >>>>> thing.The slow roll I'm wrapping up can fill in. >>>>> >>>>> I'm gonna link bombs and jet around a bit. >>>>> >>>>> Let's kind of root of this at "shards with no leader, replicas not >>>>> recovering, non functional Overseer etc" >>>>> >>>>> You can't really get anywhere without that at least workable. You >>>>> can't get to scale and performance and understandability and many >>>>> necessary >>>>> improvements without it at least semi solid. You can't obtain reliability >>>>> and greatness without it being ridiculously solid. To test and verify and >>>>> explore that, you'll have quite a challenge avoiding joining my contest. >>>>> So >>>>> it's a good lead. But also, it's an undercurrent through the system. The >>>>> connections and loop backs and dependencies and cycles - you can try to >>>>> say >>>>> what about this or that, but in the end we end up talking about the same >>>>> thing. The core behavior of the whole thing, that underpins and connects >>>>> everything. That can be addressed in many different ways, keeping things, >>>>> removing things, changing things - you could do it many ways. "The >>>>> overseer >>>>> is not the problem", it must sound funny to you. But the Overseer as a >>>>> concept has no problem. Even the poor current design can essentially work. >>>>> 30 designs can work better, other designs can work too. The root issues >>>>> around it don't stop at its borders. Anyway, let's do it. I'll dump the >>>>> path here. Let's try to blast a litte. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> >>>>> On Sun, Apr 11, 2021 at 7:16 AM Ilan Ginzburg <[email protected]> >>>>>> wrote: >>>>>> >>>>> Do you consider performance/scale of SolrCloud or corectness (shards >>>>>>> with no leader, replicas not recovering, non functional Overseer etc) as >>>>>>> the most important areas for improvement if we were to move to a v2? >>>>>>> >>>>>>> >>>>>>> Ilan >>>>>>> >>>>>>> Le sam. 10 avr. 2021 à 22:37, Mark Miller <[email protected]> a >>>>>>> écrit : >>>>>>> >>>>>>>> Now a I was saying ... SolrCloud2. Well, some people don’t like the >>>>>>>> name - fair enough - I don’t like purple much and I didn’t invent >>>>>>>> either... >>>>>>>> so Solr2. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> I have some thoughts, a few things to offer. Yes, the guy behind a >>>>>>>> lot of the current garbage. They asked me my thoughts on this thing on >>>>>>>> day >>>>>>>> one - I told them I don’t have a single one, let me have a go and I’ll >>>>>>>> get >>>>>>>> back to you. Done and done. >>>>>>>> >>>>>>>> >>>>>>>> So yeah, Solr2. Wait, what’s that ... ? Loud dance floor out here - >>>>>>>> if I could have the floor for just a moment. I’ve put just a little >>>>>>>> bit of >>>>>>>> time and effort in before grabbing the mic. A modicum. My ego, my job, >>>>>>>> my >>>>>>>> status, my history, really not part of the equation, so if I could >>>>>>>> request >>>>>>>> a few lines before the rebuttal and correction and redirection. No >>>>>>>> hurry for those things, they are patient characters. >>>>>>>> >>>>>>>> >>>>>>>> Anyway, give me a moment, and in return I’ll spare you the long >>>>>>>> winded diversions. >>>>>>>> >>>>>>>> >>>>>>>> Having considered Solr2 for some time, I see all kinds of >>>>>>>> roadblocks and restraints and limitations. I see a path fraught with >>>>>>>> the >>>>>>>> potential to mimic previous paths. And my goodness, I’m getting old. >>>>>>>> Fresh, >>>>>>>> promising paths please. So Ive invested some time and effort to >>>>>>>> establish >>>>>>>> an escape route from the safe, conservative hack and slash through the >>>>>>>> jungle that gets less safe the deeper we dive. I’d simply tell you >>>>>>>> about >>>>>>>> it, but jungle stories are all heart and no soul and we will all trade >>>>>>>> them >>>>>>>> all day. >>>>>>>> >>>>>>>> >>>>>>>> So I’ve got some notes and code and maps and journals and crap >>>>>>>> instead. Compiled from a couple expeditions. From before spelunking >>>>>>>> made me >>>>>>>> an old man. So my machete isn’t solo diving any new trails, single >>>>>>>> handedly >>>>>>>> slaying jungle cats any longer. The blade just keeps getting duller. >>>>>>>> >>>>>>>> >>>>>>>> But I’ve got these previous materials. I’ll lay them out. Take a >>>>>>>> look. If we can explore and discuss with just a passing courtesy of >>>>>>>> respect >>>>>>>> for our relative time and investment and focus put in before letting >>>>>>>> loose, >>>>>>>> I’m sure we can move beyond basic software counter meandering quickly >>>>>>>> enough to actually enter the jungle. >>>>>>>> >>>>>>>> >>>>>>>> I’ll try and lay out some evidence to help here. For instance, that >>>>>>>> zookeeper is not the issue. That the overseer is not the issue. That >>>>>>>> modules and multitudes of features are not the issue. Previous efforts >>>>>>>> are >>>>>>>> not the issue. Object oriented development and agile frameworks are >>>>>>>> not the >>>>>>>> issue. And yet all remain pertinent software development conversation. >>>>>>>> Bike >>>>>>>> shed rainbows. You can make beautiful multicolored bike sheds with all >>>>>>>> of >>>>>>>> them. >>>>>>>> >>>>>>>> >>>>>>>> But discuss colors with someone else please. I can’t do it anymore. >>>>>>>> But I can discuss some jungle hack and slash momentum and trail >>>>>>>> blazing. Give me a moment. Time is fairly irrelevant on this >>>>>>>> dance floor. Tomorrow. Next year. 3 years. Solr 42. It’s all the same >>>>>>>> timeline when the time comes. Hell, my brain feels equally everywhere >>>>>>>> in >>>>>>>> all the timelines anyway. So one moment, and I’ll point you to my >>>>>>>> speech, >>>>>>>> my journals, my old jungle dune buggy - I did spend a moment or two >>>>>>>> once or >>>>>>>> twice preparing the damn thing - just a moment or two, just 1, 2 or 3 >>>>>>>> times >>>>>>>> - and then the microphone is all yours. But please, don’t talk to me >>>>>>>> about >>>>>>>> rainbows or shades of magenta, I’m a jungle explorer, not a Sherwin >>>>>>>> Williams employee. Don’t cover me with paint - I’m a roto spinning >>>>>>>> paint >>>>>>>> spewing machine. And I’d love to spare you the paint ball war as well. >>>>>>>> >>>>>>>> >>>>>>>> Links coming. >>>>>>>> >>>>>>>> >>>>>>>> Mark >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> - Mark >>>>>>>> >>>>>>>> http://about.me/markrmiller >>>>>>>> >>>>>>> -- >>>>>> - Mark >>>>>> >>>>>> http://about.me/markrmiller >>>>>> >>>>> >>>> >>>> -- >>>> - Mark >>>> >>>> http://about.me/markrmiller >>>> >>> >>> >>> -- >>> - Mark >>> >>> http://about.me/markrmiller >>> >> -- >> - Mark >> >> http://about.me/markrmiller >> > -- > - Mark > > http://about.me/markrmiller > -- - Mark http://about.me/markrmiller
