Thanks. Features lists are as hard as time frames. I keep a boat load of various experiments and dead ends and potential on the side that would just muddy the waters until the basics are right. And the basics take 95% of the time and effort. So it’s 11:59 pm before I know what gets pulled out, say nothing about what gets to go in.
It’s really just punches to the face until 30 seconds left and then it’s all the sugar in one shot. ForkJoin, full on async, all the last mile on async, JCTools and altérnate implementation experimentation, everything that’s half way impossible to get right until hammer on the system like you only care about performance and scale because thats the only way to expose the ridiculous number of issues that lurk in the slow and resource heavy envs. A decade or two of whack a mole (done it) or a death drive to efficiency. You try to play around with async IO and suspending requests when everything else is not right and you just add new problems. So any list beyond fast and good was beyond what I could commit to. Anyway, I couldn’t fully introduce the best stuff without walking through fire - and given I’m about burned up when I get to it and when you add it all your tests and stability goes out the window because every leap forward exposed a new world of test problems and tying together at the moment that they are preparing to deport you and your about dead. The whole ordeal is such a sh*t show. But crawled out somehow. Now if I touch the thing, it’s like washing my nice car instead of a large rock on my head that seems to be getting heavier. Only way I could a reference for myself to cheat in the future. I’m sure it will allow some cheating by others. On Sat, Apr 24, 2021 at 4:46 PM Ishan Chattopadhyaya < [email protected]> wrote: > This is the most comprehensive list of improvements we've seen so far. I > know from our conversations having discussed many of these goals, but I'm > glad to see them here as a list. > Thanks for your work, Mark! > > On Sat, Apr 24, 2021 at 11:05 PM Mark Miller <[email protected]> > wrote: > >> *Ive taken a step back on this branch for a bit as I engage in other >> things and catch a breath.* >> >> Before I dump some of my notes on the background of some of the >> technical stuff, I’ll likely make it a bit easier to take the branch for a >> spin and put up the remaining code I have. >> >> The branch is essentially as named. A reference branch for Solr scale, >> performance, and stability. An investigation into what I missed on >> SolrCloud and a preparation to not miss next time. >> >> The high level goals and deliverables mostly boil down to: >> >> >> - Heavily reduced GC and memory usage and leak sweeps. >> >> >> - Heavily reduced reliance on huge amounts of unnecessary threads >> and context switching and problematic thread management. >> >> >> - Large gains in performance and efficiency across the board. >> >> >> - Large advances in Zookeeper usage and behavior and efficiency. >> >> >> - Fast and efficient multi collection support, scaling to 1000’s of >> collections and 10s of thousands of cores with relative ease compared to >> the past. >> >> >> - Hardened and improved recovery and leadership election paths. >> >> >> - Fast and stable tests, both standard and nightly. >> >> >> - Large improvements in indexing performance and efficiency, >> especially when indexing to multiple replicas. >> >> >> - Connection use and stability and efficiency improvements. >> >> >> - Async update and query paths. >> >> >> - Improved and hardened HTTP2 support through the system. >> >> >> - Optional async servlet requests, with optional use of async IO. >> >> >> - Improved and hardened startup / shutdown and cluster restarts. >> >> >> - Efficiencies and improvements around dealing with overload and >> request priority. >> >> >> - Improvements and changes and starting paths to allow for further >> and larger scale while retaining resource control and performance. >> >> >> And a variety of other things, though it won’t all end up 100% >> finished. >> >> It will essentially power the next phase of my dev career in Java. But >> there may be some fallout for others as well. >> >> -- >> - Mark >> >> http://about.me/markrmiller >> > -- - Mark http://about.me/markrmiller
