Right, that’s essentially going to be the value outside of me. Whatever anyone wants to make of it. Not much value being made of it now because who can see it? I have a reasonable view from a billion hours of logs and profilers and monitors and comparisons and measurements and watching each advance. A few others have some thoughts given what I’ve shared with them, but really it’s just a big opaque code ball to most anyone - almost wildly out of date, an unstable old push, or in between something or other as I look at the next advancement.
But the real value in it is not going anywhere. JMH will be available to run against main and that branch. It will have the new build and run against the latest Lucene. Many, many things are drastically faster, more efficient and more solid. Done the way I want to do it or was stuck doing it. Not the way you or she might. When it’s possible, curious people will be able to see some of the differences. See an implementation that’s behind those differences. Who knows, maybe be inspired to make some improvements. Meanwhile, changes will get in the way of my responsibilities. HTTP2, JavaBin (the I’m an consumer of inputstreams that disregards contracts), the Fast* classes (Fast only in name), various stuff that seems to be moving forward will end up causing me issues, and that knowledge will leak over. I don’t see any realistic game plan for a team at this point. But it will become easy to see why I call the branch a reference for speed, scale and stability of Solr/SolrCloud and I will be forced to use it for some things, and some others may find their own interest. Mike has already show interest. But he’s intelligently skeptical by nature. Many developers are. But giving a helping hand to skeptics requires development of its own and that’s been item last. MRM On Tue, Jun 8, 2021 at 6:06 PM Mike Drob <[email protected]> wrote: > I’ve picked up a few of the improvements and brought them over to main > branch, fixed a few busy waits, improved some concurrent access, minor > stuff really. I’m currently busy trying to digest and experiment with the > jetty changes Mark mentioned earlier, so hopefully that’s another piece > that can come in early. There’s some more components, but I don’t want to > promise anything since it’s all competing with other priorities as well. > > Mike > > On Tue, Jun 8, 2021 at 5:43 PM Jan Høydahl <[email protected]> wrote: > >> +1 to apply same code formatting on the ref branch to be able to compare. >> >> Solr still suffers from sloow and unstable tests. I whsh there were some >> lessons learnt from the ref-branch that could be ported over. I don't know >> if you separated out "Integration" tests from unit tests or something that >> we could bring over. Or got rid of some more bad busy-wait / sleep >> patterns? Or perhaps getting rid of unnecessary CloudCluster >> initializations, unneccessary waiting for cluster state or whatever? If not >> copy-pastable, having the biggest wins written down as short recipies of >> how to fix the test, would help a lot I guess. Or perhaps the test speedup >> all depends on the rest of the improvements made in non-test code... That's >> what it sounds like to me, that you cannot have an improvement in >> isolation, you need to do it all in one go :( >> >> Jan >> >> 8. jun. 2021 kl. 23:42 skrev David Smiley <[email protected]>: >> >> Sorry to hear that Mark. I hope it might be useful for little bits 'n >> pieces. Occasionally I'm looking at some class, maybe a test, and I >> quickly do a comparison to the ref branch to wonder what you did there. >> There is a grand source code reformatting that is underway; we should apply >> that processing to that branch so that the comparisons remain useful. >> >> Uwe: In light of Mark's comments, I think you should disable and/or >> remove the CI build jobs. >> >> ~ David Smiley >> Apache Lucene/Solr Search Developer >> http://www.linkedin.com/in/davidwsmiley >> >> >> On Tue, Jun 8, 2021 at 1:27 PM Mark Miller <[email protected]> wrote: >> >>> At this point I doubt the ref branch is going to help anyone much other >>> than me as the need arises from any assignment i may have or in sharing >>> information with others as the need arises (I put together a presentation >>> on Jetty from it for others on my team a bit ago.) >>> >>> Extracting anything but isolated help for what I may need from it is >>> currently beyond my ability. >>> >>> * It address and fixes thousands of bugs and issues of varying >>> importance and interconnectedness. Where would you even start. >>> >>> * It fully fully embraces, fixes and extends HTTP2 support, async, async >>> IO, Jetty. Doing that at all reasonably requires a tremendous amount of >>> work as each is very sensitive to getting things very close to right all >>> over the map. Even after spending ridiculous amounts of time on core >>> issues, that work is heavy to make solid. >>> >>> * It speeds up and hardens test after test after test. Ridiculous parts >>> and efforts involved, connected to everything else. >>> >>> * It minimizes resources and objects and GC tremendously everywhere, >>> moves more off heap, mmaps transaction log files, parallelizes tons more, >>> and leaves Lucene at the top of garbage generation stack (though Lucene is >>> certainly reasonable there). Again, where do you start. >>> >>> * And then it does some things that build on having everything else >>> below it. >>> >>> Other than a few items and hanging chads, it does or opens up whatever I >>> ever wanted for SolrCloud. And the effort is not simply each of the many >>> many items - it’s the crazy pain staking time and care working through all >>> of the issues and connectedness and surface area for everything. >>> >>> When I get some time to set it up for others, it will offer an alternate >>> view of what Solr could do, but ive imagined for a while it will mostly >>> offer me things and occasionally some sharing and demoing for what some >>> others have been interested in. And as I have things that I have to do, it >>> provides me with a working map and model of what the issues are, what I may >>> need to get around them, and where the current stuff stands. >>> >>> It’s what I would do with SolrCloud. It’s focused on what I think the >>> problems are, what I think matters to the end result. I don’t even often >>> see strong alignment on that with anyone other than silently across >>> timelines with Dat. Always find something I hadn’t noticed from Dat and >>> think, man, that guy is on my page. >>> >>> >>> MRM >>> >>> On Fri, Jun 4, 2021 at 8:51 AM Jan Høydahl <[email protected]> >>> wrote: >>> >>>> Hi, >>>> >>>> We have an upcoming committers meeting soon, and will likely touch on >>>> the topic of Reference Branch. >>>> As we could probably waste the whole meeting on that one topic, I'm >>>> starting this thread to "warm up". >>>> >>>> As for me, I'm totally not up to date on what the status is. I don't >>>> even know if anyone have been looking at the branch at all lately. >>>> *Is there anyone who can give a short status update on what the state >>>> is*, who is working on it, what is the next steps, what are the >>>> challenges/blockers etc? >>>> >>>> As 9.0 is coming up, and the project plans for further changes, perhaps >>>> even reorganization of the git folder structure, move to Java 17 on main >>>> branch etc etc, I think there is still a potential window between now and 6 >>>> months ahead where porting code from ref-branch is still doable. After that >>>> it will become more and more problematic. >>>> >>>> Jan >>>> >>> -- >>> - Mark >>> >>> http://about.me/markrmiller >>> >> >> -- - Mark http://about.me/markrmiller
