By the way, I did hear about the hack day and some Gradle testing, which is great, and very useful. Totally needed, but we also need a bit of a more deep core developer view of things vs the old build as well. The type of stuff that’s much harder to tease out than verifying all the new build targets and such. Of course a lot of that can come after we switch, but I have a sneaky feeling some core devs will have deep opinions about certain things.
Ill add a new task for default config setup. Mark On Sun, Sep 15, 2019 at 5:12 PM Mark Miller <markrmil...@gmail.com> wrote: > I'll just detail it out here as well: > > You can configure parallelism in ~/.gradle/gradle.properties > > org.gradle.workers.max=2 > tests_jvms=5 > > org.gradle.workers.max is controlled by gradle and defaults to the number > of cores detected - I wish I could change to divided by 2. > test_jvms is controlled by us and defaults to the number of cores detected > / 2. > > org.gradle.workers.max controls the total number of jvms that will be run > in parallel - for tasks or tests or whatever gradle is doing. > test_jvms controls how many parallel jvms a module will use for tests, but > that is also limited by org.gradle.workers.max > > You should try setting both to cores / 2 and work down from there if > needed. > > When running tests across multiple modules, org.gradle.workers.max is the > actual limit for test jvms spun up because each module is only limited to > test_jvms, but ALL tasks are limited to org.gradle.workers.max and module > tasks are run in parallel by default. > > By setting them the same, we get similar behavior whether we run tests > from the root directory of the project (all tests) or from a single module > (say solr-core). > > On Sun, Sep 15, 2019 at 5:03 PM Mark Miller <markrmil...@gmail.com> wrote: > >> I've added more about that here: >> https://cwiki.apache.org/confluence/display/SOLR/Intro+to+the+Gradle+build >> >> It's configurable, but difficult for us to choose a default based on >> cores as far as I can tell and gradles default, which is based on cores >> detected, is too high, especially because of hyperthreading. >> >> Probably the best we can do is default it to a hard 2 or 4 and let people >> raise it depending on what wait you want to error. In both cases the >> majority of people will want to change it. >> >> On Sun, Sep 15, 2019 at 3:39 PM Gus Heck <gus.h...@gmail.com> wrote: >> >>> Not sure if you heard, but about a half a dozen folks tried it out on >>> macs and one on windows at the hack day on Tuesday before Activate. It >>> caused some scrambling for sharing of power bricks (a single run of the >>> tests eats 70% of a fully charged 2018 macbook pro battery in 45 min), but >>> the good news is it only failed on well known flaky tests and on the one >>> windows machine and that in the PDF building for the ref guide (and there >>> was that small bit with the error message and the AtomicBoolean that I >>> fixed). I've heard some opinions that maybe we don't need the PDF version >>> in the future, The one bit of feedback that came out of it was it would be >>> nice to have a task that tweaked the configs things to not peg the >>> processor quite so efficiently :). Certainly we do want the mode for full >>> utilization to be available, but a background mode would be good too in >>> case folks need to do other work. >>> >>> -Gus >>> >>> On Sun, Sep 15, 2019 at 3:24 PM Mark Miller <markrmil...@gmail.com> >>> wrote: >>> >>>> Okay, I've tried to spread that link a little on social media as well. >>>> >>>> Please do some experimentation. Especially those of you that use or >>>> know more esoteric things about the build. The basics are pretty solid, >>>> most have been working for months now, it's the corners and crannies that >>>> I'm more concerned about. You could have built and run tests a few months >>>> ago and said, nice, it's working, but then I've done 10,000 things since >>>> then that were necessary. You may easily not realize things are a problem >>>> until you really dig into something. >>>> >>>> The more that devs can start trying out Gradle now, the less overlap of >>>> the two build systems we will need. >>>> >>>> The idea of the overlap is that it will almost force many of us to >>>> start trying things out - at which point we will start to understand any >>>> key things that are missing, key problems or bugs, etc. If I just make the >>>> switch and let the cards fall were they may, those cards are almost >>>> certainly going to be very disruptive and annoying to a lot of people. >>>> >>>> It will also give us a chance to start rolling CI and other tools over >>>> to using the Gradle build while ant+ivy are still available and in charge. >>>> I don't think it's a great idea to try and do this all at one moment, that >>>> will be difficult to coordinate and be more disruptive to less interested >>>> devs. We would like to keep the dual build situation contained and short >>>> though. That's why I have a plan to pull out if we don't make enough >>>> progress by a month or twos time (closer to a month). >>>> >>>> If we have enough experimentation ahead of time, the overlap can be >>>> very short - we start moving things like jenkins jobs over and when we are >>>> done and confident the world is not ending, we make some adjustments so >>>> that gradle owns the build (eg the dependency tree) and then remove >>>> ant+ivy+maven. >>>> >>>> Some things like the smoke tester and what not *can* come shortly after >>>> we switch I think - we have until the 9 release to truly get everything in >>>> order (keeping in mind that we are still developing so some things are >>>> still critical). But we need CI and other basics all moved over when we >>>> flip the ownership switch. >>>> >>>> - Mark >>>> >>>> On Sun, Sep 15, 2019 at 12:28 PM Mark Miller <markrmil...@gmail.com> >>>> wrote: >>>> >>>>> I've started to put together a little guide to help people ramp up >>>>> here: >>>>> https://cwiki.apache.org/confluence/display/SOLR/Intro+to+the+Gradle+build >>>>> >>>>> On Sat, Sep 14, 2019 at 9:09 PM Mark Miller <markrmil...@gmail.com> >>>>> wrote: >>>>> >>>>>> https://issues.apache.org/jira/browse/SOLR-13452 >>>>>> Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to >>>>>> Gradle. >>>>>> >>>>>> Hey all. >>>>>> >>>>>> Here is my Lucene and Solr 'move to Gradle' plans. >>>>>> >>>>>> * I plan on trying to commit the Gradle build by 9/30 >>>>>> * In the meantime I'll do a bit more work on packaging, publishing, >>>>>> and dependencies. >>>>>> * It would be great if others could start digging in a bit as well. >>>>>> * I plan to try and sidecar Gradle to begin with. Once that happens, >>>>>> I really need others to start digging in as we have a lot to do (switch >>>>>> ci >>>>>> and other tools to the new build, figure out new release docs and issues, >>>>>> finish the Solr documentation module (I'll try and get to that earlier >>>>>> than >>>>>> later), and generally make things pleasant across as many dev envs as we >>>>>> can. >>>>>> * Beyond that, there will likely be a longer tail of smaller issues >>>>>> or missing things even after we make the switch. >>>>>> * I'll put some effort into the side car experiment for another month >>>>>> or two perhaps (maybe not much in November) >>>>>> * If things are not looking like we will be able to flip the switch, >>>>>> I'll look at pulling out Gradle. >>>>>> >>>>>> - Mark >>>>>> >>>>> >>>>> >>>>> -- >>>>> - Mark >>>>> >>>>> http://about.me/markrmiller >>>>> >>>> >>>> >>>> -- >>>> - Mark >>>> >>>> http://about.me/markrmiller >>>> >>> >>> >>> -- >>> http://www.needhamsoftware.com (work) >>> http://www.the111shift.com (play) >>> >> >> >> -- >> - Mark >> >> http://about.me/markrmiller >> > > > -- > - Mark > > http://about.me/markrmiller > -- - Mark http://about.me/markrmiller
