Re: Gradle build optimization
> Huh, I thought by "cd lucene/core" would be equivalent to "-p lucene/core" > from the root checkout? I don't think it is. It's a wrapper script - I'm not sure it recognizes where it's run from. > real0m4.802s > user0m2.988s > sys 0m0.189s > > Not much better? Ok. In that case, it's something else. I have a strong suspicion that gradle's test task scans through all the classes or does something weird. You could try running with a build scan so that there are more details on which phase actually takes those 5 seconds. D. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Gradle build optimization
[Adding Jerome back on CC -- not sure he's sub'd] More below: On Wed, Oct 20, 2021 at 10:37 AM Dawid Weiss wrote: > > I do! I am now using pure Gradle/Lucene clean checkout defaults, from > quite a while back (last time you scolded me ;) ) > > Oh, I never scold. I politely pass over the torch of wisdom (or > bullwaste, depending on the day). :) > Perfect! I love it :) > > But it's still slower than ant on 8.x was/is, plus it makes me pay a > warmup penalty the first time (at defaults anyways). > > Part of the startup penalty is the evaluation of various scripts and > settings that never get used. There are gradle-sque ways of avoiding > that (lazy evaluation) but they do increase code complexity (in my > opinion). The first-time compilation should be improved by moving > scripts into buildSrc (or plugins). Again: this removes the clarity of > individual aspect-based scripts (again: my opinion). > OK > > so then I tested on main, with JDK15, also in cwd lucene/core: "time > ../../gradlew test --tests > "org.apache.lucene.index.TestIndexWriter.testGetCommitData" > -Ptests.seed=D708CEE0862DB94C > ignored": > > You're running the test task in *each and every submodule* that > declares it, then filter for a specific test case. Try this: > Huh, I thought by "cd lucene/core" would be equivalent to "-p lucene/core" from the root checkout? time ../../gradlew -p lucene/core test --tests > "org.apache.lucene.index.TestIndexWriter.testGetCommitData" > -Ptests.seed=D708CEE0862DB94C > ignored > I tried that but the "lucene/core" seems to incorrectly "stack up": beast3:core[main]$ time ../../gradlew test -p lucene/core --tests "org.apache.lucene.index.TestIndexWriter.testGetCommitData" -Ptests.seed=D708CEE0862DB94C Starting a Gradle Daemon (subsequent builds will be faster) FAILURE: Build failed with an exception. * What went wrong: The specified project directory '/l/trunk/lucene/core/lucene/core' does not exist. I think you either "cd lucene/core" or you stay in ROOT and pass -p lucene:core:test (or -p lucene/core?)? > or full scoped task :lucene:core:test. Should be slightly better. > OK I tried that, in ROOT of the checkout "time ./gradlew lucene:core:test --tests "org.apache.lucene.index.TestIndexWriter.testGetCommitData" -Ptests.seed=D708CEE0862DB94C > ignore" (discard first warmup, then:) real0m4.514s user0m3.090s sys 0m0.186s real0m4.587s user0m3.029s sys 0m0.186s real0m4.802s user0m2.988s sys 0m0.189s Not much better? > Also, I'm running on a 128 core crazy beast of a box (Ryzen Threadripper > 3990X), 256 GB RAM, fast SSD, 10g networking, etc. :) > I'm jealous. > LOL well there are even faster CPUs now I think, so I'm jealous of them! Turtles all the way down ... > > Also I want to thank you for migrating us to Gradle in the first place > > No need to thank anybody. It's fun. > Awesome! > > But I really don't like waiting :) And yes maybe I just should learn > how to use fancy IDE debuggers instead of SOP + rerun many times ;) > > IntelliJ works very well for me with Lucene (especially if gradle is > not used for compilation/ test launching). You may also look at this - > never tried it but it looks like something your all-green terminals > may look forward to: > > > https://docs.gradle.org/current/userguide/command_line_interface.html#sec:continuous_build Whoa, that does look interesting! Thanks for the pointer. Mike
Re: Gradle build optimization
I'm glad to see that this initiative creates a lot of excitement! I however need to narrow down a bit the scope of expectations as my plan is not to go too deep into the project's logic (won't cover tests taking too long to run for instance) but rather applying some techniques to avoid task execution when not needed. This will happen without involving extra complexity hopefully. We'll probably discuss that with the PR. So far, here is the Jira ticket, feel free to drop your comments in there as well: https://issues.apache.org/jira/browse/LUCENE-10195 Best, Jerome On Wed, Oct 20, 2021 at 4:42 PM Dawid Weiss wrote: > > The fully scoped one doesn't enjoy tab completion in the shell though. > > So I always do path-based "-p lucene/core" because it is less typing > > and less mistakes. > > Funny, it's exactly why I prefer -p as well. > > D. >
Re: Gradle build optimization
> The fully scoped one doesn't enjoy tab completion in the shell though. > So I always do path-based "-p lucene/core" because it is less typing > and less mistakes. Funny, it's exactly why I prefer -p as well. D. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Gradle build optimization
On Wed, Oct 20, 2021 at 10:37 AM Dawid Weiss wrote: > You're running the test task in *each and every submodule* that > declares it, then filter for a specific test case. Try this: > > time ../../gradlew -p lucene/core test --tests > "org.apache.lucene.index.TestIndexWriter.testGetCommitData" > -Ptests.seed=D708CEE0862DB94C > ignored > > or full scoped task :lucene:core:test. Should be slightly better. > The fully scoped one doesn't enjoy tab completion in the shell though. So I always do path-based "-p lucene/core" because it is less typing and less mistakes. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Gradle build optimization
> I do! I am now using pure Gradle/Lucene clean checkout defaults, from quite > a while back (last time you scolded me ;) ) Oh, I never scold. I politely pass over the torch of wisdom (or bullwaste, depending on the day). :) > But it's still slower than ant on 8.x was/is, plus it makes me pay a warmup > penalty the first time (at defaults anyways). Part of the startup penalty is the evaluation of various scripts and settings that never get used. There are gradle-sque ways of avoiding that (lazy evaluation) but they do increase code complexity (in my opinion). The first-time compilation should be improved by moving scripts into buildSrc (or plugins). Again: this removes the clarity of individual aspect-based scripts (again: my opinion). > so then I tested on main, with JDK15, also in cwd lucene/core: "time > ../../gradlew test --tests > "org.apache.lucene.index.TestIndexWriter.testGetCommitData" > -Ptests.seed=D708CEE0862DB94C > ignored": You're running the test task in *each and every submodule* that declares it, then filter for a specific test case. Try this: time ../../gradlew -p lucene/core test --tests "org.apache.lucene.index.TestIndexWriter.testGetCommitData" -Ptests.seed=D708CEE0862DB94C > ignored or full scoped task :lucene:core:test. Should be slightly better. > Also, I'm running on a 128 core crazy beast of a box (Ryzen Threadripper > 3990X), 256 GB RAM, fast SSD, 10g networking, etc. :) I'm jealous. > Also I want to thank you for migrating us to Gradle in the first place No need to thank anybody. It's fun. > But I really don't like waiting :) And yes maybe I just should learn how to > use fancy IDE debuggers instead of SOP + rerun many times ;) IntelliJ works very well for me with Lucene (especially if gradle is not used for compilation/ test launching). You may also look at this - never tried it but it looks like something your all-green terminals may look forward to: https://docs.gradle.org/current/userguide/command_line_interface.html#sec:continuous_build Dawid - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Gradle build optimization
It is very possible I am holding gradlew wrong, like this: https://i2.wp.com/www.adatosystems.com/wp-content/uploads/2019/03/wrong_tool.jpg Mike McCandless http://blog.mikemccandless.com On Wed, Oct 20, 2021 at 10:21 AM Michael McCandless < luc...@mikemccandless.com> wrote: > Jerome, if you are not sub'd to the Lucene dev list, then you maybe missed > some of the replies here. I'm CC'ing you back on here. > > More responses below: > > On Wed, Oct 20, 2021 at 8:45 AM Dawid Weiss wrote: > >> >> >>> I was just annoyed yesterday at how long running a single test case >>> takes. >>> >> >> Use the background daemon, Mike. >> > > I do! I am now using pure Gradle/Lucene clean checkout defaults, from > quite a while back (last time you scolded me ;) ) > > And the defaults enable the daemon I think? Yeah, just confirmed. > > But it's still slower than ant on 8.x was/is, plus it makes me pay a > warmup penalty the first time (at defaults anyways). I just tested 8.x in > cwd lucene/core and JDK8: "time ant test -Dtestcase=TestIndexWriter > -Dtestmethod=testGetCommitData > ignored" three times: > > real0m2.609s > > user0m10.583s > > sys 0m0.590s > > > real0m2.585s > > user0m10.329s > > sys 0m0.827s > > > real0m2.610s > > user0m10.458s > > sys 0m0.606s > > > > so then I tested on main, with JDK15, also in cwd lucene/core: "time > ../../gradlew test --tests > "org.apache.lucene.index.TestIndexWriter.testGetCommitData" > -Ptests.seed=D708CEE0862DB94C > ignored": > > ** (first time discarded due to "daemon" warmup its fire up) > > real0m4.447s > > user0m2.956s > > sys 0m0.217s > > > real0m4.399s > > user0m3.028s > > sys 0m0.233s > > > real0m4.404s > > user0m3.021s > > sys 0m0.181s > > > Now I know there are many differences besides ant/gradle! Maybe JDK15's > startup time got worse. The random seed does not carry over. Lucene's > sources are different, etc. And maybe I should care about a second or two > :) But I do! > > Also, I'm running on a 128 core crazy beast of a box (Ryzen Threadripper > 3990X), 256 GB RAM, fast SSD, 10g networking, etc. :) > > Also I want to thank you for migrating us to Gradle in the first place -- > I know that was a crazy amount of work! I'm trying not to complain (too > much). But I really don't like waiting :) And yes maybe I just should > learn how to use fancy IDE debuggers instead of SOP + rerun many times ;) > > Mike McCandless > > http://blog.mikemccandless.com >
Re: Gradle build optimization
Jerome, if you are not sub'd to the Lucene dev list, then you maybe missed some of the replies here. I'm CC'ing you back on here. More responses below: On Wed, Oct 20, 2021 at 8:45 AM Dawid Weiss wrote: > > >> I was just annoyed yesterday at how long running a single test case takes. >> > > Use the background daemon, Mike. > I do! I am now using pure Gradle/Lucene clean checkout defaults, from quite a while back (last time you scolded me ;) ) And the defaults enable the daemon I think? Yeah, just confirmed. But it's still slower than ant on 8.x was/is, plus it makes me pay a warmup penalty the first time (at defaults anyways). I just tested 8.x in cwd lucene/core and JDK8: "time ant test -Dtestcase=TestIndexWriter -Dtestmethod=testGetCommitData > ignored" three times: real0m2.609s user0m10.583s sys 0m0.590s real0m2.585s user0m10.329s sys 0m0.827s real0m2.610s user0m10.458s sys 0m0.606s so then I tested on main, with JDK15, also in cwd lucene/core: "time ../../gradlew test --tests "org.apache.lucene.index.TestIndexWriter.testGetCommitData" -Ptests.seed=D708CEE0862DB94C > ignored": ** (first time discarded due to "daemon" warmup its fire up) real0m4.447s user0m2.956s sys 0m0.217s real0m4.399s user0m3.028s sys 0m0.233s real0m4.404s user0m3.021s sys 0m0.181s Now I know there are many differences besides ant/gradle! Maybe JDK15's startup time got worse. The random seed does not carry over. Lucene's sources are different, etc. And maybe I should care about a second or two :) But I do! Also, I'm running on a 128 core crazy beast of a box (Ryzen Threadripper 3990X), 256 GB RAM, fast SSD, 10g networking, etc. :) Also I want to thank you for migrating us to Gradle in the first place -- I know that was a crazy amount of work! I'm trying not to complain (too much). But I really don't like waiting :) And yes maybe I just should learn how to use fancy IDE debuggers instead of SOP + rerun many times ;) Mike McCandless http://blog.mikemccandless.com
Re: Gradle build optimization
Btw. this is a placeholder for some of the known bits that require solving. https://issues.apache.org/jira/browse/LUCENE-9871 It'd be awesome to see some help from you guys at Gradle to solve the non-trivial things: - test runner and early/ internal JVM output (the issue I mentioned before), - test runner test case ordering optimization (load balancing between worker JVMs); currently the long-tail test case can slow down builds significantly. - support for user-local gradle.properties or computed properties there (we currently generate this file because it'll be slightly different for every developer, this requires a non-optimal first run), - support for project-controlled up-to-date mechanism (versioned artifact checksums) that is not bypassed on the first run. This is needed for generated resources - I didn't figure out how to do it other than with ugly hacks. If any of these is of interest to you, I can provide more pointers, of course. Let me know! Dawid On Wed, Oct 20, 2021 at 1:48 PM Jerome Prinet wrote: > Hi Lucene team! > I am a software engineer at Gradle and I'd like to help you optimize > Lucene's Gradle build. > I have already been working on this topic and I'd like to open a PR. > Should I start by creating a JIRA issue first? > Please let me know how to proceed. > Kind regards, > > -- > > Jerome Prinet > Senior Software Engineer > Gradle > W. gradle.com > >
Re: Gradle build optimization
On Wed, Oct 20, 2021 at 8:52 AM Dawid Weiss wrote: > > Can you tell us a bit more about which directions these mentioned > optimizations apply to? If it's about build caches then it's already > been discussed as a contribution to Solr: Sure, something like that is definitely worthy of discussion first. I don't understand why build caches are ever needed. If the outputs are up-to-date with the inputs, then it should say UP-TO-DATE. If I ask for "clean", then I really mean it, and expect the next run to be slower. These are simple rules for any build system (Makefile, etc). Gradle is no exception. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Gradle build optimization
Also, run an explicit project task (or narrow down the project with -p); if you're running a generic 'test' over all projects it may have to run/ scan all the classes. If you provide a full command line you used it'll be easier why it takes so long. D. On Wed, Oct 20, 2021 at 2:44 PM Dawid Weiss wrote: > > >> >> I was just annoyed yesterday at how long running a single test case takes. > > > Use the background daemon, Mike. > > D. > - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Gradle build optimization
Can you tell us a bit more about which directions these mentioned optimizations apply to? If it's about build caches then it's already been discussed as a contribution to Solr: https://issues.apache.org/jira/browse/SOLR-15603 Also, it'd be great to see some feedback for more fundamental issues within gradle -- like this one: https://issues.apache.org/jira/browse/LUCENE-9120 https://github.com/gradle/gradle/issues/11609 (this is marked as closed -- incorrectly; it's still a bug and always has been). Dawid On Wed, Oct 20, 2021 at 1:48 PM Jerome Prinet wrote: > > Hi Lucene team! > I am a software engineer at Gradle and I'd like to help you optimize Lucene's > Gradle build. > I have already been working on this topic and I'd like to open a PR. > Should I start by creating a JIRA issue first? > Please let me know how to proceed. > Kind regards, > > -- > > Jerome Prinet > Senior Software Engineer > Gradle > W. gradle.com > > - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Gradle build optimization
> I was just annoyed yesterday at how long running a single test case takes. > Use the background daemon, Mike. D.
Re: Wiki/Documentation for luceneutil
Hi Praveen, Unfortunately we do not have good docs for luceneutil. Maybe start a page on Confluence? Here's a quick explanation. After you set up your performance script (e.g. localrun.py or perf.py or so), and then run it (something like python3 localrun.py -source wikimedium10m), it will first index all docs (takes a while, depending on how concurrent your hardware is), then it will run 20 separate JVMs, where each JVM is running multiple concurrent "search tasks" (simple TermQuery, various kinds of BooleanQuery, primary key lookup, etc.) recording latency. After each JVM finishes, it will print a table of the running results so far. One row per task. Baseline mean and std-dev QPS and same for competitor are shown. Then the pct change (baseline mean versus competitor mean) is shown, as well as a confidence measure (p-value). Nightly benchmarks run these same tools and collect results over time, making charts showing the changes, here: https://home.apache.org/~mikemccand/lucenebench It's been "running" for more than a decade now! Mike McCandless http://blog.mikemccandless.com On Wed, Oct 20, 2021 at 7:43 AM Praveen Nishchal wrote: > Hi, > > Please help me to find Wiki/Documentation for luceneutil on how to > interpret the stats so presented. Is there a way to obtain a summary table > of all the runs just like lucene benchmark stats table for index/search? > > Thanks, > Praveen >
Re: Gradle build optimization
Whoa, hi Jerome, yes please :) I was just annoyed yesterday at how long running a single test case takes. Please create an issue and reference it (in all caps, i.e. LUCENE-X) in the description. Thanks, Mike McCandless http://blog.mikemccandless.com On Wed, Oct 20, 2021 at 7:48 AM Jerome Prinet wrote: > Hi Lucene team! > I am a software engineer at Gradle and I'd like to help you optimize > Lucene's Gradle build. > I have already been working on this topic and I'd like to open a PR. > Should I start by creating a JIRA issue first? > Please let me know how to proceed. > Kind regards, > > -- > > Jerome Prinet > Senior Software Engineer > Gradle > W. gradle.com > >
Gradle build optimization
Hi Lucene team! I am a software engineer at Gradle and I'd like to help you optimize Lucene's Gradle build. I have already been working on this topic and I'd like to open a PR. Should I start by creating a JIRA issue first? Please let me know how to proceed. Kind regards, -- Jerome Prinet Senior Software Engineer Gradle W. gradle.com
Re: How to run Lucene 9 tests in eclipse?
Thanks Adrien for pointing me in the right direction :) On Tue, Oct 12, 2021 at 11:47 AM Adrien Grand wrote: > Hi Praveen, > > Have you seen this page on the wiki? > https://cwiki.apache.org/confluence/display/LUCENE/DeveloperTips#DeveloperTips-TipstoconfigureIDEs > > By running `gradlew tasks`, you should see a section about IDE tasks that > help with importing the project in Eclipse. For me running tests then works > just like it does for any other Eclipse project. > > On Tue, Oct 12, 2021 at 7:42 AM Praveen Nishchal > wrote: > >> Hi All, >> >> How can we run Lucene9 tests via eclipse? I am able to run all the tests >> through the command line but not with eclipse. >> >> Please advise! >> > > > -- > Adrien >
Wiki/Documentation for luceneutil
Hi, Please help me to find Wiki/Documentation for luceneutil on how to interpret the stats so presented. Is there a way to obtain a summary table of all the runs just like lucene benchmark stats table for index/search? Thanks, Praveen
