Great work Dawid. Just trying out solr tests now. I see that it is not very smart in parallel scheduling. But that is probably something we can improve over time? Is there some env.var or global setting to force e.g. -Ptests.seed=deadbeef for faster runs? We probably don’t need to randomize everything everywhere every time?
Also, I like your moving of build files from the source tree. Will soon get used to gradlew -p solr/packaging assemble. Wonder how this will work with Admin UI changes. Hopefully you can just run a new assemble and reload browser on each change. Jan > 3. des. 2019 kl. 09:35 skrev Dawid Weiss <dawid.we...@gmail.com>: > >> gradlew -p lucene test >> That took my machine 27 minutes! I can see it used like 7 threads when I'd >> rather it use 3-4. That's probably why. > > I capped the execution of parallel tests to this > (gradle/testing/defaults-tests.gradle): > > // Set up default parallel execution limits. > maxParallelForks = (int) Math.max(1, > Math.min(Runtime.runtime.availableProcessors() / 2.0, 3.0)) > > but this limit applies per-project and gradle runs with N workers in > parallel mode (where N equals the number of cores). For example, on my > 8-core Linux machine (SSD drive) the test time for > > ./gradlew -p lucene cleanTest test -Ptests.seed=28BC12BA7C2CD894 > > is 11m 32s. If I limit max workers to 4: > > ./gradlew -p lucene cleanTest test -Ptests.seed=28BC12BA7C2CD894 > --max-workers=4 > > the build time is XXX. > > It is obvious we need to fine-tune this but it's not obvious what the > defaults should be because there are many dimensions to performance: > I/O congestion, CPU cores, overall memory bandwidth and operating > system (underlying filesystem implementation mostly, I believe). Also, > as Mark pointed out before, the test runner is slightly different in > Gradle and doesn't load-balance tests too efficiently (no work > stealing). > > Finally, all the above also doesn't mean we can only think of > improving gradle performance and not tests themselves... Some of them > are just (very) slow. :) > > If you're willing to experiment then try to run with a varying number > of workers (and identical seed to keep the tests the same). Then see > which one turns out to be the best for you (and report back). I think > it'll be half the number of cores (effectively physical cores) unless > you have a very beefy machine when memory bandwidth and I/O comes into > play. > > gradlew --max-workers N -Ptests.seed=deadbeef > >> gradlew :helpAnt >> This failed: -- java.io.FileNotFoundException > > Corrected, apologies. > > Dawid > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
