Great work Dawid. Just trying out solr tests now.
I see that it is not very smart in parallel scheduling. But that is probably 
something we can improve over time?
Is there some env.var or global setting to force e.g. -Ptests.seed=deadbeef for 
faster runs?
We probably don’t need to randomize everything everywhere every time?

Also, I like your moving of build files from the source tree. Will soon get 
used to gradlew -p solr/packaging assemble.
Wonder how this will work with Admin UI changes. Hopefully you can just run a 
new assemble and reload browser on each change.

Jan

> 3. des. 2019 kl. 09:35 skrev Dawid Weiss <dawid.we...@gmail.com>:
> 
>> gradlew -p lucene test
>> That took my machine 27 minutes!  I can see it used like 7 threads when I'd 
>> rather it use 3-4.  That's probably why.
> 
> I capped the execution of parallel tests to this
> (gradle/testing/defaults-tests.gradle):
> 
>      // Set up default parallel execution limits.
>      maxParallelForks = (int) Math.max(1,
> Math.min(Runtime.runtime.availableProcessors() / 2.0, 3.0))
> 
> but this limit applies per-project and gradle runs with N workers in
> parallel mode (where N equals the number of cores). For example, on my
> 8-core Linux machine (SSD drive) the test time for
> 
> ./gradlew -p lucene cleanTest test -Ptests.seed=28BC12BA7C2CD894
> 
> is 11m 32s. If I limit max workers to 4:
> 
> ./gradlew -p lucene cleanTest test -Ptests.seed=28BC12BA7C2CD894 
> --max-workers=4
> 
> the build time is XXX.
> 
> It is obvious we need to fine-tune this but it's not obvious what the
> defaults should be because there are many dimensions to performance:
> I/O congestion, CPU cores, overall memory bandwidth and operating
> system (underlying filesystem implementation mostly, I believe). Also,
> as Mark pointed out before, the test runner is slightly different in
> Gradle and doesn't load-balance tests too efficiently (no work
> stealing).
> 
> Finally, all the above also doesn't mean we can only think of
> improving gradle performance and not tests themselves... Some of them
> are just (very) slow. :)
> 
> If you're willing to experiment then try to run with a varying number
> of workers (and identical seed to keep the tests the same). Then see
> which one turns out to be the best for you (and report back). I think
> it'll be half the number of cores (effectively physical cores) unless
> you have a very beefy machine when memory bandwidth and I/O comes into
> play.
> 
> gradlew --max-workers N -Ptests.seed=deadbeef
> 
>> gradlew :helpAnt
>> This failed: -- java.io.FileNotFoundException
> 
> Corrected, apologies.
> 
> Dawid
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to