if test suites can be identified which only run true functional unit
tests without booting up the whole module system, sure why not. But lets
try to be careful there and not add more failure modes :)
CI needs more reliability at the moment, not less. Devs second guessing
the test results is not a good situation to be in - i don't even have a
restart button, i usually just ignore a failure and hope someone
restarts it.
As a sidenote: build itself is a big slice of the total time.
If you take a look at:
https://github.com/mbien/netbeans/actions/runs/1388038850
You see build + commit validation takes together about 17 mins (rest is
setup, tar.gz and upload).
The secondary jobs which only do extract + test (using the same tar from
the primary job) are fairly fast and can run in parallel (they don't do
full testing to not waste apache resources on pipeline experiments). JDK
8 is also not the fastest starting JVM around the block - secondary test
jobs should be faster by default.
(JDK 16+17 tests are the fastest since they all fail early ;))
regards,
michael
On 03.11.21 04:20, Jaroslav Tulach wrote:
Hi.
One of the reasons why our CI takes so long is the old habit to execute
each of our tests in a separate JVM. I am aware (now after five years
working on GraalVM) that JVM doesn't do anything useful for the first two
seconds. It starts in interpreter mode, collects profiles, invokes C1
compiler and (in the worst) case also C2 compiler. Only then the JVM
reaches the point of "peak performance" when it is actually useful. However
before reaching the point many of our tests are over. As a result running:
$ ant -f java/java.source.base/ test
just took 9.5 minutes on my computer. The cure is simple, if I change the
forkMode:
diff --git a/nbbuild/templates/common.xml b/nbbuild/templates/common.xml
index 377b6db0d1a4..ec063c3ba7b4 100644
--- a/nbbuild/templates/common.xml
+++ b/nbbuild/templates/common.xml
@@ -740,7 +740,7 @@
<attribute name="test.type"/>
<attribute name="disable.apple.ui" default="false"/>
<sequential>
- <junit showoutput="true" fork="true"
failureproperty="tests.failed" errorproperty="tests.failed"
filtertrace="${test.filter.trace}"
tempdir="${build.test.@{test.type}.results.dir}"
timeout=
"${test.timeout}" jvm="${test.nbjdk.java}">
+ <junit showoutput="true" fork="true" forkmode="perBatch"
failureproperty="tests.failed" errorproperty="tests.failed"
filtertrace="${test.filter.trace}" tempdir="${build.test.@{test.type}.re
sults.dir}" timeout="${test.timeout}" jvm="${test.nbjdk.java}">
<batchtest todir="${build.test.@{test.type}.results.dir}">
<fileset dir="${build.test.@{test.type}.classes.dir}"
includes="${test.includes}" excludes="${test.excludes}"/>
</batchtest>
then the test suite runs in 4.5 minutes. I believe similar speed up could
be achieved in other test suites as well.
Of course, there is a drawback. By running the tests in a single JVM they
may start to influence each other. Sure, unit tests shouldn't do that, but
that is easier said than done, especially with [Injectable Singletons](
http://wiki.apidesign.org/wiki/Injectable_Singleton) - used everywhere in
NetBeans - and their limited co-existence capabilities. As such we probably
need an opt-in for modules/tests to turn the single JVM mode on.
Does making CI faster worth it? Anyone willing to give it a try?
-jt
PS: Please avoid [Pentium-like jokes](
https://twitter.com/chainq/status/948725922642976768) that it is "faster
but incorrect". With opt-in, it can be faster and correct.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
