Silly question, when indicating CI, does this mean Travis CI, the Github Actions/builds, or Apache CI (Jenkins) pipeline? Assume in any case the results of change in the specific project tests might be similar but may require different implementations of CI side configuration if needed.
The single thread seems appealing to speed things up, but assume cases like testing different jdk versions would still fall under the scope of needing a new JVM. In past troubleshooting on one CI issue (believe it was one of the Gradle Project tests during one of the Travis CI tests), it seemed like a possible concurrency issue at the time, so assume in that case having single thread might reduce the potential multithreads issues while different tests working with same lock/shared files as the same time. Eric Bresie [email protected] On Wed, Nov 3, 2021 at 12:48 AM Michael Bien <[email protected]> wrote: > if test suites can be identified which only run true functional unit > tests without booting up the whole module system, sure why not. But lets > try to be careful there and not add more failure modes :) > > CI needs more reliability at the moment, not less. Devs second guessing > the test results is not a good situation to be in - i don't even have a > restart button, i usually just ignore a failure and hope someone > restarts it. > > > As a sidenote: build itself is a big slice of the total time. > > If you take a look at: > https://github.com/mbien/netbeans/actions/runs/1388038850 > > You see build + commit validation takes together about 17 mins (rest is > setup, tar.gz and upload). > > The secondary jobs which only do extract + test (using the same tar from > the primary job) are fairly fast and can run in parallel (they don't do > full testing to not waste apache resources on pipeline experiments). JDK > 8 is also not the fastest starting JVM around the block - secondary test > jobs should be faster by default. > > (JDK 16+17 tests are the fastest since they all fail early ;)) > > regards, > michael > > > On 03.11.21 04:20, Jaroslav Tulach wrote: > > Hi. > > One of the reasons why our CI takes so long is the old habit to execute > > each of our tests in a separate JVM. I am aware (now after five years > > working on GraalVM) that JVM doesn't do anything useful for the first two > > seconds. It starts in interpreter mode, collects profiles, invokes C1 > > compiler and (in the worst) case also C2 compiler. Only then the JVM > > reaches the point of "peak performance" when it is actually useful. > However > > before reaching the point many of our tests are over. As a result > running: > > > > $ ant -f java/java.source.base/ test > > > > just took 9.5 minutes on my computer. The cure is simple, if I change the > > forkMode: > > > > diff --git a/nbbuild/templates/common.xml b/nbbuild/templates/common.xml > > index 377b6db0d1a4..ec063c3ba7b4 100644 > > --- a/nbbuild/templates/common.xml > > +++ b/nbbuild/templates/common.xml > > @@ -740,7 +740,7 @@ > > <attribute name="test.type"/> > > <attribute name="disable.apple.ui" default="false"/> > > <sequential> > > - <junit showoutput="true" fork="true" > > failureproperty="tests.failed" errorproperty="tests.failed" > > filtertrace="${test.filter.trace}" > > tempdir="${build.test.@{test.type}.results.dir}" > > timeout= > > "${test.timeout}" jvm="${test.nbjdk.java}"> > > + <junit showoutput="true" fork="true" forkmode="perBatch" > > failureproperty="tests.failed" errorproperty="tests.failed" > > filtertrace="${test.filter.trace}" tempdir="${build.test.@{test.type}.re > > sults.dir}" timeout="${test.timeout}" jvm="${test.nbjdk.java}"> > > <batchtest todir="${build.test.@ > {test.type}.results.dir}"> > > > > <fileset dir="${build.test.@ > {test.type}.classes.dir}" > > includes="${test.includes}" excludes="${test.excludes}"/> > > </batchtest> > > > > then the test suite runs in 4.5 minutes. I believe similar speed up could > > be achieved in other test suites as well. > > > > Of course, there is a drawback. By running the tests in a single JVM they > > may start to influence each other. Sure, unit tests shouldn't do that, > but > > that is easier said than done, especially with [Injectable Singletons]( > > http://wiki.apidesign.org/wiki/Injectable_Singleton) - used everywhere > in > > NetBeans - and their limited co-existence capabilities. As such we > probably > > need an opt-in for modules/tests to turn the single JVM mode on. > > > > Does making CI faster worth it? Anyone willing to give it a try? > > -jt > > > > PS: Please avoid [Pentium-like jokes]( > > https://twitter.com/chainq/status/948725922642976768) that it is "faster > > but incorrect". With opt-in, it can be faster and correct. > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > For further information about the NetBeans mailing lists, visit: > https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists > > > >
