Yes, there is -Dtest-unit-sys-prop.ignore.random.failures=true , which skips tests marked with the @RandomlyFails annotation. That seems not to be the problem here--I've been skipping these, as does Jenkins. (And the OrderHid.testSnd test from question 3 is _not_ marked with @RandomlyFails.)
-- Eirik -----Original Message----- From: Tim Boudreau <niftin...@gmail.com> Sent: Wednesday, July 25, 2018 7:21 PM To: dev@netbeans.incubator.apache.org Subject: Re: Does Travis and/or Jenkins run the NetBeans test suite? I don't know that this is the answer, but I know there used to be a mechanism for marking tests that fail randomly or were known to fail but hadn't been fixed, so the test suite didn't count them, or flagged an unexpected pass. Maybe that's being used? -Tim On Wed, Jul 25, 2018 at 12:54 PM Eirik Bakke <eba...@ultorg.com> wrote: > Partially--my follow-up questions are: > 1) Am I correct that tests in editor.lib2 (and many other editor.* > modules) are _not_ included in the tests run by Jenkins (at > https://builds.apache.org/job/incubator-netbeans-linux ), since they > are not part of the "platform" cluster? > 2) Are anyone else able to run the tests in editor.lib2 without error? > (I get "NoClassDefFoundError: org/netbeans/editor/GuardedDocument" > errors > there.) > 3) Why does Jenkins not detect the failures in > org.netbeans.junit.OrderHid.testSnd? (Seen in the console log e.g. for > https://builds.apache.org/job/incubator-netbeans-linux/496 ) > > -- Eirik > > -----Original Message----- > From: Geertjan Wielenga <geertjan.wiele...@googlemail.com.INVALID> > Sent: Wednesday, July 25, 2018 2:35 PM > To: dev <dev@netbeans.incubator.apache.org> > Subject: Re: Does Travis and/or Jenkins run the NetBeans test suite? > > But does this not answer the question: > > FYI: > > https://builds.apache.org/job/incubator-netbeans-linux/ > > and > > https://builds.apache.org/job/incubator-netbeans-windows > > run platform tests. > > > > Gj > > > > On Wed, Jul 18, 2018 at 1:43 PM, Eirik Bakke <eba...@ultorg.com> wrote: > > > Looking at the "last stable build" of incubator-netbeans-linux, this > > one reports "no failures", yet has a failure in > > org.netbeans.junit.OrderHid.testSnd > > (and a few more also in the nbjunit module): > > > > https://builds.apache.org/job/incubator-netbeans-linux/480 > > > > Is there somehting special about the nbjunit module that makes > > Jenkins ignore test failures there, and only there? (I don't see it > > in the list of tests at > > https://builds.apache.org/job/incubator-netbeans-linux/480/ > > testReport/ .) > > > > These Jenkins builds also omit all the tests in the "ide" cluster, right? > > E.g. most of the editor.* modules? When I run tests in editor.lib2 > > myself, I get several errors [1]. I'm trying to figure out whether > > this is due to my own setup, or whether it's a problem in the > > codebase > itself. > > > > -- Eirik > > [1] All which boil down to "task failed due to: > java.lang.NoClassDefFoundError: > > org/netbeans/editor/GuardedDocument", e.g. in > org.netbeans.modules.editor. > > lib2.highlighting.HighlightingManagerTest . Seems like something I'm > > doing wrong with the build--unless there's an error in a static > > initializer somewhere. I tried a clean build of the entire > > incubator-netbeans repo, both on MacOS and Windows. I used "ant > > -Dtest-unit-sys-prop.ignore.random.failures=true > > -Dtest.modules=editor.lib2 test" and inspected the output for failures. > > > > On 7/18/18, 6:24 AM, "Jaroslav Tulach" <jaroslav.tul...@gmail.com > <mailto: > > jaroslav.tul...@gmail.com>> wrote: > > > > FYI: > > https://builds.apache.org/job/incubator-netbeans-linux/ > > and > > https://builds.apache.org/job/incubator-netbeans-windows > > run platform tests. > > > > -jt > > > > > > Dne neděle 15. července 2018 18:47:51 CEST, Eirik Bakke napsal(a): > > When I make a pull request on GitHub, there is a nice little > > checkmark saying "All checks have passed", with a link to a Travis CI build > > (e.g. > > https://travis-ci.org/apache/incubator-netbeans/builds/ > > 395547620?utm_source > > =github_status&utm_medium=notification ). > > Looking at the raw output of the Travis build, am I correct that > > this does _not_ actually run the NetBeans test suite? I searched the > > console output and did not find expected messages such as "Tests run:" > > or "do-junit" or "[junit]". > > Is this also the case for the Jenkins builds at > > https://builds.apache.org/view/Incubator%20Projects/job/ > > incubator-netbeans-> release ? > > Is the current codebase supposed to pass all tests at this point? > > When I check out the 9.0-vc3 tag, for instance, both of the > > following fail with various errors: > > ant commit-validation > > ant -Dtest-unit-sys-prop.ignore.random.failures=true test Are these > > supposed to work? > > -- Eirik > > > > > > > > > > > > > -- http://timboudreau.com
