RE: Does Travis and/or Jenkins run the NetBeans test suite?
Yes, there is -Dtest-unit-sys-prop.ignore.random.failures=true , which skips tests marked with the @RandomlyFails annotation. That seems not to be the problem here--I've been skipping these, as does Jenkins. (And the OrderHid.testSnd test from question 3 is _not_ marked with @RandomlyFails.) -- Eirik -Original Message- From: Tim Boudreau Sent: Wednesday, July 25, 2018 7:21 PM To: dev@netbeans.incubator.apache.org Subject: Re: Does Travis and/or Jenkins run the NetBeans test suite? I don't know that this is the answer, but I know there used to be a mechanism for marking tests that fail randomly or were known to fail but hadn't been fixed, so the test suite didn't count them, or flagged an unexpected pass. Maybe that's being used? -Tim On Wed, Jul 25, 2018 at 12:54 PM Eirik Bakke wrote: > Partially--my follow-up questions are: > 1) Am I correct that tests in editor.lib2 (and many other editor.* > modules) are _not_ included in the tests run by Jenkins (at > https://builds.apache.org/job/incubator-netbeans-linux ), since they > are not part of the "platform" cluster? > 2) Are anyone else able to run the tests in editor.lib2 without error? > (I get "NoClassDefFoundError: org/netbeans/editor/GuardedDocument" > errors > there.) > 3) Why does Jenkins not detect the failures in > org.netbeans.junit.OrderHid.testSnd? (Seen in the console log e.g. for > https://builds.apache.org/job/incubator-netbeans-linux/496 ) > > -- Eirik > > -Original Message- > From: Geertjan Wielenga > Sent: Wednesday, July 25, 2018 2:35 PM > To: dev > Subject: Re: Does Travis and/or Jenkins run the NetBeans test suite? > > But does this not answer the question: > > FYI: > > https://builds.apache.org/job/incubator-netbeans-linux/ > > and > > https://builds.apache.org/job/incubator-netbeans-windows > > run platform tests. > > > > Gj > > > > On Wed, Jul 18, 2018 at 1:43 PM, Eirik Bakke wrote: > > > Looking at the "last stable build" of incubator-netbeans-linux, this > > one reports "no failures", yet has a failure in > > org.netbeans.junit.OrderHid.testSnd > > (and a few more also in the nbjunit module): > > > > https://builds.apache.org/job/incubator-netbeans-linux/480 > > > > Is there somehting special about the nbjunit module that makes > > Jenkins ignore test failures there, and only there? (I don't see it > > in the list of tests at > > https://builds.apache.org/job/incubator-netbeans-linux/480/ > > testReport/ .) > > > > These Jenkins builds also omit all the tests in the "ide" cluster, right? > > E.g. most of the editor.* modules? When I run tests in editor.lib2 > > myself, I get several errors [1]. I'm trying to figure out whether > > this is due to my own setup, or whether it's a problem in the > > codebase > itself. > > > > -- Eirik > > [1] All which boil down to "task failed due to: > java.lang.NoClassDefFoundError: > > org/netbeans/editor/GuardedDocument", e.g. in > org.netbeans.modules.editor. > > lib2.highlighting.HighlightingManagerTest . Seems like something I'm > > doing wrong with the build--unless there's an error in a static > > initializer somewhere. I tried a clean build of the entire > > incubator-netbeans repo, both on MacOS and Windows. I used "ant > > -Dtest-unit-sys-prop.ignore.random.failures=true > > -Dtest.modules=editor.lib2 test" and inspected the output for failures. > > > > On 7/18/18, 6:24 AM, "Jaroslav Tulach" > jaroslav.tul...@gmail.com>> wrote: > > > > FYI: > > https://builds.apache.org/job/incubator-netbeans-linux/ > > and > > https://builds.apache.org/job/incubator-netbeans-windows > > run platform tests. > > > > -jt > > > > > > Dne neděle 15. července 2018 18:47:51 CEST, Eirik Bakke napsal(a): > > When I make a pull request on GitHub, there is a nice little > > checkmark saying "All checks have passed", with a link to a Travis CI build > > (e.g. > > https://travis-ci.org/apache/incubator-netbeans/builds/ > > 395547620?utm_source > > =github_status_medium=notification ). > > Looking at the raw output of the Travis build, am I correct that > > this does _not_ actually run the NetBeans test suite? I searched the > > console output and did not find expected messages such as "Tests run:" > > or "do-junit" or "[junit]". > > Is this also the case for the Jenkins builds at > > https://builds.apache.org/view/Incubator%20Projects/job/ > > incubator-netbeans-> release ? > > Is the current codebase supposed to pass all tests at this point? > > When I check out the 9.0-vc3 tag, for instance, both of the > > following fail with various errors: > > ant commit-validation > > ant -Dtest-unit-sys-prop.ignore.random.failures=true test Are these > > supposed to work? > > -- Eirik > > > > > > > > > > > > > -- http://timboudreau.com
Re: Does Travis and/or Jenkins run the NetBeans test suite?
I don't know that this is the answer, but I know there used to be a mechanism for marking tests that fail randomly or were known to fail but hadn't been fixed, so the test suite didn't count them, or flagged an unexpected pass. Maybe that's being used? -Tim On Wed, Jul 25, 2018 at 12:54 PM Eirik Bakke wrote: > Partially--my follow-up questions are: > 1) Am I correct that tests in editor.lib2 (and many other editor.* > modules) are _not_ included in the tests run by Jenkins (at > https://builds.apache.org/job/incubator-netbeans-linux ), since they are > not part of the "platform" cluster? > 2) Are anyone else able to run the tests in editor.lib2 without error? (I > get "NoClassDefFoundError: org/netbeans/editor/GuardedDocument" errors > there.) > 3) Why does Jenkins not detect the failures in > org.netbeans.junit.OrderHid.testSnd? (Seen in the console log e.g. for > https://builds.apache.org/job/incubator-netbeans-linux/496 ) > > -- Eirik > > -Original Message- > From: Geertjan Wielenga > Sent: Wednesday, July 25, 2018 2:35 PM > To: dev > Subject: Re: Does Travis and/or Jenkins run the NetBeans test suite? > > But does this not answer the question: > > FYI: > > https://builds.apache.org/job/incubator-netbeans-linux/ > > and > > https://builds.apache.org/job/incubator-netbeans-windows > > run platform tests. > > > > Gj > > > > On Wed, Jul 18, 2018 at 1:43 PM, Eirik Bakke wrote: > > > Looking at the "last stable build" of incubator-netbeans-linux, this > > one reports "no failures", yet has a failure in > > org.netbeans.junit.OrderHid.testSnd > > (and a few more also in the nbjunit module): > > > > https://builds.apache.org/job/incubator-netbeans-linux/480 > > > > Is there somehting special about the nbjunit module that makes Jenkins > > ignore test failures there, and only there? (I don't see it in the > > list of tests at > > https://builds.apache.org/job/incubator-netbeans-linux/480/ > > testReport/ .) > > > > These Jenkins builds also omit all the tests in the "ide" cluster, right? > > E.g. most of the editor.* modules? When I run tests in editor.lib2 > > myself, I get several errors [1]. I'm trying to figure out whether > > this is due to my own setup, or whether it's a problem in the codebase > itself. > > > > -- Eirik > > [1] All which boil down to "task failed due to: > java.lang.NoClassDefFoundError: > > org/netbeans/editor/GuardedDocument", e.g. in > org.netbeans.modules.editor. > > lib2.highlighting.HighlightingManagerTest . Seems like something I'm > > doing wrong with the build--unless there's an error in a static > > initializer somewhere. I tried a clean build of the entire > > incubator-netbeans repo, both on MacOS and Windows. I used "ant > > -Dtest-unit-sys-prop.ignore.random.failures=true > > -Dtest.modules=editor.lib2 test" and inspected the output for failures. > > > > On 7/18/18, 6:24 AM, "Jaroslav Tulach" > jaroslav.tul...@gmail.com>> wrote: > > > > FYI: > > https://builds.apache.org/job/incubator-netbeans-linux/ > > and > > https://builds.apache.org/job/incubator-netbeans-windows > > run platform tests. > > > > -jt > > > > > > Dne neděle 15. července 2018 18:47:51 CEST, Eirik Bakke napsal(a): > > When I make a pull request on GitHub, there is a nice little checkmark > > saying "All checks have passed", with a link to a Travis CI build (e.g. > > https://travis-ci.org/apache/incubator-netbeans/builds/ > > 395547620?utm_source > > =github_status_medium=notification ). > > Looking at the raw output of the Travis build, am I correct that this > > does _not_ actually run the NetBeans test suite? I searched the > > console output and did not find expected messages such as "Tests run:" > > or "do-junit" or "[junit]". > > Is this also the case for the Jenkins builds at > > https://builds.apache.org/view/Incubator%20Projects/job/ > > incubator-netbeans-> release ? > > Is the current codebase supposed to pass all tests at this point? When > > I check out the 9.0-vc3 tag, for instance, both of the following fail > > with various errors: > > ant commit-validation > > ant -Dtest-unit-sys-prop.ignore.random.failures=true test Are these > > supposed to work? > > -- Eirik > > > > > > > > > > > > > -- http://timboudreau.com
RE: Does Travis and/or Jenkins run the NetBeans test suite?
Partially--my follow-up questions are: 1) Am I correct that tests in editor.lib2 (and many other editor.* modules) are _not_ included in the tests run by Jenkins (at https://builds.apache.org/job/incubator-netbeans-linux ), since they are not part of the "platform" cluster? 2) Are anyone else able to run the tests in editor.lib2 without error? (I get "NoClassDefFoundError: org/netbeans/editor/GuardedDocument" errors there.) 3) Why does Jenkins not detect the failures in org.netbeans.junit.OrderHid.testSnd? (Seen in the console log e.g. for https://builds.apache.org/job/incubator-netbeans-linux/496 ) -- Eirik -Original Message- From: Geertjan Wielenga Sent: Wednesday, July 25, 2018 2:35 PM To: dev Subject: Re: Does Travis and/or Jenkins run the NetBeans test suite? But does this not answer the question: FYI: > https://builds.apache.org/job/incubator-netbeans-linux/ > and > https://builds.apache.org/job/incubator-netbeans-windows > run platform tests. > Gj On Wed, Jul 18, 2018 at 1:43 PM, Eirik Bakke wrote: > Looking at the "last stable build" of incubator-netbeans-linux, this > one reports "no failures", yet has a failure in > org.netbeans.junit.OrderHid.testSnd > (and a few more also in the nbjunit module): > > https://builds.apache.org/job/incubator-netbeans-linux/480 > > Is there somehting special about the nbjunit module that makes Jenkins > ignore test failures there, and only there? (I don't see it in the > list of tests at > https://builds.apache.org/job/incubator-netbeans-linux/480/ > testReport/ .) > > These Jenkins builds also omit all the tests in the "ide" cluster, right? > E.g. most of the editor.* modules? When I run tests in editor.lib2 > myself, I get several errors [1]. I'm trying to figure out whether > this is due to my own setup, or whether it's a problem in the codebase itself. > > -- Eirik > [1] All which boil down to "task failed due to: > java.lang.NoClassDefFoundError: > org/netbeans/editor/GuardedDocument", e.g. in org.netbeans.modules.editor. > lib2.highlighting.HighlightingManagerTest . Seems like something I'm > doing wrong with the build--unless there's an error in a static > initializer somewhere. I tried a clean build of the entire > incubator-netbeans repo, both on MacOS and Windows. I used "ant > -Dtest-unit-sys-prop.ignore.random.failures=true > -Dtest.modules=editor.lib2 test" and inspected the output for failures. > > On 7/18/18, 6:24 AM, "Jaroslav Tulach" jaroslav.tul...@gmail.com>> wrote: > > FYI: > https://builds.apache.org/job/incubator-netbeans-linux/ > and > https://builds.apache.org/job/incubator-netbeans-windows > run platform tests. > > -jt > > > Dne neděle 15. července 2018 18:47:51 CEST, Eirik Bakke napsal(a): > When I make a pull request on GitHub, there is a nice little checkmark > saying "All checks have passed", with a link to a Travis CI build (e.g. > https://travis-ci.org/apache/incubator-netbeans/builds/ > 395547620?utm_source > =github_status_medium=notification ). > Looking at the raw output of the Travis build, am I correct that this > does _not_ actually run the NetBeans test suite? I searched the > console output and did not find expected messages such as "Tests run:" > or "do-junit" or "[junit]". > Is this also the case for the Jenkins builds at > https://builds.apache.org/view/Incubator%20Projects/job/ > incubator-netbeans-> release ? > Is the current codebase supposed to pass all tests at this point? When > I check out the 9.0-vc3 tag, for instance, both of the following fail > with various errors: > ant commit-validation > ant -Dtest-unit-sys-prop.ignore.random.failures=true test Are these > supposed to work? > -- Eirik > > > > > >
Re: Does Travis and/or Jenkins run the NetBeans test suite?
But does this not answer the question: FYI: > https://builds.apache.org/job/incubator-netbeans-linux/ > and > https://builds.apache.org/job/incubator-netbeans-windows > run platform tests. > Gj On Wed, Jul 18, 2018 at 1:43 PM, Eirik Bakke wrote: > Looking at the "last stable build" of incubator-netbeans-linux, this one > reports "no failures", yet has a failure in > org.netbeans.junit.OrderHid.testSnd > (and a few more also in the nbjunit module): > > https://builds.apache.org/job/incubator-netbeans-linux/480 > > Is there somehting special about the nbjunit module that makes Jenkins > ignore test failures there, and only there? (I don't see it in the list of > tests at https://builds.apache.org/job/incubator-netbeans-linux/480/ > testReport/ .) > > These Jenkins builds also omit all the tests in the "ide" cluster, right? > E.g. most of the editor.* modules? When I run tests in editor.lib2 myself, > I get several errors [1]. I'm trying to figure out whether this is due to > my own setup, or whether it's a problem in the codebase itself. > > -- Eirik > [1] All which boil down to "task failed due to: > java.lang.NoClassDefFoundError: > org/netbeans/editor/GuardedDocument", e.g. in org.netbeans.modules.editor. > lib2.highlighting.HighlightingManagerTest . Seems like something I'm > doing wrong with the build--unless there's an error in a static initializer > somewhere. I tried a clean build of the entire incubator-netbeans repo, > both on MacOS and Windows. I used "ant > -Dtest-unit-sys-prop.ignore.random.failures=true > -Dtest.modules=editor.lib2 test" and inspected the output for failures. > > On 7/18/18, 6:24 AM, "Jaroslav Tulach" jaroslav.tul...@gmail.com>> wrote: > > FYI: > https://builds.apache.org/job/incubator-netbeans-linux/ > and > https://builds.apache.org/job/incubator-netbeans-windows > run platform tests. > > -jt > > > Dne neděle 15. července 2018 18:47:51 CEST, Eirik Bakke napsal(a): > When I make a pull request on GitHub, there is a nice little checkmark > saying "All checks have passed", with a link to a Travis CI build (e.g. > https://travis-ci.org/apache/incubator-netbeans/builds/ > 395547620?utm_source > =github_status_medium=notification ). > Looking at the raw output of the Travis build, am I correct that this does > _not_ actually run the NetBeans test suite? I searched the console output > and did not find expected messages such as "Tests run:" or "do-junit" or > "[junit]". > Is this also the case for the Jenkins builds at > https://builds.apache.org/view/Incubator%20Projects/job/ > incubator-netbeans-> release ? > Is the current codebase supposed to pass all tests at this point? When I > check out the 9.0-vc3 tag, for instance, both of the following fail with > various errors: > ant commit-validation > ant -Dtest-unit-sys-prop.ignore.random.failures=true test > Are these supposed to work? > -- Eirik > > > > > >
Re: Does Travis and/or Jenkins run the NetBeans test suite?
Looking at the "last stable build" of incubator-netbeans-linux, this one reports "no failures", yet has a failure in org.netbeans.junit.OrderHid.testSnd (and a few more also in the nbjunit module): https://builds.apache.org/job/incubator-netbeans-linux/480 Is there somehting special about the nbjunit module that makes Jenkins ignore test failures there, and only there? (I don't see it in the list of tests at https://builds.apache.org/job/incubator-netbeans-linux/480/testReport/ .) These Jenkins builds also omit all the tests in the "ide" cluster, right? E.g. most of the editor.* modules? When I run tests in editor.lib2 myself, I get several errors [1]. I'm trying to figure out whether this is due to my own setup, or whether it's a problem in the codebase itself. -- Eirik [1] All which boil down to "task failed due to: java.lang.NoClassDefFoundError: org/netbeans/editor/GuardedDocument", e.g. in org.netbeans.modules.editor.lib2.highlighting.HighlightingManagerTest . Seems like something I'm doing wrong with the build--unless there's an error in a static initializer somewhere. I tried a clean build of the entire incubator-netbeans repo, both on MacOS and Windows. I used "ant -Dtest-unit-sys-prop.ignore.random.failures=true -Dtest.modules=editor.lib2 test" and inspected the output for failures. On 7/18/18, 6:24 AM, "Jaroslav Tulach" mailto:jaroslav.tul...@gmail.com>> wrote: FYI: https://builds.apache.org/job/incubator-netbeans-linux/ and https://builds.apache.org/job/incubator-netbeans-windows run platform tests. -jt Dne neděle 15. července 2018 18:47:51 CEST, Eirik Bakke napsal(a): When I make a pull request on GitHub, there is a nice little checkmark saying "All checks have passed", with a link to a Travis CI build (e.g. https://travis-ci.org/apache/incubator-netbeans/builds/395547620?utm_source =github_status_medium=notification ). Looking at the raw output of the Travis build, am I correct that this does _not_ actually run the NetBeans test suite? I searched the console output and did not find expected messages such as "Tests run:" or "do-junit" or "[junit]". Is this also the case for the Jenkins builds at https://builds.apache.org/view/Incubator%20Projects/job/incubator-netbeans-> release ? Is the current codebase supposed to pass all tests at this point? When I check out the 9.0-vc3 tag, for instance, both of the following fail with various errors: ant commit-validation ant -Dtest-unit-sys-prop.ignore.random.failures=true test Are these supposed to work? -- Eirik
Re: Does Travis and/or Jenkins run the NetBeans test suite?
That would be good... I an interested in doing this for Groovy Support as well... maybe we can figure out a standard way how to do this. -Sven Jan Lahoda schrieb am Mi., 18. Juli 2018, 08:05: > FWIW, I'd like to set-up testing of the java.completion module on various > JDKs (and eventually other Java-related modules), but I didn't get to that > yet. > > Jan > > On Wed, Jul 18, 2018 at 6:24 AM, Jaroslav Tulach < > jaroslav.tul...@gmail.com> > wrote: > > > FYI: > > https://builds.apache.org/job/incubator-netbeans-linux/ > > and > > https://builds.apache.org/job/incubator-netbeans-windows > > run platform tests. > > > > -jt > > > > > > Dne neděle 15. července 2018 18:47:51 CEST, Eirik Bakke napsal(a): > > > When I make a pull request on GitHub, there is a nice little checkmark > > > saying "All checks have passed", with a link to a Travis CI build (e.g. > > > https://travis-ci.org/apache/incubator-netbeans/builds/ > > 395547620?utm_source > > > =github_status_medium=notification ). > > > > > > Looking at the raw output of the Travis build, am I correct that this > > does > > > _not_ actually run the NetBeans test suite? I searched the console > output > > > and did not find expected messages such as "Tests run:" or "do-junit" > or > > > "[junit]". > > > > > > Is this also the case for the Jenkins builds at > > > https://builds.apache.org/view/Incubator%20Projects/job/ > > incubator-netbeans-> release ? > > > > > > Is the current codebase supposed to pass all tests at this point? When > I > > > check out the 9.0-vc3 tag, for instance, both of the following fail > with > > > various errors: > > > > > > ant commit-validation > > > ant -Dtest-unit-sys-prop.ignore.random.failures=true test > > > > > > Are these supposed to work? > > > > > > -- Eirik > > > > > > > > > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@netbeans.incubator.apache.org > > For additional commands, e-mail: dev-h...@netbeans.incubator.apache.org > > > > For further information about the NetBeans mailing lists, visit: > > https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists > > > > > > > > >
Re: Does Travis and/or Jenkins run the NetBeans test suite?
FYI: https://builds.apache.org/job/incubator-netbeans-linux/ and https://builds.apache.org/job/incubator-netbeans-windows run platform tests. -jt Dne neděle 15. července 2018 18:47:51 CEST, Eirik Bakke napsal(a): > When I make a pull request on GitHub, there is a nice little checkmark > saying "All checks have passed", with a link to a Travis CI build (e.g. > https://travis-ci.org/apache/incubator-netbeans/builds/395547620?utm_source > =github_status_medium=notification ). > > Looking at the raw output of the Travis build, am I correct that this does > _not_ actually run the NetBeans test suite? I searched the console output > and did not find expected messages such as "Tests run:" or "do-junit" or > "[junit]". > > Is this also the case for the Jenkins builds at > https://builds.apache.org/view/Incubator%20Projects/job/incubator-netbeans-> > release ? > > Is the current codebase supposed to pass all tests at this point? When I > check out the 9.0-vc3 tag, for instance, both of the following fail with > various errors: > > ant commit-validation > ant -Dtest-unit-sys-prop.ignore.random.failures=true test > > Are these supposed to work? > > -- Eirik - To unsubscribe, e-mail: dev-unsubscr...@netbeans.incubator.apache.org For additional commands, e-mail: dev-h...@netbeans.incubator.apache.org For further information about the NetBeans mailing lists, visit: https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Does Travis and/or Jenkins run the NetBeans test suite?
I would answer if I knew the answer, best would be for Jan Lahoda to answer this. Gj On Sunday, July 15, 2018, Eirik Bakke wrote: > When I make a pull request on GitHub, there is a nice little checkmark > saying "All checks have passed", with a link to a Travis CI build (e.g. > https://travis-ci.org/apache/incubator-netbeans/builds/ > 395547620?utm_source=github_status_medium=notification ). > > Looking at the raw output of the Travis build, am I correct that this does > _not_ actually run the NetBeans test suite? I searched the console output > and did not find expected messages such as "Tests run:" or "do-junit" or > "[junit]". > > Is this also the case for the Jenkins builds at https://builds.apache.org/ > view/Incubator%20Projects/job/incubator-netbeans-release ? > > Is the current codebase supposed to pass all tests at this point? When I > check out the 9.0-vc3 tag, for instance, both of the following fail with > various errors: > > ant commit-validation > ant -Dtest-unit-sys-prop.ignore.random.failures=true test > > Are these supposed to work? > > -- Eirik > >
Does Travis and/or Jenkins run the NetBeans test suite?
When I make a pull request on GitHub, there is a nice little checkmark saying "All checks have passed", with a link to a Travis CI build (e.g. https://travis-ci.org/apache/incubator-netbeans/builds/395547620?utm_source=github_status_medium=notification ). Looking at the raw output of the Travis build, am I correct that this does _not_ actually run the NetBeans test suite? I searched the console output and did not find expected messages such as "Tests run:" or "do-junit" or "[junit]". Is this also the case for the Jenkins builds at https://builds.apache.org/view/Incubator%20Projects/job/incubator-netbeans-release ? Is the current codebase supposed to pass all tests at this point? When I check out the 9.0-vc3 tag, for instance, both of the following fail with various errors: ant commit-validation ant -Dtest-unit-sys-prop.ignore.random.failures=true test Are these supposed to work? -- Eirik
