You should instead have a fast profile, with a subset of the testsuite to run on every commit and branch for these cases. I looked on Jenkins and having many builds taking 3 Hours each won't really scale on the lab anyway. Failures will only make things worse there.
The lab is usually not powerful for long running tests. And a full profile that should run as part of a full run. (say.. once a day instead of every commit), or any interval you chose. I don't think you should hide tests though.. as that is like pushing dirt under the rug.. (even if you say to enable it later... as in anything in life temporary solutions endup being definitive usually). As any System dealing with times and asynchronous flaky and races are part of the day. One thing I did in ActiveMQ Artemis was to write a Rule where the test is retried. You could also add retries to tests in cases where it is acceptable... but be careful to not just hide bugs away in this case as well. If you are interested, on artemis, Look for usages on https://github.com/apache/activemq-artemis/blob/master/artemis-commons/src/test/java/org/apache/activemq/artemis/utils/RetryRule.java You need to activate a profile in artemis for the retryRule to work. On Fri, Mar 12, 2021 at 1:56 PM JB Onofré <[email protected]> wrote: > > Yes agree. I’m launching new builds ;) > > > Le 12 mars 2021 à 19:51, Christopher Shannon > > <[email protected]> a écrit : > > > > Just running it by itself on the command line and also in the IDE. The full > > build takes a while and if it's breaking with that then it's probably some > > other test that isn't cleaning up properly in between runs. > > > >> On Fri, Mar 12, 2021 at 1:47 PM JB Onofré <[email protected]> wrote: > >> > >> Did you try in a full build or the test individually ? I’m running a new > >> build. > >> > >>> Le 12 mars 2021 à 19:38, Christopher Shannon < > >> [email protected]> a écrit : > >>> > >>> I've been running the DurableSyncNetworkBridgeTest several times on my > >> box > >>> and it always passes. > >>> > >>>> On Fri, Mar 12, 2021 at 1:25 PM Christopher Shannon < > >>>> [email protected]> wrote: > >>>> > >>>> Ideally it would be better to fix tests than to simply exclude them. > >> These > >>>> tests were added for a reason I would presume (I know I had worked on > >> the > >>>> durable sync stuff in the past) so randomly turning off tests could > >> lead to > >>>> missing errors. > >>>> > >>>> > >>>> > >>>> On Fri, Mar 12, 2021 at 12:57 PM Jean-Baptiste Onofre <[email protected]> > >>>> wrote: > >>>> > >>>>> I’m adding these tests to be fixed/improved: > >>>>> > >>>>> FailoverDurableSubTransactionTest.testFailoverCommitListener > >>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionPropagate > >>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionWithBridgeOffline > >>>>> > >>>>> Let me create the Jira and create a PR to exclude the tests and verify > >>>>> Jenkins is happy. > >>>>> > >>>>> Regards > >>>>> JB > >>>>> > >>>>>> Le 12 mars 2021 à 16:14, Jonathan Gallimore < > >>>>> [email protected]> a écrit : > >>>>>> > >>>>>> I'm +1 on the actions :). > >>>>>> > >>>>>> Jon > >>>>>> > >>>>>> On Fri, Mar 12, 2021 at 3:11 PM Jean-Baptiste Onofre <[email protected] > >>> > >>>>>> wrote: > >>>>>> > >>>>>>> Sure, thanks for the help ! > >>>>>>> > >>>>>>> Just waiting for some feedback before starting the "actions" ;) > >>>>>>> > >>>>>>> Regards > >>>>>>> JB > >>>>>>> > >>>>>>>> Le 12 mars 2021 à 14:29, Jonathan Gallimore < > >>>>>>> [email protected]> a écrit : > >>>>>>>> > >>>>>>>> I ran into this test failing yesterday: > >>>>>>>> > >>>>>>> > >>>>> > >> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java > >>>>>>>> - I'd be happy to try and contribute a fix. Would you like to assign > >>>>> the > >>>>>>>> JIRA to me? > >>>>>>>> > >>>>>>>> Jon > >>>>>>>> > >>>>>>>> On Fri, Mar 12, 2021 at 12:58 PM Jean-Baptiste Onofre < > >>>>> [email protected]> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> Hi guys, > >>>>>>>>> > >>>>>>>>> Now that we have Jenkinsfile in our repo, and we use Jenkins > >>>>> pipeline, > >>>>>>> we > >>>>>>>>> dramatically improved our build: the build is executed for each > >>>>>>>>> PullRequests or commit on the main branch. > >>>>>>>>> > >>>>>>>>> However, we have lot of failing tests, causing quite systematically > >>>>> the > >>>>>>>>> build failing on ci-builds.apache.org. > >>>>>>>>> > >>>>>>>>> We really need to have a clean, accurate and stable build: it will > >>>>>>> improve > >>>>>>>>> the issue detection and simplify the review, especially for > >>>>>>> PullRequests. > >>>>>>>>> > >>>>>>>>> I ran several builds on my machine (with different docker > >> containers) > >>>>>>> and > >>>>>>>>> I already identified some failing/flaky tests: > >>>>>>>>> > >>>>>>>>> - > >>>>>>>>> > >>>>>>> > >>>>> > >> activemq-leveldb-store/src/test/java/org/apache/activemq/leveldb/test/ElectingLevelDBStoreTest.java > >>>>>>>>> is not a big deal as I have a PR removing leveled completely > >>>>>>>>> - > >>>>>>>>> > >>>>>>> > >>>>> > >> activemq-stomp/src/test/java/org/apache/activemq/transport/stomp/Stomp11NIOSSLTest.java. > >>>>>>>>> Chris did an improvement, but I still have some flakiness here. > >>>>>>>>> - > >>>>>>>>> > >>>>>>> > >>>>> > >> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java > >>>>>>>>> > >>>>>>>>> I propose the following action plan: > >>>>>>>>> > >>>>>>>>> 1. Create the Jira for each failing/flaky tests > >>>>>>>>> 2. Exclude the tests (in surefire plugin configuration) to have a > >>>>> "green > >>>>>>>>> light" on Jenkins. > >>>>>>>>> 3. For each Jira, we work on a PullRequest, to be sure that Jenkins > >>>>> is > >>>>>>>>> still "happy". > >>>>>>>>> > >>>>>>>>> Anyone willing to help on (3) is welcome ! > >>>>>>>>> > >>>>>>>>> If there’s no objection, I will start with (1) and (2). > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> Regards > >>>>>>>>> JB > >>>>>>> > >>>>>>> > >>>>> > >>>>> > >> > >> > -- Clebert Suconic
