Hi guys, I’ve updated my local branch according to your comments:
1. I’ve cleanup the profiles and introduce/rename a fast profile that executes all unit tests in modules but exclude the activemq-unit-tests and karaf-itests. 2. I’m keeping the smoke test profile 3. I’ve created a tobefixed profile that include all flaky tests I’ve identified 4. I’ve updated Jenkinsfile to use fast profile on PR I will create the PR soon. Regards JB > Le 13 mars 2021 à 06:05, Jean-Baptiste Onofre <[email protected]> a écrit : > > Hi, > > We already have "fast" profile, and it’s good idea to use this profile on > Jenkins by default and move some tests here. > > For instance, I don’t think it’s require to launch all activemq-unit-test by > default but I would keep the tests in each module (they are fast and doesn’t > need whole broker infra). > > About RetryRule, I did that in Karaf as well, let me see if it helps for > ActiveMQ. > > Thanks ! > I will improve this way. > > Regards > JB > >> Le 12 mars 2021 à 20:31, Clebert Suconic <[email protected]> a écrit >> : >> >> You should instead have a fast profile, with a subset of the testsuite >> to run on every commit and branch for these cases. I looked on Jenkins >> and having many builds taking 3 Hours each won't really scale on the >> lab anyway. Failures will only make things worse there. >> >> The lab is usually not powerful for long running tests. >> >> And a full profile that should run as part of a full run. (say.. once >> a day instead of every commit), or any interval you chose. >> >> I don't think you should hide tests though.. as that is like pushing >> dirt under the rug.. (even if you say to enable it later... as in >> anything in life temporary solutions endup being definitive usually). >> >> As any System dealing with times and asynchronous flaky and races are >> part of the day. One thing I did in ActiveMQ Artemis was to write a >> Rule where the test is retried. You could also add retries to tests in >> cases where it is acceptable... but be careful to not just hide bugs >> away in this case as well. >> >> If you are interested, on artemis, Look for usages on >> https://github.com/apache/activemq-artemis/blob/master/artemis-commons/src/test/java/org/apache/activemq/artemis/utils/RetryRule.java >> >> >> You need to activate a profile in artemis for the retryRule to work. >> >> On Fri, Mar 12, 2021 at 1:56 PM JB Onofré <[email protected]> wrote: >>> >>> Yes agree. I’m launching new builds ;) >>> >>>> Le 12 mars 2021 à 19:51, Christopher Shannon >>>> <[email protected]> a écrit : >>>> >>>> Just running it by itself on the command line and also in the IDE. The >>>> full >>>> build takes a while and if it's breaking with that then it's probably some >>>> other test that isn't cleaning up properly in between runs. >>>> >>>>> On Fri, Mar 12, 2021 at 1:47 PM JB Onofré <[email protected]> wrote: >>>>> >>>>> Did you try in a full build or the test individually ? I’m running a new >>>>> build. >>>>> >>>>>> Le 12 mars 2021 à 19:38, Christopher Shannon < >>>>> [email protected]> a écrit : >>>>>> >>>>>> I've been running the DurableSyncNetworkBridgeTest several times on my >>>>> box >>>>>> and it always passes. >>>>>> >>>>>>> On Fri, Mar 12, 2021 at 1:25 PM Christopher Shannon < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>> Ideally it would be better to fix tests than to simply exclude them. >>>>> These >>>>>>> tests were added for a reason I would presume (I know I had worked on >>>>> the >>>>>>> durable sync stuff in the past) so randomly turning off tests could >>>>> lead to >>>>>>> missing errors. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Mar 12, 2021 at 12:57 PM Jean-Baptiste Onofre >>>>>>> <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> I’m adding these tests to be fixed/improved: >>>>>>>> >>>>>>>> FailoverDurableSubTransactionTest.testFailoverCommitListener >>>>>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionPropagate >>>>>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionWithBridgeOffline >>>>>>>> >>>>>>>> Let me create the Jira and create a PR to exclude the tests and verify >>>>>>>> Jenkins is happy. >>>>>>>> >>>>>>>> Regards >>>>>>>> JB >>>>>>>> >>>>>>>>> Le 12 mars 2021 à 16:14, Jonathan Gallimore < >>>>>>>> [email protected]> a écrit : >>>>>>>>> >>>>>>>>> I'm +1 on the actions :). >>>>>>>>> >>>>>>>>> Jon >>>>>>>>> >>>>>>>>> On Fri, Mar 12, 2021 at 3:11 PM Jean-Baptiste Onofre >>>>>>>>> <[email protected] >>>>>> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Sure, thanks for the help ! >>>>>>>>>> >>>>>>>>>> Just waiting for some feedback before starting the "actions" ;) >>>>>>>>>> >>>>>>>>>> Regards >>>>>>>>>> JB >>>>>>>>>> >>>>>>>>>>> Le 12 mars 2021 à 14:29, Jonathan Gallimore < >>>>>>>>>> [email protected]> a écrit : >>>>>>>>>>> >>>>>>>>>>> I ran into this test failing yesterday: >>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java >>>>>>>>>>> - I'd be happy to try and contribute a fix. Would you like to assign >>>>>>>> the >>>>>>>>>>> JIRA to me? >>>>>>>>>>> >>>>>>>>>>> Jon >>>>>>>>>>> >>>>>>>>>>> On Fri, Mar 12, 2021 at 12:58 PM Jean-Baptiste Onofre < >>>>>>>> [email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi guys, >>>>>>>>>>>> >>>>>>>>>>>> Now that we have Jenkinsfile in our repo, and we use Jenkins >>>>>>>> pipeline, >>>>>>>>>> we >>>>>>>>>>>> dramatically improved our build: the build is executed for each >>>>>>>>>>>> PullRequests or commit on the main branch. >>>>>>>>>>>> >>>>>>>>>>>> However, we have lot of failing tests, causing quite systematically >>>>>>>> the >>>>>>>>>>>> build failing on ci-builds.apache.org. >>>>>>>>>>>> >>>>>>>>>>>> We really need to have a clean, accurate and stable build: it will >>>>>>>>>> improve >>>>>>>>>>>> the issue detection and simplify the review, especially for >>>>>>>>>> PullRequests. >>>>>>>>>>>> >>>>>>>>>>>> I ran several builds on my machine (with different docker >>>>> containers) >>>>>>>>>> and >>>>>>>>>>>> I already identified some failing/flaky tests: >>>>>>>>>>>> >>>>>>>>>>>> - >>>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>> activemq-leveldb-store/src/test/java/org/apache/activemq/leveldb/test/ElectingLevelDBStoreTest.java >>>>>>>>>>>> is not a big deal as I have a PR removing leveled completely >>>>>>>>>>>> - >>>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>> activemq-stomp/src/test/java/org/apache/activemq/transport/stomp/Stomp11NIOSSLTest.java. >>>>>>>>>>>> Chris did an improvement, but I still have some flakiness here. >>>>>>>>>>>> - >>>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java >>>>>>>>>>>> >>>>>>>>>>>> I propose the following action plan: >>>>>>>>>>>> >>>>>>>>>>>> 1. Create the Jira for each failing/flaky tests >>>>>>>>>>>> 2. Exclude the tests (in surefire plugin configuration) to have a >>>>>>>> "green >>>>>>>>>>>> light" on Jenkins. >>>>>>>>>>>> 3. For each Jira, we work on a PullRequest, to be sure that Jenkins >>>>>>>> is >>>>>>>>>>>> still "happy". >>>>>>>>>>>> >>>>>>>>>>>> Anyone willing to help on (3) is welcome ! >>>>>>>>>>>> >>>>>>>>>>>> If there’s no objection, I will start with (1) and (2). >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Regards >>>>>>>>>>>> JB >>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>> >>>>> >>> >> >> >> -- >> Clebert Suconic >
