Hi, We already have "fast" profile, and it’s good idea to use this profile on Jenkins by default and move some tests here.
For instance, I don’t think it’s require to launch all activemq-unit-test by default but I would keep the tests in each module (they are fast and doesn’t need whole broker infra). About RetryRule, I did that in Karaf as well, let me see if it helps for ActiveMQ. Thanks ! I will improve this way. Regards JB > Le 12 mars 2021 à 20:31, Clebert Suconic <[email protected]> a écrit : > > You should instead have a fast profile, with a subset of the testsuite > to run on every commit and branch for these cases. I looked on Jenkins > and having many builds taking 3 Hours each won't really scale on the > lab anyway. Failures will only make things worse there. > > The lab is usually not powerful for long running tests. > > And a full profile that should run as part of a full run. (say.. once > a day instead of every commit), or any interval you chose. > > I don't think you should hide tests though.. as that is like pushing > dirt under the rug.. (even if you say to enable it later... as in > anything in life temporary solutions endup being definitive usually). > > As any System dealing with times and asynchronous flaky and races are > part of the day. One thing I did in ActiveMQ Artemis was to write a > Rule where the test is retried. You could also add retries to tests in > cases where it is acceptable... but be careful to not just hide bugs > away in this case as well. > > If you are interested, on artemis, Look for usages on > https://github.com/apache/activemq-artemis/blob/master/artemis-commons/src/test/java/org/apache/activemq/artemis/utils/RetryRule.java > > > You need to activate a profile in artemis for the retryRule to work. > > On Fri, Mar 12, 2021 at 1:56 PM JB Onofré <[email protected]> wrote: >> >> Yes agree. I’m launching new builds ;) >> >>> Le 12 mars 2021 à 19:51, Christopher Shannon >>> <[email protected]> a écrit : >>> >>> Just running it by itself on the command line and also in the IDE. The full >>> build takes a while and if it's breaking with that then it's probably some >>> other test that isn't cleaning up properly in between runs. >>> >>>> On Fri, Mar 12, 2021 at 1:47 PM JB Onofré <[email protected]> wrote: >>>> >>>> Did you try in a full build or the test individually ? I’m running a new >>>> build. >>>> >>>>> Le 12 mars 2021 à 19:38, Christopher Shannon < >>>> [email protected]> a écrit : >>>>> >>>>> I've been running the DurableSyncNetworkBridgeTest several times on my >>>> box >>>>> and it always passes. >>>>> >>>>>> On Fri, Mar 12, 2021 at 1:25 PM Christopher Shannon < >>>>>> [email protected]> wrote: >>>>>> >>>>>> Ideally it would be better to fix tests than to simply exclude them. >>>> These >>>>>> tests were added for a reason I would presume (I know I had worked on >>>> the >>>>>> durable sync stuff in the past) so randomly turning off tests could >>>> lead to >>>>>> missing errors. >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Mar 12, 2021 at 12:57 PM Jean-Baptiste Onofre <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> I’m adding these tests to be fixed/improved: >>>>>>> >>>>>>> FailoverDurableSubTransactionTest.testFailoverCommitListener >>>>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionPropagate >>>>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionWithBridgeOffline >>>>>>> >>>>>>> Let me create the Jira and create a PR to exclude the tests and verify >>>>>>> Jenkins is happy. >>>>>>> >>>>>>> Regards >>>>>>> JB >>>>>>> >>>>>>>> Le 12 mars 2021 à 16:14, Jonathan Gallimore < >>>>>>> [email protected]> a écrit : >>>>>>>> >>>>>>>> I'm +1 on the actions :). >>>>>>>> >>>>>>>> Jon >>>>>>>> >>>>>>>> On Fri, Mar 12, 2021 at 3:11 PM Jean-Baptiste Onofre <[email protected] >>>>> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Sure, thanks for the help ! >>>>>>>>> >>>>>>>>> Just waiting for some feedback before starting the "actions" ;) >>>>>>>>> >>>>>>>>> Regards >>>>>>>>> JB >>>>>>>>> >>>>>>>>>> Le 12 mars 2021 à 14:29, Jonathan Gallimore < >>>>>>>>> [email protected]> a écrit : >>>>>>>>>> >>>>>>>>>> I ran into this test failing yesterday: >>>>>>>>>> >>>>>>>>> >>>>>>> >>>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java >>>>>>>>>> - I'd be happy to try and contribute a fix. Would you like to assign >>>>>>> the >>>>>>>>>> JIRA to me? >>>>>>>>>> >>>>>>>>>> Jon >>>>>>>>>> >>>>>>>>>> On Fri, Mar 12, 2021 at 12:58 PM Jean-Baptiste Onofre < >>>>>>> [email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi guys, >>>>>>>>>>> >>>>>>>>>>> Now that we have Jenkinsfile in our repo, and we use Jenkins >>>>>>> pipeline, >>>>>>>>> we >>>>>>>>>>> dramatically improved our build: the build is executed for each >>>>>>>>>>> PullRequests or commit on the main branch. >>>>>>>>>>> >>>>>>>>>>> However, we have lot of failing tests, causing quite systematically >>>>>>> the >>>>>>>>>>> build failing on ci-builds.apache.org. >>>>>>>>>>> >>>>>>>>>>> We really need to have a clean, accurate and stable build: it will >>>>>>>>> improve >>>>>>>>>>> the issue detection and simplify the review, especially for >>>>>>>>> PullRequests. >>>>>>>>>>> >>>>>>>>>>> I ran several builds on my machine (with different docker >>>> containers) >>>>>>>>> and >>>>>>>>>>> I already identified some failing/flaky tests: >>>>>>>>>>> >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>> >>>>>>> >>>> activemq-leveldb-store/src/test/java/org/apache/activemq/leveldb/test/ElectingLevelDBStoreTest.java >>>>>>>>>>> is not a big deal as I have a PR removing leveled completely >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>> >>>>>>> >>>> activemq-stomp/src/test/java/org/apache/activemq/transport/stomp/Stomp11NIOSSLTest.java. >>>>>>>>>>> Chris did an improvement, but I still have some flakiness here. >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>> >>>>>>> >>>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java >>>>>>>>>>> >>>>>>>>>>> I propose the following action plan: >>>>>>>>>>> >>>>>>>>>>> 1. Create the Jira for each failing/flaky tests >>>>>>>>>>> 2. Exclude the tests (in surefire plugin configuration) to have a >>>>>>> "green >>>>>>>>>>> light" on Jenkins. >>>>>>>>>>> 3. For each Jira, we work on a PullRequest, to be sure that Jenkins >>>>>>> is >>>>>>>>>>> still "happy". >>>>>>>>>>> >>>>>>>>>>> Anyone willing to help on (3) is welcome ! >>>>>>>>>>> >>>>>>>>>>> If there’s no objection, I will start with (1) and (2). >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Regards >>>>>>>>>>> JB >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>> >>>> >> > > > -- > Clebert Suconic
