Re: [xwiki-devs] [Proposal] Rerun flaky tests several times?
+1 On 11/06/2019 09:34, Thomas Mortagne wrote: +1 On Mon, Jun 10, 2019 at 11:43 AM Vincent Massol wrote: Hi devs, I was looking for a rerun optin in JUnit5 or Surefire but it seems it doesn’t exist for Junit5: * https://github.com/junit-team/junit5/issues/1558 * https://issues.apache.org/jira/browse/SUREFIRE-1584 However, there’s an extension to do that: * https://github.com/artsok/rerunner-jupiter Also WIP for JUnit Pioneer: * https://github.com/junit-pioneer/junit-pioneer/pull/110 So my proposal FTM is to try https://github.com/artsok/rerunner-jupiter and to start putting a repeat of 3 for JUnit5 tests that we know are flaky. For JUnit4 tests we could simply use: * https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html Note that this would also allow us to correct flaky tests that don’t set up the initial conditions (fixture) correctly. Second note: This is not an endorsement that we should accept flaky tests. They’re a plague and we need to work on them and fix them. This is just a momentary workaround to identify how flaky a test is and to try to have a better solution towards moving to a Continuous Delivery solution. WDYT? Thanks -Vincent -- Simon Urli Software Engineer at XWiki SAS [email protected] More about us at http://www.xwiki.com
Re: [xwiki-devs] [Proposal] Rerun flaky tests several times?
+1 On Mon, Jun 10, 2019 at 11:43 AM Vincent Massol wrote: > > Hi devs, > > I was looking for a rerun optin in JUnit5 or Surefire but it seems it doesn’t > exist for Junit5: > * https://github.com/junit-team/junit5/issues/1558 > * https://issues.apache.org/jira/browse/SUREFIRE-1584 > > However, there’s an extension to do that: > * https://github.com/artsok/rerunner-jupiter > > Also WIP for JUnit Pioneer: > * https://github.com/junit-pioneer/junit-pioneer/pull/110 > > So my proposal FTM is to try https://github.com/artsok/rerunner-jupiter and > to start putting a repeat of 3 for JUnit5 tests that we know are flaky. > > For JUnit4 tests we could simply use: > * > https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html > > Note that this would also allow us to correct flaky tests that don’t set up > the initial conditions (fixture) correctly. > > Second note: This is not an endorsement that we should accept flaky tests. > They’re a plague and we need to work on them and fix them. This is just a > momentary workaround to identify how flaky a test is and to try to have a > better solution towards moving to a Continuous Delivery solution. > > WDYT? > > Thanks > -Vincent > -- Thomas Mortagne
Re: [xwiki-devs] [Proposal] Rerun flaky tests several times?
Hi, I would really like to see something like this in our build, specially since I've always wanted us to enable the surefire flag that would re-execute a failed test. Now, it's not very clear to me, but both the JUnit Pioneer thing and the rerunner-jupiter one seem to be proposing an annotation that needs to be set apriori on a test, otherwise the plugin/feature does not do its job? I understand that, due to JUnit5, things might still be work in progress, but I'd like to clarify on the intent that, AFAIU, any test that fails should be rerun to check if it's a flaky test or an actual failure and not only tests marked before (since it would slightly defeat the purpose of automatically detecting and marking flaky tests). +1 to try to automate at least the flagging of such plagues (that are flaky tests) :) Thanks, Eduard On Mon, Jun 10, 2019 at 12:43 PM Vincent Massol wrote: > Hi devs, > > I was looking for a rerun optin in JUnit5 or Surefire but it seems it > doesn’t exist for Junit5: > * https://github.com/junit-team/junit5/issues/1558 > * https://issues.apache.org/jira/browse/SUREFIRE-1584 > > However, there’s an extension to do that: > * https://github.com/artsok/rerunner-jupiter > > Also WIP for JUnit Pioneer: > * https://github.com/junit-pioneer/junit-pioneer/pull/110 > > So my proposal FTM is to try https://github.com/artsok/rerunner-jupiter > and to start putting a repeat of 3 for JUnit5 tests that we know are flaky. > > For JUnit4 tests we could simply use: > * > https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html > > Note that this would also allow us to correct flaky tests that don’t set > up the initial conditions (fixture) correctly. > > Second note: This is not an endorsement that we should accept flaky tests. > They’re a plague and we need to work on them and fix them. This is just a > momentary workaround to identify how flaky a test is and to try to have a > better solution towards moving to a Continuous Delivery solution. > > WDYT? > > Thanks > -Vincent > >
