Hi, I would really like to see something like this in our build, specially since I've always wanted us to enable the surefire flag that would re-execute a failed test.
Now, it's not very clear to me, but both the JUnit Pioneer thing and the rerunner-jupiter one seem to be proposing an annotation that needs to be set apriori on a test, otherwise the plugin/feature does not do its job? I understand that, due to JUnit5, things might still be work in progress, but I'd like to clarify on the intent that, AFAIU, any test that fails should be rerun to check if it's a flaky test or an actual failure and not only tests marked before (since it would slightly defeat the purpose of automatically detecting and marking flaky tests). +1 to try to automate at least the flagging of such plagues (that are flaky tests) :) Thanks, Eduard On Mon, Jun 10, 2019 at 12:43 PM Vincent Massol <vinc...@massol.net> wrote: > Hi devs, > > I was looking for a rerun optin in JUnit5 or Surefire but it seems it > doesn’t exist for Junit5: > * https://github.com/junit-team/junit5/issues/1558 > * https://issues.apache.org/jira/browse/SUREFIRE-1584 > > However, there’s an extension to do that: > * https://github.com/artsok/rerunner-jupiter > > Also WIP for JUnit Pioneer: > * https://github.com/junit-pioneer/junit-pioneer/pull/110 > > So my proposal FTM is to try https://github.com/artsok/rerunner-jupiter > and to start putting a repeat of 3 for JUnit5 tests that we know are flaky. > > For JUnit4 tests we could simply use: > * > https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html > > Note that this would also allow us to correct flaky tests that don’t set > up the initial conditions (fixture) correctly. > > Second note: This is not an endorsement that we should accept flaky tests. > They’re a plague and we need to work on them and fix them. This is just a > momentary workaround to identify how flaky a test is and to try to have a > better solution towards moving to a Continuous Delivery solution. > > WDYT? > > Thanks > -Vincent > >
