Hi David, Your message didn't make this clear, but you are saying that Jenkins does _not_ support the flakyFailure element and hence this information will be completely missing from the Jenkins report. Have we considered including the flakyFailure information ourselves? I have seen that being done and it seems strictly better than totally ignoring it.
Ismael On Mon, Feb 12, 2024 at 12:11 AM David Jacot <dja...@confluent.io.invalid> wrote: > Hi folks, > > I have been playing with `reports.junitXml.mergeReruns` setting in gradle > [1]. From the gradle doc: > > > When mergeReruns is enabled, if a test fails but is then retried and > succeeds, its failures will be recorded as <flakyFailure> instead of > <failure>, within one <testcase>. This is effectively the reporting > produced by the surefire plugin of Apache Maven™ when enabling reruns. If > your CI server understands this format, it will indicate that the test was > flaky. If it does not, it will indicate that the test succeeded as it will > ignore the <flakyFailure> information. If the test does not succeed (i.e. > it fails for every retry), it will be indicated as having failed whether > your tool understands this format or not. > > With this, we get really close to having green builds [2] all the time. > There are only a few tests which are too flaky. We should address or > disable those. > > I think that this would help us a lot because it would reduce the noise > that we get in pull requests. At the moment, there are just too many failed > tests reported so it is really hard to know whether a pull request is > actually fine or not. > > [1] applies it to both unit and integration tests. Following the discussion > in the `github build queue` thread, it may be better to only apply it to > the integration tests. Being stricter with unit tests would make sense. > > This does not mean that we should continue our effort to reduce the number > of flaky tests. For this, I propose to keep using Gradle Entreprise. It > provides a nice report for them that we can leverage. > > Thoughts? > > Best, > David > > [1] https://github.com/apache/kafka/pull/14862 > [2] > > https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14862/19/tests >
