Sounds good. I am supportive of this change. Ismael
On Mon, Feb 12, 2024 at 7:43 AM David Jacot <dja...@confluent.io.invalid> wrote: > Hi Bruno, > > Yes, you're right. Sorry for the typo. > > Hi Ismael, > > You're right. Jenkins does not support the flakyFailure element and > hence the information is not at all in the Jenkins report. I am still > experimenting with printing the flaky tests somewhere. I will update this > thread if I get something working. In the meantime, I wanted to gauge > whether there is support for it. > > Cheers, > David > > On Mon, Feb 12, 2024 at 3:59 PM Ismael Juma <m...@ismaeljuma.com> wrote: > > > Hi David, > > > > Your message didn't make this clear, but you are saying that Jenkins does > > _not_ support the flakyFailure element and hence this information will be > > completely missing from the Jenkins report. Have we considered including > > the flakyFailure information ourselves? I have seen that being done and > it > > seems strictly better than totally ignoring it. > > > > Ismael > > > > On Mon, Feb 12, 2024 at 12:11 AM David Jacot <dja...@confluent.io.invalid > > > > wrote: > > > > > Hi folks, > > > > > > I have been playing with `reports.junitXml.mergeReruns` setting in > gradle > > > [1]. From the gradle doc: > > > > > > > When mergeReruns is enabled, if a test fails but is then retried and > > > succeeds, its failures will be recorded as <flakyFailure> instead of > > > <failure>, within one <testcase>. This is effectively the reporting > > > produced by the surefire plugin of Apache Maven™ when enabling reruns. > If > > > your CI server understands this format, it will indicate that the test > > was > > > flaky. If it does not, it will indicate that the test succeeded as it > > will > > > ignore the <flakyFailure> information. If the test does not succeed > (i.e. > > > it fails for every retry), it will be indicated as having failed > whether > > > your tool understands this format or not. > > > > > > With this, we get really close to having green builds [2] all the time. > > > There are only a few tests which are too flaky. We should address or > > > disable those. > > > > > > I think that this would help us a lot because it would reduce the noise > > > that we get in pull requests. At the moment, there are just too many > > failed > > > tests reported so it is really hard to know whether a pull request is > > > actually fine or not. > > > > > > [1] applies it to both unit and integration tests. Following the > > discussion > > > in the `github build queue` thread, it may be better to only apply it > to > > > the integration tests. Being stricter with unit tests would make sense. > > > > > > This does not mean that we should continue our effort to reduce the > > number > > > of flaky tests. For this, I propose to keep using Gradle Entreprise. It > > > provides a nice report for them that we can leverage. > > > > > > Thoughts? > > > > > > Best, > > > David > > > > > > [1] https://github.com/apache/kafka/pull/14862 > > > [2] > > > > > > > > > https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14862/19/tests > > > > > >
