I will say that I have also seen tests that seem to be more flaky intermittently. It may be ok for some time and suddenly the CI is overloaded and we see issues. I have also seen the CI struggling with running out of space recently, so I wonder if we can also try to improve things on that front.
FWIW, I noticed, filed, or commented on several flaky test JIRAs last week. I'm happy to try to get to green builds, but everyone needs to be on board. https://issues.apache.org/jira/browse/KAFKA-15529 https://issues.apache.org/jira/browse/KAFKA-14806 https://issues.apache.org/jira/browse/KAFKA-14249 https://issues.apache.org/jira/browse/KAFKA-15798 https://issues.apache.org/jira/browse/KAFKA-15797 https://issues.apache.org/jira/browse/KAFKA-15690 https://issues.apache.org/jira/browse/KAFKA-15699 https://issues.apache.org/jira/browse/KAFKA-15772 https://issues.apache.org/jira/browse/KAFKA-15759 https://issues.apache.org/jira/browse/KAFKA-15760 https://issues.apache.org/jira/browse/KAFKA-15700 I've also seen that kraft transactions tests often flakily see that the producer id is not allocated and times out. I can file a JIRA for that too. Hopefully this is a place we can start from. Justine On Sat, Nov 11, 2023 at 11:35 AM Ismael Juma <m...@ismaeljuma.com> wrote: > On Sat, Nov 11, 2023 at 10:32 AM John Roesler <vvcep...@apache.org> wrote: > > > In other words, I’m biased to think that new flakiness indicates > > non-deterministic bugs more often than it indicates a bad test. > > > > My experience is exactly the opposite. As someone who has tracked many of > the flaky fixes, the vast majority of the time they are an issue with the > test. > > Ismael >
