Thanks David! This is awesome, really glad to see this effort to reduce test flakiness.
One question -- are tests automatically sorted into these buckets or do we have to manually move them? And if so, how does that work (eg a test in "main" becomes flaky) On Tue, Feb 25, 2025 at 3:20 PM David Arthur <mum...@gmail.com> wrote: > > Can we merge the PR if only flaky or new tests fail? > > I agree with Ismael that new tests must be solid -- no flakiness should be > expected when adding a test. Obviously, we will miss things, so we have to > tolerate them on trunk (along with environmental flaky factors). > > If there are existing *unrelated* tests that are flaky on a PR, that is > fine. Ideally, each failing test or flaky tests on a PR should be > investigated. > > For PRs: > "new" tests -- no flakiness > "flaky" tests -- expect some flakiness, still look at these failures to > make sure the PR didn't make it worse > "main" tests -- normal amounts of flakiness, still look at these failures > to make sure the PR didn't make it worse (and file a Jira to report flaky, > if applicable) > > --- > > > BTW If your PRs have failed build scans, try merging in trunk. > > [image: image.png] > > -David A > > > On Mon, Feb 24, 2025 at 7:16 PM Ismael Juma <m...@ismaeljuma.com> wrote: > >> > >> > Can we merge the PR if only flaky or new tests fail? >> >> >> We certainly cannot merge if new tests fail - the goal is to ensure new >> tests are solid. >> >> For the flaky ones, I'm not sure how we intend to use these. I would >> prefer >> if we only merge when the PR status is green. Otherwise, we often end up >> merging things we shouldn't (by accident). >> >> Ismael >> >> On Mon, Feb 24, 2025 at 2:50 PM Chia-Ping Tsai <chia7...@gmail.com> >> wrote: >> >> > hi David >> > >> > Thanks for all your improvement. I do love the new test suites! >> > >> > one small question: >> > Can we merge the PR if only flaky or new tests fail? Sometimes, I list >> > tickets for flaky (or unrelated) tests before merging. However, since we >> > now have a separate test suite for stable tests (non-flaky, non-new), I >> > assume the new condition is that "stable tests must pass"? >> > >> > Best, >> > Chia-Ping >> > >> > >> > >> > >> > Ismael Juma <m...@ismaeljuma.com> 於 2025年2月25日 週二 上午6:24寫道： >> > >> > > Thanks David - this is another important improvement to our CI >> pipeline >> > and >> > > is super helpful for the project and community. >> > > >> > > Ismael >> > > >> > > On Mon, Feb 24, 2025 at 2:15 PM David Arthur <mum...@gmail.com> >> wrote: >> > > >> > > > Hey everyone, just wanted to inform you all that we just merged >> > > KAFKA-18748 >> > > > >> > > > https://github.com/apache/kafka/pull/18770 >> > > > >> > > > This splits our CI workflow into more parallel jobs which run >> subsets >> > of >> > > > the tests with different settings. The JUnit tests are now split >> into >> > > > "new", "flaky", and the remainder. >> > > > >> > > > "New" tests are what we previously called auto-quarantined tests. >> > > > >> > > > On PR builds, "new" tests are anything that do not exist on trunk. >> They >> > > are >> > > > run with zero tolerance for flakiness. >> > > > >> > > > On trunk builds, "new" tests are anything added in the last 7 days. >> > They >> > > > are run with some tolerance for flakiness. >> > > > >> > > > The point of this is to discourage flaky tests from being added to >> > trunk. >> > > > >> > > > Please update your PRs with trunk and let me know if you see any >> > > weirdness. >> > > > Feel free to tag me in the PR, reply to this thread, or email me >> > directly >> > > > with questions. >> > > > >> > > > Thanks! >> > > > David A >> > > > >> > > >> > >> > > > -- > David Arthur >
