> > Can we merge the PR if only flaky or new tests fail?
We certainly cannot merge if new tests fail - the goal is to ensure new tests are solid. For the flaky ones, I'm not sure how we intend to use these. I would prefer if we only merge when the PR status is green. Otherwise, we often end up merging things we shouldn't (by accident). Ismael On Mon, Feb 24, 2025 at 2:50 PM Chia-Ping Tsai <chia7...@gmail.com> wrote: > hi David > > Thanks for all your improvement. I do love the new test suites! > > one small question: > Can we merge the PR if only flaky or new tests fail? Sometimes, I list > tickets for flaky (or unrelated) tests before merging. However, since we > now have a separate test suite for stable tests (non-flaky, non-new), I > assume the new condition is that "stable tests must pass"? > > Best, > Chia-Ping > > > > > Ismael Juma <m...@ismaeljuma.com> 於 2025年2月25日 週二 上午6:24寫道： > > > Thanks David - this is another important improvement to our CI pipeline > and > > is super helpful for the project and community. > > > > Ismael > > > > On Mon, Feb 24, 2025 at 2:15 PM David Arthur <mum...@gmail.com> wrote: > > > > > Hey everyone, just wanted to inform you all that we just merged > > KAFKA-18748 > > > > > > https://github.com/apache/kafka/pull/18770 > > > > > > This splits our CI workflow into more parallel jobs which run subsets > of > > > the tests with different settings. The JUnit tests are now split into > > > "new", "flaky", and the remainder. > > > > > > "New" tests are what we previously called auto-quarantined tests. > > > > > > On PR builds, "new" tests are anything that do not exist on trunk. They > > are > > > run with zero tolerance for flakiness. > > > > > > On trunk builds, "new" tests are anything added in the last 7 days. > They > > > are run with some tolerance for flakiness. > > > > > > The point of this is to discourage flaky tests from being added to > trunk. > > > > > > Please update your PRs with trunk and let me know if you see any > > weirdness. > > > Feel free to tag me in the PR, reply to this thread, or email me > directly > > > with questions. > > > > > > Thanks! > > > David A > > > > > >
