> To kickstart this effort, we can publish a list of such tickets in the > community and assign one or more committers the role of a «shepherd" for each > ticket.
Please, do it. We can use specific label to effectively filter those tickets. > 13 нояб. 2023 г., в 15:16, Divij Vaidya <divijvaidy...@gmail.com> написал(а): > > Thanks for bringing this up David. > > My primary concern revolves around the possibility that the currently > disabled tests may remain inactive indefinitely. We currently have > unresolved JIRA tickets for flaky tests that have been pending for an > extended period. I am inclined to support the idea of disabling these tests > temporarily and merging changes only when the build is successful, provided > there is a clear plan for re-enabling them in the future. > > To address this issue, I propose the following measures: > > 1\ Foster a supportive environment for new contributors within the > community, encouraging them to take on tickets associated with flaky tests. > This initiative would require individuals familiar with the relevant code > to offer guidance to those undertaking these tasks. Committers should > prioritize reviewing and addressing these tickets within their available > bandwidth. To kickstart this effort, we can publish a list of such tickets > in the community and assign one or more committers the role of a "shepherd" > for each ticket. > > 2\ Implement a policy to block minor version releases until the Release > Manager (RM) is satisfied that the disabled tests do not result in gaps in > our testing coverage. The RM may rely on Subject Matter Experts (SMEs) in > the specific code areas to provide assurance before giving the green light > for a release. > > 3\ Set a community-wide goal for 2024 to achieve a stable Continuous > Integration (CI) system. This goal should encompass projects such as > refining our test suite to eliminate flakiness and addressing > infrastructure issues if necessary. By publishing this goal, we create a > shared vision for the community in 2024, fostering alignment on our > objectives. This alignment will aid in prioritizing tasks for community > members and guide reviewers in allocating their bandwidth effectively. > > -- > Divij Vaidya > > > > On Sun, Nov 12, 2023 at 2:58 AM Justine Olshan <jols...@confluent.io.invalid> > wrote: > >> I will say that I have also seen tests that seem to be more flaky >> intermittently. It may be ok for some time and suddenly the CI is >> overloaded and we see issues. >> I have also seen the CI struggling with running out of space recently, so I >> wonder if we can also try to improve things on that front. >> >> FWIW, I noticed, filed, or commented on several flaky test JIRAs last week. >> I'm happy to try to get to green builds, but everyone needs to be on board. >> >> https://issues.apache.org/jira/browse/KAFKA-15529 >> https://issues.apache.org/jira/browse/KAFKA-14806 >> https://issues.apache.org/jira/browse/KAFKA-14249 >> https://issues.apache.org/jira/browse/KAFKA-15798 >> https://issues.apache.org/jira/browse/KAFKA-15797 >> https://issues.apache.org/jira/browse/KAFKA-15690 >> https://issues.apache.org/jira/browse/KAFKA-15699 >> https://issues.apache.org/jira/browse/KAFKA-15772 >> https://issues.apache.org/jira/browse/KAFKA-15759 >> https://issues.apache.org/jira/browse/KAFKA-15760 >> https://issues.apache.org/jira/browse/KAFKA-15700 >> >> I've also seen that kraft transactions tests often flakily see that the >> producer id is not allocated and times out. >> I can file a JIRA for that too. >> >> Hopefully this is a place we can start from. >> >> Justine >> >> On Sat, Nov 11, 2023 at 11:35 AM Ismael Juma <m...@ismaeljuma.com> wrote: >> >>> On Sat, Nov 11, 2023 at 10:32 AM John Roesler <vvcep...@apache.org> >> wrote: >>> >>>> In other words, I’m biased to think that new flakiness indicates >>>> non-deterministic bugs more often than it indicates a bad test. >>>> >>> >>> My experience is exactly the opposite. As someone who has tracked many of >>> the flaky fixes, the vast majority of the time they are an issue with the >>> test. >>> >>> Ismael >>> >>
