At this point it is clear that we will probably never be able to remove some level of flakiness from our tests. For me the questions are: 1) Where do we draw the line for a release ? and 2) How do we maintain that line over time?
In my opinion, not all flakies are equals. Some fails every 10 runs, some fails 1 in a 1000 runs. I would personally draw the line based on that metric. With the circleci tasks that Andres has added we can easily get that information for a given test. We can start by putting the bar at a lower level and raise the level over time when most of the flakies that we hit are above that level. At the same time we should make sure that we do not introduce new flakies. One simple approach that has been mentioned several time is to run the new tests added by a given patch in a loop using one of the CircleCI tasks. That would allow us to minimize the risk of introducing flaky tests. We should also probably revert newly committed patch if we detect that they introduced flakies. What do you think? Le dim. 7 août 2022 à 12:24, Mick Semb Wever <m...@apache.org> a écrit : > > > With that said, I guess we can just revise on a regular basis what exactly >> are the last flakes and not numbers which also change quickly up and down >> with the first change in the Infra. >> > > > +1, I am in favour of taking a pragmatic approach. > > If flakies are identified and triaged enough that, with correlation from > both CI systems, we are confident that no legit bugs are behind them, I'm > in favour of going beta. > > I still remain in favour of somehow incentivising reducing other flakies > as well. Flakies that expose poor/limited CI infra, and/or tests that are > not as resilient as they could be, are still noise that indirectly reduce > our QA (and increase efforts to find and tackle those legit runtime > problems). Interested in hearing input from others here that have been > spending a lot of time on this front. > > Could it work if we say: all flakies must be ticketed, and test/infra > related flakies do not block a beta release so long as there are fewer than > the previous release? The intent here being pragmatic, but keeping us on a > "keep the campground cleaner" trajectory… > >
