Is it possible to make the presubmit auto retry all failed tests a few times? (and maybe generate a report of a list of flakey tests). Then you don't need to disable/isolate the flakey tests.
If this is not possible, or hard to setup, then manually moving them to a different suite sounds like a good idea. On Tue, Mar 16, 2021 at 2:11 PM Pablo Estrada <[email protected]> wrote: > Hi all, > In Beam, we sometimes hit the issue of having one or two test cases that > are particularly flaky, and we deactivate them. > This is completely reasonable to me, because we need to keep good testing > signal on our primary suites. > The danger of deactivating these tests is that, although we have good > practices to file JIRA issues to re-enable them, it is still easy for these > issues and tests to be forgotten. > Of course, ideally, the solution is "do not forget old deactivated tests" > - and we should adopt practices to ensure that. > > I think, to strengthen our practices, we can reinforce them with a > pragmatic choice: Instead of fully deactivating tests, we can make them run > in a separate suite of Flaky tests. Why would this help? > > - It would allow us to make sure that flaky tests continue to *be able to > run*. > - It would remind us that we have flaky tests that need fixing. > - It would allow us to experiment fixes to these tests on the Flaky suite, > and once they're reliable, move them to the main suite. > > Does this make sense to others? > Best > -P. >
