I expect the suite to be permared, right? Because of some thing or another flaking at all times.
Kenn On Tue, Mar 16, 2021 at 2:13 PM Alex Amato <[email protected]> wrote: > Is it possible to make the presubmit auto retry all failed tests a few > times? (and maybe generate a report of a list of flakey tests). > Then you don't need to disable/isolate the flakey tests. > > If this is not possible, or hard to setup, then manually moving them to a > different suite sounds like a good idea. > > On Tue, Mar 16, 2021 at 2:11 PM Pablo Estrada <[email protected]> wrote: > >> Hi all, >> In Beam, we sometimes hit the issue of having one or two test cases that >> are particularly flaky, and we deactivate them. >> This is completely reasonable to me, because we need to keep good testing >> signal on our primary suites. >> The danger of deactivating these tests is that, although we have good >> practices to file JIRA issues to re-enable them, it is still easy for these >> issues and tests to be forgotten. >> Of course, ideally, the solution is "do not forget old deactivated tests" >> - and we should adopt practices to ensure that. >> >> I think, to strengthen our practices, we can reinforce them with a >> pragmatic choice: Instead of fully deactivating tests, we can make them run >> in a separate suite of Flaky tests. Why would this help? >> >> - It would allow us to make sure that flaky tests continue to *be able >> to run*. >> - It would remind us that we have flaky tests that need fixing. >> - It would allow us to experiment fixes to these tests on the Flaky >> suite, and once they're reliable, move them to the main suite. >> >> Does this make sense to others? >> Best >> -P. >> >
