Hi all, In Beam, we sometimes hit the issue of having one or two test cases that are particularly flaky, and we deactivate them. This is completely reasonable to me, because we need to keep good testing signal on our primary suites. The danger of deactivating these tests is that, although we have good practices to file JIRA issues to re-enable them, it is still easy for these issues and tests to be forgotten. Of course, ideally, the solution is "do not forget old deactivated tests" - and we should adopt practices to ensure that.
I think, to strengthen our practices, we can reinforce them with a pragmatic choice: Instead of fully deactivating tests, we can make them run in a separate suite of Flaky tests. Why would this help? - It would allow us to make sure that flaky tests continue to *be able to run*. - It would remind us that we have flaky tests that need fixing. - It would allow us to experiment fixes to these tests on the Flaky suite, and once they're reliable, move them to the main suite. Does this make sense to others? Best -P.
