Hi All, There are a number of tests in our system that are either flaky or permanently red. I am suggesting to add, if not all, then most of the tests (style, unit, integration, etc) to the release validation step. In this way, we will add a regular cadence to ensuring greenness and no flaky tests in Beam.
There are a number of ways of implementing this, but what I think might work the best is to set up a process that either manually or automatically creates a JIRA for the failing test and assigns it to a component tagged with the release number. The release can then continue when all JIRAs are closed by either fixing the failure or manually testing to ensure no adverse side effects (this is in case there are environmental issues in the testing infrastructure or otherwise). Thanks for reading, what do you think? - Is there another, easier way to ensure that no test failures go unfixed? - Can the process be automated? - What am I missing? Regards, Sam
