Re: What to do about issues that track flaky tests?

2022-09-15 Thread Brian Hulette via dev
I agree with Austin on this one, it makes sense to be realistic, but I'm concerned about just blanket reducing the priority on all flakes. Two classes of issues that could certainly be dropped to P2: - Issues tracking flakes that have not been sickbayed yet (e.g. https://github.com/apache/beam/issu

Re: What to do about issues that track flaky tests?

2022-09-14 Thread Manu Zhang
Agreed. I also mentioned in a previous email that some issues have been open for a long time (before being migrated to GitHub) and it's possible that those tests can pass constantly now. We may double check and close them since reopening is just one click. Manu On Thu, Sep 15, 2022 at 6:58 AM Aus

Re: What to do about issues that track flaky tests?

2022-09-14 Thread Austin Bennett
+1 to being realistic -- proper labels are worthwhile. Though, some flaky tests probably should be P1, and just because isn't addressed in a timely manner doesn't mean it isn't a P1 - though, it does mean it wasn't addressed. On Wed, Sep 14, 2022 at 1:19 PM Kenneth Knowles wrote: > I would li

What to do about issues that track flaky tests?

2022-09-14 Thread Kenneth Knowles
I would like to make this alert email actionable. I went through most of these issues. About half are P1 "flake" issues. I don't think magically expecting them to be deflaked is helpful. So I have a couple ideas: 1. Exclude "flake" P1s from this email. This is what we used to do. But then... are