On Wed, Feb 21, 2018 at 3:26 PM, Erick Erickson <[email protected]> wrote: > Yonik: > > What I'm frustrated by now is that variations on these themes haven't > cured the problem, and it's spun out of control and is getting worse.
I understand, but what problem(s) are you trying to solve? Just because we are frustrated doesn't mean that *any* change is positive. Some changes can have a definite negative affect on software quality. You didn't respond to the main thrust of my message, so let me try to explain it again more succinctly: Flakey Test Problems: a) Flakey tests create so much noise that people no longer pay attention to the automated reporting via email. b) When running unit tests manually before a commit (i.e. "ant test") a flakey test can fail. Solutions: We cloud fix (a) by marking as flakey and having a new target "non-flakey" that is run by the jenkins jobs that are currently run continuously. For (b) "ant test" should still include the flakey tests since it's better to have to re-run a seemingly unrelated test to determine if one broke something rather than increase committed bugs due to loss of test coverage. It's a pain, but perhaps it should be. It's a real problem that needs fixing and @Ignoring it won't work as a better mechanism to get it fixed. Sweeping it under the rug would seem to ensure that it gets less attention. And we can *always* decide to prevent new flakey tests, regardless of what we do about the existing flakey tests. Mark's tool is a good way to see what the current list of flakey tests is. -Yonik --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
