On Sat, Mar 18, 2017 at 05:42:20PM -0500, Scott Cheloha wrote: > My thinking is that statically ordered regression runs can mask > bugs in the software under test and the tests themselves.
I would expect to find more issues in the tests than in the software. > In general, a test can put your system into a state that allows a > subsequent test to pass when it would have otherwise failed. Unfortunately I fear a lot of our tests do that. > Randomly ordered regression runs could help to weed these issues > out. Even if no such bugs exist, your randomly ordered regression > runs are stronger statements about your software's correctness and > the independence of each test case. Independent tests cases are useful when fixing tests or doing test driven development. For finding regressions or getting more tests cases for the software they are not so relevant. Did you have a look how many of our regress tests have independent targets? Do you plan to improve this situation? Having a feature that would fail for 80% of our tests does not make sense. I think you try to implement the second step before the first. Look at the tests and figure out wether you want to fix all of them. I expect a lot of work. After this has happened, we could add your target. bluhm
