> Randomness makes it difficult to correlate a failure to the commit that made > the test to fail (as was pointed out earlier in the discussion). If each > execution path is different, it may very well be that a failure you > experience is introduced several commits ago, so it may not be your fault.
This is true only to a certain degree. If you don't randomize all you do is essentially run a fixed scenario. This protects you against a regression in this particular state, but it doesn't help in discovering new corner cases or environment quirks, which would be prohibitive to run as a full Cartesian product of all possibilities. So there is a tradeoff here and most folks in this project have agreed to it. If you look at how many problems randomization have helped discover I think it's a good tradeoff. Finally: your scenario can be actually reproduced with ease. Run the tests with a fixed seed before you apply a patch and after you apply it... if there is no regression you can assume your patch is fine (but it doesn't mean it won't fail later on on a different seed, which nobody will blame you for). Dawid --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
