> I see your point about "bringing up bugs nobody thought to cover manually", > but it also has cons - e.g. violating the principal that tests > should be > (easily) repeatable (you will/can end up with tests the sometimes fail and > sometimes succeed, and you have to dig out the > random values of the tests > that fail in order to be able to repeat/reconstruct the fail)
Randomized tests should be identical in their execution given the same seed, it's the same principle as with regular tests but expands on different code paths every time you execute with a different seed. They are not a replacement for boundary condition tests, they're a complementary thing that should allow picking things you haven't thought of. Sure, in case of a failure you need to find the seed that caused the problem but that doesn't seem like a lot of effort given the potential profit. If you want identical runs -- fix the initial seed. If you have a non-deterministic test for a given fixed seed, it'd be equally non-deterministic if no randomization was used, it's just a flawed test (or inherently non-deterministic by nature so assertions should be relaxed). Dawid --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
