On Monday, 23 January 2017 at 01:52:29 UTC, Chris Wright wrote:
On Sun, 22 Jan 2017 20:18:11 +0000, Mark wrote:
Have you considered adding randomized tests to Phobos?
Randomized testing is an interesting strategy to use alongside
deterministic testing. It produces more coverage over time.
However, any given test run only has a fraction of the coverage
that you see over a large number of runs.
In other words, if the randomized tests catch something, you
don't know who dun it. This is generally considered a bad thing.
Phobos does have some tests that use a PRNG. They all use a
fixed seed. This is a shortcut to coming up with arbitrary test
data; it's not a way to increase overall coverage.
I think the right way to do it is to have a nightly randomized
test build, but since I'm not willing to do the work, I don't
have much say.
This. So much this.
Unit tests that loop over many randomly generated input test
vectors are just a waste of everybody's CPU time.
Don't get me wrong: fuzzing is also necessary. But it relies on
an arbitrary time limit, which is hardly compatible with keeping
a test suite fast. Which means it should be done in a another
validation process than unit tests.
