On Monday, 23 January 2017 at 01:52:29 UTC, Chris Wright wrote:
On Sun, 22 Jan 2017 20:18:11 +0000, Mark wrote:
Have you considered adding randomized tests to Phobos?

Randomized testing is an interesting strategy to use alongside deterministic testing. It produces more coverage over time. However, any given test run only has a fraction of the coverage that you see over a large number of runs.

In other words, if the randomized tests catch something, you don't know who dun it. This is generally considered a bad thing.

Phobos does have some tests that use a PRNG. They all use a fixed seed. This is a shortcut to coming up with arbitrary test data; it's not a way to increase overall coverage.

I think the right way to do it is to have a nightly randomized test build, but since I'm not willing to do the work, I don't have much say.

This. So much this.
Unit tests that loop over many randomly generated input test vectors are just a waste of everybody's CPU time.

Don't get me wrong: fuzzing is also necessary. But it relies on an arbitrary time limit, which is hardly compatible with keeping a test suite fast. Which means it should be done in a another validation process than unit tests.

Reply via email to