Julian, would you like to answer the question in the second quoted paragraph?
Daniel Shahaf wrote on Wed, Jan 08, 2020 at 14:24:39 +0000: > Julian Foad wrote on Wed, Jan 08, 2020 at 10:14:42 +0000: > > What? > > Using a PRNG makes the test code harder to read and to maintain, makes it > more difficult to diagnose FAILs, and increases the risk of a bug in the test > code. What benefits are we getting that offset these risks and costs? > > The way in which the PRNG algorithm is used is outside the PRNG's API > promises. > PRNG's make *no* promises on their outputs when used with a fixed seed. In > light of this, what reason do we have to believe that the test achieves good > coverage? > > It's not necessary to use a PRNG to test a large/diverse set of inputs. You > can achieve with plain old combinatorics (e.g., define 300 rangelist variables > and test various combinations of them). I'd actually expect that to have > better coverage than a PRNG with a fixed seed. > > By default, I would have expected the tests to be written without a PRNG, and > I > don't understand why the way the tests are is better. > > Cheers, > > Daniel
