Daniel Shahaf wrote:
Julian Foad wrote on Tue, Jan 07, 2020 at 09:47:48 +0000:
For now, I propose to make each test use a repeatable sequence, independent
of the other tests. I think that will be enough; the options I mentioned
can be added if and when there is a demand for them.
I don't understand why you're fixing the seed.
For repeatability, all you need is to output the seed to stdout and to be able
to re-run the tests with a particular value of the seed:
The suggestion was that we should prefer the regression test suite to be
deterministic, running the same fixed set of tests each time. Repeating
by default, not just repeatable on request.
The other way (non-repeating by default, and repeatable on request) is a
possible alternative that we could choose. Indeed, a few (six or so)
other tests in the suite are currently written that way. However we
don't really have the test infrastructure for handling that nicely. We
could add some, starting with the pseudocode you posted, but we haven't
yet and it would probably want a bit more than that to make an overall
good experience.
Conversely, the effect of fixing the seed is to make every single test run test
exactly the same sequence of API calls on the same sequence of inputs. That
means the test is not merely repeatable, but also deterministic
Yes, like nearly all of our test suite.
— which begs
the question, why do we need a PRNG at all? What do we gain
We gain the ability to test a large, diverse set of combinations of
inputs. Call it an "test case generator" instead of a "PRNG", if that
helps.
in exchange for the
costs and risks associated with using a PRNG in the test suite?
What?
- Julian
