On Thu, Apr 05, 2018 at 08:05:39AM +0200, Gábor Boskovits wrote:
> Actually running tests test the behaviour of a software. Unfortunately
> reproducible build does not guarantee reproducible behaviour.
> Furthermore there are still cases, where the environment is
> not the same around these running software, like hardware or
> kernel configuration settings leaking into the environment.
> These can be spotted by running tests. Nondeterministic
> failures can also be spotted more easily. There are a lot of
> packages where pulling tests can be done, I guess, but probably not
> for all of them. WDYT?
If that were a real problem we should not be providing substitutes -
same problem. With substitutes we also provide software with tests
that have been run once (at least).
We should not forbid people to run tests. But I don't think it should
be the default once tests have been run in a configuation.
Think of it as functional programming. In my opinion rerunning tests
can be cached.
My point is that we should not overestimate/overdo the idea of
leakage. Save the planet. We have responsibility.