On Mar 30, 2020, at 17:34, Han-Wen Nienhuys <hanw...@gmail.com> wrote: > > On Sun, Mar 29, 2020 at 11:45 PM Dan Eble <d...@faithful.be> wrote: >> >> On Mar 29, 2020, at 17:39, Han-Wen Nienhuys <hanw...@gmail.com> wrote: >>>> test-output-distance was removed on the grounds that the self-test >>>> serves the same purpose, but I don't see how it does. >>> >>> Could you elaborate? What failure scenario are you worried about? >> >> My question is, how does the self-test "verify that the regression tests >> have, in fact, run"? I don't see how it could do more than verify that >> itself has run. > > How does test-output-distance verify that the regression tests have run? > > The verification doesn't come from the test file. It comes from the > fact that someone is looking at the test output.
(Rhetorically) How much verification comes from looking at the output since you removed the test? > The self-test gives us assurance that the test result index.html file > is making sense. Yes, and that does not overlap with the purpose of test-output-distance. > You can verify that the tests were run by checking > that the index.html file is there. I think the only thing that the presence of index.html implies is that output-distance.py was run. > For further context, I am interested in automating our test setup > further: I want to have CI that can post feedback ("test failed!") on > code reviews automatically. test-output-distance is a test that always > fails, which makes that more complicated. This is what the description of the commit that removed test-output-distance ought to have been. I've been trying to point out that something of value was lost. This last statement of yours finally gives something to weigh that value against. -- Dan