Moritz Lenz wrote:
jerry gay wrote:
A combined harness is much better in terms of reporting.
Yes.
the tests we expect to pass reliably should be the default tests we
run. we expect all spectest_regression tests to pass reliably. the
default test target should always be named 'test'. it seems natural
that we add spectest_regression to the 'test' makefile target.
additionally, this would have possibly prevented the 74 failures
post-mdd-merge, since allison didn't know about the additional test
target in the makefile.
well, if reading the README is too much even for our architect then we
shouldn't assume that anybody else does ;-)
Another thing that would be helpful for languages in trunk is something
like a TESTME file. It should briefly say exactly what steps a core
developer should take to test that their changes haven't broken the
language, and if failures are expected or all tests should pass. Also,
anything strange like having multiple test harnesses running in sequence
instead of aggregating the results in one report. (That one caught me on
Rakudo's 'make test' too. I thought all the test were passing, and then
found that the final "All tests pass" report was hiding earlier failures
in a different summary.) The README is quite verbose and intended for
people who want to use the language. Even after reading it, it's not
straightforward to decide what to test and whether failures are relevant.
What would be really ideal is if core developers could just run 'make
languagetest' in the repository root and get a single report of all the
language failures, and know for sure that any failures are their
responsibility. But, we're a long way away from that.
Allison
