Waldek Hebisch <[email protected]> writes: > Martin Rubey wrote: >> Waldek Hebisch <[email protected]> writes: >> >> > Attached is a patch that adds support for expected failures to >> > testing framework. IMHO currently testing framework has >> > several problem but this was probably the worst one. >> >> I'll try not to take this personal. I always thought that my package >> was quite an improvement about *no* package >> > > Sure. Maybe we should say more about achivements (and current > framework is one of them). But certainly we need to discuss > problems, without that it is hard to improve things...
No problem, and thanks for the kudos! >> I assume that what you add is a grand total over all testcases - or is >> it a grand total over all testsuites? Please make this clearer in the >> output. >> >> >> ============================================================================= >> testsuite | testcases: failed (total) | tests: failed (total) >> bugs2009 0 (19) 0 (69) >> unexpected failures: 0 >> expected failures: 5 >> unexpected passes: 0 >> total tests: 69 >> > > I want total of all tests: if one test of a testcase is expected > failure and say three other pass classifying the whole testcase > as expected failure would be (IMHO) misleading. So I want > summary which ignores testsuite/testcase boundaries. OK. I agree that this was misleading. I mainly tried to copy Christian Aistleitner's unittest package for Aldor. So, the rules where: * if a test fails, the whole testcase fails (i.e., testcases are a very small unit) I guess that "expected" failures don't make a testcase fail now, right? * above that, there are only testsuites. > Note that my plan is to have 'make check' rule in Makefiles which > gives a single compact result summarizing all tests. Since we run > each test file separately the first step is to get summaries for > files. My assumption was actually that the files are all run in the same image, and statistics() would then output a grand total for all testsuites, too. But that was probably too optimistic. > Later we can add some (probably awk) script to collect data > from files and produce final numbers. My main concern is that numbers > are easy to parse for programs, I hope that human will have little > need to look at them. We can add a line and extra header to make > clear that the new summary is separate from the old one: > ... > ========================================================== > File summary. > unexpected failures: 0 > expected failures: 5 > unexpected passes: 0 > total tests: 69 OK. Martin
-- You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/fricas-devel?hl=en.
