On 03/31/2012 10:23 PM, Bob Friesenhahn wrote: > On Sat, 31 Mar 2012, Stefano Lattarini wrote: >>> >> Well, automake being automake, it only depends on portable awk[1] ;-) >> OK, portable "nawk" actually (as found by the autoconf builtin macro >> AC_PROG_AWK), but I don't know of any non-museum machine lacking that. > > Good. > >> The new feature is already documented in the automake manual on the >> 'master' branch; and this documentation should be release-quality >> already. But maybe I should make this new feature more visible before >> the 1.12 release? If yes, how do you suggest to do so? > > Certainly mentioning and discussing it again on the 'automake' list > (happening now) is useful. > Let's continue the discussion then :-)
> The previous discussion I found was from almost a year ago before > development of the feature was started. > Communication failure on my part then :-( The feature's development was basically complete by the end of August 2011 (as mandated by the GSoC deadlines). And Automake has been happily using TAP in its own testsuite since then! >> As for the implementation quality, there's still an annoying glitch in >> that you need to *manually* copy the automake-provided 'tap-driver.sh' >> script yourself into your package[2] -- "automake --add-missing" won't >> work -- and define some variables by hand; all of this is documented >> in the manual, and should hopefully be fixed for automake 1.12.1. > > This does not seem like a big problem. > > Russ Allbery's C TAP Harness looks reasonable as well. It also > requires adding a single file (.c) to the source tree and a rule > to build it as a test program. > Consider that the whole point of the TAP support in Automake is that the testsuite harness present in Automake-generated makefiles (the one that gets launched by "make check") will act as TAP consumer itself. Thus the "harness" part of Russ' package is likely of little or no use in conjunction with the newer Automake test harness -- but of course you can reuse the part of his project that provide a TAP *producer* for C and/or Shell programs. > The philosophy of TAP is rather different than Automake's existing > test suite. > Yes, but not too radically IMHO; in fact, the TAP consumer implemented by Automake reuses most of the code and interfaces of the 'parallel-tests' test harness (that was an early design choice, and so far I have been very happy with it). > These differences may prove challenging when switching from Automake > tests to TAP: > Note that you don't need any "flag day switch" for a conversion to TAP: "simple tests" (whose outcome is purely based on their exit status) and TAP tests can happily co-exists (as they currently do in the Automake's own testsuite). This too is an early, deliberate design choice. > * Every TAP test needs to somehow know its unique test number (so it > can print a result "ok 35"). > Actually, not really: $ cat foo.sh #!/bin/sh echo 1..3 echo 'ok' echo 'ok # SKIP' echo 'not ok # TODO' $ prove ./foo.sh ./foo.sh .. ok All tests successful. Files=1, Tests=3, 1 wallclock secs ( 0.04 usr + 0.00 sys = 0.04 CPU) Result: PASS But as it's reasonable, if you provide a test number, that must be correct (this should be undeniably recognized as a feature IMO); see: $ cat bar.sh #!/bin/sh echo 1..3 echo 'ok 4' echo 'ok # SKIP' echo 'not ok # TODO' $ prove ./bar.sh ./bar.sh .. All 3 subtests passed (less 1 skipped subtest: 2 okay) Test Summary Report ------------------- ./bar.sh (Wstat: 0 Tests: 3 Failed: 0) Parse errors: Tests out of sequence. Found (4) but expected (1) Files=1, Tests=3, 0 wallclock secs ( 0.04 usr + 0.00 sys = 0.04 CPU) Result: FAIL > If the tests are all in the same > module then they can simply increment an integer but if they are > not then there is a problem (or they can always plan for one > test). > Apart from the fact that I'd advise against writing your TAP stream by hand (you should use a small but higher-level module or library or set of functions to do so), the situation you describe shouldn't be a problem in any case: Automake-produced Makefiles fire off a *new* TAP-parsing process for each TAP-based test script in $(TESTS) [1], so that you would only need to worry about the ordering/numbering of the test cases in the script you are modifying. [1] This is also extremely important in order to preserve the ability to run tests in parallel -- one of the major selling points of the Automake testsuite harness. > * TAP tests need to know if they are expected to pass or fail. > > A major feature of how GraphicsMagick is using the Automake tests > framework is that it determines which tests are XFAIL based on what > features were configured into the software. The list of tests which > are expected to fail is produced based on Automake conditionals. > It is useful for the user to see tests XFAIL as a reminder that the > configuration may be missing something they wanted. How is this best > handled for TAP tests? > That is highly dependent on the exact details of how your testsuite is organized; if you have time to present more precise/targeted questions in this respect, I'd be happy to help. (And I see now that Russ has already given some good advice, the I feel to endorse). > Bob Regards, Stefano