Here's an example of why TAP is a good way to go for testing.
Here at Yahoo! we run a *lot* of tests to see if our stuff is working
right. Not only do we test the mainline code, but we also test variants to
see which ones the users like better, or help get better targeted ads in
the paid search stuff (hey, it's how search engines make money!).
So a lot of tests, and a lot of variations, and you're talking about real
problems trying to look at the TAP output and figure out if there are any
patterns in the failures - things change fast, so there are always
failures.
What I've done is create a little class that allows you to tag TAP
descriptions. Since descriptions are "just text" to TAP and to TAP
parsers, I can put anything I want in there. Test::Description::Tagged
(I'd appreciate suggestions for a better name and/or namespace) lets you
do this:
use Test::Description::Tagged;
my $desc = new Test::Description::Tagged;
my @servers = qw(alpha beta gamma delta epsilon);
my @queries = qw(foo bar baz quux);
foreach my $server (@servers) {
foreach my $query (@queries) {
$desc->tag(server=>$server, query=>$query);
my $page = run_query($server, $query);
like $page, qr/not broken/, $desc->comment('not broken');
like $page, qr/Page 1 of/, $desc->comment('page count');
...
}
}
Your TAP output looks like this:
1..36
ok 1 - [server:alpha][query:foo]||not broken
ok 2 - [server:alpha][query:foo]||page count
...
A small program based on TAPx::Parser, simple_matrix, can now read the
tagged TAP and generate HTML reports of tagX vs. tagY. It's an obvious
extension to select and reject tags.
We're expecting to deploy this pretty soon, and if it works out well, I'll
send it on to CPAN.
Big win for TAP: because it's an open protocol, I can extend it for my own
personal needs, but it's still TAP: Test::Harness-based code will still
like it, etc.
--- Joe M.
