For some variety, you might want to take a look at the JavaScript
implemention of the Test::* family, which we managed to build
asyncronous testing into, so you can do things like onTimeout
pseudo-threads that will happen asyncronously from your main testing.
Now granted, that doesn't deal with retesting, but it does have to deal
with issues like "how do I know I'm at the end of a test run" and other
various things.
There might be some interesting things that could be pulled back into
the Perl ones and applied to parts of your problem.
Adam K
Joe McMahon wrote:
Here's a scenario we have here at Yahoo!.
The code we're testing depends on XML feeds from backend servers, which
may sometimes be overloaded and not respond. The frontend servers work
around this, but it would be better if we could fail a test, wait a
bit, then go back and run it again a few times until either it
eventually passes or never passes at all.
Test::Builder does support "time travel" via current_test() - i.e., you
reset the test number, and Test::Builder forgets the intervening tests.
Which would be great, except the test output has already gone off to
STDOUT/STDERR, so the followup Test::Harness-based code that reads the
TAP gets confused.
For example, if I run the following dummy.t
print <<EOS;
1..1
not ok 1 ... failed once
not ok 1 ... failed twice
ok 1 ... worked
EOS
under prove -v, I get
dummy....1..1
not ok 1 ... failed once
not ok 1 ... failed twice
ok 1 ... worked
Test output counter mismatch [test 3]
Don't know which tests failed: got 1 ok, expected 1
Failed Test Stat Wstat Total Fail Failed List of Failed
------------------------------------------------------------------------
-------
dummy.t 1 ?? % ??
Failed 1/1 test scripts, 0.00% okay. 0/1 subtests failed, 100.00% okay.
The question is, is this valid TAP? The TAP document implies but does
not explicitly state that the numbers must be in strictly ascending
order. Test::Builder implies that repeated numbers, or reused numbers,
should be treated as "forgotten" tests. Is Test::Builder wrong? If so,
what is the best way to deal with tests that might eventually succeed
if retried? Should Test::Harness just believe the TAP input and not
count tests itself?
Obviously, you can wrap up the actual test in a retry loop/function,
but this doesn't match up with the simplicity of Test::More and related
testing methods. It seems like the only way to address this is to
subclass Test::Builder (or write a new class) that buffers up the test
output and only outputs it after tests are "committed" (i.e., "I've run
this N times and am sticking with the final result").
Or am I stretching TAP too far? Thoughts?
--- Joe M.
