Test::Harness doesn't mind if you don't have numbers on your tests
(not sure if this is by design or just by implementation) so this
"test script"
print "ok a hello\n";
print "not ok b hello\n";
print "1..2\n";
Gives
t/a....FAILED test 2
Failed 1/2 tests, 50.00% okay
Failed Test Stat Wstat Total Fail Failed List of Failed
-------------------------------------------------------------------------------
t/a.t 2 1 50.00% 2
Failed 1/1 test scripts, 0.00% okay. 1/2 subtests failed, 50.00% okay.
make: *** [test_dynamic] Error 255
so (assuming this is valid TAP) you might be able to get away with
just overriding a bit of Test::Builder so that it outputs letters
rather than numbers,
F
On 12/21/05, Joe McMahon <[EMAIL PROTECTED]> wrote:
> Here's a scenario we have here at Yahoo!.
>
> The code we're testing depends on XML feeds from backend servers, which
> may sometimes be overloaded and not respond. The frontend servers work
> around this, but it would be better if we could fail a test, wait a
> bit, then go back and run it again a few times until either it
> eventually passes or never passes at all.
>
> Test::Builder does support "time travel" via current_test() - i.e., you
> reset the test number, and Test::Builder forgets the intervening tests.
> Which would be great, except the test output has already gone off to
> STDOUT/STDERR, so the followup Test::Harness-based code that reads the
> TAP gets confused.
>
> For example, if I run the following dummy.t
>
> print <<EOS;
> 1..1
> not ok 1 ... failed once
> not ok 1 ... failed twice
> ok 1 ... worked
> EOS
>
> under prove -v, I get
> dummy....1..1
> not ok 1 ... failed once
> not ok 1 ... failed twice
> ok 1 ... worked
> Test output counter mismatch [test 3]
> Don't know which tests failed: got 1 ok, expected 1
> Failed Test Stat Wstat Total Fail Failed List of Failed
> ------------------------------------------------------------------------
> -------
> dummy.t 1 ?? % ??
> Failed 1/1 test scripts, 0.00% okay. 0/1 subtests failed, 100.00% okay.
>
> The question is, is this valid TAP? The TAP document implies but does
> not explicitly state that the numbers must be in strictly ascending
> order. Test::Builder implies that repeated numbers, or reused numbers,
> should be treated as "forgotten" tests. Is Test::Builder wrong? If so,
> what is the best way to deal with tests that might eventually succeed
> if retried? Should Test::Harness just believe the TAP input and not
> count tests itself?
>
> Obviously, you can wrap up the actual test in a retry loop/function,
> but this doesn't match up with the simplicity of Test::More and related
> testing methods. It seems like the only way to address this is to
> subclass Test::Builder (or write a new class) that buffers up the test
> output and only outputs it after tests are "committed" (i.e., "I've run
> this N times and am sticking with the final result").
>
> Or am I stretching TAP too far? Thoughts?
>
> --- Joe M.
>
>
