What I find surprising is the concept that a TODO test is assumed to fail.
If I mark a test (block) as TODO, I would have thought by flagging some
tests as not done yet then for the purposes of PASS/FAIL I don't care
what the result it.
Broken or incomplete code may still have a failure mode where it returns
"ok". That said, I think it's enough of a novelty that it's worth
showing some sort of warning for.
But that's just me, and I don't use TODO tests on anger (yet)
Adam K
Ovid wrote:
The following line is giving me pause:
ok 9 Elegy 9B # TOdO
That's an 'unexpectedly succeeded' test ('bonus', in the Test::Harness world).
Right now, if that read 'not ok # TODO', TAPx::Parser would have this:
passed true
actual_passed false
todo_failed false
But since it reads 'ok', I've reversed the sense:
passed false
actual_passed true
todo_failed true
In this case, Test::Harness and friends report that 'ok 9 # todo' is passing,
not failing, but I'm reporting the opposite result. I think my behavior is
more correct because I'm trying to write things so that someone who forgets
writes a bad harness will still see what's going on. For example, let's say
someone only wants to see failing tests:
# the 'source' key is new. You no longer have to manually create a stream
my $parser = TAPx::Parser->new( { source => $test_file } );
while ( my $result = $parser->next ) {
print $result->as_string if ! $parser->passed;
}
With that, they'll never notice that 'ok # TODO' has unexpectedly succeeded if I
adopt the current behavior of Test::Harness unless they explicitly remember to
check the $parser->todo_failed method.
I propose that 'ok # TODO' tests be reported as failures. I think it would be
good if Test::Harness also adopted this strategy because right now, if you see
that tests unexpectedly succeeded, you don't know which tests they are and you
have to try and grep through the TAP output manually.
Thoughts? Is this going to break a lot of stuff (I suspect it might).
Cheers,
Ovid
