On Friday 16 February 2007, Ovid wrote:
> Can anyone else comment on this? I understand exactly where Shlomi is
> coming from and I know that he's wrong in his thoughts, but there is
> certainly a difference of opinion regarding 'correct' behavior.
>
Actually, I wasn't claiming the current behaviour of Test::Run and
Test::Harness is the correct one. I only claimed that Test::Run behaves the
same as T::H and that TAPx::Parser broke it in this respect.
> > I find it unlikely that my code differs in interpreation from
> > Test::Harness in
> > this regard. After all, my code is derived from Test::Harness, and I
> > don't
> > recall changing the tests inside t/test-harness.t. Furthermore, the
> > results
> > specification for the test for 'bignum' is the same in both places.
>
> I can understand that. In this case, I'm not sure where Test::Harness
> goes wrong, but it's wrong. After conversation with Schwern, it was
> determined that any tests run which are more than the 'planned' tests
> are 'not ok'. I think Test::Harness is wrong in this regard and that's
> possibly part of the reason why the prove output is so strange.
Very well - I guess I agree with that. I'll wait until the new TAPx::Parser
has a stable version on CPAN and modify the tests and behaviour of Test::Run
accordingly while bumping the required version of TAPx::Parser to the new
one. (Due to the incompatibility changes).
>
> > > $ prove badplan
> > > badplan...FAILED tests 3-4
> > >
> > > Failed 2/2 tests, 0.00% okay
> > > Failed Test Stat Wstat Total Fail List of Failed
> > > ------------------------------------------------------------
> > > badplan 2 2 3-4
> > > Failed 1/1 test scripts. -2/2 subtests failed.
> > > Files=1, Tests=2, 0 wallclock secs ( 0.01 cusr + 0.01 csys =
> >
> > 0.02
> >
> > > CPU)
> > > Failed 1/1 test programs. -2/2 subtests failed.
>
> First it says that 2/2 failed and that 0.00% were OK and later says
> that -2/2 failed. Both 2/2 and -2/2 are wrong, since 2/4 tests
> 'failed'. 2 tests were planned, 4 were run. We don't have any way of
> knowing if the plan is wrong or if the final two tests were run in
> error. Only a human looking at the code can determine this. Thus, the
> 'prove' output is incorrect (and contradictory and non-sensical).
Yes, you're right.
>
> (Note: this isn't meant to be a slam on Schwern or Andy Lester.
> They've both done great work with a code base which has evolved into a
> corner.)
Sure.
>
> > > $ runtests badplan
> > > badplan.... All 2 subtests passed
> > >
> > > Test Summary Report
> > > -------------------
> > > badplan (Wstat: 0 Tests: 4 Failed: 2)
> > > Failed tests: 3-4
> > > Parse errors: Bad plan. You planned 2 tests but ran 4.
> > > Files=1, Tests=4, 0 wallclock secs ( 0.00 cusr + 0.00 csys =
> >
> > 0.00
> >
> > > CPU)
> > >
> > > That's much clearer.
>
> As pointed out, 'runtests' output is clearer, non-contradictory,
> documented, and tells the programmer what happened, unlike 'runprove'
> or 'prove'. This is one case where I think that following the exact
> behavior of T::H and 'prove' is an error.
OK.
>
> > It does, but I still don't know how to emulate the Test::Harness
> > behaviour using the new TAPx::Parser API.
>
> Admittedly, I don't know your design goals, but I don't think you
> should emulate broken behavior. You've done enough work on Test::Run
> that I suspect that fixing confusing output is not a bad thing.
>
> If you really want to emulate the behavior, check 'has_todo' and
> 'actual_ok' in TAPx::Parser::Result::Test. Those might help.
No, that's OK - I'll change the behaviour.
>
> > And furthermore, I'm not sure I'll be
> > able to
> > rely on the new API until a stable version of TAPx::Parser 0.5x is
> > released,
>
> That's fair. We're working on getting a stable release (there are
> enough changes that it might bump up to 0.6).
>
> Thanks again for pointing this out. In writing more tests for it, I
> found a subtle bug with my implementation. It won't fix your problem
> and might even make things worse. (I was comparing
> $parser->tests_planned to $result->test_num -- which might be wrong if
> tests are misnumbered -- instead of to $parser->tests_run).
>
You're welcome.
Thanks and regards,
Shlomi Fish
---------------------------------------------------------------------
Shlomi Fish [EMAIL PROTECTED]
Homepage: http://www.shlomifish.org/
Chuck Norris wrote a complete Perl 6 implementation in a day but then
destroyed all evidence with his bare hands, so no one will know his secrets.
