On Thu, Apr 11, 2002 at 10:37:10AM +0900, Curt Sampson wrote:
> On Tue, 9 Apr 2002, Michael G Schwern wrote:
>
> > > I'm not sure exactly what the purpose of this is; your test will
> > > still fail if it dies even when not in a lives_ok block, right?
> >
> > It'll fail and take the whole rest of the test program with it. Some
> > testing systems like to abort the test script on failure. Perl's
> > doesn't.
>
> I'm not entirely sure I buy this, since the framework seems perfectly
> happy to tell me that something is wrong whether I complete all
> the tests in a script or not. But it's hardly a point worth arguing.
Well, yes. But you'd like to see the results of the rest of the rest
of the tests. Not simply up to where your first failure was.
Testing Doctrine says that any tests after the first failure are
suspect because if one interface is failing, anything derived from
that will also be wrong. In practice this is rarely the case and
there's plenty of useful data to be gotten from the full run.
In practice if your test cases are getting so large that any of
this matters you have Other Problems.
> > Aegis, in particular, pretty much requires tests to always
> > exit gracefully, passing or not.
>
> As perl does: when it dies, it exits "gracefully" with error code
> 255 indicating that the test failed? Or am I misunderstanding how
> Aegis works?
Aegis works by reading the exit code of the test. 0 is success, 1 is
failure, anything is "something went horribly wrong". The thing to
keep in mind is Aegis was designed by a pre-ANSI C programmer, so any
run-time errors that cause the program to abort are an indication that
something is Very Wrong.
The reason this becomes important is Aegis has two modes of testing.
The first is your normal "everything should pass". But the second is
interesting. When your change in Aegis is fixing a bug Aegis requires
that the new tests against the old code *fail*, and that the new tests
against the new code succeed.
It has to fail gracefully, exiting in a predictable manner, in order
to be reasonably sure that new tests will catch the bug. If the test
simply dies you don't know if that was because it caught a SIGINT or
it segfaulted or what.
Which becomes a Royal Pain In The Ass when fixing the bug involves
adding new methods and then you have to do things like:
if( Some::Class->can('this_new_method') ) {
...some tests involving this_new_method()...
}
else {
fail('this_new_method not defined');
}
which becomes Rapidly Tiresome. This is why SKIP and TODO blocks work
the way they do (even though they're not directly applicable) and why
Test::More will exit with the number of tests which failed (makes it
much easier to write a simple Aegis perl testing hook).
<XUnity stuff snipped>
> (And yes, it's not a co-incidence that this looks a lot like Beck
> and Gamma's unit test frameworks for Java and Smalltalk.)
>
> Thoughts?
See also Test::Unit and Test::Inline. Test::Unit because it's XUnit
in Perl. Test::Inline because each block of embedded tests runs in
it's own lexical scope.
--
Michael G. Schwern <[EMAIL PROTECTED]> http://www.pobox.com/~schwern/
Perl Quality Assurance <[EMAIL PROTECTED]> Kwalitee Is Job One
Obscenity is the last resort of the illiterate, Mother Fucker
-- KAL
