I believe I now know how to move towards no longer using STDERR for failure
information display AND keep compatibility with existing test scripts, even
those not written using Test::Builder or Test.pm AND not require
Test::Builder, Test.pm and TH not be upgraded in lock step AND not introduce
ambiguity to TAP AND (most importantly) not effect what diagnostics people
installing tests with "make test" see.

That last one is the most critical use case, people who are just running
existing tests (whether they wrote them or they're someone else's) and upgrade
their modules from CPAN and don't really think about it.  The information they
receive to their eyeballs and brain should remain the same.

It goes a little like this.

Let's assume a simple test like this:

    print "1..1\n";
    print "# Information\n";
    print "not ok 1\n";
    print STDERR "# Failure\n";

With the existing system a user sees:

    Oh god test one failed!
    # Failure

"Oh god test one faled!" comes from the harness.  "# Failure" is on STDERR.
What's important is that this is what the user sees.

Ok, now let's say we change the harness to display diagnostics.  The user sees:

    Oh god test one failed!
    # Information
    # Failure

A change in what is displayed to the user.  So no good.


Let's say we add in a new syntax which means "display this verbatim".  I'm
going to say its if a line starts with a ! just for discussion purposes.

We change our test:

    print "TAP version 15\n";
    print "1..1\n";
    print "# Information\n";
    print "not ok 1\n";
    print "! Failure\n";

And we upgrade the harness to recognize this new syntax and display it.

    Oh god test one failed!
    ! Failure

We're no longer using STDERR yet we retain the ability to differentiate
between a straight comment and information for the user.  It would be
implemented in Test::Builder by making diag() go to STDOUT and having it
prefix messages with a ! instead of a #.

That's the new producer/new parser case and it works fine.  What about the old
producer/new parser case?  Well we have an old style script that looks like 
this:

    print "1..1\n";
    print "# Information\n";
    print "not ok 1\n";
    print STDERR "# Failure\n";

And the upgraded harness, which still ignores STDERR, displays it like this:

    Oh god test one failed!
    # Failure

So no change, an old style producer still works with a new style parser.  This
is good, it means the parser can be upgraded without having to change
everyone's existing tests.  We have a much greater control over the parser
because currently there is only one in wide-spread use, Test::Harness.

Now what about a new style producer with an old style parser?

    print "TAP version 15\n";
    print "1..1\n";
    print "# Information\n";
    print "not ok 1\n";
    print "! Failure\n";

And the old harness displays...

    Oh god test one failed!

Uh oh, loss of information.  The old parser simply ignores the "! Failure",
thinks its junk.  That's not good.  One possible work around is to have the
new producer print information to STDERR as well as STDOUT but that means
we're all noisy and gross and it something I'd like to avoid.

A better work around is to simply avoid this case.  As mentioned before we
have much more control over the parser then the producer, since there's only
one parser (Test::Harness) and lots of producers including ad-hoc ones.  We
upgrade Test::Harness first, we can do that because old producer / new harness
works fine.  Then we upgrade Test::Builder and Test.pm to be new-style
producers and have them depend on the new version of Test::Harness.

Voila!  STDERR is eliminated from new style producers.  There's no heuristics
or ambiguity necessary to determine what should and should not be displayed.
Existing tests continue to display just as they did.  We retain the ability to
put undisplayed comments in the TAP stream.

Comments, issues, problems?

Reply via email to