utApia
The diag() debate raged on in pdx tonight. Of course, the sides are roughly in agreement about most things, but with differing priorities and ideas about particulars of the implementation. Perhaps it's time to collect the issues and do some thinking. Fundamentals: 1. Anything on STDERR cannot be meaningful to a tap consumer. Facts: 1. STDERR has historically been used for (non-parsable) failure messages. 2. Completely synchronizing two filehandles is not possible at the harness level (unless the harness is the kernel, then *maybe*.) 3. Merging STDOUT and STDERR breaks starting with `warn ok;` Wants: 1. Identifiable (tagged) error messages as a formal part of TAP. ~1.5 Display errors synchronously in relation to failing tests. 2. Backwards parser/producer compatibility (mostly with expectations created by fact 1.) Please correct and clarify as needed. I've tried to cook this down into the essentials. If there is a significant fact or want which is not a subset of the above, then I have completely misunderstood. If I've got the fundamentals wrong, then I'm done -- stick a fork in me. If I've got the facts wrong, correct me; otherwise, they're uh... facts. At the moment, what I'm seeing is differences in priorities placed on wants #1 and #2 and/or how much of which want you're willing to give up for the other. Forget for a moment about 'Undefined variable' and such warnings (except to remember that they exist and can ruin your day if the protocol gets in their way (I think that's the cannot be meaningful part of the fundamental.)) --Eric -- Beware of bugs in the above code; I have only proved it correct, not tried it. --Donald Knuth --- http://scratchcomputing.com ---
Re: utApia
On 15 Mar 2007, at 09:20, Eric Wilhelm wrote: At the moment, what I'm seeing is differences in priorities placed on wants #1 and #2 and/or how much of which want you're willing to give up for the other. Right. Agreed. So let's nail this down to specific actions. My plan for TAP::Parser - assuming Ovid et al agree is: 1) Disable OUT/ERR merging by default. Enabled by a --merge flag to runtests. Documentation says: using this flag destroys the important distinction between output and errors. Errors may be parsed as TAP. Bad things may happen as a result. We already have the merging implementation and some people are finding it valuable. So we don't destroy it but we make it non-default. 2) Implement TAP grammar for structured diagnostic information. This is patently a good idea. The only friction is that Test::Builder et al don't yet have support. We have to start somewhere. Unless anyone can demonstrate that either of these actions actually move us further away from our goals I'm going to get started on them, probably tomorrow. OK? -- Andy Armstrong, hexten.net
Re: No, sending diag() to STDOUT won't work. Yet again.
In article [EMAIL PROTECTED], Michael G Schwern [EMAIL PROTECTED] wrote: Piping all diagnostics to STDOUT solves nothing except maybe allowing runtests to display warnings again. You still can't tell the difference between a comment (what currently is # foo printed to STDOUT) and a failure diagnostic (what currently is # foo printed to STDERR) and diagnostics associated with a TODO test (which is # foo printed to STDOUT). I'm not advocating any change here because I'm perfectly happy with what we have now, but isn't the problem there that # means too many things? If a comment started with a # (because this is perl), and other things had some other sigil, would anyone be arguing about which filehandle they are on? Again, I'm not saying that anyone should do any work to change this, but after reading the links Schwern provided and checking the wiki, it seems that people talk about the format of the diagnostic but not the thing to signal it.
Eliminating STDERR without any disruption.
I believe I now know how to move towards no longer using STDERR for failure information display AND keep compatibility with existing test scripts, even those not written using Test::Builder or Test.pm AND not require Test::Builder, Test.pm and TH not be upgraded in lock step AND not introduce ambiguity to TAP AND (most importantly) not effect what diagnostics people installing tests with make test see. That last one is the most critical use case, people who are just running existing tests (whether they wrote them or they're someone else's) and upgrade their modules from CPAN and don't really think about it. The information they receive to their eyeballs and brain should remain the same. It goes a little like this. Let's assume a simple test like this: print 1..1\n; print # Information\n; print not ok 1\n; print STDERR # Failure\n; With the existing system a user sees: Oh god test one failed! # Failure Oh god test one faled! comes from the harness. # Failure is on STDERR. What's important is that this is what the user sees. Ok, now let's say we change the harness to display diagnostics. The user sees: Oh god test one failed! # Information # Failure A change in what is displayed to the user. So no good. Let's say we add in a new syntax which means display this verbatim. I'm going to say its if a line starts with a ! just for discussion purposes. We change our test: print TAP version 15\n; print 1..1\n; print # Information\n; print not ok 1\n; print ! Failure\n; And we upgrade the harness to recognize this new syntax and display it. Oh god test one failed! ! Failure We're no longer using STDERR yet we retain the ability to differentiate between a straight comment and information for the user. It would be implemented in Test::Builder by making diag() go to STDOUT and having it prefix messages with a ! instead of a #. That's the new producer/new parser case and it works fine. What about the old producer/new parser case? Well we have an old style script that looks like this: print 1..1\n; print # Information\n; print not ok 1\n; print STDERR # Failure\n; And the upgraded harness, which still ignores STDERR, displays it like this: Oh god test one failed! # Failure So no change, an old style producer still works with a new style parser. This is good, it means the parser can be upgraded without having to change everyone's existing tests. We have a much greater control over the parser because currently there is only one in wide-spread use, Test::Harness. Now what about a new style producer with an old style parser? print TAP version 15\n; print 1..1\n; print # Information\n; print not ok 1\n; print ! Failure\n; And the old harness displays... Oh god test one failed! Uh oh, loss of information. The old parser simply ignores the ! Failure, thinks its junk. That's not good. One possible work around is to have the new producer print information to STDERR as well as STDOUT but that means we're all noisy and gross and it something I'd like to avoid. A better work around is to simply avoid this case. As mentioned before we have much more control over the parser then the producer, since there's only one parser (Test::Harness) and lots of producers including ad-hoc ones. We upgrade Test::Harness first, we can do that because old producer / new harness works fine. Then we upgrade Test::Builder and Test.pm to be new-style producers and have them depend on the new version of Test::Harness. Voila! STDERR is eliminated from new style producers. There's no heuristics or ambiguity necessary to determine what should and should not be displayed. Existing tests continue to display just as they did. We retain the ability to put undisplayed comments in the TAP stream. Comments, issues, problems?
Re: Eliminating STDERR without any disruption.
Michael G Schwern wrote: print TAP version 15\n; print 1..1\n; print # Information\n; print not ok 1\n; print ! Failure\n; I'd really not like to see meaningful punctuation. How about diag Failure\n. Or even levels of keywords debug/info/notice/warning/ err/crit/alert/emerg (stolen from syslog.h). Oh, and yaml if that idea lives.
Re: Eliminating STDERR without any disruption.
Yitzchak Scott-Thoennes wrote: Michael G Schwern wrote: print TAP version 15\n; print 1..1\n; print # Information\n; print not ok 1\n; print ! Failure\n; I'd really not like to see meaningful punctuation. I'm going to say its if a line starts with a ! just for discussion purposes. - Just for discussion purposes - I haven't given any thought to the real syntax yet. How about diag Failure\n. Or even levels of keywords debug/info/notice/warning/ err/crit/alert/emerg (stolen from syslog.h). That's an interesting idea. My worry is making it human readable. not ok 2 err Test failed in foo.t line 2 err got: foo err expected: bar ok 3 It doesn't leap out at you the way something like this does. not ok 2 !!! Test failed in foo.t line 2 !!! got: foo !!! expected: bar ok 3 Oh, and yaml if that idea lives. The TAP diagnostic syntax is largely orthogonal to this although it makes most uses of displayed diagnostics obsolete.
