Re: Eliminating STDERR without any disruption.
On 19 Mar 2007, at 02:05, A. Pagaltzis wrote: Hmm, I never got a subscription confirmation request. Check the pending subscriber list, I should be in there. I've just manually subscribed you. -- Andy Armstrong, hexten.net
Re: Eliminating STDERR without any disruption.
On Sun, Mar 18, 2007 at 01:06:48AM +, Andy Armstrong wrote: And in fact now I see how simple the proposal is I realise that even mentioning how it might be implemented is completely unnecessary. Sorry folks - it's been a long day :) Well then, you should spend less time working actually writing code, and more time relaxing and being sociable by replying to e-mail on lists. :-) Meetings - the practical alternative to work. Nicholas Clark
Re: Eliminating STDERR without any disruption.
Andy Armstrong wrote: On 18 Mar 2007, at 01:19, Michael G Schwern wrote: [snip] In the TAP stream the association between a block of YAML and a particular test could be made explicit by putting the test number in the block. not ok 5 --- test: 5 got: 23 expected: 42 ... This would be optional. How's that? I'd rather just make it explicit that each test result may be followed by zero or one YAML documents containing information relating to that test. In effect a test and its optional diagnostics should be an atom I think. I worry about situations where multiple producers are simultaneously outputting to one stream as in with threaded or forked producers. I guess this is going to be a mess no matter what, there has to be some sort of coordination and that's outside of our domain.
Re: Eliminating STDERR without any disruption.
On Sun, Mar 18, 2007 at 01:37:36AM +, Andy Armstrong wrote: Are we expecting a YAML reader / writer to be core anytime soon? Not that I'm aware of. Nicholas Clark
Re: Eliminating STDERR without any disruption.
Ovid wrote: Which brings me to my next point: we appear to have no attracted any people from other programming communities to the discussion about the next version of TAP, but I hear that some are using TAP::Parser any way.Does anyone know of groups that are *regularly* using the TAP::Parser for test suites in other languages? I know of xmms2, but that's about it. I'd like to find more for some talks I have coming up and I'd also like to find a way to encourage other TAP producers to use this. The darcs folks (Haskell) write some of their tests in Perl. They still use Test::Harness. That's all I can think of.
Re: Eliminating STDERR without any disruption.
Ovid wrote: Amusingly, when I was at last year's Google Test Automation Conference, lots of folks were talking about their XML output from their test harnesses and many of them weren't happy with it (having to wait for a well-formed XML document sucks, particularly when a human can read the partially formed document and still understand things). I wound up giving a lightning talk on TAP in hopes of reeling in the people who were unhappy with XML, but no dice. It would be nice to know what those other systems do that TAP does not. Steal their ideas. Another problem is we're not even on most people's maps. Here's one example. http://opensourcetesting.org Might it be time we start up a TAP-only mailing list? If nothing else it would unclog perl-qa from those who want to talk about something other than TAP.
Re: Eliminating STDERR without any disruption.
On Sunday 18 March 2007, Michael G Schwern wrote: Ovid wrote: Amusingly, when I was at last year's Google Test Automation Conference, lots of folks were talking about their XML output from their test harnesses and many of them weren't happy with it (having to wait for a well-formed XML document sucks, particularly when a human can read the partially formed document and still understand things). I wound up giving a lightning talk on TAP in hopes of reeling in the people who were unhappy with XML, but no dice. It would be nice to know what those other systems do that TAP does not. Steal their ideas. Another problem is we're not even on most people's maps. Here's one example. http://opensourcetesting.org Might it be time we start up a TAP-only mailing list? If nothing else it would unclog perl-qa from those who want to talk about something other than TAP. I think that's a good idea. I believe many people here got tired of the TAP-related discussions, and they grew to unpreceded volume. Regards, Shlomi Fish - Shlomi Fish [EMAIL PROTECTED] Homepage:http://www.shlomifish.org/ Chuck Norris wrote a complete Perl 6 implementation in a day but then destroyed all evidence with his bare hands, so no one will know his secrets.
Re: Eliminating STDERR without any disruption.
On 18 Mar 2007, at 21:12, Michael G Schwern wrote: Another problem is we're not even on most people's maps. Here's one example. http://opensourcetesting.org Might it be time we start up a TAP-only mailing list? If nothing else it would unclog perl-qa from those who want to talk about something other than TAP. +1 I volunteer to host it if necessary. -- Andy Armstrong, hexten.net
Re: Eliminating STDERR without any disruption.
* Andy Armstrong [EMAIL PROTECTED] [2007-03-18 22:45]: I volunteer to host it if necessary. That’s not a bad idea; a dedicated domain for TAP without any mention of Perl in the name is probably a good marketing move. I don’t mean that we need to ashamedly hide the fact that we’re Perlers, but I think it would be wise not to give people cues based on which they can quickly superficially dismiss TAP as irrelevant to their own work, if we want it to catch on. Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Re: Eliminating STDERR without any disruption.
* A. Pagaltzis [EMAIL PROTECTED] [2007-03-18 23:43]: * Andy Armstrong [EMAIL PROTECTED] [2007-03-18 22:45]: I volunteer to host it if necessary. That’s not a bad idea; a dedicated domain for TAP without any mention of Perl in the name is probably a good marketing move. I don’t mean that we need to ashamedly hide the fact that we’re Perlers, but I think it would be wise not to give people cues based on which they can quickly superficially dismiss TAP as irrelevant to their own work, if we want it to catch on. Oh, and it would be nice if the TAP::Parser list (or whatever the tapx list is about) would be rolled back into the TAP list; I didn’t know that list even existed until the recent crossposts, and I think this sort of fragmentation is unwise. Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Re: Eliminating STDERR without any disruption.
A. Pagaltzis wrote: * A. Pagaltzis [EMAIL PROTECTED] [2007-03-18 23:43]: * Andy Armstrong [EMAIL PROTECTED] [2007-03-18 22:45]: I volunteer to host it if necessary. That’s not a bad idea; a dedicated domain for TAP without any mention of Perl in the name is probably a good marketing move. I don’t mean that we need to ashamedly hide the fact that we’re Perlers, but I think it would be wise not to give people cues based on which they can quickly superficially dismiss TAP as irrelevant to their own work, if we want it to catch on. Oh, and it would be nice if the TAP::Parser list (or whatever the tapx list is about) would be rolled back into the TAP list; I didn’t know that list even existed until the recent crossposts, and I think this sort of fragmentation is unwise. Those two paragraphs would seem to contradict themselves. On the one hand we want a TAP list in which we can discuss the protocol independent of its implementation or Perl. Then its said to merge the discussion about the details of the TAP::Parser Perl implementation into the Perl and implementation independent TAP mailing list. Does not compute. What did you intend to gain by merging the TAP::Parser list into the TAP list?
Re: Eliminating STDERR without any disruption.
Andy Armstrong wrote: On 18 Mar 2007, at 21:12, Michael G Schwern wrote: Another problem is we're not even on most people's maps. Here's one example. http://opensourcetesting.org Might it be time we start up a TAP-only mailing list? If nothing else it would unclog perl-qa from those who want to talk about something other than TAP. +1 I volunteer to host it if necessary. Fine by me as long as it has list archives.
Re: Eliminating STDERR without any disruption.
* Michael G Schwern [EMAIL PROTECTED] [2007-03-18 22:10]: Andy Armstrong wrote: There's a lot of wisdom in Ghostbusters of which this is just one example. uhhh... oh. ROTFL! Exactly my reaction. Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Re: Eliminating STDERR without any disruption.
A. Pagaltzis wrote: What did you intend to gain by merging the TAP::Parser list into the TAP list? Well, as long as the discussions about TAP remain firmly on the TAP list, that might be OK. If the TAP::Parser list deals purely with implementation details that are truly irrelevant to TAP per se, maybe they shoud stay separate. But the volume of discussion might not be high enough to sustain multiple lists, even if the topical split is correct, and having discussions about details of TAP implementation in several different language on the same TAP list might be healthy for the protocol. I see this mistake made in lots of messageboards on the web, where people eagerly set up a hundred forums for specific topics, and then none of them gets any traffic. It’s better to throw it all together at first, and split the discussion only when the volume necessitates it. What's happened in the past is the TAP::Parser folks find an ambiguity in the protocol and then they ask on perl-qa. Granted this was before there was a tap-parser list, but I expect this to continue. I don't know if you've looked at the TAP::Parser list archives but its mostly commit messages. What would be useful is in digging out Ovid's numerous questions for clarification a few months ago on perl-qa about TAP and fixing up the TAP spec based on the discussions therein.
Re: Eliminating STDERR without any disruption.
* Andy Armstrong [EMAIL PROTECTED] [2007-03-19 00:35]: On 18 Mar 2007, at 23:10, A. Pagaltzis wrote: * Andy Armstrong [EMAIL PROTECTED] [2007-03-19 00:05]: http://testanything.org/pipermail/tap-l/ Subscribed! Can't see you on the subscribers list. Did it not work? Hmm, I never got a subscription confirmation request. Check the pending subscriber list, I should be in there. Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Re: Eliminating STDERR without any disruption.
Some other ideas I saw in a real system: - distinguish between failures in the configuration of the system under test and the actual tes this would yield levels such as conf_error and conf_warning - In addition I am not sure if some of these calls should automatically bail out as being fatal errors Gabor
Re: Eliminating STDERR without any disruption.
In article [EMAIL PROTECTED], Michael G Schwern [EMAIL PROTECTED] wrote: A. Pagaltzis wrote: The most bangs I can count instantly by looking at them is four. For five bangs and up, all I see is âlots of bangs.â I have to count character by character to tell them apart. Visually, I canât distinguish `fatal` from `fail` at all. Another problem is that Iâd never remember the exact hierarchy. So with your proposal Iâd have to count bangs for any message of import, and then go look which number means what. Do you, as a human, have to exactly distinguish them? Isn't the content of the message and the rough number of bangs enough? not ok 1 ! Failed test in foo.t line 2 ok 2 !!! WHOA! The fabric of the universe just broke down! !!! This should never happen! Please contact the author immediately! if you're going to use a different starting character for these messages, how about a [ ? Follow the start of the string by a real word: not ok 1 [fail] Failed test in foo.t line 2 ok 2 [fatal] WHOA! The fabric of the universe just broke down! [damn it, Jim, I'm a doctor, not a programmer!] This should never happen! Please contact the author immediately! It doesn't set it off visually as nicely as the repeated bangs, but that's not the goal here. People want to read this stuff programmatically. :)
Re: Eliminating STDERR without any disruption.
On 17 Mar 2007, at 17:36, brian d foy wrote: if you're going to use a different starting character for these messages, how about a [ ? Follow the start of the string by a real word: not ok 1 [fail] Failed test in foo.t line 2 ok 2 [fatal] WHOA! The fabric of the universe just broke down! [damn it, Jim, I'm a doctor, not a programmer!] This should never happen! Please contact the author immediately! It doesn't set it off visually as nicely as the repeated bangs, but that's not the goal here. People want to read this stuff programmatically. :) I'm still not clear what this notation provides that we can't do with the new YAML machine readable diagnostic syntax. What are the supposed benefits? Concision? not ok 1 --- message: Failed test in foo.t line 2 severity: fail data: expected: this got: that ... ok 2 --- message: WHOA! The fabric of the universe just broke down! severity: fatal ... -- Andy Armstrong, hexten.net
Re: Eliminating STDERR without any disruption.
Andy Armstrong wrote:
I'm still not clear what this notation provides that we can't do with
the new YAML machine readable diagnostic syntax. What are the supposed
benefits? Concision?
Yeah, brevity. Pretty much. And human readability. YAML is pretty good and
all but some text prefixed with some bangs is always going to be easier to read.
Compare what this hypothetical code might produce:
notice(Attention.);
notice(Attention.);
notice(Testicles.);
notice(That is all.);
Attention.
Attention.
Testicles.
That is all.
vs
---
message: Attention.
severity: notice
...
---
message: Attention.
severity: notice
...
---
message: Testicles.
severity: notice
...
---
message: That is all.
severity: notice
...
Of course you could be careful and put everything together in one notice()
call producing one YAML block but that's not always possible and the user has
to be careful is not a great way to start out a protocol. The syntax
makes it unnecessary.
Also the messages are free-form and not associated with any particular test.
Consider:
pass(this is a test);
notice(My hovercraft is full of eels.);
With the ! proposal it produces...
ok 1 - this is a test
My hovercraft is full of eels.
With YAML it produces...
ok 1 - this is a test
---
message: My hovercraft is full of eels.
severity: notice
...
Note how similar that is to:
pass(this is a test, { message = My hovercraft is full of eels. });
ok 1 - this is a test
---
message: My hovercraft is full of eels.
severity: pass
...
Though a parser should be able to tell the difference via the severity I'm
worried this is shaving a bit too fine.
not ok 1
---
message: Failed test in foo.t line 2
severity: fail
data:
expected: this
got: that
...
ok 2
---
message: WHOA! The fabric of the universe just broke down!
severity: fatal
...
*psst* Remember to indent
not ok 1
---
message: WHOA! The fabric of the universe just broke down!
severity: fatal
...
Re: Eliminating STDERR without any disruption.
brian d foy wrote: if you're going to use a different starting character for these messages, how about a [ ? Follow the start of the string by a real word: not ok 1 [fail] Failed test in foo.t line 2 ok 2 [fatal] WHOA! The fabric of the universe just broke down! [damn it, Jim, I'm a doctor, not a programmer!] This should never happen! Please contact the author immediately! It doesn't set it off visually as nicely as the repeated bangs, but that's not the goal here. People want to read this stuff programmatically. :) Well, we do want TAP to remain human readable otherwise we'd just say the hell with it and use XML or something and what a cold, hard dystopia that would be. I think something along the lines that brian suggests is the right thing. Characterized by * A visually distinctive syntactical element * A short, unique label indicating the level How about [*label*] or even just **label**? We could even say, for brevity, that no label == notice. not ok 1 **fail** Failed test in foo.t line 2 ok 2 **pass** Some information ok 3 We're going to connect to the Internet now. Hold onto your hats! ok 4
Re: Eliminating STDERR without any disruption.
* Michael G Schwern [EMAIL PROTECTED] [2007-03-18 00:55]: How about [*label*] or even just **label**? We could even say, for brevity, that no label == notice. not ok 1 **fail** Failed test in foo.t line 2 ok 2 **pass** Some information ok 3 We're going to connect to the Internet now. Hold onto your hats! ok 4 Works for me. Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Re: Eliminating STDERR without any disruption.
Andy Armstrong wrote:
( I'm going to be calling the YAML diagnostic syntax YAMLish and I
reckon this proposal should be called bang+ :)
I'm calling it the logging proposal for lack of anything better. The bangs
are now gone.
Yeah, brevity. Pretty much. And human readability. YAML is pretty
good and
all but some text prefixed with some bangs is always going to be
easier to read.
OK, well it wouldn't be too hard to modify the YAMLish reader / writer
to handle this syntax too.
You mentioned that one wiki, too, and it confused me. What does the YAML
reader have to do with the bang syntax? Are you proposing something like:
not ok 1
---
Failed test in foo.t line 23.
got: this
exepected: that
...
That's something I'd very, very much like to avoid. It means TAP parsers now
have to write their own customized YAML parsers and we lose the benefits of
using an established syntax.
But I have to say I'm not 100% convinced. Weighing the (arguable)
readability advantage against the use of an established format (which is
still pretty readable) I think I come down on the YAML side. Of course
that might be because I've spent the last couple of days writing the
YAMLish reader / writer :)
Its not either / or. They're complimentary. Most test failure diagnostics
are going to be structured and use structured YAML diagnostics. But sometimes
you just need to print a line of text. And outputting a full blown YAML
document for that is visual overkill. For that there's the logging syntax.
The general problem being that we don't know whether a YAML block is
associated with a test or is just a disassociated comment?
Yes, which is something that's probably going to have to be fixed as I can see
a reason to want structured diagnostics not associated with a test. I think
you mentioned that, too.
From the producer side I'm hoping to be able to come up with a clean way
of marshalling all the diagnostics associated with a particular test and
then outputting them as a single YAML document presumably either just
before the next test or at the end of the test run in the case of the
last test.
Oh, that's easy.
is( 23, 42, { foo = bar, name = is 23 42? });
If the name argument is a hash ref we interpret it instead as extra
diagnostics.
*psst* Remember to indent
not ok 1
---
message: WHOA! The fabric of the universe just broke down!
severity: fatal
...
I must have missed the rationale for indenting the YAML. What's that about?
Several reasons. Its easier for the human to still see the ok and not ok
lines around all that YAML so you can still do eyeball parsing.
not ok 1
---
got: this
expected: that
...
not ok 2
---
got: up
expected: down
...
vs
not ok 1
---
got: this
expected: that
...
not ok 2
---
got: up
expected: down
...
It prevents confusion with existing parsers.
$ cat ~/tmp/foo.t
#!/usr/bin/perl -w
print END;
1..1
ok 1
---
ok: 1
...
END
$ prove ~/tmp/foo.t
/Users/schwern/tmp/fooFAILED test 2
Failed 1/1 tests, 0.00% okay
It prevents any existing TAP producer which might be dumping YAML to STDOUT
from being parsed.
Finally it allows the ... to be accidentally omitted without the YAML parser
swallowing up all the remaining tests. Sort of a fail-safe.
not ok 1
---
got: this
expected: that
ok 2
The TAP parser simply starts at /^\s+---\s*$/ and ends at either /^\s+...\s*$/
or the next test line. This means the TAP parser needs no knowledge of YAML,
just the start and end of document delimiters. It also allows the TAP parser
to pass a complete document to the YAML parser, the YAML parser does not have
to stream. Means you can use most off-the-shelf YAML parsers.
I've been wondering why you're doing all the work altering YAML::Tiny into a
stream parser. It should not be necessary and it means TAP::Parser has to
maintain its own YAML parser.
Re: Eliminating STDERR without any disruption.
On 18 Mar 2007, at 00:36, Michael G Schwern wrote:
OK, well it wouldn't be too hard to modify the YAMLish reader /
writer
to handle this syntax too.
You mentioned that one wiki, too, and it confused me. What does
the YAML
reader have to do with the bang syntax? Are you proposing
something like:
not ok 1
---
Failed test in foo.t line 23.
got: this
exepected: that
...
No - just expressing myself badly. What I meant was more like 'we
could plug this syntax in in the same way that we do the YAMLish
reader / writer'.
That's something I'd very, very much like to avoid. It means TAP
parsers now
have to write their own customized YAML parsers and we lose the
benefits of
using an established syntax.
Yes. Sorry for the confusion.
But I have to say I'm not 100% convinced. Weighing the (arguable)
readability advantage against the use of an established format
(which is
still pretty readable) I think I come down on the YAML side. Of
course
that might be because I've spent the last couple of days writing the
YAMLish reader / writer :)
Its not either / or. They're complimentary. Most test failure
diagnostics
are going to be structured and use structured YAML diagnostics.
But sometimes
you just need to print a line of text. And outputting a full blown
YAML
document for that is visual overkill. For that there's the logging
syntax.
Ah. Penny drops. Thanks.
The general problem being that we don't know whether a YAML block is
associated with a test or is just a disassociated comment?
Yes, which is something that's probably going to have to be fixed
as I can see
a reason to want structured diagnostics not associated with a
test. I think
you mentioned that, too.
From the producer side I'm hoping to be able to come up with a
clean way
of marshalling all the diagnostics associated with a particular
test and
then outputting them as a single YAML document presumably either just
before the next test or at the end of the test run in the case of the
last test.
Oh, that's easy.
is( 23, 42, { foo = bar, name = is 23 42? });
If the name argument is a hash ref we interpret it instead as
extra diagnostics.
That's OK in the simple case but you might want to see if the test
failed and then do some work to build the diagnostic data. So you'd
have to optionally be able to
is ( 23, 42 );
my $info = { foo = bar, name = is 23 42? };
details( $info );
I must have missed the rationale for indenting the YAML. What's
that about?
Several reasons. Its easier for the human to still see the ok
and not ok
lines around all that YAML so you can still do eyeball parsing.
[snip good reasons]
OK, I'll make it work like that. Sorry I haven't been paying more
attention :)
The TAP parser simply starts at /^\s+---\s*$/ and ends at either /^
\s+...\s*$/
or the next test line. This means the TAP parser needs no
knowledge of YAML,
just the start and end of document delimiters. It also allows the
TAP parser
to pass a complete document to the YAML parser, the YAML parser
does not have
to stream. Means you can use most off-the-shelf YAML parsers.
I think it's worthwhile to define the subset of YAML that we expect
to see. If you just say it's YAML then potentially you're requiring
the baggage of a hugely complex parser.
I've been wondering why you're doing all the work altering
YAML::Tiny into a
stream parser. It should not be necessary and it means TAP::Parser
has to
maintain its own YAML parser.
That's not the only reason. YAML::Tiny is more about reading and
writing config files than safely round-tripping arbitrary data
structures. So, for example, it doesn't handle unprintable characters
in hash keys correctly. I could fix that and also
Write doesn't quote unprintables correctly (fix supplied)
http://rt.cpan.org/Public/Bug/Display.html?id=25509
but by then it starts to look more sensible to implement our own well
defined subset of YAML that's designed for safe round tripping of
whatever data is thrown at it.
--
Andy Armstrong, hexten.net
Re: Eliminating STDERR without any disruption.
On 18 Mar 2007, at 01:03, Andy Armstrong wrote: No - just expressing myself badly. What I meant was more like 'we could plug this syntax in in the same way that we do the YAMLish reader / writer'. And in fact now I see how simple the proposal is I realise that even mentioning how it might be implemented is completely unnecessary. Sorry folks - it's been a long day :) -- Andy Armstrong, hexten.net
Re: Eliminating STDERR without any disruption.
Michael G Schwern wrote: How about diag Failure\n. Or even levels of keywords debug/info/notice/warning/ err/crit/alert/emerg (stolen from syslog.h). That's an interesting idea. My worry is making it human readable. not ok 2 err Test failed in foo.t line 2 err got: foo err expected: bar ok 3 It doesn't leap out at you the way something like this does. not ok 2 !!! Test failed in foo.t line 2 !!! got: foo !!! expected: bar ok 3 What if the number of bangs indicated the severity level? We already do this thing things like: !! REALLY IMPORTANT MESSAGE !!! I don't know if we need all 8 levels used in syslog. I'm not sure where the distinction comes between Emergency, Alert, Critical and Error when it comes to testing. But its a good start. Some undefined levels we can define later seems like a good idea, leaves some wiggle room in the protocol. http://tools.ietf.org/html/rfc3164 Numerical Severity Code 0 Emergency: system is unusable 1 Alert: action must be taken immediately 2 Critical: critical conditions 3 Error: error conditions 4 Warning: warning conditions 5 Notice: normal but significant condition 6 Informational: informational messages 7 Debug: debug-level messages We would, of course, invert the numerics. Granted, syslog is pretty old technology. Anyone familiar with the current thinking on this sort of stuff?
Re: Eliminating STDERR without any disruption.
On 16 Mar 2007, at 07:53, Michael G Schwern wrote: [snip] I don't know if we need all 8 levels used in syslog. I'm not sure where the distinction comes between Emergency, Alert, Critical and Error when it comes to testing. But its a good start. Some undefined levels we can define later seems like a good idea, leaves some wiggle room in the protocol. http://tools.ietf.org/html/rfc3164 Numerical Severity Code 0 Emergency: system is unusable 1 Alert: action must be taken immediately 2 Critical: critical conditions 3 Error: error conditions 4 Warning: warning conditions 5 Notice: normal but significant condition 6 Informational: informational messages 7 Debug: debug-level messages We would, of course, invert the numerics. Granted, syslog is pretty old technology. Anyone familiar with the current thinking on this sort of stuff? [snip] Maybe use the levels from Log4J, Log::Log4perl, et al? fatal error warn info debug ? Adrian
Re: Eliminating STDERR without any disruption.
Adrian Howard wrote: Maybe use the levels from Log4J, Log::Log4perl, et al? fatal error warn info debug Ok, maybe take that and tailor it more to testing. Here it is in order of severity. The recommended display level would be warn. fatal !!! There's an error in the TAP producer and it has to shut down. Test::More's WHOA THERE! messages are another. fail !! Info about a test which failed. This would be all existing failure diagnostics, the Bail out! message and end of failing test script messages like You ran 3 but said you'd run 5. warn ! There's an ambiguity in the use of the TAP producer. Test::More's messages like you named a test with a number, confusing. notice Normal but significant condition (from syslog). This is some text you want the user to see that isn't associated with a particular test. Trying to connect to the Internet. The next test might take a long time. pass !!! Info about a test which passed. info !! General information about the test that the user probably doesn't need to see. This is currently done with a # foo comment sent to STDOUT. debug ! Debugging information.
Re: Eliminating STDERR without any disruption.
* Michael G Schwern [EMAIL PROTECTED] [2007-03-16 11:55]: fatal !!! fail !! warn ! notice pass !!! info !! debug ! The most bangs I can count instantly by looking at them is four. For five bangs and up, all I see is “lots of bangs.” I have to count character by character to tell them apart. Visually, I can’t distinguish `fatal` from `fail` at all. Another problem is that I’d never remember the exact hierarchy. So with your proposal I’d have to count bangs for any message of import, and then go look which number means what. I would say that at least `pass`, `info` and `debug` should have different punctuation. So my minimal proposal is something like the following: fatal fail !!! warn !! notice ! pass ? info @ debug # But I’d also say that these are not a strict hierarchy: you can sort it in several slightly different ways depending on what you’re looking out for. So it would be worthwhile to make the punctuation less regular and more self-explanatory. fatal X!!X fail !! warn ! notice ? pass !? info @ debug # I’d also want an option to append something like ' [warn]' to warning messages, though. Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Re: Eliminating STDERR without any disruption.
A. Pagaltzis wrote: The most bangs I can count instantly by looking at them is four. For five bangs and up, all I see is “lots of bangs.” I have to count character by character to tell them apart. Visually, I can’t distinguish `fatal` from `fail` at all. Another problem is that I’d never remember the exact hierarchy. So with your proposal I’d have to count bangs for any message of import, and then go look which number means what. Do you, as a human, have to exactly distinguish them? Isn't the content of the message and the rough number of bangs enough? not ok 1 ! Failed test in foo.t line 2 ok 2 !!! WHOA! The fabric of the universe just broke down! !!! This should never happen! Please contact the author immediately! Right now we have no means to distinguish and there's no problem when you're eyeballing it. not ok 1 # Failed test in foo.t line 2 ok 2 # WHOA THERE! The fabric of the universe just broke down! # This should never happen! Please contact the author immediately! That said, I'm not happy with having so many bangs either. I would say that at least `pass`, `info` and `debug` should have different punctuation. So my minimal proposal is something like the following: fatal fail !!! warn !! notice ! pass ? info @ debug # But I’d also say that these are not a strict hierarchy: you can sort it in several slightly different ways depending on what you’re looking out for. So it would be worthwhile to make the punctuation less regular and more self-explanatory. fatal X!!X fail !! warn ! notice ? pass !? info @ debug # I'm wary of adding in too much punctuation. Its really not clear what levels !?, ?, X!!X and @ would mean, or even that they're log messages, without looking it up. And yes, the fixed set of bangs worries me. There's no room to add a new level in later. I would like to use labels instead of numbers if possible. I’d also want an option to append something like ' [warn]' to warning messages, though. This is something I'd leave up to your TAP harness to do. Although there's this possibility: 1..1 !info! Temp dir is /tmp/aslkdjoiu3 not ok 1 !fail! Failed test in foo.t line 3. !fail! got: 42 !fail! expected: 23 Though I still think that's not as visually distinctive as: 1..1 ! Temp dir is /tmp/aslkdjoiu3 not ok 1 !! Failed test in foo.t line 3. !! got: 42 !! expected: 23
Re: Eliminating STDERR without any disruption.
* Michael G Schwern [EMAIL PROTECTED] [2007-03-17 01:35]: Its really not clear what levels !?, ?, X!!X and @ would mean, or even that they're log messages, without looking it up. I suppose that’s true, although that situation is not really different from the bangs. * Michael G Schwern [EMAIL PROTECTED] [2007-03-17 02:05]: It seems to all be a consequence of mapping numbers (of bangs) to meaning, something I usually rant about avoiding. So some way to put a string label into the protocol while being visually distinctive seems the way to go. Agreed. I’m not married to the suggestion I made. Although I did think that !? was quite a speaking “name.” :-) Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Eliminating STDERR without any disruption.
I believe I now know how to move towards no longer using STDERR for failure information display AND keep compatibility with existing test scripts, even those not written using Test::Builder or Test.pm AND not require Test::Builder, Test.pm and TH not be upgraded in lock step AND not introduce ambiguity to TAP AND (most importantly) not effect what diagnostics people installing tests with make test see. That last one is the most critical use case, people who are just running existing tests (whether they wrote them or they're someone else's) and upgrade their modules from CPAN and don't really think about it. The information they receive to their eyeballs and brain should remain the same. It goes a little like this. Let's assume a simple test like this: print 1..1\n; print # Information\n; print not ok 1\n; print STDERR # Failure\n; With the existing system a user sees: Oh god test one failed! # Failure Oh god test one faled! comes from the harness. # Failure is on STDERR. What's important is that this is what the user sees. Ok, now let's say we change the harness to display diagnostics. The user sees: Oh god test one failed! # Information # Failure A change in what is displayed to the user. So no good. Let's say we add in a new syntax which means display this verbatim. I'm going to say its if a line starts with a ! just for discussion purposes. We change our test: print TAP version 15\n; print 1..1\n; print # Information\n; print not ok 1\n; print ! Failure\n; And we upgrade the harness to recognize this new syntax and display it. Oh god test one failed! ! Failure We're no longer using STDERR yet we retain the ability to differentiate between a straight comment and information for the user. It would be implemented in Test::Builder by making diag() go to STDOUT and having it prefix messages with a ! instead of a #. That's the new producer/new parser case and it works fine. What about the old producer/new parser case? Well we have an old style script that looks like this: print 1..1\n; print # Information\n; print not ok 1\n; print STDERR # Failure\n; And the upgraded harness, which still ignores STDERR, displays it like this: Oh god test one failed! # Failure So no change, an old style producer still works with a new style parser. This is good, it means the parser can be upgraded without having to change everyone's existing tests. We have a much greater control over the parser because currently there is only one in wide-spread use, Test::Harness. Now what about a new style producer with an old style parser? print TAP version 15\n; print 1..1\n; print # Information\n; print not ok 1\n; print ! Failure\n; And the old harness displays... Oh god test one failed! Uh oh, loss of information. The old parser simply ignores the ! Failure, thinks its junk. That's not good. One possible work around is to have the new producer print information to STDERR as well as STDOUT but that means we're all noisy and gross and it something I'd like to avoid. A better work around is to simply avoid this case. As mentioned before we have much more control over the parser then the producer, since there's only one parser (Test::Harness) and lots of producers including ad-hoc ones. We upgrade Test::Harness first, we can do that because old producer / new harness works fine. Then we upgrade Test::Builder and Test.pm to be new-style producers and have them depend on the new version of Test::Harness. Voila! STDERR is eliminated from new style producers. There's no heuristics or ambiguity necessary to determine what should and should not be displayed. Existing tests continue to display just as they did. We retain the ability to put undisplayed comments in the TAP stream. Comments, issues, problems?
Re: Eliminating STDERR without any disruption.
Michael G Schwern wrote: print TAP version 15\n; print 1..1\n; print # Information\n; print not ok 1\n; print ! Failure\n; I'd really not like to see meaningful punctuation. How about diag Failure\n. Or even levels of keywords debug/info/notice/warning/ err/crit/alert/emerg (stolen from syslog.h). Oh, and yaml if that idea lives.
Re: Eliminating STDERR without any disruption.
Yitzchak Scott-Thoennes wrote: Michael G Schwern wrote: print TAP version 15\n; print 1..1\n; print # Information\n; print not ok 1\n; print ! Failure\n; I'd really not like to see meaningful punctuation. I'm going to say its if a line starts with a ! just for discussion purposes. - Just for discussion purposes - I haven't given any thought to the real syntax yet. How about diag Failure\n. Or even levels of keywords debug/info/notice/warning/ err/crit/alert/emerg (stolen from syslog.h). That's an interesting idea. My worry is making it human readable. not ok 2 err Test failed in foo.t line 2 err got: foo err expected: bar ok 3 It doesn't leap out at you the way something like this does. not ok 2 !!! Test failed in foo.t line 2 !!! got: foo !!! expected: bar ok 3 Oh, and yaml if that idea lives. The TAP diagnostic syntax is largely orthogonal to this although it makes most uses of displayed diagnostics obsolete.
