Re: The most important TAP tasks.
* Michael G Schwern [EMAIL PROTECTED] [2007-03-09T18:17:57]
*) TAP diagnostic format
http://perl-qa.yi.org/index.php/TAP_diagnostic_syntax
There is no way to output diagnostics in TAP. The stuff Test::More spits
out to STDERR are unparsable comments indented for humans. Its not TAP.
This is the single biggest TAP deficiency.
Diagnostics need not be only for failures. There's nothing which says you
can't output them for passes.
So, under what circumstances do we display the diagnostics data regarding test
line/file? Something like either:
* if other diagnostics were given, we add these and display it
* if the test fails, we add these and display it
Assuming that diagnostics are passed as a hashref, and someone passes line or
file entries (or whatever the standard name is), presumably we croak. Or, we
make the standard name obnoxious enough that by using it you are clearly
stating that you know you're about to shoot your foot. ('TAP/testline')
--
rjbs
Re: The most important TAP tasks.
On 14 Mar 2007, at 16:04, Ricardo SIGNES wrote: * Michael G Schwern [EMAIL PROTECTED] [2007-03-09T18:17:57] *) TAP diagnostic format http://perl-qa.yi.org/index.php/TAP_diagnostic_syntax There is no way to output diagnostics in TAP. The stuff Test::More spits out to STDERR are unparsable comments indented for humans. Its not TAP. This is the single biggest TAP deficiency. Diagnostics need not be only for failures. There's nothing which says you can't output them for passes. [snip] Or for other reasons. E.g. Test::Class outputs some diagnostics in verbose mode that apply to a whole set of tests. Adrian
Re: The most important TAP tasks.
Ricardo SIGNES wrote:
* Michael G Schwern [EMAIL PROTECTED] [2007-03-09T18:17:57]
*) TAP diagnostic format
http://perl-qa.yi.org/index.php/TAP_diagnostic_syntax
There is no way to output diagnostics in TAP. The stuff Test::More spits
out to STDERR are unparsable comments indented for humans. Its not TAP.
This is the single biggest TAP deficiency.
Diagnostics need not be only for failures. There's nothing which says you
can't output them for passes.
So, under what circumstances do we display the diagnostics data regarding test
line/file? Something like either:
* if other diagnostics were given, we add these and display it
* if the test fails, we add these and display it
(Talking with rjbs in chat discovered that when he says display here he
means the TAP producer outputting diagnostics, not the harness displaying them)
The TAP producer should probably spit out anything that might be useful
always. This means the file, line and name on all tests.
The nice part is that decision is entirely up to the displayer and we give
them enough information to decide. A tty oriented display which has only one
output stream will probably want to go with display only diagnostics for
failing tests by default but provide switches to see them all. Something
webby like the HTML matrix used by Pugs can show all diagnostics as part of a
tooltip when the user hovers over a link.
One concern is that diagnostics for passing tests will clutter up the raw TAP
output for those who like to read the stream of ok and not ok.
Fortunately, with TAP::Parser, you can put together a filtered TAP reader
which can read in the raw TAP, strip out whatever you don't want to see, and
spit out the filtered TAP.
Assuming that diagnostics are passed as a hashref, and someone passes line
or
file entries (or whatever the standard name is), presumably we croak. Or,
we
make the standard name obnoxious enough that by using it you are clearly
stating that you know you're about to shoot your foot. ('TAP/testline')
Conflicting keywords... I'm inclined to say that the default case should be
that the standard keys should be the shorter and most readable since that's
going to be the majority case.
Guarding against a user passing in a file and line diagnostic with a different
meaning (maybe they mean the file they're currently reading from and the line
they're on) can be done by the TAP producer. That is, if they say:
is $line, foo, { name = the foo line, file = $file, line = $. };
Test::Builder can define file and line here to go under extensions so
the output might look like:
not ok 23 #---
got: bar
expected: foo
file: test.t
line: 68
name: the foo line
extensions:
file: my_file
line: 23
Re: The most important TAP tasks.
On 9 Mar 2007, at 23:17, Michael G Schwern wrote: *) A TAP version http://perl-qa.yi.org/index.php/TAP_version Before we add any TAP extensions we need a way to identify what version of TAP we're outputting. Discuss and implement. Done. The supported syntax is as per that page. In the absence of a version it defaults to 3. It's impossible to explicitly set a version 4. -- Andy Armstrong, hexten.net
Re: The most important TAP tasks.
Andy Armstrong wrote: On 9 Mar 2007, at 23:17, Michael G Schwern wrote: *) A TAP version http://perl-qa.yi.org/index.php/TAP_version Before we add any TAP extensions we need a way to identify what version of TAP we're outputting. Discuss and implement. Done. The supported syntax is as per that page. In the absence of a version it defaults to 3. It's impossible to explicitly set a version 4. I'm counting back through TAP changes and I'm up to version 6. I'll do a final sweep through the changes to t/TEST and Test::Harness to find out what else was added so we can solidify this for historical purposes. Or we can just declare the version that introduces TAP version to be 10 and backfill the rest later.
Re: The most important TAP tasks.
On 10 Mar 2007, at 22:57, Michael G Schwern wrote: I'm counting back through TAP changes and I'm up to version 6. I'll do a final sweep through the changes to t/TEST and Test::Harness to find out what else was added so we can solidify this for historical purposes. Or we can just declare the version that introduces TAP version to be 10 and backfill the rest later. It'd be nice to avoid a gap unless it's a major hassle. -- Andy Armstrong, hexten.net
The most important TAP tasks.
There's a lot of noise and very little signal about TAP lately. Absolutely no real TAP extensions have been implemented. The TAP document remains in exactly the same form it was 9 months ago. http://search.cpan.org/~petdance/TAP-1.00/TAP.pm We're spinning our wheels and going nowhere. I'm going to declare the most important TAP tasks and until they're done I really don't want to hear about anything else. If you have a bright TAP idea archive it on the wiki until later. Here they are, in no particular order. *) TAP::Parser / TAP::Harness / Test::Harness 3.0 http://www.hexten.net/mailman/listinfo/tapx-dev We need a flexible TAP parser so folks can do all the things we dream we might be able to do with TAP. TAP::Parser is it. It needs work. There's a mailing list and repo. Andy Armstrong, Ovid and Eric Willhelm are doing all the work so far. Help them. *) Improve the TAP standards document http://search.cpan.org/~petdance/TAP-1.00/TAP.pm Andy Lester wrote a draft of the TAP standard. There have been no releases since. There's mistakes, typos, ommissions, you name it. Work on it. Put it on a POD wiki to allow easier collaborative editing. Talk with Ovid about where the ambiguities are, he's had to work them out for TAP::Parser. *) A TAP version http://perl-qa.yi.org/index.php/TAP_version Before we add any TAP extensions we need a way to identify what version of TAP we're outputting. Discuss and implement. *) TAP diagnostic format http://perl-qa.yi.org/index.php/TAP_diagnostic_syntax There is no way to output diagnostics in TAP. The stuff Test::More spits out to STDERR are unparsable comments indented for humans. Its not TAP. This is the single biggest TAP deficiency. There is a proposal which solves several problems at once. Discuss and implement a patch for Test::Builder to allow authors to output explicit diagnostics.
Re: The most important TAP tasks.
On Mar 9, 2007, at 5:17 PM, Michael G Schwern wrote: since. There's mistakes, typos, ommissions, you name it. Work on it. Put it on a POD wiki to allow easier collaborative editing. Talk with Ovid about where the ambiguities are, he's had to work them out for TAP::Parser. Better yet, I can make it available in public Subversion so that you can edit it, if you're so inclined. -- Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
