TAP for skipping all tests
http://search.cpan.org/dist/Test-Harness/lib/Test/Harness/TAP.pod implies that if I want to skip all tests, I need something like this format: 1..0 # SKIP why not? I've also noticed that all of the Test::Harness tests which exercise this behavior use '1..0', even though this is only implied, not required. It seems to me that specifying how many tests you have should be fine: 1..13 # SKIP 'cuz thirteen is scary, dude Is there a reason for the '1..0' syntax or is that merely a convention? Cheers, Ovid -- Buy the book -- http://www.oreilly.com/catalog/perlhks/ Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/
Re: TAP for skipping all tests
On Wed, Jan 24, 2007 at 04:43:52AM -0800, Ovid wrote: http://search.cpan.org/dist/Test-Harness/lib/Test/Harness/TAP.pod implies that if I want to skip all tests, I need something like this format: 1..0 # SKIP why not? I've also noticed that all of the Test::Harness tests which exercise this behavior use '1..0', even though this is only implied, not required. It seems to me that specifying how many tests you have should be fine: 1..13 # SKIP 'cuz thirteen is scary, dude Is there a reason for the '1..0' syntax or is that merely a convention? My guess is that the number of test run may depend on things determined after the point you decided to skip all the tests. Steve Peters [EMAIL PROTECTED]
Re: TAP for skipping all tests
--- Steve Peters [EMAIL PROTECTED] wrote: My guess is that the number of test run may depend on things determined after the point you decided to skip all the tests. Sounds reasonable, but I also think it should be optional. Cheers, Ovid -- Buy the book -- http://www.oreilly.com/catalog/perlhks/ Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/
Test::More and Fatal
Has anyone ever used Test::More and Fatal together? I have a test script, where each test builds upon the work of the previous step (it's part of the Subversion Perl bindings test suite, and it checks out files, makes changes to them, commits them, and so on). If any of these tests fails I need to abort the whole .t file (but not necessarily the entire test run, so BAIL_OUT isn't appropriate). I thought something like: use Test::More; use Fatal qw(ok is isa_ok open print close ...); should do the trick. Anyone used this approach before, or done anything similar? N
Re: Test::More and Fatal
On Jan 24, 2007, at 9:42 AM, Nik Clayton wrote:
Has anyone ever used Test::More and Fatal together?
I have a test script, where each test builds upon the work of the
previous step (it's part of the Subversion Perl bindings test
suite, and it checks out files, makes changes to them, commits
them, and so on).
If any of these tests fails I need to abort the whole .t file (but
not necessarily the entire test run, so BAIL_OUT isn't appropriate).
I thought something like:
use Test::More;
use Fatal qw(ok is isa_ok open print close ...);
should do the trick. Anyone used this approach before, or done
anything similar?
N
I don't like that very much. The implicit die() on test functions
will probably confuse subsequent readers of the code. Furthermore, I
cannot believe that ALL of your tests are critical. It's better,
IMHO, to explicitly mention the critical tests and abort explicitly
on failure. For example, I do the following (in your case, change
BAIL_OUT to die):
use Test::More tests = 348;
sub bail { BAIL_OUT('Failed a critical test'); }
my $mech = Test::WWW::Mechanize-new();
$mech-get_ok('http://localhost/', 'get home page') || bail;
$mech-content_contains('Home');
...
In this example, failing to fetch HTML is clearly bad news for all
subsequent tests and the C || bail is good self-documentation.
Other tests are less critical, so should not fail.
Chris
--
Chris Dolan, Software Developer, http://www.chrisdolan.net/
Public key: http://www.chrisdolan.net/public.key
vCard: http://www.chrisdolan.net/ChrisDolan.vcf
Re: Test::More and Fatal
Chris Dolan wrote: I don't like that very much. The implicit die() on test functions will probably confuse subsequent readers of the code. # Cause the listed functions to die on error use Fatal qw(...); Furthermore, I cannot believe that ALL of your tests are critical. 150 or so out of 158. So the ... or die redundant error message: $!; gets quite old quite fast. N
Re: Test::More and Fatal
On Jan 24, 2007, at 1:25 PM, Nik Clayton wrote:
Chris Dolan wrote:
I don't like that very much. The implicit die() on test functions
will probably confuse subsequent readers of the code.
# Cause the listed functions to die on error
use Fatal qw(...);
Furthermore, I cannot believe that ALL of your tests are critical.
150 or so out of 158. So the
... or die redundant error message: $!;
gets quite old quite fast.
N
Wow, 150/158? That's a high fraction. I still think the following
is somewhat more readable, but you are clearly in a different space
from the testing I usually do, so perhaps MHO doesn't apply.
sub bail { die redundant error message: $!; }
then 150 instances of ... or bail; One advantage of that approach
is that you can add cleanup code to bail() if needed, although you
could probably do that in an END{} block instead.
Chris
--
Chris Dolan, Software Developer, http://www.chrisdolan.net/
Public key: http://www.chrisdolan.net/public.key
vCard: http://www.chrisdolan.net/ChrisDolan.vcf
Re: TAP for skipping all tests
Ovid wrote: http://search.cpan.org/dist/Test-Harness/lib/Test/Harness/TAP.pod implies that if I want to skip all tests, I need something like this format: 1..0 # SKIP why not? I've also noticed that all of the Test::Harness tests which exercise this behavior use '1..0', even though this is only implied, not required. That's a mistake in the TAP document. Its the # SKIP which is actually optional. It seems to me that specifying how many tests you have should be fine: 1..13 # SKIP 'cuz thirteen is scary, dude Is there a reason for the '1..0' syntax or is that merely a convention? Like most of TAP, its convention turned standard. Conceptually I guess the idea was to declare that you're going to run 0 tests, which is exactly what happens. The # SKIP directive probably came in after. More to the point, 1..13 # SKIP will break Test::Harness. Doesn't anyone try these things? $ cat ~/tmp/skip_all.t #!/usr/bin/perl -w print OUT; 1..2 # skip because OUT $ perl -MTest::Harness -wle 'runtests shift' ~/tmp/skip_all.t /Users/schwern/tmp/skip_allFAILED before any test output arrived Failed Test Stat Wstat Total Fail List of Failed --- /Users/schwern/tmp/skip_all.t ?? ?? ?? Failed 1/1 test scripts. 2/2 subtests failed. Files=1, Tests=2, 0 wallclock secs ( 0.00 cusr + 0.01 csys = 0.01 CPU) Failed 1/1 test programs. 2/2 subtests failed. What appears to be going on here is Test::Harness is expecting you to run 2 tests and does not see them. It does not account for the fact that you said to skip all. The passing flag is never set because that defaults to false and is only set when it sees a test. In contrast if we run one test, we get a failure because it expects 2. $ perl -MTest::Harness -wle 'runtests shift' ~/tmp/skip_all.t /Users/schwern/tmp/skip_allFAILED test 2 Failed 1/2 tests, 50.00% okay Failed Test Stat Wstat Total Fail List of Failed --- /Users/schwern/tmp/skip_all.t21 2 Failed 1/1 test scripts. 1/2 subtests failed. Files=1, Tests=2, 0 wallclock secs ( 0.00 cusr + 0.00 csys = 0.00 CPU) Failed 1/1 test programs. 1/2 subtests failed. And if we run the correct amount it passes, not skips. $ perl -MTest::Harness -wle 'runtests shift' ~/tmp/skip_all.t /Users/schwern/tmp/skip_allok All tests successful. Files=1, Tests=2, 0 wallclock secs ( 0.00 cusr + 0.00 csys = 0.00 CPU) So, in short, it will break Test::Harness and probably others.
Re: A Modest Request
James E Keenan wrote: What with all the activity on this list in the last week (TAPx::Parser about to morph into Test::Harness), it's all been more than I can keep up with. I would like to suggest that one or more or the hackerati who are working on all these revisions to our core testing functionality write an article about what's happening. Such an article might appear in draft form on use.perl.org, and then get published on www.perl.com. Or hey, we've got that wiki to write down all these notes that nobody uses. http://perl-qa.yi.org/index.php/Special:Recentchanges
Re: Test::More and Fatal
On 1/24/07, Nik Clayton [EMAIL PROTECTED] wrote: Has anyone ever used Test::More and Fatal together? I have a test script, where each test builds upon the work of the previous step (it's part of the Subversion Perl bindings test suite, and it checks out files, makes changes to them, commits them, and so on). If any of these tests fails I need to abort the whole .t file (but not necessarily the entire test run, so BAIL_OUT isn't appropriate). I thought something like: use Test::More; use Fatal qw(ok is isa_ok open print close ...); You can't use Fatal on print(). Josh
Re: A Modest Request
On 1/23/07, James E Keenan [EMAIL PROTECTED] wrote: I would like to suggest that one or more or the hackerati who are working on all these revisions to our core testing functionality write an article about what's happening. I second that emotion. -- Shawn Boyette [EMAIL PROTECTED]
Re: Test::More and Fatal
* Nik Clayton [EMAIL PROTECTED] [2007-01-24 16:45]:
Anyone used this approach before, or done anything similar?
I haven’t, but it seems perfectly reasonable if you can live with
Fatal’s annoying stack traces.
Another option would be something like
sub ensure { $_[ 0 ] or die broken promises }
# ...
ensure isa_ok $foo, 'Frobnitzer';
ensure is $foo-bar, $bar-bar, Barbar is foobar;
which is still repetitive, but IMHO much less of a pain than
trailing `or bail` clauses, particularly as it’s miles easier
to add and remove en masse using a good editor.
Regards,
--
Aristotle Pagaltzis // http://plasmasturm.org/
functional literate programming
Hey folks, Konobi pointed me at an interesting talk about writing customer readable tests that were also executable. The video is from the Google London Test Automation Conference. (The code is in some ugly language... ;) http://video.google.com/videoplay?docid=1505469784301926538q=automated+testing+conference+literate+java+testing And it starts: assert_that page has title Google, has title YAPH; http://rafb.net/p/PeiWl968.html Cheers, Luke
