Ok, here's what I've decided about the changes to the way Test::Builder
handles plans. None of this requires TAP or Test::Harness changes.
* The restriction on requiring a plan before running a test will be lifted.
use Test::More;
pass();
plan tests => 2;
pass();
The above will be ok. It will result in the following legal TAP.
ok 1
ok 2
1..2
This allows more flexibility in planning when the number of tests is
calculated rather than hard coded.
By popular (or at least long and loud ;) request, a Test::Builder switch will
be added to turn the plan requirement back on.
* An "until done" plan will be added.
The oft requested, safer cousin of "no_plan" will be added. This is like
no_plan in that it does not require a fixed test count, but rather that
testing proceed until an "I'm done testing" marker is reached.
What that marker will be called, and what this sort of plan will be called, is
still up in the air. But it might look something like this:
use Test::More plan 'until_done';
...your tests...
done_testing();
If done_testing() is not seen, the test is a failure. More testing after
done_testing() is also a failure. I'm not certain how these will be expressed
as TAP, but there are two basic options.
First is to use the plan to express that something is missing or too many
tests were run.
# Two successful tests and done_testing()
ok 1
ok 2
1..2
# Two successful tests, but no done_testing() seen.
ok 1
ok 2
1..3
# Two successful tests, done_testing(), then an unexpected third test
ok 1
ok 2
ok 3
1..2
The second option is to express done_testing() as it's own test.
# Two successful tests and the done_testing() seen.
ok 1
ok 2
ok 3 - "done testing" seen
1..3
# Two successful tests, but no done_testing() seen.
ok 1
ok 2
not ok 3 - "done testing" seen
1..3
# Two successful tests, done_testing(), then an unexpected third
ok 1
ok 2
ok 3 - "done testing" seen
ok 4
1..3
After writing it out, I like the latter. By expressing "done testing" as a
test it makes it clear what's going on.
There are issues surrounding testing modules which add tests in the END phase
like Test::NoWarnings. These tests would come after the test is declared
done. What will probably be needed is a "wait, I'm not really done" switch to
allow things like this to happen after the user has already declared they're
done.
Unless someone can think of a more seemless way to do it?
* "skip_all" must still come first.
The following will be an error.
use Test::More;
pass("First test");
plan skip_all => "Because";
pass("Another test");
Test::Builder *could* guess at your meaning and skip the remaining tests, but
I don't like the ambiguity.
Fortunately, there will be an easier way to do it...
* skip_rest() is in.
The oft requested skip_rest() will be available as both a Test::More function
and Test::Builder method. The tipping point was how simple it makes the
skip_all decision code. Right now you have to write this:
use Test::More;
if( $^O eq 'SomeOS' ) {
plan skip_all => "Tests don't apply to SomeOS":
}
else {
plan tests => 3;
}
It's ugly and verbose and the decision to skip all has to come before you do
any testing at all. Testing which might effect your decision to skip.
skip_rest() makes this much simpler.
use Test::More tests => 3;
skip_rest("Tests don't apply to SomeOS") if $^O eq 'SomeOS';
The output would look like this:
1..3
ok 1 # SKIP Tests don't apply to SomeOS
ok 2 # SKIP Tests don't apply to SomeOS
ok 3 # SKIP Tests don't apply to SomeOS
The downside is a lot more TAP is output, fortunately a good TAP harness will
see the repeated SKIP reason and compress that down into one message. The
upside is the harness knows how many tests were skipped.
* A way to change the plan will be added.
Probably at the Test::Builder level at first to allow other Test module
authors to play with it while I work out the Test::More interface, if any.
This will finally allow users to experiment with block level plans and
building up the plan a bit at a time. There will likely be a "change the
plan" and "increment the plan" methods.
Due to the current limitations of TAP it will require that the plan comes at
the end of the TAP. Fortunately, if TAP becomes more flexible this can change.
To be clear, this will be different from the normal way to declare a plan to
catch double plan mistakes. That is, the following will still be an error.
plan tests => 5; # 1..2..5!
plan tests => 3; # No, 3 sir!
--
You know what the chain of command is? It's the chain I go get and beat you
with 'til you understand who's in ruttin' command here.
-- Jayne Cobb, "Firefly"
