Re: Perl 6 and Test.pm's skip() function
# from Ovid
# on Wednesday 21 January 2009 10:47:
multi sub skip() is export() { proclaim(1, # SKIP); }
multi sub skip($desc) is export() { proclaim(1, # SKIP ~
$desc); } multi sub skip($count, $desc) is export() {
for 1..$count {
proclaim(1, # SKIP ~ $desc);
}
}
Passing a number of tests to skip() is an untested failure waiting to
happen.
Perhaps it is still no easy task to count the tests in a block, but
making humans count them is just asking for errors.
--Eric
--
Beware of bugs in the above code; I have only proved it correct, not
tried it.
--Donald Knuth
---
http://scratchcomputing.com
---
Re: Perl 6 and Test.pm's skip() function
- Original Message From: Eric Wilhelm scratchcomput...@gmail.com Passing a number of tests to skip() is an untested failure waiting to happen. Perhaps it is still no easy task to count the tests in a block, but making humans count them is just asking for errors. We already have 'plan tests = 33'. People count tests all the time. It's a fact of life. If the programmer gets the skip count wrong and they have a plan, the tests will fail. If they have a plan and the programmer omits the skip count, how can we possibly know how many tests to skip in order to match the plan? The programmer still has to count if the programmer wants a plan. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Tech blog- http://use.perl.org/~Ovid/journal/ Twitter - http://twitter.com/OvidPerl Official Perl 6 Wiki - http://www.perlfoundation.org/perl6
Re: Perl 6 and Test.pm's skip() function
# from Ovid
# on Thursday 22 January 2009 02:00:
The programmer still has to count if the programmer wants a plan.
I'm not sure anybody *wants* a plan. A way to ensure that every test
ran or accurate progress reporting, yes. It seems to me that some are
just willing to suffer counting their tests to achieve that.
Pretend for a moment that the number of tests could automatically be
counted by the interpreter (e.g. at the parse/compile stage.) Would we
still have the manual counting?
I suppose if($whatever) { some_test() } else { alternate_test()}
would complicate automatic counting. But, you have to go down one
branch. The only impossible spot is when tests are inside e.g. a
runtime dispatched method, no? (And, given the procedural paradigm,
that seems to be an odd case.)
--Eric
--
Any sufficiently advanced incompetence is indistinguishable from malice.
--The Napoleon-Clarke Law
---
http://scratchcomputing.com
---
Re: Perl 6 and Test.pm's skip() function
- Original Message From: Eric Wilhelm scratchcomput...@gmail.com I'm not sure anybody *wants* a plan. snip Lots of people want plans. Lots of people don't want plans. That's not an argument I expect anybody is going to *win* (even if they're right). This has been argued to death. Many times. Over and over. You can argue it again now, but unless there's some unusually compelling point which makes the Perl-QA community collectively slap their foreheads and say duh! Why have we overlooked this for so many years?, then I'm not going to argue the point. Of course, you might just have that compelling point we've all missed (and given that testing in Perl is still in its infancy in many ways, I wouldn't be at all surprised if you do). In short, if someone doesn't want a plan, just use 'no_plan' or Test::Most's all_done function :) Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Tech blog- http://use.perl.org/~Ovid/journal/ Twitter - http://twitter.com/OvidPerl Official Perl 6 Wiki - http://www.perlfoundation.org/perl6
Re: Perl 6 and Test.pm's skip() function
# from Michael Peters
# on Thursday 22 January 2009 09:55:
I suppose if($whatever) { some_test() } else { alternate_test()}
would complicate automatic counting. But, you have to go down one
branch.
But there's no protection that one branch doesn't have a different
number of tests than the other.
If you can see all of the function calls at compile-time, you can count
them. If one branch has a different count than the other, that's an
error unless the shorter branch contains a skip statement?
The only impossible spot is when tests are inside e.g. a
runtime dispatched method, no? (And, given the procedural paradigm,
that seems to be an odd case.)
No, that's not odd at all. Any data driven testing system will be that
way. Tests are run based on some config file (I know Ovid uses YAML
for something like this), etc.
That still doesn't imply that we can't somehow count the number of tests
with a computer instead of relying on humans to screw it up. If some
combination of static analysis and early runtime can come up with a
count, then it becomes possible to automatically plan without counting.
So, whatever constructs make that impossible might be worth pondering.
--Eric
--
...the bourgeoisie were hated from both ends: by the proles, because
they had all the money, and by the intelligentsia, because of their
tendency to spend it on lawn ornaments.
--Neal Stephenson
---
http://scratchcomputing.com
---
Re: Perl 6 and Test.pm's skip() function
On Jan 22, 2009, at 11:06 AM, Eric Wilhelm wrote: That still doesn't imply that we can't somehow count the number of tests with a computer instead of relying on humans to screw it up. If some combination of static analysis and early runtime can come up with a count, then it becomes possible to automatically plan without counting. There will be loops with tests in them, and the number of iterations of the loop will be independent of the code in the test script, making it impossible to actually count the number of tests with a computer until the tests have actually been run. Which is how no_plan works. Best, David
Re: Perl 6 and Test.pm's skip() function
On Jan 22, 2009, at 1:27 PM, Eric Wilhelm wrote: I personally use no_plan only because I can't be bothered to manually count things and don't want to assume that the number of tests run on *my* computer is somehow a universal constant. I'm glad you find no_plan useful. Many others do as well. I use plans consistently, and have written a policy in Perl::Critic::Bangs that checks for no_plan (I think that's where I put it) and flags it as an error. The assumptions you're talking about are not assumptions. They are well-worn tenets of TAP and Perl testing that many people use and rely on. xoa -- Andy Lester = a...@petdance.com = www.petdance.com = AIM:petdance
Re: Perl 6 and Test.pm's skip() function
# from David E. Wheeler # on Thursday 22 January 2009 11:15: That still doesn't imply that we can't somehow count the number of tests with a computer instead of relying on humans to screw it up. If some combination of static analysis and early runtime can come up with a count, then it becomes possible to automatically plan without counting. There will be loops with tests in them, and the number of iterations of the loop will be independent of the code in the test script, making it impossible to actually count the number of tests with a computer until the tests have actually been run. Which is how no_plan works. This is true, but Perl 6 is supposed to be a lovely new language with perhaps the ability to define a function which runs a loop over a block given an iterator, right? So, if such a function were to treat all of the blocks contained tests as subtests, one could call this '1' and merrily carry on with the static counting before having executed anything. --Eric -- Atavism n: The recurrence of any peculiarity or disease of an ancestor in a subsequent generation, usually due to genetic recombination. --- http://scratchcomputing.com ---
Re: Perl 6 and Test.pm's skip() function
# from Andy Lester # on Thursday 22 January 2009 11:31: I personally use no_plan only because I can't be bothered to manually count things and don't want to assume that the number of tests run on *my* computer is somehow a universal constant. I'm glad you find no_plan useful. Many others do as well. Perhaps I'm being unclear. I do not find either 'no_plan' or 'plan' to be useful in their current state. --Eric -- If the collapse of the Berlin Wall had taught us anything, it was that socialism alone was not a sustainable economic model. --Robert Young --- http://scratchcomputing.com ---
Re: Perl 6 and Test.pm's skip() function
On Jan 22, 2009, at 1:34 PM, Eric Wilhelm wrote: I personally use no_plan only because I can't be bothered to manually count things and don't want to assume that the number of tests run on *my* computer is somehow a universal constant. I'm glad you find no_plan useful. Many others do as well. Perhaps I'm being unclear. I do not find either 'no_plan' or 'plan' to be useful in their current state. Yes, but many others do. -- Andy Lester = a...@petdance.com = www.petdance.com = AIM:petdance
Re: numeric plans - feature or symptom?
# from Andy Lester # on Thursday 22 January 2009 11:35: Perhaps I'm being unclear. I do not find either 'no_plan' or 'plan' to be useful in their current state. Yes, but many others do. Well, are we just accepting limitations and refusing to dream? ewilhelm the computer must use a $number. does the *human* actually want to use a $number? Theory This human does. soh i do. i always want to know if more tests than i expect are running Theory Because the computer can get it wrong. ewilhelm no, the computer doesn't get things wrong Theory It does because of loops. ... ewilhelm ok, so everybody who likes to manually set a static number just doesn't *trust* the computer? Theory Correct. If we were to pretend that tests could be declared in such a way that a plan could be derived from static analysis and perhaps a bit of startup computation, would there be any reason to do otherwise? If the answer is yes, please explain how the reason does not cite a limitation of the implementation. Thanks, Eric -- The only thing that could save UNIX at this late date would be a new $30 shareware version that runs on an unexpanded Commodore 64. --Don Lancaster (1991) --- http://scratchcomputing.com ---
Re: numeric plans - feature or symptom?
- Original Message From: Eric Wilhelm scratchcomput...@gmail.com Perhaps I'm being unclear. I do not find either 'no_plan' or 'plan' to be useful in their current state. Yes, but many others do. Well, are we just accepting limitations and refusing to dream? Show us the code. No, don't show us an ideal API; show us the real, actual code. Even a simple proof of concept would be fine. Seriously. Show us the code. Saying what if ... is a wonderful thought exercise and I'm VERY sympathetic to your point of view (my objections are largely stemming from practicality), but I don't believe that the if can be realized. So show us the code. Seriously. I would love to see something come even close to working. I expect I would learn a lot from it. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Tech blog- http://use.perl.org/~Ovid/journal/ Twitter - http://twitter.com/OvidPerl Official Perl 6 Wiki - http://www.perlfoundation.org/perl6
Re: numeric plans - feature or symptom?
2009/1/22 Eric Wilhelm scratchcomput...@gmail.com: # from Andy Lester # on Thursday 22 January 2009 11:35: Perhaps I'm being unclear. I do not find either 'no_plan' or 'plan' to be useful in their current state. Yes, but many others do. Well, are we just accepting limitations and refusing to dream? ewilhelm the computer must use a $number. does the *human* actually want to use a $number? Theory This human does. soh i do. i always want to know if more tests than i expect are running Theory Because the computer can get it wrong. ewilhelm no, the computer doesn't get things wrong Theory It does because of loops. ... ewilhelm ok, so everybody who likes to manually set a static number just doesn't *trust* the computer? Theory Correct. If we were to pretend that tests could be declared in such a way that a plan could be derived from static analysis and perhaps a bit of startup computation, would there be any reason to do otherwise? If the answer is yes, please explain how the reason does not cite a limitation of the implementation. Assuming the static analysis was correct, it would always produce the correct number thus would be equivalent to no_plan. For me, the purpose of the plan is not to detect failures that cause early exits - it can do that but the test harness also looks at exit codes etc. The purpose of the plan is to ensure that the test I just added really do run. I am not terribly happy with plans as they currently exist, loops make them a bit of a pain, I'd really like to be able to have blocks with subplans etc to ease that but I'm not going to start on that again. The point is that plans are a checksum, making sure that the test script you wrote tests as many things as you think it does. Having spent most of the last 5 years using python and pyunit I have missed plans and have once or twice been bitten by tests that weren't actually being called and a plan would have saved me, F Thanks, Eric -- The only thing that could save UNIX at this late date would be a new $30 shareware version that runs on an unexpanded Commodore 64. --Don Lancaster (1991) --- http://scratchcomputing.com ---
Re: numeric plans - feature or symptom?
On Jan 22, 2009, at 1:08 PM, Fergal Daly wrote: Assuming the static analysis was correct, it would always produce the correct number thus would be equivalent to no_plan. For me, the purpose of the plan is not to detect failures that cause early exits - it can do that but the test harness also looks at exit codes etc. The purpose of the plan is to ensure that the test I just added really do run. Great points, Fergal, thanks. I am not terribly happy with plans as they currently exist, loops make them a bit of a pain, I'd really like to be able to have blocks with subplans etc to ease that but I'm not going to start on that again. Test::Class helps a lot with this. Best, David
Re: numeric plans - feature or symptom?
# from Ovid
# on Thursday 22 January 2009 13:01:
Show us the code.
No, don't show us an ideal API; show us the real, actual code. Even
a simple proof of concept would be fine. Seriously. Show us the
code.
I'm not sure whether this would qualify as either an ideal API or real
actual serious code. I happen to be working on some other real actual
seriously code today, but anyhow:
sub test () {
my $subref = shift;
local $Something::something;
$subref-();
finished();
}
test {
...
ok(1);
ok(1);
...
};
test {
...
ok(1);
};
Or thereabouts. The business of skipping, todoing, counting, planning,
and ensuring that all tests actually run is going to involve various
details and possibly even get into the limitations of TAP -- but you
now have every chunk of tests setup for later/encapsulated evaluation.
One way to punt toward that would be to treat each block as a single
test ala is_deeply(). A similar approach would handle loops.
This perhaps loses some granularity within each block. It also assumes
perl 5 limitations. Possibly neither issue sticks in perl 6.
One could even simply stack the blocks and finish the test file with
run_tests_now(), which would essentially start the whole planning
process.
Other details of the implementation will be up to the implementor. If
that's me, I suppose I should have learned by now not to bother making
a suggestion.
--Eric
--
Everything should be made as simple as possible, but no simpler.
--Albert Einstein
---
http://scratchcomputing.com
---
Re: numeric plans - feature or symptom?
- Original Message
From: Eric Wilhelm scratchcomput...@gmail.com
Other details of the implementation will be up to the implementor. If
that's me, I suppose I should have learned by now not to bother making
a suggestion.
Eric, I know I come across as rather brusque at times and I apologize; it's
nothing personal! I really do want to understand where you're coming from. The
problem I have is that I've heard this argument many, many times in the past
and it's always gone nowhere. In this case, I simply don't understand how your
code solves the concerns people have. Sometimes I'm just slow in understanding
what someone is saying and I
think that sometimes your mind works fast enough that you jump from
point to point when some of us (me) need to crawl.
For example, with your code (as I understand it):
test {
my $manager = Feed::Manager-new($some_uri);
foreach my $resource ($manager-resources) {
ok my $result = $manager-get($resource), $resource should work;
}
}
Imagine that the Feed::Manager connects to an outside resource I don't have
control over and one day they add a new resource. I now have a new test and
need to know that something has changed. I don't see how your code solves
that. One can argue that this is a stupid way to test, but I argue that I
don't want to tell people how to test. However, this seems so completely
orthogonal to your point of view that it's clear I'm missing something that
you've already understood :(
Cheers,
Ovid
--
Buy the book - http://www.oreilly.com/catalog/perlhks/
Tech blog- http://use.perl.org/~Ovid/journal/
Twitter - http://twitter.com/OvidPerl
Official Perl 6 Wiki - http://www.perlfoundation.org/perl6
Re: numeric plans - feature or symptom?
On Jan 22, 2009, at 2:24 PM, Eric Wilhelm wrote: Or thereabouts. The business of skipping, todoing, counting, planning, and ensuring that all tests actually run is going to involve various details and possibly even get into the limitations of TAP -- but you now have every chunk of tests setup for later/encapsulated evaluation. One way to punt toward that would be to treat each block as a single test ala is_deeply(). A similar approach would handle loops. You would need to modify TAP to support sub-tests. Or you can just use Test::Class or Test::Block to divide your tests up into blocks. This perhaps loses some granularity within each block. It also assumes perl 5 limitations. Possibly neither issue sticks in perl 6. Yeah, I want that granularity. That's part of the point of the planned count of tests. Best, David
Re: Perl 6 and Test.pm's skip() function
* Eric Wilhelm scratchcomput...@gmail.com [2009-01-22 18:55]: Pretend for a moment that the number of tests could automatically be counted by the interpreter (e.g. at the parse/compile stage.) There’s no need to pretend. Either you can tell us how to solve the halting problem and then it’s possible, or you can’t and then it’s not. Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Re: Perl 6 and Test.pm's skip() function
* David E. Wheeler da...@kineticode.com [2009-01-22 20:20]: There will be loops with tests in them, and the number of iterations of the loop will be independent of the code in the test script, making it impossible to actually count the number of tests with a computer until the tests have actually been run. Which is how no_plan works. The nice thing about data-driven tests is that in most cases the test program can programmatically derive the number of tests from the test data so it can set up a plan without the programmer having to count. Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Re: Perl 6 and Test.pm's skip() function
* Eric Wilhelm scratchcomput...@gmail.com [2009-01-22 18:55]: I'm not sure anybody *wants* a plan. I do. A way to ensure that every test ran or accurate progress reporting, yes. I also want to be sure that no unexpected extra tests ran. It seems to me that some are just willing to suffer counting their tests to achieve that. I am willing to suffer counting tests in order to have my tests counted, yes. Although most of the time I find ways not to have to do that, and still be able to declare a plan. Data-driven tests work wonders for this. The rest of the time I use a variety of tricks to reduce the pain, such as this one: http://perl-qa.hexten.net/wiki/index.php/TestFAQ#How_do_I_update_the_plan_as_I_go.3F Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Re: numeric plans - feature or symptom?
Ovid wrote:
For example, with your code (as I understand it):
test {
my $manager = Feed::Manager-new($some_uri);
foreach my $resource ($manager-resources) {
ok my $result = $manager-get($resource), $resource should
work;
}
}
Imagine that the Feed::Manager connects to an outside resource I don't
have control over and one day they add a new resource. I now have a
new test and need to know that something has changed. I don't see how
your code solves that. One can argue that this is a stupid way to
test, but I argue that I don't want to tell people how to test.
However, this seems so completely orthogonal to your point of view
that it's clear I'm missing something that you've already understood
:(
The point Mr. Wilhelm has brought up sounds very similar to something
I have been trying (and failing so far) to achieve lately. If it were
done for p6 that'd be great but I think its possible for p5 as well.
The goal is simple: be able to easily alter the plan at any time.
Some nice things become possible if this goal is met.
1. In an xUnit style testing setup, the methods can be decorated with
their plan contribution. The plan can then be built and set before
the TestRunner kicks off the run.
This can be done right now and is done by modules like Test::Class.
The problem comes when any method declares no_plan. When that
happens then the whole plan is shot and has to be set to no_plan.
Thus a major gripe of mine with Test::Class. To me, having a plan
set to no_plan is a sorry exit strategy when the current, very
limited, way of planning fails.
If the plan was easy to alter and the method that declared no_plan
up front can determine its plan contribution at runtime then the
plan can be recalculated and set to a real value.
2. The above loop example would work just fine. The code would just
alter its plan contribution as soon as it could.
To make this happen in p5-land Test::Builder has to support true
plan at end. It would be nice if Test::Builder defaulted to sending
the plan at the end cause then it would allow this plan altering idea
to work if someone sent the plan out at the beginning of the TAP.
Or allow the second TAP plan, if one is sent, to override the first
one - I don't know. It would also be nice if it had some sort of
plan accounting infrastructure to make it as easy as possible to
keep track of the plan and sub-plans and their histories. This way
the test could just give a new count or a delta and the plan could
recompute itself.
Right now the plan is effectively a one-shot. You either know it up
front or you don't. And if you don't, you lose virtually all the
benefits of planning. Why not allow the plan to reflect what the test
expects at any given time?
The module I've been working on this with is Test::Able.
Its here: http://github.com/jdv/test-able/tree/master
Thanks,
jdv
Let us reflect on the Halting Problem (was Re: Perl 6 and Test.pm's skip() function)
Eric Wilhelm wrote:
The only impossible spot is when tests are inside e.g. a
runtime dispatched method, no? (And, given the procedural paradigm,
that seems to be an odd case.)
No, that's not odd at all. Any data driven testing system will be that
way. Tests are run based on some config file (I know Ovid uses YAML
for something like this), etc.
That still doesn't imply that we can't somehow count the number of tests
with a computer instead of relying on humans to screw it up. If some
combination of static analysis and early runtime can come up with a
count, then it becomes possible to automatically plan without counting.
So, whatever constructs make that impossible might be worth pondering.
With apologies to Godwin: As a programming discussion grows longer, the
probability of someone thinking they can solve the Halting Problem approaches
one.
Why can't the computer count the number of tests? Because it CAN NOT KNOW how
many tests there are. Here's why:
pass() for 1..rand 5;
Here is a more realistic example.
my @things = $fh;
for my $thing (@things) {
is $thing, something;
}
Here's another:
$object-run_tests;
Because, in Perl and other languages, until you run it you can't know what
class $object is going to be, or what its inheritance tree will look like, and
once you do figure out which run_tests() will run (if any) you're back to the
problem of figuring out how many tests are in run_tests().
If we were using a mathematically provable functional programming language,
say Haskell, there are some cases where you can do static analysis and figure
out how many tests are going to run. If you got really clever I'll bet you
could do it for most cases. But when a test involves any randomness or
outside input it is impossible to know. The best you can do is know when you
can not know.
For those who don't believe me, feel free to push this rock up and down the
hill of your choice until enlightenment or death occurs.
For those of you in the I can write something good enough camp, feel free to
use Shlomi's Test::Count.
http://search.cpan.org/perldoc?Test::Count
--
164. There is no such thing as a were-virgin.
-- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
http://skippyslist.com/list/
Re: Test::Builder plan at end
Justin DeVuyst wrote:
Hello,
I was told this might be a place to get information about
upcoming Test::Builder changes.
I'd like to know if and when Test::Builder will officially
support true plan at end. The current version of
Test::Builder reports 1..$seen_tests instead of
1..$expected_tests.
I've had to resort to a hack like this in one of my modules
to get around the deficiency:
my $original_sub = \Test::Builder::_ending;
*Test::Builder::_ending = sub {
my $builder = shift;
$builder-expected_tests( $builder-{Expected_Tests} );
$builder-no_header( 1 );
return $self_builder-$original_sub( @_ );
};
Which I would like to dump ASAP.
Yes, this is the deferred plan.
http://code.google.com/p/test-more/issues/detail?id=1
I don't plan on supporting it until TB2, but if someone produced a complete
patch I'd take it.
--
I have a date with some giant cartoon robots and booze.
Re: Let us reflect on the Halting Problem (was Re: Perl 6 and Test.pm's skip() function)
On Jan 22, 2009, at 5:22 PM, Michael G Schwern wrote:
Because, in Perl and other languages, until you run it you can't
know what
class $object is going to be, or what its inheritance tree will look
like, and
once you do figure out which run_tests() will run (if any) you're
back to the
problem of figuring out how many tests are in run_tests().
I'm in complete agreement with you here, but just to clarify something
that became clear to me only when Eric and I discussed it on IRC, what
Eric is thinking of is basically turning a loop of unknown length into
a single test. So to use your examples, it would be:
test {
for my $thing (@things) {
is $thing, something;
}
}
Somehow, in this example, the `is` would be a subtest, and therefor
uncounted. The call to test() is a single test.
Personally, I think this is a bad idea, because I *want* to count how
many tests are run for @things. But what Eric seems to be talking
about is some kind of subtest, AFAICT.
If we were using a mathematically provable functional programming
language,
say Haskell, there are some cases where you can do static analysis
and figure
out how many tests are going to run. If you got really clever I'll
bet you
could do it for most cases. But when a test involves any randomness
or
outside input it is impossible to know. The best you can do is know
when you
can not know.
Yeah, I think Eric is thinking that those bits with randomness would
be counted as only one test, run in a call to `test {}`.
Best,
David
Re: Let us stop rehashing plans
Please, can we stop going over plans again? Every minute spent yapping about whether plans are good or not is a minute that could be spent doing something useful, like working on Test.pm for Perl 6. This has come up a few times a year for the last five years at least, and I am not exaggerating. The horse is dead. Also, Hitler. xoxo, Andy -- Andy Lester = a...@petdance.com = www.petdance.com = AIM:petdance
