Re: Test Reforms
Randal L. Schwartz Don't remove all examples from the distro. At 30K feet, they were very helpful for me while I was trying to write code, but couldn't get to the website. Or maybe make a simple one-stop-download link on the website. they can at least be packaged separately. in fact, a -doc package could include all the sample code and recipes from the web
Re: Test Reforms
At some point, the cookbook and possibly the tutorials should be bundled into CPAN distributions. Perhaps POE::Tutorials or POE::Examples I'm almost tempted to suggest that should be left to each distro... in RH one often finds packages like -doc containing all that stuff.
Re: Test Reforms
At some point, the cookbook and possibly the tutorials should be bundled into CPAN distributions. Perhaps POE::Tutorials or POE::Examples I'm almost tempted to suggest that should be left to each distro... in RH one often finds packages like -doc containing all that stuff.
Re: Test Reforms
Rocco == Rocco Caputo [EMAIL PROTECTED] writes: Rocco POE still ships with about 200K of samples, and it will continue to Rocco until they're replaced by cookbook recipes or tutorials. The plans Rocco have been on display for months at Rocco http://poe.perl.org/?POE_RFCs/Deprecate_Sample_Program I've read that, and disagree. Even if you have only a few dozen examples in the distro, you should consider the person who is on an expensive connection, or gets the CPAN via sneakernet. Don't remove all examples from the distro. At 30K feet, they were very helpful for me while I was trying to write code, but couldn't get to the website. Or maybe make a simple one-stop-download link on the website. -- Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 [EMAIL PROTECTED] URL:http://www.stonehenge.com/merlyn/ Perl/Unix/security consulting, Technical writing, Comedy, etc. etc. See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
Re: Test Reforms
Rocco == Rocco Caputo [EMAIL PROTECTED] writes: Rocco Added to the plan: Rocco At some point, the cookbook and possibly the tutorials should be Rocco bundled into CPAN distributions. Perhaps POE::Tutorials or Rocco POE::Examples, so the low-tech and high-altitude crowds can access Rocco them away from a proper internet. How we maintain both at once is Rocco an interesting exercise, and ideas are welcome. Spot on. Thanks. -- Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 [EMAIL PROTECTED] URL:http://www.stonehenge.com/merlyn/ Perl/Unix/security consulting, Technical writing, Comedy, etc. etc. See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
Re: Test Reforms
On Sat, Feb 21, 2004 at 01:13:30AM -0500, sungo wrote:
On (02/07 15:53), Rocco Caputo wrote:
I think the tests should be generated at make test time. We get a
smaller distribution this way.
you've never been terribly concerned about a really small distribution
before. we've shipped 10K of examples until really recently. i'd much
rather ship 10K of real honest to god easy to read and maintain tests.
POE still ships with about 200K of samples, and it will continue to
until they're replaced by cookbook recipes or tutorials. The plans
have been on display for months at
http://poe.perl.org/?POE_RFCs/Deprecate_Sample_Program
Once the generation phase is done, Test::Harness executes specific and
generated tests as usual.
what happens when test generation fails? how do we include tests for the
test generation framework?
As I see it, test generation is an exercise in templating. The
proposal suggests combining a pool of perhaps 20 reusable test
programs---one for each of POE::Kernel's reusable APIs---against each
available runtime environment. The result would be maybe a hundred
small programs that look virtually identical except that they use
different modules and require different tests.
This idea is based on POE's current t/27_poll.t. If IO::Poll is
available, it's loaded so POE::Loop::Poll will be activated. Then it
tests the select() API, which exercises the loop's filehandle
watchers. If you've never seen it:
#!/usr/bin/perl -w
# $Id: 27_poll.t,v 1.10 2004/01/31 06:58:30 rcaputo Exp $
# Rerun t/04_selects.t but with IO::Poll instead.
use strict;
use lib qw(./mylib ../mylib ../lib ./lib);
use TestSetup;
BEGIN {
eval 'use IO::Poll';
test_setup(0, IO::Poll is needed for these tests)
if length($@) or not exists $INC{'IO/Poll.pm'};
test_setup(0, IO::Poll 0.05 or newer is needed for these tests)
if $IO::Poll::VERSION 0.05;
}
require 't/04_selects.t';
exit;
So, what do we do if templating fails? Well...
1. If POE's favorite templating language isn't available...
POE does not need a full-featured, general-purpose templating
library. The test generator should include some basic, line-based
tag replacement and be done with it. No problem.
2. If the test generator contains one or more bugs...
A large number of the test files will come out broken. Like the
Balrog, or constipation, they shall not pass. In theory, the
release technician will be bright enough notice something is wrong
while testing a distribution before releasing it.
We can always hire a new release technician if the current one's
excessively stupid or careless.
3. If there's not enough disk space to write the test files...
The test generator reports a disk full error, just like tar or
unzip would.
4. If the filesystem doesn't like the filenames used...
The test generator reports a cannot open file: $! error, just
like tar or unzip would.
Fixing a generator is probably easier than maintaining a hundred
discrete files.
5. If Perl is broken...
It happens. The tests will probably fail either way, so POE's
neither better nor worse off.
6. If a god smites the machine...
The bad disk block would have been there for tar or unzip to
extract a file into. The meteorite would have hit the server farm
anyway. POE is neither better nor worse off.
7. If something unanticipated happens...
Sorry, I just don't know. That's the problem with unanticipated
failure modes.
Are there failure modes I'm missing?
The major difference between revisions is that Sean pointed out some
test environments that I missed. So we have a potential for many more
test files, but we also test more of POE.
then i completely misread it. or it changed drastically from the time
you showed it to me in irc to the time you sent it to the list.
It's probably changed since you last read it. There's this iterative
cycle of release, gather feedback, edit, and re-release going on. It
happens even when no one's paying attention to the list.
my opinion is still that you are suggesting a massive new complexity of
autogenerating self-building tests that isnt really necessary. this
level of complexity will form a roadblock to long term maintenance and
developer understand. unless of course you're planning on the only
person doing long term maintenance or developing on this being you. if
that's what you're thinking, go for it. do whatever you want.
I don't see the massive complexity you do. The proposal is for a
program that combines a set of test libraries (such as selects.t and
alarms.t) with a set of event loops (Gtk, Tk, Event, IO::Poll,
select), generating from them:
#!/usr/bin/perl -w
use strict;
use lib qw(./mylib ../mylib ../lib ./lib);
use Gtk;
require selects.t; # and a second one with alarms.t
exit;
and
#!/usr/bin/perl -w
use strict;
use lib qw(./mylib ../mylib ../lib ./lib);
use
Re: Test Reforms
On (02/07 15:53), Rocco Caputo wrote: I think the tests should be generated at make test time. We get a smaller distribution this way. you've never been terribly concerned about a really small distribution before. we've shipped 10K of examples until really recently. i'd much rather ship 10K of real honest to god easy to read and maintain tests. Once the generation phase is done, Test::Harness executes specific and generated tests as usual. what happens when test generation fails? how do we include tests for the test generation framework? The major difference between revisions is that Sean pointed out some test environments that I missed. So we have a potential for many more test files, but we also test more of POE. then i completely misread it. or it changed drastically from the time you showed it to me in irc to the time you sent it to the list. my opinion is still that you are suggesting a massive new complexity of autogenerating self-building tests that isnt really necessary. this level of complexity will form a roadblock to long term maintenance and developer understand. unless of course you're planning on the only person doing long term maintenance or developing on this being you. if that's what you're thinking, go for it. do whatever you want. -- Matt Cashner http://eekeek.org eek at eekeek dot org
Re: Test Reforms
This thread has stalled. I made more revisions to http://poe.perl.org/?POE_RFCs/Test_Reforms while waiting for feedback. -- Rocco Caputo - [EMAIL PROTECTED] - http://poe.perl.org/
Re: Test Reforms
On Fri, Feb 06, 2004 at 11:57:26PM -0500, sungo wrote: On (02/04 04:14), Rocco Caputo wrote: Thanks for the reply. http://poe.perl.org/?POE_RFCs/Test_Reforms has been harshly (and incompletely, but it's like 03:00 here) revised based For perspective: The goal of updating POE's test is to maximize test coverage. This is an offshoot of the XS project, which requires a comprehensive test suite to verify that the plug-in XS replacements work identical to the base modules. POE is funky, almost self-modifying. It adapts to both the event loop you're using and the operating system you're running it on. This presents interesting issues, which seem best addressed by testing POE in as many configurations as possible. To reiterate: The goal is maximal test coverage. Is this an unrealistic goal? Is there a better way to achieve this goal? Is there a better goal? so, we've gone from our current test suite which is not complete but workable to a proposed test suite which involved lots of meta tests being sourced and run by various environments to a dynamically generated (at dist time? at run time?) test suite. I think the tests should be generated at make test time. We get a smaller distribution this way. how do we even begin to support such a massive framework? is there some kind of meta language to describe what sections of modules its built for? is there some kind of preprocessor? The specific tests are hand-written, and they don't really count towards the weird aspect of the proposed test framework. The weird bits consist of a set of meta test files and a program that's run before Test::Harness. The mystery program generates a stub file for every combination of runtime environment options. These stubs Crequire one meta test apiece. Once the generation phase is done, Test::Harness executes specific and generated tests as usual. this all feels really crazy to me. i prefer the last revision to this one. at least it didnt require massive new code (that itself will need tested) just to exist. The major difference between revisions is that Sean pointed out some test environments that I missed. So we have a potential for many more test files, but we also test more of POE. -- Rocco Caputo - [EMAIL PROTECTED] - http://poe.perl.org/
Re: Test Reforms
On (02/04 04:14), Rocco Caputo wrote: Thanks for the reply. http://poe.perl.org/?POE_RFCs/Test_Reforms has been harshly (and incompletely, but it's like 03:00 here) revised based so, we've gone from our current test suite which is not complete but workable to a proposed test suite which involved lots of meta tests being sourced and run by various environments to a dynamically generated (at dist time? at run time?) test suite. how do we even begin to support such a massive framework? is there some kind of meta language to describe what sections of modules its built for? is there some kind of preprocessor? this all feels really crazy to me. i prefer the last revision to this one. at least it didnt require massive new code (that itself will need tested) just to exist. -- Matt Cashner http://eekeek.org eek at eekeek dot org
Re: Test Reforms
On Tue, Feb 03, 2004 at 04:04:10PM -0500, Sean Egan wrote: I read the page and I like the idea of a plan for standardized test. Howver, I am a little concerned with some parts of the plan. The unit tests sound good but POE is a whole environment. So having unit tests are a good thing for those parts that can be isolated, but overall will have limited impact on the over all test framework. The meta tests are just generic loop tests. I think you should recharacterized the meta tests into two parts. The first, loop implementation specific tests and the generic loop tests. I know that is already sorta in there but the structure, ie calling them z0 for both the implementation specific loop tests and the generic loop tests, doesn't make the distinction clear. Further, other pluggable parts of POE need generic tests. Like POE::Queue and what not. My thoughts are along these lines: - unit tests for those parts which are independent of the POE environment. - Specific tests for Loop and Queue implementations. - generic tests for Loop and Queue (run over each implementation) - Tests for each of the other standard components. I would like to help more, but I rarely log into IRC, and most of this kind of discussion seems to go on there. Thanks for the reply. http://poe.perl.org/?POE_RFCs/Test_Reforms has been harshly (and incompletely, but it's like 03:00 here) revised based on your feedback. I hope it captures the essence of your suggestions, and I look forward to further refinements (especially in the incomplete sections) (by anyone, not just Sean). I think you're mistaken about IRC. A lot of discussion winds up rotting in obscurity on POE's wiki. It contains a globally visible (and editable) repository of design notes that nobody else uses. I even maintain a to-do list covering the goals for the next few POE releases on it, although they tend to slip pretty badly. So in practice the wiki's a flop. I'm willing to post design documents somewhere more effective. Matt Cashner has suggested they go to this mailing list. They can be upwards of a few hundred lines, but I'm willing to try it. Suggestions. If you got 'em, flaunt 'em. -- Rocco Caputo - [EMAIL PROTECTED] - http://poe.perl.org/
Re: Test Reforms
On (02/04 04:14), Rocco Caputo wrote: So in practice the wiki's a flop. i don't think i'd say that. the Cookbook alone makes it a success. but the wiki is not a project management or team collaboration tool. its just a web site. esp since watching the wiki for new content is an active endeavor. one must reload the changes page often to notice change. -- Matt Cashner http://eekeek.org eek at eekeek dot org
Re: Test Reforms
Rocco Caputo wrote:
I gave it a shot. Devel::Cover has come a long way since I last looked
at it. It only introduces failures in three of the tests.
The results are temporarily available at http://nopaste.snit.ch:8001/541
Yep, it sure has. You want to generate the HTML report next, which will
let you drill down to the exact lines of source that have been followed,
show you which branches where followed, and why (eg, with if (foo() ||
bar()) {} it will tell you whether or not it ever branched because
bar() !foo()). Which is extremely useful for helping decide what
conditions your test suite doesn't test, based on the way the code is
actually structured...
Sam.
Re: Test Reforms
I read the page and I like the idea of a plan for standardized test. Howver, I am a little concerned with some parts of the plan. The unit tests sound good but POE is a whole environment. So having unit tests are a good thing for those parts that can be isolated, but overall will have limited impact on the over all test framework. The meta tests are just generic loop tests. I think you should recharacterized the meta tests into two parts. The first, loop implementation specific tests and the generic loop tests. I know that is already sorta in there but the structure, ie calling them z0 for both the implementation specific loop tests and the generic loop tests, doesn't make the distinction clear. Further, other pluggable parts of POE need generic tests. Like POE::Queue and what not. My thoughts are along these lines: - unit tests for those parts which are independent of the POE environment. - Specific tests for Loop and Queue implementations. - generic tests for Loop and Queue (run over each implementation) - Tests for each of the other standard components. I would like to help more, but I rarely log into IRC, and most of this kind of discussion seems to go on there. -Sean. On Mon, 2 Feb 2004, Rocco Caputo wrote: Good morning. Tests are a vital part of the POE project. Its current tests succeed at covering about 70% of POE's its instrumentable breakpoints, but they do so without any coherent plan. Starting around version 0.30, we'll be reorganizing POE's test files and expanding their scope. We'll be migrating the tests from an old, homegrown library to Test::More in the process. The entire effort hasn't been charted out, but it will probably span multiple releases, I've put the first draft online for consideration before we start scheduling the work. If you're familiar with test design, if you think a better set of POE tests would help you out, or if you're just curious, please visit http://poe.perl.org/?POE_RFCs/Test_Reforms and comment on it here. Thank you. -- Rocco Caputo - [EMAIL PROTECTED] - http://poe.perl.org/
Re: Test Reforms
On Tue, Feb 03, 2004 at 10:13:04AM +1300, Sam Vilain wrote: Rocco Caputo wrote: Tests are a vital part of the POE project. Its current tests succeed at covering about 70% of POE's its instrumentable breakpoints, but they do so without any coherent plan. Have you tried running the test suite with Devel::Cover ? It replaces the Perl_runops() loop with a version that accounts which lines of the source code are actually run by a series of test scripts... much better than the approach taken by Devel::Coverage, for instance. This can tell you whether it really is 70% or more like 30% :-). I gave it a shot. Devel::Cover has come a long way since I last looked at it. It only introduces failures in three of the tests. The results are temporarily available at http://nopaste.snit.ch:8001/541 -- Rocco Caputo - [EMAIL PROTECTED] - http://poee.prl.org/
