On Fri, Sep 01, 2000 at 01:06:24AM -0500, Brent LaVelle wrote:
> A long time ago on p5p I suggested dumping the current test scheme in
> favor of a general purpose testing system where you could test things
> programs like:
I remember it, or at least I selectively remember parts of it. But I
thought the idea was to build an extra level of testing to compliment
rather than usurp the current t/*.t tests. The current tests try to
test simple parts of the language first, before moving on to other more
complicated parts. That's why some of them are written in a fairly
strange fashion. I don't think we can (yet) assume that there will be
an older, working perl available for our use. But I do think that once
perl has passed its t/*.t tests, we can then use it for further testing
purposes.
At the time you first mentioned this I offered some testing code I had,
subject to it being released under an appropriate licence. That has now
happened (was is late in 1998 or 1997 when you first mentioned this?)
and so I make the offer again. This time though, I will hopefully have
a little time to make it happen if this idea is deemed sound, and I
expect that it will be, generally.
I will make a few comments, with particular reference to the code which
I have, not that I expect it to totally suitable, but with the idea that
it could be used as a base upon which to build, or just to get ideas
about what is and isn't a good idea.
The code I have is all Perl. It runs on multiple platforms across a
network, well a few Unices and NT anyway. It runs a set of tests, each
of which may have a number of steps. The tests and steps in each test
can be decided upon at runtime to allow for platform and other
environment differences. The tests are written as Perl data structures
which allows subroutines to be embedded in the tests, the output of
these subroutines determining the nature of the test. Test output can
be checked against golden results, with allowances made for certain
differences, and at the end you get a pretty HTML report giving you all
sorts of information about the tests, what passed, what the exit
statuses were, how long it took...
> The test system should be able to collect metrics. What tests are good
> for what.
>
> The metrics should be used to slice and dice the tests. Based on
[ snip ]
> When I proposed this before nobody seemed to get fired up, they where
> worried about the expensive fork/exec that it added. I'm worried about
> 50000 lines of Perl code that test the same 20% of the C (or whatever)
> code that Perl is written in but take 100 times longer than the 500
> lines needed to test that 20%.
Aha. Are you suggesting code coverage be integrated? Code coverage
might also be useful at the Perl level to test Perl modules. If it can
be done right, this whole package might be valuable to module authors in
the same way that t/*.t is now.
> Below the bottom line: the test system should be aware of the Perl
> version but be and independent product that produces a set of tests for
> Perl, t/ tests. This test system would be a general purpose tool and
> take Perl testing to a new level.
I agree, although I also still see a place for the t/*.t tests.
--
Paul Johnson - [EMAIL PROTECTED]
