On Wednesday 01 December 2004 02:03 CET Justin Mason wrote:
> sounds interesting! few points:
>
> - it'd be worth checking out Test::More et al to see what they
> recommend. However, I don't think we can actually *move* to
> using Test::More or similar instead of Test, until they're
> reliably in the perl core -- so far, they're certainly not
> included in 5.6.1 so that's out :(
Jepp, unfortunately. Test::More and Test::Simple are in there since 5.8.0 I
think. Test::Builder would also have benn nice.
> - separating "tmp" and "log" -- +1
>
> - helpers -- +1. although note that I've spotted an issue with that.
> if you have:
>
> ok_if_blah($foo, $bar);
> sub ok_if_blah {
> ...
> ok ($condition);
> }
>
> that is not as useful as
>
> ok (if_blah($foo, $bar));
> sub if_blah {
> ...
> return $condition;
> }
>
> why? because ok() lists the line number of the ok() acll that
> passed/failed. if ok_if_blah() is used, the line number will
> be inside the ok_if_blah() function, which is useless. (see
> "uris.t" for an example of how the latter function can work btw.)
Oh, good point.
> - it'd be *really* nice to clear up the stuff we currently do on
> every test script, where it runs about 3 fork()/exec()s before the
> testing starts. that's slow.
Definitely.
> - "groups" of tests using "nn_test.t" -- +1 . however I'd like to
> see the simple fast tests run first, even if they're not the same
> logical grouping... e.g. running --lint early makes a lot of sense.
basic_lint.t was a candidate for 10_basic_lint.t on my list, where tests
start at 1x with 1x being the basic ones :)
> - what'd be great would be a simple config file for the test suite;
> it's silly to have to specify (and add code for) all possible
> test-suite-configuration settings in Makefile.PL! for example,
> when we need to do LDAP tests, it'd be much easier to have a
> config file for the test suite that all the LDAP config can
> be put into. (along with the bayes SQL db config, booleans for
> "do bayes tests"/ "do net tests" / "do razor2 test" etc.)
I thought about creating a directory test/flags (on demand). In there the
Makefile.PL could touch files which are named after the categories.
For handling the categories in Makefile.PL I though about something like a
space seperated list of atoms. The default would be something like "basic
bayes sql" (where any unconfigured tests are skipped, too). Then you could
either run Makefile.PL with TEST_FILTER="foo bar" or if that one isn't
given it would ask you for a list of tests to run.
That lists would follow the following logic: Each of these atoms can be
given either alone or with one of the prefixes [+-]. If no prefix is
given, the list is overridden. + adds one to the existing list, - removes
one. So if you want to run net tests but no bayes ones, you would say
"+net -bayes". For the devs' convenience there would be one meta-category
called "all" :)
> - I'm not keen on the idea of requiring a new wrapper "run" script
> to run tests, though -- I think something along the lines of
> "2 lines of boilerplate at top of script" should be able to
> do that just fine though.
In my eyes tests should be run only via
make test TEST_FILES=t/10_basic_lint.t
anyway. That ensures that the build you test against is always up-to-date.
Everybody who doesn't like to type that much can create his own
alias/wrapper :)
But you're right, now that I thought about it, the run script (which would
have been only two or three lines long anyway) isn't necessary at all. The
new boilerplate will probably look like this (I'll also add a skel for new
tests):
#!/usr/bin/perl -w -I. -It
use SATest;
plan tests => 1;
# any other inits and helpers here
sub main {
# this code is run
}
Have you noticed that our tests don't run in taint mode? Test::Harness
propagates any -[tT] switch given on the shebang line, I guess we should
add it (not blindly of course, first making sure that everything still
works).
Cheers,
Malte
--
[SGT] Simon G. Tatham: "How to Report Bugs Effectively"
<http://www.chiark.greenend.org.uk/~sgtatham/bugs.html>
[ESR] Eric S. Raymond: "How To Ask Questions The Smart Way"
<http://www.catb.org/~esr/faqs/smart-questions.html>
