-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
sounds interesting! few points:
- - it'd be worth checking out Test::More et al to see what they
recommend. However, I don't think we can actually *move* to
using Test::More or similar instead of Test, until they're
reliably in the perl core -- so far, they're certainly not
included in 5.6.1 so that's out :(
- - separating "tmp" and "log" -- +1
- - helpers -- +1. although note that I've spotted an issue with that.
if you have:
ok_if_blah($foo, $bar);
sub ok_if_blah {
...
ok ($condition);
}
that is not as useful as
ok (if_blah($foo, $bar));
sub if_blah {
...
return $condition;
}
why? because ok() lists the line number of the ok() acll that
passed/failed. if ok_if_blah() is used, the line number will
be inside the ok_if_blah() function, which is useless. (see
"uris.t" for an example of how the latter function can work btw.)
- - it'd be *really* nice to clear up the stuff we currently do on
every test script, where it runs about 3 fork()/exec()s before the
testing starts. that's slow.
- - "groups" of tests using "nn_test.t" -- +1 . however I'd like to
see the simple fast tests run first, even if they're not the same
logical grouping... e.g. running --lint early makes a lot of sense.
- - what'd be great would be a simple config file for the test suite;
it's silly to have to specify (and add code for) all possible
test-suite-configuration settings in Makefile.PL! for example,
when we need to do LDAP tests, it'd be much easier to have a
config file for the test suite that all the LDAP config can
be put into. (along with the bayes SQL db config, booleans for
"do bayes tests"/ "do net tests" / "do razor2 test" etc.)
- - I'm not keen on the idea of requiring a new wrapper "run" script
to run tests, though -- I think something along the lines of
"2 lines of boilerplate at top of script" should be able to
do that just fine though.
- --j.
Malte S. Stretz writes:
> Moin,
>
> I think I found something to kill some time on: Cleaning up the test
> framework. Currently each test is created by copy-and-pasting some old one
> as a skeleton and modify it according to the needs. This introduced some
> kludgy stuff, like the three line preamble each test has and sometimes the
> chdir("t") stuff which shouldn't be necessary because that's already done
> in sa_t_init().
>
> I'd like to replace the current medley with some cleaner framework with
> which its easier to write tests.
>
> The first change would replace the current three-line preamble with one
> single 'use SATest'. That's possible with the attached patch and some
> additional code. This will mean that the executable attribute on all those
> tests (which you guys always forget to set anyway :) will disappear.
> Instead there will be a short script called "run" or something which you
> can use to execute single tests.
>
> The next thing is that the tests are currently run in more or less random
> order (yeah, alphabetic but logically randomly). I'd like to introduce a
> schmema like the one we have for the rule files, ie. names like nn_test.t
> where nn is a number, the first digit roughly indicating some group (basic,
> spamd, bayes, ...), the second the precedence within that group.
>
> Then its time for bug 3818 which actually led me to this plan :) I'm not
> sure if that should be based on the grouping mentioned above or some
> additional keywords for each test, but I'd like to make the disabling (and
> enabling) of single tests more flexible. So, say, some Bayes SQL test
> could have the keywords "bayes sql net" -- only if all three keywords are
> enabled, the test is run.
>
> I hope a slimmer and documented (and thus easier to understand) framework
> for the tests will make people write more tests. On the other hand do I
> not want to increase the number of tests run for the mere mortal who
> installs SA via CPAN and already has to wait ages. So I'd like to keyword
> some of the tests as "basic" (or whatever keyword) and only those tests are
> run per default. All other tests would be used by us devs, people who we
> ask to debug one of their bug reports aund the BuildBots.
>
> Oh, and another thing is separating the real temp data from the log data, so
> the latter is better to find. Also make it easier to use the dirs by
> feeding some vars to the scripts.
>
> Plus maybe some helper routines like ok_if_equal() (yeah, I've been working
> a lot with JUnit lately).
>
> And I almost forgot: Making the scripts actually compliant with
> Test::Harness -- currently any unknown output on stdout is just ignored but
> the man page has a big fat warning that this might change in future. And
> we really ought to give reasons why tests are skipped.
>
> Any other things you'd like to see?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Exmh CVS
iD8DBQFBrRhOMJF5cimLx9ARAgdSAJkB8+LqbwxZ+tZxRxK0OdVrjkuTBgCgsv/2
AlopQRZ+iLelURqj+LG7p8Y=
=eAL4
-----END PGP SIGNATURE-----
