Re: RFC: Test::ManyParams
On Fri, Aug 02, 2002 at 08:16:17AM +0200, Janek Schleicher wrote: > Ilya Martynov wrote at Fri, 02 Aug 2002 07:42:44 +0200: > > >> On Wed, 31 Jul 2002 21:52:17 +0200, Janek Schleicher <[EMAIL PROTECTED]> >said: > > > > JS> [..snip..] > > > > JS> Thinking in general, > > JS> there could be also some other features included. > > JS> Let's think we'd like to test the creation of big pictures, > > JS> perhaps 5_000 x 5_000. > > JS> It could take a while to make a test for all pixels, > > JS> but we also would like to test some of them (randomly choosen), > > JS> to avoid systematic error. > > > > Test results should be easily reproducible. I don't think having > > randomly choosen tests is good idea. I think having randomly chosen repeatable tests is an excellent idea. Over the course of many people making many test runs explore far more of parameter space than any single systematic test permutation device could hope to achieve. > srand could be our friend. Which is how I'm doing it at work now. I call srand with a random number. (I'm getting mine from /dev/urandom, but I suspect that calling rand() and using that to prime srand will achieve sufficient randomness for these purposes. (ie you get to run one of 65536 sequences, which is better than running 1 of 1 sequence) My tests at work generate a seed this way before starting. They call srand with the seed, and store the seed. If the test fails, they print out what the seed was on STDERR. (as a hack in an END block that checks the exit code after Test::Builder has set it in its END block) If the test is run with no arguments it randomises (as above). Else it treats the argument as a chosen seed, and uses that to call srand() to repeat a previous run. This way I can make test a few times and it runs different random parameters each time. And if a test fails, it prints out the seed, and I can re-run repeatedly by hand until I work out why there is a bug. And fix it. I've found bugs this way that I wouldn't have spotted early by just running a fixed set of parameters in my tests each time. Mainly because I make test many times during the day as I make each incremental change (so each make test has to be fast) which immediately picks up on big bugs, but the incremental effect can find really obscure bugs that only crop up for a small proportion of possible input. > However, it's not important for me that the parameters are really randomly choosen - > > allthough I still would prefer to have some before I release a module - > but I'd like to write tests to avoid systematic mistakes, > while it would need too long to test all scenarios. This is the problem that I have, and I think I've found a solution that works for me. Nicholas Clark
Re: RFC: Test::ManyParams
-BEGIN PGP SIGNED MESSAGE- Moin, On 02-Aug-02 Nicholas Clark carved into stone: > On Fri, Aug 02, 2002 at 08:16:17AM +0200, Janek Schleicher wrote: >> Ilya Martynov wrote at Fri, 02 Aug 2002 07:42:44 +0200: >> >> >> On Wed, 31 Jul 2002 21:52:17 +0200, Janek Schleicher >> >> <[EMAIL PROTECTED]> said: >> > >> > JS> [..snip..] >> > >> > JS> Thinking in general, >> > JS> there could be also some other features included. >> > JS> Let's think we'd like to test the creation of big pictures, >> > JS> perhaps 5_000 x 5_000. >> > JS> It could take a while to make a test for all pixels, >> > JS> but we also would like to test some of them (randomly choosen), >> > JS> to avoid systematic error. >> > >> > Test results should be easily reproducible. I don't think having >> > randomly choosen tests is good idea. > > I think having randomly chosen repeatable tests is an excellent idea. > Over the course of many people making many test runs explore far more > of parameter space than any single systematic test permutation device > could hope to achieve. [snip a lot of nice text] > >> srand could be our friend. > > Which is how I'm doing it at work now. > I call srand with a random number. (I'm getting mine from /dev/urandom, > but I suspect that calling rand() and using that to prime srand will > achieve sufficient randomness for these purposes. (ie you get to run one > of 65536 sequences, which is better than running 1 of 1 sequence) rand() is 48 bits on Linux, not 32, right? ;) So it is even better ;) > This is the problem that I have, and I think I've found a solution that > works for me. I also do this in Math::BigInt's testsuite (suggestion by Paul Green) and it is really nice; systematic tests almost always fail to address the more strange cornercases that your forgot to put in the list of things to test. Random testing really finds these obscure bugs, especially when your code done by the tests is non-trivial. Cheers, Tels - -- perl -MDev::Bollocks -e'print Dev::Bollocks->rand(),"\n"' conveniently revolutionize one-to-one action-items http://bloodgate.com/perl My current Perl projects PGP key available on http://bloodgate.com/tels.asc or via email -BEGIN PGP SIGNATURE- Version: GnuPG v1.0.6 (GNU/Linux) Comment: When cryptography is outlawed, bayl bhgynjf jvyy unir cevinpl. iQEVAwUBPUqbZncLPEOTuEwVAQESeAf8CHZlCYgPZCb5zYwvIhpay5fKJAMDoWuM sDac8AIJ24u+z1TIaW8273z/sPmW7OjC9/oQDDdro7aVo+WRsGYpG+roOsGIbcj0 V9+o/RNMDzDyX/Rxcn0kCdmhCgqf9L4af/dX/xciXRJEvzkAsyYSVbVjkrvUy+F7 gEC9r1GkLXH7dKPiNaYjyLEGwN5ccPV8KZb88l1C/lcKc7M26prd0bm9M9Py7LZW i3AUvAitxqCmrTq2bPtc4WGwYk2qaC/bn6GcudDQgvhgO4HhdvTJZc/Sn/Ltl1qm 2uI9qKDe6GFsntgQWRXrznhnuVIk9ZPO45zCQ/yNIVlMnPY4Mn5YWQ== =vFF/ -END PGP SIGNATURE-
Re: RFC: Test::ManyParams
Nicholas Clark wrote at Fri, 02 Aug 2002 11:06:47 +0200: >> srand could be our friend. > > Which is how I'm doing it at work now. > I call srand with a random number. (I'm getting mine from /dev/urandom, > but I suspect that calling rand() and using that to prime srand will > achieve sufficient randomness for these purposes. (ie you get to run one > of 65536 sequences, which is better than running 1 of 1 sequence) > > My tests at work generate a seed this way before starting. They call srand > with the seed, and store the seed. > If the test fails, they print out what the seed was on STDERR. > (as a hack in an END block that checks the exit code after Test::Builder > has set it in its END block) With a "most_ok" function realising randomness test, I would only need to check the result of this function :-)) > If the test is run with no arguments it randomises (as above). Else it > treats the argument as a chosen seed, and uses that to call srand() to > repeat a previous run. > > This way I can make test a few times and it runs different random parameters > each time. And if a test fails, it prints out the seed, and I can re-run > repeatedly by hand until I work out why there is a bug. And fix it. Very elegant. I'll follow that way. [Allthough, I believe I'll take a syntax like use Test::ManyParams;# like srand(0) or srand(42) use Test::ManyParams 'randomized'# with srand(rand()) use Test::ManyParams 'randomized', $seed; # srand($seed) to hide the hacks] Thank you, Greetings, Janek
Re: RFC: Test::ManyParams
-BEGIN PGP SIGNED MESSAGE- Moin, On 02-Aug-02 Janek Schleicher carved into stone: > Nicholas Clark wrote at Fri, 02 Aug 2002 11:06:47 +0200: > Very elegant. > I'll follow that way. > [Allthough, I believe I'll take a syntax like > > use Test::ManyParams;# like srand(0) or srand(42) > use Test::ManyParams 'randomized'# with srand(rand()) > use Test::ManyParams 'randomized', $seed; # srand($seed) te@null:~> perl -e 'print rand(),"\n"' 0.159625336368666 te@null:~> perl -e 'print rand(),"\n"' 0.292230773325176 te@null:~> perl -e 'print rand(),"\n"' 0.708889858870865 te@null:~> This means perl does something like srand(rand()) for you automatically. I would suggest that Test::ManyParams leaves srand() alone, unless the user want's a specific seed: use Test::ManyParams; # 1 nothing or like case 2 use Test::ManyParams seed; # 2 seed = srand(rand()) done by T::MP use Test::ManyParams seed => 5; # 3 srand(5) seed or randomized are both good, I like the shorter one ;) You can take your pick. A nice idea is to make the choosen seed available via print "$Test::ManyParams::seed\n"; although strictly that should not be necc, since surely T::MP will print it for the testsuite? (Also, I don't know how to find out the seed for the case 1, so probaböy case 1 and 2 should do the same and just do a seed = srand(rand()). (I did not quite understand your srand(0) or srand(42) example...what is that supposed to accomplish?) Best wishes, Tels - -- perl -MDev::Bollocks -e'print Dev::Bollocks->rand(),"\n"' conveniently build virtual web-readiness http://bloodgate.com/perl My current Perl projects PGP key available on http://bloodgate.com/tels.asc or via email -BEGIN PGP SIGNATURE- Version: GnuPG v1.0.6 (GNU/Linux) Comment: When cryptography is outlawed, bayl bhgynjf jvyy unir cevinpl. iQEVAwUBPUsGhHcLPEOTuEwVAQFVIwf9FY2b1Ob92OnKpS081phFwv9XhttQA+xk 6kQaQhjpfmGyMTCML3FzYsMhuI3ratW7CkMPxWjuJGlYwaRHxlNFU9l0Q3kT5zMl 6rDRjh0jWlVa7TqzA8cYMbbtKFWCimAg78d/OnDx9XQNjEoka4RTIaAgLHdgx00S PY6RVDpgH4Su1Me2zQbTABDNzK6sxaawkQL1+jEGBwkV4hIHydvggEKFjZ+usKNa UutxaOWonWVIpR6EUpGtfa3fwY6MH49gmSFriOA3KwYzX9GtXR4I8pP84nuEVBpp VJYzwUKKHnOnjQnUvFnJXw2L7a0JdfwPmHhXmgoc+yG6pgbVpC+dow== =fMuj -END PGP SIGNATURE-
Re: RFC: Test::ManyParams
Tels wrote at Sat, 03 Aug 2002 00:25:54 +0200: > te@null:~> perl -e 'print rand(),"\n"' > 0.159625336368666 > te@null:~> perl -e 'print rand(),"\n"' > 0.292230773325176 > te@null:~> perl -e 'print rand(),"\n"' > 0.708889858870865 > te@null:~> > > This means perl does something like srand(rand()) for you automatically. I knew. I wanted to impress that now "randomizing" is on. It was more a style of commenting. However, you're right, it's confusing. > I would suggest that Test::ManyParams leaves srand() alone, unless the user > want's a specific seed: > > use Test::ManyParams; # 1 nothing or like case 2 > use Test::ManyParams seed; # 2 seed = srand(rand()) done by T::MP > use Test::ManyParams seed => 5; # 3 srand(5) That reads really good, especially when written as Test::ManyParams seed => undef; :-) > seed or randomized are both good, I like the shorter one ;) You can take > your pick. Yep. I'll implement both (as synonyms). 'seed' is short and 'randomized' describes what it's talking about. > A nice idea is to make the choosen seed available via > > print "$Test::ManyParams::seed\n"; ACK. But now, when I read the idea, I have to think to the possible scenario, that a test scripts uses another test script and both are using Test::ManyParams with an own seeding could destroy a bit the concept. So, the seed and used random numbers should depend on the callers filename and package. Could be that $Test::ManyParams::seed becomes a tied variable, as I wouldn't like to write a method 'seed' :-). > although strictly that should not be necc, since surely T::MP will print it > for the testsuite? (Also, I don't know how to find out the seed for the case > 1, so probaböy case 1 and 2 should do the same and just do a > seed = srand(rand()). > > (I did not quite understand your srand(0) or srand(42) example...what is > that supposed to accomplish?) I first thought - now I've changed, that the default behaviour should be a fixed seed. As it wouldn't play any important role what fixed seed is taken, I imagined the canonical 0 or even the sense of life :-) Thanks a lot, Cheerio, Janek
Thoughts from TPC: Testing perlguts with Inline::C
More chronicles of brainstorming from TPC. Last year at TPC, Hugo asked for ideas about testing the internal C functions of Perl. Testing perl itself is ok, but if we can test at an even finer grained level of the C API many bugs can be more easily caught. Additionally the documentation will be improved in the process (can't test it if you don't know what it's supposed to do) and the XS mechanism might also be improved as more people have to touch it to test it. At the time, there was nothing to do this except write tests in C, and that sucks. It's hard enough to find people to write tests in Perl. Then Ingy comes along with Inline::C. One of the tricks he pulled off at YAPC::Europe was writing a simple Inline module which exposed all the internal C functions as perl functions. AH HA! Now we can test C functions just like any other Perl function. So, this is a request to place Inline::C into 5.9 to facilitate easy testing of the C API. -- Michael G. Schwern <[EMAIL PROTECTED]>http://www.pobox.com/~schwern/ Perl Quality Assurance <[EMAIL PROTECTED]> Kwalitee Is Job One IMHO bugs in Perl 5 shouldn't carry over to Perl 6. (Unless, of course, we *like* the bugs... ;) -- Ken Fox in <[EMAIL PROTECTED]>
Re: Thoughts from TPC: Testing perlguts with Inline::C
On Thu, 1 Aug 2002, Michael G Schwern wrote: > More chronicles of brainstorming from TPC. > > Last year at TPC, Hugo asked for ideas about testing the internal C > functions of Perl. Testing perl itself is ok, but if we can test at an even > finer grained level of the C API many bugs can be more easily caught. > Additionally the documentation will be improved in the process (can't test > it if you don't know what it's supposed to do) and the XS mechanism might > also be improved as more people have to touch it to test it. > > At the time, there was nothing to do this except write tests in C, and that > sucks. It's hard enough to find people to write tests in Perl. That's not strictly correct. There is nothing to stop an XS module exporting the perl API and testing it using the normal testing modules. If the Perl API is straightforward the XS interface will be straightforward too (as will the Inline interface). perl 5.8 has XS::APItest (and XS::Typemap - this module test quite a bit of the perl API in order to even run) as a very simple start of that. The main difficulty is actually in routines that have complicated arguments and initialisation requirements. I'm not objecting to Inline::C in the core, I'm just suggesting that the argument of "now we can test the API" is not the argument we should be making for it since you can already do it and it's not very difficult. The only real difference is that Inline generates the XS automatically - presumably from the apidoc information. > So, this is a request to place Inline::C into 5.9 to facilitate easy testing > of the C API. -- Tim Jenness JAC software http://www.jach.hawaii.edu/~timj
