>>>>> On Fri, 27 Nov 2009 14:24:53 +0800, Sergey Aleynikov >>>>> <[email protected]> said:
> Hello, > I'm just entering automated perl testing, so have a question related > to this topic. I want to test as much possible combinations as > possible on a single machine, > but there's no glue what really matters and what - not. So, currently > i've chosen pairs mymalloc/nothreads and nomymalloc/threaded for perls > 5.8.8, 5.8.9, 5.10.1, 5.11.2, on > i386 and amd64 Freebsd systems. But, is this a correct choice? > Shouldn't i, for example, add 64-bit-int options to any of theese > configs, or make new triplets (so, varying 3 params)? > Probably, this should go to a new section of testing faq? My take on this is that we are in a position where all computers in the world will not be able to run all possible combinations of our input parameters. [...insert endless blurb as to why...] Like always when there is a large array to explore randomness helps to spot problem areas. [...insert endless blurb about evolution...] Whenever a problem area is discovered it pays to put the spot on this area until it is cleaned up and then move elsewhere. So as soon as you find a hobby horse that discovered a single bug, concentrate your search on it to find similar cases. Whenever you discover that you and another tester are doing the same thing, go somewhere else. It's really boring to see two identical results. It's like searching mushrooms on the same grid square that somebody else has just visited before you. Bottom line is we cannot tell you what to explore. You must experiment and watch your results and watch the results of the others in the whole endavour. Reading recent RT tickets is always interesting. Writing new RT tickets is helpful for the community. If there is one thing you should avoid then it is to test outdated distros that are superceded by a higher version. While it is good to have old versions installed so you can study the impact on other modules it is nothing but noise if you're testing Archive-Tar-0.23 when the current version is 1.54[0] -- andreas [0] http://www.nntp.perl.org/group/perl.cpan.testers/2009/11/msg6143241.html
