Re: Private tests
-BEGIN PGP SIGNED MESSAGE- Moin, On Monday 14 November 2005 18:21, Chris Dolan wrote: Hello all, I've just published an article about public vs. private regression tests. I've defined private tests as t/*.t files that are for the author only and don't go in MANIFEST. Naturally, those don't get as much publicity as tests included in CPAN distributions. In the article I advocate that some tests should be private. In particular, 1) those that test non-critical aspects of a module (like documentation and coding style) 2) those that are too expensive to run often 3) those that require special software or customization In my conclusion I describe a possible system where authors publish the results of private tests with their distributions as a trust- based kwalitee system. That is, authors assert kwalitee rather than be judged for it. http://www.chrisdolan.net/talk/index.php/2005/11/14/private- regression-tests/ Both positive and negative feedback is very welcome! Private tests will only be run by the author, meaning they will be only run on a very small subset of all systems the modules can be used on. This limits their usefullness quite a bit. Case ein point: I can test my modules on linux, 32 bit, unthreaded, under unicode, and under perl 5.8.x. Thats about it, everything else gets really really complicated for me to set up and maintain/test. So, no win32, no mac, now solaris, no irix, no perl 5.6.x, no iso-something, no EBDIC (or however it is spelled), no threading, no 64 bit, no SMP system. As for 1), these things should matter (the broken window analogy) and you would be surprised to know how these tests can pass on your system, and still fail on other systesm (forget to include the .pod file in MANIFEST is the most obvious one). As for 2), random testing should be employed (Math::BigInt does this, it runs 256 or so tests with random number patterns (and thus known results like 2 * A - A == A). The tests are quite fast, but they cover only a small subset of potential values. However, since each system and user runs a new, different random set, you end up with a really huge testing number being run. (Yes, this has found some bugs) And for 3), this might be the only point I can think that private tests are usefull (I have a private testset for Graph::Easy that I use from time to time, it is not public mainly because it fails/hangs/takes forever and is work-in-progress). However, I have to actually read your article to find out what your proposal solves (compared to me just running thetest once in a while :) Hope that was usefull :) Best wishes, Tels - -- Signed on Tue Nov 15 11:04:21 2005 with key 0x93B84C15. Visit my photo gallery at http://bloodgate.com/photos/ PGP key on http://bloodgate.com/tels.asc or per email. Now, _you_ behave! -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.4 (GNU/Linux) iQEVAwUBQ3m0pHcLPEOTuEwVAQF6lQf8CtubDMQjLdCBcEGNczxZj2Y1kTVhOU7Z bvweeJe4RWFfmd2JMw7dwiu3Sjb57hNlVkl4LwN+7vx3tm3DsnhRUoMHvkDtCddC 8bfxpLcOi8WMlySAud+unKnpZVwlwn2rZ/enu2Dd01QKOgOQkBr1HWTJUguPv4No eWT3UiEZhV4hU764gtF7a8raHjbvxpTJcNk22KHnRmyTKX+SugCyI0qkmIQrFntl cQWXyA9GV7V+5bK5/Sp2eWv2MXX3fhNDxtZywkmqun+6/YhPgSDJQp3FcKThZFYy WxPXsrXVIXFJbbtkSs+PGe18VdXqEFCHOI29H1+9gyiC3FW3N6AARw== =GA3y -END PGP SIGNATURE-
Re: Private tests
On Tue, 2005-11-15 at 15:23 -0600, Chris Dolan wrote: After reading some of the insightful comments posted on my blog, I've been convinced that the private tests should be included in the CPAN distribution, but disabled in some way (perhaps via a file extension other than .t?). That way, resourceful or interested users can run the tests but average users don't need to. I posted a small Module::Build subclass that shows one way to do this to Perl Monks: http://perlmonks.org/?node_id=508160 Perhaps a better approach is to store these tests in a subdirectory of t/. -- c
Re: Private tests
Le mardi 15 novembre 2005 à 15:23, Chris Dolan écrivait: After reading some of the insightful comments posted on my blog, I've been convinced that the private tests should be included in the CPAN distribution, but disabled in some way (perhaps via a file extension other than .t?). That way, resourceful or interested users can run the tests but average users don't need to. What about a special environment variable, like RUN_PRIVATE_TESTS? -- Philippe BooK Bruhat When you do not think for yourself, no one thinks for you... (Moral from Groo The Wanderer #61 (Epic))
Re: Private tests
Philippe 'BooK' Bruhat wrote: Le mardi 15 novembre 2005 à 15:23, Chris Dolan écrivait: After reading some of the insightful comments posted on my blog, I've been convinced that the private tests should be included in the CPAN distribution, but disabled in some way (perhaps via a file extension other than .t?). That way, resourceful or interested users can run the tests but average users don't need to. What about a special environment variable, like RUN_PRIVATE_TESTS? I've been working on a concept of taggable tests on some of my larger commercial stuff, integrating with the Test::More skip() function, and some form of environment variables does indeed seem the best way to do such a thing. For example, with applications that talk to the database, a lot of the time I want to be able to just test all the code that doesn't need a live production database, or need a database with the production schema, or need a schema that it is allowed to modify, and so on and so forth. So I have a bunch of environment variables like TEST_TAG_DB_RO TEST_TAG_DB_RW, TEST_TAG_DB_SCHEMA and so on and so forth. It seems to be working OK, I'm sure there's be a Test::Tags module in a year or something. Adam K
Re: Private tests
Adam Kennedy wrote: What about a special environment variable, like RUN_PRIVATE_TESTS? I've been working on a concept of taggable tests on some of my larger commercial stuff, integrating with the Test::More skip() function, and some form of environment variables does indeed seem the best way to do such a thing. Isn't Test::Manifest sort of geared to do this kind of thing? (Or could be extended to do so?)
Re: Private tests
On Tue, 2005-11-15 at 22:33 -0600, Chris Dolan wrote: Beware that M::B has a recursive mode for finding tests. It's set by the author, so you should be safe in this case, but it's a point worth remembering. I haven't looked at the code again just now, but wouldn't overriding find_test_files() and filtering out specific subdirectories based on active environment variables do the trick? -- c
