# from Ovid on Sunday 19 August 2007 05:44 am: >there are two broad categories of tests: > >1. Tests which absolutely, definitely, unquestionably, beyond a > shadow of a doubt, MUST NEVER FAIL. I'll call these MNF tests. > >2. Everything else.
In an effort to come up with more convenient terminology, I offer the following acronyms: MUST: 'Mandatory Unit and Standard Tests' EXTRA: 'Exhaustive or eXternal Tests Requiring Additional info' Feel free to suggests more expressive/fun/memorable definitions and/or better terms. >MNF tests mean that there's a real, honest-to-god problem in the >software. It doesn't mean that someone forgot =back tag. It doesn't >mean that network tests fail because the network went down partway >through the test. It doesn't mean that yes, they have your favorite >GUI module installed but aren't running X. If an MNF test fails, you >can't trust the module and the author should be notified. MUST tests will (and should) fail when the module is not adequately portable (PPC vs x86) or cross-platform (qdos vs unix-like), is incompatible with an older Perl (5.0.5), incompatible with a currently installed version of a dependency, or otherwise triggers a bug which makes it unusable on the target system (redhat.) MUST failures are treated as an absolute assertion that the software will not work and the install should be aborted (no matter how deep the dependency tree.) As many have noted, MUST failures tend to give a lot of users a bad day. If the tests don't actually qualify as MUST, then this is a "bad day" which could have been avoided. From outside the CPAN author community, anything observed as "doesn't work" makes Perl look bad or hard to use. EXTRA tests come in many flavors and whether or not to run them depends on many things. Where the MUST tests have to assume lowest-common- denominator (e.g. "I, joe-not-a-programmer need to install this to satisfy a dependency on VMS running perl 5.0.5 and will never report a bug"), the EXTRA tests can be a bit more aggressive about testing exotic and meta bits. The line between MUST and EXTRA can be a fuzzy one. In some cases, the author might think they are MUST, but it is important to see the end-user point of view. EXTRA tests do things like: * check your pod/coverage/kwalitee * use the gui * use the network * use the soundcard (a distracting no-no in office environments) * use the modem to make a phone call * login to your webservices account and test-query the server * create/delete database/tables in a shared RDBMS * require user input/setup * run for 24 hours * check "optional" features (frontends, plugins or etc.) * require "additional" dependencies (may not work on the target) EXTRA does not have to mean that end-users or CPAN smokers will never run those tests. Rather, if we can adequately define what and how to run, we will probably end up with more opt-in and better distributed testing. If we can reach some consensus, we can even go so far as to document some recommendations. http://perl-qa.hexten.net/wiki/index.php/Best_Practices_for_Testing --Eric -- As an old bass player friend of mine used to say: throw money, don't clap. --Tony Parisi --------------------------------------------------- http://scratchcomputing.com ---------------------------------------------------
