On Sun, Feb 19, 2006 at 10:22:20PM +1100, Adam Kennedy wrote: > > 2. Incompatible packaging. > Packaging unwraps, but missing files for the testing scheme.
You may want to split this into a result that contains no test suite at all (UNKNOWN) and one that has missing files according to the MANIFEST. Currently the second only gets flagged to the tester, and is usually only sent to the author, if the tester makes a point of it, or the CPAN testing FAILs. > 5. Installer missing dependency. > For the installer itself. > > 6. Installer waits forever. > Intended to cover the "non-skippable interactive question" This needs to cover misuse of fork() and alarm. Too many distributions assume these work on Windows. They don't. At least not on any Windows platform I've ever used from Win95 to WinXP. They usually occur either in the Makefile.PL or the test suite, rarely in the actual module code. They may also fail on other OSs. Although they could potentially be covered in point 5. > 12. System is incompatible with the package. > Linux::, Win32::, Mac:: modules. Irreconcilable differences. Not sure how you would cover this, but point 12 seems to possibly fit. POSIX.pm is created for the platform it's installed on. A recent package I was testing, File::Flock (which is why I can't install PITA) attempted to use the macro EWOULDBLOCK. Windows doesn't support this, and there doesn't seem to be a suitable way to detect this properly. This is just one example, I've come across others during testing. Also note that the name alone does not signify it will not work on other platforms. There are some Linix:: distros that work on Windows. > 14. Tests run, and some/all tests fail. > The normal FAIL case due to test failures. Unless you can guarantee the @INC paths are correct when testing, this should be split into two. The first is simply the standard FAIL result. The second is the result of the fact that although the distribution states the minimum version of a dependancy, the installer has either failed to find it or found the wrong version. This is a problem currently with CPANPLUS, and is unfortunately difficult to track down. It's part of the "bogus dependancy" checks. > Installation and Completion > --------------------------- > > 16. All Tests Pass, did not attempt install. > > 17. All Tests Pass, but installation failed. > > 18. All Tests Pass, installation successful. And another one, test suite reports success, but failures occurred. This is usually the result of the use of Test.pm. I've been told that due to legacy systems, test scripts using Test.pm must always pass, even if there are failures. There are still a few distributions that are submitted to CPAN like this. Barbie.
