On Dec 18, 2007 9:59 PM, Chris Dolan <[EMAIL PROTECTED]> wrote: > Does anyone know how the false negative rates compare for cpan-tester > smokers vs. CPAN::Reporter users? I've found the former to be > enormously valuable for cross-platform testing (especially David > Cantrell and Slaven Rezic), but I have seen very little feedback via > the latter at all.
I believe that Cantrell and Rezic are using a CPAN::Reporter-based smoke testing setup (i.e. not CPAN::YACSmoke). In the past, there have been a number of "false negatives" on CPANPLUS based testers due to issues with Build.PL only distros and the way that CPANPLUS behaves when Module::Build isn't installed. Rezic provoked a huge number of failures by smoking distributions on Perl 5.005_05 -- anything that uses "our $VERSION" blows up during Makefile.PL or Build.PL due to how module versions are determined by eval(). More recent versions of the toolchain can deal with it and recent versions of CPAN::Reporter check for those specific errors and flag them NA instead of FAIL. I think the headaches from these issues were one reason that prompted Schwern to declare the death of 5.005. There have also been numerous "missing library" failures -- which is actually a legitimate gap in how the toolchain handles external dependencies and thus how smokers handle them. Thankfully, Cantrell wrote Devel::CheckLib to try to make that easier, though it still means that module authors needs to do some work. It feels to me like much of the development work on CPAN::Reporter for a while has mostly been fixing various edge cases that cause false FAIL reports. It's complaint driven development -- with annoyed CPAN authors being the main complainants. Overall, my guidance to people is to just treat test reports as data -- not value judgments. It just means that some person with some particular configuration of perl wasn't successful running tests. If module authors don't like getting reports, they're welcome to filter them to their trash. That said, there has been some design discussions about switching to web-based test report submission, which would allow a central clearinghouse where authors could choose whether or not to receive reports instead of relying on emails from individual testers. See http://search.cpan.org/perldoc?CPAN::Testers for more details. David
