On Wed, Jul 04, 2007 at 08:04:41PM -0400, DAGOLDEN wrote: > > Overall, I see the logic in what you suggest. Please document it more > fully on the wiki. When I was writing CPAN::Reporter, it was > extremely hard to find any documentation. At best, I was reading > source for CPANPLUS and trying to understand what it was doing.
It is documented on the wiki: http://cpantest.grango.org/cgi-bin/pages.cgi?act=wiki-page&pagename=Reports And was previously available in Audrey's adapted perl.com article, which was bundled with the pre-0.50 CPANPLUS documentation. > >> Additionally, there is no well-defined way that I've ever seen > >> documented for a distribution to specify what platform it runs on or > >> to gracefully bail out if it's on the wrong platform. > > > >See File::FDPasser, it's the example I use in my Preparing For CPAN > >talk. There are others, but this seems to handle it better than most. > > > >This was the reason I was prompted to patch CPANPLUS for "OS > >Unsupported" or "No support for OS", so that authors could die > >appropriately if they knew their module wouldn't work with that OS. > > This needs to be much better documented -- e.g. in EU::MakeMaker and > Module::Build, probably in the PAUSE FAQ, etc. If it works today, > it's probably because of cargo cult repetition of what someone saw in > some other module. It was promoted several years ago when it was introduced, and I have mentioned in several talks since, but yes it should be on the wiki too :) I'll add that and any other perhaps not so obvious hooks. > >If they bail out of a test with the string "OS Unsupported", then that > >is a valid NA report. Anything else is unknown and should result in the > >report being abandoned, or possibly sending only to the author so they > >can at least understand why it bailed. > > So is "UNKNOWN" really unknown or "NOTESTS"? To me, UNKNOWN is > supposed to be exactly that -- an unknown result. UNKNOWN should perhaps have been labelled as NOTESTS, but for whatever reasons the instigators of CPAN Testing, used UNKNOWN. UNKNOWN doesn't mean an unknown result. If the build/configuration fails that should generate a FAIL report. This then would allow the author to verify that the distribution is supposed to work on that particular perl/platform or whether there was some other error that caused it to fail. > Overall, I get the sense that you (and/or perhaps the original CPAN > Testers creators) intended to have "NA" and "UNKNOWN" have very > specific meanings that differ from what those terms mean colloquially. > That's unfortunate. The test grades were instigated long before I got involved, and have represented the following for a long time: PASS - built and passed all tests FAIL - failed to build or failed one or more tests UNKNOWN - built, but had no test suite NA - not applicable to this platform and/or version of Perl > More generally, I think it's a design flaw to not have a grade to > capture things that don't fall cleanly into one or another category. This was something that was discussed several years ago. My suggestion was for an 'UNGRADED' report (although thats a bad choice for a name), which could indicate something didn't work right, but that it's unclear whether the environment or distribution is at fault. The feeling was for the reports to be ignored and for the testers to either investigate further or email just the author for clarification. > Too much of what happens in determining grades is heuristics -- just > discarding things that don't fit make the whole system brittle. Discarding reports that are due to the tester or tool chain ignoring the requirements from the author (such as failing to install a prerequisite), is reasonable. Anything else that isn't NA or UNKNOWN described above should be a FAIL. To my mind the 4 grades are appropriate for all the scenarios. Expanding these or making sub-grades is making work for the author, CPAN Testers, the tools, the reporting mechanisms and the data presentation websites. > I'd like to discuss a common and comprehensive approach > for the other issues and see if we can't come up with something that > is more robust and better defined and documented. If the report itself contains more information, that's fine, but I still think the basic grades have some value as they are now. Changing that is going to lead to confusion. Previous reports are unlikely to get regraded. I think a seperate mailing list is better for discussion, but the wiki is the better place to record the outcomes of those discussions. I'll see about setting something up. Barbie. -- Birmingham Perl Mongers - http://birmingham.pm.org Miss Barbell Productions - http://www.missbarbell.co.uk
