Okay following the various threads here is my take on things. 1) Design Decisions
CPANPLUS was written to try and cope with distributions that didn't have a build file (Makefile.PL/Build.PL) or had a build file (Build,PL) but failed to install Module::Build. The design was well meant and was trying to enable an install in the event that a basic Makefile.PL could cope. In some circumstances this has worked, but now that more developers are coding specifically for Module::Build, where even a fallback Makefile.PL is likely to have problems, and the fact that 5.10 now has both CPANPLUS and Module::Build in core, it seems a decision that is proving more traumatic that it first was. Perhaps now is the time to remove that decision and fail gracefully in some way that allows the user and/or tools to know that a prerequisite failure has occurred. 2) UNKNOWN reports As Graham was one of the original instigators of cpan-testers, then his vision of UNKNOWN may well have been a bit more encompassing than it since has become. The current usage is to alert when a distribution has no tests. Such that a distribution builds and runs 'make' fine, but 'make test' can't find a test suite. We still have a notable amount of reports producing UNKNOWNs, but more recently these have been against older versions of distributions that have since incorporated a test suite. However, Graham's initial vision may well suit elsewhere, which I will come back to. 3) Build/Make FAILs This is a very tricky area. There is currently no well defined interface for understanding failures at this stage. To reduce hassle for authors we allowed 'exit 0' with no Makefile to mean that something went wrong, how it's out of our control so ignore it. I know some disagree with exit 0, but in these instances it's appropriate because you are terminating the process in a controlled fashion. The ultimate goal of creating Makefile hasn't happened, but the process has exited cleanly. If we designing from scratch it might well have been useful to use to create some standard error codes (perhaps similar to the HTTP codes), but what's done is done. The current scenario of ignore any reports is the right thing to do here, as the author is basically telling the user/tester, you cannot procede without manually reviewing the reason for the failure. However, there are situations where the author gets it wrong, either in the build or the make stage. How can we discern the difference? To some we should ignore those reports too as they are of no use. However, in many situations I believe they are, as it's how we come to understand why the NA report has more value. It has also highlight some really bad magic some authors put into their build files. Going back to Graham's vision, perhaps UNKNOWN is appropriate here. The reason being that good authors will already have a test suite that runs, so receiving an UNKNOWN will then likely indicate that something further up the chain has gone awry, and if they wish to ignore them they can filter them out. The user visiting the CPAN Testers reports site will then see a shift to UNKNOWN rather than FAIL, which may look better to some authors, but will still require the user to investigate further. 4) Old distributions Another irritation for authors is to receive reports for distributions that have since been fixed. This has been much more of a problem in recent times as many of the testers have recently built new environment and are testing all of CPAN. I don't see the value of sending authors reports for distributions that have since got a PASS report. This is something that CPANPLUS did attempt to do, and with CPAN-YACSmoke it has a built in feature that it will continue to test previous versions only if the current version being tested (which may not be the latest version) FAILs, and will stop when it finally gets a PASS. Because CPAN-YACSmoke keeps it's own database of what it's tested, it can avoid sending the author a FAIL report for anything earlier it tests. If I recall correctly CPAN-Reporter also has something similar. Once the CPAN Testers API is up and running, it will also be easier to query with a particular distribtion/version/platform/perl key, whether a result has been posted for that key, or for a distribution/platform/perl what is the highest version to receive a PASS. 5) Mailing Authors We have all pretty much agreed that having a central alerting system, that an author can set preferences against, is a good thing. At the moment too many authors are complaining of receiving too many reports. As such, with the new CPAN Testers Reports, David Wheeler's patch to allow for non-PASS.rss feeds will probably make life a little easier. At least for now. Personally I don't use any feed reader, and I suspect I'm not alone, so cutting off emails completely right now is likely to get FAIL reports going unnoticed by the very authors we actually want to reach. However, if we were to implement the FAIL => UNKNOW in (3) above, and stopped CCing any author on UNKNOWN, this may be reduce the flood enough to tide us over until the preferences and central alerting can be implemented. There is also the problem about multiple reports for the same platform/perl. As mentioned in (4) CPANPLUS already detects whether the author has been sent 2 FAIL reports and suppresses the CC to the author for any further. Again this will be easier to spot with the CPAN Testers API. However, we don't have that yet, but we do have the http://www.cpantesters.org/show/DISTRIBUTION.yml, which does include this information and is how CPANPLUS works. Could this be implemented in other smokers? Preferably in a way that can easily switched to the API once it's working. That's all I can think of at the moment, but there are probably other bits I've missed that need looking at. After a month of reworking code for the database, the reports website and the stats website, I'm pretty much drained and need a break. However, it seems others have the desire to belittle those efforts, so sorry if I don't feel like publishing the rest of the work anytime soon. Cheers, Barbie. -- Birmingham Perl Mongers <http://birmingham.pm.org> Memoirs Of A Roadie <http://barbie.missbarbell.co.uk>
