On Wed, Sep 3, 2008 at 7:18 PM, Barbie <[EMAIL PROTECTED]> wrote: > Okay following the various threads here is my take on things. > > 1) Design Decisions > > Perhaps now is the time to remove that decision and fail gracefully in > some way that allows the user and/or tools to know that a prerequisite > failure has occurred.
It looks like CPANPLUS dev versions are supposed to add CPANPLUS::Dist::Build as a configure_requires now. That doesn't seem to be working yet, but it doesn't create a Makefile.PL any longer. My bias here is to just suppress these FAILs for now and trust that future release of CPANPLUS will do the right thing. > 3) Build/Make FAILs > > This is a very tricky area. There is currently no well defined interface > for understanding failures at this stage. To reduce hassle for authors > we allowed 'exit 0' with no Makefile to mean that something went wrong, > how it's out of our control so ignore it. I know some disagree with exit > 0, but in these instances it's appropriate because you are terminating > the process in a controlled fashion. The ultimate goal of creating > Makefile hasn't happened, but the process has exited cleanly. If we > designing from scratch it might well have been useful to use to create > some standard error codes (perhaps similar to the HTTP codes), but > what's done is done. > > The current scenario of ignore any reports is the right thing to do > here, as the author is basically telling the user/tester, you cannot > procede without manually reviewing the reason for the failure. I agree it's tricky. I still think "exit 0" is appropriate, but I see how others disagree. Instead of changing that recommendation, I'd like to find a way that authors can just "die" if they feel it's more appropriate and that we'll handle that condition without FAIL as well. If we later decide that "exit 0" without Makefile or Build should be reported as UNKNOWN instead of being discarded, then we can change that later and not make authors have to fix anything. > Another irritation for authors is to receive reports for distributions > that have since been fixed. This has been much more of a problem in > recent times as many of the testers have recently built new environment > and are testing all of CPAN. I don't see the value of sending authors > reports for distributions that have since got a PASS report. This is If you mean don't test 1.01 when 1.02 has a PASS, then I agree. My approach with CPAN::Reporter::Smoker is to test only the latest release -- even if "latest" was 5+ years ago. I think we should discourage testing older distributions entirely. Note that this is potentially something that we could "enforce" by checking if a distribution is no longer the most current and discarding reports that refer to older releases. But that's tricky logic and I'd rather defer that until we have more centralized control. > We have all pretty much agreed that having a central alerting system, > that an author can set preferences against, is a good thing. At the > moment too many authors are complaining of receiving too many reports. > As such, with the new CPAN Testers Reports, David Wheeler's patch to > allow for non-PASS.rss feeds will probably make life a little easier. At > least for now. Personally I don't use any feed reader, and I suspect I'm > not alone, so cutting off emails completely right now is likely to get > FAIL reports going unnoticed by the very authors we actually want to > reach. > > However, if we were to implement the FAIL => UNKNOW in (3) above, and > stopped CCing any author on UNKNOWN, this may be reduce the flood enough > to tide us over until the preferences and central alerting can be > implemented. I'd rather see a smart central remailer that watches the incoming reports and emails an author about the *first* FAIL for a distribution-platform-perl tuple. The rest can go into the stats, but we can spare the author more than one FAIL report per platform per release. > There is also the problem about multiple reports for the same > platform/perl. As mentioned in (4) CPANPLUS already detects whether the > author has been sent 2 FAIL reports and suppresses the CC to the author > for any further. Again this will be easier to spot with the CPAN Testers > API. However, we don't have that yet, but we do have the > http://www.cpantesters.org/show/DISTRIBUTION.yml, which does include > this information and is how CPANPLUS works. Could this be implemented in > other smokers? Preferably in a way that can easily switched to the API > once it's working. As I said, I'd rather see no CC'ing at all and have one of us write a quick smart remailer as described on a server somewhere. (Sounds like a good excuse for me to finally learn POE, though I'd bet that BinGOs could hack one out in an afternoon.) However, I could add the CC surpression logic directly into Test::Reporter -- have it refuse to CC on PASS/NA/UNKNOWN and only CC on FAIL if there are 0-2 FAIL reports for that platform/perl in the YAML for the distribution. (Note -- that URL is .yaml, not .yml) That would at least collapse the effort to one module, not all the clients. I still think we should kill all CC'ing at the Test::Reporter level and set up a central remailer. It's crude compared to what we want for 2.0, but it'll be easier to improve and fix over time than having to get everyone to upgrade their clients yet again. > That's all I can think of at the moment, but there are probably other > bits I've missed that need looking at. After a month of reworking code > for the database, the reports website and the stats website, I'm pretty > much drained and need a break. However, it seems others have the desire > to belittle those efforts, so sorry if I don't feel like publishing the > rest of the work anytime soon. I understand how you can be drained -- that's a lot of work without (yet) many kudos. But I really do think that improving the reporting and stats will be seen as a positive and if we can quickly follow up with some triage to the false alarms, then I think we'll get some credit for our responsiveness and zeal. Regards, David
