On Thu, Sep 4, 2008 at 10:18 AM, David Cantrell <[EMAIL PROTECTED]> wrote: > On Thu, Sep 04, 2008 at 12:18:57AM +0100, Barbie wrote: >> Going back to Graham's vision, perhaps UNKNOWN is appropriate here. > > I'd not want to overload the meaning of 'unknown' here. Splitting FAIL > into PL_FAIL, MAKE_FAIL and (TEST_)FAIL would be better, I think.
In the long-run, I agree. In the short-run, doing so would break all the stats/matrix/reporting tools that expect only PASS, FAIL, NA and UNKNOWN. Since UNKNOWN is still relatively rare (~2%) and is a fairly useless grade anyway, I think that re-using it for PL/make failures is a quick win because (mostly) only the clients have to be changed. > This is partly because CPANdeps assumes that UNKNOWN means that the > module will install, for the purposes of calculating its half-arsed > probability of success. This sounds like the one other thing that would need to be changed. > However, PL_FAIL and MAKE_FAIL should never be sent automatically - they > should receive at least a cursory look from the tester, as they are > almost always things like trying to install DBD::mysql without the MySQL > libraries and headers. That works for manual testers, but I'm not sure it'll get traction with high volume smoke testers. -- David
