On 7/5/07, Barbie <[EMAIL PROTECTED]> wrote:
It is documented on the wiki:
http://cpantest.grango.org/cgi-bin/pages.cgi?act=wiki-page&pagename=Reports
And was previously available in Audrey's adapted perl.com article, which
was bundled with the pre-0.50 CPANPLUS documentation.
Operative word is "was". Thankfully, I think the wiki can now become
definitive documentation with more than a sentence fragment of
explanation cut & pasted between articles.
> More generally, I think it's a design flaw to not have a grade to
> capture things that don't fall cleanly into one or another category.
This was something that was discussed several years ago. My suggestion
was for an 'UNGRADED' report (although thats a bad choice for a name),
which could indicate something didn't work right, but that it's unclear
whether the environment or distribution is at fault. The feeling was
for the reports to be ignored and for the testers to either investigate
further or email just the author for clarification.
With increased smoking and automation of testing with
CPANPLUS/CPAN::Reporter, testers just aren't investigating. I think
'UNGRADED' or 'NOGRADE' would work. It's better than 'ERROR' or
anything like that. It clearly signals that something went wrong.
Stats could ignore it, but I'd like to see it hit the list and get
procmail filtered to my inbox so I can investigate.
> Too much of what happens in determining grades is heuristics -- just
> discarding things that don't fit make the whole system brittle.
Discarding reports that are due to the tester or tool chain ignoring
the requirements from the author (such as failing to install a
prerequisite), is reasonable. Anything else that isn't NA or UNKNOWN
described above should be a FAIL.
Defaulting to FAIL will certainly prompt authors to complain if they
think the testing tools got it wrong. That will help surface bugs.
Personally, I think UNKNOWN is a nice way to do that, but that's open
to debate.
Out of curiosity, I just looked at CPANPLUS::Internals::Report and it
looks like it also downgrades FAIL to NA if specified prereqs are not
satisfied. (From the Changelog, it may have been inspired by
CPAN::Reporter.)
To my mind the 4 grades are appropriate for all the scenarios. Expanding
these or making sub-grades is making work for the author, CPAN Testers,
the tools, the reporting mechanisms and the data presentation websites.
If that is the binding constraint, I'd suggest using 'UNKNOWN' as the
fallback grade if a grade couldn't be otherwise determined by the
testing tools. It's rare enough anyway that it's not going to have
much impact on overall testing statistics and it's less annoying to
authors (who sometimes take FAIL personally). And the meaning is
clear. "No tests" fall into that same general category -- the testing
tools couldn't determine a result. This would broaden rather than
*change* the definition.
For unsatisfied prerequisites, I think it makes sense for the testing
tools to just discard the result and that's on my Todo list for
CPAN::Reporter.
If the report itself contains more information, that's fine, but I still
think the basic grades have some value as they are now. Changing that is
going to lead to confusion. Previous reports are unlikely to get
regraded.
I think the addition of a few extra UNKNOWNS won't cause confusion.
As I said, they should be quite rare.
I think a seperate mailing list is better for discussion, but the wiki
is the better place to record the outcomes of those discussions. I'll
see about setting something up.
I'd suggest that the conclusions reached should be captured on the
wiki with some of the same "should", "should not", "may", "must", etc.
language used for RFC's. E.g. Testing tools may send fail reports to
authors. Testing tools should not send pass reports to authors.
Testing tools must discard test results with unsatisfied
prerequisites.
What do you think?
David
