On Tue, Sep 2, 2008 at 3:53 PM, Graham Barr <[EMAIL PROTECTED]> wrote:
>> exit if $ENV{AUTOMATED_TESTING};
>
> Which removed the usefulness of those that do testing correctly and submit
> useful reports
My point was that authors can opt-out if "keeping up" is too annoying.
I would hope that authors that want the benefits of CPAN Testers
would be willing to put a tiny bit of extra effort in.
Remember -- this whole thread started with "why exit 0?" Is that
really too much to ask an author with particularly unusual
requirements to learn and use?
> True. However I would contest that the Makefile.PL or Build.PL cannot be
> "known" to be a failure of the distribution, so the "Artificial
> Intelligence" that you have programmed into CPAN testers is flawed.
>
> The UNKNOWN response was specifically added to CPAN testers to indicate that
> it is not known if the distribution passes the test suite.
As far as I can tell, those definitions were never clearly specified
anywhere and so CPANPLUS became a form of "case law" that I followed
when creating CPAN::Reporter. (Now a definition exists on the wiki.)
Personally, I agree that having a single FAIL cover PL, make and test
phases is not particularly useful. I don't know whether using UNKNOWN
is the right thing for PL/make failures, but I'd be happy to make
changes if there were a sufficient consensus as to the right changes.
Since Barbie is the keeper of the stats (and now the
cpantesters.perl.org site), I'd like him to be in on that consensus,
of course.
For consistency we would need to get Jos Boumans to buy in as well and
patch CPANPLUS to make parallel changes or else to patch CPANPLUS to
use CPAN::Reporter (or an extract of its grading logic). That's part
of my long-term hopes for "CPAN Testers 2.0" (coming "by Christmas").
This is a discussion better taken forwards on cpan-testers-discuss, as
it's an entire debate in itself.
> Well when distributions that DO specify in META.yml what prerequisites are
> needs and the distribution still has fail reports due to "Cannot locate
> Foo/Bar.pm" when it was in the META.yml then I consider that a bug in the
> testing and/or reporting and providing invalid results to the very users
> CPAN testers was intented to help
Cantrell pointed out that this can happen with system("perl ...") in
code or tests. I should add that I believe that both CPANPLUS and
CPAN::Reporter discard reports instead of sending if they notice that
any prerequisites haven't been satisfied. For CPAN::Reporter, this
even includes configure_requires.
David
