On Sep 2, 2008, at 4:09 PM, David Golden wrote:
On Tue, Sep 2, 2008 at 3:53 PM, Graham Barr <[EMAIL PROTECTED]> wrote:
exit if $ENV{AUTOMATED_TESTING};
Which removed the usefulness of those that do testing correctly and
submit
useful reports
My point was that authors can opt-out if "keeping up" is too annoying.
I would hope that authors that want the benefits of CPAN Testers
would be willing to put a tiny bit of extra effort in.
Actually CPAN testers has been getting so annoying not useful recent,
I am inclined to add that one line
Remember -- this whole thread started with "why exit 0?" Is that
really too much to ask an author with particularly unusual
requirements to learn and use?
Yes. The normal thing for any application is to exit non-zero when
there is an error.
An error during Makefile.PL or Build.PL is not necessarily an error
with the distribution. Calling die because of some system dependence
is perfectly valid thing todo.
If Makefile.PL start returning zero, when they really failed, then
anyone writing their own scripts can no longer depend on exit status
to determine if things went well and have to start probing for the
existence of file. That is just wrong.
Why should an error, which may not be related to the distribution, be
reported as a FAIL ?
Why does cpan testers have to ask authors to do non standard things
just to play nice with them ?
True. However I would contest that the Makefile.PL or Build.PL
cannot be
"known" to be a failure of the distribution, so the "Artificial
Intelligence" that you have programmed into CPAN testers is flawed.
The UNKNOWN response was specifically added to CPAN testers to
indicate that
it is not known if the distribution passes the test suite.
As far as I can tell, those definitions were never clearly specified
anywhere and so CPANPLUS became a form of "case law" that I followed
when creating CPAN::Reporter. (Now a definition exists on the wiki.)
When CPAN testers started a lot was done by agreement of a few people
who were doing the testing. It was when people started to automate it
and not bother to ask why things were done in particular ways and made
up their own definition.
Personally, I agree that having a single FAIL cover PL, make and test
phases is not particularly useful. I don't know whether using UNKNOWN
is the right thing for PL/make failures, but I'd be happy to make
changes if there were a sufficient consensus as to the right changes.
Since Barbie is the keeper of the stats (and now the
cpantesters.perl.org site), I'd like him to be in on that consensus,
of course.
First you need to define what a fail in the make stage is and I would
advocate that just a non-zero exit is not a failure.
For consistency we would need to get Jos Boumans to buy in as well and
patch CPANPLUS to make parallel changes or else to patch CPANPLUS to
use CPAN::Reporter (or an extract of its grading logic). That's part
of my long-term hopes for "CPAN Testers 2.0" (coming "by Christmas").
This is a discussion better taken forwards on cpan-testers-discuss, as
it's an entire debate in itself.
Well when distributions that DO specify in META.yml what
prerequisites are
needs and the distribution still has fail reports due to "Cannot
locate
Foo/Bar.pm" when it was in the META.yml then I consider that a bug
in the
testing and/or reporting and providing invalid results to the very
users
CPAN testers was intented to help
Cantrell pointed out that this can happen with system("perl ...") in
code or tests. I should add that I believe that both CPANPLUS and
CPAN::Reporter discard reports instead of sending if they notice that
any prerequisites haven't been satisfied. For CPAN::Reporter, this
even includes configure_requires.
Then why do I still keep getting such reports ?
Graham.
