# from David Golden
# on Wednesday 03 September 2008 14:09:
> if ... CPAN Testers is ...
> to help authors improve quality
> rather than ...
> to give users a guarantee about ... any given platform.
>
>... "high quality" author -- ... tester has a broken or misconfigured
>toolchain The false positive ratio will approach 100%.
>
>__Repetition is desensitizing__ ... [more data will] improve confidence
>in the aggregate result. ... However, ... only need to hear it once.
>
>... fail during PL or make ... valuable information to
>know when distributions can't build ... [or] ... don't pass tests. ...
>not just skip reporting. ... [but] false positives that provoke
>complaint are toolchain issues during PL or make/Build.
>
>... no easy way to distinguish the phase ... subject of an email.
>
>I... partitions the FAIL space in a way that makes it easier for
>authors to focus on which ever part of the PL/make/test cycle...
Yes. Important points. Excellent assessment.
>__What we can fix now and what we can't__
>...
>However, as long as the CPAN Testers system has individual testers
>emailing authors, there is little we can do to address the problem of
>repetition.
This is true. For me, the mail is such a completely random data point
(no PASS mail, not all of the FAIL mail, etc) that I just delete it.
>Once that is more or less reliable, we could restart email
>notifications from that central source if people felt that nagging is
>critical to improve quality. Personally, I'm coming around to the
>idea that it's not the right way to go culturally for the community.
Send one mail to each new author telling them about the setup and how to
get rss or configure it to mail them, etc. Telling people about
something they don't know *once* is usually helpful.
But with the per-tester direct mail, the recipient is powerless to stop
it, and feeling powerless tends to make people angry.
In the bigger scheme of things, I sort of think all new PAUSE authors
should get a welcome e-basket with links, advice, etc, along with
perhaps an optional (default?) monthly newsletter about new tools or
something.
But, the trend generally seems to be away from mail and towards web
services. I'm in favor.
>Some of these proposal would be easier in CPAN Testers 2.0, which will
>provide reports as structured data instead of email text, but if "exit
>0" is a straw that is breaking the Perl camel's back now, then we
>can't ignore 1.0 to work on 2.0 as I'm not sure anyone will care
>anymore by the time it's done.
What are the issues in that time? Code or non-code?
>What we can't do easily is get the testers community to upgrade to
>newer versions of the tools. That is still going to be a matter of
>announcements and proselytizing and so on. But I think we can make a
>good case for it, and if we can get the top 10 or so testers to
>upgrade across all their testing machines then I think we'll make a
>huge dent in the false positives that are undermining support for CPAN
>Testers as a tool for Perl software quality.
Yes. Further, if you can detect new tools from old, you have enough
information to filter the results.
Hah! You know, you have their e-mail address, right? You can just send
them nagging mail about how their tools are FAIL! ;-)
So, seriously.... Testing is good. Knowing that weird platforms fail or
pass is good. Knowing that old toolchains fail is even useful to some
extent, but I think the significant variables are different for
different users.
So, what are these use cases?
Let's pretend that I'm a real jerk of an author and I only care about
whether my code installs on a perl 5.8.8+ (a *real* perl -- no funky
vendor patches) with a fully updated and properly configured toolchain
and the clock set to within 0.65s of NTP time. It would then be useful
for me to be able to mark (for all the world to see) which of the
machines reporting were using "supported" configurations. That would
both show *me* a valid fail and let my users know whether their
configuration was supported.
The user could still see passes and fails (hey, they could even perhaps
login and set a preference for what to show.) And remember that the
user is also often an author who is considering whether a dependency is
a good choice for them or not. If they need to support perl 4 and like
having wonky vendor perls and purposefully set their clock 5min fast
and configure CPAN.pm for prefer_installer => rand, seeing those
results are helpful -- but seeing my implied "unsupported
configuration" flag is also helpful.
Now, I'm not really a jerk, so I don't need the thing about the clock.
I can probably live with false results from some redhat perl for a while
too. But I'm going to pretend that we do not have a "broken or
impossible to upgrade" toolchain until it comes true. My users do not
have to share that delusion, but they do have to humor me.
So, what do you show the author and/or users by default? That is
possibly still rather sticky, but making it adjustable (perhaps
according to the maintainer by default) should allow reasonable people
to be happy enough.
>For those who read this to the end, thank you for your attention to
>what is surely becoming a tedious subject.
Thank you. Not at all tedious.
--Eric
--
[...proprietary software is better than gpl because...] "There is value
in having somebody you can write checks to, and they fix bugs."
--Mike McNamara (president of a commercial software company)
---------------------------------------------------
http://scratchcomputing.com
---------------------------------------------------
