Here is the tl;dr version:
I agree there are two use cases here. I don't think the solution requires
setting up a whole parallel test gathering system, alpha modules don't pollute
CPAN test data any more than alpha perls do, they pollute the reports. The
data is fine, the reporting needs to change.
I'm pretty sure Metabase has enough resolution right now to determine when a
test has an alpha dependency, maybe not all the way down the dependency chain
but at least direct dependencies. It doesn't seem any different from how
tests from alpha perls are currently handled. That seems a whole lot simpler
than setting up a parallel test gathering system, and reporting servers, and
getting people to use it.
Unless I'm not understanding what a Metabase clone would be?
A simple first-order fix would be to add something to the report emails
indicating if a report contains an alpha dependency. Something people can
easily filter on, either with eyeballs or robots.
A second-order approximation would be to add a switch to choose whether to
include reports with alpha dependencies, just like is already done with alpha
perls. Default it to off. Then authors who don't want to be bothered won't
see the alpha failures and they can keep their green bars.
On 2012.2.10 2:02 PM, David Golden wrote:
> The point of testing is not to find bugs. The point of testing is to
> provide information that helps people FIX bugs. Otherwise, why
> bother?
A refinement rather than a disagreement. :)
> Consider two types of failure reports:
>
> (1) dist D failure on a stable toolchain T
>
> (2) dist D failure on an alpha toolchain T'
>
>>From D's author's perspective, (1) is useful data and indicates things
> to be fixed. However (2) is annoying junk data -- not unlike any
> error due to a CPAN Testers's machine being borked (e.g. out of disk
> space).
I don't entirely agree. It presumes that T' is at fault, which is often true,
but it may be that D is using some undocumented aspect of T which has changed
in T'.
It is also useful for D to be aware that T might about to release a new
version that breaks D, no matter what the reason for the change, and act
appropriately. That could be informing T or altering D.
Either way, D is affected if T goes stable without the problem being fixed.
>>From T's author's perspective, (2) is useful data, particularly when
> the result is different from (1). (i.e. regression testing) It
> indicates things to be fixed -- or at worst things that break back
> compatibility.
On this we agree, tests showing a difference between (1) and (2) are very
important information to T.
> Both are valuable, just not in the same data set, which you
> acknowledge. But if we pollute useful set (1) with results from (2),
> then authors will find CPAN Testers less useful and may start ignoring
> failure reports, which results in fewer bugs FIXED.
...
> Could we automate sending alpha toolchain reports somewhere else?
> Given enough tuits, anything is possible. In the next few months I
> hope to make it trivial to set up Metabase clones for any sort of
> isolated data gathering. (Then someone just has to provide servers or
> funding for the same.)
I agree there are two use cases here. I don't think the solution requires
setting up a whole parallel test gathering system, the data remains the same.
It's the reporting that needs to change. I'm pretty sure Metabase has enough
resolution right now to determine when a test has an alpha dependency. It
doesn't seem any different from how tests from alpha perls are currently
handled. That seems a whole lot simpler than setting up a parallel test
gathering system, and reporting servers, and getting people to use it.
Unless I'm not understanding what a Metabase clone is?
> But at the moment, CPAN Testers is not a hammer to be used for any and
> all testing needs, so, yes, it would be nice to have people that are
> smoke testing with alpha toolchain modules please stop, since that
> take a lot fewer tuits and dollars to accomplish.
I would say the integrity of the toolchain is more important than authors
getting some extra emails. I'd rather they got some "false" negatives before
a broken stable toolchain release than real failures after a stable toolchain
release.
If Andreas hadn't pointed out the 100+ dependency failures that the CPAN
Testers analysis tools discovered, I'd have thought TB1.5 was a lot closer to
release. The CPAN cascade failure from a premature TB1.5 release is
unthinkable. We've experienced it before. 0.71 fixed use_ok() but caused
dependency havoc. It took six days to release a fix. Before that there was
0.61 which changed failure diagnostic reporting, broke Test::Builder::Tester
and the cascade took out a chunk of CPAN. That took two weeks to fix. All of
this would have been avoided with alpha smoking.
A simple first-order fix would be to add something to the report emails
indicating if a report contains an alpha dependency. Something people can
easily filter on, either with eyeballs or robots.
--
You are wicked and wrong to have broken inside and peeked at the
implementation and then relied upon it.
-- tchrist in <31832.969261130@chthon>
