On Fri, Feb 10, 2012 at 3:34 PM, Michael G Schwern <schw...@pobox.com> wrote: > I strongly disagree. It is absolutely appropriate to smoke alpha releases. > If we're not testing alphas then what's the point of an alpha? If you only > smoke stables, the damage is already done! > > The point of testing is to find bugs. This only happens if tests are allowed > to FAIL. If tests aren't failing, or if your system cannot gracefully handle > failing tests, you're NOT FINDING BUGS defeating the whole point of the > system.
The point of testing is not to find bugs. The point of testing is to provide information that helps people FIX bugs. Otherwise, why bother? Consider two types of failure reports: (1) dist D failure on a stable toolchain T (2) dist D failure on an alpha toolchain T' >From D's author's perspective, (1) is useful data and indicates things to be fixed. However (2) is annoying junk data -- not unlike any error due to a CPAN Testers's machine being borked (e.g. out of disk space). >From T's author's perspective, (2) is useful data, particularly when the result is different from (1). (i.e. regression testing) It indicates things to be fixed -- or at worst things that break back compatibility. Both are valuable, just not in the same data set, which you acknowledge. But if we pollute useful set (1) with results from (2), then authors will find CPAN Testers less useful and may start ignoring failure reports, which results in fewer bugs FIXED. Right now, we have a culture where we trust testers to have reasonably sane setups, rather than impose control through either rules or some technical solution. Could we automate sending alpha toolchain reports somewhere else? Given enough tuits, anything is possible. In the next few months I hope to make it trivial to set up Metabase clones for any sort of isolated data gathering. (Then someone just has to provide servers or funding for the same.) But at the moment, CPAN Testers is not a hammer to be used for any and all testing needs, so, yes, it would be nice to have people that are smoke testing with alpha toolchain modules please stop, since that take a lot fewer tuits and dollars to accomplish. David
