Ricardo Signes <perl.test...@rjbs.manxome.org> writes: >> When a test passes with perl 5.14 and fails with 5.16, then we will >> probably be interested in the results with 5.15 (e.g. >> http://matrix.cpantesters.org/?dist=Attribute-Util%201.06 ) > > I'm not sure that this is really a good example. The question is whether > anyone should be *still* testing and sending reports against old dev versions. > > So, the way this would be useful is if: > > 1. we have known passes for 5.14 > 2. we have known failures for 5.16 > 3. we have /no existing reports/ for 5.15 > 4. it's useful to know what commit changed things from pass to fail > > The first three conditions need to be true, *and* somebody needs to be smoking > *both* the "still works" earlier 5.15.x *and* the "now fails" 5.15.(x+1), and > that still only gets you to a 30 day window of commits to test. > > It seems like a lot of noise with only a very, very small chance of producing > signal. Meanwhile, we have git-bisect to quickly get to the actual commit! > > I'm delighted to have those smoke reports rolling in during 5.15, though, so > we > CPAN authors can see what 5.16 might break. Once 5.16 is out the door, > though, > it hardly seems to have any value.
Needless to say, I fully agree with you. Thank you for spelling the full argument out so clearly. -- andreas
