Re: Measuring water quality of the CPAN river
On 11 May 2015, at 01:47, Kent Fredric kentfred...@gmail.com wrote: So the quality of a dist could be measured indirectly by the failure rate of its dependents. This was kind of the basis of the “River Smoker” idea that Tux and and I discussed late on the last day of the QAH: http://neilb.org/2015/04/24/cpan-downstream-testing.html Or as an analogy, we have 2 sampling points in the river 100m apart. If we sample at point 1, we can't sense the fecal matter because it enters the river downstream of the sampling point. But the fecal matter is sampled at point 2, so by conjecture, either point 2 created it, or it entered between points 1 and 2. Sampling across the river at point 2 helps infer where the probable fecal matter is. Sort of a cpan bisest, or rather a cpan testers bisect: look at 2 or more CPAN Testers fails where the only difference is an upriver version number. Neil
Re: Measuring water quality of the CPAN river
On 11 May 2015 at 19:20, Neil Bowers neil.bow...@cogendo.com wrote: look at 2 or more CPAN Testers fails where the only difference is an upriver version number. my point didn't pertain to upriver versions changing, but the observation that upriver modules can have buggy code that is only broken on certain architectures, and have no tests to expose that fact. Thus, any thing consuming that module, regardless the version its at, will have failure rates on CPAN for that architecture that the author of the downstream module didn't anticipate. But the problem is the upriver module, and its not a symptom exposed by upriver *changes*, but fundamental issues in that upriver was *alway* broken. Most common case of this I've seen is when upriver is only coded so it works on a specific NV size, and on different NV sizes , behaviour is unpredictable. Authors of downstream modules who have the magical NV size don't notice the problem and ship. And soon after, they get failures pertaining to their NV size *IF* they had tests. This can go on and on and you can get things 3+ deps away exposing an error that is truely in an upstream module simply due to the intermediate modules not exposing it due to their absence of tests. Obviously the accuracy of any such metric gets weaker the further it gets from the real point of origin. And even at D=2, its already rather vague. -- Kent *KENTNL* - https://metacpan.org/author/KENTNL
Re: Measuring water quality of the CPAN river
On 11 May 2015 at 12:37, Neil Bowers neil.bow...@cogendo.com wrote: These are pretty simple, and have some problems, so I’m hoping someone might come up with something more useful. Random thought while reading: backflow could be a thing to check for. Some dists don't have tests, and still have deps, causing a false-pass. So the quality of a dist could be measured indirectly by the failure rate of its depedents. Or as an analogy, we have 2 sampling points in the river 100m apart. If we sample at point 1, we can't sense the fecal matter because it enters the river downstream of the sampling point. But the fecal matter is sampled at point 2, so by conjecture, either point 2 created it, or it entered between points 1 and 2. Sampling across the river at point 2 helps infer where the probable fecal matter is. -- Kent *KENTNL* - https://metacpan.org/author/KENTNL
Measuring water quality of the CPAN river
One of the goals of the CPAN River model is to get us to focus on cleaning up the river, starting at the headwaters first. To that end, I’ve been thinking about how we might measure “river quality”, and have written up two ideas so far: http://neilb.org/2015/05/11/measuring-cpan-quality.html http://neilb.org/2015/05/11/measuring-cpan-quality.html These are pretty simple, and have some problems, so I’m hoping someone might come up with something more useful. Here are the 6 dists in the most upstream category that have = 2% failures on CPAN Testers podlators IO File-Path IPC-Cmd XSLoader Locale-Maketext-Simple I think I saw that File::Path was worked on at the NY hackathon last weekend, so maybe that will drop off this list soon ... Neil
Re: Measuring water quality of the CPAN river
On Sun, May 10, 2015 at 8:37 PM, Neil Bowers neil.bow...@cogendo.com wrote: These are pretty simple, and have some problems, so I’m hoping someone might come up with something more useful. It's really hard to examine specific cases with CT down at the moment, but I think we could be considering a number of dimensions. Brainstorming: * fails tests now – i.e. blocking automated dep installations – this is your existing metric * possibly an OS-specific variation of the above * stability over time – possibly your existing metric, but taking last N stable releases or all stable releases in last N years * something looking at bug queues – possibly open tickets proportional to number of downriver dependents; possibly looking at open vs close ratios * process metrics: e.g. trial release before stable * Kwalitee (might not tell us much, but easy to examine) * Proportional diff size between stable releases David -- David Golden x...@xdg.me Twitter/IRC: @xdg
