I don't think I have time for this today, but I wanted to write it down. The biggest problem we have with traditional reporting of bug counts in bugzilla is that inflow and outflow is largely determined by qualities other than actual code quality. In traditional bug tracking, number of QA people is relatively fixed, number of tests run relatively fixed/known, and amount of triage and fixing are also known. On the other hand, we have spikes of all sorts- bugday? yup, that made us X% less buggy in one day. Yeah. Right.[1] Sun stops filing bugs against HEAD? Or suddenly pours more resources into triage? Did our fix rate really go up, or the quality of new releases really go down, if those things happen? not really. So we have to figure out some way to control for those things.
We've traditionally had no way of controlling for these things, which has always made me very reluctant to use our stats for anything important. However, it struck me yesterday that most of these factors are fairly constant across all of our core desktop apps. Bugday? we look at all apps. Sun/Novell/RedHat/Ubuntu starts/stops paying attention? Fairly constant across all apps. New distro releases? Fairly constant across all apps. So... I think maybe a really interesting stat to look at to find 'ailing' apps is 'percentage of open bugs in the same class of apps.' What I'd like to see is a graph of where the X axis is 'percentage of all open bugs that are open in this application' and Y is date. So, for example, data points for today: * isgnome open bugs (inc. unconfirmed): 10595 * evo open bugs (ditto): 2503, 23.6% of total * nautilus open bugs (etc.): 847, 7.99% of total etc. So, get that percentage data over time (not easy, unfortunately, but probably not impossible) and graph it. Was evo 26% of open bugs last week, and now 23%? Very good sign for them. Nautilus was 7%, now 8%? Not good for them. The trendlines you'll see there will indicate which apps are improving and which declining in terms of quality, in a way that controls for the worst fluctuations that our open system invariably produces. Could also whip this into meaningful reports for maintainers- 'controlled for other fluctuations, your # of open bugs was up/down foo% last week.' Seems like that would be good. Or so it seems to me. If anyone wants to whip this up it would rock ;) Luis (can make database dumps available on request for people who want to play with this) [1] It *did* make the bugtrack less full of bugs by a nice amount, but that is not the same as less buggy. _______________________________________________ Gnome-bugsquad mailing list [email protected] http://mail.gnome.org/mailman/listinfo/gnome-bugsquad
