Re: [openstack-dev] [nova] 2 weeks in the bug tracker
On Mon, Sep 22, 2014 at 7:51 AM, Jay S. Bryant wrote: > > On 09/21/2014 07:37 PM, Matt Riedemann wrote: >> >> When I'm essentially +2 on a change but for a small issue like typos in >> the commit message, the need for a note in the code or a test (or change to >> a test), I've been doing those myself lately and then will give the +2. If >> the change already has a +2 and I'd be +W but for said things, I'm more >> inclined lately to approve and then push a dependent patch on top of it with >> the changes to keep things from stalling. >> >> This might be a change in my workflow just because we're late in the >> release and want good bug fixes getting into the release candidates, it >> could be because of the weekly tirade of how the project is going down the >> toilet and we don't get enough things reviewed/approved, I'm not sure, but >> my point is I agree with making it socially acceptable to rewrite the commit >> message as part of the review. > > Matt, > > This is consistent with what I have been doing for Cinder as well. I know > there are some people who prefer I not touch the commit messages and I > respect those requests, but otherwise I make changes to keep the process > moving. > > Jay This is also consistent with how I've been doing things in Ironic and I have been encouraging the core team to use their judgement when doing this as well -- especially when it's a patch from someone we know won't get back to it for a while (eg, because they're on vacation) or someone that has already OK'd this workflow (eg, other members of the core team, and regular developers we know). -Devananda ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] 2 weeks in the bug tracker
On Sep 23, 2014, at 1:24 PM, Mark McClain wrote: > > On Sep 19, 2014, at 9:13 AM, Sean Dague wrote: > >> Cross interaction with Neutron and Cinder remains racey. We are pretty >> optimistic on when resources will be available. Even the event interface >> with Neutron hasn't fully addressed this. I think a really great Design >> Summit session would be Nova + Neutron + Cinder to figure out a shared >> architecture to address this. I'd expect this to be at least a double >> session. > > +1000 > > mark It’s possible we could add some features to tooz (a new Oslo synchronization library) to help with this. Julien is the lead on tooz. Doug ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] 2 weeks in the bug tracker
On Sep 19, 2014, at 9:13 AM, Sean Dague wrote: > Cross interaction with Neutron and Cinder remains racey. We are pretty > optimistic on when resources will be available. Even the event interface > with Neutron hasn't fully addressed this. I think a really great Design > Summit session would be Nova + Neutron + Cinder to figure out a shared > architecture to address this. I'd expect this to be at least a double > session. +1000 mark___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] 2 weeks in the bug tracker
On Fri, 2014-09-19 at 09:13 -0400, Sean Dague wrote: > I've spent the better part of the last 2 weeks in the Nova bug tracker > to try to turn it into something that doesn't cause people to run away > screaming. I don't remember exactly where we started at open bug count 2 > weeks ago (it was north of 1400, with > 200 bugs in new, but it might > have been north of 1600), but as of this email we're at < 1000 open bugs > (I'm counting Fix Committed as closed, even though LP does not), and ~0 > new bugs (depending on the time of the day). > > == Philosophy in Triaging == > > I'm going to lay out the philosophy of triaging I've had, because this > may also set the tone going forward. > > A bug tracker is a tool to help us make a better release. It does not > exist for it's own good, it exists to help. Which means when evaluating > what stays in and what leaves we need to evaluate if any particular > artifact will help us make a better release. But also more importantly > realize that there is a cost for carrying every artifact in the tracker. > Resolving duplicates gets non linearly harder as the number of artifacts > go up. Triaging gets non-linearly hard as the number of artifacts go up. > > With this I was being somewhat pragmatic about closing bugs. An old bug > that is just a stacktrace is typically not useful. An old bug that is a > vague sentence that we should refactor a particular module (with no > specifics on the details) is not useful. A bug reported against a very > old version of OpenStack where the code has changed a lot in the > relevant area, and there aren't responses from the author, is not > useful. Not useful bugs just add debt, and we should get rid of them. > That makes the chance of pulling a random bug off the tracker something > that you could actually look at fixing, instead of mostly just stalling out. > > So I closed a lot of stuff as Invalid / Opinion that fell into those camps. > > == Keeping New Bugs at close to 0 == > > After driving the bugs in the New state down to zero last week, I found > it's actually pretty easy to keep it at 0. > > We get 10 - 20 new bugs a day in Nova (during a weekday). Of those ~20% > aren't actually a bug, and can be closed immediately. ~30% look like a > bug, but don't have anywhere near enough information in them, and > flipping them to incomplete with questions quickly means we have a real > chance of getting the right info. ~10% are fixable in < 30 minutes worth > of work. And the rest are real bugs, that seem to have enough to dive > into it, and can be triaged into Confirmed, set a priority, and add the > appropriate tags for the area. > > But, more importantly, this means we can filter bug quality on the way > in. And we can also encourage bug reporters that are giving us good > stuff, or even easy stuff, as we respond quickly. > > Recommendation #1: we adopt a 0 new bugs policy to keep this from > getting away from us in the future. > > == Our worse bug reporters are often core reviewers == > > I'm going to pick on Dan Prince here, mostly because I have a recent > concrete example, however in triaging the bug queue much of the core > team is to blame (including myself). > > https://bugs.launchpad.net/nova/+bug/1368773 is a terrible bug. Also, it > was set incomplete and no response. I'm almost 100% sure it's a dupe of > the multiprocess bug we've been tracking down but it's so terse that you > can't get to the bottom of it. This bug was filed as a result of a cryptic (to me at the time) gate unit test failure that occurred in this review: https://review.openstack.org/#/c/120099/ I mistakenly grabbed the last timeout error instead of looking at the original timeout. Within 30 minutes or so of my post Matt Riedemann had correctly classified it as https://bugs.launchpad.net/nova/+bug/1357578 I've added some extra data and marked it as a dup. Dan > > There were a ton of 2012 nova bugs that were basically "post it notes". > Oh, "we should refactor this function". Full stop. While those are fine > for personal tracking, their value goes to zero probably 3 months after > they are files, especially if the reporter stops working on the issue at > hand. Nova has plenty of "wouldn't it be great if we... " ideas. I'm not > convinced using bugs for those is useful unless we go and close them out > aggressively if they stall. > > Also, if Nova core can't file a good bug, it's hard to set the example > for others in our community. > > Recommendation #2: hey, Nova core, lets be better about filing the kinds > of bugs we want to see! mkay! > > Recommendation #3: Let's create a tag for "personal work items" or > something for these class of TODOs people are leaving themselves that > make them a ton easier to cull later when they stall and no one else has > enough context to pick them up. > > == Tags == > > The aggressive tagging that Tracy brought into the project has been > awesome. It definitely helps slice out into better function
Re: [openstack-dev] [nova] 2 weeks in the bug tracker
On 09/21/2014 07:37 PM, Matt Riedemann wrote: On 9/19/2014 8:13 AM, Sean Dague wrote: I've spent the better part of the last 2 weeks in the Nova bug tracker to try to turn it into something that doesn't cause people to run away screaming. I don't remember exactly where we started at open bug count 2 weeks ago (it was north of 1400, with > 200 bugs in new, but it might have been north of 1600), but as of this email we're at < 1000 open bugs (I'm counting Fix Committed as closed, even though LP does not), and ~0 new bugs (depending on the time of the day). == Philosophy in Triaging == I'm going to lay out the philosophy of triaging I've had, because this may also set the tone going forward. A bug tracker is a tool to help us make a better release. It does not exist for it's own good, it exists to help. Which means when evaluating what stays in and what leaves we need to evaluate if any particular artifact will help us make a better release. But also more importantly realize that there is a cost for carrying every artifact in the tracker. Resolving duplicates gets non linearly harder as the number of artifacts go up. Triaging gets non-linearly hard as the number of artifacts go up. With this I was being somewhat pragmatic about closing bugs. An old bug that is just a stacktrace is typically not useful. An old bug that is a vague sentence that we should refactor a particular module (with no specifics on the details) is not useful. A bug reported against a very old version of OpenStack where the code has changed a lot in the relevant area, and there aren't responses from the author, is not useful. Not useful bugs just add debt, and we should get rid of them. That makes the chance of pulling a random bug off the tracker something that you could actually look at fixing, instead of mostly just stalling out. So I closed a lot of stuff as Invalid / Opinion that fell into those camps. == Keeping New Bugs at close to 0 == After driving the bugs in the New state down to zero last week, I found it's actually pretty easy to keep it at 0. We get 10 - 20 new bugs a day in Nova (during a weekday). Of those ~20% aren't actually a bug, and can be closed immediately. ~30% look like a bug, but don't have anywhere near enough information in them, and flipping them to incomplete with questions quickly means we have a real chance of getting the right info. ~10% are fixable in < 30 minutes worth of work. And the rest are real bugs, that seem to have enough to dive into it, and can be triaged into Confirmed, set a priority, and add the appropriate tags for the area. But, more importantly, this means we can filter bug quality on the way in. And we can also encourage bug reporters that are giving us good stuff, or even easy stuff, as we respond quickly. Recommendation #1: we adopt a 0 new bugs policy to keep this from getting away from us in the future. == Our worse bug reporters are often core reviewers == I'm going to pick on Dan Prince here, mostly because I have a recent concrete example, however in triaging the bug queue much of the core team is to blame (including myself). https://bugs.launchpad.net/nova/+bug/1368773 is a terrible bug. Also, it was set incomplete and no response. I'm almost 100% sure it's a dupe of the multiprocess bug we've been tracking down but it's so terse that you can't get to the bottom of it. There were a ton of 2012 nova bugs that were basically "post it notes". Oh, "we should refactor this function". Full stop. While those are fine for personal tracking, their value goes to zero probably 3 months after they are files, especially if the reporter stops working on the issue at hand. Nova has plenty of "wouldn't it be great if we... " ideas. I'm not convinced using bugs for those is useful unless we go and close them out aggressively if they stall. Also, if Nova core can't file a good bug, it's hard to set the example for others in our community. Recommendation #2: hey, Nova core, lets be better about filing the kinds of bugs we want to see! mkay! Recommendation #3: Let's create a tag for "personal work items" or something for these class of TODOs people are leaving themselves that make them a ton easier to cull later when they stall and no one else has enough context to pick them up. == Tags == The aggressive tagging that Tracy brought into the project has been awesome. It definitely helps slice out into better functional areas. Here is the top of our current official tag list (and bug count): 95 compute 83 libvirt 74 api 68 vmware 67 network 41 db 40 testing 40 volumes 36 ec2 35 icehouse-backport-potential 32 low-hanging-fruit 31 xenserver 25 ironic 23 hyper-v 16 cells 14 scheduler 12 baremetal 9 ceph 9 security 8 oslo ... So, good stuff. However I think we probably want to take a further step and attempt to get champions for tags. So that tag owners would ensure their bug list looks sane, and actually spend some time fixing them. It's pretty clear, for instance, that the e
Re: [openstack-dev] [nova] 2 weeks in the bug tracker
On 9/19/2014 8:13 AM, Sean Dague wrote: I've spent the better part of the last 2 weeks in the Nova bug tracker to try to turn it into something that doesn't cause people to run away screaming. I don't remember exactly where we started at open bug count 2 weeks ago (it was north of 1400, with > 200 bugs in new, but it might have been north of 1600), but as of this email we're at < 1000 open bugs (I'm counting Fix Committed as closed, even though LP does not), and ~0 new bugs (depending on the time of the day). == Philosophy in Triaging == I'm going to lay out the philosophy of triaging I've had, because this may also set the tone going forward. A bug tracker is a tool to help us make a better release. It does not exist for it's own good, it exists to help. Which means when evaluating what stays in and what leaves we need to evaluate if any particular artifact will help us make a better release. But also more importantly realize that there is a cost for carrying every artifact in the tracker. Resolving duplicates gets non linearly harder as the number of artifacts go up. Triaging gets non-linearly hard as the number of artifacts go up. With this I was being somewhat pragmatic about closing bugs. An old bug that is just a stacktrace is typically not useful. An old bug that is a vague sentence that we should refactor a particular module (with no specifics on the details) is not useful. A bug reported against a very old version of OpenStack where the code has changed a lot in the relevant area, and there aren't responses from the author, is not useful. Not useful bugs just add debt, and we should get rid of them. That makes the chance of pulling a random bug off the tracker something that you could actually look at fixing, instead of mostly just stalling out. So I closed a lot of stuff as Invalid / Opinion that fell into those camps. == Keeping New Bugs at close to 0 == After driving the bugs in the New state down to zero last week, I found it's actually pretty easy to keep it at 0. We get 10 - 20 new bugs a day in Nova (during a weekday). Of those ~20% aren't actually a bug, and can be closed immediately. ~30% look like a bug, but don't have anywhere near enough information in them, and flipping them to incomplete with questions quickly means we have a real chance of getting the right info. ~10% are fixable in < 30 minutes worth of work. And the rest are real bugs, that seem to have enough to dive into it, and can be triaged into Confirmed, set a priority, and add the appropriate tags for the area. But, more importantly, this means we can filter bug quality on the way in. And we can also encourage bug reporters that are giving us good stuff, or even easy stuff, as we respond quickly. Recommendation #1: we adopt a 0 new bugs policy to keep this from getting away from us in the future. == Our worse bug reporters are often core reviewers == I'm going to pick on Dan Prince here, mostly because I have a recent concrete example, however in triaging the bug queue much of the core team is to blame (including myself). https://bugs.launchpad.net/nova/+bug/1368773 is a terrible bug. Also, it was set incomplete and no response. I'm almost 100% sure it's a dupe of the multiprocess bug we've been tracking down but it's so terse that you can't get to the bottom of it. There were a ton of 2012 nova bugs that were basically "post it notes". Oh, "we should refactor this function". Full stop. While those are fine for personal tracking, their value goes to zero probably 3 months after they are files, especially if the reporter stops working on the issue at hand. Nova has plenty of "wouldn't it be great if we... " ideas. I'm not convinced using bugs for those is useful unless we go and close them out aggressively if they stall. Also, if Nova core can't file a good bug, it's hard to set the example for others in our community. Recommendation #2: hey, Nova core, lets be better about filing the kinds of bugs we want to see! mkay! Recommendation #3: Let's create a tag for "personal work items" or something for these class of TODOs people are leaving themselves that make them a ton easier to cull later when they stall and no one else has enough context to pick them up. == Tags == The aggressive tagging that Tracy brought into the project has been awesome. It definitely helps slice out into better functional areas. Here is the top of our current official tag list (and bug count): 95 compute 83 libvirt 74 api 68 vmware 67 network 41 db 40 testing 40 volumes 36 ec2 35 icehouse-backport-potential 32 low-hanging-fruit 31 xenserver 25 ironic 23 hyper-v 16 cells 14 scheduler 12 baremetal 9 ceph 9 security 8 oslo ... So, good stuff. However I think we probably want to take a further step and attempt to get champions for tags. So that tag owners would ensure their bug list looks sane, and actually spend some time fixing them. It's pretty clear, for instance, that the ec2 bugs are just piling up, and very few fixes comin
Re: [openstack-dev] [nova] 2 weeks in the bug tracker
I think we are limited by the statuses available in launchpad, which doesn't have a stale status. [1] 1. https://help.launchpad.net/Bugs/Statuses On Sat, Sep 20, 2014 at 7:25 PM, Preston L. Bannister wrote: > This is great. On the point of: > >> If an Incomplete bug has no response after 30 days it's fair game to >> close (Invalid, Opinion, Won't Fix). > > > How about "Stale" ... since that is where it is. (How hard to add a state?) > > > > > On Fri, Sep 19, 2014 at 6:13 AM, Sean Dague wrote: >> >> I've spent the better part of the last 2 weeks in the Nova bug tracker >> to try to turn it into something that doesn't cause people to run away >> screaming. I don't remember exactly where we started at open bug count 2 >> weeks ago (it was north of 1400, with > 200 bugs in new, but it might >> have been north of 1600), but as of this email we're at < 1000 open bugs >> (I'm counting Fix Committed as closed, even though LP does not), and ~0 >> new bugs (depending on the time of the day). >> >> == Philosophy in Triaging == >> >> I'm going to lay out the philosophy of triaging I've had, because this >> may also set the tone going forward. >> >> A bug tracker is a tool to help us make a better release. It does not >> exist for it's own good, it exists to help. Which means when evaluating >> what stays in and what leaves we need to evaluate if any particular >> artifact will help us make a better release. But also more importantly >> realize that there is a cost for carrying every artifact in the tracker. >> Resolving duplicates gets non linearly harder as the number of artifacts >> go up. Triaging gets non-linearly hard as the number of artifacts go up. >> >> With this I was being somewhat pragmatic about closing bugs. An old bug >> that is just a stacktrace is typically not useful. An old bug that is a >> vague sentence that we should refactor a particular module (with no >> specifics on the details) is not useful. A bug reported against a very >> old version of OpenStack where the code has changed a lot in the >> relevant area, and there aren't responses from the author, is not >> useful. Not useful bugs just add debt, and we should get rid of them. >> That makes the chance of pulling a random bug off the tracker something >> that you could actually look at fixing, instead of mostly just stalling >> out. >> >> So I closed a lot of stuff as Invalid / Opinion that fell into those >> camps. >> >> == Keeping New Bugs at close to 0 == >> >> After driving the bugs in the New state down to zero last week, I found >> it's actually pretty easy to keep it at 0. >> >> We get 10 - 20 new bugs a day in Nova (during a weekday). Of those ~20% >> aren't actually a bug, and can be closed immediately. ~30% look like a >> bug, but don't have anywhere near enough information in them, and >> flipping them to incomplete with questions quickly means we have a real >> chance of getting the right info. ~10% are fixable in < 30 minutes worth >> of work. And the rest are real bugs, that seem to have enough to dive >> into it, and can be triaged into Confirmed, set a priority, and add the >> appropriate tags for the area. >> >> But, more importantly, this means we can filter bug quality on the way >> in. And we can also encourage bug reporters that are giving us good >> stuff, or even easy stuff, as we respond quickly. >> >> Recommendation #1: we adopt a 0 new bugs policy to keep this from >> getting away from us in the future. >> >> == Our worse bug reporters are often core reviewers == >> >> I'm going to pick on Dan Prince here, mostly because I have a recent >> concrete example, however in triaging the bug queue much of the core >> team is to blame (including myself). >> >> https://bugs.launchpad.net/nova/+bug/1368773 is a terrible bug. Also, it >> was set incomplete and no response. I'm almost 100% sure it's a dupe of >> the multiprocess bug we've been tracking down but it's so terse that you >> can't get to the bottom of it. >> >> There were a ton of 2012 nova bugs that were basically "post it notes". >> Oh, "we should refactor this function". Full stop. While those are fine >> for personal tracking, their value goes to zero probably 3 months after >> they are files, especially if the reporter stops working on the issue at >> hand. Nova has plenty of "wouldn't it be great if we... " ideas. I'm not >> convinced using bugs for those is useful unless we go and close them out >> aggressively if they stall. >> >> Also, if Nova core can't file a good bug, it's hard to set the example >> for others in our community. >> >> Recommendation #2: hey, Nova core, lets be better about filing the kinds >> of bugs we want to see! mkay! >> >> Recommendation #3: Let's create a tag for "personal work items" or >> something for these class of TODOs people are leaving themselves that >> make them a ton easier to cull later when they stall and no one else has >> enough context to pick them up. >> >> == Tags == >> >> The aggressive tagging that Tracy brou
Re: [openstack-dev] [nova] 2 weeks in the bug tracker
This is great. On the point of: If an Incomplete bug has no response after 30 days it's fair game to > close (Invalid, Opinion, Won't Fix). How about "Stale" ... since that is where it is. (How hard to add a state?) On Fri, Sep 19, 2014 at 6:13 AM, Sean Dague wrote: > I've spent the better part of the last 2 weeks in the Nova bug tracker > to try to turn it into something that doesn't cause people to run away > screaming. I don't remember exactly where we started at open bug count 2 > weeks ago (it was north of 1400, with > 200 bugs in new, but it might > have been north of 1600), but as of this email we're at < 1000 open bugs > (I'm counting Fix Committed as closed, even though LP does not), and ~0 > new bugs (depending on the time of the day). > > == Philosophy in Triaging == > > I'm going to lay out the philosophy of triaging I've had, because this > may also set the tone going forward. > > A bug tracker is a tool to help us make a better release. It does not > exist for it's own good, it exists to help. Which means when evaluating > what stays in and what leaves we need to evaluate if any particular > artifact will help us make a better release. But also more importantly > realize that there is a cost for carrying every artifact in the tracker. > Resolving duplicates gets non linearly harder as the number of artifacts > go up. Triaging gets non-linearly hard as the number of artifacts go up. > > With this I was being somewhat pragmatic about closing bugs. An old bug > that is just a stacktrace is typically not useful. An old bug that is a > vague sentence that we should refactor a particular module (with no > specifics on the details) is not useful. A bug reported against a very > old version of OpenStack where the code has changed a lot in the > relevant area, and there aren't responses from the author, is not > useful. Not useful bugs just add debt, and we should get rid of them. > That makes the chance of pulling a random bug off the tracker something > that you could actually look at fixing, instead of mostly just stalling > out. > > So I closed a lot of stuff as Invalid / Opinion that fell into those camps. > > == Keeping New Bugs at close to 0 == > > After driving the bugs in the New state down to zero last week, I found > it's actually pretty easy to keep it at 0. > > We get 10 - 20 new bugs a day in Nova (during a weekday). Of those ~20% > aren't actually a bug, and can be closed immediately. ~30% look like a > bug, but don't have anywhere near enough information in them, and > flipping them to incomplete with questions quickly means we have a real > chance of getting the right info. ~10% are fixable in < 30 minutes worth > of work. And the rest are real bugs, that seem to have enough to dive > into it, and can be triaged into Confirmed, set a priority, and add the > appropriate tags for the area. > > But, more importantly, this means we can filter bug quality on the way > in. And we can also encourage bug reporters that are giving us good > stuff, or even easy stuff, as we respond quickly. > > Recommendation #1: we adopt a 0 new bugs policy to keep this from > getting away from us in the future. > > == Our worse bug reporters are often core reviewers == > > I'm going to pick on Dan Prince here, mostly because I have a recent > concrete example, however in triaging the bug queue much of the core > team is to blame (including myself). > > https://bugs.launchpad.net/nova/+bug/1368773 is a terrible bug. Also, it > was set incomplete and no response. I'm almost 100% sure it's a dupe of > the multiprocess bug we've been tracking down but it's so terse that you > can't get to the bottom of it. > > There were a ton of 2012 nova bugs that were basically "post it notes". > Oh, "we should refactor this function". Full stop. While those are fine > for personal tracking, their value goes to zero probably 3 months after > they are files, especially if the reporter stops working on the issue at > hand. Nova has plenty of "wouldn't it be great if we... " ideas. I'm not > convinced using bugs for those is useful unless we go and close them out > aggressively if they stall. > > Also, if Nova core can't file a good bug, it's hard to set the example > for others in our community. > > Recommendation #2: hey, Nova core, lets be better about filing the kinds > of bugs we want to see! mkay! > > Recommendation #3: Let's create a tag for "personal work items" or > something for these class of TODOs people are leaving themselves that > make them a ton easier to cull later when they stall and no one else has > enough context to pick them up. > > == Tags == > > The aggressive tagging that Tracy brought into the project has been > awesome. It definitely helps slice out into better functional areas. > Here is the top of our current official tag list (and bug count): > > 95 compute > 83 libvirt > 74 api > 68 vmware > 67 network > 41 db > 40 testing > 40 volumes > 36 ec2 > 35 icehouse-backport-potential > 32 low-hang
Re: [openstack-dev] [nova] 2 weeks in the bug tracker
Hi all, FWIW, a quick and dirty solution is here: http://xsnippet.org/360188/ :) Thanks, Roman On Fri, Sep 19, 2014 at 2:03 PM, Ben Nemec wrote: > On 09/19/2014 08:13 AM, Sean Dague wrote: >> I've spent the better part of the last 2 weeks in the Nova bug tracker >> to try to turn it into something that doesn't cause people to run away >> screaming. I don't remember exactly where we started at open bug count 2 >> weeks ago (it was north of 1400, with > 200 bugs in new, but it might >> have been north of 1600), but as of this email we're at < 1000 open bugs >> (I'm counting Fix Committed as closed, even though LP does not), and ~0 >> new bugs (depending on the time of the day). >> >> == Philosophy in Triaging == >> >> I'm going to lay out the philosophy of triaging I've had, because this >> may also set the tone going forward. >> >> A bug tracker is a tool to help us make a better release. It does not >> exist for it's own good, it exists to help. Which means when evaluating >> what stays in and what leaves we need to evaluate if any particular >> artifact will help us make a better release. But also more importantly >> realize that there is a cost for carrying every artifact in the tracker. >> Resolving duplicates gets non linearly harder as the number of artifacts >> go up. Triaging gets non-linearly hard as the number of artifacts go up. >> >> With this I was being somewhat pragmatic about closing bugs. An old bug >> that is just a stacktrace is typically not useful. An old bug that is a >> vague sentence that we should refactor a particular module (with no >> specifics on the details) is not useful. A bug reported against a very >> old version of OpenStack where the code has changed a lot in the >> relevant area, and there aren't responses from the author, is not >> useful. Not useful bugs just add debt, and we should get rid of them. >> That makes the chance of pulling a random bug off the tracker something >> that you could actually look at fixing, instead of mostly just stalling out. >> >> So I closed a lot of stuff as Invalid / Opinion that fell into those camps. >> >> == Keeping New Bugs at close to 0 == >> >> After driving the bugs in the New state down to zero last week, I found >> it's actually pretty easy to keep it at 0. >> >> We get 10 - 20 new bugs a day in Nova (during a weekday). Of those ~20% >> aren't actually a bug, and can be closed immediately. ~30% look like a >> bug, but don't have anywhere near enough information in them, and >> flipping them to incomplete with questions quickly means we have a real >> chance of getting the right info. ~10% are fixable in < 30 minutes worth >> of work. And the rest are real bugs, that seem to have enough to dive >> into it, and can be triaged into Confirmed, set a priority, and add the >> appropriate tags for the area. >> >> But, more importantly, this means we can filter bug quality on the way >> in. And we can also encourage bug reporters that are giving us good >> stuff, or even easy stuff, as we respond quickly. >> >> Recommendation #1: we adopt a 0 new bugs policy to keep this from >> getting away from us in the future. > > We have this policy in TripleO, and to help keep it fresh in people's > minds Roman Podolyaka (IIRC) wrote an untriaged-bot for the IRC channel > that periodically posts a list of any New bugs. I've found it very > helpful, so it's probably worth getting that into infra somewhere so > other people can use it too. > >> >> == Our worse bug reporters are often core reviewers == >> >> I'm going to pick on Dan Prince here, mostly because I have a recent >> concrete example, however in triaging the bug queue much of the core >> team is to blame (including myself). >> >> https://bugs.launchpad.net/nova/+bug/1368773 is a terrible bug. Also, it >> was set incomplete and no response. I'm almost 100% sure it's a dupe of >> the multiprocess bug we've been tracking down but it's so terse that you >> can't get to the bottom of it. >> >> There were a ton of 2012 nova bugs that were basically "post it notes". >> Oh, "we should refactor this function". Full stop. While those are fine >> for personal tracking, their value goes to zero probably 3 months after >> they are files, especially if the reporter stops working on the issue at >> hand. Nova has plenty of "wouldn't it be great if we... " ideas. I'm not >> convinced using bugs for those is useful unless we go and close them out >> aggressively if they stall. >> >> Also, if Nova core can't file a good bug, it's hard to set the example >> for others in our community. >> >> Recommendation #2: hey, Nova core, lets be better about filing the kinds >> of bugs we want to see! mkay! >> >> Recommendation #3: Let's create a tag for "personal work items" or >> something for these class of TODOs people are leaving themselves that >> make them a ton easier to cull later when they stall and no one else has >> enough context to pick them up. >> >> == Tags == >> >> The aggressive tagging that Tracy
Re: [openstack-dev] [nova] 2 weeks in the bug tracker
On 09/19/2014 05:03 PM, Ben Nemec wrote: > On 09/19/2014 08:13 AM, Sean Dague wrote: >> I've spent the better part of the last 2 weeks in the Nova bug tracker >> to try to turn it into something that doesn't cause people to run away >> screaming. I don't remember exactly where we started at open bug count 2 >> weeks ago (it was north of 1400, with > 200 bugs in new, but it might >> have been north of 1600), but as of this email we're at < 1000 open bugs >> (I'm counting Fix Committed as closed, even though LP does not), and ~0 >> new bugs (depending on the time of the day). >> >> == Philosophy in Triaging == >> >> I'm going to lay out the philosophy of triaging I've had, because this >> may also set the tone going forward. >> >> A bug tracker is a tool to help us make a better release. It does not >> exist for it's own good, it exists to help. Which means when evaluating >> what stays in and what leaves we need to evaluate if any particular >> artifact will help us make a better release. But also more importantly >> realize that there is a cost for carrying every artifact in the tracker. >> Resolving duplicates gets non linearly harder as the number of artifacts >> go up. Triaging gets non-linearly hard as the number of artifacts go up. >> >> With this I was being somewhat pragmatic about closing bugs. An old bug >> that is just a stacktrace is typically not useful. An old bug that is a >> vague sentence that we should refactor a particular module (with no >> specifics on the details) is not useful. A bug reported against a very >> old version of OpenStack where the code has changed a lot in the >> relevant area, and there aren't responses from the author, is not >> useful. Not useful bugs just add debt, and we should get rid of them. >> That makes the chance of pulling a random bug off the tracker something >> that you could actually look at fixing, instead of mostly just stalling out. >> >> So I closed a lot of stuff as Invalid / Opinion that fell into those camps. >> >> == Keeping New Bugs at close to 0 == >> >> After driving the bugs in the New state down to zero last week, I found >> it's actually pretty easy to keep it at 0. >> >> We get 10 - 20 new bugs a day in Nova (during a weekday). Of those ~20% >> aren't actually a bug, and can be closed immediately. ~30% look like a >> bug, but don't have anywhere near enough information in them, and >> flipping them to incomplete with questions quickly means we have a real >> chance of getting the right info. ~10% are fixable in < 30 minutes worth >> of work. And the rest are real bugs, that seem to have enough to dive >> into it, and can be triaged into Confirmed, set a priority, and add the >> appropriate tags for the area. >> >> But, more importantly, this means we can filter bug quality on the way >> in. And we can also encourage bug reporters that are giving us good >> stuff, or even easy stuff, as we respond quickly. >> >> Recommendation #1: we adopt a 0 new bugs policy to keep this from >> getting away from us in the future. > > We have this policy in TripleO, and to help keep it fresh in people's > minds Roman Podolyaka (IIRC) wrote an untriaged-bot for the IRC channel > that periodically posts a list of any New bugs. I've found it very > helpful, so it's probably worth getting that into infra somewhere so > other people can use it too. Get me the url for the source code and the name you want the thing to be called and I will cook up the patch to get it into stackforge. Thanks Ben, Anita. > >> >> == Our worse bug reporters are often core reviewers == >> >> I'm going to pick on Dan Prince here, mostly because I have a recent >> concrete example, however in triaging the bug queue much of the core >> team is to blame (including myself). >> >> https://bugs.launchpad.net/nova/+bug/1368773 is a terrible bug. Also, it >> was set incomplete and no response. I'm almost 100% sure it's a dupe of >> the multiprocess bug we've been tracking down but it's so terse that you >> can't get to the bottom of it. >> >> There were a ton of 2012 nova bugs that were basically "post it notes". >> Oh, "we should refactor this function". Full stop. While those are fine >> for personal tracking, their value goes to zero probably 3 months after >> they are files, especially if the reporter stops working on the issue at >> hand. Nova has plenty of "wouldn't it be great if we... " ideas. I'm not >> convinced using bugs for those is useful unless we go and close them out >> aggressively if they stall. >> >> Also, if Nova core can't file a good bug, it's hard to set the example >> for others in our community. >> >> Recommendation #2: hey, Nova core, lets be better about filing the kinds >> of bugs we want to see! mkay! >> >> Recommendation #3: Let's create a tag for "personal work items" or >> something for these class of TODOs people are leaving themselves that >> make them a ton easier to cull later when they stall and no one else has >> enough context to pick them up. >> >> =
Re: [openstack-dev] [nova] 2 weeks in the bug tracker
On 09/19/2014 08:13 AM, Sean Dague wrote: > I've spent the better part of the last 2 weeks in the Nova bug tracker > to try to turn it into something that doesn't cause people to run away > screaming. I don't remember exactly where we started at open bug count 2 > weeks ago (it was north of 1400, with > 200 bugs in new, but it might > have been north of 1600), but as of this email we're at < 1000 open bugs > (I'm counting Fix Committed as closed, even though LP does not), and ~0 > new bugs (depending on the time of the day). > > == Philosophy in Triaging == > > I'm going to lay out the philosophy of triaging I've had, because this > may also set the tone going forward. > > A bug tracker is a tool to help us make a better release. It does not > exist for it's own good, it exists to help. Which means when evaluating > what stays in and what leaves we need to evaluate if any particular > artifact will help us make a better release. But also more importantly > realize that there is a cost for carrying every artifact in the tracker. > Resolving duplicates gets non linearly harder as the number of artifacts > go up. Triaging gets non-linearly hard as the number of artifacts go up. > > With this I was being somewhat pragmatic about closing bugs. An old bug > that is just a stacktrace is typically not useful. An old bug that is a > vague sentence that we should refactor a particular module (with no > specifics on the details) is not useful. A bug reported against a very > old version of OpenStack where the code has changed a lot in the > relevant area, and there aren't responses from the author, is not > useful. Not useful bugs just add debt, and we should get rid of them. > That makes the chance of pulling a random bug off the tracker something > that you could actually look at fixing, instead of mostly just stalling out. > > So I closed a lot of stuff as Invalid / Opinion that fell into those camps. > > == Keeping New Bugs at close to 0 == > > After driving the bugs in the New state down to zero last week, I found > it's actually pretty easy to keep it at 0. > > We get 10 - 20 new bugs a day in Nova (during a weekday). Of those ~20% > aren't actually a bug, and can be closed immediately. ~30% look like a > bug, but don't have anywhere near enough information in them, and > flipping them to incomplete with questions quickly means we have a real > chance of getting the right info. ~10% are fixable in < 30 minutes worth > of work. And the rest are real bugs, that seem to have enough to dive > into it, and can be triaged into Confirmed, set a priority, and add the > appropriate tags for the area. > > But, more importantly, this means we can filter bug quality on the way > in. And we can also encourage bug reporters that are giving us good > stuff, or even easy stuff, as we respond quickly. > > Recommendation #1: we adopt a 0 new bugs policy to keep this from > getting away from us in the future. We have this policy in TripleO, and to help keep it fresh in people's minds Roman Podolyaka (IIRC) wrote an untriaged-bot for the IRC channel that periodically posts a list of any New bugs. I've found it very helpful, so it's probably worth getting that into infra somewhere so other people can use it too. > > == Our worse bug reporters are often core reviewers == > > I'm going to pick on Dan Prince here, mostly because I have a recent > concrete example, however in triaging the bug queue much of the core > team is to blame (including myself). > > https://bugs.launchpad.net/nova/+bug/1368773 is a terrible bug. Also, it > was set incomplete and no response. I'm almost 100% sure it's a dupe of > the multiprocess bug we've been tracking down but it's so terse that you > can't get to the bottom of it. > > There were a ton of 2012 nova bugs that were basically "post it notes". > Oh, "we should refactor this function". Full stop. While those are fine > for personal tracking, their value goes to zero probably 3 months after > they are files, especially if the reporter stops working on the issue at > hand. Nova has plenty of "wouldn't it be great if we... " ideas. I'm not > convinced using bugs for those is useful unless we go and close them out > aggressively if they stall. > > Also, if Nova core can't file a good bug, it's hard to set the example > for others in our community. > > Recommendation #2: hey, Nova core, lets be better about filing the kinds > of bugs we want to see! mkay! > > Recommendation #3: Let's create a tag for "personal work items" or > something for these class of TODOs people are leaving themselves that > make them a ton easier to cull later when they stall and no one else has > enough context to pick them up. > > == Tags == > > The aggressive tagging that Tracy brought into the project has been > awesome. It definitely helps slice out into better functional areas. > Here is the top of our current official tag list (and bug count): > > 95 compute > 83 libvirt > 74 api > 68 vmware > 67 net
Re: [openstack-dev] [nova] 2 weeks in the bug tracker
On Fri, Sep 19, 2014 at 11:13 PM, Sean Dague wrote: > I've spent the better part of the last 2 weeks in the Nova bug tracker > to try to turn it into something that doesn't cause people to run away > screaming. I don't remember exactly where we started at open bug count 2 > weeks ago (it was north of 1400, with > 200 bugs in new, but it might > have been north of 1600), but as of this email we're at < 1000 open bugs > (I'm counting Fix Committed as closed, even though LP does not), and ~0 > new bugs (depending on the time of the day). > > == Philosophy in Triaging == > > I'm going to lay out the philosophy of triaging I've had, because this > may also set the tone going forward. > > A bug tracker is a tool to help us make a better release. It does not > exist for it's own good, it exists to help. Which means when evaluating > what stays in and what leaves we need to evaluate if any particular > artifact will help us make a better release. But also more importantly > realize that there is a cost for carrying every artifact in the tracker. > Resolving duplicates gets non linearly harder as the number of artifacts > go up. Triaging gets non-linearly hard as the number of artifacts go up. > > With this I was being somewhat pragmatic about closing bugs. An old bug > that is just a stacktrace is typically not useful. An old bug that is a > vague sentence that we should refactor a particular module (with no > specifics on the details) is not useful. A bug reported against a very > old version of OpenStack where the code has changed a lot in the > relevant area, and there aren't responses from the author, is not > useful. Not useful bugs just add debt, and we should get rid of them. > That makes the chance of pulling a random bug off the tracker something > that you could actually look at fixing, instead of mostly just stalling out. > > So I closed a lot of stuff as Invalid / Opinion that fell into those camps. > > == Keeping New Bugs at close to 0 == > > After driving the bugs in the New state down to zero last week, I found > it's actually pretty easy to keep it at 0. > > We get 10 - 20 new bugs a day in Nova (during a weekday). Of those ~20% > aren't actually a bug, and can be closed immediately. ~30% look like a > bug, but don't have anywhere near enough information in them, and > flipping them to incomplete with questions quickly means we have a real > chance of getting the right info. ~10% are fixable in < 30 minutes worth > of work. And the rest are real bugs, that seem to have enough to dive > into it, and can be triaged into Confirmed, set a priority, and add the > appropriate tags for the area. On the bugs which would take less than 30 minutes, is that because they're not bugs, or are they just trivial? It would be cool to be adding the low-hanging-fruit tag to those bugs if you're not, because we should just fix them. > But, more importantly, this means we can filter bug quality on the way > in. And we can also encourage bug reporters that are giving us good > stuff, or even easy stuff, as we respond quickly. > > Recommendation #1: we adopt a 0 new bugs policy to keep this from > getting away from us in the future. Agreed, this was a goal we used to have back in the day and I'd like to bring it back. > == Our worse bug reporters are often core reviewers == > > I'm going to pick on Dan Prince here, mostly because I have a recent > concrete example, however in triaging the bug queue much of the core > team is to blame (including myself). > > https://bugs.launchpad.net/nova/+bug/1368773 is a terrible bug. Also, it > was set incomplete and no response. I'm almost 100% sure it's a dupe of > the multiprocess bug we've been tracking down but it's so terse that you > can't get to the bottom of it. > > There were a ton of 2012 nova bugs that were basically "post it notes". > Oh, "we should refactor this function". Full stop. While those are fine > for personal tracking, their value goes to zero probably 3 months after > they are files, especially if the reporter stops working on the issue at > hand. Nova has plenty of "wouldn't it be great if we... " ideas. I'm not > convinced using bugs for those is useful unless we go and close them out > aggressively if they stall. > > Also, if Nova core can't file a good bug, it's hard to set the example > for others in our community. > > Recommendation #2: hey, Nova core, lets be better about filing the kinds > of bugs we want to see! mkay! > > Recommendation #3: Let's create a tag for "personal work items" or > something for these class of TODOs people are leaving themselves that > make them a ton easier to cull later when they stall and no one else has > enough context to pick them up. I think we also get a lot of bugs filed almost immediately before a fix. Sort of like a tracking mechanism for micro-features. Do we want to continue doing that, or do we want to just let smallish things land without a bug? > == Tags == > > The aggressive tagging that Tracy brought i
Re: [openstack-dev] [nova] 2 weeks in the bug tracker
Sorry for top-posting. Just want to say thanks for writing this up and I agree with all the points and recommendations you made. -jay On 09/19/2014 09:13 AM, Sean Dague wrote: I've spent the better part of the last 2 weeks in the Nova bug tracker to try to turn it into something that doesn't cause people to run away screaming. I don't remember exactly where we started at open bug count 2 weeks ago (it was north of 1400, with > 200 bugs in new, but it might have been north of 1600), but as of this email we're at < 1000 open bugs (I'm counting Fix Committed as closed, even though LP does not), and ~0 new bugs (depending on the time of the day). == Philosophy in Triaging == I'm going to lay out the philosophy of triaging I've had, because this may also set the tone going forward. A bug tracker is a tool to help us make a better release. It does not exist for it's own good, it exists to help. Which means when evaluating what stays in and what leaves we need to evaluate if any particular artifact will help us make a better release. But also more importantly realize that there is a cost for carrying every artifact in the tracker. Resolving duplicates gets non linearly harder as the number of artifacts go up. Triaging gets non-linearly hard as the number of artifacts go up. With this I was being somewhat pragmatic about closing bugs. An old bug that is just a stacktrace is typically not useful. An old bug that is a vague sentence that we should refactor a particular module (with no specifics on the details) is not useful. A bug reported against a very old version of OpenStack where the code has changed a lot in the relevant area, and there aren't responses from the author, is not useful. Not useful bugs just add debt, and we should get rid of them. That makes the chance of pulling a random bug off the tracker something that you could actually look at fixing, instead of mostly just stalling out. So I closed a lot of stuff as Invalid / Opinion that fell into those camps. == Keeping New Bugs at close to 0 == After driving the bugs in the New state down to zero last week, I found it's actually pretty easy to keep it at 0. We get 10 - 20 new bugs a day in Nova (during a weekday). Of those ~20% aren't actually a bug, and can be closed immediately. ~30% look like a bug, but don't have anywhere near enough information in them, and flipping them to incomplete with questions quickly means we have a real chance of getting the right info. ~10% are fixable in < 30 minutes worth of work. And the rest are real bugs, that seem to have enough to dive into it, and can be triaged into Confirmed, set a priority, and add the appropriate tags for the area. But, more importantly, this means we can filter bug quality on the way in. And we can also encourage bug reporters that are giving us good stuff, or even easy stuff, as we respond quickly. Recommendation #1: we adopt a 0 new bugs policy to keep this from getting away from us in the future. == Our worse bug reporters are often core reviewers == I'm going to pick on Dan Prince here, mostly because I have a recent concrete example, however in triaging the bug queue much of the core team is to blame (including myself). https://bugs.launchpad.net/nova/+bug/1368773 is a terrible bug. Also, it was set incomplete and no response. I'm almost 100% sure it's a dupe of the multiprocess bug we've been tracking down but it's so terse that you can't get to the bottom of it. There were a ton of 2012 nova bugs that were basically "post it notes". Oh, "we should refactor this function". Full stop. While those are fine for personal tracking, their value goes to zero probably 3 months after they are files, especially if the reporter stops working on the issue at hand. Nova has plenty of "wouldn't it be great if we... " ideas. I'm not convinced using bugs for those is useful unless we go and close them out aggressively if they stall. Also, if Nova core can't file a good bug, it's hard to set the example for others in our community. Recommendation #2: hey, Nova core, lets be better about filing the kinds of bugs we want to see! mkay! Recommendation #3: Let's create a tag for "personal work items" or something for these class of TODOs people are leaving themselves that make them a ton easier to cull later when they stall and no one else has enough context to pick them up. == Tags == The aggressive tagging that Tracy brought into the project has been awesome. It definitely helps slice out into better functional areas. Here is the top of our current official tag list (and bug count): 95 compute 83 libvirt 74 api 68 vmware 67 network 41 db 40 testing 40 volumes 36 ec2 35 icehouse-backport-potential 32 low-hanging-fruit 31 xenserver 25 ironic 23 hyper-v 16 cells 14 scheduler 12 baremetal 9 ceph 9 security 8 oslo ... So, good stuff. However I think we probably want to take a further step and attempt to get champions for tags. So that tag owners would ensure their bug list looks sane,
Re: [openstack-dev] [nova] 2 weeks in the bug tracker
On 09/19/2014 09:58 AM, Sylvain Bauza wrote: >> == Keeping New Bugs at close to 0 == >> >> After driving the bugs in the New state down to zero last week, I found >> it's actually pretty easy to keep it at 0. >> >> We get 10 - 20 new bugs a day in Nova (during a weekday). Of those ~20% >> aren't actually a bug, and can be closed immediately. ~30% look like a >> bug, but don't have anywhere near enough information in them, and >> flipping them to incomplete with questions quickly means we have a real >> chance of getting the right info. ~10% are fixable in < 30 minutes worth >> of work. And the rest are real bugs, that seem to have enough to dive >> into it, and can be triaged into Confirmed, set a priority, and add the >> appropriate tags for the area. >> >> But, more importantly, this means we can filter bug quality on the way >> in. And we can also encourage bug reporters that are giving us good >> stuff, or even easy stuff, as we respond quickly. >> >> Recommendation #1: we adopt a 0 new bugs policy to keep this from >> getting away from us in the future. > > Agreed, provided we can review all the new bugs each week. So I actually don't think this works if it's a weekly thing. Keeping new bugs at 0 really has to be daily because the response to bug reports sets up the expected cadence with the reporter. If you flip back new bugs in < 6 or 8 hrs, there is a decent chance they are still on their same work shift, and the context is still in their head (or even the situation is still existing). Once you pass 24hrs the chance of that goes way down. And, realistically, I've found that when I open the bug tracker in the morning and there are 5 bugs, that's totally doable over the first cup of coffee. Poking the bug tracker a couple more times during the day is all that's needed to keep it there. >> == Our worse bug reporters are often core reviewers == >> >> I'm going to pick on Dan Prince here, mostly because I have a recent >> concrete example, however in triaging the bug queue much of the core >> team is to blame (including myself). >> >> https://bugs.launchpad.net/nova/+bug/1368773 is a terrible bug. Also, it >> was set incomplete and no response. I'm almost 100% sure it's a dupe of >> the multiprocess bug we've been tracking down but it's so terse that you >> can't get to the bottom of it. >> >> There were a ton of 2012 nova bugs that were basically "post it notes". >> Oh, "we should refactor this function". Full stop. While those are fine >> for personal tracking, their value goes to zero probably 3 months after >> they are files, especially if the reporter stops working on the issue at >> hand. Nova has plenty of "wouldn't it be great if we... " ideas. I'm not >> convinced using bugs for those is useful unless we go and close them out >> aggressively if they stall. >> >> Also, if Nova core can't file a good bug, it's hard to set the example >> for others in our community. >> >> Recommendation #2: hey, Nova core, lets be better about filing the kinds >> of bugs we want to see! mkay! >> >> Recommendation #3: Let's create a tag for "personal work items" or >> something for these class of TODOs people are leaving themselves that >> make them a ton easier to cull later when they stall and no one else has >> enough context to pick them up. > > I would propose to set their importance as "Wishlist" then. I would > leave the tags for setting which components are impacted. Maybe. I honestly don't think core team members should file wishlist bugs at all. That really means feature and means a spec. Or it means just do it (for refactoring). >> == Tags == >> >> The aggressive tagging that Tracy brought into the project has been >> awesome. It definitely helps slice out into better functional areas. >> Here is the top of our current official tag list (and bug count): >> >> 95 compute >> 83 libvirt >> 74 api >> 68 vmware >> 67 network >> 41 db >> 40 testing >> 40 volumes >> 36 ec2 >> 35 icehouse-backport-potential >> 32 low-hanging-fruit >> 31 xenserver >> 25 ironic >> 23 hyper-v >> 16 cells >> 14 scheduler >> 12 baremetal >> 9 ceph >> 9 security >> 8 oslo >> ... >> >> So, good stuff. However I think we probably want to take a further step >> and attempt to get champions for tags. So that tag owners would ensure >> their bug list looks sane, and actually spend some time fixing them. >> It's pretty clear, for instance, that the ec2 bugs are just piling up, >> and very few fixes coming in. Cells seems like it's in the same camp (a >> bunch of recent bugs have been cells related, it looks like a lot more >> deployments are trying it). >> >> Probably the most important thing in tag owners would be cleaning up the >> bugs in the tag. Realizing that 2 bugs were actually the same bug. >> Cleaning up descriptions / titles / etc so that people can move forward >> on them. >> >> Recommendation #4: create tag champions > > +1. That said, some bugs can be having more than 1 tag (for example, > compute/conductor/scheduler)
Re: [openstack-dev] [nova] 2 weeks in the bug tracker
On Fri, Sep 19, 2014 at 09:13:29AM -0400, Sean Dague wrote: > I've spent the better part of the last 2 weeks in the Nova bug tracker > to try to turn it into something that doesn't cause people to run away > screaming. I don't remember exactly where we started at open bug count 2 > weeks ago (it was north of 1400, with > 200 bugs in new, but it might > have been north of 1600), but as of this email we're at < 1000 open bugs > (I'm counting Fix Committed as closed, even though LP does not), and ~0 > new bugs (depending on the time of the day). > > == Philosophy in Triaging == > > I'm going to lay out the philosophy of triaging I've had, because this > may also set the tone going forward. > > A bug tracker is a tool to help us make a better release. It does not > exist for it's own good, it exists to help. Which means when evaluating > what stays in and what leaves we need to evaluate if any particular > artifact will help us make a better release. But also more importantly > realize that there is a cost for carrying every artifact in the tracker. > Resolving duplicates gets non linearly harder as the number of artifacts > go up. Triaging gets non-linearly hard as the number of artifacts go up. > > With this I was being somewhat pragmatic about closing bugs. An old bug > that is just a stacktrace is typically not useful. An old bug that is a > vague sentence that we should refactor a particular module (with no > specifics on the details) is not useful. A bug reported against a very > old version of OpenStack where the code has changed a lot in the > relevant area, and there aren't responses from the author, is not > useful. Not useful bugs just add debt, and we should get rid of them. > That makes the chance of pulling a random bug off the tracker something > that you could actually look at fixing, instead of mostly just stalling out. > > So I closed a lot of stuff as Invalid / Opinion that fell into those camps. > > == Keeping New Bugs at close to 0 == > > After driving the bugs in the New state down to zero last week, I found > it's actually pretty easy to keep it at 0. > > We get 10 - 20 new bugs a day in Nova (during a weekday). Of those ~20% > aren't actually a bug, and can be closed immediately. ~30% look like a > bug, but don't have anywhere near enough information in them, and > flipping them to incomplete with questions quickly means we have a real > chance of getting the right info. ~10% are fixable in < 30 minutes worth > of work. And the rest are real bugs, that seem to have enough to dive > into it, and can be triaged into Confirmed, set a priority, and add the > appropriate tags for the area. > > But, more importantly, this means we can filter bug quality on the way > in. And we can also encourage bug reporters that are giving us good > stuff, or even easy stuff, as we respond quickly. > > Recommendation #1: we adopt a 0 new bugs policy to keep this from > getting away from us in the future. > > == Our worse bug reporters are often core reviewers == > > I'm going to pick on Dan Prince here, mostly because I have a recent > concrete example, however in triaging the bug queue much of the core > team is to blame (including myself). > > https://bugs.launchpad.net/nova/+bug/1368773 is a terrible bug. Also, it > was set incomplete and no response. I'm almost 100% sure it's a dupe of > the multiprocess bug we've been tracking down but it's so terse that you > can't get to the bottom of it. > > There were a ton of 2012 nova bugs that were basically "post it notes". > Oh, "we should refactor this function". Full stop. While those are fine > for personal tracking, their value goes to zero probably 3 months after > they are files, especially if the reporter stops working on the issue at > hand. Nova has plenty of "wouldn't it be great if we... " ideas. I'm not > convinced using bugs for those is useful unless we go and close them out > aggressively if they stall. > > Also, if Nova core can't file a good bug, it's hard to set the example > for others in our community. > > Recommendation #2: hey, Nova core, lets be better about filing the kinds > of bugs we want to see! mkay! Not just Nova core. From my observation, most often bugs are often filed in a hurry while going through a Elastic Search/logstash, etc. I updated the below wiki that I wrote a while ago with a reference to this email (thanks for taking the time to write it): https://wiki.openstack.org/wiki/BugFilingRecommendations Reproducing the content of *why* the time to file a more informative bug (for those who prefer to read plain-text). Nothing radically new, just a gentle reminder: - Useful for new test engineers who do not have all the context of a bug. - Useful for documentation writers to help them write correct errata text/release notes. - Useful for non-technical folks reading the bugs/RFEs. Clear information saves a heck of a lot of time! - Useful for folks like product and program managers wh
Re: [openstack-dev] [nova] 2 weeks in the bug tracker
Le 19/09/2014 15:13, Sean Dague a écrit : I've spent the better part of the last 2 weeks in the Nova bug tracker to try to turn it into something that doesn't cause people to run away screaming. I don't remember exactly where we started at open bug count 2 weeks ago (it was north of 1400, with > 200 bugs in new, but it might have been north of 1600), but as of this email we're at < 1000 open bugs (I'm counting Fix Committed as closed, even though LP does not), and ~0 new bugs (depending on the time of the day). == Philosophy in Triaging == I'm going to lay out the philosophy of triaging I've had, because this may also set the tone going forward. A bug tracker is a tool to help us make a better release. It does not exist for it's own good, it exists to help. Which means when evaluating what stays in and what leaves we need to evaluate if any particular artifact will help us make a better release. But also more importantly realize that there is a cost for carrying every artifact in the tracker. Resolving duplicates gets non linearly harder as the number of artifacts go up. Triaging gets non-linearly hard as the number of artifacts go up. With this I was being somewhat pragmatic about closing bugs. An old bug that is just a stacktrace is typically not useful. An old bug that is a vague sentence that we should refactor a particular module (with no specifics on the details) is not useful. A bug reported against a very old version of OpenStack where the code has changed a lot in the relevant area, and there aren't responses from the author, is not useful. Not useful bugs just add debt, and we should get rid of them. That makes the chance of pulling a random bug off the tracker something that you could actually look at fixing, instead of mostly just stalling out. So I closed a lot of stuff as Invalid / Opinion that fell into those camps. == Keeping New Bugs at close to 0 == After driving the bugs in the New state down to zero last week, I found it's actually pretty easy to keep it at 0. We get 10 - 20 new bugs a day in Nova (during a weekday). Of those ~20% aren't actually a bug, and can be closed immediately. ~30% look like a bug, but don't have anywhere near enough information in them, and flipping them to incomplete with questions quickly means we have a real chance of getting the right info. ~10% are fixable in < 30 minutes worth of work. And the rest are real bugs, that seem to have enough to dive into it, and can be triaged into Confirmed, set a priority, and add the appropriate tags for the area. But, more importantly, this means we can filter bug quality on the way in. And we can also encourage bug reporters that are giving us good stuff, or even easy stuff, as we respond quickly. Recommendation #1: we adopt a 0 new bugs policy to keep this from getting away from us in the future. Agreed, provided we can review all the new bugs each week. == Our worse bug reporters are often core reviewers == I'm going to pick on Dan Prince here, mostly because I have a recent concrete example, however in triaging the bug queue much of the core team is to blame (including myself). https://bugs.launchpad.net/nova/+bug/1368773 is a terrible bug. Also, it was set incomplete and no response. I'm almost 100% sure it's a dupe of the multiprocess bug we've been tracking down but it's so terse that you can't get to the bottom of it. There were a ton of 2012 nova bugs that were basically "post it notes". Oh, "we should refactor this function". Full stop. While those are fine for personal tracking, their value goes to zero probably 3 months after they are files, especially if the reporter stops working on the issue at hand. Nova has plenty of "wouldn't it be great if we... " ideas. I'm not convinced using bugs for those is useful unless we go and close them out aggressively if they stall. Also, if Nova core can't file a good bug, it's hard to set the example for others in our community. Recommendation #2: hey, Nova core, lets be better about filing the kinds of bugs we want to see! mkay! Recommendation #3: Let's create a tag for "personal work items" or something for these class of TODOs people are leaving themselves that make them a ton easier to cull later when they stall and no one else has enough context to pick them up. I would propose to set their importance as "Wishlist" then. I would leave the tags for setting which components are impacted. == Tags == The aggressive tagging that Tracy brought into the project has been awesome. It definitely helps slice out into better functional areas. Here is the top of our current official tag list (and bug count): 95 compute 83 libvirt 74 api 68 vmware 67 network 41 db 40 testing 40 volumes 36 ec2 35 icehouse-backport-potential 32 low-hanging-fruit 31 xenserver 25 ironic 23 hyper-v 16 cells 14 scheduler 12 baremetal 9 ceph 9 security 8 oslo ... So, good stuff. However I think we probably want to take a further step and attempt to get champions for tags. S
[openstack-dev] [nova] 2 weeks in the bug tracker
I've spent the better part of the last 2 weeks in the Nova bug tracker to try to turn it into something that doesn't cause people to run away screaming. I don't remember exactly where we started at open bug count 2 weeks ago (it was north of 1400, with > 200 bugs in new, but it might have been north of 1600), but as of this email we're at < 1000 open bugs (I'm counting Fix Committed as closed, even though LP does not), and ~0 new bugs (depending on the time of the day). == Philosophy in Triaging == I'm going to lay out the philosophy of triaging I've had, because this may also set the tone going forward. A bug tracker is a tool to help us make a better release. It does not exist for it's own good, it exists to help. Which means when evaluating what stays in and what leaves we need to evaluate if any particular artifact will help us make a better release. But also more importantly realize that there is a cost for carrying every artifact in the tracker. Resolving duplicates gets non linearly harder as the number of artifacts go up. Triaging gets non-linearly hard as the number of artifacts go up. With this I was being somewhat pragmatic about closing bugs. An old bug that is just a stacktrace is typically not useful. An old bug that is a vague sentence that we should refactor a particular module (with no specifics on the details) is not useful. A bug reported against a very old version of OpenStack where the code has changed a lot in the relevant area, and there aren't responses from the author, is not useful. Not useful bugs just add debt, and we should get rid of them. That makes the chance of pulling a random bug off the tracker something that you could actually look at fixing, instead of mostly just stalling out. So I closed a lot of stuff as Invalid / Opinion that fell into those camps. == Keeping New Bugs at close to 0 == After driving the bugs in the New state down to zero last week, I found it's actually pretty easy to keep it at 0. We get 10 - 20 new bugs a day in Nova (during a weekday). Of those ~20% aren't actually a bug, and can be closed immediately. ~30% look like a bug, but don't have anywhere near enough information in them, and flipping them to incomplete with questions quickly means we have a real chance of getting the right info. ~10% are fixable in < 30 minutes worth of work. And the rest are real bugs, that seem to have enough to dive into it, and can be triaged into Confirmed, set a priority, and add the appropriate tags for the area. But, more importantly, this means we can filter bug quality on the way in. And we can also encourage bug reporters that are giving us good stuff, or even easy stuff, as we respond quickly. Recommendation #1: we adopt a 0 new bugs policy to keep this from getting away from us in the future. == Our worse bug reporters are often core reviewers == I'm going to pick on Dan Prince here, mostly because I have a recent concrete example, however in triaging the bug queue much of the core team is to blame (including myself). https://bugs.launchpad.net/nova/+bug/1368773 is a terrible bug. Also, it was set incomplete and no response. I'm almost 100% sure it's a dupe of the multiprocess bug we've been tracking down but it's so terse that you can't get to the bottom of it. There were a ton of 2012 nova bugs that were basically "post it notes". Oh, "we should refactor this function". Full stop. While those are fine for personal tracking, their value goes to zero probably 3 months after they are files, especially if the reporter stops working on the issue at hand. Nova has plenty of "wouldn't it be great if we... " ideas. I'm not convinced using bugs for those is useful unless we go and close them out aggressively if they stall. Also, if Nova core can't file a good bug, it's hard to set the example for others in our community. Recommendation #2: hey, Nova core, lets be better about filing the kinds of bugs we want to see! mkay! Recommendation #3: Let's create a tag for "personal work items" or something for these class of TODOs people are leaving themselves that make them a ton easier to cull later when they stall and no one else has enough context to pick them up. == Tags == The aggressive tagging that Tracy brought into the project has been awesome. It definitely helps slice out into better functional areas. Here is the top of our current official tag list (and bug count): 95 compute 83 libvirt 74 api 68 vmware 67 network 41 db 40 testing 40 volumes 36 ec2 35 icehouse-backport-potential 32 low-hanging-fruit 31 xenserver 25 ironic 23 hyper-v 16 cells 14 scheduler 12 baremetal 9 ceph 9 security 8 oslo ... So, good stuff. However I think we probably want to take a further step and attempt to get champions for tags. So that tag owners would ensure their bug list looks sane, and actually spend some time fixing them. It's pretty clear, for instance, that the ec2 bugs are just piling up, and very few fixes coming in. Cells seems like it's in the same camp
