Re: [openstack-dev] [bugs] definition of triaged
Robert Collins wrote: On 16 December 2013 23:56, Thierry Carrez thie...@openstack.org wrote: I like the first and third parts. Not really convinced with the second part, though. You'll have a lot of Confirmed bugs without proposed approach (like 99% of them) so asking core to read them all and scan them for a proposed approach sounds like a waste of time. There seems to So, I'm trying to reconcile: - the goal of moving bugs into 'triaged' - which really is: - keeping a pipeline of low hanging fruit as an onramp - apparently some documentation needs too, though I'd rather push /all/ those into DocImpact tags on changes, + those bugs that are solely documentation issues. - the goal of identifying critical bugs rapidly - the goal of steering bugs to the right subteam (e.g. vendor interests) Agree on your 3 goals. But I would argue that, in our setting, the value of the second and third goal is much higher than the value of the first one. We need *some* easy/analyzed bugs in the onramp pipeline, but we don't need all of them. [...] I'm *entirely* happy with saying that anyone with the experience to do it can move things up to Triaged - I see no problem there, but there is a huge problem if we have any step in the process's inner loop that requires O(bugs) tasks. [...] I completely agree. I felt like *your* approach for the second phase was O(bugs), which is why I disagreed with it :) You proposed: Daily tasks - second layer - -core current and previous members 1. Assess the proposed approach in Confirmed+High[1] bugs 1.1. If good, move to Triaged 1.2 If not, suggest what would make the approach good[2] 2. If appropriate add low-hanging-fruit tag IIUC that means going through each and every Confirmed+High[1] bug to check if there is a proposed approach in them, and move them to Triaged if it's any good. This is O(confirmedbugs). My proposal (and the current state of things) is like this: Anyone can propose an approach to a random bug and set the bug to Triaged when he does. Anyone. This is not an O(bugs) effort, this is 0 effort for your core members. I assert there is not enough value in assessing the proposed approach for it to be worth core time. Anyone should be able to propose a solution and, if they are sure enough about it, set the bug to Triaged. That creates enough Triaged bugs for your on-ramp pipeline. Code review will be there to catch the corner case bad solution. Core developers are just human beings that sign up to do a lot of code reviews. Not deities that need to vouch every proposed solution in every bug before a human may be tempted to solve it. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [bugs] definition of triaged
On 16 December 2013 23:56, Thierry Carrez thie...@openstack.org wrote: Robert Collins wrote: https://wiki.openstack.org/wiki/BugTriage is a 10 step process: thats not something we can sensible do over coffee in the morning. Totally agree with that. The idea of those 10 steps was not meant as something you could ever complete, it was more a way to state there is no point in doing step 8 unless step 1 is covered. I agree that splitting the process into separate daily / cycle-y processes (as you propose below) is a better way to ever get it done. Ok, cool. I like the first and third parts. Not really convinced with the second part, though. You'll have a lot of Confirmed bugs without proposed approach (like 99% of them) so asking core to read them all and scan them for a proposed approach sounds like a waste of time. There seems to So, I'm trying to reconcile: - the goal of moving bugs into 'triaged' - which really is: - keeping a pipeline of low hanging fruit as an onramp - apparently some documentation needs too, though I'd rather push /all/ those into DocImpact tags on changes, + those bugs that are solely documentation issues. - the goal of identifying critical bugs rapidly - the goal of steering bugs to the right subteam (e.g. vendor interests) be more value in asserting that there is a proposed approach in the bug, rather than there is a core-approved approach in this bug. The definition of Triaged on the Bugs wiki page is 'Triaged The bug comments contain a full analysis on how to properly fix the issue ' Fundamentally for a process to work at scale we can't depend on 'all triagers naturally figure out which bugs need to be Triaged and which only need to be Confirmed'. *regardless* of definition of triaged, we need a system where we don't exhaust people every few weeks reviewing *every open bug* to figure out which ones have been missed. I'm *entirely* happy with saying that anyone with the experience to do it can move things up to Triaged - I see no problem there, but there is a huge problem if we have any step in the process's inner loop that requires O(bugs) tasks. Furthermore, I'm not convinced we really need to spend core time assessing the proposed approach. You can spend core time on suggesting a proposed approach (i.e. turning Confirmed into Triaged), though. So if I rephrased Daily tasks - second layer - -core current and previous members 1. Review Confirmed+High[1] bugs 1.1. If there is a full analysis on how to properly fix the issue move to Triaged 1.2 If not, add one. 2. If appropriate add low-hanging-fruit tag You'd be happy? The reason I didn't write it like that is that this burns *more* core time. I was trying to propose a process where we can scale a competent but non-core team up without demanding that -core commit to boiling the ocean. The different between the phrasing I had and the one you seem to be proposing is that rather than promoting and guiding on bugs, -core are being asked to /do/ on every bug. That places a lot of load on -core, *or* it means we get many less bugs triaged. If we say that anyone with reasonable competency can do it, we could say: Daily tasks - second layer - more experienced folk 1. Review Confirmed+High[1] bugs 1.1. If there is a full analysis on how to properly fix the issue move to Triaged 1.2 If not, add one. 2. If appropriate add low-hanging-fruit tag You basically have the following states for a bug: A - Brand new B - WAT (incomplete, ask the reporter for more info) C - I don't have the expertise to judge (but I can add tag) D - Yes this is definitely a bug, here is its priority E - This is a bug and here is a suggested approach to fix it F - I'm core and I bless that way to fix it G - I started working on that fix H - The fix is in code review I - The fix landed in master J - The fix landed in a milestone K - The fix landed in a release That is way too many states, especially if you rely on humans to set them. My experience is that humans deal with 3 bug states correctly, but start to fail doing it consistently if you ask them to set 4 or more. I would quibble about that list - for instance tagging and expertise to judge project impact tag properly are IME the same. There are many dimensions of input to bugs, and what you're calling states there are really the combination of things from different dimensions. Timeline interactions for instance, don't change the state of a bug, but they change whether we want it to show up in bug searches. In the current setup (constrained by what Launchpad supports) we use tags for C, we ignore F, we combine G+H, and we combine J+K: A - New B - Incomplete C - (New + tagged) D - Confirmed E - Triaged F - (Triaged + comment) G - In progress H - In progress (automatically set) I - Fix committed (automatically set) J - Fix released (automatically set) K - (Fix Released + remilestoning) (automatically set) I think less states (and
Re: [openstack-dev] [bugs] definition of triaged
On 13/12/13 19:13, Thierry Carrez wrote: Robert Collins wrote: Confirmed The bug was reproduced or confirmed as a genuine bug Triaged The bug comments contain a full analysis on how to properly fix the issue From wiki.openstack.org/wiki/Bugs Putting aside the difficulty of complete reproduction sometimes, I don't understand the use of Triaged here. In LP they mean: Confirmed Verified by someone other than the reporter. Triaged Verified by the bug supervisor. So our meaning is very divergent. I'd like us to consolidate on the standard meaning - which is that the relative priority of having a doctor [developer] attack the problem has been assessed. I'm the one who established, a long time ago, this divergence between Launchpad's classic use of those statuses and OpenStack's. In my experience NOBODY ever uses Confirmed with the LP meaning, so I figured we should use it for something more useful: to describe how advanced you are in the resolution of the issue. This is why I proposed (and we used) the following distinction, as described in https://wiki.openstack.org/wiki/BugTriage : Confirmed bugs are genuine bugs but nobody really looked into the best way to fix them yet. They are confirmed, priority-assigned, tagged Triaged bugs are bugs which are analyzed and have a clear way forward to resolve them, just missing someone to actually write the patch That way developers could pick ready to fix bugs by searching Triaged bugs rather than Confirmed ones. Specifically: - we should use Triaged to indicate that: - we have assigned a priority - we believe it's a genuine bug - we have routed[tagged] it to what is probably the right place [vendor driver/low-hanging-fruit etc] - we should use Incomplete if we aren't sure that its a bug and need the reporter to tell us more to be sure - triagers shouldn't ever set 'confirmed' - thats reserved solely for end users to tell us that more than one user is encountering the problem. I can see how that works for Ubuntu. But did you ever see, in OpenStack, an end user tell us that they /also encountered/ the problem ? The end result of your proposal is that we stop using Confirmed and use Triaged instead (to describe the exact same thing). We lose the ability to use Triaged to indicate a more analyzed state. I'm not sure that's a net win, or worth asking anyone to change their habits, or worth changing the BugTriage wikipage (and/or any other page that repeated it). I'll gladly admit that my meaning of Triaged was not used that much and that it could be replaced by something more useful. But merely using triaged for our old meaning of confirmed (and stop using confirmed) sounds like change for the sake of change. Just want to back Thierry here - find the current Triaged/Confirmed distinction quite useful. In the documentation case, Triaged often means that there's a note of which files to edit and/or has text included in the bug or linked to somewhere that can be used to write the content needed. This exactly matches with: Triaged bugs are bugs which are analyzed and have a clear way forward to resolve them, just missing someone to actually write the patch and it appears to be working well for facilitating the long tail contributors, who might just drop by for a patch or two. Sometimes, we can work out that the issue is real but it might be a particularly complex fix across multiple books, or need a developer's input to work out what the actual behaviour should be. These get set as 'confirmed' during the Triage process. Like others have noted, I've not seen cases where users have marked bugs as 'confirmed'. Even our tamer in-community operators tend to wait for a TriagePerson to set it to 'confirmed' - and this is for mere typos in docs :) I guess, in short - I'm not sure there's an issue on 'triaged' vs 'confirmed'? Though, the idea in later emails than this that existing bugs should be looked at only once or twice a cycle worries me :) I'll leave that for the more wisened to debate for now ... Regards, Tom ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [bugs] definition of triaged
Robert Collins wrote: Confirmed The bug was reproduced or confirmed as a genuine bug Triaged The bug comments contain a full analysis on how to properly fix the issue From wiki.openstack.org/wiki/Bugs Putting aside the difficulty of complete reproduction sometimes, I don't understand the use of Triaged here. In LP they mean: Confirmed Verified by someone other than the reporter. Triaged Verified by the bug supervisor. So our meaning is very divergent. I'd like us to consolidate on the standard meaning - which is that the relative priority of having a doctor [developer] attack the problem has been assessed. I'm the one who established, a long time ago, this divergence between Launchpad's classic use of those statuses and OpenStack's. In my experience NOBODY ever uses Confirmed with the LP meaning, so I figured we should use it for something more useful: to describe how advanced you are in the resolution of the issue. This is why I proposed (and we used) the following distinction, as described in https://wiki.openstack.org/wiki/BugTriage : Confirmed bugs are genuine bugs but nobody really looked into the best way to fix them yet. They are confirmed, priority-assigned, tagged Triaged bugs are bugs which are analyzed and have a clear way forward to resolve them, just missing someone to actually write the patch That way developers could pick ready to fix bugs by searching Triaged bugs rather than Confirmed ones. Specifically: - we should use Triaged to indicate that: - we have assigned a priority - we believe it's a genuine bug - we have routed[tagged] it to what is probably the right place [vendor driver/low-hanging-fruit etc] - we should use Incomplete if we aren't sure that its a bug and need the reporter to tell us more to be sure - triagers shouldn't ever set 'confirmed' - thats reserved solely for end users to tell us that more than one user is encountering the problem. I can see how that works for Ubuntu. But did you ever see, in OpenStack, an end user tell us that they /also encountered/ the problem ? The end result of your proposal is that we stop using Confirmed and use Triaged instead (to describe the exact same thing). We lose the ability to use Triaged to indicate a more analyzed state. I'm not sure that's a net win, or worth asking anyone to change their habits, or worth changing the BugTriage wikipage (and/or any other page that repeated it). I'll gladly admit that my meaning of Triaged was not used that much and that it could be replaced by something more useful. But merely using triaged for our old meaning of confirmed (and stop using confirmed) sounds like change for the sake of change. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [bugs] definition of triaged
On 12/12/2013 04:46 PM, Robert Collins wrote: Hi, I'm trying to overhaul the bug triage process for nova (initially) to make it much lighter and more effective. I'll be sending a more comprehensive mail shortly before you do, let's agree what we're trying to solve. Perhaps you were going to cover that in your later message, but it wouldn't hurt discussing it now. I actually didn't think our process was that broken. It's more that I feel we need a person leading a small team that is working on it reguarly. The idea with the tagging approach was to break up the triage problem into smaller work queues. I haven't kept up with the tagging part and would really like to hand that off. Then some of the work queues aren't getting triaged as regularly as they need to. I'd like to see a small team making this a high priority with some of their time each week. With all of that said, if you think an overhaul of the process is necessary to get to the end goal of a more well triaged bug queue, then I'm happy to entertain it. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [bugs] definition of triaged
On Thu, Dec 12, 2013 at 3:46 PM, Robert Collins robe...@robertcollins.netwrote: Hi, I'm trying to overhaul the bug triage process for nova (initially) to make it much lighter and more effective. I'll be sending a more comprehensive mail shortly but one thing that has been giving me pause is this: Confirmed The bug was reproduced or confirmed as a genuine bug Triaged The bug comments contain a full analysis on how to properly fix the issue From wiki.openstack.org/wiki/Bugs Putting aside the difficulty of complete reproduction sometimes, I don't understand the use of Triaged here. In LP they mean: Confirmed Verified by someone other than the reporter. Triaged Verified by the bug supervisor. So our meaning is very divergent. I'd like us to consolidate on the standard meaning - which is that the relative priority of having a doctor [developer] attack the problem has been assessed. Specifically: - we should use Triaged to indicate that: - we have assigned a priority - we believe it's a genuine bug - we have routed[tagged] it to what is probably the right place ++ that's exactly how I use it, with some emphasis on believe which I use to differentiate from this from Confirmed ... [vendor driver/low-hanging-fruit etc] - we should use Incomplete if we aren't sure that its a bug and need the reporter to tell us more to be sure - triagers shouldn't ever set 'confirmed' - thats reserved solely for end users to tell us that more than one user is encountering the problem. As a triager, if I put my user hat on and am able to reproduce a bug, I'll mark Confirmed, otherwise it's just Triaged. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- -Dolph ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
