Hi,
I'd like to start my feedback with a request.
It'd help me to get a big-picture of the stuff that surrounds this
email. Things I'd like to see is information about who's been consulted
going in to this. Also, which threads about bug lifecycle got looked at.
It'd also be nice to see how this one should play into more
forward-looking work.
I know that I wasn't consulted because I didn't insist. I did have the
chance, though. Others might feel easier if they had similar insight.
Also, the question about "is this the problem to solve" benefit from
context. This one might just be a dependency of the ideas on how to get
to more heavy stuff.
And now to remember all that when I write my own heavy-weight stuff soon.
More details inline.
On 30/01/16 00:45, Emma Humphries wrote:
Bug Program Next Steps
Over the last week, I’ve asked you to step up and identify developers
who will be responsible for bugs triaged into their component (in
Firefox, Core, Toolkit, Fennec iOS, and Fennec Android.)
Why This Matters
Bugs are a unit of quality we can use to see how we’re doing.
We believe that the number of outstanding, actionable bugs is the best
metric of code quality available, and that how this number changes
over time will be a strong indicator of the evolving quality of
Firefox. Actionablebugs are those with the status NEW for which the
next action required - if any - is unambiguous: does the bug need
immediate attention, will it be worked on for a future release, will
it be worked on at all?
There are two parts to maintaining the value of that metric. First, we
want to make better assertions about the quality of our releases by
making clear decisions about which bugs must be fixed for each release
(urgent) and actively tracking those bugs. The other is the number of
good bugs filed by our community. Filing a bug report is a gateway to
greater participation in the Mozilla project, and we owe it to our
community to make quick and clear decisions about each bug.
Making decisions on new bugs quickly helps us avoid point releases,
and gives positive feedback to people filing bugs so that they will
file more good bugs.
There's a school of thought that values a non-actionable bug over a bug
not filed.
I know that for my personal crashers, for example, I look at stack
traces, and have no clue what could be going wrong there. Or how I could
provide value in figuring it out.
I do force myself to file bugs on them at times, though. 'Cause if I
don't file 'em, nobody will, and then there's even less chance to figure
something out.
I think this is a concern beyond crashes, as we struggle to find a
balance between not having insight on how things work, and oth, a
gazillion of useless bugs that just say "doesn't work".
What’s Your Role
Starting in the second quarter of this year, if you’ve taken on a
component, I’m expecting you or your team to look at the bugs which
landed in the component on a more frequent basis than a weekly triage.
In February, we’re starting a pilot with four groups of components
where we’ll get the process changes and field tested, so that we can
we can take the changes to all the affected bugzilla components for
review and comment before we implement them across all of the work on
Firefox.
How are those four groups chosen?
Hold on, we already have a weekly triage!
That’s fantastic, but a weekly pace means we miss bugs that affect
upcoming releases. So I’m expecting you to scan that list of inbound
bugs daily for the urgent ones (I’ll define urgent below) and put them
into one of the states described in the next section, the others can
go into your regular triage.
At Your Regular Triage
You’ll look at the bugs which landed in your component and decide on
how to act on them using the states described in the next section.
We don’t have a regular triage
This is a process which you’ll need to start, and the bug program team
will help with this.
This is potentially a lot of work for one person
Looking at the urgent bugs does not have to be one person’s task. You
can have a rotation of people doing this. Look at the Core::Graphics
<https://wiki.mozilla.org/Platform/GFX/TriageSchedule>triage wiki for
an example of what you could be doing.
Bug States
Initially, these states will be marked in bugzilla.mozilla.org
<http://bugzilla.mozilla.org> using whiteboard tags during the pilot.
The bugzilla team will be making further changes once we’ve settled on
a process.
You’ll be looking at bugs in your component as they land, in your
component. We expect most of these will be NEW bugs, but some will be
in UNCONFIRMED.
There are four states you’ll need to decide to put each bug, and in
your reviews between your team’s weekly triages, we want you to be on
the watch for bugs with characteristics which make getting it in front
of someone urgent: these are bugs with crash, topcrash, regression, or
dataloss keywords; crash logs linked in comments; references to
mozregression results; and others.
The bug should not remain in this state after your review of it.
You’ll need to decide which of the following states you’ll move this
bug into, if you can’t you’ll need to be taking action: such as
getting someone to run mozregression, need info’ing a domain expert,
looking at checkins, and whatever else techniques you have to get a
bug reduced.
Once you have an understanding of the bug, you should assign it to one
of these states.
I'd ask you to make additions to these states for two use-cases:
Ran out of time
As you're asking for this to be done on a daily basis, I'd expect it to
be time-bound. We do have bugs that just take a day or two to figure out
what the next step is.
Scheme doesn't work well for this bug
I think this is an important option as we're learning how this works out
for us.
I'm also generally concerned how UX bugs or crashes would fit into these
buckets. UX bugs, and possibly any other flavor of ideation, have the
majority of work associated with "should we do this or not". And crashes
as a single crash stack are hardly ever actionable.
Urgent
Assigned to a developer, release tracking flags nominated, and set at
priority `P1`. If the bug is being worked on by somebody from outside
your core team, a team mentor should be assigned.
All these need to be set for the bug when you assign it to this state.
This state is for bugs you find in your daily review which need a
developer immediately.
If the bug is not in need of immediate attention, then your team’s
process should land the bug in one of the following states.
Backlog
This is a NEW bug that the team acknowledges is a bug, but is not a
current priority, but will consider taking on. If the bug contains
regression, crash, topcrash, or similar keywords and metadata, then
the team can explain why it’s not a high priority.
Is a Bug, Not Prioritized
This is a terminal state for a NEW bug. We acknowledge the bug exists,
it affects people, but it is not important enough to warrant working
on it. The team will review and accept patches from the community for
this bug report.
Closed
This is a terminal state for a NEW bug. After review, the bug is not
something that can be worked on, or describes existing and expected
behavior.
Other States
There are other states we’re looking at for bugs. These will cover a
bug as it’s worked on, as ASSIGNED as is doesn’t provide useful
information as to progress; flag bugs that have stalled, or needing
code reviews, or sheriff attention; bugs that are on a release train;
and bugs with fixes in general or ESR version.
Timeline
Now: finish finding component responsible parties
February: pilot review of NEW bugs with four groups of components,
draft new process
March: comment period for new process, finalize process
Q2: implement new process across all components
Q3: all newly triaged bugs following the new process
Are there good metrics to use to evaluate the success and impact of this
process?
To pick up on what I said earlier, getting a category for "didn't work"
could be one way to measure, or at least enable us to systematically
investigate areas for improvements.
Axel
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
