Re: Changes to how we track regressions in Bugzilla
Thanks for everyone's feedback. We're starting on the implementation in these bugs https://bugzilla.mozilla.org/show_bug.cgi?id=1542378 (enter bug form), https://bugzilla.mozilla.org/show_bug.cgi?id=1545258 (create new field), and https://bugzilla.mozilla.org/show_bug.cgi?id=1543876 (migrate existing metadata). I will follow up when we are ready to enable the field in production Bugzilla. -- Emma On Mon, Apr 29, 2019 at 3:15 PM Emma Humphries wrote: > We are making more changes to how we report and track regressions in > Bugzilla. > > > There are multiple fields and keywords we use to track regressions in > Bugzilla. It’s confusing and the fields are used inconsistently across > teams. > > > We are consolidating these fields to reduce confusion and provide a > consistent way to track these bugs. This change will also improve our > understanding of defects by humans, tools and automation. > > > == The Change > > > You have already seen the first part of this work. We’ve added the > regresses and regressed_by fields to identify the bug associated with the > changeset that caused a regression. That step eliminated the confusion over > if a bug caused a regression or was a dependency. > > Currently we use the regression and regressionwindow-wanted keywords to > indicate regressions and look for the changeset responsible. We also have > the has-regression-range field (which is used less frequently.) > > We’re replacing the regression and regressionwindow-wanted keywords, and > the has-regression-range field with is_regression, which has the values: > > >- > >yes-range-known >- > >yes-range-needed >- > >yes-range-not-needed >- > >unsure >- > >no >- > >'---' > > We have yes-range-not-needed as a value because there are some > long-standing regressions we know of, and also to preserve history on > older, resolved regression bugs for which we may not know the cause or for > which finding the range would be too expensive or difficult to find. > == Usage > > > The default value of is_regression is '---'. > > If you believe a bug is a regression, but don’t know what commit > introduced it, set is_regression to yes-range-needed. > > If you are not sure if a bug is a regression, set is_regression to unsure. > > > Once you’ve found the changeset that introduced the regression, set > is_regression to yes-range-known and set the regressed_by field to the > bug containing the changeset. > > If a bug turns out to not be a regression, set is_regression to no and > make sure regressed_by is empty. > > If you set is_regression to yes-range-not-needed, then you’re asserting > that there’s not a changeset you can revert to fix the regression. > > If you are looking for regressions to find ranges for, search on > is_regression = yes-range-needed. > > Bugs with is_regression = unsure need help confirming if they are > regressions. If you can confirm it’s a regression set the value to > yes-range-needed, or yes-range-known. If not, set the value to no. > > We will notify triage owners through setting a needinfo request via the > Autonag Bot when these fields are in an inconsistent state (see below). > > > == Notes > > > The is_regression field is intended for bugs of type defect. > > We will be updating our UI for bug entry so that when setting a bug > > So that we don’t lose history in the move, we will have to note older, > resolved regression bugs for which we don’t have values for the > regresses/regressed_by fields. If the bug is a regression which was RESOLVED > FIXED or RESOLVED VERIFIED, we will mark it as yes-range-known, and look > into backfilling the field using the information in the blocked_by field. > > These changes were discussed, at length, on Bugzilla ( > https://bugzilla.mozilla.org/show_bug.cgi?id=1534280) > > > == Feedback > > > I'll be reading your feedback on this on this thread, or in the bugzilla > bug and will respond later this week. > > > -- Emma > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Changes to how we track regressions in Bugzilla
When I was emailing this yesterday, I neglected to shout out folks for their help: Calixte suggested the improvements, and the plan benefited from Marco's reflecting on what we were trying to achieve. I recommend reading the thread https://bugzilla.mozilla.org/show_bug.cgi?id=1534280 to follow the evolution of the process. -- Emma On Mon, Apr 29, 2019 at 3:15 PM Emma Humphries wrote: > We are making more changes to how we report and track regressions in > Bugzilla. > > > There are multiple fields and keywords we use to track regressions in > Bugzilla. It’s confusing and the fields are used inconsistently across > teams. > > > We are consolidating these fields to reduce confusion and provide a > consistent way to track these bugs. This change will also improve our > understanding of defects by humans, tools and automation. > > > == The Change > > > You have already seen the first part of this work. We’ve added the > regresses and regressed_by fields to identify the bug associated with the > changeset that caused a regression. That step eliminated the confusion over > if a bug caused a regression or was a dependency. > > Currently we use the regression and regressionwindow-wanted keywords to > indicate regressions and look for the changeset responsible. We also have > the has-regression-range field (which is used less frequently.) > > We’re replacing the regression and regressionwindow-wanted keywords, and > the has-regression-range field with is_regression, which has the values: > > >- > >yes-range-known >- > >yes-range-needed >- > >yes-range-not-needed >- > >unsure >- > >no >- > >'---' > > We have yes-range-not-needed as a value because there are some > long-standing regressions we know of, and also to preserve history on > older, resolved regression bugs for which we may not know the cause or for > which finding the range would be too expensive or difficult to find. > == Usage > > > The default value of is_regression is '---'. > > If you believe a bug is a regression, but don’t know what commit > introduced it, set is_regression to yes-range-needed. > > If you are not sure if a bug is a regression, set is_regression to unsure. > > > Once you’ve found the changeset that introduced the regression, set > is_regression to yes-range-known and set the regressed_by field to the > bug containing the changeset. > > If a bug turns out to not be a regression, set is_regression to no and > make sure regressed_by is empty. > > If you set is_regression to yes-range-not-needed, then you’re asserting > that there’s not a changeset you can revert to fix the regression. > > If you are looking for regressions to find ranges for, search on > is_regression = yes-range-needed. > > Bugs with is_regression = unsure need help confirming if they are > regressions. If you can confirm it’s a regression set the value to > yes-range-needed, or yes-range-known. If not, set the value to no. > > We will notify triage owners through setting a needinfo request via the > Autonag Bot when these fields are in an inconsistent state (see below). > > > == Notes > > > The is_regression field is intended for bugs of type defect. > > We will be updating our UI for bug entry so that when setting a bug > > So that we don’t lose history in the move, we will have to note older, > resolved regression bugs for which we don’t have values for the > regresses/regressed_by fields. If the bug is a regression which was RESOLVED > FIXED or RESOLVED VERIFIED, we will mark it as yes-range-known, and look > into backfilling the field using the information in the blocked_by field. > > These changes were discussed, at length, on Bugzilla ( > https://bugzilla.mozilla.org/show_bug.cgi?id=1534280) > > > == Feedback > > > I'll be reading your feedback on this on this thread, or in the bugzilla > bug and will respond later this week. > > > -- Emma > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Changes to how we track regressions in Bugzilla
On 4/29/19 6:15 PM, Emma Humphries wrote: We’re replacing the regression and regressionwindow-wanted keywords, and the has-regression-range field with is_regression Emma, Thank you, the new setup seems a lot clearer about the state of the bug in terms of regression tracking! -Boris ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Changes to how we track regressions in Bugzilla
We are making more changes to how we report and track regressions in Bugzilla. There are multiple fields and keywords we use to track regressions in Bugzilla. It’s confusing and the fields are used inconsistently across teams. We are consolidating these fields to reduce confusion and provide a consistent way to track these bugs. This change will also improve our understanding of defects by humans, tools and automation. == The Change You have already seen the first part of this work. We’ve added the regresses and regressed_by fields to identify the bug associated with the changeset that caused a regression. That step eliminated the confusion over if a bug caused a regression or was a dependency. Currently we use the regression and regressionwindow-wanted keywords to indicate regressions and look for the changeset responsible. We also have the has-regression-range field (which is used less frequently.) We’re replacing the regression and regressionwindow-wanted keywords, and the has-regression-range field with is_regression, which has the values: - yes-range-known - yes-range-needed - yes-range-not-needed - unsure - no - '---' We have yes-range-not-needed as a value because there are some long-standing regressions we know of, and also to preserve history on older, resolved regression bugs for which we may not know the cause or for which finding the range would be too expensive or difficult to find. == Usage The default value of is_regression is '---'. If you believe a bug is a regression, but don’t know what commit introduced it, set is_regression to yes-range-needed. If you are not sure if a bug is a regression, set is_regression to unsure. Once you’ve found the changeset that introduced the regression, set is_regression to yes-range-known and set the regressed_by field to the bug containing the changeset. If a bug turns out to not be a regression, set is_regression to no and make sure regressed_by is empty. If you set is_regression to yes-range-not-needed, then you’re asserting that there’s not a changeset you can revert to fix the regression. If you are looking for regressions to find ranges for, search on is_regression = yes-range-needed. Bugs with is_regression = unsure need help confirming if they are regressions. If you can confirm it’s a regression set the value to yes-range-needed, or yes-range-known. If not, set the value to no. We will notify triage owners through setting a needinfo request via the Autonag Bot when these fields are in an inconsistent state (see below). == Notes The is_regression field is intended for bugs of type defect. We will be updating our UI for bug entry so that when setting a bug So that we don’t lose history in the move, we will have to note older, resolved regression bugs for which we don’t have values for the regresses/regressed_by fields. If the bug is a regression which was RESOLVED FIXED or RESOLVED VERIFIED, we will mark it as yes-range-known, and look into backfilling the field using the information in the blocked_by field. These changes were discussed, at length, on Bugzilla ( https://bugzilla.mozilla.org/show_bug.cgi?id=1534280) == Feedback I'll be reading your feedback on this on this thread, or in the bugzilla bug and will respond later this week. -- Emma ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
