Re: Blocking bugs process

2015-07-13 Thread Ian Booth 


On 14/07/15 09:30, Martin Packman wrote:
> Thank you for responding Ian.
> 
> On 13/07/2015, Ian Booth  wrote:
>>>
>>> == Definition of blocking bugs ==
>>> Master and all release branches should always be in a releasable
>>> state. If a bug must be fixed for the next minor release, it is
>>> considered a ‘blocker’ and will prevent all landing on that branch.
>>>
>>
>> I don't agree with the above definition. There are always many bugs which
>> must be fixed for the next minor release - these are assigned to the relevant
>> milestones and marked high or critical.
> 
> So, my argument over this was a pretty strict interpretation of
> "must". There are lots of bugs with less-than-critical severity that
> get targeted at the next minor milestone, and I think that's perfect
> reasonable. However, there are comparatively few bugs that could not
> be deferred if, for instance, we discovered a security issue and had
> to rush out a new minor release immediately.
> 
> From my perspective, blockers are things that break CI enough to
> hinder our ability to do a release, or issues that if we allow into
> release code will break users in unrecoverable ways. I know Aaron
> prefers a much wider interpretation however.
> 
>> In the case of bug 1468653, this has been
>> under investigation for 2 weeks and even though we are holding up the 1.24.3
>> release for it, if it were a blocker the whole production line would have
>> stalled unnecessarily.
> 
> That's an interesting bug. It seems with a lot of pain (manually
> restarting things) it is actually repairable, and does involve a
> pretty specific setup. I don't think I'd have added the blocker tag to
> it, but that may not be the consensus.

By the definition given

"If a bug must be fixed for the next minor release, it is considered a ‘blocker’
and will prevent all landing on that branch."

that bug and any other that we say we must include in a release would block
landings. That's the bit I'm having an issue with. I think landings need to be
blocked when appropriate, but not by that definition.

> 
>> With follow on changes, the problem is
>> quite isolated so landing fixes for other release critical issues should not
>> be prevented.
> 
> The fact it's a somewhat isolated problem is important. What I really
> want to avoid is the case where we leave say, upgrades broken as a
> whole, and keep landing code. That makes it impossible to tell if
> following changes also have upgrade problems, or compound the existing
> issue. Likewise, if we have CI tests failing, subsequent landings make
> identifying and deal with further regressions vastly more painful, and
> hose our release process.
>

Depends on the changes. I think we should be pragmatic and make considered
decisions. I guess that's why we have the jfdi flag.


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Blocking bugs process

2015-07-13 Thread Martin Packman 
Thank you for responding Ian.

On 13/07/2015, Ian Booth  wrote:
>>
>> == Definition of blocking bugs ==
>> Master and all release branches should always be in a releasable
>> state. If a bug must be fixed for the next minor release, it is
>> considered a ‘blocker’ and will prevent all landing on that branch.
>>
>
> I don't agree with the above definition. There are always many bugs which
> must be fixed for the next minor release - these are assigned to the relevant
> milestones and marked high or critical.

So, my argument over this was a pretty strict interpretation of
"must". There are lots of bugs with less-than-critical severity that
get targeted at the next minor milestone, and I think that's perfect
reasonable. However, there are comparatively few bugs that could not
be deferred if, for instance, we discovered a security issue and had
to rush out a new minor release immediately.

From my perspective, blockers are things that break CI enough to
hinder our ability to do a release, or issues that if we allow into
release code will break users in unrecoverable ways. I know Aaron
prefers a much wider interpretation however.

> In the case of bug 1468653, this has been
> under investigation for 2 weeks and even though we are holding up the 1.24.3
> release for it, if it were a blocker the whole production line would have
> stalled unnecessarily.

That's an interesting bug. It seems with a lot of pain (manually
restarting things) it is actually repairable, and does involve a
pretty specific setup. I don't think I'd have added the blocker tag to
it, but that may not be the consensus.

> With follow on changes, the problem is
> quite isolated so landing fixes for other release critical issues should not
> be prevented.

The fact it's a somewhat isolated problem is important. What I really
want to avoid is the case where we leave say, upgrades broken as a
whole, and keep landing code. That makes it impossible to tell if
following changes also have upgrade problems, or compound the existing
issue. Likewise, if we have CI tests failing, subsequent landings make
identifying and deal with further regressions vastly more painful, and
hose our release process.

Martin

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Blocking bugs process

2015-07-13 Thread Ian Booth 
> 
> == Definition of blocking bugs ==
> Master and all release branches should always be in a releasable
> state. If a bug must be fixed for the next minor release, it is
> considered a ‘blocker’ and will prevent all landing on that branch.
>

I don't agree with the above definition. There are always many bugs which must
be fixed for the next minor release - these are assigned to the relevant
milestones and marked high or critical. We often hold a minor release until all
required bugs are fixed; this doesn't make them blockers and should not hold up
efforts to land fixes for those and other such bugs.

I thought a blocker was defined as a critical tagged ci+blocker. So it's
possible to have a critical bug, which would block a release, but not tagged as
a blocker. Also, as and stated, even some high bugs are required to be fixed
before we release (the current upgrade bug 1468653 a case in point) but by the
above definition would be blockers. In the case of bug 1468653, this has been
under investigation for 2 weeks and even though we are holding up the 1.24.3
release for it, if it were a blocker the whole production line would have
stalled unnecessarily.

> We block for two reasons:
> * To prevent problems from becoming compounded by follow-on changes.
> * As a stop-the-line, all-hands-on-deck signal to get more eyes on the 
> problem.
>

So taking bug 1468653 again (which is not currently a blocker but would be by
the above definition), putting more eyes on the problem would not help - the
right people are already looking at it. With follow on changes, the problem is
quite isolated so landing fixes for other release critical issues should not be
prevented.

So in summary, I agree with the need to block but not the literal interpretation
of the broad scope of the definition used above.

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Blocking bugs process

2015-07-13 Thread Nate Finch 
 Awesome :)

On Mon, Jul 13, 2015 at 2:29 PM Martin Packman 
wrote:

> On 13/07/2015, Nate Finch  wrote:
> > Can you put this in the wiki?
>
> Done, with Aaron's addition:
>
> 
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Blocking bugs process

2015-07-13 Thread Martin Packman 
On 13/07/2015, Nate Finch  wrote:
> Can you put this in the wiki?

Done, with Aaron's addition:



-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Blocking bugs process

2015-07-13 Thread Nate Finch 
Can you put this in the wiki?

On Thu, Jul 9, 2015 at 6:33 PM Martin Packman 
wrote:

> The QA team have been trying to hammer out a clearer process over
> blocking bugs, and have put together the document below for
> discussion. We'll be handling bugs as described here unless anyone has
> serious objections.
>
> Thanks!
>
> Martin
>
>
> == Definition of blocking bugs ==
> Master and all release branches should always be in a releasable
> state. If a bug must be fixed for the next minor release, it is
> considered a ‘blocker’ and will prevent all landing on that branch.
>
> We block for two reasons:
> * To prevent problems from becoming compounded by follow-on changes.
> * As a stop-the-line, all-hands-on-deck signal to get more eyes on the
> problem.
>
> A regression is a bug that is present in a version of juju that is not
> present in older juju versions.
> We are strict about regressions because our goal is to land these
> changes into Ubuntu, which is treating them as though they were
> bugfix-only releases.
>
> Regressions compared to juju versions going back to 1.18 prevent
> releases. This includes CLI or API incompatibility and other behaviour
> changes. Consistently failing tests will also prevent releases.
> Although ideally all regressions would block, regressions with limited
> impact, such as single test failures, do not initially need to block
> landings. To prevent branches remaining unreleasable for long periods,
> these bugs will be updated to block after a week.
>
> == Handling blocking bugs and regressions ==
>
> * File a bug
> * Mark as critical, target against next minor release.
> * Tag "blocker" unless it is a regression with limited impact.
> * Tag "ci" if it causes a CI test to fail.
> * If a particular revision introduced the issue, subscribe the author
> to the bug.
> * If trunk is blocked, alert the #juju-dev IRC channel or mailing juju-dev
> list.
> * If the bug was not tagged "blocker" but is not fixed within a week,
> tag it "blocker".
>
> == Unblocking ==
>
> * All bugs tagged "ci blocker" will be marked fix-released when the
> branch has a blessed tip.
> * QA will mark all other blockers fix-released when they determine
> them to be fixed.
> * Exceptions are raised to the release team.
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Blocking bugs process

2015-07-13 Thread Aaron Bentley 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 2015-07-09 06:33 PM, Martin Packman wrote:
> == Unblocking ==
> 
> * All bugs tagged "ci blocker" will be marked fix-released when
> the branch has a blessed tip. * QA will mark all other blockers
> fix-released when they determine them to be fixed. * Exceptions are
> raised to the release team.

I'd like to add "The blocker tag should not be removed except by the
person who added it or through raising an exception to the release team."

Aaron

-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQEcBAEBAgAGBQJVo+k7AAoJEK84cMOcf+9hX1kH/i3HwwiBts10NGrNQEYN2apT
dXFMfxd5MfaynjQrxJmL9hkh2zGq9M5KgxjgQKcwifcvh3RL/MWIu2N9Gtxnxee4
AO7sIrrGc0g9ECQ7WXLteGVSzqkyoRd+Q2N0h6e5P+1RrMMOU7rV/WtbnZ74BqXJ
/NmXN+cy/2Pkfef8DjPFFyj13sQud/fh1RDqCXa9Pml7UTTI7zhQYgQrbjmhgKZD
FdcrWzBYa5hNvThR5XX0rIAu0OhS1wySEH8KMwlav9IMCMSSoEI61d59jN3spdeW
apl+6ODI59eX6lxiwctFwSfkAf/1tKALyk0uNPPFmcTGYZ1Mq3BYfN65tRfLkFE=
=Nv+Z
-END PGP SIGNATURE-

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev