Operations Working Group - First Meeting

[Gastón Kleiman]

Hi all,

Thank you for responding to my previous emails - I think that we have
quorum for a new working group!

Many of you who have expressed interest seem to be in Europe, so I tried
schedule the first meeting at a time that I hope will be friendly for
people in both GMT+1 and GMT-8:

*Date:* Wednesday July 25th from 9:00-10:00 AM PDT
*Agenda:*
https://docs.google.com/document/d/1XjJfoksz70vbTvvT6FQ1t_J0SD1XIoipmYSvEHJfXt8/
*Zoom URL:* https://zoom.us/j/310132146


You can also find the event in the Mesos Community Calendar.

Feel free to add more topics to the agenda.

Looking forward to meeting you all next week,

-Gastón

Re: Backport Policy

[Vinod Kone]

I like how you summarized it Greg and I would vote for leaving the decision
to the committer too. In addition to what others mentioned, I think
committer should've the responsibility because if things break in a point
release (after it is released), it is the committer and contributor who are
on the hook to triage and fix it and not the release manager.

Having said that, if "during" the release process (i.e., cutting an RC)
these backports cause delays for a release manager in getting the release
out (e.g., CI flakiness introduced due to backports), release manager could
be the ultimate arbiter on whether such a backport should be reverted or
fixed by the committer/contributor. Hopefully such issues are caught much
before a release process is started (e.g., CI running against release
branches).

On Mon, Jul 16, 2018 at 1:28 PM Jie Yu  wrote:

> Greg, I like your idea of adding a prescriptive "policy" when evaluating
> whether a bug fix should be backported, and leave the decision to committer
> (because they have the most context, and avoid a bottleneck in the
> process).
>
> - Jie
>
> On Mon, Jul 16, 2018 at 11:24 AM, Greg Mann  wrote:
>
> > My impression is that we have two opposing schools of thought here:
> >
> >1. Backport as little as possible, to avoid unforeseen consequences
> >2. Backport as much as proves practical, to eliminate bugs in
> >supported versions
> >
> > Do other people agree with this assessment?
> >
> > If so, how can we find common ground? One possible solution would be to
> > leave the decision on backporting up to the committer, without
> specifying a
> > project-wide policy. This seems to be the status quo, and would lead to
> > some variation across committers regarding what types of fixes are
> > backported. We could also choose to delegate the decision to the release
> > manager; I favor leaving the decision with the committer, to eliminate
> the
> > burden on release managers.
> >
> > Here's a thought: rather than defining a prescriptive "policy" that we
> > expect committers to abide by, we could enumerate in the documentation
> the
> > competing concerns that we expect committers to consider when making
> > decisions on backports. The committing docs could read something like:
> >
> > "When bug fixes are committed to master, the committer should evaluate
> the
> > fix to determine whether or not it should be backported to supported
> > versions. This is left to the committer, but they are expected to weigh
> the
> > following concerns when making the decision:
> >
> >- Every backported change comes with a risk of unintended
> >consequences. The change should be carefully evaluated to ensure that
> such
> >side-effects are highly unlikely.
> >- As the complexity of applying a backport increases due to merge
> >conflicts, the likelihood of unintended consequences also increases.
> Bug
> >fixes which require extensive rebasing should only be backported when
> the
> >bug is critical enough to warrant the risk.
> >- Users of supported versions benefit greatly from the resolution of
> >bugs in point releases. Thus, whenever concerns #1 and #2 can be
> allayed
> >for a given bug fix, it should be backported."
> >
> >
> > Cheers,
> > Greg
> >
> >
> > On Mon, Jul 16, 2018 at 3:06 AM, Alex Rukletsov 
> > wrote:
> >
> >> Back porting as little as possible is the ultimate goal for me. My
> >> reasons are closely aligned with what Andrew wrote above.
> >>
> >> If we agree on this strategy, the next question is how to enforce it. My
> >> intuition is that committers will lean towards back porting their
> patches
> >> in arguable cases, because humans tend to overestimate the importance of
> >> their personal work. Delegating the decision in such cases to a release
> >> manager in my opinion will help us enforce the strategy of minimal
> number
> >> backports. As a bonus, the release manager will have a much better
> >> understanding of what's going on with the release, keyword: "more
> >> ownership".
> >>
> >> On Sat, Jul 14, 2018 at 12:07 AM, Andrew Schwartzmeyer <
> >> and...@schwartzmeyer.com> wrote:
> >>
> >>> I believe I fall somewhere between Alex and Ben.
> >>>
> >>> As for deciding what to backport or not, I lean toward Alex's view of
> >>> backporting as little as possible (and agree with his criteria). My
> >>> reasoning is that all changes can have unforeseen consequences, which I
> >>> believe is something to be actively avoided in already released
> versions.
> >>> The reason for backporting patches to fix regressions is the same as
> the
> >>> reason to avoid backporting as much as possible: keep behavior
> consistent
> >>> (and safe) within a release. With that as the goal of a branch in
> >>> maintenance mode, it makes sense to fix regressions, and make
> exceptions to
> >>> fix CVEs and other critical/blocking issues.
> >>>
> >>> As for who should decide what to backport, I lean toward Ben's view of
> >>> the burden b

Re: Backport Policy

[Jie Yu]

Greg, I like your idea of adding a prescriptive "policy" when evaluating
whether a bug fix should be backported, and leave the decision to committer
(because they have the most context, and avoid a bottleneck in the process).

- Jie

On Mon, Jul 16, 2018 at 11:24 AM, Greg Mann  wrote:

> My impression is that we have two opposing schools of thought here:
>
>1. Backport as little as possible, to avoid unforeseen consequences
>2. Backport as much as proves practical, to eliminate bugs in
>supported versions
>
> Do other people agree with this assessment?
>
> If so, how can we find common ground? One possible solution would be to
> leave the decision on backporting up to the committer, without specifying a
> project-wide policy. This seems to be the status quo, and would lead to
> some variation across committers regarding what types of fixes are
> backported. We could also choose to delegate the decision to the release
> manager; I favor leaving the decision with the committer, to eliminate the
> burden on release managers.
>
> Here's a thought: rather than defining a prescriptive "policy" that we
> expect committers to abide by, we could enumerate in the documentation the
> competing concerns that we expect committers to consider when making
> decisions on backports. The committing docs could read something like:
>
> "When bug fixes are committed to master, the committer should evaluate the
> fix to determine whether or not it should be backported to supported
> versions. This is left to the committer, but they are expected to weigh the
> following concerns when making the decision:
>
>- Every backported change comes with a risk of unintended
>consequences. The change should be carefully evaluated to ensure that such
>side-effects are highly unlikely.
>- As the complexity of applying a backport increases due to merge
>conflicts, the likelihood of unintended consequences also increases. Bug
>fixes which require extensive rebasing should only be backported when the
>bug is critical enough to warrant the risk.
>- Users of supported versions benefit greatly from the resolution of
>bugs in point releases. Thus, whenever concerns #1 and #2 can be allayed
>for a given bug fix, it should be backported."
>
>
> Cheers,
> Greg
>
>
> On Mon, Jul 16, 2018 at 3:06 AM, Alex Rukletsov 
> wrote:
>
>> Back porting as little as possible is the ultimate goal for me. My
>> reasons are closely aligned with what Andrew wrote above.
>>
>> If we agree on this strategy, the next question is how to enforce it. My
>> intuition is that committers will lean towards back porting their patches
>> in arguable cases, because humans tend to overestimate the importance of
>> their personal work. Delegating the decision in such cases to a release
>> manager in my opinion will help us enforce the strategy of minimal number
>> backports. As a bonus, the release manager will have a much better
>> understanding of what's going on with the release, keyword: "more
>> ownership".
>>
>> On Sat, Jul 14, 2018 at 12:07 AM, Andrew Schwartzmeyer <
>> and...@schwartzmeyer.com> wrote:
>>
>>> I believe I fall somewhere between Alex and Ben.
>>>
>>> As for deciding what to backport or not, I lean toward Alex's view of
>>> backporting as little as possible (and agree with his criteria). My
>>> reasoning is that all changes can have unforeseen consequences, which I
>>> believe is something to be actively avoided in already released versions.
>>> The reason for backporting patches to fix regressions is the same as the
>>> reason to avoid backporting as much as possible: keep behavior consistent
>>> (and safe) within a release. With that as the goal of a branch in
>>> maintenance mode, it makes sense to fix regressions, and make exceptions to
>>> fix CVEs and other critical/blocking issues.
>>>
>>> As for who should decide what to backport, I lean toward Ben's view of
>>> the burden being on the committer. I don't think we should add more work
>>> for release managers, and I think the committer/shepherd obviously has the
>>> most understanding of the context around changes proposed for backport.
>>>
>>> Here's an example of a recent bugfix which I backported:
>>> https://reviews.apache.org/r/67587/ (for MESOS-3790)
>>>
>>> While normally I believe this change falls under "avoid due to
>>> unforeseen consequences," I made an exception as the bug was old, circa
>>> 2015, (indicating it had been an issue for others), and was causing
>>> recurring failures in testing. The fix itself was very small, meaning it
>>> was easier to evaluate for possible side effects, so I felt a little safer
>>> in that regard. The effect of not having the fix was a fatal and undesired
>>> crash, which furthermore left troublesome side effects on the system (you
>>> couldn't bring the agent back up). And lastly, a dependent project (DC/OS)
>>> wanted it in their next bump, which necessitated backporting to the release
>>> they were pullin

Re: Backport Policy

[Greg Mann]

My impression is that we have two opposing schools of thought here:

   1. Backport as little as possible, to avoid unforeseen consequences
   2. Backport as much as proves practical, to eliminate bugs in supported
   versions

Do other people agree with this assessment?

If so, how can we find common ground? One possible solution would be to
leave the decision on backporting up to the committer, without specifying a
project-wide policy. This seems to be the status quo, and would lead to
some variation across committers regarding what types of fixes are
backported. We could also choose to delegate the decision to the release
manager; I favor leaving the decision with the committer, to eliminate the
burden on release managers.

Here's a thought: rather than defining a prescriptive "policy" that we
expect committers to abide by, we could enumerate in the documentation the
competing concerns that we expect committers to consider when making
decisions on backports. The committing docs could read something like:

"When bug fixes are committed to master, the committer should evaluate the
fix to determine whether or not it should be backported to supported
versions. This is left to the committer, but they are expected to weigh the
following concerns when making the decision:

   - Every backported change comes with a risk of unintended consequences.
   The change should be carefully evaluated to ensure that such side-effects
   are highly unlikely.
   - As the complexity of applying a backport increases due to merge
   conflicts, the likelihood of unintended consequences also increases. Bug
   fixes which require extensive rebasing should only be backported when the
   bug is critical enough to warrant the risk.
   - Users of supported versions benefit greatly from the resolution of
   bugs in point releases. Thus, whenever concerns #1 and #2 can be allayed
   for a given bug fix, it should be backported."


Cheers,
Greg


On Mon, Jul 16, 2018 at 3:06 AM, Alex Rukletsov  wrote:

> Back porting as little as possible is the ultimate goal for me. My reasons
> are closely aligned with what Andrew wrote above.
>
> If we agree on this strategy, the next question is how to enforce it. My
> intuition is that committers will lean towards back porting their patches
> in arguable cases, because humans tend to overestimate the importance of
> their personal work. Delegating the decision in such cases to a release
> manager in my opinion will help us enforce the strategy of minimal number
> backports. As a bonus, the release manager will have a much better
> understanding of what's going on with the release, keyword: "more
> ownership".
>
> On Sat, Jul 14, 2018 at 12:07 AM, Andrew Schwartzmeyer <
> and...@schwartzmeyer.com> wrote:
>
>> I believe I fall somewhere between Alex and Ben.
>>
>> As for deciding what to backport or not, I lean toward Alex's view of
>> backporting as little as possible (and agree with his criteria). My
>> reasoning is that all changes can have unforeseen consequences, which I
>> believe is something to be actively avoided in already released versions.
>> The reason for backporting patches to fix regressions is the same as the
>> reason to avoid backporting as much as possible: keep behavior consistent
>> (and safe) within a release. With that as the goal of a branch in
>> maintenance mode, it makes sense to fix regressions, and make exceptions to
>> fix CVEs and other critical/blocking issues.
>>
>> As for who should decide what to backport, I lean toward Ben's view of
>> the burden being on the committer. I don't think we should add more work
>> for release managers, and I think the committer/shepherd obviously has the
>> most understanding of the context around changes proposed for backport.
>>
>> Here's an example of a recent bugfix which I backported:
>> https://reviews.apache.org/r/67587/ (for MESOS-3790)
>>
>> While normally I believe this change falls under "avoid due to unforeseen
>> consequences," I made an exception as the bug was old, circa 2015,
>> (indicating it had been an issue for others), and was causing recurring
>> failures in testing. The fix itself was very small, meaning it was easier
>> to evaluate for possible side effects, so I felt a little safer in that
>> regard. The effect of not having the fix was a fatal and undesired crash,
>> which furthermore left troublesome side effects on the system (you couldn't
>> bring the agent back up). And lastly, a dependent project (DC/OS) wanted it
>> in their next bump, which necessitated backporting to the release they were
>> pulling in.
>>
>> I think in general we should backport only as necessary, and leave it on
>> the committers to decide if backporting a particular change is necessary.
>>
>>
>> On 07/13/2018 12:54 am, Alex Rukletsov wrote:
>>
>>> This is exactly where our views differ, Ben : )
>>>
>>> Ideally, I would like a release manager to have more ownership and less
>>> manual work. In my imagination, a release manager has m

Re: Backport Policy

[Alex Rukletsov]

Back porting as little as possible is the ultimate goal for me. My reasons
are closely aligned with what Andrew wrote above.

If we agree on this strategy, the next question is how to enforce it. My
intuition is that committers will lean towards back porting their patches
in arguable cases, because humans tend to overestimate the importance of
their personal work. Delegating the decision in such cases to a release
manager in my opinion will help us enforce the strategy of minimal number
backports. As a bonus, the release manager will have a much better
understanding of what's going on with the release, keyword: "more
ownership".

On Sat, Jul 14, 2018 at 12:07 AM, Andrew Schwartzmeyer <
and...@schwartzmeyer.com> wrote:

> I believe I fall somewhere between Alex and Ben.
>
> As for deciding what to backport or not, I lean toward Alex's view of
> backporting as little as possible (and agree with his criteria). My
> reasoning is that all changes can have unforeseen consequences, which I
> believe is something to be actively avoided in already released versions.
> The reason for backporting patches to fix regressions is the same as the
> reason to avoid backporting as much as possible: keep behavior consistent
> (and safe) within a release. With that as the goal of a branch in
> maintenance mode, it makes sense to fix regressions, and make exceptions to
> fix CVEs and other critical/blocking issues.
>
> As for who should decide what to backport, I lean toward Ben's view of the
> burden being on the committer. I don't think we should add more work for
> release managers, and I think the committer/shepherd obviously has the most
> understanding of the context around changes proposed for backport.
>
> Here's an example of a recent bugfix which I backported:
> https://reviews.apache.org/r/67587/ (for MESOS-3790)
>
> While normally I believe this change falls under "avoid due to unforeseen
> consequences," I made an exception as the bug was old, circa 2015,
> (indicating it had been an issue for others), and was causing recurring
> failures in testing. The fix itself was very small, meaning it was easier
> to evaluate for possible side effects, so I felt a little safer in that
> regard. The effect of not having the fix was a fatal and undesired crash,
> which furthermore left troublesome side effects on the system (you couldn't
> bring the agent back up). And lastly, a dependent project (DC/OS) wanted it
> in their next bump, which necessitated backporting to the release they were
> pulling in.
>
> I think in general we should backport only as necessary, and leave it on
> the committers to decide if backporting a particular change is necessary.
>
>
> On 07/13/2018 12:54 am, Alex Rukletsov wrote:
>
>> This is exactly where our views differ, Ben : )
>>
>> Ideally, I would like a release manager to have more ownership and less
>> manual work. In my imagination, a release manager has more power and
>> control about dates, features, backports and everything that is related to
>> "their" branch. I would also like us to back port as little as possible,
>> to
>> simplify testing and releasing patch versions.
>>
>> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler 
>> wrote:
>>
>> +user, I probably it would be good to hear from users as well.
>>>
>>> Please see the original proposal as well as Alex's proposal and let us
>>> know
>>> your thoughts.
>>>
>>> To continue the discussion from where Alex left off:
>>>
>>> > Other bugs and significant improvements, e.g., performance, may be back
>>> ported,
>>> the release manager should ideally be the one who decides on this.
>>>
>>> I'm a little puzzled by this, why is the release manager involved? As we
>>> already document, backports occur when the bug is fixed, so this happens
>>> in
>>> the steady state of development, not at release time. The release manager
>>> only comes in at the time of the release itself, at which point all
>>> backports have already happened and the release manager handles the
>>> release
>>> process. Only blocker level issues can stop the release and while the
>>> release manager has a strong say, we should generally agree on what
>>> consists of a release blocking issue.
>>>
>>> Just to clarify my workflow, I generally backport every bug fix I commit
>>> that applies cleanly, right after I commit it to master (with the
>>> exceptions I listed below).
>>>
>>> On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov 
>>> wrote:
>>>
>>> > I would like to back port as little as possible. I suggest the
>>> following
>>> > criteria:
>>> >
>>> > * By default, regressions are back ported to existing release
>>> branches. A
>>> > bug is considered a regression if the functionality is present in the
>>> > previous minor or patch version and is not affected by the bug there.
>>> >
>>> > * Critical and blocker issues, e.g., a CVE, can be back ported.
>>> >
>>> > * Other bugs and significant improvements, e.g., performance, may be
>>> back
>>> > ported, the release manager