Re: Backport Policy

[Andrew Schwartzmeyer]

I believe I fall somewhere between Alex and Ben.

As for deciding what to backport or not, I lean toward Alex's view of 
backporting as little as possible (and agree with his criteria). My 
reasoning is that all changes can have unforeseen consequences, which I 
believe is something to be actively avoided in already released 
versions. The reason for backporting patches to fix regressions is the 
same as the reason to avoid backporting as much as possible: keep 
behavior consistent (and safe) within a release. With that as the goal 
of a branch in maintenance mode, it makes sense to fix regressions, and 
make exceptions to fix CVEs and other critical/blocking issues.


As for who should decide what to backport, I lean toward Ben's view of 
the burden being on the committer. I don't think we should add more work 
for release managers, and I think the committer/shepherd obviously has 
the most understanding of the context around changes proposed for 
backport.


Here's an example of a recent bugfix which I backported: 
https://reviews.apache.org/r/67587/ (for MESOS-3790)


While normally I believe this change falls under "avoid due to 
unforeseen consequences," I made an exception as the bug was old, circa 
2015, (indicating it had been an issue for others), and was causing 
recurring failures in testing. The fix itself was very small, meaning it 
was easier to evaluate for possible side effects, so I felt a little 
safer in that regard. The effect of not having the fix was a fatal and 
undesired crash, which furthermore left troublesome side effects on the 
system (you couldn't bring the agent back up). And lastly, a dependent 
project (DC/OS) wanted it in their next bump, which necessitated 
backporting to the release they were pulling in.


I think in general we should backport only as necessary, and leave it on 
the committers to decide if backporting a particular change is 
necessary.


On 07/13/2018 12:54 am, Alex Rukletsov wrote:

This is exactly where our views differ, Ben : )

Ideally, I would like a release manager to have more ownership and less
manual work. In my imagination, a release manager has more power and
control about dates, features, backports and everything that is related 
to
"their" branch. I would also like us to back port as little as 
possible, to

simplify testing and releasing patch versions.

On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler  
wrote:



+user, I probably it would be good to hear from users as well.

Please see the original proposal as well as Alex's proposal and let us 
know

your thoughts.

To continue the discussion from where Alex left off:

> Other bugs and significant improvements, e.g., performance, may be back
ported,
the release manager should ideally be the one who decides on this.

I'm a little puzzled by this, why is the release manager involved? As 
we
already document, backports occur when the bug is fixed, so this 
happens in
the steady state of development, not at release time. The release 
manager

only comes in at the time of the release itself, at which point all
backports have already happened and the release manager handles the 
release

process. Only blocker level issues can stop the release and while the
release manager has a strong say, we should generally agree on what
consists of a release blocking issue.

Just to clarify my workflow, I generally backport every bug fix I 
commit

that applies cleanly, right after I commit it to master (with the
exceptions I listed below).

On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov 
wrote:

> I would like to back port as little as possible. I suggest the following
> criteria:
>
> * By default, regressions are back ported to existing release branches. A
> bug is considered a regression if the functionality is present in the
> previous minor or patch version and is not affected by the bug there.
>
> * Critical and blocker issues, e.g., a CVE, can be back ported.
>
> * Other bugs and significant improvements, e.g., performance, may be back
> ported, the release manager should ideally be the one who decides on
this.
>
> On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone 
wrote:
>
> > Ben, thanks for the clarification. I'm in agreement with the points you
> > made.
> >
> > Once we have consensus, would you mind updating the doc?
> >
> > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler 
> > wrote:
> >
> > > I realized recently that we aren't all on the same page with
> backporting.
> > > We currently only document the following:
> > >
> > > "Typically the fix for an issue that is affecting supported releases
> > lands
> > > on the master branch and is then backported to the release
branch(es).
> In
> > > rare cases, the fix might directly go into a release branch without
> > landing
> > > on master (e.g., fix / issue is not applicable to master)." [1]
> > >
> > > This leaves room for interpretation about what lies outside of
> "typical".
> > > Here's the simplest way I can explain what I stick to, and I'd like

Re: Backport Policy

[Greg Mann]

It seems to me that putting the burden of deciding on backports on the
release manager would actually increase the amount of work required. Simply
cutting the release on a particular date is pretty quick - however,
examining tickets to determine whether or not a particular fix should be
backported seems like more effort?

The backport policy strikes me as a community decision, rather than an
individual decision. The community discusses (as we're doing now :) and
settles on a policy, which individual committers then execute. Putting each
release manager in charge of the backport policy for that particular
release would lead to inconsistency across releases. I'm not sure that this
is a terrible thing, but it's not a particularly good thing either.

I would propose that we establish a single backport policy that we all do
our best to execute, with an understanding that there will always be room
for exceptions in some situations.

I like the idea of backporting all bug fixes which apply relatively
cleanly. In addition, very critical bug fixes are worth backporting even
when extensive work is required to backport them.


Alex, could you elaborate on why you would like to backport as little as
possible? I would like to better understand your motivations there :)

Cheers,
Greg


On Fri, Jul 13, 2018 at 2:40 PM, Jie Yu  wrote:

> I typically backport all bug fixes that cleanly apply and the risk is low.
> It's a judgement call, but many of the time, you can easily tell the risk
> is low.
>
> I think my argument on why we want to do this is "why not". I want our
> software to have less bugs!
>
> Letting release manager decides which patch to backport or not does not
> scale. Some release managers might even become dormant after a while.
>
> - Jie
>
> On Fri, Jul 13, 2018 at 12:54 AM, Alex Rukletsov 
> wrote:
>
>> This is exactly where our views differ, Ben : )
>>
>> Ideally, I would like a release manager to have more ownership and less
>> manual work. In my imagination, a release manager has more power and
>> control about dates, features, backports and everything that is related to
>> "their" branch. I would also like us to back port as little as possible,
>> to
>> simplify testing and releasing patch versions.
>>
>> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler 
>> wrote:
>>
>> > +user, I probably it would be good to hear from users as well.
>> >
>> > Please see the original proposal as well as Alex's proposal and let us
>> know
>> > your thoughts.
>> >
>> > To continue the discussion from where Alex left off:
>> >
>> > > Other bugs and significant improvements, e.g., performance, may be
>> back
>> > ported,
>> > the release manager should ideally be the one who decides on this.
>> >
>> > I'm a little puzzled by this, why is the release manager involved? As we
>> > already document, backports occur when the bug is fixed, so this
>> happens in
>> > the steady state of development, not at release time. The release
>> manager
>> > only comes in at the time of the release itself, at which point all
>> > backports have already happened and the release manager handles the
>> release
>> > process. Only blocker level issues can stop the release and while the
>> > release manager has a strong say, we should generally agree on what
>> > consists of a release blocking issue.
>> >
>> > Just to clarify my workflow, I generally backport every bug fix I commit
>> > that applies cleanly, right after I commit it to master (with the
>> > exceptions I listed below).
>> >
>> > On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov 
>> > wrote:
>> >
>> > > I would like to back port as little as possible. I suggest the
>> following
>> > > criteria:
>> > >
>> > > * By default, regressions are back ported to existing release
>> branches. A
>> > > bug is considered a regression if the functionality is present in the
>> > > previous minor or patch version and is not affected by the bug there.
>> > >
>> > > * Critical and blocker issues, e.g., a CVE, can be back ported.
>> > >
>> > > * Other bugs and significant improvements, e.g., performance, may be
>> back
>> > > ported, the release manager should ideally be the one who decides on
>> > this.
>> > >
>> > > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone 
>> > wrote:
>> > >
>> > > > Ben, thanks for the clarification. I'm in agreement with the points
>> you
>> > > > made.
>> > > >
>> > > > Once we have consensus, would you mind updating the doc?
>> > > >
>> > > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler > >
>> > > > wrote:
>> > > >
>> > > > > I realized recently that we aren't all on the same page with
>> > > backporting.
>> > > > > We currently only document the following:
>> > > > >
>> > > > > "Typically the fix for an issue that is affecting supported
>> releases
>> > > > lands
>> > > > > on the master branch and is then backported to the release
>> > branch(es).
>> > > In
>> > > > > rare cases, the fix might directly go into a release branch
>> without
>> > > > landing
>> > > > > on 

Re: Backport Policy

[Jie Yu]

I typically backport all bug fixes that cleanly apply and the risk is low.
It's a judgement call, but many of the time, you can easily tell the risk
is low.

I think my argument on why we want to do this is "why not". I want our
software to have less bugs!

Letting release manager decides which patch to backport or not does not
scale. Some release managers might even become dormant after a while.

- Jie

On Fri, Jul 13, 2018 at 12:54 AM, Alex Rukletsov 
wrote:

> This is exactly where our views differ, Ben : )
>
> Ideally, I would like a release manager to have more ownership and less
> manual work. In my imagination, a release manager has more power and
> control about dates, features, backports and everything that is related to
> "their" branch. I would also like us to back port as little as possible, to
> simplify testing and releasing patch versions.
>
> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler 
> wrote:
>
> > +user, I probably it would be good to hear from users as well.
> >
> > Please see the original proposal as well as Alex's proposal and let us
> know
> > your thoughts.
> >
> > To continue the discussion from where Alex left off:
> >
> > > Other bugs and significant improvements, e.g., performance, may be back
> > ported,
> > the release manager should ideally be the one who decides on this.
> >
> > I'm a little puzzled by this, why is the release manager involved? As we
> > already document, backports occur when the bug is fixed, so this happens
> in
> > the steady state of development, not at release time. The release manager
> > only comes in at the time of the release itself, at which point all
> > backports have already happened and the release manager handles the
> release
> > process. Only blocker level issues can stop the release and while the
> > release manager has a strong say, we should generally agree on what
> > consists of a release blocking issue.
> >
> > Just to clarify my workflow, I generally backport every bug fix I commit
> > that applies cleanly, right after I commit it to master (with the
> > exceptions I listed below).
> >
> > On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov 
> > wrote:
> >
> > > I would like to back port as little as possible. I suggest the
> following
> > > criteria:
> > >
> > > * By default, regressions are back ported to existing release
> branches. A
> > > bug is considered a regression if the functionality is present in the
> > > previous minor or patch version and is not affected by the bug there.
> > >
> > > * Critical and blocker issues, e.g., a CVE, can be back ported.
> > >
> > > * Other bugs and significant improvements, e.g., performance, may be
> back
> > > ported, the release manager should ideally be the one who decides on
> > this.
> > >
> > > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone 
> > wrote:
> > >
> > > > Ben, thanks for the clarification. I'm in agreement with the points
> you
> > > > made.
> > > >
> > > > Once we have consensus, would you mind updating the doc?
> > > >
> > > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler 
> > > > wrote:
> > > >
> > > > > I realized recently that we aren't all on the same page with
> > > backporting.
> > > > > We currently only document the following:
> > > > >
> > > > > "Typically the fix for an issue that is affecting supported
> releases
> > > > lands
> > > > > on the master branch and is then backported to the release
> > branch(es).
> > > In
> > > > > rare cases, the fix might directly go into a release branch without
> > > > landing
> > > > > on master (e.g., fix / issue is not applicable to master)." [1]
> > > > >
> > > > > This leaves room for interpretation about what lies outside of
> > > "typical".
> > > > > Here's the simplest way I can explain what I stick to, and I'd like
> > to
> > > > hear
> > > > > what others have in mind:
> > > > >
> > > > > * By default, bug fixes at any level should be backported to
> existing
> > > > > release branches if it affects those releases. Especially
> important:
> > > > > crashes, bugs in non-experimental features.
> > > > >
> > > > > * Exceptional cases that can omit backporting: difficult to
> backport
> > > > fixes
> > > > > (especially if the bugs are deemed of low priority), bugs in
> > > experimental
> > > > > features.
> > > > >
> > > > > * Exceptional non-bug cases that can be backported: performance
> > > > > improvements.
> > > > >
> > > > > I realize that there is a ton of subtlety here (even in terms of
> > which
> > > > > things are defined as bugs). But I hope we can lay down a policy
> that
> > > > gives
> > > > > everyone the right mindset for common cases and then discuss corner
> > > cases
> > > > > on-demand in the future.
> > > > >
> > > > > [1] http://mesos.apache.org/documentation/latest/versioning/
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Release Apache Mesos 1.6.1 (rc2)

[Vinod Kone]

+1 (binding)

Ran through ASF CI. Red builds were known health check / check flaky tests.

*Revision*: ae82dd5cc6f415916702897acfd3085b6387b118

   - refs/tags/1.6.1-rc2

Configuration Matrix gcc clang
centos:7 --verbose --enable-libevent --enable-ssl autotools
[image: Failed]

[image: Not run]
cmake
[image: Success]

[image: Not run]
--verbose autotools
[image: Failed]

[image: Not run]
cmake
[image: Success]

[image: Not run]
ubuntu:14.04 --verbose --enable-libevent --enable-ssl autotools
[image: Failed]

[image: Failed]

cmake
[image: Success]

[image: Success]

--verbose autotools
[image: Success]

[image: Success]

cmake
[image: Success]

[image: Success]



On Fri, Jul 13, 2018 at 12:48 PM Chun-Hung Hsiao 
wrote:

> +1 (binding)
>
> Tested on our internal CI. All green.
> Tested on my Mac with both autotools and CMake, with gRPC enabled.
> Failed tests:
>
> HealthCheckTest.ROOT_INTERNET_CURL_HealthyTaskViaHTTPWithContainerImage
> HealthCheckTest.ROOT_INTERNET_CURL_HealthyTaskViaHTTPSWithContainerImage
> HealthCheckTest.ROOT_INTERNET_CURL_HealthyTaskViaTCPWithContainerImage
> FetcherCacheTest.LocalUncachedExtract
> FetcherCacheHttpTest.HttpMixed
>
> MesosContainerizer/DefaultExecutorTest.ROOT_INTERNET_CURL_DockerTaskWithFileURI
> MesosContainerizer/DefaultExecutorTest.ROOT_LaunchGroupFailure
>
> LauncherAndIsolationParam/PersistentVolumeDefaultExecutor.ROOT_PersistentResources
>
> LauncherAndIsolationParam/PersistentVolumeDefaultExecutor.ROOT_TaskSandboxPersistentVolume
>
> LauncherAndIsolationParam/PersistentVolumeDefaultExecutor.ROOT_TasksSharingViaSandboxVolumes
>
> LauncherAndIsolationParam/PersistentVolumeDefaultExecutor.ROOT_TaskGroupsSharingViaSandboxVolumes
>
> LauncherAndIsolationParam/PersistentVolumeDefaultExecutor.ROOT_HealthCheckUsingPersistentVolume
>
> All of the above tests require the `filesystem/linux` isolator so are
> supposed to fail 

Re: [VOTE] Release Apache Mesos 1.6.1 (rc2)

[Chun-Hung Hsiao]

+1 (binding)

Tested on our internal CI. All green.
Tested on my Mac with both autotools and CMake, with gRPC enabled.
Failed tests:

HealthCheckTest.ROOT_INTERNET_CURL_HealthyTaskViaHTTPWithContainerImage
HealthCheckTest.ROOT_INTERNET_CURL_HealthyTaskViaHTTPSWithContainerImage
HealthCheckTest.ROOT_INTERNET_CURL_HealthyTaskViaTCPWithContainerImage
FetcherCacheTest.LocalUncachedExtract
FetcherCacheHttpTest.HttpMixed
MesosContainerizer/DefaultExecutorTest.ROOT_INTERNET_CURL_DockerTaskWithFileURI
MesosContainerizer/DefaultExecutorTest.ROOT_LaunchGroupFailure
LauncherAndIsolationParam/PersistentVolumeDefaultExecutor.ROOT_PersistentResources
LauncherAndIsolationParam/PersistentVolumeDefaultExecutor.ROOT_TaskSandboxPersistentVolume
LauncherAndIsolationParam/PersistentVolumeDefaultExecutor.ROOT_TasksSharingViaSandboxVolumes
LauncherAndIsolationParam/PersistentVolumeDefaultExecutor.ROOT_TaskGroupsSharingViaSandboxVolumes
LauncherAndIsolationParam/PersistentVolumeDefaultExecutor.ROOT_HealthCheckUsingPersistentVolume

All of the above tests require the `filesystem/linux` isolator so are
supposed to fail on a Mac.

On Thu, Jul 12, 2018 at 8:50 AM Greg Mann  wrote:

> Yep, I trimmed the list to make it more digestible and provided a list of
> "notable" bug fixes. The entire list of changes can be found in the
> CHANGELOG.
>
> On Thu, Jul 12, 2018, 7:56 AM Chun-Hung Hsiao 
> wrote:
>
>> Seems you missed MESOS-9049. And this seems not just a bug fix release
>> because of MESOS-8934? ;)
>>
>> On Wed, Jul 11, 2018, 9:37 PM Greg Mann  wrote:
>>
>>> Whoops, I forgot to include the list of changes included in this release
>>> - sorry!
>>>
>>> 1.6.1-rc2 includes the following notable bug fixes:
>>>
>>>   * [MESOS-3790] - ZooKeeper connection should retry on `EAI_NONAME`.
>>>   * [MESOS-8830] - Agent gc on old slave sandboxes could empty
>>> persistent volume data
>>>   * [MESOS-8871] - Agent may fail to recover if the agent dies before
>>> image store cache checkpointed.
>>>   * [MESOS-8904] - Master crash when removing quota.
>>>   * [MESOS-8936] - Implement a Random Sorter for offer allocations.
>>>   * [MESOS-8945] - Master check failure due to CHECK_SOME(providerId).
>>>   * [MESOS-8963] - Executor crash trying to print container ID.
>>>   * [MESOS-8980] - mesos-slave can deadlock with docker pull.
>>>   * [MESOS-8986] - `slave.available()` in the allocator is expensive and
>>> drags down allocation performance.
>>>   * [MESOS-8987] - Master asks agent to shutdown upon auth errors.
>>>   * [MESOS-9002] - GCC 8.1 build failure in os::Fork::Tree.
>>>   * [MESOS-9024] - Mesos master segfaults with stack overflow under load.
>>>   * [MESOS-9025] - The container which joins CNI network and has
>>> checkpoint enabled will be mistakenly destroyed by agent.
>>>
>>> Cheers,
>>> Greg
>>>
>>> On Wed, Jul 11, 2018 at 6:15 PM, Greg Mann  wrote:
>>>
 Hi all,

 Please vote on releasing the following candidate as Apache Mesos 1.6.1.


 1.6.1 includes the following:

 
 *Announce major features here*
 *Announce major bug fixes here*

 The CHANGELOG for the release is available at:

 https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.6.1-rc2

 

 The candidate for Mesos 1.6.1 release is available at:

 https://dist.apache.org/repos/dist/dev/mesos/1.6.1-rc2/mesos-1.6.1.tar.gz

 The tag to be voted on is 1.6.1-rc2:
 https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=1.6.1-rc2

 The SHA512 checksum of the tarball can be found at:

 https://dist.apache.org/repos/dist/dev/mesos/1.6.1-rc2/mesos-1.6.1.tar.gz.sha512

 The signature of the tarball can be found at:

 https://dist.apache.org/repos/dist/dev/mesos/1.6.1-rc2/mesos-1.6.1.tar.gz.asc

 The PGP key used to sign the release is here:
 https://dist.apache.org/repos/dist/release/mesos/KEYS

 The JAR is in a staging repository here:
 https://repository.apache.org/content/repositories/orgapachemesos-1230

 Please vote on releasing this package as Apache Mesos 1.6.1!

 The vote is open until Mon Jul 16 18:15:00 PDT 2018 and passes if a
 majority of at least 3 +1 PMC votes are cast.

 [ ] +1 Release this package as Apache Mesos 1.6.1
 [ ] -1 Do not release this package because ...

 Thanks,
 Greg

>>>
>>>

Re: Backport Policy

[Alex Rukletsov]

This is exactly where our views differ, Ben : )

Ideally, I would like a release manager to have more ownership and less
manual work. In my imagination, a release manager has more power and
control about dates, features, backports and everything that is related to
"their" branch. I would also like us to back port as little as possible, to
simplify testing and releasing patch versions.

On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler  wrote:

> +user, I probably it would be good to hear from users as well.
>
> Please see the original proposal as well as Alex's proposal and let us know
> your thoughts.
>
> To continue the discussion from where Alex left off:
>
> > Other bugs and significant improvements, e.g., performance, may be back
> ported,
> the release manager should ideally be the one who decides on this.
>
> I'm a little puzzled by this, why is the release manager involved? As we
> already document, backports occur when the bug is fixed, so this happens in
> the steady state of development, not at release time. The release manager
> only comes in at the time of the release itself, at which point all
> backports have already happened and the release manager handles the release
> process. Only blocker level issues can stop the release and while the
> release manager has a strong say, we should generally agree on what
> consists of a release blocking issue.
>
> Just to clarify my workflow, I generally backport every bug fix I commit
> that applies cleanly, right after I commit it to master (with the
> exceptions I listed below).
>
> On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov 
> wrote:
>
> > I would like to back port as little as possible. I suggest the following
> > criteria:
> >
> > * By default, regressions are back ported to existing release branches. A
> > bug is considered a regression if the functionality is present in the
> > previous minor or patch version and is not affected by the bug there.
> >
> > * Critical and blocker issues, e.g., a CVE, can be back ported.
> >
> > * Other bugs and significant improvements, e.g., performance, may be back
> > ported, the release manager should ideally be the one who decides on
> this.
> >
> > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone 
> wrote:
> >
> > > Ben, thanks for the clarification. I'm in agreement with the points you
> > > made.
> > >
> > > Once we have consensus, would you mind updating the doc?
> > >
> > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler 
> > > wrote:
> > >
> > > > I realized recently that we aren't all on the same page with
> > backporting.
> > > > We currently only document the following:
> > > >
> > > > "Typically the fix for an issue that is affecting supported releases
> > > lands
> > > > on the master branch and is then backported to the release
> branch(es).
> > In
> > > > rare cases, the fix might directly go into a release branch without
> > > landing
> > > > on master (e.g., fix / issue is not applicable to master)." [1]
> > > >
> > > > This leaves room for interpretation about what lies outside of
> > "typical".
> > > > Here's the simplest way I can explain what I stick to, and I'd like
> to
> > > hear
> > > > what others have in mind:
> > > >
> > > > * By default, bug fixes at any level should be backported to existing
> > > > release branches if it affects those releases. Especially important:
> > > > crashes, bugs in non-experimental features.
> > > >
> > > > * Exceptional cases that can omit backporting: difficult to backport
> > > fixes
> > > > (especially if the bugs are deemed of low priority), bugs in
> > experimental
> > > > features.
> > > >
> > > > * Exceptional non-bug cases that can be backported: performance
> > > > improvements.
> > > >
> > > > I realize that there is a ton of subtlety here (even in terms of
> which
> > > > things are defined as bugs). But I hope we can lay down a policy that
> > > gives
> > > > everyone the right mindset for common cases and then discuss corner
> > cases
> > > > on-demand in the future.
> > > >
> > > > [1] http://mesos.apache.org/documentation/latest/versioning/
> > > >
> > >
> >
>