Thanks for clarifying the backporting policy, BenM! I totally agree with the changes proposed for the backporting policy, but I realize two more scenarios that are more clear to me yet:
- There are some bugs that are not fixable (due to legacy technical decisions), and we end up with fixing the issue by a semantic/behavior change in a new release. Do we expect this semantic/behavior change being backported? - There might be some bugs that root cause is unknown yet, but it did impact on a couple releases. If we decide to add some commits for debugging purpose (e.g., a new debugging endpoint, or more logging), should we also allow these patches to be backported? For #2, I think we should do the backporting, but for #1, maybe more discussion is needed since it relates to whether the user has to upgrade or not. Cheers, Gilbert On Tue, Jul 17, 2018 at 4:26 PM, Lawrence Rau <larry...@mac.com> wrote: > I don’t have a big stake in, however, one opinion is if a large commercial > enterprise was using a specific release that is working the desire is often > to only upgrade if necessary. Necessary can be for a number of reasons > including new features; however if a new feature is not needed the > compelling reason to upgrade is to fix a specific problem that is causing > issues. Thus keeping a maintenance release stable is very important and > reducing the chance for, while fixing one problem, introducing another. > > Often a clear classification of severity of the problem would dictate the > need to make a change. (yes these can be subjective, but some guidance > would be better than nothing). > > It might be good to give committers guidance on back porting things that > have a high impact on improving a problem. Fixing a crashing bug, fixing a > degenerative performance issue, etc, where these issues have no easy/viable > work around. Nice to have fixes aren’t, always, worth updating to. > > There can be an argument to respond with a “then don’t upgrade” but if > changing the release with “nice to have’s” and several point releases later > when a critical bug is fixed then the org if forced to accept the risk of > the nice to have’s. > > just an opinion. > …larry > > > On Jul 17, 2018, at 3:00 PM, Chun-Hung Hsiao <chhs...@mesosphere.io> > wrote: > > I just have a comment on a special case: > If a fix for a flaky test is easy to backport, > IMO we probably should backport it, > otherwise if someone backports another critical fix in the future, > it would take them extra effort to check all CI failures. > > On Mon, Jul 16, 2018 at 11:39 AM Vinod Kone <vinodk...@apache.org> wrote: > >> I like how you summarized it Greg and I would vote for leaving the >> decision >> to the committer too. In addition to what others mentioned, I think >> committer should've the responsibility because if things break in a point >> release (after it is released), it is the committer and contributor who >> are >> on the hook to triage and fix it and not the release manager. >> >> Having said that, if "during" the release process (i.e., cutting an RC) >> these backports cause delays for a release manager in getting the release >> out (e.g., CI flakiness introduced due to backports), release manager >> could >> be the ultimate arbiter on whether such a backport should be reverted or >> fixed by the committer/contributor. Hopefully such issues are caught much >> before a release process is started (e.g., CI running against release >> branches). >> >> On Mon, Jul 16, 2018 at 1:28 PM Jie Yu <yujie....@gmail.com> wrote: >> >> > Greg, I like your idea of adding a prescriptive "policy" when evaluating >> > whether a bug fix should be backported, and leave the decision to >> committer >> > (because they have the most context, and avoid a bottleneck in the >> > process). >> > >> > - Jie >> > >> > On Mon, Jul 16, 2018 at 11:24 AM, Greg Mann <g...@mesosphere.io> wrote: >> > >> > > My impression is that we have two opposing schools of thought here: >> > > >> > > 1. Backport as little as possible, to avoid unforeseen consequences >> > > 2. Backport as much as proves practical, to eliminate bugs in >> > > supported versions >> > > >> > > Do other people agree with this assessment? >> > > >> > > If so, how can we find common ground? One possible solution would be >> to >> > > leave the decision on backporting up to the committer, without >> > specifying a >> > > project-wide policy. This seems to be the status quo, and would lead >> to >> > > some variation across committers regarding what types of fixes are >> > > backported. We could also choose to delegate the decision to the >> release >> > > manager; I favor leaving the decision with the committer, to eliminate >> > the >> > > burden on release managers. >> > > >> > > Here's a thought: rather than defining a prescriptive "policy" that we >> > > expect committers to abide by, we could enumerate in the documentation >> > the >> > > competing concerns that we expect committers to consider when making >> > > decisions on backports. The committing docs could read something like: >> > > >> > > "When bug fixes are committed to master, the committer should evaluate >> > the >> > > fix to determine whether or not it should be backported to supported >> > > versions. This is left to the committer, but they are expected to >> weigh >> > the >> > > following concerns when making the decision: >> > > >> > > - Every backported change comes with a risk of unintended >> > > consequences. The change should be carefully evaluated to ensure >> that >> > such >> > > side-effects are highly unlikely. >> > > - As the complexity of applying a backport increases due to merge >> > > conflicts, the likelihood of unintended consequences also >> increases. >> > Bug >> > > fixes which require extensive rebasing should only be backported >> when >> > the >> > > bug is critical enough to warrant the risk. >> > > - Users of supported versions benefit greatly from the resolution >> of >> > > bugs in point releases. Thus, whenever concerns #1 and #2 can be >> > allayed >> > > for a given bug fix, it should be backported." >> > > >> > > >> > > Cheers, >> > > Greg >> > > >> > > >> > > On Mon, Jul 16, 2018 at 3:06 AM, Alex Rukletsov <a...@mesosphere.com> >> > > wrote: >> > > >> > >> Back porting as little as possible is the ultimate goal for me. My >> > >> reasons are closely aligned with what Andrew wrote above. >> > >> >> > >> If we agree on this strategy, the next question is how to enforce >> it. My >> > >> intuition is that committers will lean towards back porting their >> > patches >> > >> in arguable cases, because humans tend to overestimate the >> importance of >> > >> their personal work. Delegating the decision in such cases to a >> release >> > >> manager in my opinion will help us enforce the strategy of minimal >> > number >> > >> backports. As a bonus, the release manager will have a much better >> > >> understanding of what's going on with the release, keyword: "more >> > >> ownership". >> > >> >> > >> On Sat, Jul 14, 2018 at 12:07 AM, Andrew Schwartzmeyer < >> > >> and...@schwartzmeyer.com> wrote: >> > >> >> > >>> I believe I fall somewhere between Alex and Ben. >> > >>> >> > >>> As for deciding what to backport or not, I lean toward Alex's view >> of >> > >>> backporting as little as possible (and agree with his criteria). My >> > >>> reasoning is that all changes can have unforeseen consequences, >> which I >> > >>> believe is something to be actively avoided in already released >> > versions. >> > >>> The reason for backporting patches to fix regressions is the same as >> > the >> > >>> reason to avoid backporting as much as possible: keep behavior >> > consistent >> > >>> (and safe) within a release. With that as the goal of a branch in >> > >>> maintenance mode, it makes sense to fix regressions, and make >> > exceptions to >> > >>> fix CVEs and other critical/blocking issues. >> > >>> >> > >>> As for who should decide what to backport, I lean toward Ben's view >> of >> > >>> the burden being on the committer. I don't think we should add more >> > work >> > >>> for release managers, and I think the committer/shepherd obviously >> has >> > the >> > >>> most understanding of the context around changes proposed for >> backport. >> > >>> >> > >>> Here's an example of a recent bugfix which I backported: >> > >>> https://reviews.apache.org/r/67587/ (for MESOS-3790) >> > >>> >> > >>> While normally I believe this change falls under "avoid due to >> > >>> unforeseen consequences," I made an exception as the bug was old, >> circa >> > >>> 2015, (indicating it had been an issue for others), and was causing >> > >>> recurring failures in testing. The fix itself was very small, >> meaning >> > it >> > >>> was easier to evaluate for possible side effects, so I felt a little >> > safer >> > >>> in that regard. The effect of not having the fix was a fatal and >> > undesired >> > >>> crash, which furthermore left troublesome side effects on the system >> > (you >> > >>> couldn't bring the agent back up). And lastly, a dependent project >> > (DC/OS) >> > >>> wanted it in their next bump, which necessitated backporting to the >> > release >> > >>> they were pulling in. >> > >>> >> > >>> I think in general we should backport only as necessary, and leave >> it >> > on >> > >>> the committers to decide if backporting a particular change is >> > necessary. >> > >>> >> > >>> >> > >>> On 07/13/2018 12:54 am, Alex Rukletsov wrote: >> > >>> >> > >>>> This is exactly where our views differ, Ben : ) >> > >>>> >> > >>>> Ideally, I would like a release manager to have more ownership and >> > less >> > >>>> manual work. In my imagination, a release manager has more power >> and >> > >>>> control about dates, features, backports and everything that is >> > related >> > >>>> to >> > >>>> "their" branch. I would also like us to back port as little as >> > >>>> possible, to >> > >>>> simplify testing and releasing patch versions. >> > >>>> >> > >>>> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler < >> bmah...@apache.org> >> > >>>> wrote: >> > >>>> >> > >>>> +user, I probably it would be good to hear from users as well. >> > >>>>> >> > >>>>> Please see the original proposal as well as Alex's proposal and >> let >> > us >> > >>>>> know >> > >>>>> your thoughts. >> > >>>>> >> > >>>>> To continue the discussion from where Alex left off: >> > >>>>> >> > >>>>> > Other bugs and significant improvements, e.g., performance, may >> be >> > >>>>> back >> > >>>>> ported, >> > >>>>> the release manager should ideally be the one who decides on this. >> > >>>>> >> > >>>>> I'm a little puzzled by this, why is the release manager >> involved? As >> > >>>>> we >> > >>>>> already document, backports occur when the bug is fixed, so this >> > >>>>> happens in >> > >>>>> the steady state of development, not at release time. The release >> > >>>>> manager >> > >>>>> only comes in at the time of the release itself, at which point >> all >> > >>>>> backports have already happened and the release manager handles >> the >> > >>>>> release >> > >>>>> process. Only blocker level issues can stop the release and while >> the >> > >>>>> release manager has a strong say, we should generally agree on >> what >> > >>>>> consists of a release blocking issue. >> > >>>>> >> > >>>>> Just to clarify my workflow, I generally backport every bug fix I >> > >>>>> commit >> > >>>>> that applies cleanly, right after I commit it to master (with the >> > >>>>> exceptions I listed below). >> > >>>>> >> > >>>>> On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov < >> a...@mesosphere.com >> > > >> > >>>>> wrote: >> > >>>>> >> > >>>>> > I would like to back port as little as possible. I suggest the >> > >>>>> following >> > >>>>> > criteria: >> > >>>>> > >> > >>>>> > * By default, regressions are back ported to existing release >> > >>>>> branches. A >> > >>>>> > bug is considered a regression if the functionality is present >> in >> > the >> > >>>>> > previous minor or patch version and is not affected by the bug >> > there. >> > >>>>> > >> > >>>>> > * Critical and blocker issues, e.g., a CVE, can be back ported. >> > >>>>> > >> > >>>>> > * Other bugs and significant improvements, e.g., performance, >> may >> > be >> > >>>>> back >> > >>>>> > ported, the release manager should ideally be the one who >> decides >> > on >> > >>>>> this. >> > >>>>> > >> > >>>>> > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone < >> vinodk...@apache.org >> > > >> > >>>>> wrote: >> > >>>>> > >> > >>>>> > > Ben, thanks for the clarification. I'm in agreement with the >> > >>>>> points you >> > >>>>> > > made. >> > >>>>> > > >> > >>>>> > > Once we have consensus, would you mind updating the doc? >> > >>>>> > > >> > >>>>> > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler < >> > >>>>> bmah...@apache.org> >> > >>>>> > > wrote: >> > >>>>> > > >> > >>>>> > > > I realized recently that we aren't all on the same page with >> > >>>>> > backporting. >> > >>>>> > > > We currently only document the following: >> > >>>>> > > > >> > >>>>> > > > "Typically the fix for an issue that is affecting supported >> > >>>>> releases >> > >>>>> > > lands >> > >>>>> > > > on the master branch and is then backported to the release >> > >>>>> branch(es). >> > >>>>> > In >> > >>>>> > > > rare cases, the fix might directly go into a release branch >> > >>>>> without >> > >>>>> > > landing >> > >>>>> > > > on master (e.g., fix / issue is not applicable to master)." >> [1] >> > >>>>> > > > >> > >>>>> > > > This leaves room for interpretation about what lies outside >> of >> > >>>>> > "typical". >> > >>>>> > > > Here's the simplest way I can explain what I stick to, and >> I'd >> > >>>>> like >> > >>>>> to >> > >>>>> > > hear >> > >>>>> > > > what others have in mind: >> > >>>>> > > > >> > >>>>> > > > * By default, bug fixes at any level should be backported to >> > >>>>> existing >> > >>>>> > > > release branches if it affects those releases. Especially >> > >>>>> important: >> > >>>>> > > > crashes, bugs in non-experimental features. >> > >>>>> > > > >> > >>>>> > > > * Exceptional cases that can omit backporting: difficult to >> > >>>>> backport >> > >>>>> > > fixes >> > >>>>> > > > (especially if the bugs are deemed of low priority), bugs in >> > >>>>> > experimental >> > >>>>> > > > features. >> > >>>>> > > > >> > >>>>> > > > * Exceptional non-bug cases that can be backported: >> performance >> > >>>>> > > > improvements. >> > >>>>> > > > >> > >>>>> > > > I realize that there is a ton of subtlety here (even in >> terms >> > of >> > >>>>> which >> > >>>>> > > > things are defined as bugs). But I hope we can lay down a >> > policy >> > >>>>> that >> > >>>>> > > gives >> > >>>>> > > > everyone the right mindset for common cases and then discuss >> > >>>>> corner >> > >>>>> > cases >> > >>>>> > > > on-demand in the future. >> > >>>>> > > > >> > >>>>> > > > [1] http://mesos.apache.org/documentation/latest/ >> versioning/ >> > >>>>> > > > >> > >>>>> > > >> > >>>>> > >> > >>>>> >> > >>>>> >> > >>> >> > >> >> > > >> > >> > >
