On 5/2/23 15:00, Ilya Maximets wrote: > On 5/2/23 06:50, Han Zhou wrote: >> >> >> On Fri, Apr 28, 2023 at 11:38 AM Mark Michelson <[email protected] >> <mailto:[email protected]>> wrote: >>> >>> Sorry for the top-post, but trying to write my responses inline at this >>> point is basically impossible for me if I want to actually make any >>> salient points :) Instead, I'll address the various topics one-by-one >>> here at the top. >>> >>> First, I'll weigh in on some of the easier topics: >>> >>> * More frequent minor releases >>> >>> I 100% agree that we should have more frequent minor releases, >>> especially on the LTS branches. I made a proposal on this list in >>> November 2021 here >>> https://mail.openvswitch.org/pipermail/ovs-dev/2021-November/389581.html >>> <https://mail.openvswitch.org/pipermail/ovs-dev/2021-November/389581.html> >>> that would have made monthly point releases happen. The email got no >>> response, so I assumed it was unwanted and moved on. At this point, >>> maybe we don't necessarily need to put out new minor releases that >>> frequently, but having it be more regular would make it more worthwhile >>> to backport fixes to supported branches. >>> >>> * Less frequent major releases >>> >>> I agree with this too. When OVN split from OVS, we wanted a more >>> frequent release cadence since we were adding lots of new features and >>> optimizations into OVN, and CMSes were eager to consume these changes. >>> Three years later and the number of massive features and optimizations >>> in OVN has slowed down. The larger changes that we would like to make >>> could benefit from a longer lead time before release. It's been >>> suggested to move down to three major releases a year, but I'd even be >>> happy with two. >> >> +1 for two major releases a year. > > I'd suggest keeping .03 and .09 releases then. This way OVN will be > able to use latest stable OVS submodule, if necessary. >
From the OVS release-process.rst, the release dates are: +---------------+----------------+-------------------------------------+ | T + 5.5 | Aug 15, Feb 15 | Release version x.y.0 | +---------------+----------------+-------------------------------------+ From the OVN release-process.rst, the release dates for .03 and .09 are: +---------+-------------+-----------------+---------+ | Release | Soft Freeze | Branch Creation | Release | +---------+-------------+-----------------+---------+ | 23.03.0 | Feb 3 | Feb 17 | Mar 3 | +---------+-------------+-----------------+---------+ ... +---------+-------------+-----------------+---------+ | 23.09.0 | Aug 4 | Aug 18 | Sep 1 | +---------+-------------+-----------------+---------+ Assuming no OVS release delays it means that we need to bump the OVS submodule 2 weeks before the OVN release. In theory, that means less time to test potential changes due to OVS libraries. In practice, however, it's probably not that bad. I'm voting +1 for two major releases a year and I agree that those should be .03 and .09. >> >>> >>> * Shorter LTS support window >>> >>> I, personally, would largely be in favor of this since supporting old >>> versions is hard and can be risky. However, I think CMSes would not be >>> thrilled about being forced to upgrade more frequently. In fact, I'm >>> pretty sure Red Hat's OpenStack team was already not happy that they >>> *only* get two years of full support on an LTS. If the concern about the >>> length of LTS support is about the number of backports required, then I >>> think having more infrequent major releases would solve the problem for >>> us instead, and we can keep our current support windows. >>> >>> * Fewer unnecessary refactors >>> >>> I'm not sure this happens much, tbh. Typically, when a refactor happens, >>> it's also introducing new functionality or an optimization. The refactor >>> is there in order to more easily facilitate the accompanying change. >>> While there may be some refactors that exist solely for code "niceness" >>> (and I may have even contributed some), I can't think of any. I'm >>> hesitant to forbid refactors that don't add anything new, since they may >>> help maintainability of the code in the future. I think the best way to >>> approach this is a case-by-case basis. >>> >>> >>> OK, now for the harder topic. First, I'll make a hypothetical release >>> timeline. Pretend for a moment that It's December 2024. With the current >>> release schedule, the following releases would exist: >>> >>> 24.12 >>> 24.09 >>> 24.06 >>> 24.03 (LTS) >>> 23.12 >>> 23.09 >>> 23.06 >>> 23.03 >>> 22.12 >>> 22.09 >>> 22.06 >>> 22.03 (LTS) >>> >>> Based on our current documentation, the following releases would be >>> under support (based on the "backporting" documentation cited by Ilya): >>> >>> 24.12 >>> 24.09 >>> 24.06 >>> 24.03 (LTS) >>> >>> The following releases would have "critical" bug fix support >>> >>> 22.03 (LTS) >>> >>> And the following releases would be considered unsupported >>> >>> 23.12 >>> 23.09 >>> 23.06 >>> 23.03 >>> 22.12 >>> 22.09 >>> 22.06 >>> >>> So now let's consider some hypotheticals. >>> >>> Hypothetical 1: A bug is reported against OVN. Its root cause is >>> discovered, and it turns out it's been in every release of OVN ever. >>> Where should the fix be backported? >>> >>> Answer 1: Based on current policy, the bug fix should be backported to >>> all branches currently under support (24.03, 24.06, 24.09, and 24.12). >>> Even though it may be possible to backport the patches further, there's >>> not much point to it since we will never make any more minor releases on >>> the unsupported branches. However, I don't think anyone would NACK a >>> backport to an unsupported branch if it applies cleanly and does not >>> risk regression. >>> >> >> Agree. Just want to emphasize that the major point of the discussion for >> this case is when we are at the point of 25.12 (or 23.12), we will need to >> backport to 8 branches. Changing to two major releases per year would reduce >> the load. I think if we agree with this, we need to change as soon as we >> can. Even if we change it now, i.e. let the next release be 23.09, we will >> still need to maintain 6 branches for a period of time and decrease to 4 >> gradually. > > I'd suggest keeping the current 23.06 and 23.09. Keep only .03 and .09 > onward. > >> >>> Hypothetical 2: A critical security issues is reported against OVN. Its >>> root cause is discovered, and it turns out it's been in every release of >>> OVN ever. Where should the fix be backported? >>> >>> Answer 2: By the letter of the documentation, the fix would be applied >>> to all versions in Hypothetical 1, as well as 22.03. The discussion in >>> this thread has questioned whether the unsupported branches between the >>> LTSes should also have the fix applied, since this is not explicitly >>> covered by the documentation. My intention when writing the LTS policy >>> was that the answer was NO. The reason for this is that if a newer LTS >>> exists, then the preferred upgrade path for someone running an old LTS >>> would be to upgrade to the next LTS directly. And those that run any of >>> the releases between LTSes are also encouraged to upgrade to the newest >>> LTS. While that was my intention, it sounds like other devs in this >>> thread disagree and think that the fix should also be applied to the >>> unsupported releases between the LTSes. >>> >> Other than the consideration for upgrades, Ilya also mentioned that skipping >> branches "makes it harder to bisect in which branch the issue was >> introduced". I think it makes sense, but I don't remember facing this issue >> for OVN, so I'm not sure how important this factor will be. > > > We need to define the word "supported". "Supported" from the user > perspective means forming of new minor releases from this branch. > We can still apply changes to the branch while backporting "all the > way down", but users may not see them, because the new minor release > was provided only for the old LTS and not for in-between branches. > I think this part was confusing for me. I assumed that if a branch was newer than LTS then it was implicitly also "supported". I guess that, according to your definition, I was wrong. > >> >>> >>> One last point I want to bring up is that there is nothing in our >>> documentation that says how long a "standard" release is supported. >>> Based on the backporting document that Ilya cited, it makes it sound as >>> though standard releases are supported up to the point of releasing a >>> new LTS. I think this should be rethought. Consider in the above example >>> the 23.12 release. It exists for three months, then the 24.03 LTS >>> release is made. Does that mean that a three month old release >>> immediately becomes unsupported because there's now a new LTS? >>> Meanwhile, the 22.06 release will have received 18 months of bug fixes. >>> To me, that makes little sense. I think we should codify standard >>> releases as follows: >>> >>> "Standard releases will receive full support for one year from the time >>> of their release. This means that bug fixes and security fixes will be >>> applied to this version, and minor releases will be made periodically >>> for one year after the release is made. Once the support window ends, >>> bugfixes and security fixes *may* be applied to the branches from which >>> standard releases are created, but no further minor releases will be >>> made. The reason that fixes may be applied to the unsupported branches >>> is to help facilitate backports to older supported LTS releases." >>> >> >> Thanks for bringing up this point. For a branch like 22.06, it would >> actually be supported for 21 months (much longer than 1 year), because we >> don't skip branches when backporting down to the last LTS release. So even >> if we have this one-year policy, the non-LTS releases still are not created >> equal. I think this may need to be clarified clearly, so that users >> understand that releases closer to LTS have shorter support time, while the >> minimum is one year. If I understand correctly, OVS is doing this >> differently and making each release supported for two years. > > The difference between OVS and OVN is that OVS supports old LTS for one > release cycle after the new LTS is designated. New LTS is designated > every 2 years at the moment of of release of the next version, i.e. > 2.17 became LTS at the day of release of 3.0. The same day 2.13 became > old LTS. And 2.13 went EOL at the day 3.1 was released. This gives > about 1 year window for users to migrate from the previous LTS to > yet-to-be a new LTS (for the first release cycle) or to the already > new LTS (the second release cycle). 1 year of transitional period lines > up with 2 releases per year strategy. > > OTOH, OVN says "1 year of critical fixes" and that spans 4 OVN releases. > So, OVN can't drop old LTS after 1 cycle or even 2. That generates > strange branches between new LTS and old LTS that are fairly old and are > not formally maintained. And these branches seem to generate most of the > confusion. > Moving to 2 releases per year should solve that problem if OVN adopts the > same release strategy as OVS. > I'm not sure I was clear enough earlier in the discussion but I was actually trying to advocate for less releases too by pointing to the number of new features that were accepted since 22.03. There were 5 NEWS items per release and I don't think that's a very large number. Reducing the number of releases to 2 per year would make it that we introduce ~10 new features per release. I think that's acceptable. > OVS also promises to provide minor releases for the last 4 released branches > once in a while. So, 2 years of minor releases for everyone. However, > since these branches "are not formally maintained", releases are happening > way less frequently than minor releases on LTS and the latest stable branch. > > For OVN with the current release cadence the equivalent policy would mean > providing minor releases once in a while for the last 8 releases. And > provide frequent minor releases for LTSes and the latest released branch. > > Branches that are newer than the oldest LTS, but older than 2 years, IMO, > should still receive bug fix backports for the sake of repository consistency, > but minor releases should not be formed on these branches. > This is causing a lot of work today, but should not be that burdensome > with reduced release cadence. > > E.g. for OVS at any given time I need to backport fixes at most to 6 > branches (there is a couple of weeks between the major release and the final > minor release for old LTS, where the number is 7, but it can be discarded). > Throughout 2 year cycle the number is 4, 4, 5, 6. Then it repeats. There is > only one completely unsupported branch at the point of 6, for which minor > releases are not supposed to be provided. But it's not hard to backport > there, > since I'm already doing backports to 5 other branches, even if the release is > not going to be formed. At most 3 branches need frequent minor releases. > Throughout 2 year cycle the number is 2, 2, 2, 3. > > The table at the last slide has some visual representation: > > https://www.openvswitch.org/support/ovscon2020/slides/Community-updates-and-new-LTS-process.pdf > >> >> Thanks, >> Han >> >>> Note that this now conflicts with the "backporting" documentation since >>> it says we will backport to all branches back to the most recent LTS. I >>> think we should still make a best effort to backport fixes to all >>> branches since the most recent LTS. However, I don't think it should be >>> required if the effort or risk of the backport is too much. So I think >>> the backporting documentation should be updated to reflect that if we >>> agree on the proposed lifetime of standard releases. > > Usually backporting to in-between branches is not that hard. There might > be exceptions, if the design changed multiple times. I'd still prefer > backporting to all branches though. Me too. As you pointed out earlier OVN gets refactor patches more often than OVS so backporting fixes is not always as easy as with OVS. Aside from refactoring less (which I still hope is achievable in the future) reducing the number of branches makes it easier for maintainers to backport everywhere. Regards, Dumitru > > My 2c. > > Best regards, Ilya Maximets. > >>> >>> >>> On 4/18/23 13:35, Han Zhou wrote: >>>> > > On Tue, Apr 18, 2023 at 10:33 AM Han Zhou <[email protected] >>>> <mailto:[email protected]> >>>> <mailto:[email protected] <mailto:[email protected]>>> wrote: >>>> > >>>> > >>>> > >>>> > On Tue, Apr 18, 2023 at 1:50 AM Dumitru Ceara <[email protected] >>>> <mailto:[email protected]> >>>> <mailto:[email protected] <mailto:[email protected]>>> wrote: >>>> > > >>>> > > On 4/17/23 21:46, Han Zhou wrote: >>>> > > > On Mon, Apr 17, 2023 at 12:01 PM Ilya Maximets >>>> <[email protected] <mailto:[email protected]> <mailto:[email protected] >>>> <mailto:[email protected]>>> wrote: >>>> > > >> >>>> > > >> On 4/17/23 19:20, Han Zhou wrote: >>>> > > >>> >>>> > > >>> >>>> > > >>> On Mon, Apr 17, 2023 at 9:24 AM Numan Siddique <[email protected] >>>> <mailto:[email protected]> >>>> <mailto:[email protected] <mailto:[email protected]>> <mailto: >>>> > > > [email protected] <mailto:[email protected]> <mailto:[email protected] >>>> <mailto:[email protected]>>>> wrote: >>>> > > >>>> >>>> > > >>>> On Mon, Apr 17, 2023 at 5:57 AM Dumitru Ceara >>>> <[email protected] <mailto:[email protected]> <mailto:[email protected] >>>> <mailto:[email protected]>> >>>> > > > <mailto:[email protected] <mailto:[email protected]> >>>> <mailto:[email protected] <mailto:[email protected]>>>> wrote: >>>> > > >>>>> >>>> > > >>>>> On 4/17/23 11:55, Dumitru Ceara wrote: >>>> > > >>>>>> On 4/14/23 18:01, Han Zhou wrote: >>>> > > >>>>>>> >>>> > > >>>>>>> >>>> > > >>>>>>> On Fri, Apr 14, 2023 at 2:16 AM Dumitru Ceara >>>> <[email protected] <mailto:[email protected]> <mailto:[email protected] >>>> <mailto:[email protected]>> >>>> > > > <mailto:[email protected] <mailto:[email protected]> >>>> <mailto:[email protected] <mailto:[email protected]>>> >>>> > > >>>>>>> <mailto:[email protected] <mailto:[email protected]> >>>> <mailto:[email protected] <mailto:[email protected]>> >>>> <mailto:[email protected] <mailto:[email protected]> >>>> <mailto:[email protected] <mailto:[email protected]>>>>> wrote: >>>> > > >>>>>>>> >>>> > > >>>>>>>> On 4/14/23 04:15, Han Zhou wrote: >>>> > > >>>>>>>>> On Thu, Apr 13, 2023 at 12:44 PM Han Zhou <[email protected] >>>> <mailto:[email protected]> >>>> <mailto:[email protected] <mailto:[email protected]>> >>>> > > > <mailto:[email protected] <mailto:[email protected]> <mailto:[email protected] >>>> <mailto:[email protected]>>> >>>> > > >>>>>>> <mailto:[email protected] <mailto:[email protected]> >>>> <mailto:[email protected] <mailto:[email protected]>> >>>> <mailto:[email protected] <mailto:[email protected]> <mailto:[email protected] >>>> <mailto:[email protected]>>>>> wrote: >>>> > > >>>>>>>>>> >>>> > > >>>>>>>>> I backported this series to the last release branch-23.03 >>>> and >>>> > > > the LTS >>>> > > >>>>>>>>> branch-22.03. >>>> > > >>>>>>>>> The code base has changed a lot so was mostly manually >>>> > > > backported >>>> > > >>>>>>> instead >>>> > > >>>>>>>>> of cherry-pick. >>>> > > >>>>>>>>> >>>> > > >>>>>>>> >>>> > > >>>>>>>> Thanks for that but I think we shouldn't skip stable >>>> branches in >>>> > > >>>>>>>> between. We likely still have users on those that won't >>>> upgrade >>>> > > > to the >>>> > > >>>>>>>> latest 23.03 but instead to a new z-stream release (when that >>>> > > > happens) >>>> > > >>>>>>>> of the stable branch. >>>> > > >>>>>>>> >>>> > > >>>>>>>> I cherry picked your backports and made sure the tests >>>> pass and >>>> > > > then >>>> > > >>>>>>>> pushed them to branches 22.12, 22.09 and 22.06. >>>> > > >>>>>>> >>>> > > >>>>>>> Thanks Dumitru for helping backporting. I was uncertain about >>>> > > > whether we >>>> > > >>>>>>> should continue backporting without skipping branches, as our >>>> > > > growing >>>> > > >>>>>>> number of released branches makes this approach increasingly >>>> > > > difficult >>>> > > >>>>>>> to sustain. >>>> > > >>>>>>> >>>> > > >>>>>>> The release process document >>>> > > >>>>>>> >>>> (https://docs.ovn.org/en/latest/internals/release-process.html >>>> <https://docs.ovn.org/en/latest/internals/release-process.html> >>>> <https://docs.ovn.org/en/latest/internals/release-process.html >>>> <https://docs.ovn.org/en/latest/internals/release-process.html>> < >>>> > > > https://docs.ovn.org/en/latest/internals/release-process.html >>>> <https://docs.ovn.org/en/latest/internals/release-process.html> >>>> <https://docs.ovn.org/en/latest/internals/release-process.html >>>> <https://docs.ovn.org/en/latest/internals/release-process.html>>> >>>> > > >>>>>>> >>>> <https://docs.ovn.org/en/latest/internals/release-process.html >>>> <https://docs.ovn.org/en/latest/internals/release-process.html> >>>> <https://docs.ovn.org/en/latest/internals/release-process.html >>>> <https://docs.ovn.org/en/latest/internals/release-process.html>> < >>>> > > > https://docs.ovn.org/en/latest/internals/release-process.html >>>> <https://docs.ovn.org/en/latest/internals/release-process.html> >>>> <https://docs.ovn.org/en/latest/internals/release-process.html >>>> <https://docs.ovn.org/en/latest/internals/release-process.html>>>>) doesn't >>>> > > >>>>>>> specify how long we should maintain non-LTS branches. My >>>> > > > understanding >>>> > > >>>>>>> was that we should primarily maintain LTS and the latest >>>> releases >>>> > > > for >>>> > > >>>>>>> regular bug fixes, while critical/security fixes would be >>>> applied >>>> > > > to all >>>> > > >>>>>>> branches. It appears we have different interpretations of this >>>> > > > process. >>>> > > >>>>>>> While it's beneficial to backport to all branches, I >>>> thought this >>>> > > > was >>>> > > >>>>>>> more of a convenience when there weren't too many branches to >>>> > > > manage. In >>>> > > >>>>>>> the worst case, a patch may require manual backporting to each >>>> > > > branch if >>>> > > >>>>>>> new features make the codebase in each branch unique. I also >>>> > > > question >>>> > > >>>>>>> what differentiates an LTS branch if we're backporting to all >>>> > > > branches >>>> > > >>>>>>> anyway. My assumption was that users opting for a non-LTS >>>> branch >>>> > > > should >>>> > > >>>>>>> be prepared to upgrade to newer releases, given the >>>> "short-term" >>>> > > >>>>>>> maintenance of these branches. Is my understanding incorrect? >>>> > > >>>>>>> >>>> > > >>>>>> Hi Han, >>>> > > >>>>>> >>>> > > >>>>>> I changed the subject of the email to better reflect the new >>>> > > > discussion. >>>> > > >>>>>> >>>> > > >>>>>> There's this statement in the release-process document: "LTS >>>> > > > releases >>>> > > >>>>>> will receive bug fixes until the point that another LTS is >>>> > > > released. At >>>> > > >>>>>> that point, the old LTS will receive an additional year of >>>> > > > critical and >>>> > > >>>>>> security fixes." >>>> > > >>>>>> >>>> > > >>>>>> Which further differentiates an LTS release from a non-LTS >>>> release. >>>> > > >>> >>>> > > >>> If you are saying "the old LTS will receive an additional year of >>>> > > > critical and security fixes" is the different part, I agree. But >>>> still, in >>>> > > > this case, for critical and security fixes, when applied to the >>>> LTS for the >>>> > > > *additional* year, do you agree that we will skip non-LTS >>>> releases? If not >>>> > > > skipping, then it makes no difference. >>>> > > >>> >>>> > > >>>>>> >>>> > > >>>>>>> Perhaps we should discuss and formalize our backporting >>>> practice >>>> > > > going >>>> > > >>>>>>> forward, as the number of branches will only continue to >>>> grow in >>>> > > > the >>>> > > >>>>>>> coming months and years. cc @Mark Michelson >>>> > > >>>>>>> <mailto:[email protected] <mailto:[email protected]> >>>> <mailto:[email protected] <mailto:[email protected]>> >>>> <mailto:[email protected] <mailto:[email protected]> >>>> <mailto:[email protected] <mailto:[email protected]>>>> @Numan >>>> > > > Siddique <mailto:[email protected] <mailto:[email protected]> >>>> <mailto:[email protected] <mailto:[email protected]>> >>>> <mailto:[email protected] <mailto:[email protected]> <mailto:[email protected] >>>> <mailto:[email protected]>>>> >>>> > > >>>>>>> >>>> > > >>>>>> >>>> > > >>>>>> Checking the OVS/OVN release-process documents again it's >>>> indeed >>>> > > > not >>>> > > >>>>>> explicitly specified that non-LTS stable release branches are >>>> > > > supposed >>>> > > >>>>>> to get all bug fixes. But the OVS document has this addition: >>>> > > >>>>>> >>>> > > >>>>>> "While branches other than LTS and the latest release are not >>>> > > > formally >>>> > > >>>>>> maintained, the OVS project usually provides stable releases >>>> for >>>> > > > these >>>> > > >>>>>> branches for at least 2 years, i.e. stable releases are >>>> provided >>>> > > > for the >>>> > > >>>>>> last 4 release branches. However, these branches may not >>>> include >>>> > > > all >>>> > > >>>>>> the fixes that LTS has in case backporting is not >>>> straightforward >>>> > > > and >>>> > > >>>>>> developers are not willing to spend their time on that (this >>>> mostly >>>> > > >>>>>> affects branches that are older than the LTS, because >>>> backporting >>>> > > > to LTS >>>> > > >>>>>> implies backporting to all intermediate branches)." >>>> > > >>>>>> >>>> > > >>>>>> This last part really makes me think that we SHOULD backport >>>> to all >>>> > > >>>>>> intermediate branches: "... because backporting to LTS >>>> > > >>>>>> implies backporting to all intermediate branches". >>>> > > >>>>>> >>>> > > >>>> >>>> > > >>>> I agree with this. >>>> > > >>>> >>>> > > >>> If we consistently backport to all intermediate releases, the LTS >>>> > > > release appears to function more as a reference point in the >>>> series of >>>> > > > releases, rather than a version with additional maintenance >>>> support. This >>>> > > > is ok if we all agree to it. >>>> > > >>> >>>> > > >>>>>> Maybe we should make this rule (or some other backport >>>> strategy we >>>> > > >>>>>> decide to use from this point on) explicit indeed. >>>> > > >> >>>> > > >> First of all, the backporting process is already documented in OVN: >>>> > > >> >>>> > > > >>>> https://docs.ovn.org/en/latest/internals/contributing/backporting-patches.html >>>> >>>> <https://docs.ovn.org/en/latest/internals/contributing/backporting-patches.html> >>>> >>>> <https://docs.ovn.org/en/latest/internals/contributing/backporting-patches.html >>>> >>>> <https://docs.ovn.org/en/latest/internals/contributing/backporting-patches.html>> >>>> > > >> >>>> > > >> "The maintainer first applies the patch to master, then >>>> backports the >>>> > > >> patch to each older affected tree, as far back as it goes or at >>>> least >>>> > > >> to all currently supported branches. This is usually each >>>> branch back >>>> > > >> to the most recent LTS release branch." >>>> > > > >>>> > > > Thanks for the pointer, and sorry for missing it earlier. Probably >>>> we >>>> > > > should combine these two documents for OVN because both mentioned >>>> some part >>>> > > > of backport processes, and remove the irrelevant part (like >>>> > > > userspace/kernel related discussion for OVS). >>>> > > > >>>> > > >>>> > > I think this still needs clarification: "... then backports the >>>> patch to >>>> > > each older affected tree, as far back as it goes or at least to all >>>> > > currently supported branches". We still don't mention that all >>>> branches >>>> > > between main/master and the LTS are currently supported. I always >>>> > > assumed they are but it's obviously not clear since we're having this >>>> > > conversation. >>>> > > >>>> > > >> >>>> > > >> No-one updated the documentation in general since migration from >>>> master >>>> > > >> to main, but that's not the point. >>>> > > >> >>>> > > >> Backports are done for each intermediate branch to avoid >>>> regressions >>>> > > >> on upgrades, which may be very bad for users and not a great >>>> thing to >>>> > > >> have in general (also makes it harder to bisect in which branch the >>>> > > >> issue was introduced). >>>> > > >>>> > > +1 >>>> > > >>>> > > That's exactly the reason why I added the missing backports of the >>>> > > original bug fix. >>>> > > >>>> > > >> >>>> > > >> This should remain this way, I think, unless OVN decides that >>>> intermediate >>>> > > >> branches are not supported at all and never get any backports. >>>> i.e. the >>>> > > >> release schema similar to DPDK, where all xx.11 releases are >>>> supported for >>>> > > >> a few years, but intermediate releases are fully abandoned the >>>> moment the >>>> > > >> next one is out. >>>> > > >> >>>> > > >> >>>> > > >> "the LTS release appears to function more as a reference point >>>> in the >>>> > > >> series of releases, rather than a version with additional >>>> maintenance >>>> > > >> support" >>>> > > >> >>>> > > >> The reason why it appears this way is a bit orthogonal. The point >>>> > > >> of extra maintenance support is preparing of official minor >>>> releases. >>>> > > >> Minor releases on LTS and the latest stable should be much more >>>> > > >> frequent than on other branches that are not fully supported. >>>> > > >> For example, I'm trying to release a new minor version of OVS >>>> LTS and >>>> > > >> the latest stable every ~2 months, while other "unsupported" >>>> branches >>>> > > >> may receive minor releases once in 4-6 months. >>>> > > >> >>>> > > >> The problem with OVN project that makes us ask these questions >>>> is that >>>> > > >> OVN doesn't really prepare any minor releases any regularly at all, >>>> > > >> while it should. OVN 22.03 LTS received 2 stables releases >>>> since release >>>> > > >> in March 2022 after people asking for them. In comparison, OVS >>>> 2.17 LTS >>>> > > >> received 6 new minor versions within approximately same time frame. >>>> > > >> >>>> > > >>>> > > It does seem like we should have more predictable minor release >>>> schedules. >>>> > > >>>> > > >> So, the original point of a long term support was to keep >>>> backporting >>>> > > >> fixes all the way down and prepare minor releases more frequently >>>> on >>>> > > >> this one particular branch. And backporting all the way down, >>>> according >>>> > > >> to the current documentation, means backporting to all intermediate >>>> > > >> branches. >>>> > > >> >>>> > > > Thanks for the explanation. Now I see the point of a LTS release. >>>> Releasing >>>> > > > minor versions more frequently does make it more useful. >>>> > > > So according to the current documented process we should not skip >>>> branches >>>> > > > even for the "additional" year of a LTS, although the DPDK >>>> approach seems >>>> > > > more attractive to me considering the amount of release branches >>>> between >>>> > > > two LTS in OVN. >>>> > > > >>>> > > >>>> > > If we're careful in backporting only "very important fixes" (as Ilya >>>> > > called them below) to the older LTS then even if we need to backport >>>> to >>>> > > all intermediate branches the maintainer effort should still be >>>> reasonable. >>>> > > >>>> > This may sound reasonable but I would not count on it. What we can do >>>> is try our best to ensure quality and avoid bugs as much as we can, but >>>> beyond that there is not much we can control. >>>> > When bugs need to be fixed, the importance of the bug fix mostly >>>> depends on the bug itself (how users are impacted), not how we evaluate >>>> it. (We may tend to backport less bug fixes when there are more branches >>>> to maintain, but if we do that it would be wrong.) >>>> > >>>> > > >>>> >>>> > > >>>> +1 on for this. >>>> > > >>>> >>>> > > >>>> If we have these branches >>>> > > >>>> >>>> > > >>>> - Latest release (R9) >>>> > > >>>> - R8 >>>> > > >>>> - R7 >>>> > > >>>> - R6 (LTS) >>>> > > >>>> - R5 >>>> > > >>>> - R4 >>>> > > >>>> - R3 >>>> > > >>>> - R2 (LTS) >>>> > > >>>> - R1 >>>> > > >>>> >>>> > > >>>> IMO it should be enough to backport bug fixes up to the latest >>>> LTS >>>> > > >>>> (which is R6) in the above example. >>>> > > >>> >>>> > > >>> This example looks not bad, but it is not how OVN would be >>>> released, >>>> > > > because for OVN "LTS releases are scheduled to be released once >>>> every two >>>> > > > years". So it will look like: >>>> > > >>> - Latest release (R17) >>>> > > >>> ... >>>> > > >>> - R12 >>>> > > >>> - R11 >>>> > > >>> - R10 (LTS) >>>> > > >>> ... >>>> > > >>> - R4 >>>> > > >>> - R3 >>>> > > >>> - R2 (LTS) >>>> > > >>> - R1 >>>> > > >>> >>>> > > >>> We will always have to backport from R17 down to R10 in this >>>> example (8 >>>> > > > branches). >>>> > > >>> >>>> > > >>>> In case we decide to backport to R2 as well, then I think we >>>> should >>>> > > >>>> not skip intermediate branches (R5, R4 and R3) >>>> > > >>>> and instead backport upto R2. >>>> > > >>> >>>> > > >>> I don't think we will ever want to backport to R2 in this case. >>>> If we >>>> > > > do, it would mean from R17 - R2, which includes 16 branches. >>>> > > >>> >>>> > > >>> But look at another example: >>>> > > >>> - Latest release (R13) >>>> > > >>> - R12 >>>> > > >>> - R11 >>>> > > >>> - R10 (LTS) >>>> > > >>> ... >>>> > > >>> - R4 >>>> > > >>> - R3 >>>> > > >>> - R2 (LTS) >>>> > > >>> - R1 >>>> > > >>> >>>> > > >>> Because the new LTS (R10) in this example is released less than >>>> a year, >>>> > > > according to the rule, we will support the older LTS (R2) for >>>> critical / >>>> > > > security bug fixes. >>>> > > >> >>>> > > >> >>>> > > >> Right, that's why only very important fixes should go to R2. >>>> > > >> i.e. these should actually be fixes for very severe issues that >>>> > > >> majority of users is experiencing. Fixes for some minor leaks >>>> > > >> or broken communication in corner cases should not be backported >>>> > > >> to R2. >>>> > > >> >>>> > > >> In a worst case scenario, it will be ~12 branches to backport to, >>>> > > >> because an LTS release will be alive for at most 3 years. >>>> > > >> 12 is a lot, but again, backporting that far should only happen >>>> > > >> in exceptional cases. >>>> > > >> >>>> > > >> In case of a single LTS branch, there should be maximum of 8 >>>> > > >> branches to backport to. That is a lot too, but that's what >>>> > > >> the current process suggests. >>>> > > >> >>>> > > >> The issue of backporting too far can be mitigated by being more >>>> > > >> conscious about what kind of changes are getting backported, >>>> > > >> how severe the issue they are trying to fix in comparison to the >>>> > > >> amount of work backporting and validation of stable branches >>>> > > >> requires. That's partially what frequent major releases are for. >>>> > > >> If users want to stick to stable releases instead and still want >>>> > > >> every minor issue to be fixed, maybe this is time to reduce the >>>> > > >> frequency of major releases to e.g. 3 times a year instead of 4. >>>> > > >> Or reduce the support time of LTS to e.g. 1.5 years of full support >>>> > > >> plus 0.5 years of security / critical fixes and designating every >>>> > > >> xx.03 release as an LTS. >>>> > > >> >>>> > > > I like this last proposal. Shall we discuss this again in OVN >>>> meeting? I >>>> > > > think the overhead of maintenance was not fully understood when >>>> we agreed >>>> > > > on this LTS cycle initially (I could be only me). >>>> > > > >>>> > > >>>> > > +1 >>>> > > >>>> > > Counting how many new features/news items we got per stable (.0) >>>> release >>>> > > since 22.03 (LTS) I get an average of 5 items per release. Moving to >>>> 3 >>>> > > releases per year instead of 4 shouldn't make a huge difference in the >>>> > > maintenance effort. >>>> > > >>>> > Agree that moving to 3 releases per year doesn't help much. What I >>>> liked is this part: >>>> > >>>> > ... reduce the support time of LTS to e.g. 1.5 years of full support >>>> > plus 0.5 years of security / critical fixes and designating every >>>> > xx.03 release as an LTS. >>>> > >>>> > > >> Also, OVN sometimes may be more conscious about constantly >>>> > > >> re-factoring everything. :D Sometimes it's necessary, true, but >>>> > > >> sometimes the same functionality can be achieved without reworking >>>> > > >> a ton of code just for looks of it. This may greatly reduce the >>>> > > >> need in manual backports. >>>> > > > >>>> > > > Strongly agree (but I also insist that my recent refactor for I-P >>>> should be >>>> > > > exempted :D) >>>> > > > >>>> > > >>>> > > We went through a period in which we realized some things need to be >>>> > > redesigned in order for the whole OVN model to scale (e.g., datapath >>>> > > groups, LB groups, generating LB logical flows, northd I-P). That >>>> > > would've been impossible without a certain amount of refactoring. >>>> What >>>> > > I hope is that the rate of refactoring goes down in the near future. >>>> > > >>>> Forgot to comment on this. I think we had this hope ~5 year ago, and now >>>> I have zero hope for this :D >>>> >>>> > > Maybe a compromise (to reduce the maintainer load) is to be more >>>> strict >>>> > > in extracting the refactor part in explicit commits and allowing those >>>> > > to be backported. >>>> > > >>>> > Backporting for refactoring itself would be a big burden and >>>> sometimes impossible when there are too many branches (this is also a >>>> conflict principle of "backport only very important fixes" discussed >>>> earlier). So I am not sure if this would help in practice. >>>> > >>>> > Thanks a lot for the discussion. Would love to discuss more in the >>>> next meeting. >>>> > >>>> > Han >>>> > >>>> > > >> >>>> > > >> My 2c. >>>> > > >> >>>> > > >> /me mumbles something inaudible and goes back to PTO. >>>> > > >>>> > > Enjoy and thanks for the feedback! >>>> > > >>>> > > > >>>> > > > Thanks again for educating me during your PTO. >>>> > > > Han >>>> > > > >>>> > > >> >>>> > > >> Best regards, Ilya Maximets. >>>> > > >> >>>> > > >> >>>> > > >>> I think we need to agree on whether we should skip R3 - R9. >>>> > > >>> >>>> > > >>> Thanks, >>>> > > >>> Han >>>> > > >>> >>>> > > >>>> >>>> > > >>>> +1 from to have a formal process for backports. >>>> > > >>>> >>>> > > >>>> Numan >>>> > > >>>> >>>> > > >>>> > > Regards, >>>> > > Dumitru >>>> > > >>> > _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
