> > On Tue, Apr 18, 2023 at 10:33 AM Han Zhou <[email protected]
<mailto:[email protected]>> wrote:
>
>
>
> On Tue, Apr 18, 2023 at 1:50 AM Dumitru Ceara <[email protected]
<mailto:[email protected]>> wrote:
> >
> > On 4/17/23 21:46, Han Zhou wrote:
> > > On Mon, Apr 17, 2023 at 12:01 PM Ilya Maximets
<[email protected] <mailto:[email protected]>> wrote:
> > >>
> > >> On 4/17/23 19:20, Han Zhou wrote:
> > >>>
> > >>>
> > >>> On Mon, Apr 17, 2023 at 9:24 AM Numan Siddique <[email protected]
<mailto:[email protected]> <mailto:
> > > [email protected] <mailto:[email protected]>>> wrote:
> > >>>>
> > >>>> On Mon, Apr 17, 2023 at 5:57 AM Dumitru Ceara
<[email protected] <mailto:[email protected]>
> > > <mailto:[email protected] <mailto:[email protected]>>> wrote:
> > >>>>>
> > >>>>> On 4/17/23 11:55, Dumitru Ceara wrote:
> > >>>>>> On 4/14/23 18:01, Han Zhou wrote:
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Fri, Apr 14, 2023 at 2:16 AM Dumitru Ceara
<[email protected] <mailto:[email protected]>
> > > <mailto:[email protected] <mailto:[email protected]>>
> > >>>>>>> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> On 4/14/23 04:15, Han Zhou wrote:
> > >>>>>>>>> On Thu, Apr 13, 2023 at 12:44 PM Han Zhou <[email protected]
<mailto:[email protected]>
> > > <mailto:[email protected] <mailto:[email protected]>>
> > >>>>>>> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>> I backported this series to the last release branch-23.03 and
> > > the LTS
> > >>>>>>>>> branch-22.03.
> > >>>>>>>>> The code base has changed a lot so was mostly manually
> > > backported
> > >>>>>>> instead
> > >>>>>>>>> of cherry-pick.
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>> Thanks for that but I think we shouldn't skip stable
branches in
> > >>>>>>>> between. We likely still have users on those that won't
upgrade
> > > to the
> > >>>>>>>> latest 23.03 but instead to a new z-stream release (when that
> > > happens)
> > >>>>>>>> of the stable branch.
> > >>>>>>>>
> > >>>>>>>> I cherry picked your backports and made sure the tests
pass and
> > > then
> > >>>>>>>> pushed them to branches 22.12, 22.09 and 22.06.
> > >>>>>>>
> > >>>>>>> Thanks Dumitru for helping backporting. I was uncertain about
> > > whether we
> > >>>>>>> should continue backporting without skipping branches, as our
> > > growing
> > >>>>>>> number of released branches makes this approach increasingly
> > > difficult
> > >>>>>>> to sustain.
> > >>>>>>>
> > >>>>>>> The release process document
> > >>>>>>>
(https://docs.ovn.org/en/latest/internals/release-process.html
<https://docs.ovn.org/en/latest/internals/release-process.html> <
> > > https://docs.ovn.org/en/latest/internals/release-process.html
<https://docs.ovn.org/en/latest/internals/release-process.html>>
> > >>>>>>>
<https://docs.ovn.org/en/latest/internals/release-process.html
<https://docs.ovn.org/en/latest/internals/release-process.html> <
> > > https://docs.ovn.org/en/latest/internals/release-process.html
<https://docs.ovn.org/en/latest/internals/release-process.html>>>) doesn't
> > >>>>>>> specify how long we should maintain non-LTS branches. My
> > > understanding
> > >>>>>>> was that we should primarily maintain LTS and the latest
releases
> > > for
> > >>>>>>> regular bug fixes, while critical/security fixes would be
applied
> > > to all
> > >>>>>>> branches. It appears we have different interpretations of this
> > > process.
> > >>>>>>> While it's beneficial to backport to all branches, I
thought this
> > > was
> > >>>>>>> more of a convenience when there weren't too many branches to
> > > manage. In
> > >>>>>>> the worst case, a patch may require manual backporting to each
> > > branch if
> > >>>>>>> new features make the codebase in each branch unique. I also
> > > question
> > >>>>>>> what differentiates an LTS branch if we're backporting to all
> > > branches
> > >>>>>>> anyway. My assumption was that users opting for a non-LTS
branch
> > > should
> > >>>>>>> be prepared to upgrade to newer releases, given the
"short-term"
> > >>>>>>> maintenance of these branches. Is my understanding incorrect?
> > >>>>>>>
> > >>>>>> Hi Han,
> > >>>>>>
> > >>>>>> I changed the subject of the email to better reflect the new
> > > discussion.
> > >>>>>>
> > >>>>>> There's this statement in the release-process document: "LTS
> > > releases
> > >>>>>> will receive bug fixes until the point that another LTS is
> > > released. At
> > >>>>>> that point, the old LTS will receive an additional year of
> > > critical and
> > >>>>>> security fixes."
> > >>>>>>
> > >>>>>> Which further differentiates an LTS release from a non-LTS
release.
> > >>>
> > >>> If you are saying "the old LTS will receive an additional year of
> > > critical and security fixes" is the different part, I agree. But
still, in
> > > this case, for critical and security fixes, when applied to the
LTS for the
> > > *additional* year, do you agree that we will skip non-LTS
releases? If not
> > > skipping, then it makes no difference.
> > >>>
> > >>>>>>
> > >>>>>>> Perhaps we should discuss and formalize our backporting
practice
> > > going
> > >>>>>>> forward, as the number of branches will only continue to
grow in
> > > the
> > >>>>>>> coming months and years. cc @Mark Michelson
> > >>>>>>> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>> @Numan
> > > Siddique <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>> Checking the OVS/OVN release-process documents again it's indeed
> > > not
> > >>>>>> explicitly specified that non-LTS stable release branches are
> > > supposed
> > >>>>>> to get all bug fixes. But the OVS document has this addition:
> > >>>>>>
> > >>>>>> "While branches other than LTS and the latest release are not
> > > formally
> > >>>>>> maintained, the OVS project usually provides stable releases for
> > > these
> > >>>>>> branches for at least 2 years, i.e. stable releases are provided
> > > for the
> > >>>>>> last 4 release branches. However, these branches may not
include
> > > all
> > >>>>>> the fixes that LTS has in case backporting is not
straightforward
> > > and
> > >>>>>> developers are not willing to spend their time on that (this
mostly
> > >>>>>> affects branches that are older than the LTS, because
backporting
> > > to LTS
> > >>>>>> implies backporting to all intermediate branches)."
> > >>>>>>
> > >>>>>> This last part really makes me think that we SHOULD backport
to all
> > >>>>>> intermediate branches: "... because backporting to LTS
> > >>>>>> implies backporting to all intermediate branches".
> > >>>>>>
> > >>>>
> > >>>> I agree with this.
> > >>>>
> > >>> If we consistently backport to all intermediate releases, the LTS
> > > release appears to function more as a reference point in the
series of
> > > releases, rather than a version with additional maintenance
support. This
> > > is ok if we all agree to it.
> > >>>
> > >>>>>> Maybe we should make this rule (or some other backport
strategy we
> > >>>>>> decide to use from this point on) explicit indeed.
> > >>
> > >> First of all, the backporting process is already documented in OVN:
> > >>
> > >
https://docs.ovn.org/en/latest/internals/contributing/backporting-patches.html <https://docs.ovn.org/en/latest/internals/contributing/backporting-patches.html>
> > >>
> > >> "The maintainer first applies the patch to master, then
backports the
> > >> patch to each older affected tree, as far back as it goes or at
least
> > >> to all currently supported branches. This is usually each
branch back
> > >> to the most recent LTS release branch."
> > >
> > > Thanks for the pointer, and sorry for missing it earlier. Probably we
> > > should combine these two documents for OVN because both mentioned
some part
> > > of backport processes, and remove the irrelevant part (like
> > > userspace/kernel related discussion for OVS).
> > >
> >
> > I think this still needs clarification: "... then backports the
patch to
> > each older affected tree, as far back as it goes or at least to all
> > currently supported branches". We still don't mention that all
branches
> > between main/master and the LTS are currently supported. I always
> > assumed they are but it's obviously not clear since we're having this
> > conversation.
> >
> > >>
> > >> No-one updated the documentation in general since migration from
master
> > >> to main, but that's not the point.
> > >>
> > >> Backports are done for each intermediate branch to avoid regressions
> > >> on upgrades, which may be very bad for users and not a great
thing to
> > >> have in general (also makes it harder to bisect in which branch the
> > >> issue was introduced).
> >
> > +1
> >
> > That's exactly the reason why I added the missing backports of the
> > original bug fix.
> >
> > >>
> > >> This should remain this way, I think, unless OVN decides that
intermediate
> > >> branches are not supported at all and never get any backports.
i.e. the
> > >> release schema similar to DPDK, where all xx.11 releases are
supported for
> > >> a few years, but intermediate releases are fully abandoned the
moment the
> > >> next one is out.
> > >>
> > >>
> > >> "the LTS release appears to function more as a reference point
in the
> > >> series of releases, rather than a version with additional
maintenance
> > >> support"
> > >>
> > >> The reason why it appears this way is a bit orthogonal. The point
> > >> of extra maintenance support is preparing of official minor
releases.
> > >> Minor releases on LTS and the latest stable should be much more
> > >> frequent than on other branches that are not fully supported.
> > >> For example, I'm trying to release a new minor version of OVS
LTS and
> > >> the latest stable every ~2 months, while other "unsupported"
branches
> > >> may receive minor releases once in 4-6 months.
> > >>
> > >> The problem with OVN project that makes us ask these questions
is that
> > >> OVN doesn't really prepare any minor releases any regularly at all,
> > >> while it should. OVN 22.03 LTS received 2 stables releases
since release
> > >> in March 2022 after people asking for them. In comparison, OVS
2.17 LTS
> > >> received 6 new minor versions within approximately same time frame.
> > >>
> >
> > It does seem like we should have more predictable minor release
schedules.
> >
> > >> So, the original point of a long term support was to keep
backporting
> > >> fixes all the way down and prepare minor releases more frequently on
> > >> this one particular branch. And backporting all the way down,
according
> > >> to the current documentation, means backporting to all intermediate
> > >> branches.
> > >>
> > > Thanks for the explanation. Now I see the point of a LTS release.
Releasing
> > > minor versions more frequently does make it more useful.
> > > So according to the current documented process we should not skip
branches
> > > even for the "additional" year of a LTS, although the DPDK
approach seems
> > > more attractive to me considering the amount of release branches
between
> > > two LTS in OVN.
> > >
> >
> > If we're careful in backporting only "very important fixes" (as Ilya
> > called them below) to the older LTS then even if we need to backport to
> > all intermediate branches the maintainer effort should still be
reasonable.
> >
> This may sound reasonable but I would not count on it. What we can do
is try our best to ensure quality and avoid bugs as much as we can, but
beyond that there is not much we can control.
> When bugs need to be fixed, the importance of the bug fix mostly
depends on the bug itself (how users are impacted), not how we evaluate
it. (We may tend to backport less bug fixes when there are more branches
to maintain, but if we do that it would be wrong.)
>
> > >>>>
> > >>>> +1 on for this.
> > >>>>
> > >>>> If we have these branches
> > >>>>
> > >>>> - Latest release (R9)
> > >>>> - R8
> > >>>> - R7
> > >>>> - R6 (LTS)
> > >>>> - R5
> > >>>> - R4
> > >>>> - R3
> > >>>> - R2 (LTS)
> > >>>> - R1
> > >>>>
> > >>>> IMO it should be enough to backport bug fixes up to the latest LTS
> > >>>> (which is R6) in the above example.
> > >>>
> > >>> This example looks not bad, but it is not how OVN would be
released,
> > > because for OVN "LTS releases are scheduled to be released once
every two
> > > years". So it will look like:
> > >>> - Latest release (R17)
> > >>> ...
> > >>> - R12
> > >>> - R11
> > >>> - R10 (LTS)
> > >>> ...
> > >>> - R4
> > >>> - R3
> > >>> - R2 (LTS)
> > >>> - R1
> > >>>
> > >>> We will always have to backport from R17 down to R10 in this
example (8
> > > branches).
> > >>>
> > >>>> In case we decide to backport to R2 as well, then I think we
should
> > >>>> not skip intermediate branches (R5, R4 and R3)
> > >>>> and instead backport upto R2.
> > >>>
> > >>> I don't think we will ever want to backport to R2 in this case.
If we
> > > do, it would mean from R17 - R2, which includes 16 branches.
> > >>>
> > >>> But look at another example:
> > >>> - Latest release (R13)
> > >>> - R12
> > >>> - R11
> > >>> - R10 (LTS)
> > >>> ...
> > >>> - R4
> > >>> - R3
> > >>> - R2 (LTS)
> > >>> - R1
> > >>>
> > >>> Because the new LTS (R10) in this example is released less than
a year,
> > > according to the rule, we will support the older LTS (R2) for
critical /
> > > security bug fixes.
> > >>
> > >>
> > >> Right, that's why only very important fixes should go to R2.
> > >> i.e. these should actually be fixes for very severe issues that
> > >> majority of users is experiencing. Fixes for some minor leaks
> > >> or broken communication in corner cases should not be backported
> > >> to R2.
> > >>
> > >> In a worst case scenario, it will be ~12 branches to backport to,
> > >> because an LTS release will be alive for at most 3 years.
> > >> 12 is a lot, but again, backporting that far should only happen
> > >> in exceptional cases.
> > >>
> > >> In case of a single LTS branch, there should be maximum of 8
> > >> branches to backport to. That is a lot too, but that's what
> > >> the current process suggests.
> > >>
> > >> The issue of backporting too far can be mitigated by being more
> > >> conscious about what kind of changes are getting backported,
> > >> how severe the issue they are trying to fix in comparison to the
> > >> amount of work backporting and validation of stable branches
> > >> requires. That's partially what frequent major releases are for.
> > >> If users want to stick to stable releases instead and still want
> > >> every minor issue to be fixed, maybe this is time to reduce the
> > >> frequency of major releases to e.g. 3 times a year instead of 4.
> > >> Or reduce the support time of LTS to e.g. 1.5 years of full support
> > >> plus 0.5 years of security / critical fixes and designating every
> > >> xx.03 release as an LTS.
> > >>
> > > I like this last proposal. Shall we discuss this again in OVN
meeting? I
> > > think the overhead of maintenance was not fully understood when
we agreed
> > > on this LTS cycle initially (I could be only me).
> > >
> >
> > +1
> >
> > Counting how many new features/news items we got per stable (.0)
release
> > since 22.03 (LTS) I get an average of 5 items per release. Moving to 3
> > releases per year instead of 4 shouldn't make a huge difference in the
> > maintenance effort.
> >
> Agree that moving to 3 releases per year doesn't help much. What I
liked is this part:
>
> ... reduce the support time of LTS to e.g. 1.5 years of full support
> plus 0.5 years of security / critical fixes and designating every
> xx.03 release as an LTS.
>
> > >> Also, OVN sometimes may be more conscious about constantly
> > >> re-factoring everything. :D Sometimes it's necessary, true, but
> > >> sometimes the same functionality can be achieved without reworking
> > >> a ton of code just for looks of it. This may greatly reduce the
> > >> need in manual backports.
> > >
> > > Strongly agree (but I also insist that my recent refactor for I-P
should be
> > > exempted :D)
> > >
> >
> > We went through a period in which we realized some things need to be
> > redesigned in order for the whole OVN model to scale (e.g., datapath
> > groups, LB groups, generating LB logical flows, northd I-P). That
> > would've been impossible without a certain amount of refactoring. What
> > I hope is that the rate of refactoring goes down in the near future.
> >
Forgot to comment on this. I think we had this hope ~5 year ago, and now
I have zero hope for this :D
> > Maybe a compromise (to reduce the maintainer load) is to be more strict
> > in extracting the refactor part in explicit commits and allowing those
> > to be backported.
> >
> Backporting for refactoring itself would be a big burden and
sometimes impossible when there are too many branches (this is also a
conflict principle of "backport only very important fixes" discussed
earlier). So I am not sure if this would help in practice.
>
> Thanks a lot for the discussion. Would love to discuss more in the
next meeting.
>
> Han
>
> > >>
> > >> My 2c.
> > >>
> > >> /me mumbles something inaudible and goes back to PTO.
> >
> > Enjoy and thanks for the feedback!
> >
> > >
> > > Thanks again for educating me during your PTO.
> > > Han
> > >
> > >>
> > >> Best regards, Ilya Maximets.
> > >>
> > >>
> > >>> I think we need to agree on whether we should skip R3 - R9.
> > >>>
> > >>> Thanks,
> > >>> Han
> > >>>
> > >>>>
> > >>>> +1 from to have a formal process for backports.
> > >>>>
> > >>>> Numan
> > >>>>
> >
> > Regards,
> > Dumitru
> >
