On 4/17/23 21:46, Han Zhou wrote: > On Mon, Apr 17, 2023 at 12:01 PM Ilya Maximets <[email protected]> wrote: >> >> On 4/17/23 19:20, Han Zhou wrote: >>> >>> >>> On Mon, Apr 17, 2023 at 9:24 AM Numan Siddique <[email protected] <mailto: > [email protected]>> wrote: >>>> >>>> On Mon, Apr 17, 2023 at 5:57 AM Dumitru Ceara <[email protected] > <mailto:[email protected]>> wrote: >>>>> >>>>> On 4/17/23 11:55, Dumitru Ceara wrote: >>>>>> On 4/14/23 18:01, Han Zhou wrote: >>>>>>> >>>>>>> >>>>>>> On Fri, Apr 14, 2023 at 2:16 AM Dumitru Ceara <[email protected] > <mailto:[email protected]> >>>>>>> <mailto:[email protected] <mailto:[email protected]>>> wrote: >>>>>>>> >>>>>>>> On 4/14/23 04:15, Han Zhou wrote: >>>>>>>>> On Thu, Apr 13, 2023 at 12:44 PM Han Zhou <[email protected] > <mailto:[email protected]> >>>>>>> <mailto:[email protected] <mailto:[email protected]>>> wrote: >>>>>>>>>> >>>>>>>>> I backported this series to the last release branch-23.03 and > the LTS >>>>>>>>> branch-22.03. >>>>>>>>> The code base has changed a lot so was mostly manually > backported >>>>>>> instead >>>>>>>>> of cherry-pick. >>>>>>>>> >>>>>>>> >>>>>>>> Thanks for that but I think we shouldn't skip stable branches in >>>>>>>> between. We likely still have users on those that won't upgrade > to the >>>>>>>> latest 23.03 but instead to a new z-stream release (when that > happens) >>>>>>>> of the stable branch. >>>>>>>> >>>>>>>> I cherry picked your backports and made sure the tests pass and > then >>>>>>>> pushed them to branches 22.12, 22.09 and 22.06. >>>>>>> >>>>>>> Thanks Dumitru for helping backporting. I was uncertain about > whether we >>>>>>> should continue backporting without skipping branches, as our > growing >>>>>>> number of released branches makes this approach increasingly > difficult >>>>>>> to sustain. >>>>>>> >>>>>>> The release process document >>>>>>> (https://docs.ovn.org/en/latest/internals/release-process.html < > https://docs.ovn.org/en/latest/internals/release-process.html> >>>>>>> <https://docs.ovn.org/en/latest/internals/release-process.html < > https://docs.ovn.org/en/latest/internals/release-process.html>>) doesn't >>>>>>> specify how long we should maintain non-LTS branches. My > understanding >>>>>>> was that we should primarily maintain LTS and the latest releases > for >>>>>>> regular bug fixes, while critical/security fixes would be applied > to all >>>>>>> branches. It appears we have different interpretations of this > process. >>>>>>> While it's beneficial to backport to all branches, I thought this > was >>>>>>> more of a convenience when there weren't too many branches to > manage. In >>>>>>> the worst case, a patch may require manual backporting to each > branch if >>>>>>> new features make the codebase in each branch unique. I also > question >>>>>>> what differentiates an LTS branch if we're backporting to all > branches >>>>>>> anyway. My assumption was that users opting for a non-LTS branch > should >>>>>>> be prepared to upgrade to newer releases, given the "short-term" >>>>>>> maintenance of these branches. Is my understanding incorrect? >>>>>>> >>>>>> Hi Han, >>>>>> >>>>>> I changed the subject of the email to better reflect the new > discussion. >>>>>> >>>>>> There's this statement in the release-process document: "LTS > releases >>>>>> will receive bug fixes until the point that another LTS is > released. At >>>>>> that point, the old LTS will receive an additional year of > critical and >>>>>> security fixes." >>>>>> >>>>>> Which further differentiates an LTS release from a non-LTS release. >>> >>> If you are saying "the old LTS will receive an additional year of > critical and security fixes" is the different part, I agree. But still, in > this case, for critical and security fixes, when applied to the LTS for the > *additional* year, do you agree that we will skip non-LTS releases? If not > skipping, then it makes no difference. >>> >>>>>> >>>>>>> Perhaps we should discuss and formalize our backporting practice > going >>>>>>> forward, as the number of branches will only continue to grow in > the >>>>>>> coming months and years. cc @Mark Michelson >>>>>>> <mailto:[email protected] <mailto:[email protected]>> @Numan > Siddique <mailto:[email protected] <mailto:[email protected]>> >>>>>>> >>>>>> >>>>>> Checking the OVS/OVN release-process documents again it's indeed > not >>>>>> explicitly specified that non-LTS stable release branches are > supposed >>>>>> to get all bug fixes. But the OVS document has this addition: >>>>>> >>>>>> "While branches other than LTS and the latest release are not > formally >>>>>> maintained, the OVS project usually provides stable releases for > these >>>>>> branches for at least 2 years, i.e. stable releases are provided > for the >>>>>> last 4 release branches. However, these branches may not include > all >>>>>> the fixes that LTS has in case backporting is not straightforward > and >>>>>> developers are not willing to spend their time on that (this mostly >>>>>> affects branches that are older than the LTS, because backporting > to LTS >>>>>> implies backporting to all intermediate branches)." >>>>>> >>>>>> This last part really makes me think that we SHOULD backport to all >>>>>> intermediate branches: "... because backporting to LTS >>>>>> implies backporting to all intermediate branches". >>>>>> >>>> >>>> I agree with this. >>>> >>> If we consistently backport to all intermediate releases, the LTS > release appears to function more as a reference point in the series of > releases, rather than a version with additional maintenance support. This > is ok if we all agree to it. >>> >>>>>> Maybe we should make this rule (or some other backport strategy we >>>>>> decide to use from this point on) explicit indeed. >> >> First of all, the backporting process is already documented in OVN: >> > https://docs.ovn.org/en/latest/internals/contributing/backporting-patches.html >> >> "The maintainer first applies the patch to master, then backports the >> patch to each older affected tree, as far back as it goes or at least >> to all currently supported branches. This is usually each branch back >> to the most recent LTS release branch." > > Thanks for the pointer, and sorry for missing it earlier. Probably we > should combine these two documents for OVN because both mentioned some part > of backport processes, and remove the irrelevant part (like > userspace/kernel related discussion for OVS). >
I think this still needs clarification: "... then backports the patch to each older affected tree, as far back as it goes or at least to all currently supported branches". We still don't mention that all branches between main/master and the LTS are currently supported. I always assumed they are but it's obviously not clear since we're having this conversation. >> >> No-one updated the documentation in general since migration from master >> to main, but that's not the point. >> >> Backports are done for each intermediate branch to avoid regressions >> on upgrades, which may be very bad for users and not a great thing to >> have in general (also makes it harder to bisect in which branch the >> issue was introduced). +1 That's exactly the reason why I added the missing backports of the original bug fix. >> >> This should remain this way, I think, unless OVN decides that intermediate >> branches are not supported at all and never get any backports. i.e. the >> release schema similar to DPDK, where all xx.11 releases are supported for >> a few years, but intermediate releases are fully abandoned the moment the >> next one is out. >> >> >> "the LTS release appears to function more as a reference point in the >> series of releases, rather than a version with additional maintenance >> support" >> >> The reason why it appears this way is a bit orthogonal. The point >> of extra maintenance support is preparing of official minor releases. >> Minor releases on LTS and the latest stable should be much more >> frequent than on other branches that are not fully supported. >> For example, I'm trying to release a new minor version of OVS LTS and >> the latest stable every ~2 months, while other "unsupported" branches >> may receive minor releases once in 4-6 months. >> >> The problem with OVN project that makes us ask these questions is that >> OVN doesn't really prepare any minor releases any regularly at all, >> while it should. OVN 22.03 LTS received 2 stables releases since release >> in March 2022 after people asking for them. In comparison, OVS 2.17 LTS >> received 6 new minor versions within approximately same time frame. >> It does seem like we should have more predictable minor release schedules. >> So, the original point of a long term support was to keep backporting >> fixes all the way down and prepare minor releases more frequently on >> this one particular branch. And backporting all the way down, according >> to the current documentation, means backporting to all intermediate >> branches. >> > Thanks for the explanation. Now I see the point of a LTS release. Releasing > minor versions more frequently does make it more useful. > So according to the current documented process we should not skip branches > even for the "additional" year of a LTS, although the DPDK approach seems > more attractive to me considering the amount of release branches between > two LTS in OVN. > If we're careful in backporting only "very important fixes" (as Ilya called them below) to the older LTS then even if we need to backport to all intermediate branches the maintainer effort should still be reasonable. >>>> >>>> +1 on for this. >>>> >>>> If we have these branches >>>> >>>> - Latest release (R9) >>>> - R8 >>>> - R7 >>>> - R6 (LTS) >>>> - R5 >>>> - R4 >>>> - R3 >>>> - R2 (LTS) >>>> - R1 >>>> >>>> IMO it should be enough to backport bug fixes up to the latest LTS >>>> (which is R6) in the above example. >>> >>> This example looks not bad, but it is not how OVN would be released, > because for OVN "LTS releases are scheduled to be released once every two > years". So it will look like: >>> - Latest release (R17) >>> ... >>> - R12 >>> - R11 >>> - R10 (LTS) >>> ... >>> - R4 >>> - R3 >>> - R2 (LTS) >>> - R1 >>> >>> We will always have to backport from R17 down to R10 in this example (8 > branches). >>> >>>> In case we decide to backport to R2 as well, then I think we should >>>> not skip intermediate branches (R5, R4 and R3) >>>> and instead backport upto R2. >>> >>> I don't think we will ever want to backport to R2 in this case. If we > do, it would mean from R17 - R2, which includes 16 branches. >>> >>> But look at another example: >>> - Latest release (R13) >>> - R12 >>> - R11 >>> - R10 (LTS) >>> ... >>> - R4 >>> - R3 >>> - R2 (LTS) >>> - R1 >>> >>> Because the new LTS (R10) in this example is released less than a year, > according to the rule, we will support the older LTS (R2) for critical / > security bug fixes. >> >> >> Right, that's why only very important fixes should go to R2. >> i.e. these should actually be fixes for very severe issues that >> majority of users is experiencing. Fixes for some minor leaks >> or broken communication in corner cases should not be backported >> to R2. >> >> In a worst case scenario, it will be ~12 branches to backport to, >> because an LTS release will be alive for at most 3 years. >> 12 is a lot, but again, backporting that far should only happen >> in exceptional cases. >> >> In case of a single LTS branch, there should be maximum of 8 >> branches to backport to. That is a lot too, but that's what >> the current process suggests. >> >> The issue of backporting too far can be mitigated by being more >> conscious about what kind of changes are getting backported, >> how severe the issue they are trying to fix in comparison to the >> amount of work backporting and validation of stable branches >> requires. That's partially what frequent major releases are for. >> If users want to stick to stable releases instead and still want >> every minor issue to be fixed, maybe this is time to reduce the >> frequency of major releases to e.g. 3 times a year instead of 4. >> Or reduce the support time of LTS to e.g. 1.5 years of full support >> plus 0.5 years of security / critical fixes and designating every >> xx.03 release as an LTS. >> > I like this last proposal. Shall we discuss this again in OVN meeting? I > think the overhead of maintenance was not fully understood when we agreed > on this LTS cycle initially (I could be only me). > +1 Counting how many new features/news items we got per stable (.0) release since 22.03 (LTS) I get an average of 5 items per release. Moving to 3 releases per year instead of 4 shouldn't make a huge difference in the maintenance effort. >> Also, OVN sometimes may be more conscious about constantly >> re-factoring everything. :D Sometimes it's necessary, true, but >> sometimes the same functionality can be achieved without reworking >> a ton of code just for looks of it. This may greatly reduce the >> need in manual backports. > > Strongly agree (but I also insist that my recent refactor for I-P should be > exempted :D) > We went through a period in which we realized some things need to be redesigned in order for the whole OVN model to scale (e.g., datapath groups, LB groups, generating LB logical flows, northd I-P). That would've been impossible without a certain amount of refactoring. What I hope is that the rate of refactoring goes down in the near future. Maybe a compromise (to reduce the maintainer load) is to be more strict in extracting the refactor part in explicit commits and allowing those to be backported. >> >> My 2c. >> >> /me mumbles something inaudible and goes back to PTO. Enjoy and thanks for the feedback! > > Thanks again for educating me during your PTO. > Han > >> >> Best regards, Ilya Maximets. >> >> >>> I think we need to agree on whether we should skip R3 - R9. >>> >>> Thanks, >>> Han >>> >>>> >>>> +1 from to have a formal process for backports. >>>> >>>> Numan >>>> Regards, Dumitru _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
