On 4/17/23 19:20, Han Zhou wrote: > > > On Mon, Apr 17, 2023 at 9:24 AM Numan Siddique <[email protected] > <mailto:[email protected]>> wrote: >> >> On Mon, Apr 17, 2023 at 5:57 AM Dumitru Ceara <[email protected] >> <mailto:[email protected]>> wrote: >> > >> > On 4/17/23 11:55, Dumitru Ceara wrote: >> > > On 4/14/23 18:01, Han Zhou wrote: >> > >> >> > >> >> > >> On Fri, Apr 14, 2023 at 2:16 AM Dumitru Ceara <[email protected] >> > >> <mailto:[email protected]> >> > >> <mailto:[email protected] <mailto:[email protected]>>> wrote: >> > >>> >> > >>> On 4/14/23 04:15, Han Zhou wrote: >> > >>>> On Thu, Apr 13, 2023 at 12:44 PM Han Zhou <[email protected] >> > >>>> <mailto:[email protected]> >> > >> <mailto:[email protected] <mailto:[email protected]>>> wrote: >> > >>>>> >> > >>>> I backported this series to the last release branch-23.03 and the LTS >> > >>>> branch-22.03. >> > >>>> The code base has changed a lot so was mostly manually backported >> > >> instead >> > >>>> of cherry-pick. >> > >>>> >> > >>> >> > >>> Thanks for that but I think we shouldn't skip stable branches in >> > >>> between. We likely still have users on those that won't upgrade to the >> > >>> latest 23.03 but instead to a new z-stream release (when that happens) >> > >>> of the stable branch. >> > >>> >> > >>> I cherry picked your backports and made sure the tests pass and then >> > >>> pushed them to branches 22.12, 22.09 and 22.06. >> > >> >> > >> Thanks Dumitru for helping backporting. I was uncertain about whether we >> > >> should continue backporting without skipping branches, as our growing >> > >> number of released branches makes this approach increasingly difficult >> > >> to sustain. >> > >> >> > >> The release process document >> > >> (https://docs.ovn.org/en/latest/internals/release-process.html >> > >> <https://docs.ovn.org/en/latest/internals/release-process.html> >> > >> <https://docs.ovn.org/en/latest/internals/release-process.html >> > >> <https://docs.ovn.org/en/latest/internals/release-process.html>>) >> > >> doesn't >> > >> specify how long we should maintain non-LTS branches. My understanding >> > >> was that we should primarily maintain LTS and the latest releases for >> > >> regular bug fixes, while critical/security fixes would be applied to all >> > >> branches. It appears we have different interpretations of this process. >> > >> While it's beneficial to backport to all branches, I thought this was >> > >> more of a convenience when there weren't too many branches to manage. In >> > >> the worst case, a patch may require manual backporting to each branch if >> > >> new features make the codebase in each branch unique. I also question >> > >> what differentiates an LTS branch if we're backporting to all branches >> > >> anyway. My assumption was that users opting for a non-LTS branch should >> > >> be prepared to upgrade to newer releases, given the "short-term" >> > >> maintenance of these branches. Is my understanding incorrect? >> > >> >> > > Hi Han, >> > > >> > > I changed the subject of the email to better reflect the new discussion. >> > > >> > > There's this statement in the release-process document: "LTS releases >> > > will receive bug fixes until the point that another LTS is released. At >> > > that point, the old LTS will receive an additional year of critical and >> > > security fixes." >> > > >> > > Which further differentiates an LTS release from a non-LTS release. > > If you are saying "the old LTS will receive an additional year of critical > and security fixes" is the different part, I agree. But still, in this case, > for critical and security fixes, when applied to the LTS for the *additional* > year, do you agree that we will skip non-LTS releases? If not skipping, then > it makes no difference. > >> > > >> > >> Perhaps we should discuss and formalize our backporting practice going >> > >> forward, as the number of branches will only continue to grow in the >> > >> coming months and years. cc @Mark Michelson >> > >> <mailto:[email protected] <mailto:[email protected]>> @Numan >> > >> Siddique <mailto:[email protected] <mailto:[email protected]>> >> > >> >> > > >> > > Checking the OVS/OVN release-process documents again it's indeed not >> > > explicitly specified that non-LTS stable release branches are supposed >> > > to get all bug fixes. But the OVS document has this addition: >> > > >> > > "While branches other than LTS and the latest release are not formally >> > > maintained, the OVS project usually provides stable releases for these >> > > branches for at least 2 years, i.e. stable releases are provided for the >> > > last 4 release branches. However, these branches may not include all >> > > the fixes that LTS has in case backporting is not straightforward and >> > > developers are not willing to spend their time on that (this mostly >> > > affects branches that are older than the LTS, because backporting to LTS >> > > implies backporting to all intermediate branches)." >> > > >> > > This last part really makes me think that we SHOULD backport to all >> > > intermediate branches: "... because backporting to LTS >> > > implies backporting to all intermediate branches". >> > > >> >> I agree with this. >> > If we consistently backport to all intermediate releases, the LTS release > appears to function more as a reference point in the series of releases, > rather than a version with additional maintenance support. This is ok if we > all agree to it. > >> > > Maybe we should make this rule (or some other backport strategy we >> > > decide to use from this point on) explicit indeed.
First of all, the backporting process is already documented in OVN: https://docs.ovn.org/en/latest/internals/contributing/backporting-patches.html "The maintainer first applies the patch to master, then backports the patch to each older affected tree, as far back as it goes or at least to all currently supported branches. This is usually each branch back to the most recent LTS release branch." No-one updated the documentation in general since migration from master to main, but that's not the point. Backports are done for each intermediate branch to avoid regressions on upgrades, which may be very bad for users and not a great thing to have in general (also makes it harder to bisect in which branch the issue was introduced). This should remain this way, I think, unless OVN decides that intermediate branches are not supported at all and never get any backports. i.e. the release schema similar to DPDK, where all xx.11 releases are supported for a few years, but intermediate releases are fully abandoned the moment the next one is out. "the LTS release appears to function more as a reference point in the series of releases, rather than a version with additional maintenance support" The reason why it appears this way is a bit orthogonal. The point of extra maintenance support is preparing of official minor releases. Minor releases on LTS and the latest stable should be much more frequent than on other branches that are not fully supported. For example, I'm trying to release a new minor version of OVS LTS and the latest stable every ~2 months, while other "unsupported" branches may receive minor releases once in 4-6 months. The problem with OVN project that makes us ask these questions is that OVN doesn't really prepare any minor releases any regularly at all, while it should. OVN 22.03 LTS received 2 stables releases since release in March 2022 after people asking for them. In comparison, OVS 2.17 LTS received 6 new minor versions within approximately same time frame. So, the original point of a long term support was to keep backporting fixes all the way down and prepare minor releases more frequently on this one particular branch. And backporting all the way down, according to the current documentation, means backporting to all intermediate branches. >> >> +1 on for this. >> >> If we have these branches >> >> - Latest release (R9) >> - R8 >> - R7 >> - R6 (LTS) >> - R5 >> - R4 >> - R3 >> - R2 (LTS) >> - R1 >> >> IMO it should be enough to backport bug fixes up to the latest LTS >> (which is R6) in the above example. > > This example looks not bad, but it is not how OVN would be released, because > for OVN "LTS releases are scheduled to be released once every two years". So > it will look like: > - Latest release (R17) > ... > - R12 > - R11 > - R10 (LTS) > ... > - R4 > - R3 > - R2 (LTS) > - R1 > > We will always have to backport from R17 down to R10 in this example (8 > branches). > >> In case we decide to backport to R2 as well, then I think we should >> not skip intermediate branches (R5, R4 and R3) >> and instead backport upto R2. > > I don't think we will ever want to backport to R2 in this case. If we do, it > would mean from R17 - R2, which includes 16 branches. > > But look at another example: > - Latest release (R13) > - R12 > - R11 > - R10 (LTS) > ... > - R4 > - R3 > - R2 (LTS) > - R1 > > Because the new LTS (R10) in this example is released less than a year, > according to the rule, we will support the older LTS (R2) for critical / > security bug fixes. Right, that's why only very important fixes should go to R2. i.e. these should actually be fixes for very severe issues that majority of users is experiencing. Fixes for some minor leaks or broken communication in corner cases should not be backported to R2. In a worst case scenario, it will be ~12 branches to backport to, because an LTS release will be alive for at most 3 years. 12 is a lot, but again, backporting that far should only happen in exceptional cases. In case of a single LTS branch, there should be maximum of 8 branches to backport to. That is a lot too, but that's what the current process suggests. The issue of backporting too far can be mitigated by being more conscious about what kind of changes are getting backported, how severe the issue they are trying to fix in comparison to the amount of work backporting and validation of stable branches requires. That's partially what frequent major releases are for. If users want to stick to stable releases instead and still want every minor issue to be fixed, maybe this is time to reduce the frequency of major releases to e.g. 3 times a year instead of 4. Or reduce the support time of LTS to e.g. 1.5 years of full support plus 0.5 years of security / critical fixes and designating every xx.03 release as an LTS. Also, OVN sometimes may be more conscious about constantly re-factoring everything. :D Sometimes it's necessary, true, but sometimes the same functionality can be achieved without reworking a ton of code just for looks of it. This may greatly reduce the need in manual backports. My 2c. /me mumbles something inaudible and goes back to PTO. Best regards, Ilya Maximets. > I think we need to agree on whether we should skip R3 - R9. > > Thanks, > Han > >> >> +1 from to have a formal process for backports. >> >> Numan >> >> >> >> >> >> > > >> > > CC: Ilya for input from OVS perspective. >> > >> > > CC: Ilya for input from OVS perspective. (this I actually CC-ed him..) >> > >> > > >> > > Regards, >> > > Dumitru >> > > >> > >> > _______________________________________________ >> > dev mailing list >> > [email protected] <mailto:[email protected]> >> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev >> > <https://mail.openvswitch.org/mailman/listinfo/ovs-dev> _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
