On Mon, 28 Aug 2023 at 21:40, Adrien Nader <adr...@notk.org> wrote: > > Hi, > > Finally back after pretty long vacation. :) > > On Mon, Aug 21, 2023, Mauricio Faria de Oliveira wrote: > > Hey Simon, > > > > On Thu, Aug 17, 2023 at 4:42 AM Simon Chopin <simon.cho...@canonical.com> > > wrote: > > > > > > Hi, > > > > > > Quoting Mauricio Faria de Oliveira (2023-08-15 15:20:50) > > > > Hey, > > > > > > > > I'd like to request input (initially thinking of involved teams: SRU, > > > > Foundations) > > > > on backports of some performance improvement patches to OpenSSL in > > > > Jammy. > > > > (Please feel free to comment and include others as appropriate.) > > > > > > FYI, Adrien is currently offline, he should be back in a couple of > > > weeks. Meanwhile you can get my 2¢ as his backup on openssl-related > > > matters :) > > > > > > > SEG has a customer support ticket about it, which allows us to put in > > > > the work, > > > > but of course, it is OpenSSL, an SRU, thus agreement beforehand is > > > > needed. > > > > > > > > ... > > > > > > > > The context is OpenSSL 3.0 has known, significant performance > > > > regressions [0] > > > > from OpenSSL 1.1.1, which has been addressed / still in-progress > > > > upstream: > > > > 1) some patches in the 3.0 stable branch > > > > 2) some patches in the master branch (ie, not backported to 3.0) > > > > 3) some issues still open > > > > > > > > To offset regression risk, there are benefits; e.g., 1) Performance: > > > > some improvement 2) Security: smaller delta to 3.0 branch (may help > > > > with CVE fix backports) 3) Community: possibly help with mentions of > > > > Ubuntu 22.04 in regressions > > > > > > > > IMHO, backports should be restricted to the 3.0 branch (so not to > > > > defeat 2). > > > > > > Not Security, but I'm OK with backporting patches from the master branch > > > as well, but only if we can get them in Mantic first, and wait for its > > > GA. That would give exposure to a wider range of hardware, shaking off > > > some regressions. > > > > Understood. We'll consider that option, for patches from the master branch. > > > > Hopefully, most patches may be in 3.0 stable (to land before October), > > but nonetheless, it's reassuring to know there's a way forward for > > other patches. > > I've been wanting to _ultimately_ update openssl in our stable releases. > That means updating 22.04 with another openssl LTS. Before anyone > panics: that's a really long term goal. > > Since openssl had relatively small SRUs in the past years, I had planned > to do things very progressively. At the moment we have 3.0.10 in Mantic > (I didn't check who updated it but thanks!) and 3.0.8 in Lunar. No > regression compared to Jammy's 3.0.2. I didn't read about issues on > other distros either. That's encouraging. > > We're talking about two kinds of patches: performance and bug fixes. > Upstream backports bug fixes to the 3.0 release but they don't > necessarily do it for performance fix (they probably typically don't). > There's no "complete" list for such fixes either (there's not even a > definition of which patches would qualify). > While the patches that Mauricio linked to have been backported to 3.0, a > number of the performance fixes have not and it will probably be > difficult to backport them since they can be architectural or locking > changes. > I'm pretty sure the number of performance fixes will only lower and the > situation is transitory but right now there's a demand for several > performance fixes. For instance, it seems Haproxy has encountered > redhibitory performance. > > Below are the "steps" I had thought about before my vacation (hopefully > I remember well :) ). Steps 1 and 2 could be merged or shortened > depending on the SRU team. > > 1- SRU with select patches
We have done this in the past. Including backporting lots of optimisations for armhf, s390x, ppc. It is difficult, but possible to land safely. > > There have been a number of issues reported with upstream patches but > they had low impact and/or had easy workarounds. Since there are now > other incentives to do an SRU, I plan to include everything in it. > I think all patches are in Lunar or even previous releases. > > I don't have a list for performance patches however, except for a couple > ones. I don't plan to list them by myself; instead I welcome inputs on > that. Benchmarking is simply too much work, especially considering the > kind of performance regressions from 1.1 to 3.0 that I've seen talked > about on the openssl bug tracker. That would probably be a year-long > full-time job. > > I would like to start preparing this SRU soon and I hope to have it > ready in October (that's my first SRU this large but I think/hope that's > realistic). Since it would be the first one, I plan to stick to changes > that don't require backporting work. For subsequent SRUs we could > include more patches as Simon said (i.e. patches from master provided > they go into non-LTS releases first during their development cycle). > > 2- Micro-release updates Each time we attempted this, it ended up in a disaster. Ubuntu ABI guarantees are stronger than those of upstream. Often micro-releases would change runtime behaviour of existing APIs, or break API and ABI in the ways that would be unacceptable to a binary distribution like Ubuntu causing to recompiled/rebuild/re-release all of Ubuntu and all of 3rd party software. Whilst upstream micro-release policy remains as loose as it is, I do not believe we could get to shipping upstream mciro-releases updates in Ubuntu. If upstream tightens their policies, and overall integration testing improves, I see this as quite risky. > > If all goes well, I'd like to do micro-release updates for openssl; that > would obviously include more changes and possibly include some > performance improvements. > > Note that upstream really wants us to track micro-releases and has > unambiguously stated they care a lot about stability: > https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/2019970 > > 3- Full release updates > Here be dragons. Often testing on newer interim releases is insufficient to testing on an LTS release. And often there are API/ABI fixups done all over the universe that prevent sensible backporting. Please note the 1.1.1 fiasco. The timing of upstream 1.1.1. release slipped and we agreed to ship bionic 1.1.0 and upgrade it to 1.1.1 shortly after GA to then freeze bionic on 1.1.1. for the rest of its lifetime, as if bionic GA happened with 1.1.1. We then spent years backporting and SRUing fix-ups of all the packages that got improvements to handle 1.1.1. correctly which all were missing in bionic. Despite largely APi/ABI compatibility between 1.1.0 and 1.1.1 series. Many issues remaining unresolved until later interim releases and ultimately were only fully resolved in focal. At this point I'd rather fork and maintain 3.1 in next LTS for ever than to try to ever backport things ever again. if you want more details, I can dig up all the bug reports for your analysis. Also note that v3.1 release appears to be a fork. Lots of things on master were supposed to land in v3.1 release. Then v3.1 branch got branched from much older codebase due to change in release criteria and feature priority of what to ship in v3.1. Thus subsequently you can see that on master a lot of APIs got their version introduced in bumped from 3.1 to 3.2. Thus 3.1 was a smaller than expected release, and subsequently 3.2 has many changes that were developed from before 3.1 and haven't shipped yet. > As I stated, I'd like to be able to update openssl and maybe have the > same openssl LTS version across Ubuntu LTS versions. Obviously that > won't be worked on before the previous steps are successful. The initial > motive was precisely performance regressions but also more generally the > fact that it could be less work and less risk overall (I'm worried about > patching 3.0 on 22.04 in 8 or 10 years, while most probably having to > backport whole new features). > > Best, > > -- > Adrien > > -- > Ubuntu-release mailing list > Ubuntu-release@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-release -- okurrr, Dimitri -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
