Re: RFC on Jammy SRU with OpenSSL 3.0 backports for performance
Hi Robie, On Wed, Sep 27, 2023, Robie Basak wrote: > Hi Adrien, > > Thank you for working on improving OpenSSL in our stable releases! Steve > has already said that performance improvements are acceptable. That's > fine by me as well, but of course subject to review of the specific > changes. I wanted to talk about your longer term goals though: > > On Mon, Aug 28, 2023 at 10:33:12PM +0200, Adrien Nader wrote: > > I've been wanting to _ultimately_ update openssl in our stable releases. > > That means updating 22.04 with another openssl LTS. Before anyone > > panics: that's a really long term goal. > > > > Since openssl had relatively small SRUs in the past years, I had planned > > to do things very progressively. At the moment we have 3.0.10 in Mantic > > (I didn't check who updated it but thanks!) and 3.0.8 in Lunar. No > > regression compared to Jammy's 3.0.2. I didn't read about issues on > > other distros either. That's encouraging. > > Have you done any research about previous regressions in OpenSSL we've > had? Dimitri mentioned some. I try to tag these in the bug tracker so > that they can be found later for analysis[1], and I think they're quite > interesting in themselves. I saw Dimitri's message but I was on vacation or just back from vacation and I've been constantly remembering about it but never when in front of my computer. I'll bring that up again. Thanks for mentioning it again! I wasn't aware of the regression-update tag. That's very interesting and I'll definitely keep that in mind. > Here are some that demonstrate use cases that might influence our future > decisions: > > https://bugs.launchpad.net/debian/+source/nodejs/+bug/1779863 - nodejs > upstream arrange to build against an ABI and then ship their own built > binaries to third parties who then use an Ubuntu LTS expecting that ABI. > > https://bugs.launchpad.net/ubuntu/cosmic/+source/net-snmp/+bug/1794589 - > similar to above (or perhaps the same) - nodejs upstream consider the > OpenSSL ABI version "pinned" against every upstream release for the > purpose of binary compatibility of built binary modules around their > ecosystem. > > https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1979639 - ABI > also means configuration file formatting in /etc/ssl/ > > https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/986147 - prior to > release of Precise, the move from 1.0.0 to 1.0.1 changed behaviour in a > way that broke quite a few users in what sounds like an upstream > microrelease update. This was a long time ago though. Perhaps upstream > have revised their policies on releases since. > > Also, bugs tagged bionic-openssl-1.1 (I see 109 tasks though that > includes multiple tasks against the same bugs) track the issues Dimitri > mentions caused by the introduction of 1.1 into Bionic. When searching > for these, be sure to include "Fix Released", "Invalid", "Won't Fix" and > suchlike since a simple tag search won't find tasks against stable > releases otherwise. > > Again, I appreciate you driving maintenance for OpenSSL in Ubuntu. I > hope the above will help inform your choices. If these past classes of > issues have been addressed and are unlikely to recur, that's fine too if > there's some analysis that demonstrates that! Basically, I would like that by NN we have a framework to decide whether an SRU for openssl specifically is safe. There are several criteria: I know of at least ABI, behavior and configuration. ABI is probably the only one which can be verified automatically but I don't think there's an explicit process for that at the moment. Compatibility for behavior and configuration are more difficult to assess and I know of no automated analysis for these. I checked the issues you linked to and I think ensuring the ABI is the same would have prevented two of these. The configuration one _should_ not happen with a .patch version and might be easy enough to spot in a limited set of patches. The 1.0.1 one is the most unexpected one and needs reviews and tests (I think that's actually an issue I was looking for but only had vague recollections about and couldn't find anymore in the noise after 10 years :) ). I'll have a look at the bionic-openssl-1.1 issues, thanks. I'm interesting in looking at as many possible issues in order to come up with the best rules and be sure everything is safe. Honestly, I wish we didn't have to do that but it looks like 3.0.notthatmuch releases had a lot of issues. I don't know if there will be the same interest in updating 24.04's openssl since that version will be including 2 years of fixes and only few new bugs. It's also getting far less likely that there will be further performance improvements in 3.0.x. Thinking about all of this as I type, I think I will compare the ABIs of 3.0.2 (jammy's) and 3.0.11 (mantic's) tomorrow^Wtonight. Much to my surprise, the ABIs for jammy and mantic appear identical. At least that's not immediately ruling out SRUs. I'll try to quantify th
Re: RFC on Jammy SRU with OpenSSL 3.0 backports for performance
Hi Adrien, Thank you for working on improving OpenSSL in our stable releases! Steve has already said that performance improvements are acceptable. That's fine by me as well, but of course subject to review of the specific changes. I wanted to talk about your longer term goals though: On Mon, Aug 28, 2023 at 10:33:12PM +0200, Adrien Nader wrote: > I've been wanting to _ultimately_ update openssl in our stable releases. > That means updating 22.04 with another openssl LTS. Before anyone > panics: that's a really long term goal. > > Since openssl had relatively small SRUs in the past years, I had planned > to do things very progressively. At the moment we have 3.0.10 in Mantic > (I didn't check who updated it but thanks!) and 3.0.8 in Lunar. No > regression compared to Jammy's 3.0.2. I didn't read about issues on > other distros either. That's encouraging. Have you done any research about previous regressions in OpenSSL we've had? Dimitri mentioned some. I try to tag these in the bug tracker so that they can be found later for analysis[1], and I think they're quite interesting in themselves. Here are some that demonstrate use cases that might influence our future decisions: https://bugs.launchpad.net/debian/+source/nodejs/+bug/1779863 - nodejs upstream arrange to build against an ABI and then ship their own built binaries to third parties who then use an Ubuntu LTS expecting that ABI. https://bugs.launchpad.net/ubuntu/cosmic/+source/net-snmp/+bug/1794589 - similar to above (or perhaps the same) - nodejs upstream consider the OpenSSL ABI version "pinned" against every upstream release for the purpose of binary compatibility of built binary modules around their ecosystem. https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1979639 - ABI also means configuration file formatting in /etc/ssl/ https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/986147 - prior to release of Precise, the move from 1.0.0 to 1.0.1 changed behaviour in a way that broke quite a few users in what sounds like an upstream microrelease update. This was a long time ago though. Perhaps upstream have revised their policies on releases since. Also, bugs tagged bionic-openssl-1.1 (I see 109 tasks though that includes multiple tasks against the same bugs) track the issues Dimitri mentions caused by the introduction of 1.1 into Bionic. When searching for these, be sure to include "Fix Released", "Invalid", "Won't Fix" and suchlike since a simple tag search won't find tasks against stable releases otherwise. Again, I appreciate you driving maintenance for OpenSSL in Ubuntu. I hope the above will help inform your choices. If these past classes of issues have been addressed and are unlikely to recur, that's fine too if there's some analysis that demonstrates that! Robie [1] Please, everyone should do this - the regression-update tag is useful not only to flag bugs for attention but also for retrospective analysis to help inform future decisions. signature.asc Description: PGP signature -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: RFC on Jammy SRU with OpenSSL 3.0 backports for performance
No problem Adrien, look forward to the SRU getting into the pipeline and hopefully landing! On Wed, Sep 27, 2023 at 1:43 AM Adrien Nader wrote: > Hi Rafael, > > On Fri, Sep 15, 2023, Rafael Lopez wrote: > > Thanks for working on the SRU. FWIW, I tested the packages from your PPA, > > looks good to me and results match previous tests performed. > > Let me know if I can do any further testing or assist in any way to > > expedite the SRU. > > I'm deeply sorry for taking such a long time to answer. I was fully > taken by a couple of other tasks which I really needed to finish. The > good news is that they're done and now I can mostly dedicate my time to > this. > > Thanks a lot for your tests. There were other reports that the SRU fixed > the reporters' issue so we can confidently move forward with this. I'm > at a conference until tomorrow evening but this is the next thing I'll > be working on, worst case: Thursday. > > -- > Adrien > -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: RFC on Jammy SRU with OpenSSL 3.0 backports for performance
Hi Rafael, On Fri, Sep 15, 2023, Rafael Lopez wrote: > Thanks for working on the SRU. FWIW, I tested the packages from your PPA, > looks good to me and results match previous tests performed. > Let me know if I can do any further testing or assist in any way to > expedite the SRU. I'm deeply sorry for taking such a long time to answer. I was fully taken by a couple of other tasks which I really needed to finish. The good news is that they're done and now I can mostly dedicate my time to this. Thanks a lot for your tests. There were other reports that the SRU fixed the reporters' issue so we can confidently move forward with this. I'm at a conference until tomorrow evening but this is the next thing I'll be working on, worst case: Thursday. -- Adrien -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: RFC on Jammy SRU with OpenSSL 3.0 backports for performance
Hi Adrien, Thanks for working on the SRU. FWIW, I tested the packages from your PPA, looks good to me and results match previous tests performed. Let me know if I can do any further testing or assist in any way to expedite the SRU. Regards, Rafael On Sat, Sep 9, 2023 at 1:50 AM Adrien Nader wrote: > Hi Rafael, > > On Fri, Sep 01, 2023, Rafael Lopez wrote: > > Hi Adrien, > > > > I added some results to LP#2009544 for your consideration, including > tests > > on mantic and lunar. > > > > I also tested a couple of good-looking patches in addition to PR#18151 > [1]. > > Summary below, see LP for detailed results and PPAs with patches. > > > > PR#18151 + PR#17881 [2] - up to ~2x improvement over [1] > > PR#18151 + PR#17921 [3] - up to ~2.5x improvement over [1] > > PR#18151 + PR#17881 [2] + PR#17921 [3] - up to ~4x improvement over [1] > > (based on user CPU time) > > > > PR#17921 [3] might be a good candidate to SRU in addition to [1] at some > > stage. It is a moderate change in Jammy as there are a number of > > pre-requisite patches including refactoring. However, all said > > pre-requisites are merged into 3.0 branch upstream and already included > in > > both Lunar and Mantic (you can cleanly cherry pick PR#17921 for those). > The > > performance increase is noticeable. [3] itself is not merged to 3.0 > > upstream even though the pre-reqs are, see [4] for a comment on this. > Seems > > to fit what you described regarding upstream (not) backporting > performance > > patches into 3.0. It is merged into 3.1. > > > > PR#17881 [2] also brings noticeable improvement, though less, and has > > nearly all the pre-reqs in 3.0 (and mantic/lunar). Looks like one [5] is > > missing which was not backported to the 3.0 branch, though it is in 3.1. > > > > I haven't analysed the patches in any detail, wanted to see if they were > > worth the effort performance wise first. > > > > Rafael > > > > [1] https://github.com/openssl/openssl/pull/18151 > > [2] https://github.com/openssl/openssl/pull/17881 > > [3] https://github.com/openssl/openssl/pull/17921 > > [4] > https://github.com/openssl/openssl/pull/20421#issuecomment-1457743122 > > [5] https://github.com/openssl/openssl/pull/6127 > > Thanks a lot for digging into this. > > I thought a bit about the best way forward and I came up with three or > four big steps: > > 1- SRU [1] to Jammy in addition to bugfixes > 2- Include [2] and [3] in the next Ubuntu release (also next LTS) >Per policy these cannot go straight into an SRU I think. Moreover >there are a lot of patches already for Jammy which can make resolving >conflict more difficult and error-prone, and while [3] from >openssl-3.1 applies almost completely cleanly, [2] doesn't. > 3- After OO is released, SRU [2] and [3] to Jammy > 4- Bonus: SRU the same version as OO to Jammy (_maybe_ it will make more >sense to skip step 3 but I can't predict it) > > Unfortunately we have to wait until the OO release for [2] and [3]. > > I've created a PPA for step 1 at > https://launchpad.net/~adrien-n/+archive/ubuntu/openssl-jammy-sru . It > should be ready besides last-minute debian/changelog fixes (trimming an > entry and removing ~ppa*, and triple-checking the version number). > > I need to trigger as many autopkgtests in the PPA as I can fit, which is > something I intend to do this evening and/or over the week-end but I'm > pretty confident. > > -- > Adrien > -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: RFC on Jammy SRU with OpenSSL 3.0 backports for performance
On Mon, 28 Aug 2023 at 21:40, Adrien Nader wrote: > > Hi, > > Finally back after pretty long vacation. :) > > On Mon, Aug 21, 2023, Mauricio Faria de Oliveira wrote: > > Hey Simon, > > > > On Thu, Aug 17, 2023 at 4:42 AM Simon Chopin > > wrote: > > > > > > Hi, > > > > > > Quoting Mauricio Faria de Oliveira (2023-08-15 15:20:50) > > > > Hey, > > > > > > > > I'd like to request input (initially thinking of involved teams: SRU, > > > > Foundations) > > > > on backports of some performance improvement patches to OpenSSL in > > > > Jammy. > > > > (Please feel free to comment and include others as appropriate.) > > > > > > FYI, Adrien is currently offline, he should be back in a couple of > > > weeks. Meanwhile you can get my 2¢ as his backup on openssl-related > > > matters :) > > > > > > > SEG has a customer support ticket about it, which allows us to put in > > > > the work, > > > > but of course, it is OpenSSL, an SRU, thus agreement beforehand is > > > > needed. > > > > > > > > ... > > > > > > > > The context is OpenSSL 3.0 has known, significant performance > > > > regressions [0] > > > > from OpenSSL 1.1.1, which has been addressed / still in-progress > > > > upstream: > > > > 1) some patches in the 3.0 stable branch > > > > 2) some patches in the master branch (ie, not backported to 3.0) > > > > 3) some issues still open > > > > > > > > To offset regression risk, there are benefits; e.g., 1) Performance: > > > > some improvement 2) Security: smaller delta to 3.0 branch (may help > > > > with CVE fix backports) 3) Community: possibly help with mentions of > > > > Ubuntu 22.04 in regressions > > > > > > > > IMHO, backports should be restricted to the 3.0 branch (so not to > > > > defeat 2). > > > > > > Not Security, but I'm OK with backporting patches from the master branch > > > as well, but only if we can get them in Mantic first, and wait for its > > > GA. That would give exposure to a wider range of hardware, shaking off > > > some regressions. > > > > Understood. We'll consider that option, for patches from the master branch. > > > > Hopefully, most patches may be in 3.0 stable (to land before October), > > but nonetheless, it's reassuring to know there's a way forward for > > other patches. > > I've been wanting to _ultimately_ update openssl in our stable releases. > That means updating 22.04 with another openssl LTS. Before anyone > panics: that's a really long term goal. > > Since openssl had relatively small SRUs in the past years, I had planned > to do things very progressively. At the moment we have 3.0.10 in Mantic > (I didn't check who updated it but thanks!) and 3.0.8 in Lunar. No > regression compared to Jammy's 3.0.2. I didn't read about issues on > other distros either. That's encouraging. > > We're talking about two kinds of patches: performance and bug fixes. > Upstream backports bug fixes to the 3.0 release but they don't > necessarily do it for performance fix (they probably typically don't). > There's no "complete" list for such fixes either (there's not even a > definition of which patches would qualify). > While the patches that Mauricio linked to have been backported to 3.0, a > number of the performance fixes have not and it will probably be > difficult to backport them since they can be architectural or locking > changes. > I'm pretty sure the number of performance fixes will only lower and the > situation is transitory but right now there's a demand for several > performance fixes. For instance, it seems Haproxy has encountered > redhibitory performance. > > Below are the "steps" I had thought about before my vacation (hopefully > I remember well :) ). Steps 1 and 2 could be merged or shortened > depending on the SRU team. > > 1- SRU with select patches We have done this in the past. Including backporting lots of optimisations for armhf, s390x, ppc. It is difficult, but possible to land safely. > > There have been a number of issues reported with upstream patches but > they had low impact and/or had easy workarounds. Since there are now > other incentives to do an SRU, I plan to include everything in it. > I think all patches are in Lunar or even previous releases. > > I don't have a list for performance patches however, except for a couple > ones. I don't plan to list them by myself; instead I welcome inputs on > that. Benchmarking is simply too much work, especially considering the > kind of performance regressions from 1.1 to 3.0 that I've seen talked > about on the openssl bug tracker. That would probably be a year-long > full-time job. > > I would like to start preparing this SRU soon and I hope to have it > ready in October (that's my first SRU this large but I think/hope that's > realistic). Since it would be the first one, I plan to stick to changes > that don't require backporting work. For subsequent SRUs we could > include more patches as Simon said (i.e. patches from master provided > they go into non-LTS releases first during their development cycle). >
Re: RFC on Jammy SRU with OpenSSL 3.0 backports for performance
Hi Rafael, On Fri, Sep 01, 2023, Rafael Lopez wrote: > Hi Adrien, > > I added some results to LP#2009544 for your consideration, including tests > on mantic and lunar. > > I also tested a couple of good-looking patches in addition to PR#18151 [1]. > Summary below, see LP for detailed results and PPAs with patches. > > PR#18151 + PR#17881 [2] - up to ~2x improvement over [1] > PR#18151 + PR#17921 [3] - up to ~2.5x improvement over [1] > PR#18151 + PR#17881 [2] + PR#17921 [3] - up to ~4x improvement over [1] > (based on user CPU time) > > PR#17921 [3] might be a good candidate to SRU in addition to [1] at some > stage. It is a moderate change in Jammy as there are a number of > pre-requisite patches including refactoring. However, all said > pre-requisites are merged into 3.0 branch upstream and already included in > both Lunar and Mantic (you can cleanly cherry pick PR#17921 for those). The > performance increase is noticeable. [3] itself is not merged to 3.0 > upstream even though the pre-reqs are, see [4] for a comment on this. Seems > to fit what you described regarding upstream (not) backporting performance > patches into 3.0. It is merged into 3.1. > > PR#17881 [2] also brings noticeable improvement, though less, and has > nearly all the pre-reqs in 3.0 (and mantic/lunar). Looks like one [5] is > missing which was not backported to the 3.0 branch, though it is in 3.1. > > I haven't analysed the patches in any detail, wanted to see if they were > worth the effort performance wise first. > > Rafael > > [1] https://github.com/openssl/openssl/pull/18151 > [2] https://github.com/openssl/openssl/pull/17881 > [3] https://github.com/openssl/openssl/pull/17921 > [4] https://github.com/openssl/openssl/pull/20421#issuecomment-1457743122 > [5] https://github.com/openssl/openssl/pull/6127 Thanks a lot for digging into this. I thought a bit about the best way forward and I came up with three or four big steps: 1- SRU [1] to Jammy in addition to bugfixes 2- Include [2] and [3] in the next Ubuntu release (also next LTS) Per policy these cannot go straight into an SRU I think. Moreover there are a lot of patches already for Jammy which can make resolving conflict more difficult and error-prone, and while [3] from openssl-3.1 applies almost completely cleanly, [2] doesn't. 3- After OO is released, SRU [2] and [3] to Jammy 4- Bonus: SRU the same version as OO to Jammy (_maybe_ it will make more sense to skip step 3 but I can't predict it) Unfortunately we have to wait until the OO release for [2] and [3]. I've created a PPA for step 1 at https://launchpad.net/~adrien-n/+archive/ubuntu/openssl-jammy-sru . It should be ready besides last-minute debian/changelog fixes (trimming an entry and removing ~ppa*, and triple-checking the version number). I need to trigger as many autopkgtests in the PPA as I can fit, which is something I intend to do this evening and/or over the week-end but I'm pretty confident. -- Adrien -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: RFC on Jammy SRU with OpenSSL 3.0 backports for performance
Hi Adrien, I added some results to LP#2009544 for your consideration, including tests on mantic and lunar. I also tested a couple of good-looking patches in addition to PR#18151 [1]. Summary below, see LP for detailed results and PPAs with patches. PR#18151 + PR#17881 [2] - up to ~2x improvement over [1] PR#18151 + PR#17921 [3] - up to ~2.5x improvement over [1] PR#18151 + PR#17881 [2] + PR#17921 [3] - up to ~4x improvement over [1] (based on user CPU time) PR#17921 [3] might be a good candidate to SRU in addition to [1] at some stage. It is a moderate change in Jammy as there are a number of pre-requisite patches including refactoring. However, all said pre-requisites are merged into 3.0 branch upstream and already included in both Lunar and Mantic (you can cleanly cherry pick PR#17921 for those). The performance increase is noticeable. [3] itself is not merged to 3.0 upstream even though the pre-reqs are, see [4] for a comment on this. Seems to fit what you described regarding upstream (not) backporting performance patches into 3.0. It is merged into 3.1. PR#17881 [2] also brings noticeable improvement, though less, and has nearly all the pre-reqs in 3.0 (and mantic/lunar). Looks like one [5] is missing which was not backported to the 3.0 branch, though it is in 3.1. I haven't analysed the patches in any detail, wanted to see if they were worth the effort performance wise first. Rafael [1] https://github.com/openssl/openssl/pull/18151 [2] https://github.com/openssl/openssl/pull/17881 [3] https://github.com/openssl/openssl/pull/17921 [4] https://github.com/openssl/openssl/pull/20421#issuecomment-1457743122 [5] https://github.com/openssl/openssl/pull/6127 On Wed, Aug 30, 2023 at 6:36 AM Adrien Nader wrote: > Hi, > > On Mon, Aug 28, 2023, Adrien Nader wrote: > > 1- SRU with select patches > > > > There have been a number of issues reported with upstream patches but > > they had low impact and/or had easy workarounds. Since there are now > > other incentives to do an SRU, I plan to include everything in it. > > I think all patches are in Lunar or even previous releases. > > > > I don't have a list for performance patches however, except for a couple > > ones. I don't plan to list them by myself; instead I welcome inputs on > > that. Benchmarking is simply too much work, especially considering the > > kind of performance regressions from 1.1 to 3.0 that I've seen talked > > about on the openssl bug tracker. That would probably be a year-long > > full-time job. > > > > I would like to start preparing this SRU soon and I hope to have it > > ready in October (that's my first SRU this large but I think/hope that's > > realistic). Since it would be the first one, I plan to stick to changes > > that don't require backporting work. For subsequent SRUs we could > > include more patches as Simon said (i.e. patches from master provided > > they go into non-LTS releases first during their development cycle). > > My list of patches so far for the SRU can be seen at this address: > > https://bugs.launchpad.net/ubuntu/jammy/+source/openssl > > It's likely that the list will be frozen before the end of next weeek. > > There's only one thing I'd like to also SRU and which is about removing > dead code that also unecessarily sources debconf in the postinst and > which tends to trigger the infamous bug with thousands of duplicates on > launchpad. Unfortunately I'm not sure I can do that without delaying > this SRU and I might have to leave it for a subsequent one. > > -- > Adrien > -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: RFC on Jammy SRU with OpenSSL 3.0 backports for performance
Hi, On Mon, Aug 28, 2023, Adrien Nader wrote: > 1- SRU with select patches > > There have been a number of issues reported with upstream patches but > they had low impact and/or had easy workarounds. Since there are now > other incentives to do an SRU, I plan to include everything in it. > I think all patches are in Lunar or even previous releases. > > I don't have a list for performance patches however, except for a couple > ones. I don't plan to list them by myself; instead I welcome inputs on > that. Benchmarking is simply too much work, especially considering the > kind of performance regressions from 1.1 to 3.0 that I've seen talked > about on the openssl bug tracker. That would probably be a year-long > full-time job. > > I would like to start preparing this SRU soon and I hope to have it > ready in October (that's my first SRU this large but I think/hope that's > realistic). Since it would be the first one, I plan to stick to changes > that don't require backporting work. For subsequent SRUs we could > include more patches as Simon said (i.e. patches from master provided > they go into non-LTS releases first during their development cycle). My list of patches so far for the SRU can be seen at this address: https://bugs.launchpad.net/ubuntu/jammy/+source/openssl It's likely that the list will be frozen before the end of next weeek. There's only one thing I'd like to also SRU and which is about removing dead code that also unecessarily sources debconf in the postinst and which tends to trigger the infamous bug with thousands of duplicates on launchpad. Unfortunately I'm not sure I can do that without delaying this SRU and I might have to leave it for a subsequent one. -- Adrien -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: RFC on Jammy SRU with OpenSSL 3.0 backports for performance
Hi, On Tue, Aug 15, 2023, Mauricio Faria de Oliveira wrote: > Hey, > > I'd like to request input (initially thinking of involved teams: SRU, > Foundations) > on backports of some performance improvement patches to OpenSSL in Jammy. > (Please feel free to comment and include others as appropriate.) > > SEG has a customer support ticket about it, which allows us to put in the > work, > but of course, it is OpenSSL, an SRU, thus agreement beforehand is needed. > > ... > > The context is OpenSSL 3.0 has known, significant performance regressions [0] > from OpenSSL 1.1.1, which has been addressed / still in-progress upstream: > 1) some patches in the 3.0 stable branch > 2) some patches in the master branch (ie, not backported to 3.0) > 3) some issues still open > > To offset regression risk, there are benefits; e.g., > 1) Performance: some improvement > 2) Security: smaller delta to 3.0 branch (may help with CVE fix backports) > 3) Community: possibly help with mentions of Ubuntu 22.04 in regressions > > IMHO, backports should be restricted to the 3.0 branch (so not to defeat 2). > > ... > > There are statements in this thread [1] that suggest we only backport bug > and security fixes, certainly understandable, but considering the numbers, > _perhaps_ we should consider it -- that's why I ask your opinion on this. > > For example, the test in bug [2]: > > 1) Focal, OpenSSL 1.1.1: 1.5 seconds > 2) Jammy, OpenSSL 3.0.2: 30 seconds (20x slower) > 3) Jammy, 7 cherrypicks: 5 seconds (3x slower) (PPA [3]) Can you also test this on Mantic? That would tell us if these patches cover regain as much as is possible or if there's more in subsequent patch releases. Thanks, -- Adrien -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: RFC on Jammy SRU with OpenSSL 3.0 backports for performance
Hi, Finally back after pretty long vacation. :) On Mon, Aug 21, 2023, Mauricio Faria de Oliveira wrote: > Hey Simon, > > On Thu, Aug 17, 2023 at 4:42 AM Simon Chopin > wrote: > > > > Hi, > > > > Quoting Mauricio Faria de Oliveira (2023-08-15 15:20:50) > > > Hey, > > > > > > I'd like to request input (initially thinking of involved teams: SRU, > > > Foundations) > > > on backports of some performance improvement patches to OpenSSL in Jammy. > > > (Please feel free to comment and include others as appropriate.) > > > > FYI, Adrien is currently offline, he should be back in a couple of > > weeks. Meanwhile you can get my 2¢ as his backup on openssl-related > > matters :) > > > > > SEG has a customer support ticket about it, which allows us to put in the > > > work, > > > but of course, it is OpenSSL, an SRU, thus agreement beforehand is needed. > > > > > > ... > > > > > > The context is OpenSSL 3.0 has known, significant performance regressions > > > [0] > > > from OpenSSL 1.1.1, which has been addressed / still in-progress upstream: > > > 1) some patches in the 3.0 stable branch > > > 2) some patches in the master branch (ie, not backported to 3.0) > > > 3) some issues still open > > > > > > To offset regression risk, there are benefits; e.g., 1) Performance: > > > some improvement 2) Security: smaller delta to 3.0 branch (may help > > > with CVE fix backports) 3) Community: possibly help with mentions of > > > Ubuntu 22.04 in regressions > > > > > > IMHO, backports should be restricted to the 3.0 branch (so not to > > > defeat 2). > > > > Not Security, but I'm OK with backporting patches from the master branch > > as well, but only if we can get them in Mantic first, and wait for its > > GA. That would give exposure to a wider range of hardware, shaking off > > some regressions. > > Understood. We'll consider that option, for patches from the master branch. > > Hopefully, most patches may be in 3.0 stable (to land before October), > but nonetheless, it's reassuring to know there's a way forward for > other patches. I've been wanting to _ultimately_ update openssl in our stable releases. That means updating 22.04 with another openssl LTS. Before anyone panics: that's a really long term goal. Since openssl had relatively small SRUs in the past years, I had planned to do things very progressively. At the moment we have 3.0.10 in Mantic (I didn't check who updated it but thanks!) and 3.0.8 in Lunar. No regression compared to Jammy's 3.0.2. I didn't read about issues on other distros either. That's encouraging. We're talking about two kinds of patches: performance and bug fixes. Upstream backports bug fixes to the 3.0 release but they don't necessarily do it for performance fix (they probably typically don't). There's no "complete" list for such fixes either (there's not even a definition of which patches would qualify). While the patches that Mauricio linked to have been backported to 3.0, a number of the performance fixes have not and it will probably be difficult to backport them since they can be architectural or locking changes. I'm pretty sure the number of performance fixes will only lower and the situation is transitory but right now there's a demand for several performance fixes. For instance, it seems Haproxy has encountered redhibitory performance. Below are the "steps" I had thought about before my vacation (hopefully I remember well :) ). Steps 1 and 2 could be merged or shortened depending on the SRU team. 1- SRU with select patches There have been a number of issues reported with upstream patches but they had low impact and/or had easy workarounds. Since there are now other incentives to do an SRU, I plan to include everything in it. I think all patches are in Lunar or even previous releases. I don't have a list for performance patches however, except for a couple ones. I don't plan to list them by myself; instead I welcome inputs on that. Benchmarking is simply too much work, especially considering the kind of performance regressions from 1.1 to 3.0 that I've seen talked about on the openssl bug tracker. That would probably be a year-long full-time job. I would like to start preparing this SRU soon and I hope to have it ready in October (that's my first SRU this large but I think/hope that's realistic). Since it would be the first one, I plan to stick to changes that don't require backporting work. For subsequent SRUs we could include more patches as Simon said (i.e. patches from master provided they go into non-LTS releases first during their development cycle). 2- Micro-release updates If all goes well, I'd like to do micro-release updates for openssl; that would obviously include more changes and possibly include some performance improvements. Note that upstream really wants us to track micro-releases and has unambiguously stated they care a lot about stability: https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/2019970 3- Full release updates As I stated, I'd like
Re: RFC on Jammy SRU with OpenSSL 3.0 backports for performance
Hey Simon, On Thu, Aug 17, 2023 at 4:42 AM Simon Chopin wrote: > > Hi, > > Quoting Mauricio Faria de Oliveira (2023-08-15 15:20:50) > > Hey, > > > > I'd like to request input (initially thinking of involved teams: SRU, > > Foundations) > > on backports of some performance improvement patches to OpenSSL in Jammy. > > (Please feel free to comment and include others as appropriate.) > > FYI, Adrien is currently offline, he should be back in a couple of > weeks. Meanwhile you can get my 2¢ as his backup on openssl-related > matters :) > > > SEG has a customer support ticket about it, which allows us to put in the > > work, > > but of course, it is OpenSSL, an SRU, thus agreement beforehand is needed. > > > > ... > > > > The context is OpenSSL 3.0 has known, significant performance regressions > > [0] > > from OpenSSL 1.1.1, which has been addressed / still in-progress upstream: > > 1) some patches in the 3.0 stable branch > > 2) some patches in the master branch (ie, not backported to 3.0) > > 3) some issues still open > > > > To offset regression risk, there are benefits; e.g., 1) Performance: > > some improvement 2) Security: smaller delta to 3.0 branch (may help > > with CVE fix backports) 3) Community: possibly help with mentions of > > Ubuntu 22.04 in regressions > > > > IMHO, backports should be restricted to the 3.0 branch (so not to > > defeat 2). > > Not Security, but I'm OK with backporting patches from the master branch > as well, but only if we can get them in Mantic first, and wait for its > GA. That would give exposure to a wider range of hardware, shaking off > some regressions. Understood. We'll consider that option, for patches from the master branch. Hopefully, most patches may be in 3.0 stable (to land before October), but nonetheless, it's reassuring to know there's a way forward for other patches. Thanks! -- Mauricio Faria de Oliveira -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: RFC on Jammy SRU with OpenSSL 3.0 backports for performance
Hi Steve, On Tue, Aug 15, 2023 at 4:12 PM Steve Langasek wrote: > > Hi Mauricio, > > On Tue, Aug 15, 2023 at 10:20:50AM -0300, Mauricio Faria de Oliveira wrote: > > Hey, > > > I'd like to request input (initially thinking of involved teams: SRU, > > Foundations) > > on backports of some performance improvement patches to OpenSSL in Jammy. > > (Please feel free to comment and include others as appropriate.) > > With my SRU Team hat on, I would be willing to consider such performance > fixes in an SRU for jammy. Modern systems spend a lot of time in crypto > routines, so performance certainly matters here! And while with glibc we > have to worry about whether performance improvements for one chip are going > to regress them for another, when we're talking about questions of whether > hardware AES support is being exploited by openssl vs not (as I saw was the > case for one of the referenced commits), it's a lot more straightforward. > > We also have a precedent of "hardware enablement" performance improvements > on jammy for a number of server packages, to get them built with compiler > options that significantly improve performance on particular arm64 server > hardware. Thanks for your feedback and insight, it's very helpful. > > Caveat: I have not reviewed the actual commits that you are proposing to > backport, I am only speaking for whether this is ok for SRU in principle. I > would prefer to save a more in-depth review for the SRU unapproved queue. Absolutely. SEG, too, would only work on these patches (and maybe others) more in-depth after some consensus here. Thanks again, Mauricio > > > SEG has a customer support ticket about it, which allows us to put in the > > work, > > but of course, it is OpenSSL, an SRU, thus agreement beforehand is needed. > > > ... > > > The context is OpenSSL 3.0 has known, significant performance regressions > > [0] > > from OpenSSL 1.1.1, which has been addressed / still in-progress upstream: > > 1) some patches in the 3.0 stable branch > > 2) some patches in the master branch (ie, not backported to 3.0) > > 3) some issues still open > > > > To offset regression risk, there are benefits; e.g., > > 1) Performance: some improvement > > 2) Security: smaller delta to 3.0 branch (may help with CVE fix backports) > > 3) Community: possibly help with mentions of Ubuntu 22.04 in regressions > > > > IMHO, backports should be restricted to the 3.0 branch (so not to defeat 2). > > > > ... > > > > There are statements in this thread [1] that suggest we only backport bug > > and security fixes, certainly understandable, but considering the numbers, > > _perhaps_ we should consider it -- that's why I ask your opinion on this. > > > > For example, the test in bug [2]: > > > > 1) Focal, OpenSSL 1.1.1: 1.5 seconds > > 2) Jammy, OpenSSL 3.0.2: 30 seconds (20x slower) > > 3) Jammy, 7 cherrypicks: 5 seconds (3x slower) (PPA [3]) > > > > So, these 7 clean cherry-picks from 3.0 branch [4] help significantly, > > for this customer's test-case, but we (SEG) would only analyze in more > > detail (and possibly propose such changes) based on input from other > > involved teams. (SRU, Foundations, and Security later.) > > > > Please let us know your thoughts. > > > > Thanks! > > Mauricio > > > > [0] https://github.com/openssl/openssl/issues/17627#issuecomment-1060123659 > > [1] > > https://lists.ubuntu.com/archives/ubuntu-devel-discuss/2023-May/019532.html > > [2] https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/2009544 > > [3] https://launchpad.net/~mfo/+archive/ubuntu/lp2009544 > > [4] https://github.com/openssl/openssl/pull/18151 > > > > -- > > Mauricio Faria de Oliveira > > > > -- > > Ubuntu-release mailing list > > Ubuntu-release@lists.ubuntu.com > > Modify settings or unsubscribe at: > > https://lists.ubuntu.com/mailman/listinfo/ubuntu-release > > -- > Steve Langasek Give me a lever long enough and a Free OS > Debian Developer to set it on, and I can move the world. > Ubuntu Developer https://www.debian.org/ > slanga...@ubuntu.com vor...@debian.org -- Mauricio Faria de Oliveira -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: RFC on Jammy SRU with OpenSSL 3.0 backports for performance
Hi, Quoting Mauricio Faria de Oliveira (2023-08-15 15:20:50) > Hey, > > I'd like to request input (initially thinking of involved teams: SRU, > Foundations) > on backports of some performance improvement patches to OpenSSL in Jammy. > (Please feel free to comment and include others as appropriate.) FYI, Adrien is currently offline, he should be back in a couple of weeks. Meanwhile you can get my 2¢ as his backup on openssl-related matters :) > SEG has a customer support ticket about it, which allows us to put in the > work, > but of course, it is OpenSSL, an SRU, thus agreement beforehand is needed. > > ... > > The context is OpenSSL 3.0 has known, significant performance regressions [0] > from OpenSSL 1.1.1, which has been addressed / still in-progress upstream: > 1) some patches in the 3.0 stable branch > 2) some patches in the master branch (ie, not backported to 3.0) > 3) some issues still open > > To offset regression risk, there are benefits; e.g., 1) Performance: > some improvement 2) Security: smaller delta to 3.0 branch (may help > with CVE fix backports) 3) Community: possibly help with mentions of > Ubuntu 22.04 in regressions > > IMHO, backports should be restricted to the 3.0 branch (so not to > defeat 2). Not Security, but I'm OK with backporting patches from the master branch as well, but only if we can get them in Mantic first, and wait for its GA. That would give exposure to a wider range of hardware, shaking off some regressions. -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: RFC on Jammy SRU with OpenSSL 3.0 backports for performance
Hi Mauricio, On Tue, Aug 15, 2023 at 10:20:50AM -0300, Mauricio Faria de Oliveira wrote: > Hey, > I'd like to request input (initially thinking of involved teams: SRU, > Foundations) > on backports of some performance improvement patches to OpenSSL in Jammy. > (Please feel free to comment and include others as appropriate.) With my SRU Team hat on, I would be willing to consider such performance fixes in an SRU for jammy. Modern systems spend a lot of time in crypto routines, so performance certainly matters here! And while with glibc we have to worry about whether performance improvements for one chip are going to regress them for another, when we're talking about questions of whether hardware AES support is being exploited by openssl vs not (as I saw was the case for one of the referenced commits), it's a lot more straightforward. We also have a precedent of "hardware enablement" performance improvements on jammy for a number of server packages, to get them built with compiler options that significantly improve performance on particular arm64 server hardware. Caveat: I have not reviewed the actual commits that you are proposing to backport, I am only speaking for whether this is ok for SRU in principle. I would prefer to save a more in-depth review for the SRU unapproved queue. > SEG has a customer support ticket about it, which allows us to put in the > work, > but of course, it is OpenSSL, an SRU, thus agreement beforehand is needed. > ... > The context is OpenSSL 3.0 has known, significant performance regressions [0] > from OpenSSL 1.1.1, which has been addressed / still in-progress upstream: > 1) some patches in the 3.0 stable branch > 2) some patches in the master branch (ie, not backported to 3.0) > 3) some issues still open > > To offset regression risk, there are benefits; e.g., > 1) Performance: some improvement > 2) Security: smaller delta to 3.0 branch (may help with CVE fix backports) > 3) Community: possibly help with mentions of Ubuntu 22.04 in regressions > > IMHO, backports should be restricted to the 3.0 branch (so not to defeat 2). > > ... > > There are statements in this thread [1] that suggest we only backport bug > and security fixes, certainly understandable, but considering the numbers, > _perhaps_ we should consider it -- that's why I ask your opinion on this. > > For example, the test in bug [2]: > > 1) Focal, OpenSSL 1.1.1: 1.5 seconds > 2) Jammy, OpenSSL 3.0.2: 30 seconds (20x slower) > 3) Jammy, 7 cherrypicks: 5 seconds (3x slower) (PPA [3]) > > So, these 7 clean cherry-picks from 3.0 branch [4] help significantly, > for this customer's test-case, but we (SEG) would only analyze in more > detail (and possibly propose such changes) based on input from other > involved teams. (SRU, Foundations, and Security later.) > > Please let us know your thoughts. > > Thanks! > Mauricio > > [0] https://github.com/openssl/openssl/issues/17627#issuecomment-1060123659 > [1] > https://lists.ubuntu.com/archives/ubuntu-devel-discuss/2023-May/019532.html > [2] https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/2009544 > [3] https://launchpad.net/~mfo/+archive/ubuntu/lp2009544 > [4] https://github.com/openssl/openssl/pull/18151 > > -- > Mauricio Faria de Oliveira > > -- > Ubuntu-release mailing list > Ubuntu-release@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-release -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developer https://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: PGP signature -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
