Re: Unclear Memory Leakage since OpenBSD 7.3 upgrade (nginx and MariaDB; Not consistent)
On 2023-09-25, Tobias Fiebig wrote: > On Mon, 2023-09-25 at 18:15 +0200, Rudolf Leitgeb wrote: >> Either this, or the TLS 1.3 code was always buggy, but now >> it was actually used per default. > Yes, setting up nginx with enabled tlsv1.3 on 7.2 and earlier is also > on the todo. Similarly, disabling tlsv1.3 and forcing tlsv1.3 on > earlier versions. > > Still, the earlier versions i had been running seemed to support > tlsv1.3, at least according to s_client. But the use as default might > change things. The www/nginx port has had working TLS 1.3 since 2020, I've been using it pretty much since that happened. It will be difficult to get reports from others about this on 7.3 because nginx was not updated to 1.24.0 in ports until after 7.3. FWIW I'm using 1.24.0 on -current from Jul 31st with no issues. I would try updating the problematic system to new -current (7.4-beta) with snapshot packages rather than self-built nginx. If that fixes it anyway then good. If not then we have a data point more likely to be reproducible by others, and maybe a chance of fixing before 7.4 is totally locked down. > pdns itself is not leaking, the memory is hogged by mariadb. But (given > everything runs via unix sockets) i am not using TLS in that stack at > all. This is was initially nudged me a bit towards other functions that > might be used from libressl (sha* or something used in auth maybe?). My reasonably busy traccar/mariadb 7.3 box has stable memory use from mariadb (sitting around 13G, with innodb_buffer_pool_size=12G). ... "memory leak" doesn't give much of an idea of what's going on, some data would be good. KB, MB, GB? per minute? hour? if you fire a bunch of requests at it, does that increase the rate or does it not matter? >> > > But yes, getting a specific commit there will be helpful. >> > Sadly it turns out that it is the commit i feared it would be: >> > >> > > commit 7b24b93d67daa9c16d665129fd5d3e7dbc583e4f >> > > Author: Maxim Dounin >> > > Date: Fri Mar 24 02:57:43 2023 +0300 >> > > >> > > SSL: enabled TLSv1.3 by default. >> > >> > Feared, because it basically puts me back to start w.r.t. what the >> > root >> > cause might be; Could be anything that happened to TLSv1.3 code in >> > either LibreSSL or Nginx. -- Please keep replies on the mailing list.
Re: Unclear Memory Leakage since OpenBSD 7.3 upgrade (nginx and MariaDB; Not consistent)
On Mon, 2023-09-25 at 18:15 +0200, Rudolf Leitgeb wrote: > Either this, or the TLS 1.3 code was always buggy, but now > it was actually used per default. Yes, setting up nginx with enabled tlsv1.3 on 7.2 and earlier is also on the todo. Similarly, disabling tlsv1.3 and forcing tlsv1.3 on earlier versions. Still, the earlier versions i had been running seemed to support tlsv1.3, at least according to s_client. But the use as default might change things. > Question: is there a similar > commit in your DNS server? Do you use this DNS server with > anything like TLS? pdns itself is not leaking, the memory is hogged by mariadb. But (given everything runs via unix sockets) i am not using TLS in that stack at all. This is was initially nudged me a bit towards other functions that might be used from libressl (sha* or something used in auth maybe?). But this will need some more test-setups to run for some time; I will be able to setup automation for that in the coming weeks. With best regards, Tobias > On Sun, 2023-09-24 at 21:31 +0200, Tobias Fiebig wrote: > > > > > But yes, getting a specific commit there will be helpful. > > Sadly it turns out that it is the commit i feared it would be: > > > > > commit 7b24b93d67daa9c16d665129fd5d3e7dbc583e4f > > > Author: Maxim Dounin > > > Date: Fri Mar 24 02:57:43 2023 +0300 > > > > > > SSL: enabled TLSv1.3 by default. > > > > Feared, because it basically puts me back to start w.r.t. what the > > root > > cause might be; Could be anything that happened to TLSv1.3 code in > > either LibreSSL or Nginx. > -- Dr.-Ing. Tobias Fiebig T +31 616 80 98 99 M tob...@fiebig.nl
Re: Unclear Memory Leakage since OpenBSD 7.3 upgrade (nginx and MariaDB; Not consistent)
Either this, or the TLS 1.3 code was always buggy, but now it was actually used per default. Question: is there a similar commit in your DNS server? Do you use this DNS server with anything like TLS? On Sun, 2023-09-24 at 21:31 +0200, Tobias Fiebig wrote: > > > But yes, getting a specific commit there will be helpful. > Sadly it turns out that it is the commit i feared it would be: > > > commit 7b24b93d67daa9c16d665129fd5d3e7dbc583e4f > > Author: Maxim Dounin > > Date: Fri Mar 24 02:57:43 2023 +0300 > > > > SSL: enabled TLSv1.3 by default. > > Feared, because it basically puts me back to start w.r.t. what the > root > cause might be; Could be anything that happened to TLSv1.3 code in > either LibreSSL or Nginx.
Re: Unclear Memory Leakage since OpenBSD 7.3 upgrade (nginx and MariaDB; Not consistent)
> But yes, getting a specific commit there will be helpful. Sadly it turns out that it is the commit i feared it would be: > commit 7b24b93d67daa9c16d665129fd5d3e7dbc583e4f > Author: Maxim Dounin > Date: Fri Mar 24 02:57:43 2023 +0300 > > SSL: enabled TLSv1.3 by default. Feared, because it basically puts me back to start w.r.t. what the root cause might be; Could be anything that happened to TLSv1.3 code in either LibreSSL or Nginx. I guess the next step is going through all commits of libressl between what is in 7.2 and 7.3. With best regards, Tobias
Re: Unclear Memory Leakage since OpenBSD 7.3 upgrade (nginx and MariaDB; Not consistent)
My dev environment on 7.3 is completely different but a little more standard I share about my nginx: wiz$ pkg_info nginx Information for inst:nginx-1.22.0p0 Comment: robust and small HTTP server and mail proxy server Description: This is the stable branch of nginx, as distributed by nginx.org. nginx provides FastCGI and reverse HTTP proxying, with or without caching, including simple load balancing and fault tolerance. It has a modular architecture. Filters include gzipping, byte ranges, chunked responses, and SSI. Multiple SSI inclusions within a single page can be processed in parallel if they are handled by FastCGI or proxied servers. SSL and TLS SNI are supported. Maintainer: Robert Nagy WWW: https://nginx.org/ wiz$ ldd /usr/local/sbin/nginx /usr/local/sbin/nginx: StartEnd Type Open Ref GrpRef Name 0f5043481000 0f50435b4000 exe 10 0 /usr/local/sbin/nginx 0f52fcc29000 0f52fcc6f000 rlib 01 0 /usr/local/lib/libpcre.so.3.0 0f52dabca000 0f52dac3a000 rlib 01 0 /usr/lib/libssl.so.53.2 0f52b0c0e000 0f52b0e69000 rlib 02 0 /usr/lib/libcrypto.so.50.2 0f52e1cf9000 0f52e1d17000 rlib 01 0 /usr/lib/libz.so.7.0 0f531de9f000 0f531df95000 rlib 01 0 /usr/lib/libc.so.97.0 0f53165dc000 0f53165dc000 ld.so 01 0 /usr/libexec/ld.so I also own MariaDB, 10.9.4. I underline that on my system "pkg_info mysql" or "pkg_info mariadb" return blank and back to cursor. -- Daniele Bonini Tobias Fiebig wrote: > Moin, > > I have been dealing with memory-leaks on a host running an nginx > reverse proxy for some time. The host had been running 7.2 with nginx > 1.23.1 (self compiled as i need some features not in the package) > until May, which was fine. > > After upgrading to 7.3 and nginx-1.24.0, i started to see heavy memory > leakage over time. I initially attributed this to nginx, and solved > the issue by ignoring it/throwing a bit more memory at the box for > some time. However, I started debugging it now and could trace it to > some commit between nginx 1.23.3 and 1.23.4; Currently, I am going > through all commits to see with which version it first appears. > > However, clicking around this morning, i noticed that my primary NS > shows a similar memory leakage for mysql/mariadb (runs powerdns with a > mysql backend, both from packages) since the upgrade to 7.3 in May as > well. One further host seems to show a significantly higher use of > inactive memory since 7.3. I found one more host with mariadb that now > shows a higher utilization of inactive memory (gitea+maria); However, > other maria+X instances on 7.3 run fine. > > On the PowerDNS+Mysql host I also see gradually increasing CPU load > along with the memory leakage. > > Restarting mysql and/or nginx resolves the issue for some time. I also > saw the issue on a host running maria+nextcloud, where i fixed it by > switching to postgres. Maria is still running there, but not receiving > any requests, and the memory leak is gone. > > Images: > > https://rincewind.home.aperture-labs.org/~tfiebig/pdns-maria-cpu-year.png > https://rincewind.home.aperture-labs.org/~tfiebig/pdns-maria-memory-year.png > > https://rincewind.home.aperture-labs.org/~tfiebig/nginx-rev-proxy-memory-year.png > > https://rincewind.home.aperture-labs.org/~tfiebig/gitea-maria-memory-year.png > > https://rincewind.home.aperture-labs.org/~tfiebig/nextcloud-maria-switched-to-postgres-memory-year.png > > All hosts are VMs on Linux/KVM hosts (proxmox), with a generic > qemu-kvm processor as CPU type, running 7.3. > > Hence, I am no longer 100% convinced that this is an nginx problem. To > help circling this in a bit closer (and maybe get it reproducible in a > first step): > > Did anyone else experience memory leakage on openbsd with mariadb or a > self-build >=nginx-1.23.4 (or other applications) since the upgrade to > 7.3? > > With best regards, > Tobias >
Re: Unclear Memory Leakage since OpenBSD 7.3 upgrade (nginx and MariaDB; Not consistent)
On Sun, 2023-09-24 at 13:07 +0200, Rudolf Leitgeb wrote: > This libpcre2 library seems to be the only one, which is not > used all over the place. The library itself may not even be buggy, it > may just return something, which the new versions of the caller can't > handle, or it may be unhappy with something the new callers send. > > Still: if you can tie this memory/cpu leak to the interaction between > application and that library, it should be much easier to identify > the offending commit in nginx and all the other affected programs. Yes, _if_ it is that library. I would not fully set my sights on that yet. Nginx has the benefit of having a cut somewhere that triggers this; Hopefully the tests are done soon (i went for brute force instead of binary search; takes four days but requires no interaction). But yes, getting a specific commit there will be helpful. > I just saw, that the version 8 of libpcre2 seems to be quite a bit > behind the current version: > https://github.com/PCRE2Project/pcre2/releases > > Is this intentional? I am using what comes from the packages/ports. So the intentions on that are with the maintainers there. With best regards, Tobias
Re: Unclear Memory Leakage since OpenBSD 7.3 upgrade (nginx and MariaDB; Not consistent)
This libpcre2 library seems to be the only one, which is not used all over the place. The library itself may not even be buggy, it may just return something, which the new versions of the caller can't handle, or it may be unhappy with something the new callers send. Still: if you can tie this memory/cpu leak to the interaction between application and that library, it should be much easier to identify the offending commit in nginx and all the other affected programs. I just saw, that the version 8 of libpcre2 seems to be quite a bit behind the current version: https://github.com/PCRE2Project/pcre2/releases Is this intentional? On Sun, 2023-09-24 at 12:59 +0200, Tobias Fiebig wrote: > They do, but nothing special. The common set between nginx and mysqld > is: > > /usr/local/lib/libpcre2-8.so.0.6 > /usr/lib/libssl.so.53.2 > /usr/lib/libcrypto.so.50.2 > /usr/lib/libz.so.7.0 > /usr/lib/libc.so.97.0 > /usr/libexec/ld.so > > However, an affected nginx (1.24.0) does not differ from an > unaffected > nginx (1.23.1). > > I somewhat hope that my testing through nginx commits will yield what > calls are actually involved, and may allow me to see whether mysql is > pushed into similar codepaths on affected systems (and not on > unaffected ones), likely giving a better hint as to where the issue > is. > > With best regards, > Tobias > > On Sun, 2023-09-24 at 12:53 +0200, Rudolf Leitgeb wrote: > > Do the affected programs use the same libraries? > > > > On Sun, 2023-09-24 at 09:32 +0200, Tobias Fiebig wrote: > > > After upgrading to 7.3 and nginx-1.24.0, i started to see heavy > > > memory > > > leakage over time. I initially attributed this to nginx, and > > > solved > > > the > > > issue by ignoring it/throwing a bit more memory at the box for > > > some > > > time. However, I started debugging it now and could trace it to > > > some > > > commit between nginx 1.23.3 and 1.23.4; Currently, I am going > > > through > > > all commits to see with which version it first appears. > > > > > > However, clicking around this morning, i noticed that my primary > > > NS > > > shows a similar memory leakage for mysql/mariadb (runs powerdns > > > with > > > a > > > mysql backend, both from packages) since the upgrade to 7.3 in > > > May > > > as > > > well. One further host seems to show a significantly higher use > > > of > > > inactive memory since 7.3. I found one more host with mariadb > > > that > > > now > > > shows a higher utilization of inactive memory (gitea+maria); > > > However, > > > other maria+X instances on 7.3 run fine. > > >
Re: Unclear Memory Leakage since OpenBSD 7.3 upgrade (nginx and MariaDB; Not consistent)
They do, but nothing special. The common set between nginx and mysqld is: /usr/local/lib/libpcre2-8.so.0.6 /usr/lib/libssl.so.53.2 /usr/lib/libcrypto.so.50.2 /usr/lib/libz.so.7.0 /usr/lib/libc.so.97.0 /usr/libexec/ld.so However, an affected nginx (1.24.0) does not differ from an unaffected nginx (1.23.1). I somewhat hope that my testing through nginx commits will yield what calls are actually involved, and may allow me to see whether mysql is pushed into similar codepaths on affected systems (and not on unaffected ones), likely giving a better hint as to where the issue is. With best regards, Tobias On Sun, 2023-09-24 at 12:53 +0200, Rudolf Leitgeb wrote: > Do the affected programs use the same libraries? > > On Sun, 2023-09-24 at 09:32 +0200, Tobias Fiebig wrote: > > After upgrading to 7.3 and nginx-1.24.0, i started to see heavy > > memory > > leakage over time. I initially attributed this to nginx, and solved > > the > > issue by ignoring it/throwing a bit more memory at the box for some > > time. However, I started debugging it now and could trace it to > > some > > commit between nginx 1.23.3 and 1.23.4; Currently, I am going > > through > > all commits to see with which version it first appears. > > > > However, clicking around this morning, i noticed that my primary NS > > shows a similar memory leakage for mysql/mariadb (runs powerdns > > with > > a > > mysql backend, both from packages) since the upgrade to 7.3 in May > > as > > well. One further host seems to show a significantly higher use of > > inactive memory since 7.3. I found one more host with mariadb that > > now > > shows a higher utilization of inactive memory (gitea+maria); > > However, > > other maria+X instances on 7.3 run fine. >
Re: Unclear Memory Leakage since OpenBSD 7.3 upgrade (nginx and MariaDB; Not consistent)
Do the affected programs use the same libraries? On Sun, 2023-09-24 at 09:32 +0200, Tobias Fiebig wrote: > After upgrading to 7.3 and nginx-1.24.0, i started to see heavy > memory > leakage over time. I initially attributed this to nginx, and solved > the > issue by ignoring it/throwing a bit more memory at the box for some > time. However, I started debugging it now and could trace it to some > commit between nginx 1.23.3 and 1.23.4; Currently, I am going through > all commits to see with which version it first appears. > > However, clicking around this morning, i noticed that my primary NS > shows a similar memory leakage for mysql/mariadb (runs powerdns with > a > mysql backend, both from packages) since the upgrade to 7.3 in May as > well. One further host seems to show a significantly higher use of > inactive memory since 7.3. I found one more host with mariadb that > now > shows a higher utilization of inactive memory (gitea+maria); However, > other maria+X instances on 7.3 run fine.
