Public bug reported: Source package: nginx
This is a confirmed regression introduced by the security update in USN-8398-1, which upgraded nginx to 1.24.0-2ubuntu7.10 on Ubuntu 24.04 Noble. nginx workers segfault continuously on startup when no third-party modules are loaded. This is related to but distinct from Bug #2155992 (which requires the headers-more module). https://ubuntu.com/security/notices/USN-8398-1 ### 1) Ubuntu release ``` Description: Ubuntu 24.04 LTS Release: 24.04 Codename: noble ``` ### 2) Package versions ``` nginx 1.24.0-2ubuntu7.10 (noble-security / noble-updates) nginx-common 1.24.0-2ubuntu7.10 (noble-security / noble-updates) ``` No `nginx-extras`, no `libnginx-mod-http-headers-more-filter`, and no dynamic modules loaded via `load_module` anywhere in the nginx configuration. ### 3) What I expected to happen nginx should continue to run normally after the security upgrade delivered by USN-8398-1. ### 4) What happened instead All nginx worker processes segfault immediately on startup and continue to do so continuously. The master process keeps respawning workers, which crash within seconds. nginx is entirely non-functional. This behaviour is present across multiple reboots since the upgrade was applied. Kernel log entries (from `dmesg`): ``` [28.888478] nginx[4660]: segfault at 0 ip 000056e869f68623 sp 00007ffc4bfeb868 error 4 in nginx[56e869f63000+d3000] likely on CPU 5 (core 5, socket 0) [28.890876] traps: nginx[4663] general protection fault ip:56e869f68623 sp:7ffc4bfeb868 error:0 in nginx[56e869f63000+d3000] [30.065006] nginx[4664]: segfault at f ip 000056e869f68623 sp 00007ffc4bfeb8a8 error 4 in nginx[56e869f63000+d3000] likely on CPU 7 (core 1, socket 0) [30.111606] nginx[4659]: segfault at f ip 000056e869f68623 sp 00007ffc4bfeb8a8 error 4 in nginx[56e869f63000+d3000] likely on CPU 0 (core 0, socket 0) [31.220777] nginx[4665]: segfault at f ip 000056e869f68623 sp 00007ffc4bfeb8a8 error 4 in nginx[56e869f63000+d3000] likely on CPU 2 (core 2, socket 0) [31.382838] nginx[4670]: segfault at f ip 000056e869f68623 sp 00007ffc4bfeb8a8 error 4 in nginx[56e869f63000+d3000] likely on CPU 0 (core 0, socket 0) ``` The crash offset `nginx[..+d3000]` is **identical across all workers and all reboots**. `addr2line` identifies the crashing function: ``` $ addr2line -e /usr/sbin/nginx -f 0xd3623 ngx_http_v2_string_encode ??:? ``` systemd journal entries showing heap corruption messages: ``` Jun 09 09:21:16 [redacted] nginx[469057]: double free or corruption (!prev) Jun 09 09:21:16 [redacted] nginx[469039]: worker process 469057 exited on signal 6 (core dumped) Jun 09 09:21:16 [redacted] nginx[469048]: corrupted size vs. prev_size Jun 09 09:21:16 [redacted] nginx[469039]: worker process 469048 exited on signal 6 (core dumped) Jun 09 09:21:17 [redacted] nginx[469039]: worker process 469050 exited on signal 11 (core dumped) ``` ### 5) nginx -V output ``` nginx version: nginx/1.24.0 (Ubuntu) built with OpenSSL 3.0.13 30 Jan 2024 TLS SNI support enabled configure arguments: --with-cc-opt='-g -O2 -fno-omit-frame-pointer ...' \ --with-http_ssl_module --with-http_v2_module \ [standard Ubuntu build — no third-party --add-module flags] ``` Full `nginx -V` output available on request. ### 6) nginx -t output Config test passes cleanly: ``` nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful ``` (Some unrelated `protocol options redefined` warnings on other vhosts, not related to the crashing behaviour.) ### 7) Environment - Kernel: `6.8.0-124-generic` - OpenSSL: `3.0.13` — matches the version nginx was built against exactly - No snap updates relevant to nginx or OpenSSL on the affected day - No dynamic modules loaded (`grep -r "load_module" /etc/nginx/` returns nothing) ### 8) Workaround Downgrading to `1.24.0-2ubuntu7` from `noble/main` resolves the issue immediately: ```bash sudo apt install nginx=1.24.0-2ubuntu7 nginx-common=1.24.0-2ubuntu7 sudo apt-mark hold nginx nginx-common ``` ### 9) Relationship to existing bugs This is **distinct from Bug #2155992 / #2155999**, which require `libnginx-mod-http-headers-more-filter` to be loaded. This system has no headers-more module installed or loaded at all. The crash address resolves to `ngx_http_v2_string_encode`, consistent with the HTTP/2 code path touched by USN-8398-1, suggesting the regression is broader than the headers-more case. Marking as related-to (not duplicate-of) Bug #2155992. ** Affects: nginx (Ubuntu) Importance: Undecided Status: New ** Description changed: Source package: nginx This is a confirmed regression introduced by the security update in USN-8398-1, which upgraded nginx to 1.24.0-2ubuntu7.10 on Ubuntu 24.04 Noble. nginx workers segfault continuously on startup when no third-party modules are loaded. This is related to but distinct from Bug #2155992 (which requires the headers-more module). + + https://ubuntu.com/security/notices/USN-8398-1 ### 1) Ubuntu release ``` Description: Ubuntu 24.04 LTS Release: 24.04 Codename: noble ``` ### 2) Package versions ``` nginx 1.24.0-2ubuntu7.10 (noble-security / noble-updates) nginx-common 1.24.0-2ubuntu7.10 (noble-security / noble-updates) ``` No `nginx-extras`, no `libnginx-mod-http-headers-more-filter`, and no dynamic modules loaded via `load_module` anywhere in the nginx configuration. ### 3) What I expected to happen nginx should continue to run normally after the security upgrade delivered by USN-8398-1. ### 4) What happened instead All nginx worker processes segfault immediately on startup and continue to do so continuously. The master process keeps respawning workers, which crash within seconds. nginx is entirely non-functional. This behaviour is present across multiple reboots since the upgrade was applied. Kernel log entries (from `dmesg`): ``` [28.888478] nginx[4660]: segfault at 0 ip 000056e869f68623 sp 00007ffc4bfeb868 error 4 in nginx[56e869f63000+d3000] likely on CPU 5 (core 5, socket 0) [28.890876] traps: nginx[4663] general protection fault ip:56e869f68623 sp:7ffc4bfeb868 error:0 in nginx[56e869f63000+d3000] [30.065006] nginx[4664]: segfault at f ip 000056e869f68623 sp 00007ffc4bfeb8a8 error 4 in nginx[56e869f63000+d3000] likely on CPU 7 (core 1, socket 0) [30.111606] nginx[4659]: segfault at f ip 000056e869f68623 sp 00007ffc4bfeb8a8 error 4 in nginx[56e869f63000+d3000] likely on CPU 0 (core 0, socket 0) [31.220777] nginx[4665]: segfault at f ip 000056e869f68623 sp 00007ffc4bfeb8a8 error 4 in nginx[56e869f63000+d3000] likely on CPU 2 (core 2, socket 0) [31.382838] nginx[4670]: segfault at f ip 000056e869f68623 sp 00007ffc4bfeb8a8 error 4 in nginx[56e869f63000+d3000] likely on CPU 0 (core 0, socket 0) ``` The crash offset `nginx[..+d3000]` is **identical across all workers and all reboots**. `addr2line` identifies the crashing function: ``` $ addr2line -e /usr/sbin/nginx -f 0xd3623 ngx_http_v2_string_encode ??:? ``` systemd journal entries showing heap corruption messages: ``` Jun 09 09:21:16 [redacted] nginx[469057]: double free or corruption (!prev) Jun 09 09:21:16 [redacted] nginx[469039]: worker process 469057 exited on signal 6 (core dumped) Jun 09 09:21:16 [redacted] nginx[469048]: corrupted size vs. prev_size Jun 09 09:21:16 [redacted] nginx[469039]: worker process 469048 exited on signal 6 (core dumped) Jun 09 09:21:17 [redacted] nginx[469039]: worker process 469050 exited on signal 11 (core dumped) ``` ### 5) nginx -V output ``` nginx version: nginx/1.24.0 (Ubuntu) built with OpenSSL 3.0.13 30 Jan 2024 TLS SNI support enabled configure arguments: --with-cc-opt='-g -O2 -fno-omit-frame-pointer ...' \ - --with-http_ssl_module --with-http_v2_module \ - [standard Ubuntu build — no third-party --add-module flags] + --with-http_ssl_module --with-http_v2_module \ + [standard Ubuntu build — no third-party --add-module flags] ``` Full `nginx -V` output available on request. ### 6) nginx -t output Config test passes cleanly: ``` nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful ``` (Some unrelated `protocol options redefined` warnings on other vhosts, not related to the crashing behaviour.) ### 7) Environment - Kernel: `6.8.0-124-generic` - OpenSSL: `3.0.13` — matches the version nginx was built against exactly - No snap updates relevant to nginx or OpenSSL on the affected day - No dynamic modules loaded (`grep -r "load_module" /etc/nginx/` returns nothing) ### 8) Workaround Downgrading to `1.24.0-2ubuntu7` from `noble/main` resolves the issue immediately: ```bash sudo apt install nginx=1.24.0-2ubuntu7 nginx-common=1.24.0-2ubuntu7 sudo apt-mark hold nginx nginx-common ``` ### 9) Relationship to existing bugs This is **distinct from Bug #2155992 / #2155999**, which require `libnginx-mod-http-headers-more-filter` to be loaded. This system has no headers-more module installed or loaded at all. The crash address resolves to `ngx_http_v2_string_encode`, consistent with the HTTP/2 code path touched by USN-8398-1, suggesting the regression is broader than the headers-more case. Marking as related-to (not duplicate-of) Bug #2155992. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2156028 Title: nginx workers segfault after upgrade to 1.24.0-2ubuntu7.10 -- no headers-more module loaded To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/2156028/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
