Public bug reported:
lsb_release -rd
Description: Ubuntu 26.04 LTS
Release: 26.04
After Ubuntu upgraded nginx from 1.28.3-2ubuntu1.2 to 1.28.3-2ubuntu1.3,
nginx workers began segfaulting immediately on traffic to a vhost that
uses the dynamic njs HTTP module.
The affected host uses `ngx_http_js_module.so` via `libnginx-mod-http-js` and
has njs handlers/filters that read request headers through `r.headersIn`,
including `r.headersIn.host` and
`r.headersIn.Origin`.
Timeline:
- 2026-06-09 06:02:54 UTC: unattended-upgrade upgraded nginx from
`1.28.3-2ubuntu1.2` to `1.28.3-2ubuntu1.3`.
- 2026-06-09 06:02:55 UTC: nginx was stopped/started by the package upgrade.
- 2026-06-09 06:03:18 UTC: first nginx worker segfault appeared.
- After this, requests to the affected vhost frequently failed through
Cloudflare as HTTP 520 or redirect failures because nginx workers were crashing.
- Downgrading nginx-related packages back to `1.28.3-2ubuntu1.2` stopped the
issue.
Installed package versions at the time:
- `nginx`: `1.28.3-2ubuntu1.3`
- `nginx-common`: `1.28.3-2ubuntu1.3`
- `libnginx-mod-stream`: `1.28.3-2ubuntu1.3`
- `libnginx-mod-http-js`: `0.9.4-1build2`
Relevant package/ABI observation:
- `nginx 1.28.3-2ubuntu1.3` still provides `nginx-abi-1.28.3-1`.
- `libnginx-mod-http-js 0.9.4-1build2` depends on `nginx-abi-1.28.3-1`.
- The njs module package did not appear to be rebuilt specifically for `nginx
1.28.3-2ubuntu1.3`.
Crash log evidence:
Kernel logs showed nginx worker segfaults in the njs HTTP module immediately
after the package upgrade:
nginx[97846]: segfault at 7 ip 000071bf3e1132f5 sp 00007fff9aa44620 error 4
in ngx_http_js_module.so[272f5,71bf3e10b000+ac000]
nginx[97848]: segfault at 7 ip 000071bf3e1132f5 sp 00007fff9aa44620 error 4
in ngx_http_js_module.so[272f5,71bf3e10b000+ac000]
nginx[97845]: segfault at 7 ip 000071bf3e1132f5 sp 00007fff9aa44620 error 4
in ngx_http_js_module.so[272f5,71bf3e10b000+ac000]
The backend service behind the affected vhost remained healthy when accessed
directly, so the failures were not caused by the backend application. The
crashes were in nginx workers,
specifically inside `ngx_http_js_module.so`.
Why this looks like an ABI/layout mismatch:
The `1.28.3-2ubuntu1.3` changelog says the update backports the `max_headers`
directive for CVE-2026-49975.
The backported patch adds a new field to `ngx_http_headers_in_t`:
ngx_list_t headers;
ngx_uint_t count;
ngx_table_elt_t *host;
Previously, `host` followed `headers` directly. Inserting `count`
before `host` changes the layout/offsets of `ngx_http_headers_in_t`.
A dynamic module built against the previous headers may still read the old
offset for `headers_in.host`. With the new nginx core, that old offset now
points at the new integer
`headers_in.count` field instead of the `host` pointer.
This matches the observed crash very closely. The segfault address was `7`. A
normal browser/proxy request can easily have around 7 request headers. If the
old njs module interprets
`headers_in.count == 7` as a pointer while servicing `r.headersIn.host`, it
would dereference address `0x7`, producing exactly this kind of crash:
segfault at 7 ... in ngx_http_js_module.so
Why only one vhost was visibly affected:
The failing vhost is the only high-traffic public vhost on this host that
uses njs request processing on normal traffic. Its nginx config uses directives
such as:
The middleware reads request headers through `r.headersIn`, including
`r.headersIn.host`. Other vhosts either do not use `ngx_http_js_module` or do
not exercise `r.headersIn` in the same way,
so they did not visibly trigger the crash.
Resolution/workaround:
Downgrading nginx packages back to `1.28.3-2ubuntu1.2` stopped the segfaults
and restored the affected vhost.
After the downgrade, the affected vhost returned normal HTTP 200
responses again, and no new nginx worker segfaults were observed in the
immediate post-restart logs.
The most likely fix is to rebuild `libnginx-mod-http-js` /
`ngx_http_js_module.so` against the nginx headers from `1.28.3-2ubuntu1.3`
The update appears to add `headers_in.count` into `ngx_http_headers_in_t`,
changing the offsets of later fields such as `headers_in.host`. The installed
njs HTTP module package, `libnginx-
mod-http-js 0.9.4-1build2`, appears to have been built before this nginx
update and still depends on/provides compatibility through
`nginx-abi-1.28.3-1`..
Expected behavior:
Upgrading from `nginx 1.28.3-2ubuntu1.2` to `1.28.3-2ubuntu1.3` should not
cause `ngx_http_js_module.so` worker segfaults when existing njs configs read
request headers through `r.headersIn`.
Actual behavior:
Immediately after the upgrade, nginx workers began segfaulting in
`ngx_http_js_module.so` on traffic to a vhost using njs request-header access.
Downgrading nginx back to `1.28.3-2ubuntu1.2`
resolved the issue.
Related upstream context:
The `max_headers` feature was introduced upstream in nginx after 1.28 and is
present in newer upstream versions where nginx core and modules are normally
built against matching headers. This
report is specifically about the Ubuntu 1.28.3 security backport appearing to
change a dynamic-module-visible struct layout while retaining the same
`nginx-abi-1.28.3-1` compatibility marker
and without rebuilding the njs HTTP module package.
** Affects: nginx (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2156284
Title:
nginx 1.28.3-2ubuntu1.3 backport changes ngx_http_headers_in_t layout
without ABI bump or rebuilding libnginx-mod-http-js, causing
ngx_http_js_module.so worker segfaults
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/2156284/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
