Public bug reported:
A Jammy 22.04 container on the same FIPS host isn't affected so this
seems to be isolated to Noble (onwards?)
On a host with `/proc/sys/crypto/fips_enabled=1`, `systemd-resolved`
fails to start inside an unprivileged LXD container running Noble 24.04.
The service exits 1 and the container is left with no working DNS as
`/etc/resolv.conf` is left still pointing at the `127.0.0.53` stub.
`/proc/sys/crypto/fips_enabled` isn't namespaced so the host's FIPS flag
is visible inside the container even though the container has the normal
non-FIPS crypto libraries and has no FIPS provider installed.
Unfortunately the error that bubbles up is a bit misleading as well, it
says `Cannot allocate memory`/`ENOMEM` (see "Analysis" below for more
details)
## Reproducer
* Boot up an Ubuntu Pro FIPS-Updates 24.04 (noble) instance and confirm that
FIPS is active
```
$ uname -r
6.8.0-1070-gcp-fips
$ cat /proc/sys/crypto/fips_enabled
1
```
* Install and set up LXD
```
$ sudo snap install lxd
$ sudo lxd init --auto --storage-backend dir
```
* Launch a *noble* container and check `resolved`
```
$ lxc launch ubuntu-daily:noble nobletest
<wait a bit>
$ lxc exec nobletest -- cat /proc/sys/crypto/fips_enabled
$ lxc exec nobletest -- systemctl is-active systemd-resolved
$ lxc exec nobletest -- getent hosts archive.ubuntu.com
```
* Also launch a *jammy* container on the same host to see the difference
```
$ lxc launch ubuntu-daily:jammy jammytest
<wait a bit>
$ lxc exec jammytest -- systemctl is-active systemd-resolved
$ lxc exec jammytest -- getent hosts archive.ubuntu.com
```
### Expected result
`systemd-resolved` starts in both containers and name resolution works
(same as a non-FIPS host)
#### Actual result
The *noble* container gets `fips_enabled=1` leaked in from the host and
`resolved` is dead:
```
$ lxc exec nobletest -- cat /proc/sys/crypto/fips_enabled
1
$ lxc exec nobletest -- systemctl status systemd-resolved --no-pager
x systemd-resolved.service - Network Name Resolution
Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; enabled;
preset: enabled)
Active: failed (Result: exit-code) since Mon 2026-10-05 17:43:20 UTC
Process: 476 ExecStart=/usr/lib/systemd/systemd-resolved (code=exited,
status=1/FAILURE)
Main PID: 476 (code=exited, status=1/FAILURE)
Error: 12 (Cannot allocate memory)
$ lxc exec nobletest -- journalctl -u systemd-resolved --no-pager | tail
systemd-resolved[466]: Positive Trust Anchors:
systemd-resolved[466]: . IN DS 20326 8 2
e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
systemd-resolved[466]: Negative trust anchors: home.arpa 10.in-addr.arpa ...
systemd-resolved[466]: Could not create manager: Cannot allocate memory
systemd[1]: systemd-resolved.service: Main process exited, code=exited,
status=1/FAILURE
systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is
at 5.
systemd[1]: systemd-resolved.service: Start request repeated too quickly.
```
And then all DNS resolution in the container fails:
```
$ lxc exec nobletest -- getent hosts archive.ubuntu.com
(no output)
$ lxc exec nobletest -- apt-get update
Err:1 http://archive.ubuntu.com/ubuntu noble InRelease
Temporary failure resolving 'archive.ubuntu.com'
```
But the Jammy container on the *same host* is fine, the FIPS flag leaks in in
the same way but `resolved` starts and DNS works:
```
$ lxc exec jammytest -- cat /proc/sys/crypto/fips_enabled
1
$ lxc exec jammytest -- systemctl is-active systemd-resolved
active
$ lxc exec jammytest -- getent hosts archive.ubuntu.com
2620:2d:4000:1::102 archive.ubuntu.com
2620:2d:4000:1::101 archive.ubuntu.com
[...]
```
## Analysis (big thanks to my intern :robot:)
The journal shows resolved getting past trust-anchor loading before
failing in `manager_new()`. The next big step is DNS-over-TLS
initialisation. In `src/resolve/resolved-manager.c` (v255, lines
606-614), `manager_new()` parses the config and *then* initialises DNS-
over-TLS unconditionally:
```
r = manager_parse_config_file(m);
if (r < 0)
log_warning_errno(r, "Failed to parse configuration file: %m");
#if ENABLE_DNS_OVER_TLS
r = dnstls_manager_init(m);
if (r < 0)
return r; /* aborts manager_new() -> service fails */
#endif
```
(Note this isn't gated on `dns_over_tls_mode`, even though the
configuration has already been parsed at that point and the mode is
known)
And in `src/resolve/resolved-dnstls-openssl.c`, any `SSL_CTX_new()`
failure gets flattened to `-ENOMEM`:
```
int dnstls_manager_init(Manager *manager) {
int r;
assert(manager);
ERR_load_crypto_strings();
SSL_load_error_strings();
manager->dnstls_data.ctx = SSL_CTX_new(TLS_client_method());
if (!manager->dnstls_data.ctx)
return -ENOMEM;
[...]
```
### Why Jammy isn't affected (the DNS-over-TLS backend changed)
This is the big difference and it seems to be a packaging change rather
than a code issue? From the `debian/rules` in each suite:
jammy:
build options: `-Dgnutls=true`/no `-Ddns-over-tls=`
backend: GnuTLS
source file: resolved-dnstls-gnutls.c
noble:
build options: `-Dopenssl=enabled`, `-Dcryptolib=openssl`,
`-Ddns-over-tls=openssl`
backend: OpenSSL
source file: resolved-dnstls-openssl.c
Jammy's `dnstls_manager_init()` only performs a small allocation and never
builds a TLS context, so it doesn't kick off OpenSSL's provider negotiation at
all:
```
int dnstls_manager_init(Manager *manager) {
int r;
assert(manager);
r =
gnutls_certificate_allocate_credentials(&manager->dnstls_data.cert_cred);
if (r < 0)
return -ENOMEM;
r =
gnutls_certificate_set_x509_system_trust(manager->dnstls_data.cert_cred);
if (r < 0)
log_warning("Failed to load system trust store: %s",
gnutls_strerror(r));
return 0;
}
```
Noble's equivalent calls `SSL_CTX_new()`, which is where FIPS mode is
engaged and where the failure happens. So I think it's the switch to the
OpenSSL backend is what's started this on Noble. This also explains the
observed behaviour, OpenSSL fails to initialise because it sees
`fips_enabled=1` and attempts the FIPS mode, but the container doesn't
have any FIPS providers available. So the real OpenSSL error is
discarded and replaced with an `ENOMEM`.
(Note that the error strings are loaded immediately above the failing
call so the underlying reason is available via `ERR_get_error()` and
just not used)
What's interesting though is that DNS-over-TLS isn't even enabled here?
`resolvectl status` on the host reports `-DNSOverTLS`, which is the
default (`DNSOverTLS=no`). So the service is refusing to start because
an unused TLS context couldn't be created.
Three things are at play
1) Noble's `systemd-resolved` can't start when `fips_enabled=1` but no
FIPS provider is present, a situation that arises automatically for any
container on a FIPS host since the sysctl isn't namespaced. Jammy is
unaffected because it builds the GnuTLS backend (this was introduced by
the switch to `-Ddns-over-tls=openssl` in Noble)
2) `dnstls_manager_init()` is called even when `DNSOverTLS=no`. Gating
it on `dns_over_tls_mode`, or treating its failure as non-fatal and
degrading to plain DNS would avoid the outage entirely (FWIW this looks
like the smallest/safest/easiest(?) fix
3) `SSL_CTX_new()` failures are reported as `ENOMEM`
## Impact
Any Noble 24.04 container on any FIPS-enabled host loses DNS entirely on first
boot with no actionable error message. This affects containers on FIPS hosts
generally and because the symptom is reported as
ENOMEM users are likely to chase the wrong problem.
Reproduced independently on both GCE and Azure Ubuntu Pro FIPS-Updates
Noble images
## Workaround
For now we're bypassing `resolved` in the containers by pointing
`/etc/resolv.conf` directly at the LXD bridge gateway:
```
$ lxc exec nobletest -- sh -c 'echo nameserver 10.157.252.1 > /etc/resolv.conf'
$ lxc exec nobletest -- apt-get update
Fetched 37.5 MB in 5s (7138 kB/s) # works fine woop
```
Relevant noble package versions:
```
systemd 255.4-1ubuntu8.17
openssl 3.0.13-0ubuntu3.16
libssl3t64 >= 3.0.9
```
(noble's `systemd` depends on both `libssl3t64 (>= 3.0.0)` and
`libgcrypt20 (>= 1.10.0)`)
For comparison Jammy ships systemd 249 and OpenSSL 3.0.2, but from what
I can see the real/important difference is the build configuration not
the version
** Affects: systemd (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2169731
Title:
systemd-resolved fails to start in an lxd container on a FIPS-enabled
Noble host (and reports a misleading ENOMEM)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2169731/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
