Root cause for the Jammy report in comment #4 (SSL broke across our
servers after the 20260601~22.04.1 upgrade and a later reboot). The
analysis and the reproducer below were prepared with the help of an AI
assistant (Claude).
Summary: update-ca-certificates never creates /etc/ssl/certs hash links
for a local CA file that contains more than one certificate, and
whenever the CA set changes it deletes the hash links that other tools
(c_rehash) created for such a file. Anything that looks CAs up in
/etc/ssl/certs as a directory then can no longer find that CA. A package
update that changes the CA set, like 20240203 -> 20260601, is what
triggers it.
Our setup: a local CA file in /usr/local/share/ca-certificates/
containing three certificates (an intermediate CA, its issuing
intermediate, and the root). Our configuration management also runs
c_rehash /etc/ssl/certs, which links a multi-certificate file under the
hash of its first certificate, so the intermediate CA was findable by
hash and directory-based verification worked for years.
What happened:
1. On 2026-06-17 unattended-upgrades installed ca-certificates
20260601~22.04.1. The CA set changed ("14 added, 39 removed"), so
update-ca-certificates ran "openssl rehash". The log shows: "rehash:
warning: skipping <our file>.pem, it does not contain exactly one
certificate or CRL".
2. openssl rehash removes every existing hash link in the directory before
re-creating links, and it only creates links for files with exactly one
certificate. The link c_rehash had made for our intermediate CA was deleted and
not replaced. The file itself stays in /etc/ssl/certs/ca-certific
ates.crt, so anything that uses that bundle file kept working.
3. Long-running processes kept working because OpenSSL caches the CA
certificates it has already loaded from a CA directory. A week later the
servers restarted, and services that verify through a CA directory (OpenLDAP
slapd with TLSCACertificatePath /etc/ssl/certs, and OpenSSL clients using
the default capath) failed with "unable to get local issuer certificate".
4. Re-running c_rehash restores the link, which is why the workaround in
comment #4 works. The next CA-set change removes it again.
Reproducer (fresh Ubuntu 22.04, 24.04 or 26.04 container with ca-
certificates, openssl and perl; run as root). It builds a throwaway root
-> intermediate -> leaf chain, puts the intermediate and root in one
local .crt file, and verifies the leaf by CA directory only:
w=$(mktemp -d); cd "$w"; L=/usr/local/share/ca-certificates
openssl req -x509 -newkey rsa:2048 -nodes -keyout root.key -subj "/CN=Test
Root" -days 2 -out root.pem
openssl req -newkey rsa:2048 -nodes -keyout int.key -subj "/CN=Test
Intermediate" -out int.csr
printf
'basicConstraints=critical,CA:true\nkeyUsage=critical,keyCertSign,cRLSign\n' >
ca.ext
openssl x509 -req -in int.csr -CA root.pem -CAkey root.key -CAcreateserial
-days 2 -extfile ca.ext -out int.pem
openssl req -newkey rsa:2048 -nodes -keyout leaf.key -subj
"/CN=leaf.example.test" -out leaf.csr
openssl x509 -req -in leaf.csr -CA int.pem -CAkey int.key -CAcreateserial -days
2 -out leaf.pem
cat int.pem root.pem > $L/test-chain.crt
update-ca-certificates
openssl verify -no-CAfile -no-CAstore -CApath /etc/ssl/certs leaf.pem # (A)
fails: the chain file was never linked
c_rehash /etc/ssl/certs
openssl verify -no-CAfile -no-CAstore -CApath /etc/ssl/certs leaf.pem # (B)
OK
openssl req -x509 -newkey rsa:2048 -nodes -keyout /dev/null -subj
"/CN=Unrelated CA" -days 2 -out $L/unrelated.crt
update-ca-certificates # any
CA-set change, e.g. a package update
openssl verify -no-CAfile -no-CAstore -CApath /etc/ssl/certs leaf.pem # (C)
fails again: the c_rehash link was removed
openssl verify -CAfile /etc/ssl/certs/ca-certificates.crt leaf.pem # (D)
OK: the bundle file still has the chain
Results, identical on all three releases (ca-certificates
20260601~22.04.1 with OpenSSL 3.0.2, 20260601~24.04.1 with 3.0.13,
20260601~26.04.1 with 3.5.5): (A) "unable to get local issuer
certificate", (B) OK, (C) "unable to get local issuer certificate", (D)
OK.
Why this is easy to miss: update-ca-certificates only rehashes when the
set of CA files changes, so the link survives every routine run and
disappears only on the next package update that adds or removes CAs.
Running processes hide it until they restart, which can be days later.
The only signal is the "skipping ... does not contain exactly one
certificate" warning in the apt term.log.
The man page says all .crt files below /usr/local/share/ca-certificates are
"included as implicitly trusted", and README.Debian asks for "single files
ending with .crt" without saying what happens to a file with several
certificates. In practice such a file is trusted through ca-certificates
.crt but not through the /etc/ssl/certs directory, and update-ca-certificates
removes the directory links that other tools create for it.
We've worked around it on our side by also installing each CA as a one-
certificate file next to the bundle, which openssl rehash links on every
run.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2157000
Title:
package ca-certificates 20260601~24.04.1 failed to install/upgrade:
installed ca-certificates package post-installation script subprocess
returned error exit status 1
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ca-certificates/+bug/2157000/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
