** Description changed:
Note: This bug report, analysis of the issue, and the potential fix were
created using the assistance of an LLM.
[ Impact ]
- Users on Ubuntu 24.04 noble who attempt to sign files via a PKCS#11
- token using the OpenSSL PKCS#11 engine (libengine-pkcs11-openssl)
- experience a segfault. This affects any tool that uses the engine for
- signing, including sbsign (used for UEFI Secure Boot binary signing) and
- osslsigncode. The crash occurs unconditionally on the affected OpenSSL
- versions — there is no workaround available to users without replacing
- the library.
+ OpenSSL 3.0.12 and 3.0.13 carry commit 2b74e75 ("Improved detection of
+ engine-provided private classic keys"), which runs a get1/set1 round-
+ trip on the key returned by ENGINE_load_private_key(). That round-trip
+ calls ENGINE_finish() on the key's engine and clears the binding,
+ dropping the functional reference the engine took for itself.
- The root cause is a regression introduced in OpenSSL commit 2b74e75
- (shipped in 3.0.12) and fixed in OpenSSL commit 39ea783 (not backported
- to the 3.0.x series in noble). libp11 0.4.13 (in plucky 25.04+) carries
- a runtime workaround that detects the affected OpenSSL versions and
- reroutes signing through PKEY_CRYPTO, avoiding the broken code path.
- Noble ships 0.4.12 which does not have this workaround.
+ Applications that then release their own engine reference before using
+ the key take the reference count to zero, so the PKCS#11 context is torn
+ down while the key is still in use, and signing dereferences freed
+ memory. sbsign, sbvarsign, osslsigncode and Apache mod_ssl all follow
+ that pattern. Applications that hold their reference until teardown,
+ such as the openssl CLI, wpa_supplicant, libcurl, nginx and libpq, are
+ unaffected, which is why this does not reproduce with "openssl pkeyutl".
+
+ Reported in Ubuntu here and in LP #2119094 (Google Cloud KMS and
+ DigiCert KeyLocker code signing), LP #2067163 and LP #2097330, and
+ upstream as mtrojnar/osslsigncode#410 and #316.
+
+ Upstream reverted 2b74e75 in commit
+ ad6cbe4b7f57a783a66a7ae883ea0d35ef5f82b6 on the openssl-3.0 branch,
+ which is 39ea78379826fa98e8dc8c0d2b07e2c17cd68380 on master. It was
+ released in 3.0.14 and has been carried in every 3.0.x since. noble is
+ the only supported Ubuntu series still carrying the defect. jammy has
+ 3.0.2-0ubuntu1.30, which predates the change, and resolute and stonking
+ are both past the revert.
+
+ This upload backports that revert as a single patch deleting 42 lines
+ from crypto/engine/eng_pkey.c. No other patch in noble's 86-patch stack
+ touches crypto/engine/, so it applies with no fuzz or offset.
+
+ This bug was previously fixed in libp11 rather than openssl, by
+ backporting a workaround that registers the PKCS#11 engine as the
+ process-wide default for RSA and EC. That caused its own regression, LP
+ #2163152, and has been reverted in libp11 0.4.12-1.1ubuntu2, currently
+ in noble-proposed. libp11's upstream maintainer declined to fix the side
+ effect in libp11 and recommended this backport instead, saying "The
+ clean solution for affected distributions is to backport
+ openssl/openssl#23063 and disable this libp11 workaround for that
+ patched OpenSSL build. If the distribution retains the runtime version
+ number 3.0.13, both changes are necessary because OpenSSL_version_num()
+ cannot distinguish the backport." Debian reached the same conclusion
+ independently. The intention is to release the libp11 revert and this
+ openssl upload together, so that noble never spends time in the crashing
+ state.
[ Test Plan ]
- Requires a PKCS#11 token. Tested with a YubiHSM2 device; any PKCS#11
- token with an RSA key should reproduce the issue.
+ Reproducible with SoftHSM2 and no hardware. libp11 must be
+ 0.4.12-1.1build2 or 0.4.12-1.1ubuntu2, because the workaround in
+ 0.4.12-1.1ubuntu1 masks the crash.
- 1. On a fresh noble system, install dependencies:
- sudo apt install libengine-pkcs11-openssl sbsigntool opensc
- # Also install yubihsm-connector, yubihsm-shell, yubihsm-pkcs11 from the
Yubico SDK
+ 1. sudo apt install softhsm2 opensc libengine-pkcs11-openssl sbsigntool
+ shim-signed
- 2. Start yubihsm-connector and confirm device is reachable:
- curl -s http://127.0.0.1:12345/connector/status # expect status=OK
+ 2. Create a token, import an RSA key and a certificate.
- 3. Generate a test RSA-2048 key and self-signed certificate:
- openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out test.key
- openssl req -new -x509 -key test.key -out test.crt -days 365 -subj
"/CN=Test"
+ openssl req -x509 -newkey rsa:2048 -keyout k.pem -out c.pem -days 365
-nodes -subj "/CN=Test"
+ softhsm2-util --init-token --free --label test --so-pin 1234 --pin 1234
+ softhsm2-util --import k.pem --token test --label testkey --id 01 --pin
1234
- 4. Load the key into the token (YubiHSM2 factory defaults):
- yubihsm-shell --connector http://127.0.0.1:12345 -p password --authkey 1 \
- -a put-asymmetric-key --object-id 257 --label "test-key" \
- --domains 1 --capabilities sign-pkcs --algorithm rsa2048 --in test.key
+ 3. Sign a PE binary through the engine.
- 5. Create PKCS#11 connector config:
- echo "connector = http://127.0.0.1:12345"; > yubihsm_pkcs11.conf
+ sbsign --engine pkcs11 --key
+ 'pkcs11:token=test;object=testkey;type=private;pin-value=1234' --cert
+ c.pem --output /dev/null /usr/lib/shim/shimx64.efi
- 6. Attempt to sign an EFI binary:
- YUBIHSM_PKCS11_CONF=$(pwd)/yubihsm_pkcs11.conf \
- sbsign --engine pkcs11 \
- --key "pkcs11:id=%01%01;type=private;pin-value=0001password" \
- --cert test.crt --output test-output.efi /boot/efi/EFI/ubuntu/grubx64.efi
+ Before fix, Segmentation fault, exit 139.
+ After fix, "Signing Unsigned original image", exit 0.
- Before fix: Segmentation fault (core dumped)
- After fix: signed binary produced; sbverify --cert test.crt
test-output.efi returns OK. stderr will print "Workaround for OpenSSL 3.0.13
enabled" — this is expected.
+ Any PKCS#11 token works, substituting its own key URI. Verified with
+ SoftHSM2 and with a YubiHSM 2.
[ Where problems could occur ]
- The patch adds a call to ENGINE_set_default_string(engine,
- "PKEY_CRYPTO") inside load_privkey() when an affected OpenSSL version is
- detected. This reroutes all PKEY_CRYPTO operations for the engine
- instance through the EVP_PKEY path rather than the legacy RSA_sign path.
+ - openssl is in main and on every Ubuntu system, and 1397 binary
+ packages in noble depend on libssl3t64, so any defect introduced here
+ has an unusually large blast radius. That alone justifies caution
+ regardless of how small the change is. What bounds the risk is
+ reachability rather than popularity. The reverted code sits in
+ ENGINE_load_private_key(), part of the deprecated ENGINE API, and runs
+ only when an application explicitly loads a private key through an
+ engine. Nothing on a default install does that, since it needs an engine
+ to be configured and a token or HSM to be present. The patch deletes 42
+ lines and adds none, changes no header, no symbol and no struct layout,
+ so the ABI is untouched and nothing needs rebuilding.
- Possible failure modes:
+ - The patch restores crypto/engine/eng_pkey.c to its 3.0.11 form, so any
+ OpenSSL ENGINE that relied on OpenSSL coercing its returned key to
+ legacy will behave differently. Upstream named the two regressions this
+ revert would cause before merging it, openssl#17092 and openssl#17286.
+ Both came from the out-of-tree e_nss engine, which upstream judged to be
+ engine-side bugs already worked around there. Both issues are closed,
+ neither has had activity since 2023-10-04, and e_nss is not packaged in
+ Ubuntu.
- - Deployments using foreign keys from multiple engines simultaneously:
- ENGINE_set_default_string sets the default engine for PKEY_CRYPTO
- globally for that engine instance. If an application loads private keys
- from multiple engines and relies on per-key engine dispatch, this could
- interfere. Such deployments are uncommon and would already be non-
- functional on affected OpenSSL versions (the bug causes a segfault
- regardless). The upstream comment in libp11 0.4.13 explicitly
- acknowledges this trade-off.
+ - The engines shipped in noble on amd64, taken from the release and
+ updates Contents indexes, are afalg, loader_attic and padlock from
+ libssl3t64 itself, plus pkcs11 from libengine-pkcs11-openssl, tpm2tss
+ from libengine-tpm2-tss-openssl, gost from libengine-gost-openssl, and
+ qatengine. Only libp11 changes behaviour, in the intended direction.
+ tpm2tss already builds legacy keys, so the round-trip was a no-op for
+ it. The qatengine KPT path was not exercised, since it needs QAT
+ hardware.
- - ENGINE_set_default_string failure: if the call fails on an affected
- OpenSSL version, the patch logs "Failed to set PKEY_CRYPTO default
- engine" to stderr and continues. The behaviour is then identical to the
- unpatched package — a segfault on signing. This is not worse than the
- current state but means the fix silently did not apply.
+ - The reverted code is only reachable via ENGINE_load_private_key(), so
+ TLS handshakes, certificate verification, FIPS mode and provider-only
+ stacks are untouched.
- - The workaround emits "Workaround for OpenSSL X.Y.Z enabled" to stderr
- on every key load on affected versions. Applications that parse or
- suppress stderr output may be affected by this unexpected message.
+ - Engine teardown moves from key-load time back to EVP_PKEY_free time.
+ That is the pre-3.0.12 ordering, but it is still a change for every
+ consumer of the engine path.
+
+ - OpenSSL's own test suite cannot detect this, because no test signs
+ with an engine-loaded key. The suite passing says the patch applies
+ cleanly and nothing more, so verification has to be manual.
+
+ - On the field record, the revert has been in Debian bookworm since
+ openssl 3.0.13-1~deb12u2 in July 2024 and in every upstream 3.0.x since
+ 3.0.14, with no report anywhere of a regression caused by it.
+ openssl#23137 is open and labelled a regression against engine-backed DH
+ keys, but it was filed on 2023-12-22, six weeks before the revert
+ landed, against 3.0.12, and its root cause is in crypto/dh/dh_backend.c.
+
+ Measured with SoftHSM2 and a YubiHSM 2, signing with sbsign against an
+ RSA-2048 token key.
+
+ openssl 3.0.13-0ubuntu3.15 + libp11 0.4.12-1.1build2 sbsign
exits 139
+ openssl 3.0.13-0ubuntu3.15 + libp11 0.4.12-1.1ubuntu1 sbsign
exits 0, engine left as process-wide default
+ openssl 3.0.13-0ubuntu3.16~ppa1 + libp11 0.4.12-1.1build2 sbsign
exits 0, engine not left as default
+ openssl 3.0.13-0ubuntu3.16~ppa1 + libp11 0.4.12-1.1ubuntu2~ppa1 sbsign
exits 0, engine not left as default
+
+ The last two rows used PPA builds based on 3.0.13-0ubuntu3.15, before
+ the 3.0.13-0ubuntu3.16 security update. This upload rebases the revert
+ onto that security update and will be re-verified against the -proposed
+ package.
[ Other Info ]
- - Upstream fix: https://github.com/OpenSC/libp11/pull/554 (landed in libp11
0.4.13)
- - OpenSSL regression: openssl/openssl@2b74e75, fixed in
openssl/openssl@39ea783
- - The fix is present in plucky (0.4.13) and questing (0.4.16) without
reported regressions.
- - autopkgtest results: TODO
+ - This SRU was invited in 2024. LP #2062167
+ (https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/2062167)
+ proposed exactly this commit for noble before release, saying "The
+ bigger one is ad6cbe4b...: Revert "Improved detection of engine-provided
+ private "classic" keys" ... All look safe to me." It was deferred with
+ "it's too late for this right now, considering we're in Final Freeze for
+ 24.04 ... Let's revisit as an SRU", and never filed.
+
+ - Christian Ehrhardt identified the same commit independently on LP
+ #2119094, comment #6,
+ https://bugs.launchpad.net/ubuntu/+source/libp11/+bug/2119094/comments/6
+
+ - Upstream libp11 declined to fix the workaround's side effect in libp11
+ and recommended that distributions backport the OpenSSL revert and drop
+ the workaround. OpenSC/libp11#676 is the discussion and
+ https://github.com/OpenSC/libp11/issues/676#issuecomment-5341749942 is
+ the specific recommendation.
+
+ - Debian hit the same defect and fixed it in openssl rather than libp11.
+ https://bugs.debian.org/1074764 was reported on 2024-07-02 against
+ libssl3, and openssl 3.0.13-1~deb12u2 shipped to bookworm on 2024-07-05
+ with the changelog entry "Revert "Improved detection of engine-provided
+ private "classic" keys" (Closes: #1074764)", three days later. It has
+ been carried in bookworm ever since.
+
+ - Upstream OpenSSL references.
+ - the defect,
https://github.com/openssl/openssl/commit/2b74e75331a27fc89cad9c8ea6a26c70019300b5
+ - the revert on openssl-3.0,
https://github.com/openssl/openssl/commit/ad6cbe4b7f57a783a66a7ae883ea0d35ef5f82b6
+ - the revert on master,
https://github.com/openssl/openssl/commit/39ea78379826fa98e8dc8c0d2b07e2c17cd68380
+ - the pull request it was merged under,
https://github.com/openssl/openssl/pull/23063
+ - the regressions upstream expected,
https://github.com/openssl/openssl/issues/17092 and
https://github.com/openssl/openssl/issues/17286
+ - the open engine issue that predates the revert,
https://github.com/openssl/openssl/issues/23137
+
+ - Downstream reports of the same crash,
+ https://github.com/mtrojnar/osslsigncode/issues/410 from the PuTTY
+ maintainer signing releases with a SafeNet token, and
+ https://github.com/mtrojnar/osslsigncode/issues/316
+
+ - Related Ubuntu bugs. LP #2163152 is the regression caused by the
+ libp11 workaround, reverted in 0.4.12-1.1ubuntu2. LP #2119094, LP
+ #2067163 and LP #2097330 are further reports of this crash.
+
+ - The previous description of this bug was the SRU template for the
+ libp11 workaround, which is now being reverted. You can find the old SRU
+ template description from this SRU bug's activity history.
** Description changed:
Note: This bug report, analysis of the issue, and the potential fix were
created using the assistance of an LLM.
[ Impact ]
OpenSSL 3.0.12 and 3.0.13 carry commit 2b74e75 ("Improved detection of
engine-provided private classic keys"), which runs a get1/set1 round-
trip on the key returned by ENGINE_load_private_key(). That round-trip
calls ENGINE_finish() on the key's engine and clears the binding,
dropping the functional reference the engine took for itself.
Applications that then release their own engine reference before using
the key take the reference count to zero, so the PKCS#11 context is torn
down while the key is still in use, and signing dereferences freed
memory. sbsign, sbvarsign, osslsigncode and Apache mod_ssl all follow
that pattern. Applications that hold their reference until teardown,
such as the openssl CLI, wpa_supplicant, libcurl, nginx and libpq, are
unaffected, which is why this does not reproduce with "openssl pkeyutl".
Reported in Ubuntu here and in LP #2119094 (Google Cloud KMS and
- DigiCert KeyLocker code signing), LP #2067163 and LP #2097330, and
+ DigiCert KeyLocker code signing), LP: #2067163 and LP: #2097330, and
upstream as mtrojnar/osslsigncode#410 and #316.
Upstream reverted 2b74e75 in commit
ad6cbe4b7f57a783a66a7ae883ea0d35ef5f82b6 on the openssl-3.0 branch,
which is 39ea78379826fa98e8dc8c0d2b07e2c17cd68380 on master. It was
released in 3.0.14 and has been carried in every 3.0.x since. noble is
the only supported Ubuntu series still carrying the defect. jammy has
3.0.2-0ubuntu1.30, which predates the change, and resolute and stonking
are both past the revert.
This upload backports that revert as a single patch deleting 42 lines
from crypto/engine/eng_pkey.c. No other patch in noble's 86-patch stack
touches crypto/engine/, so it applies with no fuzz or offset.
This bug was previously fixed in libp11 rather than openssl, by
backporting a workaround that registers the PKCS#11 engine as the
process-wide default for RSA and EC. That caused its own regression, LP
#2163152, and has been reverted in libp11 0.4.12-1.1ubuntu2, currently
in noble-proposed. libp11's upstream maintainer declined to fix the side
effect in libp11 and recommended this backport instead, saying "The
clean solution for affected distributions is to backport
openssl/openssl#23063 and disable this libp11 workaround for that
patched OpenSSL build. If the distribution retains the runtime version
number 3.0.13, both changes are necessary because OpenSSL_version_num()
cannot distinguish the backport." Debian reached the same conclusion
independently. The intention is to release the libp11 revert and this
openssl upload together, so that noble never spends time in the crashing
state.
[ Test Plan ]
Reproducible with SoftHSM2 and no hardware. libp11 must be
0.4.12-1.1build2 or 0.4.12-1.1ubuntu2, because the workaround in
0.4.12-1.1ubuntu1 masks the crash.
1. sudo apt install softhsm2 opensc libengine-pkcs11-openssl sbsigntool
shim-signed
2. Create a token, import an RSA key and a certificate.
- openssl req -x509 -newkey rsa:2048 -keyout k.pem -out c.pem -days 365
-nodes -subj "/CN=Test"
- softhsm2-util --init-token --free --label test --so-pin 1234 --pin 1234
- softhsm2-util --import k.pem --token test --label testkey --id 01 --pin
1234
+ openssl req -x509 -newkey rsa:2048 -keyout k.pem -out c.pem -days 365
-nodes -subj "/CN=Test"
+ softhsm2-util --init-token --free --label test --so-pin 1234 --pin 1234
+ softhsm2-util --import k.pem --token test --label testkey --id 01 --pin
1234
3. Sign a PE binary through the engine.
- sbsign --engine pkcs11 --key
+ sbsign --engine pkcs11 --key
'pkcs11:token=test;object=testkey;type=private;pin-value=1234' --cert
c.pem --output /dev/null /usr/lib/shim/shimx64.efi
- Before fix, Segmentation fault, exit 139.
- After fix, "Signing Unsigned original image", exit 0.
+ Before fix, Segmentation fault, exit 139.
+ After fix, "Signing Unsigned original image", exit 0.
Any PKCS#11 token works, substituting its own key URI. Verified with
SoftHSM2 and with a YubiHSM 2.
[ Where problems could occur ]
- openssl is in main and on every Ubuntu system, and 1397 binary
packages in noble depend on libssl3t64, so any defect introduced here
has an unusually large blast radius. That alone justifies caution
regardless of how small the change is. What bounds the risk is
reachability rather than popularity. The reverted code sits in
ENGINE_load_private_key(), part of the deprecated ENGINE API, and runs
only when an application explicitly loads a private key through an
engine. Nothing on a default install does that, since it needs an engine
to be configured and a token or HSM to be present. The patch deletes 42
lines and adds none, changes no header, no symbol and no struct layout,
so the ABI is untouched and nothing needs rebuilding.
- The patch restores crypto/engine/eng_pkey.c to its 3.0.11 form, so any
OpenSSL ENGINE that relied on OpenSSL coercing its returned key to
legacy will behave differently. Upstream named the two regressions this
revert would cause before merging it, openssl#17092 and openssl#17286.
Both came from the out-of-tree e_nss engine, which upstream judged to be
engine-side bugs already worked around there. Both issues are closed,
neither has had activity since 2023-10-04, and e_nss is not packaged in
Ubuntu.
- The engines shipped in noble on amd64, taken from the release and
updates Contents indexes, are afalg, loader_attic and padlock from
libssl3t64 itself, plus pkcs11 from libengine-pkcs11-openssl, tpm2tss
from libengine-tpm2-tss-openssl, gost from libengine-gost-openssl, and
qatengine. Only libp11 changes behaviour, in the intended direction.
tpm2tss already builds legacy keys, so the round-trip was a no-op for
it. The qatengine KPT path was not exercised, since it needs QAT
hardware.
- The reverted code is only reachable via ENGINE_load_private_key(), so
TLS handshakes, certificate verification, FIPS mode and provider-only
stacks are untouched.
- Engine teardown moves from key-load time back to EVP_PKEY_free time.
That is the pre-3.0.12 ordering, but it is still a change for every
consumer of the engine path.
- OpenSSL's own test suite cannot detect this, because no test signs
with an engine-loaded key. The suite passing says the patch applies
cleanly and nothing more, so verification has to be manual.
- On the field record, the revert has been in Debian bookworm since
openssl 3.0.13-1~deb12u2 in July 2024 and in every upstream 3.0.x since
3.0.14, with no report anywhere of a regression caused by it.
openssl#23137 is open and labelled a regression against engine-backed DH
keys, but it was filed on 2023-12-22, six weeks before the revert
landed, against 3.0.12, and its root cause is in crypto/dh/dh_backend.c.
Measured with SoftHSM2 and a YubiHSM 2, signing with sbsign against an
RSA-2048 token key.
- openssl 3.0.13-0ubuntu3.15 + libp11 0.4.12-1.1build2 sbsign
exits 139
- openssl 3.0.13-0ubuntu3.15 + libp11 0.4.12-1.1ubuntu1 sbsign
exits 0, engine left as process-wide default
- openssl 3.0.13-0ubuntu3.16~ppa1 + libp11 0.4.12-1.1build2 sbsign
exits 0, engine not left as default
- openssl 3.0.13-0ubuntu3.16~ppa1 + libp11 0.4.12-1.1ubuntu2~ppa1 sbsign
exits 0, engine not left as default
+ openssl 3.0.13-0ubuntu3.15 + libp11 0.4.12-1.1build2 sbsign
exits 139
+ openssl 3.0.13-0ubuntu3.15 + libp11 0.4.12-1.1ubuntu1 sbsign
exits 0, engine left as process-wide default
+ openssl 3.0.13-0ubuntu3.16~ppa1 + libp11 0.4.12-1.1build2 sbsign
exits 0, engine not left as default
+ openssl 3.0.13-0ubuntu3.16~ppa1 + libp11 0.4.12-1.1ubuntu2~ppa1 sbsign
exits 0, engine not left as default
The last two rows used PPA builds based on 3.0.13-0ubuntu3.15, before
the 3.0.13-0ubuntu3.16 security update. This upload rebases the revert
onto that security update and will be re-verified against the -proposed
package.
[ Other Info ]
- This SRU was invited in 2024. LP #2062167
(https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/2062167)
proposed exactly this commit for noble before release, saying "The
bigger one is ad6cbe4b...: Revert "Improved detection of engine-provided
private "classic" keys" ... All look safe to me." It was deferred with
"it's too late for this right now, considering we're in Final Freeze for
24.04 ... Let's revisit as an SRU", and never filed.
- Christian Ehrhardt identified the same commit independently on LP
#2119094, comment #6,
https://bugs.launchpad.net/ubuntu/+source/libp11/+bug/2119094/comments/6
- Upstream libp11 declined to fix the workaround's side effect in libp11
and recommended that distributions backport the OpenSSL revert and drop
the workaround. OpenSC/libp11#676 is the discussion and
https://github.com/OpenSC/libp11/issues/676#issuecomment-5341749942 is
the specific recommendation.
- Debian hit the same defect and fixed it in openssl rather than libp11.
https://bugs.debian.org/1074764 was reported on 2024-07-02 against
libssl3, and openssl 3.0.13-1~deb12u2 shipped to bookworm on 2024-07-05
with the changelog entry "Revert "Improved detection of engine-provided
private "classic" keys" (Closes: #1074764)", three days later. It has
been carried in bookworm ever since.
- Upstream OpenSSL references.
- - the defect,
https://github.com/openssl/openssl/commit/2b74e75331a27fc89cad9c8ea6a26c70019300b5
- - the revert on openssl-3.0,
https://github.com/openssl/openssl/commit/ad6cbe4b7f57a783a66a7ae883ea0d35ef5f82b6
- - the revert on master,
https://github.com/openssl/openssl/commit/39ea78379826fa98e8dc8c0d2b07e2c17cd68380
- - the pull request it was merged under,
https://github.com/openssl/openssl/pull/23063
- - the regressions upstream expected,
https://github.com/openssl/openssl/issues/17092 and
https://github.com/openssl/openssl/issues/17286
- - the open engine issue that predates the revert,
https://github.com/openssl/openssl/issues/23137
+ - the defect,
https://github.com/openssl/openssl/commit/2b74e75331a27fc89cad9c8ea6a26c70019300b5
+ - the revert on openssl-3.0,
https://github.com/openssl/openssl/commit/ad6cbe4b7f57a783a66a7ae883ea0d35ef5f82b6
+ - the revert on master,
https://github.com/openssl/openssl/commit/39ea78379826fa98e8dc8c0d2b07e2c17cd68380
+ - the pull request it was merged under,
https://github.com/openssl/openssl/pull/23063
+ - the regressions upstream expected,
https://github.com/openssl/openssl/issues/17092 and
https://github.com/openssl/openssl/issues/17286
+ - the open engine issue that predates the revert,
https://github.com/openssl/openssl/issues/23137
- Downstream reports of the same crash,
https://github.com/mtrojnar/osslsigncode/issues/410 from the PuTTY
maintainer signing releases with a SafeNet token, and
https://github.com/mtrojnar/osslsigncode/issues/316
- Related Ubuntu bugs. LP #2163152 is the regression caused by the
libp11 workaround, reverted in 0.4.12-1.1ubuntu2. LP #2119094, LP
#2067163 and LP #2097330 are further reports of this crash.
- The previous description of this bug was the SRU template for the
libp11 workaround, which is now being reverted. You can find the old SRU
template description from this SRU bug's activity history.
** Description changed:
Note: This bug report, analysis of the issue, and the potential fix were
created using the assistance of an LLM.
[ Impact ]
OpenSSL 3.0.12 and 3.0.13 carry commit 2b74e75 ("Improved detection of
engine-provided private classic keys"), which runs a get1/set1 round-
trip on the key returned by ENGINE_load_private_key(). That round-trip
calls ENGINE_finish() on the key's engine and clears the binding,
dropping the functional reference the engine took for itself.
Applications that then release their own engine reference before using
the key take the reference count to zero, so the PKCS#11 context is torn
down while the key is still in use, and signing dereferences freed
memory. sbsign, sbvarsign, osslsigncode and Apache mod_ssl all follow
that pattern. Applications that hold their reference until teardown,
such as the openssl CLI, wpa_supplicant, libcurl, nginx and libpq, are
unaffected, which is why this does not reproduce with "openssl pkeyutl".
- Reported in Ubuntu here and in LP #2119094 (Google Cloud KMS and
+ Reported in Ubuntu here and in LP: #2119094 (Google Cloud KMS and
DigiCert KeyLocker code signing), LP: #2067163 and LP: #2097330, and
upstream as mtrojnar/osslsigncode#410 and #316.
Upstream reverted 2b74e75 in commit
ad6cbe4b7f57a783a66a7ae883ea0d35ef5f82b6 on the openssl-3.0 branch,
which is 39ea78379826fa98e8dc8c0d2b07e2c17cd68380 on master. It was
released in 3.0.14 and has been carried in every 3.0.x since. noble is
the only supported Ubuntu series still carrying the defect. jammy has
3.0.2-0ubuntu1.30, which predates the change, and resolute and stonking
are both past the revert.
This upload backports that revert as a single patch deleting 42 lines
from crypto/engine/eng_pkey.c. No other patch in noble's 86-patch stack
touches crypto/engine/, so it applies with no fuzz or offset.
This bug was previously fixed in libp11 rather than openssl, by
backporting a workaround that registers the PKCS#11 engine as the
process-wide default for RSA and EC. That caused its own regression, LP
#2163152, and has been reverted in libp11 0.4.12-1.1ubuntu2, currently
in noble-proposed. libp11's upstream maintainer declined to fix the side
effect in libp11 and recommended this backport instead, saying "The
clean solution for affected distributions is to backport
openssl/openssl#23063 and disable this libp11 workaround for that
patched OpenSSL build. If the distribution retains the runtime version
number 3.0.13, both changes are necessary because OpenSSL_version_num()
cannot distinguish the backport." Debian reached the same conclusion
independently. The intention is to release the libp11 revert and this
openssl upload together, so that noble never spends time in the crashing
state.
[ Test Plan ]
Reproducible with SoftHSM2 and no hardware. libp11 must be
0.4.12-1.1build2 or 0.4.12-1.1ubuntu2, because the workaround in
0.4.12-1.1ubuntu1 masks the crash.
1. sudo apt install softhsm2 opensc libengine-pkcs11-openssl sbsigntool
shim-signed
2. Create a token, import an RSA key and a certificate.
openssl req -x509 -newkey rsa:2048 -keyout k.pem -out c.pem -days 365
-nodes -subj "/CN=Test"
softhsm2-util --init-token --free --label test --so-pin 1234 --pin 1234
softhsm2-util --import k.pem --token test --label testkey --id 01 --pin
1234
3. Sign a PE binary through the engine.
sbsign --engine pkcs11 --key
'pkcs11:token=test;object=testkey;type=private;pin-value=1234' --cert
c.pem --output /dev/null /usr/lib/shim/shimx64.efi
Before fix, Segmentation fault, exit 139.
After fix, "Signing Unsigned original image", exit 0.
Any PKCS#11 token works, substituting its own key URI. Verified with
SoftHSM2 and with a YubiHSM 2.
[ Where problems could occur ]
- openssl is in main and on every Ubuntu system, and 1397 binary
packages in noble depend on libssl3t64, so any defect introduced here
has an unusually large blast radius. That alone justifies caution
regardless of how small the change is. What bounds the risk is
reachability rather than popularity. The reverted code sits in
ENGINE_load_private_key(), part of the deprecated ENGINE API, and runs
only when an application explicitly loads a private key through an
engine. Nothing on a default install does that, since it needs an engine
to be configured and a token or HSM to be present. The patch deletes 42
lines and adds none, changes no header, no symbol and no struct layout,
so the ABI is untouched and nothing needs rebuilding.
- The patch restores crypto/engine/eng_pkey.c to its 3.0.11 form, so any
OpenSSL ENGINE that relied on OpenSSL coercing its returned key to
legacy will behave differently. Upstream named the two regressions this
revert would cause before merging it, openssl#17092 and openssl#17286.
Both came from the out-of-tree e_nss engine, which upstream judged to be
engine-side bugs already worked around there. Both issues are closed,
neither has had activity since 2023-10-04, and e_nss is not packaged in
Ubuntu.
- The engines shipped in noble on amd64, taken from the release and
updates Contents indexes, are afalg, loader_attic and padlock from
libssl3t64 itself, plus pkcs11 from libengine-pkcs11-openssl, tpm2tss
from libengine-tpm2-tss-openssl, gost from libengine-gost-openssl, and
qatengine. Only libp11 changes behaviour, in the intended direction.
tpm2tss already builds legacy keys, so the round-trip was a no-op for
it. The qatengine KPT path was not exercised, since it needs QAT
hardware.
- The reverted code is only reachable via ENGINE_load_private_key(), so
TLS handshakes, certificate verification, FIPS mode and provider-only
stacks are untouched.
- Engine teardown moves from key-load time back to EVP_PKEY_free time.
That is the pre-3.0.12 ordering, but it is still a change for every
consumer of the engine path.
- OpenSSL's own test suite cannot detect this, because no test signs
with an engine-loaded key. The suite passing says the patch applies
cleanly and nothing more, so verification has to be manual.
- On the field record, the revert has been in Debian bookworm since
openssl 3.0.13-1~deb12u2 in July 2024 and in every upstream 3.0.x since
3.0.14, with no report anywhere of a regression caused by it.
openssl#23137 is open and labelled a regression against engine-backed DH
keys, but it was filed on 2023-12-22, six weeks before the revert
landed, against 3.0.12, and its root cause is in crypto/dh/dh_backend.c.
Measured with SoftHSM2 and a YubiHSM 2, signing with sbsign against an
RSA-2048 token key.
openssl 3.0.13-0ubuntu3.15 + libp11 0.4.12-1.1build2 sbsign
exits 139
openssl 3.0.13-0ubuntu3.15 + libp11 0.4.12-1.1ubuntu1 sbsign
exits 0, engine left as process-wide default
openssl 3.0.13-0ubuntu3.16~ppa1 + libp11 0.4.12-1.1build2 sbsign
exits 0, engine not left as default
openssl 3.0.13-0ubuntu3.16~ppa1 + libp11 0.4.12-1.1ubuntu2~ppa1 sbsign
exits 0, engine not left as default
The last two rows used PPA builds based on 3.0.13-0ubuntu3.15, before
the 3.0.13-0ubuntu3.16 security update. This upload rebases the revert
onto that security update and will be re-verified against the -proposed
package.
[ Other Info ]
- - This SRU was invited in 2024. LP #2062167
+ - This SRU was invited in 2024. LP: #2062167
(https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/2062167)
proposed exactly this commit for noble before release, saying "The
bigger one is ad6cbe4b...: Revert "Improved detection of engine-provided
private "classic" keys" ... All look safe to me." It was deferred with
"it's too late for this right now, considering we're in Final Freeze for
24.04 ... Let's revisit as an SRU", and never filed.
- - Christian Ehrhardt identified the same commit independently on LP
+ - Christian Ehrhardt identified the same commit independently on LP:
#2119094, comment #6,
https://bugs.launchpad.net/ubuntu/+source/libp11/+bug/2119094/comments/6
- Upstream libp11 declined to fix the workaround's side effect in libp11
and recommended that distributions backport the OpenSSL revert and drop
the workaround. OpenSC/libp11#676 is the discussion and
https://github.com/OpenSC/libp11/issues/676#issuecomment-5341749942 is
the specific recommendation.
- Debian hit the same defect and fixed it in openssl rather than libp11.
https://bugs.debian.org/1074764 was reported on 2024-07-02 against
libssl3, and openssl 3.0.13-1~deb12u2 shipped to bookworm on 2024-07-05
with the changelog entry "Revert "Improved detection of engine-provided
private "classic" keys" (Closes: #1074764)", three days later. It has
been carried in bookworm ever since.
- Upstream OpenSSL references.
- the defect,
https://github.com/openssl/openssl/commit/2b74e75331a27fc89cad9c8ea6a26c70019300b5
- the revert on openssl-3.0,
https://github.com/openssl/openssl/commit/ad6cbe4b7f57a783a66a7ae883ea0d35ef5f82b6
- the revert on master,
https://github.com/openssl/openssl/commit/39ea78379826fa98e8dc8c0d2b07e2c17cd68380
- the pull request it was merged under,
https://github.com/openssl/openssl/pull/23063
- the regressions upstream expected,
https://github.com/openssl/openssl/issues/17092 and
https://github.com/openssl/openssl/issues/17286
- the open engine issue that predates the revert,
https://github.com/openssl/openssl/issues/23137
- Downstream reports of the same crash,
https://github.com/mtrojnar/osslsigncode/issues/410 from the PuTTY
maintainer signing releases with a SafeNet token, and
https://github.com/mtrojnar/osslsigncode/issues/316
- - Related Ubuntu bugs. LP #2163152 is the regression caused by the
- libp11 workaround, reverted in 0.4.12-1.1ubuntu2. LP #2119094, LP
- #2067163 and LP #2097330 are further reports of this crash.
+ - Related Ubuntu bugs. LP: #2163152 is the regression caused by the
+ libp11 workaround, reverted in 0.4.12-1.1ubuntu2. LP: #2119094, LP:
+ #2067163 and LP: #2097330 are further reports of this crash.
- The previous description of this bug was the SRU template for the
libp11 workaround, which is now being reverted. You can find the old SRU
template description from this SRU bug's activity history.
** Description changed:
Note: This bug report, analysis of the issue, and the potential fix were
created using the assistance of an LLM.
[ Impact ]
OpenSSL 3.0.12 and 3.0.13 carry commit 2b74e75 ("Improved detection of
engine-provided private classic keys"), which runs a get1/set1 round-
trip on the key returned by ENGINE_load_private_key(). That round-trip
calls ENGINE_finish() on the key's engine and clears the binding,
dropping the functional reference the engine took for itself.
Applications that then release their own engine reference before using
the key take the reference count to zero, so the PKCS#11 context is torn
down while the key is still in use, and signing dereferences freed
memory. sbsign, sbvarsign, osslsigncode and Apache mod_ssl all follow
that pattern. Applications that hold their reference until teardown,
such as the openssl CLI, wpa_supplicant, libcurl, nginx and libpq, are
unaffected, which is why this does not reproduce with "openssl pkeyutl".
Reported in Ubuntu here and in LP: #2119094 (Google Cloud KMS and
DigiCert KeyLocker code signing), LP: #2067163 and LP: #2097330, and
upstream as mtrojnar/osslsigncode#410 and #316.
Upstream reverted 2b74e75 in commit
ad6cbe4b7f57a783a66a7ae883ea0d35ef5f82b6 on the openssl-3.0 branch,
which is 39ea78379826fa98e8dc8c0d2b07e2c17cd68380 on master. It was
released in 3.0.14 and has been carried in every 3.0.x since. noble is
the only supported Ubuntu series still carrying the defect. jammy has
3.0.2-0ubuntu1.30, which predates the change, and resolute and stonking
are both past the revert.
This upload backports that revert as a single patch deleting 42 lines
from crypto/engine/eng_pkey.c. No other patch in noble's 86-patch stack
touches crypto/engine/, so it applies with no fuzz or offset.
This bug was previously fixed in libp11 rather than openssl, by
backporting a workaround that registers the PKCS#11 engine as the
- process-wide default for RSA and EC. That caused its own regression, LP
+ process-wide default for RSA and EC. That caused its own regression, LP:
#2163152, and has been reverted in libp11 0.4.12-1.1ubuntu2, currently
in noble-proposed. libp11's upstream maintainer declined to fix the side
effect in libp11 and recommended this backport instead, saying "The
clean solution for affected distributions is to backport
openssl/openssl#23063 and disable this libp11 workaround for that
patched OpenSSL build. If the distribution retains the runtime version
number 3.0.13, both changes are necessary because OpenSSL_version_num()
cannot distinguish the backport." Debian reached the same conclusion
independently. The intention is to release the libp11 revert and this
openssl upload together, so that noble never spends time in the crashing
state.
[ Test Plan ]
Reproducible with SoftHSM2 and no hardware. libp11 must be
0.4.12-1.1build2 or 0.4.12-1.1ubuntu2, because the workaround in
0.4.12-1.1ubuntu1 masks the crash.
1. sudo apt install softhsm2 opensc libengine-pkcs11-openssl sbsigntool
shim-signed
2. Create a token, import an RSA key and a certificate.
openssl req -x509 -newkey rsa:2048 -keyout k.pem -out c.pem -days 365
-nodes -subj "/CN=Test"
softhsm2-util --init-token --free --label test --so-pin 1234 --pin 1234
softhsm2-util --import k.pem --token test --label testkey --id 01 --pin
1234
3. Sign a PE binary through the engine.
sbsign --engine pkcs11 --key
'pkcs11:token=test;object=testkey;type=private;pin-value=1234' --cert
c.pem --output /dev/null /usr/lib/shim/shimx64.efi
Before fix, Segmentation fault, exit 139.
After fix, "Signing Unsigned original image", exit 0.
Any PKCS#11 token works, substituting its own key URI. Verified with
SoftHSM2 and with a YubiHSM 2.
[ Where problems could occur ]
- openssl is in main and on every Ubuntu system, and 1397 binary
packages in noble depend on libssl3t64, so any defect introduced here
has an unusually large blast radius. That alone justifies caution
regardless of how small the change is. What bounds the risk is
reachability rather than popularity. The reverted code sits in
ENGINE_load_private_key(), part of the deprecated ENGINE API, and runs
only when an application explicitly loads a private key through an
engine. Nothing on a default install does that, since it needs an engine
to be configured and a token or HSM to be present. The patch deletes 42
lines and adds none, changes no header, no symbol and no struct layout,
so the ABI is untouched and nothing needs rebuilding.
- The patch restores crypto/engine/eng_pkey.c to its 3.0.11 form, so any
OpenSSL ENGINE that relied on OpenSSL coercing its returned key to
legacy will behave differently. Upstream named the two regressions this
revert would cause before merging it, openssl#17092 and openssl#17286.
Both came from the out-of-tree e_nss engine, which upstream judged to be
engine-side bugs already worked around there. Both issues are closed,
neither has had activity since 2023-10-04, and e_nss is not packaged in
Ubuntu.
- The engines shipped in noble on amd64, taken from the release and
updates Contents indexes, are afalg, loader_attic and padlock from
libssl3t64 itself, plus pkcs11 from libengine-pkcs11-openssl, tpm2tss
from libengine-tpm2-tss-openssl, gost from libengine-gost-openssl, and
qatengine. Only libp11 changes behaviour, in the intended direction.
tpm2tss already builds legacy keys, so the round-trip was a no-op for
it. The qatengine KPT path was not exercised, since it needs QAT
hardware.
- The reverted code is only reachable via ENGINE_load_private_key(), so
TLS handshakes, certificate verification, FIPS mode and provider-only
stacks are untouched.
- Engine teardown moves from key-load time back to EVP_PKEY_free time.
That is the pre-3.0.12 ordering, but it is still a change for every
consumer of the engine path.
- OpenSSL's own test suite cannot detect this, because no test signs
with an engine-loaded key. The suite passing says the patch applies
cleanly and nothing more, so verification has to be manual.
- On the field record, the revert has been in Debian bookworm since
openssl 3.0.13-1~deb12u2 in July 2024 and in every upstream 3.0.x since
3.0.14, with no report anywhere of a regression caused by it.
openssl#23137 is open and labelled a regression against engine-backed DH
keys, but it was filed on 2023-12-22, six weeks before the revert
landed, against 3.0.12, and its root cause is in crypto/dh/dh_backend.c.
Measured with SoftHSM2 and a YubiHSM 2, signing with sbsign against an
RSA-2048 token key.
openssl 3.0.13-0ubuntu3.15 + libp11 0.4.12-1.1build2 sbsign
exits 139
openssl 3.0.13-0ubuntu3.15 + libp11 0.4.12-1.1ubuntu1 sbsign
exits 0, engine left as process-wide default
openssl 3.0.13-0ubuntu3.16~ppa1 + libp11 0.4.12-1.1build2 sbsign
exits 0, engine not left as default
openssl 3.0.13-0ubuntu3.16~ppa1 + libp11 0.4.12-1.1ubuntu2~ppa1 sbsign
exits 0, engine not left as default
The last two rows used PPA builds based on 3.0.13-0ubuntu3.15, before
the 3.0.13-0ubuntu3.16 security update. This upload rebases the revert
onto that security update and will be re-verified against the -proposed
package.
[ Other Info ]
- This SRU was invited in 2024. LP: #2062167
(https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/2062167)
proposed exactly this commit for noble before release, saying "The
bigger one is ad6cbe4b...: Revert "Improved detection of engine-provided
private "classic" keys" ... All look safe to me." It was deferred with
"it's too late for this right now, considering we're in Final Freeze for
24.04 ... Let's revisit as an SRU", and never filed.
- Christian Ehrhardt identified the same commit independently on LP:
#2119094, comment #6,
https://bugs.launchpad.net/ubuntu/+source/libp11/+bug/2119094/comments/6
- Upstream libp11 declined to fix the workaround's side effect in libp11
and recommended that distributions backport the OpenSSL revert and drop
the workaround. OpenSC/libp11#676 is the discussion and
https://github.com/OpenSC/libp11/issues/676#issuecomment-5341749942 is
the specific recommendation.
- Debian hit the same defect and fixed it in openssl rather than libp11.
https://bugs.debian.org/1074764 was reported on 2024-07-02 against
libssl3, and openssl 3.0.13-1~deb12u2 shipped to bookworm on 2024-07-05
with the changelog entry "Revert "Improved detection of engine-provided
private "classic" keys" (Closes: #1074764)", three days later. It has
been carried in bookworm ever since.
- Upstream OpenSSL references.
- the defect,
https://github.com/openssl/openssl/commit/2b74e75331a27fc89cad9c8ea6a26c70019300b5
- the revert on openssl-3.0,
https://github.com/openssl/openssl/commit/ad6cbe4b7f57a783a66a7ae883ea0d35ef5f82b6
- the revert on master,
https://github.com/openssl/openssl/commit/39ea78379826fa98e8dc8c0d2b07e2c17cd68380
- the pull request it was merged under,
https://github.com/openssl/openssl/pull/23063
- the regressions upstream expected,
https://github.com/openssl/openssl/issues/17092 and
https://github.com/openssl/openssl/issues/17286
- the open engine issue that predates the revert,
https://github.com/openssl/openssl/issues/23137
- Downstream reports of the same crash,
https://github.com/mtrojnar/osslsigncode/issues/410 from the PuTTY
maintainer signing releases with a SafeNet token, and
https://github.com/mtrojnar/osslsigncode/issues/316
- Related Ubuntu bugs. LP: #2163152 is the regression caused by the
libp11 workaround, reverted in 0.4.12-1.1ubuntu2. LP: #2119094, LP:
#2067163 and LP: #2097330 are further reports of this crash.
- The previous description of this bug was the SRU template for the
libp11 workaround, which is now being reverted. You can find the old SRU
template description from this SRU bug's activity history.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2158304
Title:
[SRU] sbsign (and any tool using the OpenSSL PKCS#11 engine) segfaults
when signing via a PKCS#11 token on OpenSSL 3.0.13
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/libp11/+bug/2158304/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
