Bug#969559: curl segmentation fauls on any https URL
On Fri, Sep 11, 2020 at 06:28:20PM +0200, Bernhard Übelacker wrote: > Dear Maintainer, hello Bruce Momjian, > with the last informations the issue is perfectly reproducible. > > It looks like a use after free caused by statically stored > function pointers in libengine-pkcs11-openssl / libp11. > > That led to following upstream bug: > https://github.com/OpenSC/libp11/issues/328 > > This got fixed in this commit: > > https://github.com/OpenSC/libp11/commit/e64496a198d4d2eb0310a22dc21be8b81367d319 > > This commit is not yet included in an upstream release tag. > Therefore this error is also visible in current testing. > > I hope it is ok to reassign to libengine-pkcs11-openssl. Yes, thank you for researching this and closing it. -- Bruce Momjian https://momjian.us EnterpriseDB https://enterprisedb.com The usefulness of a cup is in its emptiness, Bruce Lee
Bug#969559: curl segmentation fauls on any https URL
Dear Maintainer, hello Bruce Momjian, with the last informations the issue is perfectly reproducible. It looks like a use after free caused by statically stored function pointers in libengine-pkcs11-openssl / libp11. That led to following upstream bug: https://github.com/OpenSC/libp11/issues/328 This got fixed in this commit: https://github.com/OpenSC/libp11/commit/e64496a198d4d2eb0310a22dc21be8b81367d319 This commit is not yet included in an upstream release tag. Therefore this error is also visible in current testing. I hope it is ok to reassign to libengine-pkcs11-openssl. Kind regards, Bernhard
Bug#969559: Info received (Bug#969559: curl segmentation fauls on any https URL)
Oh, the kernel error message might be helpful: curl[4979] general protection ip:7f3a3da00bce sp:7fff5dc217d0 error:0 in libcrypto.so.1.1[7f3a3d8fe000+19e000] -- Bruce Momjian https://momjian.us EnterpriseDB https://enterprisedb.com The usefulness of a cup is in its emptiness, Bruce Lee
Bug#969559: curl segmentation fauls on any https URL
On Sun, Sep 6, 2020 at 02:37:22PM +0200, Bernhard Übelacker wrote:
> Hello Bruce Momjian,
> thanks for the details and confirmation.
>
>
> Am 05.09.20 um 17:32 schrieb Bruce Momjian,,,:
> > (gdb) print pmeth->init
> > $1 = (int (*)(EVP_PKEY_CTX *)) 0xf0e0d0c0b0a0908
>
> > gdb) print *pmeth
> > $8 = {pkey_id = 50462976, flags = 117835012, init = 0xf0e0d0c0b0a0908,
> > copy = 0x1716151413121110, cleanup = 0x1f1e1d1c1b1a1918, paramgen_init =
> > 0x98c476a19fc273a5, paramgen = 0x9cc072a593ce7fa9,
>
> The pointer init copy and cleanup are really not looking like usual
> pointers or random ...
>
> > I am using a pkcs11 hardware crypto device, and perhaps it is
> > misconfigured, but it probably shouldn't crash. This might be a library
> > bug, not sure. I will check the pkcs11's configuration now, but it used
> > to work.
>
> But I have no knowledge about such crypto hardware, therefore
> I am not sure if I can be of any more help. Maybe you could
> provide the needed packages, libraries and configuration steps
> that are needed to use such a device of yours when starting with
> a fresh debian installation?
I was just able to reproduce this failure on a fresh install of Debian
10.5/Buster. What I did was just to install pkcs11 support:
apt-get install libengine-pkcs11-openssl
and then modify /etc/ssl/openssl.cnf with the attached patch to use
pkcs11 support; 'curl https://google.com' will then segmentation fault.
This server has no pkcs11 hardware; it is an AWS instance. If you
comment out the line:
pkcs11 = pkcs11_section
curl works again. Thanks for your research so far on this.
--
Bruce Momjian https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
--- /etc/ssl/openssl.cnf.orig 2019-05-30 11:27:48.0 -0400
+++ /etc/ssl/openssl.cnf 2020-09-07 16:02:31.448309714 -0400
@@ -353,6 +353,7 @@
# identifier (optional, default: sha1)
[default_conf]
ssl_conf = ssl_sect
+engines = engine_section
[ssl_sect]
system_default = system_default_sect
@@ -360,3 +361,14 @@
[system_default_sect]
MinProtocol = TLSv1.2
CipherString = DEFAULT@SECLEVEL=2
+
+[engine_section]
+pkcs11 = pkcs11_section
+
+[pkcs11_section]
+# https://github.com/openssl/openssl/blob/master/README.ENGINE
+engine_id = pkcs11
+# same as SO_PATH
+dynamic_path = /usr/lib/x86_64-linux-gnu/engines-1.1/pkcs11.so
+MODULE_PATH = opensc-pkcs11.so
+init = 0
Bug#969559: curl segmentation fauls on any https URL
Hello Bruce Momjian,
thanks for the details and confirmation.
Am 05.09.20 um 17:32 schrieb Bruce Momjian,,,:
> (gdb) print pmeth->init
> $1 = (int (*)(EVP_PKEY_CTX *)) 0xf0e0d0c0b0a0908
> gdb) print *pmeth
> $8 = {pkey_id = 50462976, flags = 117835012, init = 0xf0e0d0c0b0a0908,
> copy = 0x1716151413121110, cleanup = 0x1f1e1d1c1b1a1918, paramgen_init =
> 0x98c476a19fc273a5, paramgen = 0x9cc072a593ce7fa9,
The pointer init copy and cleanup are really not looking like usual
pointers or random ...
> I am using a pkcs11 hardware crypto device, and perhaps it is
> misconfigured, but it probably shouldn't crash. This might be a library
> bug, not sure. I will check the pkcs11's configuration now, but it used
> to work.
But I have no knowledge about such crypto hardware, therefore
I am not sure if I can be of any more help. Maybe you could
provide the needed packages, libraries and configuration steps
that are needed to use such a device of yours when starting with
a fresh debian installation?
Kind regards,
Bernhard
Bug#969559: Info received (Bug#969559: curl segmentation fauls on any https URL)
I have checked my pkcs11 device and it is functioning properly, but curl still crashes. Fortunately I can just use 'wget' until this is fixed. -- Bruce Momjian https://momjian.us EnterpriseDB https://enterprisedb.com The usefulness of a cup is in its emptiness, Bruce Lee
Bug#969559: curl segmentation fauls on any https URL
On Sat, Sep 5, 2020 at 03:50:20PM +0200, Bernhard Übelacker wrote:
> Dear Maintainer,
> I tried to reproduce this fault, but did not get a segfault.
>
> However, I think the backtrace points to these lines:
>
> (gdb) bt
> #0 0x7769dbce in int_ctx_new () at ../crypto/evp/pmeth_lib.c:160
> #1 0x7769dcfa in EVP_PKEY_CTX_new () at
> ../crypto/evp/pmeth_lib.c:245
> #2 0x77698d44 in do_sigver_init () at ../crypto/evp/m_sigver.c:29
> #3 0x77698eab in EVP_DigestVerifyInit () at
> ../crypto/evp/m_sigver.c:97
> #4 0x775bc7d2 in ASN1_item_verify () at
> ../crypto/asn1/a_verify.c:148
> #5 0x77722490 in X509_verify () at ../crypto/x509/x_all.c:26
> ...
>
>
> https://sources.debian.org/src/openssl/1.1.1d-0+deb10u3/crypto/evp/pmeth_lib.c/#L160
>
> 159 if (pmeth->init) {
> 160 if (pmeth->init(ret) <= 0) {
> 161 ret->pmeth = NULL;
>
> As there is a check for pmeth->init being non-null, I guess
> it contains for some reason an invalid pointer.
>
>
> @Bruce Momjian,
> maybe you could install the following debug symbols packages
> `curl-dbgsym libcurl4-dbgsym libssl1.1-dbgsym` from the dbgsym
> repository described here:
>
> https://wiki.debian.org/HowToGetABacktrace#Installing_the_debugging_symbols
>
> Then run a new gdb session and when the segfault appears
> please run these commands in gdb:
> print pmeth->init
> bt full 5
Sure, here it is:
(gdb) run https://google.com
Starting program: /usr/bin/curl https://google.com
[Thread debugging using libthread_db enabled]
Using host libthread_db library
"/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x76730700 (LWP 30481)]
[Thread 0x76730700 (LWP 30481) exited]
Thread 1 "curl" received signal SIGSEGV, Segmentation fault.
0x77679bce in int_ctx_new (pkey=pkey@entry=0x556035a0,
e=0x555bd8d0, e@entry=0x0, id=, id@entry=-1) at
../crypto/evp/pmeth_lib.c:160
160 ../crypto/evp/pmeth_lib.c: No such file or directory.
(gdb) print pmeth->init
$1 = (int (*)(EVP_PKEY_CTX *)) 0xf0e0d0c0b0a0908
(gdb) bt full 5
#0 0x77679bce in int_ctx_new (pkey=pkey@entry=0x556035a0,
e=0x555bd8d0, e@entry=0x0, id=, id@entry=-1) at
../crypto/evp/pmeth_lib.c:160
ret = 0x55609810
pmeth = 0x555eaaf0
#1 0x77679cfa in EVP_PKEY_CTX_new
(pkey=pkey@entry=0x556035a0, e=e@entry=0x0) at ../crypto/evp/pmeth_lib.c:245
No locals.
#2 0x77674d44 in do_sigver_init (ctx=ctx@entry=0x556034c0,
pctx=pctx@entry=0x0, type=type@entry=0x777b1fc0 , e=e@entry=0x0,
pkey=pkey@entry=0x556035a0, ver=ver@entry=1)
at ../crypto/evp/m_sigver.c:29
No locals.
#3 0x77674eab in EVP_DigestVerifyInit
(ctx=ctx@entry=0x556034c0, pctx=pctx@entry=0x0,
type=type@entry=0x777b1fc0 , e=e@entry=0x0,
pkey=pkey@entry=0x556035a0)
at ../crypto/evp/m_sigver.c:97
No locals.
#4 0x775987d2 in ASN1_item_verify (it=0x777c3e80
, a=a@entry=0x555ff698,
signature=signature@entry=0x555ff6a8, asn=asn@entry=0x555ff610,
pkey=0x556035a0)
at ../crypto/asn1/a_verify.c:148
type = 0x777b1fc0
ctx = 0x556034c0
buf_in = 0x0
ret = -1
inl = 0
mdnid = 672
pknid = 6
inll = 0
(More stack frames follow...)
I also got this output:
gdb) print *pmeth
$8 = {pkey_id = 50462976, flags = 117835012, init = 0xf0e0d0c0b0a0908,
copy = 0x1716151413121110, cleanup = 0x1f1e1d1c1b1a1918, paramgen_init =
0x98c476a19fc273a5, paramgen = 0x9cc072a593ce7fa9,
keygen_init = 0xdabe4402cda85116, keygen = 0xdeba4006c1a45d1a,
sign_init = 0x681bf10ff0df87ae, sign = 0x6715fc03fbd58ea6, verify_init =
0x924fa56f48f1e16d, verify = 0x8d51b87353ebf875,
verify_recover_init = 0x1799a7c97f8256c6, verify_recover =
0x8b59d56cec4c296f, signctx_init = 0xe7754752753ae23d, signctx =
0x39cf0754b49ebf27, verifyctx_init = 0x48097bc25f90dc0b,
verifyctx = 0x2f1c87c1a44552ad, encrypt_init = 0x87d3b21760a6f545,
encrypt = 0xa820a64334d0d30, decrypt_init = 0x54feb4be1cf7cf7c, decrypt =
0xdfa761d2f0bbe613, derive_init = 0x7929a8e7fefa1af0,
derive = 0x40e6afb34a64a5d7, ctrl = 0x2500f59b71fe4125, ctrl_str =
0xa1c725ad5bb1388, digestsign = 0xe04ff2a999665a4e, digestverify =
0xeacdf8cdaa2b577e, check = 0xe97909bfcc79fc24,
public_check = 0x36de686d3cc21a37, param_check = 0xd, digest_custom =
0x7758ac80 }
(gdb) print pmeth->init[0]
Cannot access memory at address 0xf0e0d0c0b0a0908
(gdb) print *(pmeth->init)
Cannot access memory at address
Bug#969559: curl segmentation fauls on any https URL
Dear Maintainer,
I tried to reproduce this fault, but did not get a segfault.
However, I think the backtrace points to these lines:
(gdb) bt
#0 0x7769dbce in int_ctx_new () at ../crypto/evp/pmeth_lib.c:160
#1 0x7769dcfa in EVP_PKEY_CTX_new () at
../crypto/evp/pmeth_lib.c:245
#2 0x77698d44 in do_sigver_init () at ../crypto/evp/m_sigver.c:29
#3 0x77698eab in EVP_DigestVerifyInit () at
../crypto/evp/m_sigver.c:97
#4 0x775bc7d2 in ASN1_item_verify () at
../crypto/asn1/a_verify.c:148
#5 0x77722490 in X509_verify () at ../crypto/x509/x_all.c:26
...
https://sources.debian.org/src/openssl/1.1.1d-0+deb10u3/crypto/evp/pmeth_lib.c/#L160
159 if (pmeth->init) {
160 if (pmeth->init(ret) <= 0) {
161 ret->pmeth = NULL;
As there is a check for pmeth->init being non-null, I guess
it contains for some reason an invalid pointer.
@Bruce Momjian,
maybe you could install the following debug symbols packages
`curl-dbgsym libcurl4-dbgsym libssl1.1-dbgsym` from the dbgsym
repository described here:
https://wiki.debian.org/HowToGetABacktrace#Installing_the_debugging_symbols
Then run a new gdb session and when the segfault appears
please run these commands in gdb:
print pmeth->init
bt full 5
Kind regards,
Bernhard
# Buster/stable amd64 qemu VM
apt update
apt dist-upgrade
apt install systemd-coredump curl gdb
curl https://google.com
dpkg -l curl libc6 libcurl4 zlib1g libssl1.1
ii curl7.64.0-4+deb10u1 amd64command line tool for
transferring data with URL syntax
ii libc6:amd64 2.28-10 amd64GNU C Library: Shared
libraries
ii libcurl4:amd64 7.64.0-4+deb10u1 amd64easy-to-use client-side URL
transfer library (OpenSSL flavour)
ii libssl1.1:amd64 1.1.1d-0+deb10u3 amd64Secure Sockets Layer toolkit
- shared libraries
ii zlib1g:amd641:1.2.11.dfsg-1 amd64compression library - runtime
benutzer@debian:~$ curl https://google.com
301 Moved
301 Moved
The document has moved
https://www.google.com/;>here.
gdb -q --args curl https://google.com
b ASN1_item_verify
y
run
disassemble ASN1_item_verify
b EVP_DigestVerifyInit
cont
...
generate-core-file /tmp/core
(gdb) bt
#0 0x7769dbce in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
#1 0x77698d44 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
#2 0x775bc7d2 in ASN1_item_verify () from
/lib/x86_64-linux-gnu/libcrypto.so.1.1
#3 0x7771cfb4 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
#4 0x7771edd6 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
#5 0x7771f416 in X509_verify_cert () from
/lib/x86_64-linux-gnu/libcrypto.so.1.1
#6 0x7782fb88 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.1
#7 0x778510f3 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.1
#8 0x778536c5 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.1
#9 0x7784d143 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.1
#10 0x77838f34 in SSL_do_handshake () from
/lib/x86_64-linux-gnu/libssl.so.1.1
#11 0x77fa3240 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#12 0x77fa53f0 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#13 0x77fa61da in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#14 0x77f4d462 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#15 0x77f6f6fe in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#16 0x77f70aa9 in curl_multi_perform () from
/lib/x86_64-linux-gnu/libcurl.so.4
#17 0x77f67642 in curl_easy_perform () from
/lib/x86_64-linux-gnu/libcurl.so.4
#18 0x55569f30 in ?? ()
#19 0x5556b42a in ?? ()
#20 0xd8c4 in ?? ()
#21 0x77b5c09b in __libc_start_main (main=0xd770, argc=2,
argv=0x7fffe608, init=, fini=,
rtld_fini=, stack_end=0x7fffe5f8)
at ../csu/libc-start.c:308
#22 0xd9da in ?? ()
apt install curl-dbgsym libcurl4-dbgsym libssl1.1-dbgsym
gdb -q /usr/bin/curl --core /tmp/core
set width 0
set pagination off
(gdb) bt
#0 0x7769dbce in int_ctx_new (pkey=pkey@entry=0x55601a10,
e=e@entry=0x0, id=, id@entry=-1) at ../crypto/evp/pmeth_lib.c:160
#1 0x7769dcfa in EVP_PKEY_CTX_new (pkey=pkey@entry=0x55601a10,
e=e@entry=0x0) at ../crypto/evp/pmeth_lib.c:245
#2 0x77698d44 in do_sigver_init (ctx=ctx@entry=0x55601930,
pctx=pctx@entry=0x0, type=type@entry=0x777d5fc0 , e=e@entry=0x0,
pkey=pkey@entry=0x55601a10, ver=ver@entry=1) at ../crypto/evp/m_sigver.c:29
#3 0x77698eab in EVP_DigestVerifyInit (ctx=ctx@entry=0x55601930,
pctx=pctx@entry=0x0, type=type@entry=0x777d5fc0 , e=e@entry=0x0,
pkey=pkey@entry=0x55601a10) at ../crypto/evp/m_sigver.c:97
#4 0x775bc7d2 in ASN1_item_verify (it=0x777e7e80 ,
a=a@entry=0x555fda18, signature=signature@entry=0x555fda28,
Bug#969559: curl segmentation fauls on any https URL
Package: curl Version: 7.64.0-4+deb10u1 Severity: grave Justification: renders package unusable Dear Maintainer, *** Reporter, please consider answering these questions, where appropriate *** Simply type: $ curl https://google.com Segmentation fault or use any https URL. Here is a backtrace: 0x77679bce in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (gdb) bt #0 0x77679bce in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 #1 0x77674d44 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 #2 0x775987d2 in ASN1_item_verify () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 #3 0x776f8fb4 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 #4 0x776fadd6 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 #5 0x776fb416 in X509_verify_cert () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 #6 0x7780bb88 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1 #7 0x7782d0f3 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1 #8 0x7782f6c5 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1 #9 0x77829143 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1 #10 0x77814f34 in SSL_do_handshake () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1 #11 0x77f7f240 in ?? () from /usr/lib/x86_64-linux-gnu/libcurl.so.4 #12 0x77f813f0 in ?? () from /usr/lib/x86_64-linux-gnu/libcurl.so.4 #13 0x77f821da in ?? () from /usr/lib/x86_64-linux-gnu/libcurl.so.4 #14 0x77f29462 in ?? () from /usr/lib/x86_64-linux-gnu/libcurl.so.4 #15 0x77f4b6fe in ?? () from /usr/lib/x86_64-linux-gnu/libcurl.so.4 #16 0x77f4caa9 in curl_multi_perform () from /usr/lib/x86_64-linux-gnu/libcurl.so.4 #17 0x77f43642 in curl_easy_perform () from /usr/lib/x86_64-linux-gnu/libcurl.so.4 #18 0x55569f30 in ?? () #19 0x5556b42a in ?? () #20 0xd8c4 in ?? () #21 0x77b3809b in __libc_start_main (main=0xd770, argc=2, argv=0x7fffded8, init=, fini=, rtld_fini=, stack_end=0x7fffdec8) at ../csu/libc-start.c:308 #22 0xd9da in ?? () *** End of the template - remove these template lines *** -- System Information: Debian Release: 10.5 APT prefers stable APT policy: (500, 'stable') Architecture: amd64 (x86_64) Kernel: Linux 4.19.0-10-amd64 (SMP w/16 CPU cores) Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages curl depends on: ii libc6 2.28-10 ii libcurl4 7.64.0-4+deb10u1 ii zlib1g1:1.2.11.dfsg-1 curl recommends no packages. curl suggests no packages. -- no debconf information
