Re: 2.2.12 and rsa/ecdsa cert regression (crash on startup) ?
On Fri, Apr 02, 2021 at 08:22:16AM +, Jarno Huuskonen wrote: > Thanks William, with 2.2.12 + patch haproxy starts and serves both rsa/ecdsa > certs. > > I'm attaching a regtest patch that attempts to check that haproxy starts > with multi-bundle cert and serves both rsa/ecdsa certs. > (the test itself is not well tested so handle with care :) > (for example I'm not sure if ciphers ECDHE-RSA-AES128-GCM-SHA256 / ECDHE- > ECDSA-AES256-GCM-SHA384 are needed/usefull and work with boring/libressl). > > -Jarno > Thanks, that's a good idea, but I don't think I will integrate it as it is. I'll duplicate the set_ssl_cert.vtc one to have the full CLI part tested with it, since we don't have this test too. I'll probably emit a new 2.2 this evening with the fix and the reg-test. Thanks! -- William Lallemand
Re: 2.2.12 and rsa/ecdsa cert regression (crash on startup) ?
Hello,
On Thu, 2021-04-01 at 16:03 +0200, William Lallemand wrote:
> On Thu, Apr 01, 2021 at 02:26:07PM +0200, William Lallemand wrote:
> > On Thu, Apr 01, 2021 at 10:19:31AM +, Jarno Huuskonen wrote:
> > > Hello,
> > >
> > > I'm seeing a regression with 2.2.12 and using rsa and ecdsa certs on
> > > bind.
> > > (cert1.pem.ecdsa
> > > cert1.pem.ecdsa.ocsp
> > > cert1.pem.ocsp
> > > cert1.pem.rsa
> > > cert1.pem.rsa.ocsp
> > > )
> > >
> >
> > Thanks for the report, I can reproduce the problem, I'm investigating.
> >
>
> Could you try the attached patch?
Thanks William, with 2.2.12 + patch haproxy starts and serves both rsa/ecdsa
certs.
I'm attaching a regtest patch that attempts to check that haproxy starts
with multi-bundle cert and serves both rsa/ecdsa certs.
(the test itself is not well tested so handle with care :)
(for example I'm not sure if ciphers ECDHE-RSA-AES128-GCM-SHA256 / ECDHE-
ECDSA-AES256-GCM-SHA384 are needed/usefull and work with boring/libressl).
-Jarno
--
Jarno Huuskonen
From b0aec4e620404ea38dae0fe50046ab0f2cb48398 Mon Sep 17 00:00:00 2001
From: Jarno Huuskonen
Date: Fri, 2 Apr 2021 09:39:39 +0300
Subject: [PATCH] REGTESTS: ssl: Minimal multi-bundle certificates bind check.
This adds minimal test to check that multi-bundle (rsa/ecdsa) bind
works (for BUG/MEDIUM: ssl: ckch_inst->ctx not assigned with
multi-bundle certificates) and both rsa/ecdsa certs are served.
---
reg-tests/ssl/rsa_and_ecdsa_bind.pem.ecdsa | 1 +
reg-tests/ssl/rsa_and_ecdsa_bind.pem.rsa | 1 +
reg-tests/ssl/set_ssl_cert.vtc | 31 ++
3 files changed, 33 insertions(+)
create mode 12 reg-tests/ssl/rsa_and_ecdsa_bind.pem.ecdsa
create mode 12 reg-tests/ssl/rsa_and_ecdsa_bind.pem.rsa
diff --git a/reg-tests/ssl/rsa_and_ecdsa_bind.pem.ecdsa b/reg-tests/ssl/rsa_and_ecdsa_bind.pem.ecdsa
new file mode 12
index 0..16276ab88
--- /dev/null
+++ b/reg-tests/ssl/rsa_and_ecdsa_bind.pem.ecdsa
@@ -0,0 +1 @@
+ecdsa.pem
\ No newline at end of file
diff --git a/reg-tests/ssl/rsa_and_ecdsa_bind.pem.rsa b/reg-tests/ssl/rsa_and_ecdsa_bind.pem.rsa
new file mode 12
index 0..1b7cb2c3c
--- /dev/null
+++ b/reg-tests/ssl/rsa_and_ecdsa_bind.pem.rsa
@@ -0,0 +1 @@
+common.pem
\ No newline at end of file
diff --git a/reg-tests/ssl/set_ssl_cert.vtc b/reg-tests/ssl/set_ssl_cert.vtc
index a606b477d..022e8d6c3 100644
--- a/reg-tests/ssl/set_ssl_cert.vtc
+++ b/reg-tests/ssl/set_ssl_cert.vtc
@@ -16,6 +16,9 @@
# any SNI. The test consists in checking that the used certificate is the right one after
# updating it via a "set ssl cert" call.
#
+# listen other-rsaecdsa-ssl / other-rsaecdsa checks that haproxy can bind and serve multi-bundle
+# (rsa/ecdsa) certificate.
+#
# If this test does not work anymore:
# - Check that you have socat
@@ -74,6 +77,21 @@ haproxy h1 -conf {
bind "${tmpdir}/other-ssl.sock" ssl crt-list ${testdir}/set_default_cert.crt-list
server s1 ${s1_addr}:${s1_port}
+# check that we can bind with: rsa_and_ecdsa_bind.pem.rsa / rsa_and_ecdsa_bind.pem.ecdsa
+listen other-rsaecdsa-ssl
+bind "${tmpdir}/other-rsaecdsa-ssl.sock" ssl crt ${testdir}/rsa_and_ecdsa_bind.pem
+http-request deny deny_status 200
+server s1 ${s1_addr}:${s1_port}
+
+# use other-rsa_ecdsa-ssl to check both rsa and ecdsa certs are returned
+listen other-rsaecdsa
+bind "fd@${otherrsaecdsa}"
+http-response set-header X-SSL-Server-SHA1 %[ssl_s_sha1,hex]
+use-server s1rsa if { path_end -i .rsa }
+use-server s1ecdsa if { path_end -i .ecdsa }
+server s1rsa "${tmpdir}/other-rsaecdsa-ssl.sock" ssl verify none force-tlsv12 sni str(www.test1.com) ciphers ECDHE-RSA-AES128-GCM-SHA256
+server s1ecdsa "${tmpdir}/other-rsaecdsa-ssl.sock" ssl verify none force-tlsv12 sni str(localhost) ciphers ECDHE-ECDSA-AES256-GCM-SHA384
+
} -start
@@ -202,3 +220,16 @@ client c1 -connect ${h1_clearlst_sock} {
expect resp.http.X-SSL-Server-SHA1 == "9DC18799428875976DDE706E9956035EE88A4CB3"
expect resp.status == 200
} -run
+
+# Check that other-rsaecdsa serves both rsa and ecdsa certificate
+client c1 -connect ${h1_otherrsaecdsa_sock} {
+txreq -req GET -url /dummy.rsa
+rxresp
+expect resp.http.X-SSL-Server-SHA1 == "2195C9F0FD58470313013FC27C1B9CF9864BD1C6"
+expect resp.status == 200
+
+txreq -req GET -url /dummy.ecdsa
+rxresp
+expect resp.http.X-SSL-Server-SHA1 == "A490D069DBAFBEE66DE434BEC34030ADE8BCCBF1"
+expect resp.status == 200
+} -run
--
2.26.3
Re: 2.2.12 and rsa/ecdsa cert regression (crash on startup) ?
On Thu, Apr 01, 2021 at 02:26:07PM +0200, William Lallemand wrote:
> On Thu, Apr 01, 2021 at 10:19:31AM +, Jarno Huuskonen wrote:
> > Hello,
> >
> > I'm seeing a regression with 2.2.12 and using rsa and ecdsa certs on bind.
> > (cert1.pem.ecdsa
> > cert1.pem.ecdsa.ocsp
> > cert1.pem.ocsp
> > cert1.pem.rsa
> > cert1.pem.rsa.ocsp
> > )
> >
>
> Thanks for the report, I can reproduce the problem, I'm investigating.
>
Could you try the attached patch?
Thanks
--
William Lallemand
>From 3adeb8baf45c2f775848770b349cfa5e3fdd561b Mon Sep 17 00:00:00 2001
From: William Lallemand
Date: Thu, 1 Apr 2021 15:48:21 +0200
Subject: [PATCH] BUG/MEDIUM: ssl: ckch_inst->ctx not assigned with
multi-bundle certificates
When backporting patch 8218aed ("BUG/MINOR: ssl: Fix update of default
certificate") in 2.2, a regression was introduced. The 2.2
multi-certificate loading code does not have the same code path and this
part was not modified, introducing a segfault when trying to start
haproxy with a multi-certificate bundle.
This patch fixes the problem by setting the ckch_inst->ctx variable in
ckch_inst_new_load_multi_store().
No backport needed, 2.2 only.
---
src/ssl_sock.c | 13 ++---
1 file changed, 10 insertions(+), 3 deletions(-)
diff --git a/src/ssl_sock.c b/src/ssl_sock.c
index 14311370f3..627de34761 100644
--- a/src/ssl_sock.c
+++ b/src/ssl_sock.c
@@ -3340,9 +3340,16 @@ int ckch_inst_new_load_multi_store(const char *path, struct ckch_store *ckchs,
/* Mark a default context if none exists, using the ctx that has the most shared keys */
- if (!bind_conf->default_ctx) {
- for (i = SSL_SOCK_POSSIBLE_KT_COMBOS - 1; i >= 0; i--) {
- if (key_combos[i].ctx) {
+ for (i = SSL_SOCK_POSSIBLE_KT_COMBOS - 1; i >= 0; i--) {
+ if (key_combos[i].ctx) {
+ if (!ckch_inst->ctx) {
+/* Always keep a reference to the newly constructed SSL_CTX in the
+ * instance. This way if the instance has no SNIs, the SSL_CTX will
+ * still be linked. */
+SSL_CTX_up_ref(key_combos[i].ctx);
+ckch_inst->ctx = key_combos[i].ctx;
+ }
+ if (!bind_conf->default_ctx) {
bind_conf->default_ctx = key_combos[i].ctx;
bind_conf->default_ssl_conf = ssl_conf;
ckch_inst->is_default = 1;
--
2.26.3
Re: 2.2.12 and rsa/ecdsa cert regression (crash on startup) ?
On Thu, Apr 01, 2021 at 10:19:31AM +, Jarno Huuskonen wrote: > Hello, > > I'm seeing a regression with 2.2.12 and using rsa and ecdsa certs on bind. > (cert1.pem.ecdsa > cert1.pem.ecdsa.ocsp > cert1.pem.ocsp > cert1.pem.rsa > cert1.pem.rsa.ocsp > ) > Thanks for the report, I can reproduce the problem, I'm investigating. -- William Lallemand
2.2.12 and rsa/ecdsa cert regression (crash on startup) ?
Hello,
I'm seeing a regression with 2.2.12 and using rsa and ecdsa certs on bind.
(cert1.pem.ecdsa
cert1.pem.ecdsa.ocsp
cert1.pem.ocsp
cert1.pem.rsa
cert1.pem.rsa.ocsp
)
haproxy crashes on startup:
(gdb) bt
#0 0x7710f159 in SSL_CTX_up_ref () from /lib64/libssl.so.1.1
#1 0x0042e1a3 in ssl_sock_load_cert_sni (ckch_inst=0x9adf30,
bind_conf=bind_conf@entry=0x9a6590) at src/ssl_sock.c:2866
#2 0x0043186f in ssl_sock_load_ckchs (path=,
ssl_conf=, sni_filter=,
fcount=, err=0x7fffdb68, ckch_inst=0x7fffba08,
bind_conf=0x9a6590, ckchs=0x9a6ad0) at src/ssl_sock.c:3587
#3 ssl_sock_load_ckchs (path=, ckchs=0x9a6ad0,
bind_conf=0x9a6590, ssl_conf=, sni_filter=,
fcount=, ckch_inst=0x7fffba08, err=0x7fffdb68) at
src/ssl_sock.c:3572
#4 0x00431b84 in ssl_sock_load_cert (path=path@entry=0x9703b8
"/etc/haproxy/ssl/cert1.pem",
bind_conf=bind_conf@entry=0x9a6590, err=err@entry=0x7fffdb68) at
src/ssl_sock.c:3740
#5 0x0043bfbe in bind_parse_crt (args=0x7fffdc10,
cur_arg=, px=, conf=0x9a6590,
err=0x7fffdb68)
at src/cfgparse-ssl.c:645
#6 0x0048e57b in cfg_parse_listen (file=0x99b060
"/etc/haproxy/haproxy.cfg", linenum=116, args=0x7fffdc10, kwm=)
at src/cfgparse-listen.c:605
#7 0x0047fcab in readcfgfile (file=0x99b060
"/etc/haproxy/haproxy.cfg") at src/cfgparse.c:2087
#8 0x0052dd7c in init (argc=, argc@entry=6,
argv=, argv@entry=0x7fffe2f8) at src/haproxy.c:2050
#9 0x0041e3ca in main (argc=6, argv=0x7fffe2f8) at
src/haproxy.c:3180
(This is on rhel8:
HA-Proxy version 2.2.12-a723e77 2021/03/31 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2
2025.
Known bugs: http://www.haproxy.org/bugs/bugs-2.2.12.html
Running on: Linux 4.18.0-240.15.1.el8_3.x86_64 #1 SMP Wed Feb 3 03:12:15 EST
2021 x86_64
Build options :
TARGET = linux-glibc
CPU = generic
CC = gcc
CFLAGS = -O2 -g -Wall -Wextra -Wdeclaration-after-statement -fwrapv -Wno-
unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-
missing-field-initializers -Wno-stringop-overflow -Wno-cast-function-type -
Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -
Wnull-dereference
OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_GETADDRINFO=1 USE_OPENSSL=1
USE_ZLIB=1 USE_SYSTEMD=1
DEBUG =
Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT +PCRE2 +PCRE2_JIT
+POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +BACKTRACE -STATIC_PCRE -
STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H
+GETADDRINFO +OPENSSL -LUA +FUTEX +ACCEPT4 -CLOSEFROM +ZLIB -SLZ
+CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -
OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=2).
Built with OpenSSL version : OpenSSL 1.1.1g FIPS 21 Apr 2020
Running on OpenSSL version : OpenSSL 1.1.1g FIPS 21 Apr 2020
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"),
raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Built with PCRE2 version : 10.32 2018-09-10
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 8.3.1 20191121 (Red Hat 8.3.1-5)
Built with the Prometheus exporter as a service
)
Crash doesn't happen if I use just ecdsa or rsa cert file:
cert1.pem
cert1.pem.ocsp
(Crash also doesn't happen on 2.2.10, 2.2.11, 2.3.9 and 2.4dev(haproxy-ss-
20210401))
Git bisect points to this commit:
commit b87c8899d872843c12b3516ad51da84b22538d91
BUG/MINOR: ssl: Fix update of default certificate
Something like this config should be able to reproduce:
frontend FE_crash
bind ipv4@:443 name crashv4ssl ssl crt /etc/haproxy/ssl/cert1.pem
alpn h2,http/1.1
bind ipv6@:::443 name crashv6ssl ssl crt /etc/haproxy/ssl/cert1.pem
alpn h2,http/1.1
modehttp
default_backend BE_crash
backend BE_crash
server crash 192.168.1.105:8081 id 1 check
(And cert1.pem is multiple files:
cert1.pem.ecdsa
cert1.pem.ecdsa.ocsp
cert1.pem.ocsp
cert1.pem.rsa
cert1.pem.rsa.ocsp
)
-Jarno
--
Jarno Huuskonen
