Hi list,
we have some weird issues now, the second time, that *some* SSL sockets
seem to be broken as well as stats sockets.
HTTP seems to work fine, still, SSL ones are broken however. It happened
at least on 2.1.3 and *perhaps* on 2.1.2 as well. We're not sure whether
the first time was on 2.1.2 or 2.1.3.
The one that failed today was updated yesterday, so HAProxy has an
uptime of about 24h.
We're using threads. default + HTTP is using 1 thread, 1 is dedicated
for a TCP listener/Layer-4, one is for RSA only and all the rest is for
ECC.
A curl looks like:
# curl -kvs https://127.0.0.1:17920
* Rebuilt URL to: https://127.0.0.1:17920/
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 17920 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection:
ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* Unknown SSL protocol error in connection to 127.0.0.1:17920
* Curl_http_done: called premature == 1
* stopped the pause stream!
* Closing connection 0
Mar 27 14:26:25 somelb haproxy[21313]: 127.0.0.1:6762
[27/Mar/2020:14:26:25.972] layer4stuff layer4stuff/socket-17920-ecc
5/-1/4 0 SC 2/1/0/0/3 0/0
The config looks like this:
listen layer4stuff
mode tcp
bind :17920 process 1/2
bind :::17920 process 1/2
log global
option tcplog
# some early L4-ACLs
use-server socket-17920-rsa if !REQ_TLS_HAS_ECC
server socket-17920-rsa unix@/var/run/haproxy-17920-rsa.sock
send-proxy-v2
use-server socket-17920-ecc if REQ_TLS_HAS_ECC
server socket-17920-ecc unix@/var/run/haproxy-17920-ecc.sock
send-proxy-v2
listen tls-splitter
mode tcp
log global
option tcplog
bind unix@/var/run/haproxy-17920-rsa.sock accept-proxy user
haproxy group root mode 600 ssl crt
/etc/haproxy/haproxy.d/frontend/17920-RSA/ no-tlsv10 no-tlsv11 ciphers
ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256
alpn h2,http/1.1 process 1/3
bind unix@/var/run/haproxy-17920-ecc.sock accept-proxy user
haproxy group root mode 600 ssl crt
/etc/haproxy/haproxy.d/frontend/17920-ECC/ no-tlsv10 no-tlsv11 ciphers
ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256
alpn h2,http/1.1 process 1/4-
use-server socket-17910-h2 if { ssl_fc_alpn h2 }
server socket-17910-h2 unix@/var/run/haproxy-17910-h2.sock
send-proxy-v2
use-server socket-17910 if !{ ssl_fc_alpn h2 }
server socket-17910 unix@/var/run/haproxy-17910.sock
send-proxy-v2
frontend somefrontend
bind :17910 process 1/1
bind :::17910 process 1/1
bind unix@/var/run/haproxy-17910-h2.sock id 317910 accept-proxy
user haproxy group root mode 600 proto h2 process 1/1 # H/2 TLS uplink
bind unix@/var/run/haproxy-17910.sock id 217910 accept-proxy
user haproxy group root mode 600 process 1/1 # TLS uplink
acl is_https so_id 217910
acl is_https so_id 317910
mode http
# Some more stuff here
We've separated this broken loadbalancer for now, so if you want me to
do anything with the running process, let me know.
The above is basically the same for every other site. Our config is
quite huge.
Since the stats socket is broken as well, it may be something related to
the UNIX sockets.
The stats TCP socket is still working.
# echo help | socat - /var/run/haproxy.stat
2020/03/27 14:23:54 socat[21715] E connect(5, AF=1
"/var/run/haproxy.stat", 23): Connection refused
# ps aux|grep haproxy
root 20799 0.0 0.0 49624 7608 ? Ss Mar26 1:10
/usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
-sf 21313
haproxy 21313 5.4 0.7 887168 236788 ? Sl 13:00 4:59
/usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
-sf 15635
The problem ocurred arount 13:40 (CET, in case it matters at some point)
Any ideas so far?
I'm currently trying to find something useful in the log around that
time, whether it was a reload or something.
Debian 9.12 currently
# haproxy -vv
HA-Proxy version 2.1.3 2020/02/12 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2021.
Known bugs: http://www.haproxy.org/bugs/bugs-2.1.3.html
Build options :
TARGET = linux-glibc
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
-fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter
-Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered
-Wno-missing-field-initializers -Wtype-limits -Wshift-negative-value
-Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
OPTIONS = USE_PCRE=1 USE_PCRE_JIT= USE_LIBCRYPT=1 USE_OPENSSL=1
USE_LUA=1 USE_ZLIB=1 USE_NS= USE_SYSTEMD=1
Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER +PCRE
-PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD
-PTHREAD_PSHARED -REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY
+LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO
+OPENSSL +LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO
-NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER
+PRCTL +THREAD_DUMP -EVPORTS
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=8).
Built with OpenSSL version : OpenSSL 1.1.0l 10 Sep 2019
Running on OpenSSL version : OpenSSL 1.1.0l 10 Sep 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.3
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with the Prometheus exporter as a service
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto'
keyword)
h2 : mode=HTTP side=FE|BE mux=H2
fcgi : mode=HTTP side=BE mux=FCGI
<default> : mode=HTTP side=FE|BE mux=H1
<default> : mode=TCP side=FE|BE mux=PASS
Available services :
prometheus-exporter
Available filters :
[SPOE] spoe
[CACHE] cache
[FCGI] fcgi-app
[TRACE] trace
[COMP] compression
--
Regards,
Christian Ruppert
