Hello, We are experiencing for the past weeks a segfault on haproxy processes when reloading haproxy.
Each thread generates a coredump. Fortunately, this is the old process that crashes, so there is no production impact. The same behaviour happens with haproxy 2.0.25 compiled with OpenSSL 1.1.1l Core was generated by `/root/haproxy -sf 42184 42185 42186 42187 42188 42189 42190 42191 42192 42193 4'. Program terminated with signal 6, Aborted. #0 0x00007f414c4d4495 in raise () from /lib64/libc.so.6 (gdb) bt full #0 0x00007f414c4d4495 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f414c4d5c75 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x00007f414c5123a7 in __libc_message () from /lib64/libc.so.6 No symbol table info available. #3 0x00007f414c517dee in malloc_printerr () from /lib64/libc.so.6 No symbol table info available. #4 0x00007f414c51ac3d in _int_free () from /lib64/libc.so.6 No symbol table info available. #5 0x000000000047149c in ssl_sock_free_ssl_conf (conf=0x6f3d440) at src/ssl_sock.c:3766 No locals. #6 0x0000000000479f3d in ssl_sock_free_ssl_conf (conf=<optimized out>) at src/ssl_sock.c:3764 No locals. #7 ssl_sock_free_all_ctx (bind_conf=bind_conf@entry=0x6f3d160) at src/ssl_sock.c:5144 node = 0x6f4ac60 sni = 0x6f4ac40 #8 0x000000000047a4a1 in ssl_sock_destroy_bind_conf (bind_conf=0x6f3d160) at src/ssl_sock.c:5176 No locals. #9 0x000000000050aae8 in deinit () at src/haproxy.c:2673 haproxy -vv output : Build options : TARGET = linux-glibc CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wno-stringop-overflow -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference OPTIONS = USE_THREAD=0 USE_STATIC_PCRE=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_NS= Feature list : -51DEGREES +ACCEPT4 -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL +EPOLL -EVPORTS +FUTEX +GETADDRINFO -KQUEUE +LIBCRYPT +LINUX_SPLICE +LINUX_TPROXY +LUA -MY_ACCEPT4 -MY_EPOLL -MY_SPLICE +NETFILTER -NS -OBSOLETE_LINKER +OPENSSL -PCRE -PCRE2 -PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PRIVATE_CACHE -PTHREAD_PSHARED -REGPARM +RT -SLZ +STATIC_PCRE -STATIC_PCRE2 -SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -VSYSCALL -WURFL +ZLIB Default settings : bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with multi-threading support (MAX_THREADS=64, default=32). Built with OpenSSL version : OpenSSL 1.1.1t 7 Feb 2023 Running on OpenSSL version : OpenSSL 1.1.1t 7 Feb 2023 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3 Built with Lua version : Lua 5.4.4 Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Built with zlib version : 1.2.13 Running on zlib version : 1.2.13 Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Built with PCRE version : 8.45 2021-06-15 Running on PCRE version : 8.45 2021-06-15 PCRE library supports JIT : no (USE_PCRE_JIT not set) Encrypted password support via crypt(3): yes Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Available multiplexer protocols : (protocols marked as <default> cannot be specified using 'proto' keyword) h2 : mode=HTX side=FE|BE mux=H2 h2 : mode=HTTP side=FE mux=H2 <default> : mode=HTX side=FE|BE mux=H1 <default> : mode=TCP|HTTP side=FE|BE mux=PASS Available services : none Available filters : [SPOE] spoe [COMP] compression [CACHE] cache [TRACE] trace I understand this is linked to some SSL config file, but fail to find what. The haproxy config file is very large (> 1000 lines) with dozens of frontend/backend. It will help me to know which one is it. Any "gdb" advice how to narrow the issue down ? Thanks, Olivier