Le 5/2/23 à 13:58, Maciej Zdeb a écrit :
Hi,
I'm experiencing a segmentation fault caused by adding "disabled"
(http://docs.haproxy.org/2.4/configuration.html#4-disabled
<http://docs.haproxy.org/2.4/configuration.html#4-disabled>) to the frontend
section of haproxy configuration file. That frontend does not handle any traffic
yet.
The "disabled" keyword was used on other frontends on the same HAProxy instance
before without any issue. The difference is that this time it was added to
frontend with spoe filter. This probably causes a crash when processing requests
on a different heavily used frontend (also spoe enabled).
It crashes after 2-10s under production traffic. I couldn't replicate it with a
synthetic test, however I have a coredump and packages with HAProxy binary and
debug symbols.
(gdb) bt
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=140527117211200)
at ./nptl/pthread_kill.c:44
#1 __pthread_kill_internal (signo=6, threadid=140527117211200) at
./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=140527117211200, signo=signo@entry=6) at
./nptl/pthread_kill.c:89
#3 0x00007fcf0881a476 in __GI_raise (sig=sig@entry=6) at
../sysdeps/posix/raise.c:26
#4 0x00007fcf088007f3 in __GI_abort () at ./stdlib/abort.c:79
#5 0x0000558f48524feb in ha_panic () at src/debug.c:333
#6 0x0000558f486be1af in wdt_handler (sig=14, si=0x7fcf04e4d3f0,
arg=0x7fcf04e4d2c0) at src/wdt.c:122
#7 <signal handler called>
#8 0x0000558f48469895 in pattern_exec_match (head=0x558f69fe67b8,
smp=0x7fcf04e4e080, fill=0) at src/pattern.c:2498
#9 0x0000558f485a0bf2 in acl_exec_cond (cond=0x558f69fe69a0, px=0x558f69f67cf0,
sess=0x7fce50243cd0, strm=0x7fce5020a4b0, opt=6) at src/acl.c:1123
#10 0x0000558f483b5eea in spoe_encode_message (s=0x7fce5020a4b0,
ctx=0x7fce5026ddf0, msg=0x558f69fe2650, dir=0, buf=0x7fcf04e4e1b0,
end=0x7fce502a627c "") at src/flt_spoe.c:2185
#11 0x0000558f483b6259 in spoe_encode_messages (s=0x7fce5020a4b0,
ctx=0x7fce5026ddf0, messages=0x558f69fe23a0, dir=0, type=1) at src/flt_spoe.c:2285
#12 0x0000558f483b75a4 in spoe_process_messages (s=0x7fce5020a4b0,
ctx=0x7fce5026ddf0, messages=0x558f69fe23a0, dir=0, type=1) at src/flt_spoe.c:2726
#13 0x0000558f483b79d0 in spoe_process_event (s=0x7fce5020a4b0,
ctx=0x7fce5026ddf0, ev=SPOE_EV_ON_HTTP_REQ_FE) at src/flt_spoe.c:2842
#14 0x0000558f483b8eeb in spoe_chn_pre_analyze (s=0x7fce5020a4b0,
filter=0x7fce5025c920, chn=0x7fce5020a4c0, an_bit=16) at src/flt_spoe.c:3324
#15 0x0000558f48549976 in flt_pre_analyze (s=0x7fce5020a4b0, chn=0x7fce5020a4c0,
an_bit=16) at src/filters.c:764
#16 0x0000558f4836aaff in process_stream (t=0x7fce50242f30,
context=0x7fce5020a4b0, state=260) at src/stream.c:1918
#17 0x0000558f4857aa3c in run_tasks_from_lists (budgets=0x7fcf04e4e5a0) at
src/task.c:597
#18 0x0000558f4857b485 in process_runnable_tasks () at src/task.c:838
#19 0x0000558f484d0eb5 in run_poll_loop () at src/haproxy.c:2630
#20 0x0000558f484d1423 in run_thread_poll_loop (data=0x5) at src/haproxy.c:2805
#21 0x00007fcf0886cb43 in start_thread (arg=<optimized out>) at
./nptl/pthread_create.c:442
#22 0x00007fcf088fea00 in clone3 () at
../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
haproxy -vv
HAProxy version 2.4.22-f8e3218 2023/02/14 - https://haproxy.org/
<https://haproxy.org/>
Status: long-term supported branch - will stop receiving fixes around Q2 2026.
Known bugs: http://www.haproxy.org/bugs/bugs-2.4.22.html
<http://www.haproxy.org/bugs/bugs-2.4.22.html>
Running on: Linux 5.15.0-69-generic #76-Ubuntu SMP Fri Mar 17 17:19:29 UTC 2023
x86_64
Build options :
TARGET = linux-glibc
CPU = generic
CC = cc
CFLAGS = -O0 -g -Wall -Wextra -Wdeclaration-after-statement -fwrapv
-Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered
-Wno-missing-field-initializers -Wtype-limits -Wshift-negative-value
-Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -DMAX_SESS_STKCTR=12
OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_GETADDRINFO=1 USE_OPENSSL=1
USE_LUA=1 USE_ZLIB=1 USE_DL=1 USE_PROMEX=1
DEBUG =
Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H
-DEVICEATLAS +DL +EPOLL -EVPORTS +FUTEX +GETADDRINFO -KQUEUE +LIBCRYPT
+LINUX_SPLICE +LINUX_TPROXY +LUA -MEMORY_PROFILING +NETFILTER +NS
-OBSOLETE_LINKER +OPENSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL
-PRIVATE_CACHE -PROCCTL +PROMEX -PTHREAD_PSHARED -QUIC +RT -SLZ -STATIC_PCRE
-STATIC_PCRE2 -SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL +ZLIB
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=4).
Built with OpenSSL version : OpenSSL 1.1.1t 7 Feb 2023
Running on OpenSSL version : OpenSSL 1.1.1t 7 Feb 2023
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.5
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"),
raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Built with PCRE2 version : 10.39 2021-10-29
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 7.5.0
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTTP side=FE|BE mux=H2
flags=HTX|CLEAN_ABRT|HOL_RISK|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI
flags=HTX|HOL_RISK|NO_UPG
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
Available services : prometheus-exporter
Available filters :
[SPOE] spoe
[CACHE] cache
[FCGI] fcgi-app
[COMP] compression
[TRACE] trace
Hi Maciej,
Thanks for the report and sorry for the delay. Could you share your SPOE
configuration please ?
When a SPOE filter is configured on a disabled proxy, it is released during
configuration parsing, just like any other filters. I suspect some shared data
are released if you use the same backend for your SPOE agents.
--
Christopher Faulet
