Hi Christopher,
no problem. :) Yes I'm using the same spoe backend for multiple frontends.
This is my spoe configuration:
[abc]
spoe-agent abc-agent
messages check-abc
register-var-names x_abc_request_headers x_headers x_abc_response body
option var-prefix abc
option set-on-error error
option set-total-time total_time
timeout hello 100ms
timeout idle 41s
timeout processing 50ms
use-backend spoe-abc
spoe-message check-abc
args Key=str("123") Key2=str("Haproxy")...
acl route_to_abc path_beg -f /etc/haproxy/configs/acl/path_begin.lst
event on-frontend-http-request if route_to_abc
wt., 9 maj 2023 o 09:14 Christopher Faulet <[email protected]> napisał(a):
> Le 5/2/23 à 13:58, Maciej Zdeb a écrit :
> > Hi,
> > I'm experiencing a segmentation fault caused by adding "disabled"
> > (http://docs.haproxy.org/2.4/configuration.html#4-disabled
> > <http://docs.haproxy.org/2.4/configuration.html#4-disabled>) to the
> frontend
> > section of haproxy configuration file. That frontend does not handle any
> traffic
> > yet.
> >
> > The "disabled" keyword was used on other frontends on the same HAProxy
> instance
> > before without any issue. The difference is that this time it was added
> to
> > frontend with spoe filter. This probably causes a crash when processing
> requests
> > on a different heavily used frontend (also spoe enabled).
> >
> > It crashes after 2-10s under production traffic. I couldn't replicate it
> with a
> > synthetic test, however I have a coredump and packages with HAProxy
> binary and
> > debug symbols.
> >
> > (gdb) bt
> > #0 __pthread_kill_implementation (no_tid=0, signo=6,
> threadid=140527117211200)
> > at ./nptl/pthread_kill.c:44
> > #1 __pthread_kill_internal (signo=6, threadid=140527117211200) at
> > ./nptl/pthread_kill.c:78
> > #2 __GI___pthread_kill (threadid=140527117211200, signo=signo@entry=6)
> at
> > ./nptl/pthread_kill.c:89
> > #3 0x00007fcf0881a476 in __GI_raise (sig=sig@entry=6) at
> > ../sysdeps/posix/raise.c:26
> > #4 0x00007fcf088007f3 in __GI_abort () at ./stdlib/abort.c:79
> > #5 0x0000558f48524feb in ha_panic () at src/debug.c:333
> > #6 0x0000558f486be1af in wdt_handler (sig=14, si=0x7fcf04e4d3f0,
> > arg=0x7fcf04e4d2c0) at src/wdt.c:122
> > #7 <signal handler called>
> > #8 0x0000558f48469895 in pattern_exec_match (head=0x558f69fe67b8,
> > smp=0x7fcf04e4e080, fill=0) at src/pattern.c:2498
> > #9 0x0000558f485a0bf2 in acl_exec_cond (cond=0x558f69fe69a0,
> px=0x558f69f67cf0,
> > sess=0x7fce50243cd0, strm=0x7fce5020a4b0, opt=6) at src/acl.c:1123
> > #10 0x0000558f483b5eea in spoe_encode_message (s=0x7fce5020a4b0,
> > ctx=0x7fce5026ddf0, msg=0x558f69fe2650, dir=0, buf=0x7fcf04e4e1b0,
> > end=0x7fce502a627c "") at src/flt_spoe.c:2185
> > #11 0x0000558f483b6259 in spoe_encode_messages (s=0x7fce5020a4b0,
> > ctx=0x7fce5026ddf0, messages=0x558f69fe23a0, dir=0, type=1) at
> src/flt_spoe.c:2285
> > #12 0x0000558f483b75a4 in spoe_process_messages (s=0x7fce5020a4b0,
> > ctx=0x7fce5026ddf0, messages=0x558f69fe23a0, dir=0, type=1) at
> src/flt_spoe.c:2726
> > #13 0x0000558f483b79d0 in spoe_process_event (s=0x7fce5020a4b0,
> > ctx=0x7fce5026ddf0, ev=SPOE_EV_ON_HTTP_REQ_FE) at src/flt_spoe.c:2842
> > #14 0x0000558f483b8eeb in spoe_chn_pre_analyze (s=0x7fce5020a4b0,
> > filter=0x7fce5025c920, chn=0x7fce5020a4c0, an_bit=16) at
> src/flt_spoe.c:3324
> > #15 0x0000558f48549976 in flt_pre_analyze (s=0x7fce5020a4b0,
> chn=0x7fce5020a4c0,
> > an_bit=16) at src/filters.c:764
> > #16 0x0000558f4836aaff in process_stream (t=0x7fce50242f30,
> > context=0x7fce5020a4b0, state=260) at src/stream.c:1918
> > #17 0x0000558f4857aa3c in run_tasks_from_lists (budgets=0x7fcf04e4e5a0)
> at
> > src/task.c:597
> > #18 0x0000558f4857b485 in process_runnable_tasks () at src/task.c:838
> > #19 0x0000558f484d0eb5 in run_poll_loop () at src/haproxy.c:2630
> > #20 0x0000558f484d1423 in run_thread_poll_loop (data=0x5) at
> src/haproxy.c:2805
> > #21 0x00007fcf0886cb43 in start_thread (arg=<optimized out>) at
> > ./nptl/pthread_create.c:442
> > #22 0x00007fcf088fea00 in clone3 () at
> ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
> >
> > haproxy -vv
> > HAProxy version 2.4.22-f8e3218 2023/02/14 - https://haproxy.org/
> > <https://haproxy.org/>
> > Status: long-term supported branch - will stop receiving fixes around Q2
> 2026.
> > Known bugs: http://www.haproxy.org/bugs/bugs-2.4.22.html
> > <http://www.haproxy.org/bugs/bugs-2.4.22.html>
> > Running on: Linux 5.15.0-69-generic #76-Ubuntu SMP Fri Mar 17 17:19:29
> UTC 2023
> > x86_64
> > Build options :
> > TARGET = linux-glibc
> > CPU = generic
> > CC = cc
> > CFLAGS = -O0 -g -Wall -Wextra -Wdeclaration-after-statement -fwrapv
> > -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered
> > -Wno-missing-field-initializers -Wtype-limits -Wshift-negative-value
> > -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
> -DMAX_SESS_STKCTR=12
> > OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_GETADDRINFO=1 USE_OPENSSL=1
> > USE_LUA=1 USE_ZLIB=1 USE_DL=1 USE_PROMEX=1
> > DEBUG =
> >
> > Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY
> +CRYPT_H
> > -DEVICEATLAS +DL +EPOLL -EVPORTS +FUTEX +GETADDRINFO -KQUEUE +LIBCRYPT
> > +LINUX_SPLICE +LINUX_TPROXY +LUA -MEMORY_PROFILING +NETFILTER +NS
> > -OBSOLETE_LINKER +OPENSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL
> +PRCTL
> > -PRIVATE_CACHE -PROCCTL +PROMEX -PTHREAD_PSHARED -QUIC +RT -SLZ
> -STATIC_PCRE
> > -STATIC_PCRE2 -SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL +ZLIB
> >
> > Default settings :
> > bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
> >
> > Built with multi-threading support (MAX_THREADS=64, default=4).
> > Built with OpenSSL version : OpenSSL 1.1.1t 7 Feb 2023
> > Running on OpenSSL version : OpenSSL 1.1.1t 7 Feb 2023
> > OpenSSL library supports TLS extensions : yes
> > OpenSSL library supports SNI : yes
> > OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
> > Built with Lua version : Lua 5.3.5
> > Built with the Prometheus exporter as a service
> > Built with network namespace support.
> > Built with zlib version : 1.2.11
> > Running on zlib version : 1.2.11
> > Compression algorithms supported : identity("identity"),
> deflate("deflate"),
> > raw-deflate("deflate"), gzip("gzip")
> > Built with transparent proxy support using: IP_TRANSPARENT
> IPV6_TRANSPARENT
> > IP_FREEBIND
> > Built with PCRE2 version : 10.39 2021-10-29
> > PCRE2 library supports JIT : yes
> > Encrypted password support via crypt(3): yes
> > Built with gcc compiler version 7.5.0
> >
> > Available polling systems :
> > epoll : pref=300, test result OK
> > poll : pref=200, test result OK
> > select : pref=150, test result OK
> > Total: 3 (3 usable), will use epoll.
> >
> > Available multiplexer protocols :
> > (protocols marked as <default> cannot be specified using 'proto' keyword)
> > h2 : mode=HTTP side=FE|BE mux=H2
> > flags=HTX|CLEAN_ABRT|HOL_RISK|NO_UPG
> > fcgi : mode=HTTP side=BE mux=FCGI
> > flags=HTX|HOL_RISK|NO_UPG
> > h1 : mode=HTTP side=FE|BE mux=H1
> flags=HTX|NO_UPG
> > <default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
> > none : mode=TCP side=FE|BE mux=PASS
> flags=NO_UPG
> > <default> : mode=TCP side=FE|BE mux=PASS flags=
> >
> > Available services : prometheus-exporter
> > Available filters :
> > [SPOE] spoe
> > [CACHE] cache
> > [FCGI] fcgi-app
> > [COMP] compression
> > [TRACE] trace
>
> Hi Maciej,
>
> Thanks for the report and sorry for the delay. Could you share your SPOE
> configuration please ?
>
> When a SPOE filter is configured on a disabled proxy, it is released
> during
> configuration parsing, just like any other filters. I suspect some shared
> data
> are released if you use the same backend for your SPOE agents.
>
> --
> Christopher Faulet
>
>
