Re: Segfault on HAProxy 2.0.11 on HTX mode
Le 19/02/2020 à 11:35, Olivier D a écrit : Hello, I would like to report a segfault on HAProxy 2.0.11 ; this version has been running fine for two months, and this morning starting segfaulting over and over. Mitigation was performed by adding "no option http-use-htx" on 'defaults' block. I know it's not the latest version :) I'll update to 2.0.13 this evening. Program terminated with signal 11, Segmentation fault. #0 htx_sl_p2 (sl=) at include/common/htx.h:293 293 include/common/htx.h: No such file or directory. (gdb) bt #0 htx_sl_p2 (sl=) at include/common/htx.h:293 #1 htx_sl_req_uri (sl=) at include/common/htx.h:308 #2 assign_server (s=0xdc139f0) at src/backend.c:746 #3 0x00552114 in assign_server_and_queue (s=s@entry=0xdc139f0) at src/backend.c:977 #4 0x005556f8 in assign_server_and_queue (s=0xdc139f0) at src/backend.c:1772 #5 srv_redispatch_connect (s=s@entry=0xdc139f0) at src/backend.c:1705 #6 0x004c2cf8 in sess_prepare_conn_req (s=) at src/stream.c:1250 #7 process_stream (t=t@entry=0xd1db790, context=0xdc139f0, state=out>) at src/stream.c:2414 #8 0x00594865 in process_runnable_tasks () at src/task.c:412 #9 0x005038f7 in run_poll_loop () at src/haproxy.c:2520 #10 run_thread_poll_loop (data=data@entry=0x0) at src/haproxy.c:2641 #11 0x004653b0 in main (argc=, argv=0x7fff848ae498) at src/haproxy.c:3318 Hi all, This bug was fixed (commit 9d9d64540) and backported as far as 1.9. The crash was happening when a client error was encountered for a tarpitted connection with an HTTP load-balancing algorithm configured on the backend (for instance, balance uri). Thanks to Olivier for his help, -- Christopher Faulet
Re: Segfault on HAProxy 2.0.11 on HTX mode
Le 19/02/2020 à 17:12, Olivier D a écrit : I thought HTX was default mode since 2.0-dev3 (https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#no%20option%20http-use-htx) We don't have custom config on this, so default mode was used everywhere. Ahhh, you're right. I've mixed up the versions... > Did you make any recent changes on HAproxy or your servers ? I'm surprised the > segaults appear spontaneously after 2 months without any problem. > > > Only minor modifications in the last few days ... minor modifications may have huge impact especially if you hit an hidden bug :) Config file is auto-generated from a central server, so we always add frontends, backends or certificates. That's all. I can send you the config file, but it's 8k lines, so it wont help you much I guess. Can the coredump help you more, with the binary used ? Yes, send me everything, it could help and limits the useless round-trips. Don't forget to tell me the distro you are using. Thanks, -- Christopher Faulet
Re: Segfault on HAProxy 2.0.11 on HTX mode
Le mer. 19 févr. 2020 à 16:24, Christopher Faulet a
écrit :
> Le 19/02/2020 à 16:05, Olivier D a écrit :
> > A bug was fixed in 2.0.12 that could explain such of crashes. The
> upstream
> > commit id is eec7f8ac0 (or 0ed1e8963 in the 2.0 tree). It is related
> to the
> > GitHub issue #420.
> >
> > But I don't know if it is the same bug because I don't know how it
> is possible
> > to apply an HTTP load-balancing algo on a TCP backend. I must take a
> look at
> > your configuration. You can send it to me in private. Maybe I'll
> found
> > something
> > explaining your crashes.
> >
> >
> > I have hundreds of frontend/backends in this config. What made you think
> this is
> > related to a tcp backend ? That would help me a lot.
> >
> >
>
> Because the mentioned commit fixes a bug where it was possible to assign a
> TCP
> backend to an HTX stream. It is possible to hit this bug when dynamic
> rules are
> used to choose the backend. In such case, we are unable to detect bad
> configuration during HAProxy startup.
>
We do use some use_backend if {}, but only on http frontends (I checked).
Never on tcp.
We have a mix between "listen" blocks with "server" defined inside, and
some frontend/backend blocks. So one "listen" block may also have a
"use_backend if".
Yes, it's bad, but it has been auto-generated since we use HAProxy 1.5 and
we never rewrite this part.
So, if you have TCP frontends that can be dynamically routed to HTTP or TCP
> backends, you may hit the bug. See github issue #420.
>
I don't think it is this one. Our only tcp frontends are all formated like
this :
listen x
id 20609
bind-process 18
balance source
hash-type consistent
mode tcp
bind X.X.X.X:443
server s1 X.X.X.X:443 id 4567 check weight 5 send-proxy-v2-ssl-cn
check-ssl verify none
server s2 X.X.X.X:443 id 1234 check weight 5 send-proxy-v2-ssl-cn
check-ssl verify none
> There is another source of bugs. In HAProxy 2.0, the HTX mode is not
> enabled by
> default. If you have dynamic routing rules, be careful to have the same
> mode
> (legacy or HTX) everywhere. I will do some tests to be sure this case it
> properly handled.
>
I thought HTX was default mode since 2.0-dev3 (
https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#no%20option%20http-use-htx
)
We don't have custom config on this, so default mode was used everywhere.
>
>
> > Did you make any recent changes on HAproxy or your servers ? I'm
> surprised the
> > segaults appear spontaneously after 2 months without any problem.
> >
> >
> > Only minor modifications in the last few days ...
>
> minor modifications may have huge impact especially if you hit an hidden
> bug :)
>
Config file is auto-generated from a central server, so we always add
frontends, backends or certificates. That's all.
I can send you the config file, but it's 8k lines, so it wont help you much
I guess. Can the coredump help you more, with the binary used ?
Olivier
>
> --
> Christopher Faulet
>
Re: Segfault on HAProxy 2.0.11 on HTX mode
Le 19/02/2020 à 16:05, Olivier D a écrit : A bug was fixed in 2.0.12 that could explain such of crashes. The upstream commit id is eec7f8ac0 (or 0ed1e8963 in the 2.0 tree). It is related to the GitHub issue #420. But I don't know if it is the same bug because I don't know how it is possible to apply an HTTP load-balancing algo on a TCP backend. I must take a look at your configuration. You can send it to me in private. Maybe I'll found something explaining your crashes. I have hundreds of frontend/backends in this config. What made you think this is related to a tcp backend ? That would help me a lot. Because the mentioned commit fixes a bug where it was possible to assign a TCP backend to an HTX stream. It is possible to hit this bug when dynamic rules are used to choose the backend. In such case, we are unable to detect bad configuration during HAProxy startup. So, if you have TCP frontends that can be dynamically routed to HTTP or TCP backends, you may hit the bug. See github issue #420. There is another source of bugs. In HAProxy 2.0, the HTX mode is not enabled by default. If you have dynamic routing rules, be careful to have the same mode (legacy or HTX) everywhere. I will do some tests to be sure this case it properly handled. Did you make any recent changes on HAproxy or your servers ? I'm surprised the segaults appear spontaneously after 2 months without any problem. Only minor modifications in the last few days ... minor modifications may have huge impact especially if you hit an hidden bug :) -- Christopher Faulet
Re: Segfault on HAProxy 2.0.11 on HTX mode
Hello, Le mer. 19 févr. 2020 à 15:27, Christopher Faulet a écrit : > Le 19/02/2020 à 11:35, Olivier D a écrit : > > Hello, > > > > I would like to report a segfault on HAProxy 2.0.11 ; this version has > been > > running fine for two months, and this morning starting segfaulting over > and over. > > Mitigation was performed by adding "no option http-use-htx" on > 'defaults' block. > > > > I know it's not the latest version :) I'll update to 2.0.13 this evening. > > > > Program terminated with signal 11, Segmentation fault. > > #0 htx_sl_p2 (sl=) at include/common/htx.h:293 > > 293 include/common/htx.h: No such file or directory. > > (gdb) bt > > #0 htx_sl_p2 (sl=) at include/common/htx.h:293 > > #1 htx_sl_req_uri (sl=) at include/common/htx.h:308 > > #2 assign_server (s=0xdc139f0) at src/backend.c:746 > > #3 0x00552114 in assign_server_and_queue (s=s@entry=0xdc139f0) > at > > src/backend.c:977 > > #4 0x005556f8 in assign_server_and_queue (s=0xdc139f0) at > > src/backend.c:1772 > > #5 srv_redispatch_connect (s=s@entry=0xdc139f0) at src/backend.c:1705 > > #6 0x004c2cf8 in sess_prepare_conn_req (s=) at > > src/stream.c:1250 > > #7 process_stream (t=t@entry=0xd1db790, context=0xdc139f0, > state= > out>) at src/stream.c:2414 > > #8 0x00594865 in process_runnable_tasks () at src/task.c:412 > > #9 0x005038f7 in run_poll_loop () at src/haproxy.c:2520 > > #10 run_thread_poll_loop (data=data@entry=0x0) at src/haproxy.c:2641 > > #11 0x004653b0 in main (argc=, > argv=0x7fff848ae498) at > > src/haproxy.c:3318 > > > > > > > > Config file is very long ... If needed, a coredump + binary can be sent > on private. > > > > Hi, > > A bug was fixed in 2.0.12 that could explain such of crashes. The upstream > commit id is eec7f8ac0 (or 0ed1e8963 in the 2.0 tree). It is related to > the > GitHub issue #420. > > But I don't know if it is the same bug because I don't know how it is > possible > to apply an HTTP load-balancing algo on a TCP backend. I must take a look > at > your configuration. You can send it to me in private. Maybe I'll found > something > explaining your crashes. > I have hundreds of frontend/backends in this config. What made you think this is related to a tcp backend ? That would help me a lot. > > Did you make any recent changes on HAproxy or your servers ? I'm surprised > the > segaults appear spontaneously after 2 months without any problem. > Only minor modifications in the last few days ... I'll update to latest haproxy version to check. Olivier > > > -- > Christopher Faulet >
Re: Segfault on HAProxy 2.0.11 on HTX mode
Le 19/02/2020 à 11:35, Olivier D a écrit : Hello, I would like to report a segfault on HAProxy 2.0.11 ; this version has been running fine for two months, and this morning starting segfaulting over and over. Mitigation was performed by adding "no option http-use-htx" on 'defaults' block. I know it's not the latest version :) I'll update to 2.0.13 this evening. Program terminated with signal 11, Segmentation fault. #0 htx_sl_p2 (sl=) at include/common/htx.h:293 293 include/common/htx.h: No such file or directory. (gdb) bt #0 htx_sl_p2 (sl=) at include/common/htx.h:293 #1 htx_sl_req_uri (sl=) at include/common/htx.h:308 #2 assign_server (s=0xdc139f0) at src/backend.c:746 #3 0x00552114 in assign_server_and_queue (s=s@entry=0xdc139f0) at src/backend.c:977 #4 0x005556f8 in assign_server_and_queue (s=0xdc139f0) at src/backend.c:1772 #5 srv_redispatch_connect (s=s@entry=0xdc139f0) at src/backend.c:1705 #6 0x004c2cf8 in sess_prepare_conn_req (s=) at src/stream.c:1250 #7 process_stream (t=t@entry=0xd1db790, context=0xdc139f0, state=out>) at src/stream.c:2414 #8 0x00594865 in process_runnable_tasks () at src/task.c:412 #9 0x005038f7 in run_poll_loop () at src/haproxy.c:2520 #10 run_thread_poll_loop (data=data@entry=0x0) at src/haproxy.c:2641 #11 0x004653b0 in main (argc=, argv=0x7fff848ae498) at src/haproxy.c:3318 Config file is very long ... If needed, a coredump + binary can be sent on private. Hi, A bug was fixed in 2.0.12 that could explain such of crashes. The upstream commit id is eec7f8ac0 (or 0ed1e8963 in the 2.0 tree). It is related to the GitHub issue #420. But I don't know if it is the same bug because I don't know how it is possible to apply an HTTP load-balancing algo on a TCP backend. I must take a look at your configuration. You can send it to me in private. Maybe I'll found something explaining your crashes. Did you make any recent changes on HAproxy or your servers ? I'm surprised the segaults appear spontaneously after 2 months without any problem. -- Christopher Faulet
Segfault on HAProxy 2.0.11 on HTX mode
Hello,
I would like to report a segfault on HAProxy 2.0.11 ; this version has been
running fine for two months, and this morning starting segfaulting over and
over.
Mitigation was performed by adding "no option http-use-htx" on 'defaults'
block.
I know it's not the latest version :) I'll update to 2.0.13 this evening.
Program terminated with signal 11, Segmentation fault.
#0 htx_sl_p2 (sl=) at include/common/htx.h:293
293 include/common/htx.h: No such file or directory.
(gdb) bt
#0 htx_sl_p2 (sl=) at include/common/htx.h:293
#1 htx_sl_req_uri (sl=) at include/common/htx.h:308
#2 assign_server (s=0xdc139f0) at src/backend.c:746
#3 0x00552114 in assign_server_and_queue (s=s@entry=0xdc139f0) at
src/backend.c:977
#4 0x005556f8 in assign_server_and_queue (s=0xdc139f0) at
src/backend.c:1772
#5 srv_redispatch_connect (s=s@entry=0xdc139f0) at src/backend.c:1705
#6 0x004c2cf8 in sess_prepare_conn_req (s=) at
src/stream.c:1250
#7 process_stream (t=t@entry=0xd1db790, context=0xdc139f0,
state=) at src/stream.c:2414
#8 0x00594865 in process_runnable_tasks () at src/task.c:412
#9 0x005038f7 in run_poll_loop () at src/haproxy.c:2520
#10 run_thread_poll_loop (data=data@entry=0x0) at src/haproxy.c:2641
#11 0x004653b0 in main (argc=, argv=0x7fff848ae498)
at src/haproxy.c:3318
haproxy -vv:
HA-Proxy version 2.0.11 2019/12/11 - https://haproxy.org/
Build options :
TARGET = linux-glibc
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
-fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter
-Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered
-Wno-missing-field-initializers -Wno-implicit-fallthrough
-Wno-stringop-overflow -Wtype-limits -Wshift-negative-value
-Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
OPTIONS = USE_THREAD=0 USE_STATIC_PCRE=1 USE_OPENSSL=1 USE_LUA=1
USE_ZLIB=1 USE_NS=
Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE
-PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED
-REGPARM +STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE
+LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4
-MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO -NS +DL +RT -DEVICEATLAS
-51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=20).
Built with OpenSSL version : OpenSSL 1.1.1d 10 Sep 2019
Running on OpenSSL version : OpenSSL 1.1.1d 10 Sep 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.5
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.43 2019-02-23
Running on PCRE version : 8.43 2019-02-23
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as cannot be specified using 'proto' keyword)
h2 : mode=HTXside=FE|BE mux=H2
h2 : mode=HTTP side=FEmux=H2
: mode=HTXside=FE|BE mux=H1
: mode=TCP|HTTP side=FE|BE mux=PASS
Available services : none
Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace
Config file is very long ... If needed, a coredump + binary can be sent on
private.
Olivier
