Hi Maciej, On Wed, Sep 16, 2020 at 9:00 PM Maciej Zdeb <[email protected]> wrote:
> Hi, > > Our HAProxy (2.0.14) started to crash, so first we upgraded to 2.0.17 but > it didn't help. Below you'll find traces from coredump > > Version: > HA-Proxy version 2.0.17 2020/07/31 - https://haproxy.org/ > Build options : > TARGET = linux-glibc > CPU = generic > CC = gcc > CFLAGS = -O0 -g -fno-strict-aliasing -Wdeclaration-after-statement > -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter > -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered > -Wno-missing-field-initializers -Wno-implicit-fallthrough > -Wno-stringop-overflow -Wtype-limits -Wshift-negative-value > -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference > -DIP_BIND_ADDRESS_NO_PORT=24 -DMAX_SESS_STKCTR=12 > OPTIONS = USE_PCRE=1 USE_PCRE_JIT=1 USE_REGPARM=1 USE_GETADDRINFO=1 > USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_DL=1 > > Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER +PCRE > +PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED > +REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE > +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 > -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS > -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS > > Default settings : > bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 > > Built with multi-threading support (MAX_THREADS=64, default=4). > Built with OpenSSL version : OpenSSL 1.1.1f 31 Mar 2020 > Running on OpenSSL version : OpenSSL 1.1.1f 31 Mar 2020 > OpenSSL library supports TLS extensions : yes > OpenSSL library supports SNI : yes > OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3 > Built with Lua version : Lua 5.3.5 > Built with network namespace support. > Built with transparent proxy support using: IP_TRANSPARENT > IPV6_TRANSPARENT IP_FREEBIND > Built with zlib version : 1.2.11 > Running on zlib version : 1.2.11 > Compression algorithms supported : identity("identity"), > deflate("deflate"), raw-deflate("deflate"), gzip("gzip") > Built with PCRE version : 8.44 2020-02-12 > Running on PCRE version : 8.44 2020-02-12 > PCRE library supports JIT : yes > Encrypted password support via crypt(3): yes > > Available polling systems : > epoll : pref=300, test result OK > poll : pref=200, test result OK > select : pref=150, test result OK > Total: 3 (3 usable), will use epoll. > > Available multiplexer protocols : > (protocols marked as <default> cannot be specified using 'proto' keyword) > h2 : mode=HTTP side=FE mux=H2 > h2 : mode=HTX side=FE|BE mux=H2 > <default> : mode=HTX side=FE|BE mux=H1 > <default> : mode=TCP|HTTP side=FE|BE mux=PASS > > Available services : none > > Available filters : > [SPOE] spoe > [COMP] compression > [CACHE] cache > [TRACE] trace > > > Coredump fragment from thread1: > (gdb) bt > #0 0x000055cbbf6ed64b in h2s_notify_recv (h2s=0x7f65b8b55130) at > src/mux_h2.c:783 > #1 0x000055cbbf6edbc7 in h2s_close (h2s=0x7f65b8b55130) at > src/mux_h2.c:921 > #2 0x000055cbbf6f9745 in h2s_htx_make_trailers (h2s=0x7f65b8b55130, > htx=0x7f65a9c34f20) at src/mux_h2.c:5385 > #3 0x000055cbbf6fa48e in h2_snd_buf (cs=0x7f65b8c48a40, > buf=0x7f65d05291b8, count=2, flags=1) at src/mux_h2.c:5694 > #4 0x000055cbbf7cdde3 in si_cs_send (cs=0x7f65b8c48a40) at > src/stream_interface.c:762 > #5 0x000055cbbf7ce839 in stream_int_chk_snd_conn (si=0x7f65d0529478) at > src/stream_interface.c:1145 > #6 0x000055cbbf7cc9d6 in si_chk_snd (si=0x7f65d0529478) at > include/proto/stream_interface.h:496 > #7 0x000055cbbf7cd559 in stream_int_notify (si=0x7f65d05294d0) at > src/stream_interface.c:510 > #8 0x000055cbbf7cda33 in si_cs_process (cs=0x55cbca178f90) at > src/stream_interface.c:644 > #9 0x000055cbbf7cdfb1 in si_cs_io_cb (t=0x0, ctx=0x7f65d05294d0, state=1) > at src/stream_interface.c:817 > #10 0x000055cbbf81af32 in process_runnable_tasks () at src/task.c:415 > #11 0x000055cbbf740cc0 in run_poll_loop () at src/haproxy.c:2701 > #12 0x000055cbbf741188 in run_thread_poll_loop (data=0x1) at > src/haproxy.c:2840 > #13 0x00007f667fbdb6db in start_thread (arg=0x7f65d5556700) at > pthread_create.c:463 > #14 0x00007f667e764a3f in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 > > (gdb) bt full > #0 0x000055cbbf6ed64b in h2s_notify_recv (h2s=0x7f65b8b55130) at > src/mux_h2.c:783 > sw = 0xffffffff > #1 0x000055cbbf6edbc7 in h2s_close (h2s=0x7f65b8b55130) at > src/mux_h2.c:921 > No locals. > #2 0x000055cbbf6f9745 in h2s_htx_make_trailers (h2s=0x7f65b8b55130, > htx=0x7f65a9c34f20) at src/mux_h2.c:5385 > list = {{n = {ptr = 0x55cbbf88d18c "", len = 0}, v = {ptr = > 0x7f65a9c34f20 "�?", len = 94333580136844}}, {n = {ptr = 0x7f65d5532410 " > Oée\177", len = 94333579742112}, v = {ptr = 0x7f65a9c38e78 "�\001", len = > 140074616573728}}, {n = { > ptr = 0x55cbbf6ea203 <htx_get_blksz+24> > "\211E�\213E�\203�\002t\005\203�\005u\035H\213E�\213@ > \004\017��H\213E�\213@\004��\b%��\017", len = 81604378627}, v = {ptr = > 0x1fd00000001 <error: Cannot access memory at address 0x1fd00000001>, len = > 140074616589944}}, {n = {ptr = 0x7f65a9c34f20 "�?", len = 140075347420216}, > v = { > ptr = 0x55cbbf82bfc0 <htx_used_space+32> > "\001�H\203�\b[]�UH\211�SH\203�\bH\211}�H\213E�\213\030H\213E�H\211�����)�\211�H\203�\b[]�UH\211�H\203�\030H\211}�H\213E�H\211�����\211E�\203}�\aw\a�", > len = 140074616573728}}, {n = {ptr = 0x3fd0 <error: Cannot access memory at > address 0x3fd0>, len = 140075347420248}, v = { > ptr = 0x55cbbf82bfe8 <htx_free_space+31> > ")�\211�H\203�\b[]�UH\211�H\203�\030H\211}�H\213E�H\211�����\211E�\203}�\aw\a�", > len = 2848149280}}, {n = {ptr = 0x7f65a9c34f20 "�?", len = > 140075347420288}, v = { > ptr = 0x55cbbf82b970 <htx_get_blk+41> > "\211�H��\003H\001�H\203�\020[]�UH\211�H\211}�H\213E�\213@ > \004��\034]�UH\211�H\203�\030H\211}�H\213E�H\211������\211E�\213E�\203�\002t\005\203�\005u\035H\213E�\213@ > \004\017��H\213E�\213@\004��\b%��\017", len = 2848149280}}, {n = {ptr = > 0x7f65a9c34f20 "�?", len = 140075347420512}, v = { > ptr = 0x7f65d55324c0 "�$S�e\177", len = 94333579740367}}, {n > = {ptr = 0x1a9c34fd8 <error: Cannot access memory at address 0x1a9c34fd8>, > len = 140074616573728}, v = > [STRIPED] > > I can provide more details from coredump if needed. > I'm not one of the devs but obviously many of us using v2.0 will be interested in the answer. Assuming you do not install from packages can you please provide some more background on how you produce the binary, like if you compile then what OS and kernel is this compiled on and what OS and kernel this crashes on? Again if compiled any other custom compiled packages in use, like OpenSSL, lua etc, you might be using or have compiled haproxy against etc.? Also if this is a bug and you have hit some corner case with your config (many are using 2.0 but we have not seen crashes) you should provide a stripped down version (not too stripped though just the sensitive data) of your config too. Cheers, Igor
