Re: 1.9.5: SIGSEGV in wake_srv_chk under heavy load

[Olivier Houchard]

Hi Maksim,

On Thu, Mar 28, 2019 at 04:36:16PM +0300,  ?? wrote:
> Hi!
> 
> We accidentally got a spike of client requests to our site and under that
> heavy load to ssl-protected backends haproxy=1.9.5 had fallen down :(
> 

I think I understand how it may have happened, and hopefully fixed it with
commit to master 06f6811d9fc3f36597b686e4ca9fe7fbccf091b0
It will be backported to 1.9 later, but if you need it right now, you can
probably apply it on 1.9 as-is.

Thanks a lot for reporting !

Olivier

1.9.5: SIGSEGV in wake_srv_chk under heavy load

[Максим Куприянов]

Hi!

We accidentally got a spike of client requests to our site and under that
heavy load to ssl-protected backends haproxy=1.9.5 had fallen down :(

backtrace

Core was generated by `/usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p
/var/run/haproxy'.

Program terminated with signal SIGSEGV, Segmentation fault.

#0  wake_srv_chk (cs=cs@entry=0x0) at src/checks.c:1410

1410 src/checks.c: No such file or directory.

[Current thread is 1 (Thread 0x7f11cd14f180 (LWP 52133))]

(gdb) thread apply all bt


Thread 4 (Thread 0x7f11c972d700 (LWP 52135)):

#0  0x7f11cb8a719c in malloc () from /lib/x86_64-linux-gnu/libc.so.6

#1  0x7f11cc0ffe78 in CRYPTO_malloc () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0

#2  0x7f11cc1e7ea9 in ASN1_STRING_type_new () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0

#3  0x7f11cc1d19fa in c2i_ASN1_INTEGER () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0

#4  0x7f11cc1dd9b4 in asn1_ex_c2i () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0

#5  0x7f11cc1dde2b in ?? () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0

#6  0x7f11cc1dece8 in ?? () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0

#7  0x7f11cc1df036 in ?? () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0

#8  0x7f11cc1df241 in ?? () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0

#9  0x7f11cc1dea3d in ?? () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0

#10 0x7f11cc1df036 in ?? () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0

#11 0x7f11cc1df2ad in ?? () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0

#12 0x7f11cc1dea3d in ?? () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0

#13 0x7f11cc1df415 in ASN1_item_ex_d2i () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0

#14 0x7f11cc1df46b in ASN1_item_d2i () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0

#15 0x7f11cc52c83b in d2i_SSL_SESSION () from
/lib/x86_64-linux-gnu/libssl.so.1.0.0

#16 0x55df0ca744b8 in ssl_sock_init (conn=0x7f119c438430) at
src/ssl_sock.c:5129

#17 0x55df0cb69300 in conn_xprt_init (conn=0x7f119c438430) at
include/proto/connection.h:84

#18 tcp_connect_server (conn=0x7f119c438430, data=,
delack=) at src/proto_tcp.c:559

#19 0x55df0cb551e0 in si_connect (conn=0x7f119c438430, si=) at include/proto/stream_interface.h:492

#20 connect_server (s=s@entry=0x7f119e31d8f0) at src/backend.c:1511

#21 0x55df0cacbfba in sess_update_stream_int (s=0x7f119e31d8f0) at
src/stream.c:929

#22 process_stream (t=, context=0x7f119e31d8f0,
state=) at src/stream.c:2306

#23 0x55df0cb97c4d in process_runnable_tasks () at src/task.c:434

#24 0x55df0cb0f2a0 in run_poll_loop () at src/haproxy.c:2640

#25 run_thread_poll_loop (data=) at src/haproxy.c:2705

#26 0x7f11cc95a6ba in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0

#27 0x7f11cb92a41d in clone () from /lib/x86_64-linux-gnu/libc.so.6


Thread 3 (Thread 0x7f11c8f2c700 (LWP 52136)):

#0  0x7f11cb92b577 in shutdown () from /lib/x86_64-linux-gnu/libc.so.6

#1  0x55df0cb9bd75 in conn_sock_shutw (clean=1, c=) at
include/proto/connection.h:456

#2  mux_pt_shutw (cs=0x7f11c03c34e0, mode=CS_SHW_NORMAL) at src/mux_pt.c:233

#3  0x55df0cb61aeb in cs_shutw (mode=CS_SHW_NORMAL, cs=)
at include/proto/connection.h:493

#4  stream_int_shutw_conn (si=0x7f11c0760f98) at src/stream_interface.c:988

#5  0x55df0cacc216 in si_shutw (si=0x7f11c0760f98) at
include/proto/stream_interface.h:420

#6  process_stream (t=, context=0x7f11c0760d00,
state=) at src/stream.c:2442

#7  0x55df0cb97c4d in process_runnable_tasks () at src/task.c:434

---Type  to continue, or q  to quit---

#8  0x55df0cb0f2a0 in run_poll_loop () at src/haproxy.c:2640

#9  run_thread_poll_loop (data=) at src/haproxy.c:2705

#10 0x7f11cc95a6ba in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0

#11 0x7f11cb92a41d in clone () from /lib/x86_64-linux-gnu/libc.so.6


Thread 2 (Thread 0x7f11c9f2e700 (LWP 52134)):

#0  0x7f11cb8a2e4e in ?? () from /lib/x86_64-linux-gnu/libc.so.6

#1  0x7f11cb8a753c in free () from /lib/x86_64-linux-gnu/libc.so.6

#2  0x7f11cc1000cd in CRYPTO_free () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0

#3  0x7f11cc1bc8f1 in EVP_MD_CTX_cleanup () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0

#4  0x7f11cc126b26 in HMAC_CTX_cleanup () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0

#5  0x7f11cc12700e in ?? () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0

#6  0x7f11cc1c990c in EVP_PKEY_CTX_free () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0

#7  0x7f11cc1bc882 in EVP_MD_CTX_cleanup () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0

#8  0x7f11cc514487 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0

#9  0x7f11cc51521a in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0

#10 0x7f11cc508a58 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0

#11 0x7f11cc509603 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0

#12 0x7f11cc50a687 in ?? () from