Hi!
Yesterday one of our haproxies (1.8.5) with nbthread=8 set in its config
stuck with 800% CPU usage. Some responses were served successfully but many
of them just timed out. perf top showed this:
59.19% [.] thread_enter_sync
32.68% [.] fwrr_get_next_server
We made a core and here is a full bt:
Core was generated by `/usr/sbin/haproxy'.
#0 0x000055a9807b984b in fwrr_get_next_server (p=p@entry=0x55a982e91c40,
srvtoavoid=srvtoavoid@entry=0x55a982eb7940) at src/lb_fwrr.c:473
473 HA_SPIN_LOCK(LBPRM_LOCK, &p->lbprm.lock);
(gdb) bt
#0 0x000055a9807b984b in fwrr_get_next_server (p=p@entry=0x55a982e91c40,
srvtoavoid=srvtoavoid@entry=0x55a982eb7940) at src/lb_fwrr.c:473
#1 0x000055a98078ab5d in assign_server (s=0x7f4bbc0906a0) at
src/backend.c:604
#2 0x000055a98078be6d in assign_server_and_queue (s=s@entry=0x7f4bbc0906a0)
at src/backend.c:872
#3 0x000055a98078d333 in srv_redispatch_connect (s=s@entry=0x7f4bbc0906a0)
at src/backend.c:1284
#4 0x000055a9807369d8 in sess_prepare_conn_req (s=0x7f4bbc0906a0) at
src/stream.c:1094
#5 process_stream (t=<optimized out>) at src/stream.c:2219
#6 0x000055a9807b1e68 in process_runnable_tasks () at src/task.c:311
#7 0x000055a980766ff4 in run_poll_loop () at src/haproxy.c:2399
#8 run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:2461
#9 0x00007f4bcd00d184 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f4bcc28b03d in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) thread apply all bt
Thread 8 (Thread 0x7f4bcd888980 (LWP 133188)):
#0 fwrr_update_position (grp=0x55a982e92858, grp=0x55a982e92858,
s=0x55a982e9a2f0) at src/lb_fwrr.c:454
#1 fwrr_get_next_server (p=p@entry=0x55a982e91c40,
srvtoavoid=srvtoavoid@entry=0x55a982e9a2f0) at src/lb_fwrr.c:521
#2 0x000055a98078ab5d in assign_server (s=0x7f4bc05ed8c0) at
src/backend.c:604
#3 0x000055a98078be6d in assign_server_and_queue (s=s@entry=0x7f4bc05ed8c0)
at src/backend.c:872
#4 0x000055a98078d333 in srv_redispatch_connect (s=s@entry=0x7f4bc05ed8c0)
at src/backend.c:1284
#5 0x000055a9807369d8 in sess_prepare_conn_req (s=0x7f4bc05ed8c0) at
src/stream.c:1094
#6 process_stream (t=<optimized out>) at src/stream.c:2219
#7 0x000055a9807b1e68 in process_runnable_tasks () at src/task.c:311
#8 0x000055a980766ff4 in run_poll_loop () at src/haproxy.c:2399
#9 run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:2461
#10 0x000055a9806d1a57 in main (argc=<optimized out>, argv=<optimized out>)
at src/haproxy.c:3050
Thread 7 (Thread 0x7f4bb3fff700 (LWP 133195)):
#0 0x000055a9807bbad6 in thread_sync_barrier (barrier=0x55a980a2f4e0
<barrier>) at src/hathreads.c:110
#1 thread_enter_sync () at src/hathreads.c:123
#2 0x000055a98076705e in sync_poll_loop () at src/haproxy.c:2376
#3 run_poll_loop () at src/haproxy.c:2431
#4 run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:2461
#5 0x00007f4bcd00d184 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f4bcc28b03d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 6 (Thread 0x7f4bc89d8700 (LWP 133194)):
#0 0x000055a9807bbad6 in thread_sync_barrier (barrier=0x55a980a2f4e0
<barrier>) at src/hathreads.c:110
#1 thread_enter_sync () at src/hathreads.c:123
#2 0x000055a98076705e in sync_poll_loop () at src/haproxy.c:2376
#3 run_poll_loop () at src/haproxy.c:2431
#4 run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:2461
#5 0x00007f4bcd00d184 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f4bcc28b03d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 5 (Thread 0x7f4bc91d9700 (LWP 133193)):
#0 0x000055a9807b984b in fwrr_get_next_server (p=p@entry=0x55a982e91c40,
srvtoavoid=srvtoavoid@entry=0x55a982eb28a0) at src/lb_fwrr.c:473
#1 0x000055a98078ab5d in assign_server (s=0x7f4bc0aa4690) at
src/backend.c:604
#2 0x000055a98078be6d in assign_server_and_queue (s=s@entry=0x7f4bc0aa4690)
at src/backend.c:872
#3 0x000055a98078d333 in srv_redispatch_connect (s=s@entry=0x7f4bc0aa4690)
at src/backend.c:1284
#4 0x000055a9807369d8 in sess_prepare_conn_req (s=0x7f4bc0aa4690) at
src/stream.c:1094
#5 process_stream (t=<optimized out>) at src/stream.c:2219
#6 0x000055a9807b1e68 in process_runnable_tasks () at src/task.c:311
#7 0x000055a980766ff4 in run_poll_loop () at src/haproxy.c:2399
#8 run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:2461
#9 0x00007f4bcd00d184 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f4bcc28b03d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 4 (Thread 0x7f4bc99da700 (LWP 133192)):
---Type <return> to continue, or q <return> to quit---
#0 0x000055a9807bbad6 in thread_sync_barrier (barrier=0x55a980a2f4e0
<barrier>) at src/hathreads.c:110
#1 thread_enter_sync () at src/hathreads.c:123
#2 0x000055a98076705e in sync_poll_loop () at src/haproxy.c:2376
#3 run_poll_loop () at src/haproxy.c:2431
#4 run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:2461
#5 0x00007f4bcd00d184 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f4bcc28b03d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 3 (Thread 0x7f4bca1db700 (LWP 133191)):
#0 0x000055a9807bbad6 in thread_sync_barrier (barrier=0x55a980a2f4e0
<barrier>) at src/hathreads.c:110
#1 thread_enter_sync () at src/hathreads.c:123
#2 0x000055a98076705e in sync_poll_loop () at src/haproxy.c:2376
#3 run_poll_loop () at src/haproxy.c:2431
#4 run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:2461
#5 0x00007f4bcd00d184 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f4bcc28b03d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 2 (Thread 0x7f4bca9dc700 (LWP 133190)):
#0 thread_sync_barrier (barrier=0x55a980a2f4e0 <barrier>) at
src/hathreads.c:110
#1 thread_enter_sync () at src/hathreads.c:123
#2 0x000055a98076705e in sync_poll_loop () at src/haproxy.c:2376
#3 run_poll_loop () at src/haproxy.c:2431
#4 run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:2461
#5 0x00007f4bcd00d184 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f4bcc28b03d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 1 (Thread 0x7f4bcb1dd700 (LWP 133189)):
#0 0x000055a9807b984b in fwrr_get_next_server (p=p@entry=0x55a982e91c40,
srvtoavoid=srvtoavoid@entry=0x55a982eb7940) at src/lb_fwrr.c:473
#1 0x000055a98078ab5d in assign_server (s=0x7f4bbc0906a0) at
src/backend.c:604
#2 0x000055a98078be6d in assign_server_and_queue (s=s@entry=0x7f4bbc0906a0)
at src/backend.c:872
#3 0x000055a98078d333 in srv_redispatch_connect (s=s@entry=0x7f4bbc0906a0)
at src/backend.c:1284
#4 0x000055a9807369d8 in sess_prepare_conn_req (s=0x7f4bbc0906a0) at
src/stream.c:1094
#5 process_stream (t=<optimized out>) at src/stream.c:2219
#6 0x000055a9807b1e68 in process_runnable_tasks () at src/task.c:311
#7 0x000055a980766ff4 in run_poll_loop () at src/haproxy.c:2399
#8 run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:2461
#9 0x00007f4bcd00d184 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f4bcc28b03d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Version info:
# haproxy -vv
HA-Proxy version 1.8.5-1 2018/03/26
Copyright 2000-2018 Willy Tarreau <wi...@haproxy.org>
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4
-Wformat -Werror=format-security -D_FORTIFY_SOURCE=2
OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_THREAD=1
USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1 USE_PCRE_JIT=1 USE_TFO=1 USE_NS=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
Running on OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.1
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.31 2012-07-06
Running on PCRE version : 8.31 2012-07-06
PCRE library supports JIT : no (libpcre build without JIT?)
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace
Please take a look.
--
Best regards,
Maksim Kupriianov
