Hi List,
I thought i 'reasonably' tested some of 1.8.0's options.
Today i put it into 'production' on my secondary cluster node and notice
it takes 100% cpu... I guess i should have tried such a thing last week.
My regular config with 10 frontends and total 13 servers seems to
startup fine when 'email-alert level' is set to 'emerg' , doesnt need to
send a mail then..
Anyhow below some gdb and console output.
Config that reproduces it is pretty simple no new features used or anything.
Though the server is 'down' so it is trying to send a mail for that..
that never seems to happen though.. no mail is received.
I tried using nokqueu and nopoll, but that did not result in any
improvement..
Anything else i can provide?
Regards,
PiBa-NL / Pieter
haproxy -f /root/hap.conf -V
[WARNING] 330/204605 (14771) : config : missing timeouts for frontend
'TestMailFront'.
| While not properly invalid, you will certainly encounter various
problems
| with such a configuration. To fix this, please ensure that all
following
| timeouts are set to a non-zero value: 'client', 'connect', 'server'.
[WARNING] 330/204605 (14771) : config : missing timeouts for backend
'TestMailBack'.
| While not properly invalid, you will certainly encounter various
problems
| with such a configuration. To fix this, please ensure that all
following
| timeouts are set to a non-zero value: 'client', 'connect', 'server'.
Note: setting global.maxconn to 2000.
Available polling systems :
kqueue : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result FAILED
Total: 3 (2 usable), will use kqueue.
Available filters :
[TRACE] trace
[COMP] compression
[SPOE] spoe
Using kqueue() as the polling mechanism.
[WARNING] 330/204608 (14771) : Server TestMailBack/TestServer is DOWN,
reason: Layer4 timeout, check duration: 2009ms. 0 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 330/204608 (14771) : backend 'TestMailBack' has no server available!
Complete configuration that reproduces the issue:
mailers globalmailers
mailer ex01 192.168.0.40:25
frontend TestMailFront
bind :88
default_backend TestMailBack
backend TestMailBack
server TestServer 192.168.0.250:80 check
email-alert mailers globalmailers
email-alert level info
email-alert from haproxy@me.local
email-alert to m...@me.tld
email-alert myhostname pfs
root@:~ # haproxy -vv
HA-Proxy version 1.8.0 2017/11/26
Copyright 2000-2017 Willy Tarreau <wi...@haproxy.org>
Build options :
TARGET = freebsd
CPU = generic
CC = cc
CFLAGS = -pipe -g -fstack-protector -fno-strict-aliasing
-fno-strict-aliasing -Wdeclaration-after-statement -fwrapv
-Wno-address-of-packed-member -Wno-null-dereference -Wno-unused-label
-DFREEBSD_PORTS
OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_CPU_AFFINITY=1
USE_ACCEPT4=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_STATIC_PCRE=1
USE_PCRE_JIT=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.40 2017-01-11
Running on PCRE version : 8.40 2017-01-11
PCRE library supports JIT : yes
Built with multi-threading support.
Encrypted password support via crypt(3): yes
Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY
Built with Lua version : Lua 5.3.4
Built with OpenSSL version : OpenSSL 1.0.2k-freebsd 26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-freebsd 26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Available polling systems :
kqueue : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use kqueue.
Available filters :
[TRACE] trace
[COMP] compression
[SPOE] spoe
root@:~ #
root@:~ # /usr/local/bin/gdb --pid 14771
GNU gdb (GDB) 8.0.1 [GDB v8.0.1 for FreeBSD]
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd11.1".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 14771
Reading symbols from /usr/local/sbin/haproxy...done.
Reading symbols from /lib/libcrypt.so.5...(no debugging symbols
found)...done.
Reading symbols from /lib/libz.so.6...(no debugging symbols found)...done.
Reading symbols from /lib/libthr.so.3...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libssl.so.8...(no debugging symbols
found)...done.
Reading symbols from /lib/libcrypto.so.8...(no debugging symbols
found)...done.
Reading symbols from /usr/local/lib/liblua-5.3.so...(no debugging
symbols found)...done.
Reading symbols from /lib/libm.so.5...(no debugging symbols found)...done.
Reading symbols from /lib/libc.so.7...(no debugging symbols found)...done.
Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols
found)...done.
[Switching to LWP 100129 of process 14771]
0x00000000004fca69 in tcpcheck_main (check=0x80243b610) at src/checks.c:2598
2598 HA_SPIN_LOCK(SERVER_LOCK, &check->server->lock);
(gdb) bt
#0 0x00000000004fca69 in tcpcheck_main (check=0x80243b610) at
src/checks.c:2598
#1 0x00000000005013bd in connect_conn_chk (t=0x80242d0a0) at
src/checks.c:1548
#2 0x00000000004fff08 in process_chk_conn (t=0x80242d0a0) at
src/checks.c:2137
#3 0x00000000004ff6c7 in process_chk (t=0x80242d0a0) at src/checks.c:2283
#4 0x00000000004fa9a3 in process_email_alert (t=0x80242d0a0) at
src/checks.c:3157
#5 0x000000000058c6f6 in process_runnable_tasks () at src/task.c:229
#6 0x000000000050d70b in run_poll_loop () at src/haproxy.c:2326
#7 0x000000000050b259 in run_thread_poll_loop (data=0x802446060) at
src/haproxy.c:2375
#8 0x00000000005080fe in main (argc=4, argv=0x7fffffffeb40) at
src/haproxy.c:2910
(gdb) bt full
#0 0x00000000004fca69 in tcpcheck_main (check=0x80243b610) at
src/checks.c:2598
contentptr = 0x8024a2ef0 ""
comment = 0x802497800 "@xI\002\b"
next = 0x9
done = 0
ret = 0
step = 0
cs = 0x0
conn = 0x0
s = 0x8024d5000
t = 0x80242d0a0
head = 0x8024a2ef0
retcode = 0
#1 0x00000000005013bd in connect_conn_chk (t=0x80242d0a0) at
src/checks.c:1548
check = 0x80243b610
s = 0x8024d5000
cs = 0x0
conn = 0x0
proto = 0x1000800000000
tcp_rule = 0x802497800
ret = 8
quickack = 9100288
#2 0x00000000004fff08 in process_chk_conn (t=0x80242d0a0) at
src/checks.c:2137
check = 0x80243b610
s = 0x8024d5000
cs = 0x0
conn = 0x0
rv = 32767
ret = -6920
expired = 1
#3 0x00000000004ff6c7 in process_chk (t=0x80242d0a0) at src/checks.c:2283
check = 0x80243b610
#4 0x00000000004fa9a3 in process_email_alert (t=0x80242d0a0) at
src/checks.c:3157
---Type <return> to continue, or q <return> to quit---
check = 0x80243b610
q = 0x80243b600
alert = 0x8024a2ee0
#5 0x000000000058c6f6 in process_runnable_tasks () at src/task.c:229
t = 0x80242d0a0
i = 32767
max_processed = 200
rq_next = 0x0
local_tasks = {0x1, 0x8a5b18 <thread_exit_sync.barrier>, 0x0,
0x100eee7fb, 0x1, 0x1, 0x0, 0x0, 0x0,
0x8a5b18 <thread_exit_sync.barrier>, 0x7fffffffe700, 0x59f7f6
<thread_exit_sync+262>, 0x1, 0x1, 0xfffffffffffffffe,
0x53000008024d59e0}
local_tasks_count = 0
final_tasks_count = 0
#6 0x000000000050d70b in run_poll_loop () at src/haproxy.c:2326
next = -16518964
#7 0x000000000050b259 in run_thread_poll_loop (data=0x802446060) at
src/haproxy.c:2375
ptif = 0x89a2e8 <per_thread_init_list>
ptdf = 0x206
#8 0x00000000005080fe in main (argc=4, argv=0x7fffffffeb40) at
src/haproxy.c:2910
tids = 0x802446060
threads = 0x802446068
i = 1
err = 0
retry = 200
limit = {rlim_cur = 4012, rlim_max = 4012}
errmsg =
"\000\353\377\377\377\177\000\000h\353\377\377\377\177\000\000@\353\377\377\377\177\000\000\004\000\000\000\000\000\000\000/^\312\230\312\"\355\256
\017\212\000\000\000\000\000\070\353\377\377\377\177\000\000h\353\377\377\377\177\000\000@\353\377\377\377\177\000\000\004\000\000\000\000\000\000\000\340\352\377\377\377\177\000\000Rq\340\001\b\000\000\000\001\000\000"
pidfd = -1
(gdb) cont
Continuing.
^C
Program received signal SIGINT, Interrupt.
0x00000000004fca69 in tcpcheck_main (check=0x80243b610) at src/checks.c:2598
2598 HA_SPIN_LOCK(SERVER_LOCK, &check->server->lock);
(gdb) bt
#0 0x00000000004fca69 in tcpcheck_main (check=0x80243b610) at
src/checks.c:2598
#1 0x00000000005013bd in connect_conn_chk (t=0x80242d0a0) at
src/checks.c:1548
#2 0x00000000004fff08 in process_chk_conn (t=0x80242d0a0) at
src/checks.c:2137
#3 0x00000000004ff6c7 in process_chk (t=0x80242d0a0) at src/checks.c:2283
#4 0x00000000004fa9a3 in process_email_alert (t=0x80242d0a0) at
src/checks.c:3157
#5 0x000000000058c6f6 in process_runnable_tasks () at src/task.c:229
#6 0x000000000050d70b in run_poll_loop () at src/haproxy.c:2326
#7 0x000000000050b259 in run_thread_poll_loop (data=0x802446060) at
src/haproxy.c:2375
#8 0x00000000005080fe in main (argc=4, argv=0x7fffffffeb40) at
src/haproxy.c:2910
(gdb) Quit
(gdb) cont
Continuing.
^C
Program received signal SIGINT, Interrupt.
0x00000000004fca6b in tcpcheck_main (check=0x80243b610) at src/checks.c:2598
2598 HA_SPIN_LOCK(SERVER_LOCK, &check->server->lock);
(gdb) bt
#0 0x00000000004fca6b in tcpcheck_main (check=0x80243b610) at
src/checks.c:2598
#1 0x00000000005013bd in connect_conn_chk (t=0x80242d0a0) at
src/checks.c:1548
#2 0x00000000004fff08 in process_chk_conn (t=0x80242d0a0) at
src/checks.c:2137
#3 0x00000000004ff6c7 in process_chk (t=0x80242d0a0) at src/checks.c:2283
#4 0x00000000004fa9a3 in process_email_alert (t=0x80242d0a0) at
src/checks.c:3157
#5 0x000000000058c6f6 in process_runnable_tasks () at src/task.c:229
#6 0x000000000050d70b in run_poll_loop () at src/haproxy.c:2326
#7 0x000000000050b259 in run_thread_poll_loop (data=0x802446060) at
src/haproxy.c:2375
#8 0x00000000005080fe in main (argc=4, argv=0x7fffffffeb40) at
src/haproxy.c:2910
(gdb) cont
Continuing.
^C
Program received signal SIGINT, Interrupt.
0x00000000004fc9b9 in tcpcheck_main (check=0x80243b610) at src/checks.c:2598
2598 HA_SPIN_LOCK(SERVER_LOCK, &check->server->lock);
(gdb) bt
#0 0x00000000004fc9b9 in tcpcheck_main (check=0x80243b610) at
src/checks.c:2598
#1 0x00000000005013bd in connect_conn_chk (t=0x80242d0a0) at
src/checks.c:1548
#2 0x00000000004fff08 in process_chk_conn (t=0x80242d0a0) at
src/checks.c:2137
#3 0x00000000004ff6c7 in process_chk (t=0x80242d0a0) at src/checks.c:2283
#4 0x00000000004fa9a3 in process_email_alert (t=0x80242d0a0) at
src/checks.c:3157
#5 0x000000000058c6f6 in process_runnable_tasks () at src/task.c:229
#6 0x000000000050d70b in run_poll_loop () at src/haproxy.c:2326
#7 0x000000000050b259 in run_thread_poll_loop (data=0x802446060) at
src/haproxy.c:2375
#8 0x00000000005080fe in main (argc=4, argv=0x7fffffffeb40) at
src/haproxy.c:2910
(gdb) con
Ambiguous command "con": condition, continue.
(gdb) cont
Continuing.
^C
Program received signal SIGINT, Interrupt.
0x00000000004fca6b in tcpcheck_main (check=0x80243b610) at src/checks.c:2598
2598 HA_SPIN_LOCK(SERVER_LOCK, &check->server->lock);
(gdb) bt full
#0 0x00000000004fca6b in tcpcheck_main (check=0x80243b610) at
src/checks.c:2598
contentptr = 0x8024a2ef0 ""
comment = 0x802497800 "@xI\002\b"
next = 0x9
done = 0
ret = 0
step = 0
cs = 0x0
conn = 0x0
s = 0x8024d5000
t = 0x80242d0a0
head = 0x8024a2ef0
retcode = 0
#1 0x00000000005013bd in connect_conn_chk (t=0x80242d0a0) at
src/checks.c:1548
check = 0x80243b610
s = 0x8024d5000
cs = 0x0
conn = 0x0
proto = 0x1000800000000
tcp_rule = 0x802497800
ret = 8
quickack = 9100288
#2 0x00000000004fff08 in process_chk_conn (t=0x80242d0a0) at
src/checks.c:2137
check = 0x80243b610
s = 0x8024d5000
cs = 0x0
conn = 0x0
rv = 32767
ret = -6920
expired = 1
#3 0x00000000004ff6c7 in process_chk (t=0x80242d0a0) at src/checks.c:2283
check = 0x80243b610
#4 0x00000000004fa9a3 in process_email_alert (t=0x80242d0a0) at
src/checks.c:3157
check = 0x80243b610
q = 0x80243b600
alert = 0x8024a2ee0
#5 0x000000000058c6f6 in process_runnable_tasks () at src/task.c:229
t = 0x80242d0a0
i = 32767
max_processed = 200
rq_next = 0x0
local_tasks = {0x1, 0x8a5b18 <thread_exit_sync.barrier>, 0x0,
0x100eee7fb, 0x1, 0x1, 0x0, 0x0, 0x0,
0x8a5b18 <thread_exit_sync.barrier>, 0x7fffffffe700, 0x59f7f6
<thread_exit_sync+262>, 0x1, 0x1, 0xfffffffffffffffe,
0x53000008024d59e0}
local_tasks_count = 0
final_tasks_count = 0
#6 0x000000000050d70b in run_poll_loop () at src/haproxy.c:2326
next = -16518964
#7 0x000000000050b259 in run_thread_poll_loop (data=0x802446060) at
src/haproxy.c:2375
ptif = 0x89a2e8 <per_thread_init_list>
ptdf = 0x206
#8 0x00000000005080fe in main (argc=4, argv=0x7fffffffeb40) at
src/haproxy.c:2910
tids = 0x802446060
threads = 0x802446068
i = 1
err = 0
retry = 200
limit = {rlim_cur = 4012, rlim_max = 4012}
---Type <return> to continue, or q <return> to quit---
errmsg =
"\000\353\377\377\377\177\000\000h\353\377\377\377\177\000\000@\353\377\377\377\177\000\000\004\000\000\000\000\000\000\000/^\312\230\312\"\355\256
\017\212\000\000\000\000\000\070\353\377\377\377\177\000\000h\353\377\377\377\177\000\000@\353\377\377\377\177\000\000\004\000\000\000\000\000\000\000\340\352\377\377\377\177\000\000Rq\340\001\b\000\000\000\001\000\000"
pidfd = -1
(gdb)
