Hi Willy and Tim,
> > >> Code 134 implies the worker was killed with SIGABRT. You could check
> > >> whether there is a core dump.
> > >
> > > I don't have any core dumps.
> >
> > Check whether coredumps are enabled using `ulimit -c`, often they are
> > disabled by default, because they could contain sensitive information.
> > After the next crash you should be able to retrieve a backtrace using
> > gdb. Possibly recompile haproxy with debug symbols for it to be useful.
>
> If it happens quickly, another option might be to attach gdb to the
> process after it is started. But with multiple processes it's not very
> convenient.
>
> A few more things on the core dumps :
> - they are ignored if you have a chroot statement in the global section
> - you need not to use "user/uid/group/gid" otherwise the system also
> disables core dumps
I'm using chroot and user/group in my config, so I'm not able to share core
dumps.
> There are very few abort() calls in the code :
> - some in the thread debugging code to detect recursive locks ;
> - one in the cache applet which triggers on an impossible case very
> likely resulting from cache corruption (hence a bug)
> - a few inside the Lua library
> - a few in the HPACK decompressor, detecting a few possible bugs there
>
> Except for Lua, all of them were added during 1.8, so depending on what the
> configuration uses, there are very few possible candidates.
I added my configuration in this mail. Hopefully this will narrow down the
possible candidates.
I did some more research to the memory warnings we encounter every few days. It
seems like the haproxy processes use a lot of memory. Would haproxy with
nbthreads share this memory?
1160 haproxy 20 0 1881720 1.742g 5504 S 83.9 11.5 1:53.38 haproxy
1045 haproxy 20 0 1880120 1.740g 5572 S 71.0 11.5 1:36.62 haproxy
1104 haproxy 20 0 1880376 1.741g 6084 R 64.6 11.5 1:46.29 haproxy
1079 haproxy 20 0 1881116 1.741g 5564 S 58.1 11.5 1:42.29 haproxy
1135 haproxy 20 0 1881240 1.741g 5564 S 58.1 11.5 1:49.85 haproxy
995 haproxy 20 0 1881852 1.742g 5584 R 38.7 11.5 1:30.05 haproxy
1020 haproxy 20 0 1881448 1.741g 5516 S 25.8 11.5 1:32.20 haproxy
4926 haproxy 20 0 1881008 1.718g 2176 S 6.5 11.3 3:11.74 haproxy
8526 haproxy 20 0 1878032 6516 1304 S 0.0 0.0 2:10.04 haproxy
8529 haproxy 20 0 1880336 5208 4 S 0.0 0.0 2:34.68 haproxy
11530 haproxy 20 0 1878748 6556 1392 S 0.0 0.0 2:25.94 haproxy
26938 haproxy 20 0 1882592 6032 892 S 0.0 0.0 3:56.79 haproxy
29577 haproxy 20 0 1880480 1.738g 3132 S 0.0 11.5 2:08.74 haproxy
31124 haproxy 20 0 1880776 1.740g 4284 S 0.0 11.5 2:58.84 haproxy
7548 root 20 0 1869896 1.731g 4456 S 0.0 11.4 1008:23 haproxy
I'm using systemd to reload haproxy for new SSL certificates every few minutes.
[Service]
Environment=CONFIG=/etc/haproxy/haproxy.cfg
EnvironmentFile=-/etc/default/haproxy
ExecStartPre=/usr/sbin/haproxy -f ${CONFIG} -c -q
ExecStart=/usr/sbin/haproxy -Ws -f ${CONFIG} -p /run/haproxy.pid $EXTRAOPTS
ExecReload=/usr/sbin/haproxy -c -f ${CONFIG}
ExecReload=/bin/kill -USR2 $MAINPID
KillMode=mixed
Restart=always
Configuration:
global
log **removed hostname** syslog
maxconn 32000
ulimit-n 65536
tune.maxrewrite 2048
user haproxy
group haproxy
daemon
chroot /var/lib/haproxy
nbproc 7
maxcompcpuusage 85
spread-checks 0
ssl-default-bind-options no-sslv3
stats socket /var/run/haproxy.sock mode 400 level admin process 1
stats socket /var/run/haproxy.sock.2 mode 400 level admin process 2
stats socket /var/run/haproxy.sock.3 mode 400 level admin process 3
stats socket /var/run/haproxy.sock.4 mode 400 level admin process 4
stats socket /var/run/haproxy.sock.5 mode 400 level admin process 5
stats socket /var/run/haproxy.sock.6 mode 400 level admin process 6
stats socket /var/run/haproxy.sock.7 mode 400 level admin process 7
master-worker no-exit-on-failure
defaults
log global
timeout http-request 5s
timeout connect 2s
timeout client 125s
timeout server 125s
mode http
option dontlog-normal
option http-server-close
option tcp-smart-connect
frontend fe_http
bind ipv4@:80 backlog 65534
bind ipv6@:80 backlog 65534
bind ipv4@:443 ssl crt /etc/haproxy/ssl/invalid.pem crt /etc/haproxy/ssl/
crt /etc/haproxy/customer-ssl/ strict-sni backlog 65534
bind ipv6@:443 ssl crt /etc/haproxy/ssl/invalid.pem crt /etc/haproxy/ssl/
crt /etc/haproxy/customer-ssl/ strict-sni backlog 65534
bind-process 1-7
tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }
option forwardfor
acl secure dst_port 443
acl is_acme_request path_beg /.well-known/acme-challenge/
reqadd X-Forwarded-Proto:\ https if secure
default_backend be_reservedpage
use_backend be_acme if is_acme_request
use_backend %[req.fhdr(host),lower,map_dom(/etc/haproxy/domain2backend.map)]
compression algo gzip
maxconn 32000
http-response set-header X-Balancer lb0
listen stats
bind *:1936
bind-process 1
mode http
stats enable
stats uri /
stats admin if TRUE
backend be_acme
bind-process 1
option httpchk HEAD /ping.php HTTP/1.1\r\nHost:\ **removed hostname**
option http-server-close
option http-pretend-keepalive
server **removed hostname** **removed ip**:80 maxconn 200 inter 5000 check
backend be_reservedpage
bind-process 1
http-request set-header X-Forwarded-Host %[hdr(host)]
http-request set-header Host **removed hostname**
http-response set-header Cache-Control no-cache,\ no-store,\ must-revalidate
server **removed name** **removed ip**:80 verify none check
Thanks,
Frank
