After I updated from 1.8.8 to 1.8.9, the child process consumes 100% of one core from startup. Most of haproxy's functionality works fine (and of course much slower than before) except that commands (e.g. set state to MAINT) in the stat web page often result in a 502 bad gateway.
After some debugging, it turns out to be an issue with the resolver. I am using LXD containers and dns resolution is done by LXD which only listen on its subnet. If I change all .lxd domains in my configuration to IP addresses, everything gets back to normal. Wonder if anyone has a clue or it's a bug. Additional information: When I updated haproxy, I also updated my kernel from 4.4.0-124 to 4.4.0-127. The server has 2 cores and 2GB memory. No obvious memory leak after 3 hours of 100% CPU. [haproxy -v] HA-Proxy version 1.8.9-1ppa1~xenial 2018/05/19 Copyright 2000-2018 Willy Tarreau <wi...@haproxy.org> (I'm using Vincent Bernat's ppa) [uname -a] Linux <masked FQDN> 4.4.0-127-generic #153-Ubuntu SMP Sat May 19 10:58:46 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux [lsb_release -a] LSB Version: core-9.20160110ubuntu0.2-amd64:core-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-amd64:security-9.20160110ubuntu0.2-noarch Distributor ID: Ubuntu Description: Ubuntu 16.04.4 LTS Release: 16.04 Codename: xenial =============================================================================== [Original config file] =============================================================================== global log /dev/log local0 log /dev/log local1 notice chroot /var/lib/haproxy stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners stats timeout 30s user haproxy group haproxy daemon # Default SSL material locations ca-base /etc/ssl/certs crt-base /etc/ssl/private # Default ciphers to use on SSL-enabled listening sockets. # For more information, see ciphers(1SSL). This list is from: # https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/ # An alternative list with additional directives can be obtained from # https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS ssl-default-bind-options no-sslv3 resolvers lxd nameserver lxd 10.12.34.1:53 userlist stats-auth group admin users zinsser user zinsser password <masked> defaults log global mode http option httplog option dontlognull timeout connect 5000 timeout client 50000 timeout server 50000 errorfile 400 /etc/haproxy/errors/400.http errorfile 403 /etc/haproxy/errors/403.http errorfile 408 /etc/haproxy/errors/408.http errorfile 500 /etc/haproxy/errors/500.http errorfile 502 /etc/haproxy/errors/502.http errorfile 503 /etc/haproxy/errors/503.http errorfile 504 /etc/haproxy/errors/504.http default-server init-addr none resolvers lxd listen imap mode tcp bind *:143 server mail-main mail-main.lxd:143 send-proxy listen smtp mode tcp bind *:25 server mail-main mail-main.lxd:25 send-proxy listen submission mode tcp bind *:587 server mail-main mail-main.lxd:587 send-proxy frontend https-in mode http bind *:443 ssl crt /etc/haproxy/certs/full-priv.pem alpn h2,http/1.1 # X-Forwarded-For option forwardfor # X-Forwarded-Proto reqadd X-Forwarded-Proto:\ https # Domains acl cloud hdr_beg(host) -i cloud.<masked> acl rspamd hdr_beg(host) -i rspamd.<masked> # Url acl openproject url_beg /openproject # Stats acl stats_auth http_auth(stats-auth) acl stats_admin http_auth_group(stats-auth) admin stats http-request auth unless stats_auth stats enable stats hide-version stats uri /ha_stats stats admin if stats_admin # Backends use_backend openproject if cloud openproject use_backend cloud if cloud use_backend rspamd if rspamd default_backend wordpress frontend http-in mode http bind *:80 redirect scheme https code 301 backend wordpress mode http server wordpress wordpress.lxd:80 backend cloud mode http server cloud cloud.lxd:80 backend openproject mode http server openproject openproject.lxd:80 backend rspamd mode http server rspamd mail-filter.lxd:11334 =============================================================================== [Workaround config file] Notice that the default-server line is not commented out. =============================================================================== log /dev/log local0 log /dev/log local1 notice chroot /var/lib/haproxy stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners stats timeout 30s user haproxy group haproxy daemon # Default SSL material locations ca-base /etc/ssl/certs crt-base /etc/ssl/private # Default ciphers to use on SSL-enabled listening sockets. # For more information, see ciphers(1SSL). This list is from: # https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/ # An alternative list with additional directives can be obtained from # https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS ssl-default-bind-options no-sslv3 resolvers lxd nameserver lxd 10.12.34.1:53 userlist stats-auth group admin users zinsser user zinsser password <masked> defaults log global mode http option httplog option dontlognull timeout connect 5000 timeout client 50000 timeout server 50000 errorfile 400 /etc/haproxy/errors/400.http errorfile 403 /etc/haproxy/errors/403.http errorfile 408 /etc/haproxy/errors/408.http errorfile 500 /etc/haproxy/errors/500.http errorfile 502 /etc/haproxy/errors/502.http errorfile 503 /etc/haproxy/errors/503.http errorfile 504 /etc/haproxy/errors/504.http default-server init-addr none resolvers lxd listen imap mode tcp bind *:143 server mail-main 10.12.34.246:143 send-proxy listen smtp mode tcp bind *:25 server mail-main 10.12.34.246:25 send-proxy listen submission mode tcp bind *:587 server mail-main 10.12.34.246:587 send-proxy frontend https-in mode http bind *:443 ssl crt /etc/haproxy/certs/full-priv.pem alpn h2,http/1.1 # X-Forwarded-For option forwardfor # X-Forwarded-Proto reqadd X-Forwarded-Proto:\ https # Domains acl cloud hdr_beg(host) -i cloud.<masked> acl rspamd hdr_beg(host) -i rspamd.<masked> # Url acl openproject url_beg /openproject # Stats acl stats_auth http_auth(stats-auth) acl stats_admin http_auth_group(stats-auth) admin stats http-request auth unless stats_auth stats enable stats hide-version stats uri /ha_stats stats admin if stats_admin # Backends use_backend openproject if cloud openproject use_backend cloud if cloud use_backend rspamd if rspamd default_backend wordpress frontend http-in mode http bind *:80 redirect scheme https code 301 backend wordpress mode http server wordpress 10.12.34.46:80 backend cloud mode http server cloud 10.12.34.185:80 backend openproject mode http server openproject 10.12.34.240:80 backend rspamd mode http server rspamd 10.12.34.175:11334
