Hello,
First time mailing here, so hopefully I'm at the right place.
I use Ansible to deploy our haproxy configuration, and the following
scenario might happen:
1. Haproxy is running on an existing system
2. I execute the Ansible playbook.
3. This playbook changes the haproxy config (which will trigger a restart
of haproxy at the end of the run due to an ansible handler)
4. The same playbook also updates Haproxy due to a new package being
available from the repository. This package automatically reloads haproxy.
5. Shortly after the automatic reload from the haproxy package, Ansible
triggers a restart of haproxy due to the config change we did at 3. )
This works as expected when there are no active sessions in haproxy.
However, if there is an active session, the active session will still work,
but no new connections will be accepted ("connection refused"). The call to
restart haproxy will also hang for 1.5 minutes. Please see the following
log:
# Reload happening here due to the apt upgrade of haproxy
Oct 4 13:01:03 icinga01 systemd[1]: Reloading HAProxy Load Balancer.
Oct 4 13:01:03 icinga01 systemd[1]: Reloaded HAProxy Load Balancer.
# restart happening here due to the ansible handler
Oct 4 13:01:18 icinga01 ansible-systemd: Invoked with no_block=False
name=haproxy enabled=None daemon_reload=False state=restarted user=False
masked=None
Oct 4 13:01:18 icinga01 systemd[1]: Stopping HAProxy Load Balancer...
Oct 4 13:02:48 icinga01 systemd[1]: haproxy.service: State 'stop-sigterm'
timed out. Killing.
Oct 4 13:02:48 icinga01 systemd[1]: haproxy.service: Killing process 14410
(haproxy-systemd) with signal SIGKILL.
Oct 4 13:02:48 icinga01 systemd[1]: haproxy.service: Killing process 14413
(haproxy) with signal SIGKILL.
Oct 4 13:02:48 icinga01 systemd[1]: haproxy.service: Killing process 14414
(haproxy) with signal SIGKILL.
Oct 4 13:02:48 icinga01 systemd[1]: haproxy.service: Main process exited,
code=killed, status=9/KILL
Oct 4 13:02:48 icinga01 systemd[1]: haproxy.service: Killing process 14413
(haproxy) with signal SIGKILL.
Oct 4 13:02:48 icinga01 systemd[1]: haproxy.service: Killing process 14414
(haproxy) with signal SIGKILL.
Oct 4 13:02:48 icinga01 systemd[1]: Stopped HAProxy Load Balancer.
Oct 4 13:02:48 icinga01 systemd[1]: haproxy.service: Unit entered failed
state.
Oct 4 13:02:48 icinga01 systemd[1]: haproxy.service: Failed with result
'timeout'.
Oct 4 13:02:48 icinga01 systemd[1]: Starting HAProxy Load Balancer...
Oct 4 13:02:48 icinga01 systemd[1]: Started HAProxy Load Balancer.
Please see the environment information:
apt-cache policy haproxy
haproxy:
Installed: 1.7.9-1~bpo9+1
Candidate: 1.7.9-1~bpo9+1
Version table:
*** 1.7.9-1~bpo9+1 100
100 http://ftp.nl.debian.org/debian stretch-backports/main amd64
Packages
100 http://haproxy.debian.net stretch-backports-1.7/main amd64
Packages
100 http://httpredir.debian.org/debian stretch-backports/main amd64
Packages
100 /var/lib/dpkg/status
1.7.5-2 500
500 http://ftp.nl.debian.org/debian stretch/main amd64 Packages
This is a debian 9 amd64 installation. I have also seen the issue on Debian
8 however.
I have a minimal haproxy config to reproduce it:
cat /etc/haproxy/haproxy.cfg
global
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
listen icinga_ido
bind :::3306 v4v6
mode tcp
server icingasql01 <ip>:3306 check port 9100
server icingasql02 <ip>:3306 check port 9100
server icingasql03 <ip>:3306 check port 9100
If haproxy is running with this config, and you connect to MySQL through
haproxy (simply: mysql --h 127.0.0.1 u user -p) the session will be "open".
No need to actually perform a query.
While this session is open, the following will cause the issue:
systemctl reload haproxy && systemctl restart haproxy.service
During this time, the active connection will still work, but I'm unable to
open a new connection until the timeout 1.5 minutes later.
If I don't have any sessions open to MySQL the reload & restart will have
no noticeable delay/hang and it works as expected.
Is this a bug or is there a setting I should change to prevent this from
happening?
Thank you!
Niels Hendriks
