Verified to be working on Bionic using the provided test case and
another simpler one (simply stopping haproxy resulted in the error/143
status).
Preparing to unpack .../haproxy_1.8.8-1ubuntu0.8_amd64.deb ...
Unpacking haproxy (1.8.8-1ubuntu0.8) over (1.8.8-1ubuntu0.7) ...
Setting up haproxy (1.8.8-1ubuntu0.8) ...
Processing triggers for systemd (237-3ubuntu10.33) ...
Processing triggers for rsyslog (8.32.0-1ubuntu4) ...
root@hap1:~# for x in {1..1000}; do pkill -TERM -x haproxy ; sleep 0.001;
systemctl reset-failed haproxy.service; done
root@hap1:~# systemctl status haproxy
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset:
enabled)
Active: active (running) since Thu 2019-11-28 15:20:01 UTC; 1s ago
Docs: man:haproxy(1)
file:/usr/share/doc/haproxy/configuration.txt.gz
Process: 8076 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS
(code=exited, status=0/SUCCESS)
Main PID: 8077 (haproxy)
Tasks: 2 (limit: 4915)
CGroup: /system.slice/haproxy.service
├─8077 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid
└─8078 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid
Nov 28 15:20:01 hap1 systemd[1]: Starting HAProxy Load Balancer...
Nov 28 15:20:01 hap1 systemd[1]: Started HAProxy Load Balancer.
** Tags removed: verification-needed verification-needed-bionic
** Tags added: verification-done verification-done-bionic
--
You received this bug notification because you are a member of Ubuntu
High Availability Team, which is subscribed to haproxy in Ubuntu.
https://bugs.launchpad.net/bugs/1848902
Title:
haproxy in bionic can get stuck
Status in haproxy package in Ubuntu:
Fix Released
Status in haproxy source package in Bionic:
Fix Committed
Bug description:
[Impact]
* The master process will exit with the status of the last worker.
When the worker is killed with SIGTERM, it is expected to get 143 as an
exit status. Therefore, we consider this exit status as normal from a
systemd point of view. If it happens when not stopping, the systemd
unit is configured to always restart, so it has no adverse effect.
* Backport upstream fix - adding another accepted RC to the systemd
service
[Test Case]
* You want to install haproxy and have it running. Then sigterm it a lot.
With the fix it would restart the service all the time, well except
restart limit. But in the bad case it will just stay down and didn't
even try to restart it.
$ apt install haproxy
$ for x in {1..100}; do pkill -TERM -x haproxy ; sleep 0.1 ; done
$ systemctl status haproxy
The above is a hacky way to trigger some A/B behavior on the fix.
It isn't perfect as systemd restart counters will kick in and you
essentially check a secondary symptom.
I'd recommend to in addition run the following:
$ apt install haproxy
$ for x in {1..1000}; do pkill -TERM -x haproxy ; sleep 0.001 systemctl
reset-failed haproxy.service; done
$ systemctl status haproxy
You can do so with even smaller sleeps, that should keep the service up
and running (this isn't changing with the fix, but should work with the
new code).
[Regression Potential]
* This eventually is a conffile modification, so if there are other
modifications done by the user they will get a prompt. But that isn't a
regression. I checked the code and I can't think of another RC=143 that
would due to that "no more" detected as error. I really think other
than the update itself triggering a restart (as usual for services)
there is no further regression potential to this.
[Other Info]
* Fix already active in IS hosted cloud without issues since a while
* Also reports (comment #5) show that others use this in production as
well
---
On a Bionic/Stein cloud, after a network partition, we saw several
units (glance, swift-proxy and cinder) fail to start haproxy, like so:
root@juju-df624b-6-lxd-4:~# systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor
preset: enabled)
Active: failed (Result: exit-code) since Sun 2019-10-20 00:23:18 UTC; 1h
35min ago
Docs: man:haproxy(1)
file:/usr/share/doc/haproxy/configuration.txt.gz
Process: 2002655 ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE
$EXTRAOPTS (code=exited, status=143)
Process: 2002649 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS
(code=exited, status=0/SUCCESS)
Main PID: 2002655 (code=exited, status=143)
Oct 20 00:16:52 juju-df624b-6-lxd-4 systemd[1]: Starting HAProxy Load
Balancer...
Oct 20 00:16:52 juju-df624b-6-lxd-4 systemd[1]: Started HAProxy Load Balancer.
Oct 20 00:23:18 juju-df624b-6-lxd-4 systemd[1]: Stopping HAProxy Load
Balancer...
Oct 20 00:23:18 juju-df624b-6-lxd-4 haproxy[2002655]: [WARNING] 292/001652
(2002655) : Exiting Master process...
Oct 20 00:23:18 juju-df624b-6-lxd-4 haproxy[2002655]: [ALERT] 292/001652
(2002655) : Current worker 2002661 exited with code 143
Oct 20 00:23:18 juju-df624b-6-lxd-4 haproxy[2002655]: [WARNING] 292/001652
(2002655) : All workers exited. Exiting... (143)
Oct 20 00:23:18 juju-df624b-6-lxd-4 systemd[1]: haproxy.service: Main process
exited, code=exited, status=143/n/a
Oct 20 00:23:18 juju-df624b-6-lxd-4 systemd[1]: haproxy.service: Failed with
result 'exit-code'.
Oct 20 00:23:18 juju-df624b-6-lxd-4 systemd[1]: Stopped HAProxy Load Balancer.
root@juju-df624b-6-lxd-4:~#
The Debian maintainer came up with the following patch for this:
https://www.mail-archive.com/[email protected]/msg30477.html
Which was added to the 1.8.10-1 Debian upload and merged into upstream 1.8.13.
Unfortunately Bionic is on 1.8.8-1ubuntu0.4 and doesn't have this patch.
Please consider pulling this patch into an SRU for Bionic.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/haproxy/+bug/1848902/+subscriptions
_______________________________________________
Mailing list: https://launchpad.net/~ubuntu-ha
Post to : [email protected]
Unsubscribe : https://launchpad.net/~ubuntu-ha
More help : https://help.launchpad.net/ListHelp
