[Ubuntu-ha] [Bug 1792298] Re: keepalived: MISC healthchecker's exit status is erroneously treated as a permanent error
That patch is applied upstream in the package shipped in bionic and
later
** Also affects: keepalived (Ubuntu Xenial)
Importance: Undecided
Status: New
** Changed in: keepalived (Ubuntu)
Status: Triaged => Fix Released
** Changed in: keepalived (Ubuntu Xenial)
Status: New => Triaged
** Changed in: keepalived (Ubuntu Xenial)
Importance: Undecided => Medium
--
You received this bug notification because you are a member of Ubuntu
High Availability Team, which is subscribed to keepalived in Ubuntu.
https://bugs.launchpad.net/bugs/1792298
Title:
keepalived: MISC healthchecker's exit status is erroneously treated as
a permanent error
Status in keepalived package in Ubuntu:
Fix Released
Status in keepalived source package in Xenial:
Triaged
Bug description:
1) The release of Ubuntu we are using
$ lsb_release -rd
Description:Ubuntu 16.04.5 LTS
Release:16.04
2) The version of the package we are using
$ apt-cache policy keepalived
keepalived:
Installed: 1:1.2.24-1ubuntu0.16.04.1
...
3) What we expected to happen
MISC healthcheckers would be treated normally.
4) What happened instead
We are trying to use Ubuntu 16.04's keepalived with our own MISC
healthchecker, which is implemented to exit with exit code 3, and getting the
following log messages endlessly.
--- Note: some IP fields are masked ---
Sep 12 06:55:09 devsvr Keepalived[16705]: Healthcheck child process(34232)
died: Respawning
Sep 12 06:55:09 devsvr Keepalived[16705]: Starting Healthcheck child process,
pid=34239
Sep 12 06:55:09 devsvr Keepalived_healthcheckers[34239]: Initializing ipvs
Sep 12 06:55:09 devsvr Keepalived_healthcheckers[34239]: Registering Kernel
netlink reflector
Sep 12 06:55:09 devsvr Keepalived_healthcheckers[34239]: Registering Kernel
netlink command channel
Sep 12 06:55:09 devsvr Keepalived_healthcheckers[34239]: Opening file
'/etc/keepalived/keepalived.conf'.
Sep 12 06:55:09 devsvr Keepalived_healthcheckers[34239]: Using LinkWatch
kernel netlink reflector...
Sep 12 06:55:09 devsvr Keepalived_healthcheckers[34239]: Activating
healthchecker for service [XX.XX.XX.18]:80
Sep 12 06:55:09 devsvr Keepalived_healthcheckers[34239]: Activating
healthchecker for service [XX.XX.XX.19]:80
Sep 12 06:55:09 devsvr Keepalived_healthcheckers[34239]: Activating
healthchecker for service [XX.XX.XX.18]:443
Sep 12 06:55:09 devsvr Keepalived_healthcheckers[34239]: Activating
healthchecker for service [XX.XX.XX.19]:443
...
Sep 12 06:55:09 devsvr Keepalived_healthcheckers[34239]: Activating
healthchecker for service [XX.XX.XX.52]:443
Sep 12 06:55:09 devsvr Keepalived_healthcheckers[34239]: Activating
healthchecker for service [XX.XX.XX.53]:443
Sep 12 06:55:10 devsvr Keepalived_healthcheckers[34239]: pid 34257 exited
with permanent error CONFIG. Terminating
Sep 12 06:55:10 devsvr Keepalived_healthcheckers[34239]: Removing service
[XX.XX.XX.24]:25 from VS [YY.YY.YY.YY]:0
Sep 12 06:55:10 devsvr Keepalived_healthcheckers[34239]: Removing service
[XX.XX.XX.25]:25 from VS [YY.YY.YY.YY]:0
Sep 12 06:55:10 devsvr Keepalived_healthcheckers[34239]: Removing service
[XX.XX.XX.21]:56667 from VS [ZZ.ZZ.ZZ.ZZ]:0
Sep 12 06:55:10 devsvr Keepalived_healthcheckers[34239]: Removing service
[XX.XX.XX.52]:443 from VS [WW.WW.WW.WW]:0
Sep 12 06:55:10 devsvr Keepalived[16705]: Healthcheck child process(34239)
died: Respawning
Sep 12 06:55:10 devsvr Keepalived[16705]: Starting Healthcheck child process,
pid=34260
...
---
It looks like our MISC healthchecker's exit code 3, which should be a
valid value according to the following description, is treated as a
permanent error since it is equal to KEEPALIVED_EXIT_CONFIG defined in
keepalived's lib/scheduler.h :
---
# MISC healthchecker, run a program
MISC_CHECK
{
# External script or program
...
# exit status 2-255: svc check success, weight
# changed to 2 less than exit status.
# (for example: exit status of 255 would set
# weight to 253)
misc_dynamic
}
---
The problem, we think, have started with this patch (we did not see the
problem in Ubuntu 14.04):
Stop respawning children repeatedly after permanent error
-
https://github.com/acassen/keepalived/commit/4ae9314af448eb8ea4f3d8ef39bcc469779b0fec
The problem will be fixed by this patch (not included in Ubuntu 16.04):
Make report_child_status() check for vrrp and checker child processes
-
https://github.com/acassen/keepalived/commit/ca955a7c1a6af324428ff04e24be68a180be127f
Please consider backporting it to Ubuntu 16.04's keepalived.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1792298/+subscriptions
___
Mailing
[Ubuntu-ha] [Bug 1792298] Re: keepalived: MISC healthchecker's exit status is erroneously treated as a permanent error
Thanks for the patch and testing instructions
** Tags added: bitesize server-next
** Changed in: keepalived (Ubuntu)
Status: Incomplete => Triaged
** Changed in: keepalived (Ubuntu)
Importance: Undecided => Medium
--
You received this bug notification because you are a member of Ubuntu
High Availability Team, which is subscribed to keepalived in Ubuntu.
https://bugs.launchpad.net/bugs/1792298
Title:
keepalived: MISC healthchecker's exit status is erroneously treated as
a permanent error
Status in keepalived package in Ubuntu:
Fix Released
Status in keepalived source package in Xenial:
Triaged
Bug description:
1) The release of Ubuntu we are using
$ lsb_release -rd
Description:Ubuntu 16.04.5 LTS
Release:16.04
2) The version of the package we are using
$ apt-cache policy keepalived
keepalived:
Installed: 1:1.2.24-1ubuntu0.16.04.1
...
3) What we expected to happen
MISC healthcheckers would be treated normally.
4) What happened instead
We are trying to use Ubuntu 16.04's keepalived with our own MISC
healthchecker, which is implemented to exit with exit code 3, and getting the
following log messages endlessly.
--- Note: some IP fields are masked ---
Sep 12 06:55:09 devsvr Keepalived[16705]: Healthcheck child process(34232)
died: Respawning
Sep 12 06:55:09 devsvr Keepalived[16705]: Starting Healthcheck child process,
pid=34239
Sep 12 06:55:09 devsvr Keepalived_healthcheckers[34239]: Initializing ipvs
Sep 12 06:55:09 devsvr Keepalived_healthcheckers[34239]: Registering Kernel
netlink reflector
Sep 12 06:55:09 devsvr Keepalived_healthcheckers[34239]: Registering Kernel
netlink command channel
Sep 12 06:55:09 devsvr Keepalived_healthcheckers[34239]: Opening file
'/etc/keepalived/keepalived.conf'.
Sep 12 06:55:09 devsvr Keepalived_healthcheckers[34239]: Using LinkWatch
kernel netlink reflector...
Sep 12 06:55:09 devsvr Keepalived_healthcheckers[34239]: Activating
healthchecker for service [XX.XX.XX.18]:80
Sep 12 06:55:09 devsvr Keepalived_healthcheckers[34239]: Activating
healthchecker for service [XX.XX.XX.19]:80
Sep 12 06:55:09 devsvr Keepalived_healthcheckers[34239]: Activating
healthchecker for service [XX.XX.XX.18]:443
Sep 12 06:55:09 devsvr Keepalived_healthcheckers[34239]: Activating
healthchecker for service [XX.XX.XX.19]:443
...
Sep 12 06:55:09 devsvr Keepalived_healthcheckers[34239]: Activating
healthchecker for service [XX.XX.XX.52]:443
Sep 12 06:55:09 devsvr Keepalived_healthcheckers[34239]: Activating
healthchecker for service [XX.XX.XX.53]:443
Sep 12 06:55:10 devsvr Keepalived_healthcheckers[34239]: pid 34257 exited
with permanent error CONFIG. Terminating
Sep 12 06:55:10 devsvr Keepalived_healthcheckers[34239]: Removing service
[XX.XX.XX.24]:25 from VS [YY.YY.YY.YY]:0
Sep 12 06:55:10 devsvr Keepalived_healthcheckers[34239]: Removing service
[XX.XX.XX.25]:25 from VS [YY.YY.YY.YY]:0
Sep 12 06:55:10 devsvr Keepalived_healthcheckers[34239]: Removing service
[XX.XX.XX.21]:56667 from VS [ZZ.ZZ.ZZ.ZZ]:0
Sep 12 06:55:10 devsvr Keepalived_healthcheckers[34239]: Removing service
[XX.XX.XX.52]:443 from VS [WW.WW.WW.WW]:0
Sep 12 06:55:10 devsvr Keepalived[16705]: Healthcheck child process(34239)
died: Respawning
Sep 12 06:55:10 devsvr Keepalived[16705]: Starting Healthcheck child process,
pid=34260
...
---
It looks like our MISC healthchecker's exit code 3, which should be a
valid value according to the following description, is treated as a
permanent error since it is equal to KEEPALIVED_EXIT_CONFIG defined in
keepalived's lib/scheduler.h :
---
# MISC healthchecker, run a program
MISC_CHECK
{
# External script or program
...
# exit status 2-255: svc check success, weight
# changed to 2 less than exit status.
# (for example: exit status of 255 would set
# weight to 253)
misc_dynamic
}
---
The problem, we think, have started with this patch (we did not see the
problem in Ubuntu 14.04):
Stop respawning children repeatedly after permanent error
-
https://github.com/acassen/keepalived/commit/4ae9314af448eb8ea4f3d8ef39bcc469779b0fec
The problem will be fixed by this patch (not included in Ubuntu 16.04):
Make report_child_status() check for vrrp and checker child processes
-
https://github.com/acassen/keepalived/commit/ca955a7c1a6af324428ff04e24be68a180be127f
Please consider backporting it to Ubuntu 16.04's keepalived.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1792298/+subscriptions
___
Mailing list: https://launchpad.net/~ubuntu-ha
Post to : ubuntu-ha@lists.launchpad.net
Unsubscribe : https://launchpad.net/~ubuntu-ha
More help :
[Ubuntu-ha] [Bug 1627083] Re: ipt_CLUSTERIP is deprecated and it will removed soon, use xt_cluster instead
For strongswan, I found a reference in a 2018 workshop to work on xt_cluster support: https://wiki.strongswan.org/projects/strongswan/wiki/Linux_IPsec_Workshop_2018 No open bug reports about moving from ipt_CLUSTERIP to xt_cluster, just references in old bugs about how that was wanted, but just not done yet. For pacemaker, I couldn't find results even mentioning the problem, other than this bug. Looks like it will be some time still until ipt_CLUSTERIP is abandoned. ** Changed in: strongswan (Ubuntu) Importance: Undecided => Wishlist ** Changed in: pacemaker (Ubuntu) Importance: Undecided => Wishlist ** Changed in: strongswan (Ubuntu) Status: New => Triaged ** Changed in: pacemaker (Ubuntu) Status: New => Triaged -- You received this bug notification because you are a member of Ubuntu High Availability Team, which is subscribed to pacemaker in Ubuntu. https://bugs.launchpad.net/bugs/1627083 Title: ipt_CLUSTERIP is deprecated and it will removed soon, use xt_cluster instead Status in pacemaker package in Ubuntu: Triaged Status in strongswan package in Ubuntu: Triaged Bug description: pacemaker still uses iptable's "CLUSTERIP" -- and dmesg shows a deprecation warning: [ 15.027333] ipt_CLUSTERIP: ClusterIP Version 0.8 loaded successfully [ 15.027464] ipt_CLUSTERIP: ipt_CLUSTERIP is deprecated and it will removed soon, use xt_cluster instead ~# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination CLUSTERIP all -- anywhere proxy.charite.de CLUSTERIP hashmode=sourceip-sourceport clustermac=EF:EE:6B:F9:7B:67 total_nodes=4 local_node=2 hash_init=0 ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: pacemaker 1.1.14-2ubuntu1.1 ProcVersionSignature: Ubuntu 4.4.0-38.57-generic 4.4.19 Uname: Linux 4.4.0-38-generic x86_64 ApportVersion: 2.20.1-0ubuntu2.1 Architecture: amd64 Date: Fri Sep 23 17:26:01 2016 InstallationDate: Installed on 2014-08-19 (766 days ago) InstallationMedia: Ubuntu-Server 14.04.1 LTS "Trusty Tahr" - Release amd64 (20140722.3) SourcePackage: pacemaker UpgradeStatus: Upgraded to xenial on 2016-09-22 (1 days ago) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/pacemaker/+bug/1627083/+subscriptions ___ Mailing list: https://launchpad.net/~ubuntu-ha Post to : ubuntu-ha@lists.launchpad.net Unsubscribe : https://launchpad.net/~ubuntu-ha More help : https://help.launchpad.net/ListHelp
