Today while testing my LVS-DR I have been running into the situation where keepalived gets itself into some kind of sync issue. Basically both directors end up as master and then nothing works after that. Here is a snip from logs:
DIRECTOR 1: Jul 23 21:52:50 grp-01-00-50 Keepalived: Starting Keepalived v1.1.13 (03/26,2007) Jul 23 21:52:50 grp-01-00-50 Keepalived: Starting Healthcheck child process, pid=29059 Jul 23 21:52:50 grp-01-00-50 Keepalived_vrrp: Using MII-BMSR NIC polling thread... Jul 23 21:52:50 grp-01-00-50 Keepalived_vrrp: Netlink reflector reports IP 192.168.1.150 added Jul 23 21:52:50 grp-01-00-50 Keepalived_vrrp: Registering Kernel netlink reflector Jul 23 21:52:50 grp-01-00-50 Keepalived_vrrp: Registering Kernel netlink command channel Jul 23 21:52:50 grp-01-00-50 Keepalived_vrrp: Registering gratutious ARP shared channel Jul 23 21:52:50 grp-01-00-50 Keepalived: Starting VRRP child process, pid=29060 Jul 23 21:52:50 grp-01-00-50 kernel: IPVS: sync thread stopped! Jul 23 21:52:50 grp-01-00-50 Keepalived_vrrp: Configuration is using : 34940 Bytes Jul 23 21:52:50 grp-01-00-50 Keepalived_healthcheckers: Using MII-BMSR NIC polling thread... Jul 23 21:52:50 grp-01-00-50 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.1.150 added Jul 23 21:52:50 grp-01-00-50 Keepalived_healthcheckers: Registering Kernel netlink reflector Jul 23 21:52:50 grp-01-00-50 Keepalived_healthcheckers: Registering Kernel netlink command channel Jul 23 21:52:50 grp-01-00-50 Keepalived_healthcheckers: Configuration is using : 9125 Bytes Jul 23 21:52:50 grp-01-00-50 Keepalived_healthcheckers: Activating healtchecker for service [192.168.1.200:80] Jul 23 21:52:50 grp-01-00-50 Keepalived_healthcheckers: Activating healtchecker for service [192.168.1.201:80] Jul 23 21:52:50 grp-01-00-50 Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(8,9)] Jul 23 21:52:50 grp-01-00-50 kernel: IPVS: sync thread started: state = MASTER, mcast_ifn = eth0, syncid = 25 Jul 23 21:52:52 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE Jul 23 21:52:54 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE Jul 23 21:52:54 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs. Jul 23 21:52:54 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.240 Jul 23 21:52:54 grp-01-00-50 Keepalived_vrrp: Netlink: skipping nl_cmd msg... Jul 23 21:52:54 grp-01-00-50 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.1.240 added Jul 23 21:52:54 grp-01-00-50 : /etc/keepalived/ip_localhost (called from: keepalived): rsh 192.168.1.200 /sbin/ip addr add 192.168.1.240/32 dev lo brd + scope host Jul 23 21:52:55 grp-01-00-50 : /etc/keepalived/ip_localhost (called from: keepalived): RTNETLINK answers: File exists Jul 23 21:52:55 grp-01-00-50 : /etc/keepalived/ip_localhost (called from: keepalived): rsh 192.168.1.201 /sbin/ip addr add 192.168.1.240/32 dev lo brd + scope host Jul 23 21:52:55 grp-01-00-50 : /etc/keepalived/ip_localhost (called from: keepalived): RTNETLINK answers: File exists Jul 23 21:52:55 grp-01-00-50 : /etc/keepalived/ip_localhost (called from: keepalived): ip addr del 192.168.1.240/32 dev lo Jul 23 21:52:55 grp-01-00-50 : /etc/keepalived/ip_localhost (called from: keepalived): RTNETLINK answers: Cannot assign requested address Jul 23 21:52:59 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.240 Jul 23 21:53:39 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) Received lower prio advert, forcing new election Jul 23 21:53:39 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.240 Jul 23 21:53:41 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) Received lower prio advert, forcing new election Jul 23 21:53:41 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.240 Jul 23 21:53:43 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) Received lower prio advert, forcing new election Jul 23 21:53:43 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.240 Jul 23 21:53:45 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) Received lower prio advert, forcing new election Jul 23 21:53:45 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.240 Jul 23 21:53:47 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) Received lower prio advert, forcing new election Jul 23 21:53:47 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.240 Jul 23 21:53:49 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) Received lower prio advert, forcing new election ... forever DIRECTOR 2: Jul 23 21:53:32 grp-01-00-51 Keepalived: Starting Keepalived v1.1.13 (03/26,2007) Jul 23 21:53:32 grp-01-00-51 Keepalived: Starting Healthcheck child process, pid=27215 Jul 23 21:53:32 grp-01-00-51 Keepalived_vrrp: Using MII-BMSR NIC polling thread... Jul 23 21:53:32 grp-01-00-51 Keepalived_vrrp: Netlink reflector reports IP 192.168.1.151 added Jul 23 21:53:32 grp-01-00-51 Keepalived_vrrp: Registering Kernel netlink reflector Jul 23 21:53:32 grp-01-00-51 Keepalived_vrrp: Registering Kernel netlink command channel Jul 23 21:53:32 grp-01-00-51 Keepalived_vrrp: Registering gratutious ARP shared channel Jul 23 21:53:32 grp-01-00-51 Keepalived_vrrp: Configuration is using : 34940 Bytes Jul 23 21:53:33 grp-01-00-51 Keepalived: Starting VRRP child process, pid=27217 Jul 23 21:53:33 grp-01-00-51 Keepalived_healthcheckers: Using MII-BMSR NIC polling thread... Jul 23 21:53:33 grp-01-00-51 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.1.151 added Jul 23 21:53:33 grp-01-00-51 Keepalived_healthcheckers: Registering Kernel netlink reflector Jul 23 21:53:33 grp-01-00-51 Keepalived_healthcheckers: Registering Kernel netlink command channel Jul 23 21:53:33 grp-01-00-51 Keepalived_healthcheckers: Configuration is using : 9125 Bytes Jul 23 21:53:33 grp-01-00-51 Keepalived_healthcheckers: Activating healtchecker for service [192.168.1.200:80] Jul 23 21:53:33 grp-01-00-51 Keepalived_healthcheckers: Activating healtchecker for service [192.168.1.201:80] Jul 23 21:53:33 grp-01-00-51 Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE Jul 23 21:53:33 grp-01-00-51 Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(8,9)] Jul 23 21:53:33 grp-01-00-51 : /etc/keepalived/ip_localhost (called from: keepalived): ip addr add 192.168.1.240/32 dev lo brd + scope host Jul 23 21:53:33 grp-01-00-51 kernel: IPVS: sync thread started: state = BACKUP, mcast_ifn = eth0, syncid = 25 Jul 23 21:53:39 grp-01-00-51 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE Jul 23 21:53:41 grp-01-00-51 Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE Jul 23 21:53:41 grp-01-00-51 Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs. Jul 23 21:53:41 grp-01-00-51 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.240 Jul 23 21:53:41 grp-01-00-51 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.1.240 added Jul 23 21:53:41 grp-01-00-51 Keepalived_vrrp: Netlink: skipping nl_cmd msg... Jul 23 21:53:41 grp-01-00-51 avahi-daemon[2086]: Registering new address record for 192.168.1.240 on eth0.IPv4. Jul 23 21:53:41 grp-01-00-51 kernel: IPVS: stopping sync thread 27223 ... Jul 23 21:53:41 grp-01-00-51 : /etc/keepalived/ip_localhost (called from: keepalived): rsh 192.168.1.200 /sbin/ip addr add 192.168.1.240/32 dev lo brd + scope host Jul 23 21:53:41 grp-01-00-51 : /etc/keepalived/ip_localhost (called from: keepalived): poll: protocol failure in circuit setup Jul 23 21:53:41 grp-01-00-51 : /etc/keepalived/ip_localhost (called from: keepalived): rsh 192.168.1.201 /sbin/ip addr add 192.168.1.240/32 dev lo brd + scope host Jul 23 21:53:41 grp-01-00-51 : /etc/keepalived/ip_localhost (called from: keepalived): poll: protocol failure in circuit setup Jul 23 21:53:42 grp-01-00-51 : /etc/keepalived/ip_localhost (called from: keepalived): ip addr del 192.168.1.240/32 dev lo Jul 23 21:53:42 grp-01-00-51 kernel: IPVS: sync thread stopped! Jul 23 21:53:42 grp-01-00-51 kernel: IPVS: sync thread started: state = MASTER, mcast_ifn = eth0, syncid = 25 Jul 23 21:53:46 grp-01-00-51 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.240 [EMAIL PROTECTED] keepalived]# Anyway I found that this was a firewall issue. Once that was fixed this sync problem went away. Just thought I would post it in case someone else runs into this. Gerry _______________________________________________ LinuxVirtualServer.org mailing list - [email protected] Send requests to [EMAIL PROTECTED] or go to http://lists.graemef.net/mailman/listinfo/lvs-users
