------- Comment From [email protected] 2016-11-28 13:07 EDT-------
(In reply to comment #13)
> Dave Wilder, do you have a way to trigger bug/test fix that wouldn't require
> a full openstack + HA-VRPP setup?
Here is my test script and the keepalived config I used (generated by
openstack). Openstack is not needed to run my test script, just the
config file.
vrrp_instance VR_1 {
state BACKUP
interface ha-99a3cb02-dc
virtual_router_id 1
priority 50
garp_master_delay 60
nopreempt
advert_int 2
track_interface {
ha-99a3cb02-dc
}
virtual_ipaddress {
169.254.0.1/24 dev ha-99a3cb02-dc
}
virtual_ipaddress_excluded {
27.0.0.1/24 dev qr-bc6c9831-52
9.47.64.9/20 dev qg-9b2d21c4-59
fe80::f816:3eff:fe95:9c41/64 dev qg-9b2d21c4-59 scope link
fe80::f816:3eff:febb:d0ff/64 dev qr-bc6c9831-52 scope link
}
virtual_routes {
0.0.0.0/0 via 9.47.79.254 dev qg-9b2d21c4-59
}
}
The test script. Note: this script will not generate a working
keepalived setup but it is sufficient to demonstrate the bug and verify
the fix.
#!/bin/bash
# List of interfaces to create
Interfaces="ha-99a3cb02-dc qr-bc6c9831-52 qg-9b2d21c4-59"
# Un-comment to generate 100 extra veth pairs
# Interfaces="`seq 1 100` ha-99a3cb02-dc qr-bc6c9831-52 qg-9b2d21c4-59"
for i in $Interfaces; do
echo Creaating $i
ip link add $i type veth peer name v-$PEER
ifconfig $i up
PEER=$(($PEER+1))
done
# KEEPALIVED="/home/wilder/scratch/keepalived-1.2.24/keepalived/keepalived"
KEEPALIVED="keepalived"
CONF=$PWD/test.conf
$KEEPALIVED -d -n -l -D -f $CONF
echo Done, cleaning up
for i in $Interfaces; do
ip link del $i
done
-------
( test run showing the problem )
# ./setup
Creaating ha-99a3cb02-dc
Creaating qr-bc6c9831-52
Creaating qg-9b2d21c4-59
Starting Keepalived v1.2.24 (11/11,2016)
Opening file '/home/wilder/scratch/test.conf'.
Starting Healthcheck child process, pid=1130854
Starting VRRP child process, pid=1130855
Initializing ipvs
Netlink: error: message truncated <<<<< ****
Netlink: error: message truncated <<<<<<
Netlink: error: message truncated
Netlink: error: message truncated
Netlink: error: message truncated
Netlink: error: message truncated
Netlink reflector reports IP 192.168.0.2 added
Netlink reflector reports IP 192.168.0.2 added
Netlink reflector reports IP 172.17.0.1 added
Netlink reflector reports IP 172.17.0.1 added
Netlink reflector reports IP fe80::9abe:94ff:fe0d:f2f4 added
Netlink reflector reports IP fe80::9abe:94ff:fe0d:f2f4 added
Netlink reflector reports IP fe80::42:dbff:fe53:c725 added
Netlink reflector reports IP fe80::42:dbff:fe53:c725 added
Registering Kernel netlink reflector
Registering Kernel netlink reflector
Registering Kernel netlink command channel
Registering Kernel netlink command channel
Registering gratuitous ARP shared channel
Opening file '/home/wilder/scratch/test.conf'.
Opening file '/home/wilder/scratch/test.conf'.
Cant find interface ha-99a3cb02-dc for vrrp_instance VR_1 !!!
ha-99a3cb02-dc no match, ignoring...
VRRP is trying to assign ip address 169.254.0.1/24 to unknown ha-99a3cb02-dc
interface !!! go out and fix your conf !!!
VRRP is trying to assign ip address 9.47.64.9/20 to unknown qg-9b2d21c4-59
interface !!! go out and fix your conf !!!
VRRP is trying to assign ip address fe80::f816:3eff:fe95:9c41/64 to unknown
qg-9b2d21c4-59 interface !!! go out and fix your conf !!! <<<<<<<<*****
Cannot specify scope for IPv6 addresses (fe80::f816:3eff:febb:d0ff/64) -
ignoring scope
VRRP is trying to assign VROUTE to unknown qg-9b2d21c4-59 interface !!! go out
and fix your conf !!!
VRRP_Instance(VR_1) Unknown interface !
------< Global definitions >------
Router ID = docker
Default interface = eth0
LVS flush = false
VRRP IPv4 mcast group = 224.0.0.18
VRRP IPv6 mcast group = ff02::12
Gratuitous ARP delay = 5
Gratuitous ARP repeat = 5
Gratuitous ARP refresh timer = 0
Gratuitous ARP refresh repeat = 1
Gratuitous ARP lower priority delay = 0
Gratuitous ARP lower priority repeat = -1
Send advert after receive lower priority advert = true
Gratuitous ARP interval = 0
Gratuitous NA interval = 0
VRRP default protocol version = 2
Iptables input chain = INPUT
VRRP check unicast_src = false
VRRP skip check advert addresses = false
VRRP strict mode = false
VRRP process priority = 0
VRRP don't swap = false
Checker process priority = 0
Checker don't swap = false
Network namespace = (default)
------< SSL definitions >------
Using autogen SSL context
Using LinkWatch kernel netlink reflector...
Stopped
pid 11308 exited with permanent error CONFIG. Terminating
Stopping
Stopped
Stopped Keepalived v1.2.24 (11/11,2016)
Here is a test run with my change.
# ./setup
Creaating ha-99a3cb02-dc
Creaating qr-bc6c9831-52
Creaating qg-9b2d21c4-59
Starting Keepalived v1.2.24 (11/11,2016)
Opening file '/home/wilder/scratch/test.conf'.
Starting Healthcheck child process, pid=1131124
Starting VRRP child process, pid=1131125
Initializing ipvs
Netlink reflector reports IP 192.168.0.2 added
Netlink reflector reports IP 192.168.0.2 added
Netlink reflector reports IP 9.114.216.147 added
Netlink reflector reports IP 9.114.216.147 added
Netlink reflector reports IP 192.168.0.2 added
Netlink reflector reports IP 192.168.0.2 added
Netlink reflector reports IP 172.17.0.1 added
Netlink reflector reports IP 172.17.0.1 added
Netlink reflector reports IP 172.18.0.1 added
Netlink reflector reports IP 172.18.0.1 added
Netlink reflector reports IP fe80::9abe:94ff:fe0d:f2f4 added
Netlink reflector reports IP fd55:faaf:e1ab:3df:9abe:94ff:fe0d:f2f6 added
Netlink reflector reports IP fe80::9abe:94ff:fe0d:f2f6 added
Netlink reflector reports IP fe80::42:dbff:fe53:c725 added
Netlink reflector reports IP fe80::9abe:94ff:fe0d:f2f4 added
Netlink reflector reports IP fe80::42:93ff:fe45:7d65 added
Netlink reflector reports IP fd55:faaf:e1ab:3df:9abe:94ff:fe0d:f2f6 added
Registering Kernel netlink reflector
Netlink reflector reports IP fe80::9abe:94ff:fe0d:f2f6 added
Registering Kernel netlink command channel
Netlink reflector reports IP fe80::42:dbff:fe53:c725 added
Netlink reflector reports IP fe80::42:93ff:fe45:7d65 added
Registering gratuitous ARP shared channel
Registering Kernel netlink reflector
Registering Kernel netlink command channel
Opening file '/home/wilder/scratch/test.conf'.
Opening file '/home/wilder/scratch/test.conf'.
Cannot specify scope for IPv6 addresses (fe80::f816:3eff:fe95:9c41/64) -
ignoring scope
Cannot specify scope for IPv6 addresses (fe80::f816:3eff:febb:d0ff/64) -
ignoring scope
(VR_1): Cannot find an IP address to use for interface ha-99a3cb02-dc
------< Global definitions >------
Router ID = docker
Default interface = eth0
LVS flush = false
VRRP IPv4 mcast group = 224.0.0.18
VRRP IPv6 mcast group = ff02::12
Gratuitous ARP delay = 5
Gratuitous ARP repeat = 5
Gratuitous ARP refresh timer = 0
Gratuitous ARP refresh repeat = 1
Gratuitous ARP lower priority delay = 0
Gratuitous ARP lower priority repeat = -1
Send advert after receive lower priority advert = true
Gratuitous ARP interval = 0
Gratuitous NA interval = 0
VRRP default protocol version = 2
Iptables input chain = INPUT
VRRP check unicast_src = false
VRRP skip check advert addresses = false
VRRP strict mode = false
VRRP process priority = 0
VRRP don't swap = false
Checker process priority = 0
Checker don't swap = false
Network namespace = (default)
Regards
David Wilder
--
You received this bug notification because you are a member of Ubuntu
High Availability Team, which is subscribed to keepalived in Ubuntu.
https://bugs.launchpad.net/bugs/1642763
Title:
keepalived raising VIP apply error
Status in keepalived package in Ubuntu:
Triaged
Bug description:
== Comment: #0 - Andrew Thorstensen - 2016-11-17 09:50:25 ==
---Problem Description---
Using Ubuntu 16.04 on ppc64le, we are building a 'neutron network node' using
the VRRP configuration (built on keepalived).
Information on this OpenStack configuration can be found here:
https://wiki.openstack.org/wiki/Neutron/L3_High_Availability_VRRP
When we run, the configuration is failing to apply via keepalived.
The logs post the following:
Nov 17 02:58:31 p8test-lp1 Keepalived_vrrp[54542]: VRRP is trying to assign
VIP to unknown qr-a5f5ba96-52 interface !!! go out and fix your conf !!!
However, the device DOES exist. But the keepalived config just doesn't
always deploy it.
ii keepalived 1:1.2.19-1
ppc64el Failover and monitoring daemon for LVS clusters
This configuration sometimes works, but does sometimes fail on Ubuntu 16.04.1
---uname output---
Linux p8test-lp1 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:38:24 UTC 2016
ppc64le ppc64le ppc64le GNU/Linux
---Additional Hardware Info---
This is a Power8 system with Ubuntu 16.04.1 installed. Though we see no
indication that this is specific to Power.
Machine Type = S822L
Machine Type = 8286-42A
---Steps to Reproduce---
Install openstack. Run the network node in a VRRP HA configuration. Create
a router and assign a global IP.
== Comment: #5 - David J. Wilder - 2016-11-17 15:58:04 ==
The problem is fixed in this upstream commit:
https://github.com/acassen/keepalived/commit/9f327bbf3e86def1055a106eda0633638bda0345
On systems with a page size larger than 4096 keepalived may report:
"Netlink: error: message truncated" messages
This error was reported on a ppc64le in an OpenStack/Nutron environment.
Ppc64le is using a 64k pages size. I found that keepalived's netlink recvmsg
buffer was too small causing messages to be truncated. The size of the read
buffer for the netlink socket should be based on page size however, it should
not exceed 8192. See the comment in the patch.
I tested the fix by creating 100 veth interfaces and verifying the errors
did not return.
Signed-off-by: David Wilder <[email protected]>
Signed-off-by: Quentin Armitage <[email protected]>
...
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1642763/+subscriptions
_______________________________________________
Mailing list: https://launchpad.net/~ubuntu-ha
Post to : [email protected]
Unsubscribe : https://launchpad.net/~ubuntu-ha
More help : https://help.launchpad.net/ListHelp
