[Ubuntu-ha] [Bug 1642763] Re: keepalived raising VIP apply error
** Changed in: keepalived (Ubuntu) Assignee: Nish Aravamudan (nacc) => (unassigned) ** Changed in: keepalived (Ubuntu Xenial) Assignee: Nish Aravamudan (nacc) => (unassigned) ** Changed in: keepalived (Ubuntu Yakkety) Assignee: Nish Aravamudan (nacc) => (unassigned) -- You received this bug notification because you are a member of Ubuntu High Availability Team, which is subscribed to keepalived in Ubuntu. https://bugs.launchpad.net/bugs/1642763 Title: keepalived raising VIP apply error Status in keepalived package in Ubuntu: Fix Released Status in keepalived source package in Xenial: Fix Released Status in keepalived source package in Yakkety: Fix Released Bug description: [Impact] * keepalived on ppc64el (due to a large page size) experiences "Netlink: error: message truncated" messages. * These Netlink truncations result in keepalived thinking that the the underlying device does not exist, even though it does. [Test Case] * Creating 100 veth interfaces ppc64el and verify if "Netlink: error: message truncated" errors are emitted. If so, the bug is present. If not, the bug is fixed. [Regression Potential] * This is code issue, fixed upstream, in the keepalived code when the system page size exceeds 4096. The upstream fix was backported to all releases and should only properly limit the size of the buffer used for netlink to at most 8192 on systems with a page size greater than 8192. I believe risk of regression is very low. * Using the tests provided by David Wilder, I ran on both x86_64 and ppc64el LXD containers. Without the backported changes, I saw no issues on x86_64, and the reported issue on ppc64el (as expected, as a page size greater than 4K is required to see the buffer size issue). With the backported changes, both architectures show no issue with the provided testcase. --- == Comment: #0 - Andrew Thorstensen - 2016-11-17 09:50:25 == ---Problem Description--- Using Ubuntu 16.04 on ppc64le, we are building a 'neutron network node' using the VRRP configuration (built on keepalived). Information on this OpenStack configuration can be found here: https://wiki.openstack.org/wiki/Neutron/L3_High_Availability_VRRP When we run, the configuration is failing to apply via keepalived. The logs post the following: Nov 17 02:58:31 p8test-lp1 Keepalived_vrrp[54542]: VRRP is trying to assign VIP to unknown qr-a5f5ba96-52 interface !!! go out and fix your conf !!! However, the device DOES exist. But the keepalived config just doesn't always deploy it. ii keepalived 1:1.2.19-1 ppc64el Failover and monitoring daemon for LVS clusters This configuration sometimes works, but does sometimes fail on Ubuntu 16.04.1 ---uname output--- Linux p8test-lp1 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:38:24 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux ---Additional Hardware Info--- This is a Power8 system with Ubuntu 16.04.1 installed. Though we see no indication that this is specific to Power. Machine Type = S822L Machine Type = 8286-42A ---Steps to Reproduce--- Install openstack. Run the network node in a VRRP HA configuration. Create a router and assign a global IP. == Comment: #5 - David J. Wilder - 2016-11-17 15:58:04 == The problem is fixed in this upstream commit: https://github.com/acassen/keepalived/commit/9f327bbf3e86def1055a106eda0633638bda0345 On systems with a page size larger than 4096 keepalived may report: "Netlink: error: message truncated" messages This error was reported on a ppc64le in an OpenStack/Nutron environment. Ppc64le is using a 64k pages size. I found that keepalived's netlink recvmsg buffer was too small causing messages to be truncated. The size of the read buffer for the netlink socket should be based on page size however, it should not exceed 8192. See the comment in the patch. I tested the fix by creating 100 veth interfaces and verifying the errors did not return. Signed-off-by: David WilderSigned-off-by: Quentin Armitage ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1642763/+subscriptions ___ Mailing list: https://launchpad.net/~ubuntu-ha Post to : ubuntu-ha@lists.launchpad.net Unsubscribe : https://launchpad.net/~ubuntu-ha More help : https://help.launchpad.net/ListHelp
[Ubuntu-ha] [Bug 1642763] Re: keepalived raising VIP apply error
** Tags removed: verification-needed ** Tags added: verification-done-yakkety -- You received this bug notification because you are a member of Ubuntu High Availability Team, which is subscribed to keepalived in Ubuntu. https://bugs.launchpad.net/bugs/1642763 Title: keepalived raising VIP apply error Status in keepalived package in Ubuntu: Fix Released Status in keepalived source package in Xenial: Fix Released Status in keepalived source package in Yakkety: Fix Committed Bug description: [Impact] * keepalived on ppc64el (due to a large page size) experiences "Netlink: error: message truncated" messages. * These Netlink truncations result in keepalived thinking that the the underlying device does not exist, even though it does. [Test Case] * Creating 100 veth interfaces ppc64el and verify if "Netlink: error: message truncated" errors are emitted. If so, the bug is present. If not, the bug is fixed. [Regression Potential] * This is code issue, fixed upstream, in the keepalived code when the system page size exceeds 4096. The upstream fix was backported to all releases and should only properly limit the size of the buffer used for netlink to at most 8192 on systems with a page size greater than 8192. I believe risk of regression is very low. * Using the tests provided by David Wilder, I ran on both x86_64 and ppc64el LXD containers. Without the backported changes, I saw no issues on x86_64, and the reported issue on ppc64el (as expected, as a page size greater than 4K is required to see the buffer size issue). With the backported changes, both architectures show no issue with the provided testcase. --- == Comment: #0 - Andrew Thorstensen - 2016-11-17 09:50:25 == ---Problem Description--- Using Ubuntu 16.04 on ppc64le, we are building a 'neutron network node' using the VRRP configuration (built on keepalived). Information on this OpenStack configuration can be found here: https://wiki.openstack.org/wiki/Neutron/L3_High_Availability_VRRP When we run, the configuration is failing to apply via keepalived. The logs post the following: Nov 17 02:58:31 p8test-lp1 Keepalived_vrrp[54542]: VRRP is trying to assign VIP to unknown qr-a5f5ba96-52 interface !!! go out and fix your conf !!! However, the device DOES exist. But the keepalived config just doesn't always deploy it. ii keepalived 1:1.2.19-1 ppc64el Failover and monitoring daemon for LVS clusters This configuration sometimes works, but does sometimes fail on Ubuntu 16.04.1 ---uname output--- Linux p8test-lp1 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:38:24 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux ---Additional Hardware Info--- This is a Power8 system with Ubuntu 16.04.1 installed. Though we see no indication that this is specific to Power. Machine Type = S822L Machine Type = 8286-42A ---Steps to Reproduce--- Install openstack. Run the network node in a VRRP HA configuration. Create a router and assign a global IP. == Comment: #5 - David J. Wilder - 2016-11-17 15:58:04 == The problem is fixed in this upstream commit: https://github.com/acassen/keepalived/commit/9f327bbf3e86def1055a106eda0633638bda0345 On systems with a page size larger than 4096 keepalived may report: "Netlink: error: message truncated" messages This error was reported on a ppc64le in an OpenStack/Nutron environment. Ppc64le is using a 64k pages size. I found that keepalived's netlink recvmsg buffer was too small causing messages to be truncated. The size of the read buffer for the netlink socket should be based on page size however, it should not exceed 8192. See the comment in the patch. I tested the fix by creating 100 veth interfaces and verifying the errors did not return. Signed-off-by: David WilderSigned-off-by: Quentin Armitage ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1642763/+subscriptions ___ Mailing list: https://launchpad.net/~ubuntu-ha Post to : ubuntu-ha@lists.launchpad.net Unsubscribe : https://launchpad.net/~ubuntu-ha More help : https://help.launchpad.net/ListHelp
[Ubuntu-ha] [Bug 1642763] Re: keepalived raising VIP apply error
Hi Dave, Did you enable yakkety-proposed in your VM? Per rmadison: keepalived | 1:1.2.23-1ubuntu0.1 is available in yakkety-proposed for all architectures. Thanks, Nish -- You received this bug notification because you are a member of Ubuntu High Availability Team, which is subscribed to keepalived in Ubuntu. https://bugs.launchpad.net/bugs/1642763 Title: keepalived raising VIP apply error Status in keepalived package in Ubuntu: Fix Released Status in keepalived source package in Xenial: Fix Released Status in keepalived source package in Yakkety: Fix Committed Bug description: [Impact] * keepalived on ppc64el (due to a large page size) experiences "Netlink: error: message truncated" messages. * These Netlink truncations result in keepalived thinking that the the underlying device does not exist, even though it does. [Test Case] * Creating 100 veth interfaces ppc64el and verify if "Netlink: error: message truncated" errors are emitted. If so, the bug is present. If not, the bug is fixed. [Regression Potential] * This is code issue, fixed upstream, in the keepalived code when the system page size exceeds 4096. The upstream fix was backported to all releases and should only properly limit the size of the buffer used for netlink to at most 8192 on systems with a page size greater than 8192. I believe risk of regression is very low. * Using the tests provided by David Wilder, I ran on both x86_64 and ppc64el LXD containers. Without the backported changes, I saw no issues on x86_64, and the reported issue on ppc64el (as expected, as a page size greater than 4K is required to see the buffer size issue). With the backported changes, both architectures show no issue with the provided testcase. --- == Comment: #0 - Andrew Thorstensen - 2016-11-17 09:50:25 == ---Problem Description--- Using Ubuntu 16.04 on ppc64le, we are building a 'neutron network node' using the VRRP configuration (built on keepalived). Information on this OpenStack configuration can be found here: https://wiki.openstack.org/wiki/Neutron/L3_High_Availability_VRRP When we run, the configuration is failing to apply via keepalived. The logs post the following: Nov 17 02:58:31 p8test-lp1 Keepalived_vrrp[54542]: VRRP is trying to assign VIP to unknown qr-a5f5ba96-52 interface !!! go out and fix your conf !!! However, the device DOES exist. But the keepalived config just doesn't always deploy it. ii keepalived 1:1.2.19-1 ppc64el Failover and monitoring daemon for LVS clusters This configuration sometimes works, but does sometimes fail on Ubuntu 16.04.1 ---uname output--- Linux p8test-lp1 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:38:24 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux ---Additional Hardware Info--- This is a Power8 system with Ubuntu 16.04.1 installed. Though we see no indication that this is specific to Power. Machine Type = S822L Machine Type = 8286-42A ---Steps to Reproduce--- Install openstack. Run the network node in a VRRP HA configuration. Create a router and assign a global IP. == Comment: #5 - David J. Wilder - 2016-11-17 15:58:04 == The problem is fixed in this upstream commit: https://github.com/acassen/keepalived/commit/9f327bbf3e86def1055a106eda0633638bda0345 On systems with a page size larger than 4096 keepalived may report: "Netlink: error: message truncated" messages This error was reported on a ppc64le in an OpenStack/Nutron environment. Ppc64le is using a 64k pages size. I found that keepalived's netlink recvmsg buffer was too small causing messages to be truncated. The size of the read buffer for the netlink socket should be based on page size however, it should not exceed 8192. See the comment in the patch. I tested the fix by creating 100 veth interfaces and verifying the errors did not return. Signed-off-by: David WilderSigned-off-by: Quentin Armitage ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1642763/+subscriptions ___ Mailing list: https://launchpad.net/~ubuntu-ha Post to : ubuntu-ha@lists.launchpad.net Unsubscribe : https://launchpad.net/~ubuntu-ha More help : https://help.launchpad.net/ListHelp
[Ubuntu-ha] [Bug 1642763] Re: keepalived raising VIP apply error
Dave, would you be able to also test in 16.10? I'll do my best to setup an environment on my end, but it would be good to have you verify it as well. -- You received this bug notification because you are a member of Ubuntu High Availability Team, which is subscribed to keepalived in Ubuntu. https://bugs.launchpad.net/bugs/1642763 Title: keepalived raising VIP apply error Status in keepalived package in Ubuntu: Fix Released Status in keepalived source package in Xenial: Fix Released Status in keepalived source package in Yakkety: Fix Committed Bug description: [Impact] * keepalived on ppc64el (due to a large page size) experiences "Netlink: error: message truncated" messages. * These Netlink truncations result in keepalived thinking that the the underlying device does not exist, even though it does. [Test Case] * Creating 100 veth interfaces ppc64el and verify if "Netlink: error: message truncated" errors are emitted. If so, the bug is present. If not, the bug is fixed. [Regression Potential] * This is code issue, fixed upstream, in the keepalived code when the system page size exceeds 4096. The upstream fix was backported to all releases and should only properly limit the size of the buffer used for netlink to at most 8192 on systems with a page size greater than 8192. I believe risk of regression is very low. * Using the tests provided by David Wilder, I ran on both x86_64 and ppc64el LXD containers. Without the backported changes, I saw no issues on x86_64, and the reported issue on ppc64el (as expected, as a page size greater than 4K is required to see the buffer size issue). With the backported changes, both architectures show no issue with the provided testcase. --- == Comment: #0 - Andrew Thorstensen - 2016-11-17 09:50:25 == ---Problem Description--- Using Ubuntu 16.04 on ppc64le, we are building a 'neutron network node' using the VRRP configuration (built on keepalived). Information on this OpenStack configuration can be found here: https://wiki.openstack.org/wiki/Neutron/L3_High_Availability_VRRP When we run, the configuration is failing to apply via keepalived. The logs post the following: Nov 17 02:58:31 p8test-lp1 Keepalived_vrrp[54542]: VRRP is trying to assign VIP to unknown qr-a5f5ba96-52 interface !!! go out and fix your conf !!! However, the device DOES exist. But the keepalived config just doesn't always deploy it. ii keepalived 1:1.2.19-1 ppc64el Failover and monitoring daemon for LVS clusters This configuration sometimes works, but does sometimes fail on Ubuntu 16.04.1 ---uname output--- Linux p8test-lp1 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:38:24 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux ---Additional Hardware Info--- This is a Power8 system with Ubuntu 16.04.1 installed. Though we see no indication that this is specific to Power. Machine Type = S822L Machine Type = 8286-42A ---Steps to Reproduce--- Install openstack. Run the network node in a VRRP HA configuration. Create a router and assign a global IP. == Comment: #5 - David J. Wilder - 2016-11-17 15:58:04 == The problem is fixed in this upstream commit: https://github.com/acassen/keepalived/commit/9f327bbf3e86def1055a106eda0633638bda0345 On systems with a page size larger than 4096 keepalived may report: "Netlink: error: message truncated" messages This error was reported on a ppc64le in an OpenStack/Nutron environment. Ppc64le is using a 64k pages size. I found that keepalived's netlink recvmsg buffer was too small causing messages to be truncated. The size of the read buffer for the netlink socket should be based on page size however, it should not exceed 8192. See the comment in the patch. I tested the fix by creating 100 veth interfaces and verifying the errors did not return. Signed-off-by: David WilderSigned-off-by: Quentin Armitage ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1642763/+subscriptions ___ Mailing list: https://launchpad.net/~ubuntu-ha Post to : ubuntu-ha@lists.launchpad.net Unsubscribe : https://launchpad.net/~ubuntu-ha More help : https://help.launchpad.net/ListHelp
[Ubuntu-ha] [Bug 1642763] Re: keepalived raising VIP apply error
This bug was fixed in the package keepalived - 1:1.2.19-1ubuntu0.1 --- keepalived (1:1.2.19-1ubuntu0.1) xenial; urgency=medium * debian/patches/fix_message_truncation_with_large_pagesizes.patch: Resolve "Netlink: error: message truncated" messages. Thanks to David Wilder. Closes LP: #1642763. -- Nishanth Aravamudan Tue, 29 Nov 2016 10:31:22 -0800 ** Changed in: keepalived (Ubuntu Xenial) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu High Availability Team, which is subscribed to keepalived in Ubuntu. https://bugs.launchpad.net/bugs/1642763 Title: keepalived raising VIP apply error Status in keepalived package in Ubuntu: Fix Released Status in keepalived source package in Xenial: Fix Released Status in keepalived source package in Yakkety: Fix Committed Bug description: [Impact] * keepalived on ppc64el (due to a large page size) experiences "Netlink: error: message truncated" messages. * These Netlink truncations result in keepalived thinking that the the underlying device does not exist, even though it does. [Test Case] * Creating 100 veth interfaces ppc64el and verify if "Netlink: error: message truncated" errors are emitted. If so, the bug is present. If not, the bug is fixed. [Regression Potential] * This is code issue, fixed upstream, in the keepalived code when the system page size exceeds 4096. The upstream fix was backported to all releases and should only properly limit the size of the buffer used for netlink to at most 8192 on systems with a page size greater than 8192. I believe risk of regression is very low. * Using the tests provided by David Wilder, I ran on both x86_64 and ppc64el LXD containers. Without the backported changes, I saw no issues on x86_64, and the reported issue on ppc64el (as expected, as a page size greater than 4K is required to see the buffer size issue). With the backported changes, both architectures show no issue with the provided testcase. --- == Comment: #0 - Andrew Thorstensen - 2016-11-17 09:50:25 == ---Problem Description--- Using Ubuntu 16.04 on ppc64le, we are building a 'neutron network node' using the VRRP configuration (built on keepalived). Information on this OpenStack configuration can be found here: https://wiki.openstack.org/wiki/Neutron/L3_High_Availability_VRRP When we run, the configuration is failing to apply via keepalived. The logs post the following: Nov 17 02:58:31 p8test-lp1 Keepalived_vrrp[54542]: VRRP is trying to assign VIP to unknown qr-a5f5ba96-52 interface !!! go out and fix your conf !!! However, the device DOES exist. But the keepalived config just doesn't always deploy it. ii keepalived 1:1.2.19-1 ppc64el Failover and monitoring daemon for LVS clusters This configuration sometimes works, but does sometimes fail on Ubuntu 16.04.1 ---uname output--- Linux p8test-lp1 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:38:24 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux ---Additional Hardware Info--- This is a Power8 system with Ubuntu 16.04.1 installed. Though we see no indication that this is specific to Power. Machine Type = S822L Machine Type = 8286-42A ---Steps to Reproduce--- Install openstack. Run the network node in a VRRP HA configuration. Create a router and assign a global IP. == Comment: #5 - David J. Wilder - 2016-11-17 15:58:04 == The problem is fixed in this upstream commit: https://github.com/acassen/keepalived/commit/9f327bbf3e86def1055a106eda0633638bda0345 On systems with a page size larger than 4096 keepalived may report: "Netlink: error: message truncated" messages This error was reported on a ppc64le in an OpenStack/Nutron environment. Ppc64le is using a 64k pages size. I found that keepalived's netlink recvmsg buffer was too small causing messages to be truncated. The size of the read buffer for the netlink socket should be based on page size however, it should not exceed 8192. See the comment in the patch. I tested the fix by creating 100 veth interfaces and verifying the errors did not return. Signed-off-by: David Wilder Signed-off-by: Quentin Armitage ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1642763/+subscriptions ___ Mailing list: https://launchpad.net/~ubuntu-ha Post to : ubuntu-ha@lists.launchpad.net Unsubscribe : https://launchpad.net/~ubuntu-ha More help : https://help.launchpad.net/ListHelp
[Ubuntu-ha] [Bug 1642763] Re: keepalived raising VIP apply error
** Tags removed: targetmilestone-inin--- verification-needed ** Tags added: targetmilestone-inin16041 verification-done -- You received this bug notification because you are a member of Ubuntu High Availability Team, which is subscribed to keepalived in Ubuntu. https://bugs.launchpad.net/bugs/1642763 Title: keepalived raising VIP apply error Status in keepalived package in Ubuntu: Fix Released Status in keepalived source package in Xenial: Fix Committed Status in keepalived source package in Yakkety: Fix Committed Bug description: [Impact] * keepalived on ppc64el (due to a large page size) experiences "Netlink: error: message truncated" messages. * These Netlink truncations result in keepalived thinking that the the underlying device does not exist, even though it does. [Test Case] * Creating 100 veth interfaces ppc64el and verify if "Netlink: error: message truncated" errors are emitted. If so, the bug is present. If not, the bug is fixed. [Regression Potential] * This is code issue, fixed upstream, in the keepalived code when the system page size exceeds 4096. The upstream fix was backported to all releases and should only properly limit the size of the buffer used for netlink to at most 8192 on systems with a page size greater than 8192. I believe risk of regression is very low. * Using the tests provided by David Wilder, I ran on both x86_64 and ppc64el LXD containers. Without the backported changes, I saw no issues on x86_64, and the reported issue on ppc64el (as expected, as a page size greater than 4K is required to see the buffer size issue). With the backported changes, both architectures show no issue with the provided testcase. --- == Comment: #0 - Andrew Thorstensen - 2016-11-17 09:50:25 == ---Problem Description--- Using Ubuntu 16.04 on ppc64le, we are building a 'neutron network node' using the VRRP configuration (built on keepalived). Information on this OpenStack configuration can be found here: https://wiki.openstack.org/wiki/Neutron/L3_High_Availability_VRRP When we run, the configuration is failing to apply via keepalived. The logs post the following: Nov 17 02:58:31 p8test-lp1 Keepalived_vrrp[54542]: VRRP is trying to assign VIP to unknown qr-a5f5ba96-52 interface !!! go out and fix your conf !!! However, the device DOES exist. But the keepalived config just doesn't always deploy it. ii keepalived 1:1.2.19-1 ppc64el Failover and monitoring daemon for LVS clusters This configuration sometimes works, but does sometimes fail on Ubuntu 16.04.1 ---uname output--- Linux p8test-lp1 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:38:24 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux ---Additional Hardware Info--- This is a Power8 system with Ubuntu 16.04.1 installed. Though we see no indication that this is specific to Power. Machine Type = S822L Machine Type = 8286-42A ---Steps to Reproduce--- Install openstack. Run the network node in a VRRP HA configuration. Create a router and assign a global IP. == Comment: #5 - David J. Wilder - 2016-11-17 15:58:04 == The problem is fixed in this upstream commit: https://github.com/acassen/keepalived/commit/9f327bbf3e86def1055a106eda0633638bda0345 On systems with a page size larger than 4096 keepalived may report: "Netlink: error: message truncated" messages This error was reported on a ppc64le in an OpenStack/Nutron environment. Ppc64le is using a 64k pages size. I found that keepalived's netlink recvmsg buffer was too small causing messages to be truncated. The size of the read buffer for the netlink socket should be based on page size however, it should not exceed 8192. See the comment in the patch. I tested the fix by creating 100 veth interfaces and verifying the errors did not return. Signed-off-by: David WilderSigned-off-by: Quentin Armitage ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1642763/+subscriptions ___ Mailing list: https://launchpad.net/~ubuntu-ha Post to : ubuntu-ha@lists.launchpad.net Unsubscribe : https://launchpad.net/~ubuntu-ha More help : https://help.launchpad.net/ListHelp
[Ubuntu-ha] [Bug 1642763] Re: keepalived raising VIP apply error
Thank you for this fix. I setup an OpenStack cluster using 2 controller nodes on Power systems and Xenial. The fix worked perfectly and resolved the issue I was seeing. Version verified: 1:1.2.19-1ubuntu0.1 root@kyle-pwr-1:~# apt list --installed | grep keepalived WARNING: apt does not have a stable CLI interface. Use with caution in scripts. keepalived/xenial-proposed,now 1:1.2.19-1ubuntu0.1 ppc64el [installed] -- You received this bug notification because you are a member of Ubuntu High Availability Team, which is subscribed to keepalived in Ubuntu. https://bugs.launchpad.net/bugs/1642763 Title: keepalived raising VIP apply error Status in keepalived package in Ubuntu: Fix Released Status in keepalived source package in Xenial: Fix Committed Status in keepalived source package in Yakkety: Fix Committed Bug description: [Impact] * keepalived on ppc64el (due to a large page size) experiences "Netlink: error: message truncated" messages. * These Netlink truncations result in keepalived thinking that the the underlying device does not exist, even though it does. [Test Case] * Creating 100 veth interfaces ppc64el and verify if "Netlink: error: message truncated" errors are emitted. If so, the bug is present. If not, the bug is fixed. [Regression Potential] * This is code issue, fixed upstream, in the keepalived code when the system page size exceeds 4096. The upstream fix was backported to all releases and should only properly limit the size of the buffer used for netlink to at most 8192 on systems with a page size greater than 8192. I believe risk of regression is very low. * Using the tests provided by David Wilder, I ran on both x86_64 and ppc64el LXD containers. Without the backported changes, I saw no issues on x86_64, and the reported issue on ppc64el (as expected, as a page size greater than 4K is required to see the buffer size issue). With the backported changes, both architectures show no issue with the provided testcase. --- == Comment: #0 - Andrew Thorstensen - 2016-11-17 09:50:25 == ---Problem Description--- Using Ubuntu 16.04 on ppc64le, we are building a 'neutron network node' using the VRRP configuration (built on keepalived). Information on this OpenStack configuration can be found here: https://wiki.openstack.org/wiki/Neutron/L3_High_Availability_VRRP When we run, the configuration is failing to apply via keepalived. The logs post the following: Nov 17 02:58:31 p8test-lp1 Keepalived_vrrp[54542]: VRRP is trying to assign VIP to unknown qr-a5f5ba96-52 interface !!! go out and fix your conf !!! However, the device DOES exist. But the keepalived config just doesn't always deploy it. ii keepalived 1:1.2.19-1 ppc64el Failover and monitoring daemon for LVS clusters This configuration sometimes works, but does sometimes fail on Ubuntu 16.04.1 ---uname output--- Linux p8test-lp1 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:38:24 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux ---Additional Hardware Info--- This is a Power8 system with Ubuntu 16.04.1 installed. Though we see no indication that this is specific to Power. Machine Type = S822L Machine Type = 8286-42A ---Steps to Reproduce--- Install openstack. Run the network node in a VRRP HA configuration. Create a router and assign a global IP. == Comment: #5 - David J. Wilder - 2016-11-17 15:58:04 == The problem is fixed in this upstream commit: https://github.com/acassen/keepalived/commit/9f327bbf3e86def1055a106eda0633638bda0345 On systems with a page size larger than 4096 keepalived may report: "Netlink: error: message truncated" messages This error was reported on a ppc64le in an OpenStack/Nutron environment. Ppc64le is using a 64k pages size. I found that keepalived's netlink recvmsg buffer was too small causing messages to be truncated. The size of the read buffer for the netlink socket should be based on page size however, it should not exceed 8192. See the comment in the patch. I tested the fix by creating 100 veth interfaces and verifying the errors did not return. Signed-off-by: David WilderSigned-off-by: Quentin Armitage ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1642763/+subscriptions ___ Mailing list: https://launchpad.net/~ubuntu-ha Post to : ubuntu-ha@lists.launchpad.net Unsubscribe : https://launchpad.net/~ubuntu-ha More help : https://help.launchpad.net/ListHelp
[Ubuntu-ha] [Bug 1642763] Re: keepalived raising VIP apply error
Hello bugproxy, or anyone else affected, Accepted keepalived into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/keepalived/1:1.2.23-1ubuntu0.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Changed in: keepalived (Ubuntu Yakkety) Status: In Progress => Fix Committed ** Tags added: verification-needed ** Changed in: keepalived (Ubuntu Xenial) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu High Availability Team, which is subscribed to keepalived in Ubuntu. https://bugs.launchpad.net/bugs/1642763 Title: keepalived raising VIP apply error Status in keepalived package in Ubuntu: Fix Released Status in keepalived source package in Xenial: Fix Committed Status in keepalived source package in Yakkety: Fix Committed Bug description: [Impact] * keepalived on ppc64el (due to a large page size) experiences "Netlink: error: message truncated" messages. * These Netlink truncations result in keepalived thinking that the the underlying device does not exist, even though it does. [Test Case] * Creating 100 veth interfaces ppc64el and verify if "Netlink: error: message truncated" errors are emitted. If so, the bug is present. If not, the bug is fixed. [Regression Potential] * This is code issue, fixed upstream, in the keepalived code when the system page size exceeds 4096. The upstream fix was backported to all releases and should only properly limit the size of the buffer used for netlink to at most 8192 on systems with a page size greater than 8192. I believe risk of regression is very low. * Using the tests provided by David Wilder, I ran on both x86_64 and ppc64el LXD containers. Without the backported changes, I saw no issues on x86_64, and the reported issue on ppc64el (as expected, as a page size greater than 4K is required to see the buffer size issue). With the backported changes, both architectures show no issue with the provided testcase. --- == Comment: #0 - Andrew Thorstensen - 2016-11-17 09:50:25 == ---Problem Description--- Using Ubuntu 16.04 on ppc64le, we are building a 'neutron network node' using the VRRP configuration (built on keepalived). Information on this OpenStack configuration can be found here: https://wiki.openstack.org/wiki/Neutron/L3_High_Availability_VRRP When we run, the configuration is failing to apply via keepalived. The logs post the following: Nov 17 02:58:31 p8test-lp1 Keepalived_vrrp[54542]: VRRP is trying to assign VIP to unknown qr-a5f5ba96-52 interface !!! go out and fix your conf !!! However, the device DOES exist. But the keepalived config just doesn't always deploy it. ii keepalived 1:1.2.19-1 ppc64el Failover and monitoring daemon for LVS clusters This configuration sometimes works, but does sometimes fail on Ubuntu 16.04.1 ---uname output--- Linux p8test-lp1 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:38:24 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux ---Additional Hardware Info--- This is a Power8 system with Ubuntu 16.04.1 installed. Though we see no indication that this is specific to Power. Machine Type = S822L Machine Type = 8286-42A ---Steps to Reproduce--- Install openstack. Run the network node in a VRRP HA configuration. Create a router and assign a global IP. == Comment: #5 - David J. Wilder - 2016-11-17 15:58:04 == The problem is fixed in this upstream commit: https://github.com/acassen/keepalived/commit/9f327bbf3e86def1055a106eda0633638bda0345 On systems with a page size larger than 4096 keepalived may report: "Netlink: error: message truncated" messages This error was reported on a ppc64le in an OpenStack/Nutron environment. Ppc64le is using a 64k pages size. I found that keepalived's netlink recvmsg buffer was too small causing messages to be truncated. The size of the read buffer for the netlink socket should be based on page size however, it should not exceed 8192. See the comment in the patch. I tested the fix by creating 100 veth interfaces and verifying the errors did not
[Ubuntu-ha] [Bug 1642763] Re: keepalived raising VIP apply error
This bug was fixed in the package keepalived - 1:1.2.24-1ubuntu1 --- keepalived (1:1.2.24-1ubuntu1) zesty; urgency=medium * debian/patches/fix_message_truncation_with_large_pagesizes.patch: Resolve "Netlink: error: message truncated" messages. Thanks to David Wilder. Closes LP: #1642763. -- Nishanth Aravamudan Tue, 29 Nov 2016 09:45:12 -0800 ** Changed in: keepalived (Ubuntu) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu High Availability Team, which is subscribed to keepalived in Ubuntu. https://bugs.launchpad.net/bugs/1642763 Title: keepalived raising VIP apply error Status in keepalived package in Ubuntu: Fix Released Status in keepalived source package in Xenial: In Progress Status in keepalived source package in Yakkety: In Progress Bug description: [Impact] * keepalived on ppc64el (due to a large page size) experiences "Netlink: error: message truncated" messages. * These Netlink truncations result in keepalived thinking that the the underlying device does not exist, even though it does. [Test Case] * Creating 100 veth interfaces ppc64el and verify if "Netlink: error: message truncated" errors are emitted. If so, the bug is present. If not, the bug is fixed. [Regression Potential] * This is code issue, fixed upstream, in the keepalived code when the system page size exceeds 4096. The upstream fix was backported to all releases and should only properly limit the size of the buffer used for netlink to at most 8192 on systems with a page size greater than 8192. I believe risk of regression is very low. * Using the tests provided by David Wilder, I ran on both x86_64 and ppc64el LXD containers. Without the backported changes, I saw no issues on x86_64, and the reported issue on ppc64el (as expected, as a page size greater than 4K is required to see the buffer size issue). With the backported changes, both architectures show no issue with the provided testcase. --- == Comment: #0 - Andrew Thorstensen - 2016-11-17 09:50:25 == ---Problem Description--- Using Ubuntu 16.04 on ppc64le, we are building a 'neutron network node' using the VRRP configuration (built on keepalived). Information on this OpenStack configuration can be found here: https://wiki.openstack.org/wiki/Neutron/L3_High_Availability_VRRP When we run, the configuration is failing to apply via keepalived. The logs post the following: Nov 17 02:58:31 p8test-lp1 Keepalived_vrrp[54542]: VRRP is trying to assign VIP to unknown qr-a5f5ba96-52 interface !!! go out and fix your conf !!! However, the device DOES exist. But the keepalived config just doesn't always deploy it. ii keepalived 1:1.2.19-1 ppc64el Failover and monitoring daemon for LVS clusters This configuration sometimes works, but does sometimes fail on Ubuntu 16.04.1 ---uname output--- Linux p8test-lp1 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:38:24 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux ---Additional Hardware Info--- This is a Power8 system with Ubuntu 16.04.1 installed. Though we see no indication that this is specific to Power. Machine Type = S822L Machine Type = 8286-42A ---Steps to Reproduce--- Install openstack. Run the network node in a VRRP HA configuration. Create a router and assign a global IP. == Comment: #5 - David J. Wilder - 2016-11-17 15:58:04 == The problem is fixed in this upstream commit: https://github.com/acassen/keepalived/commit/9f327bbf3e86def1055a106eda0633638bda0345 On systems with a page size larger than 4096 keepalived may report: "Netlink: error: message truncated" messages This error was reported on a ppc64le in an OpenStack/Nutron environment. Ppc64le is using a 64k pages size. I found that keepalived's netlink recvmsg buffer was too small causing messages to be truncated. The size of the read buffer for the netlink socket should be based on page size however, it should not exceed 8192. See the comment in the patch. I tested the fix by creating 100 veth interfaces and verifying the errors did not return. Signed-off-by: David Wilder Signed-off-by: Quentin Armitage ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1642763/+subscriptions ___ Mailing list: https://launchpad.net/~ubuntu-ha Post to : ubuntu-ha@lists.launchpad.net Unsubscribe : https://launchpad.net/~ubuntu-ha More help : https://help.launchpad.net/ListHelp
[Ubuntu-ha] [Bug 1642763] Re: keepalived raising VIP apply error
On 29.11.2016 [21:30:21 -], bugproxy wrote: > --- Comment From wil...@us.ibm.com 2016-11-29 16:21 EDT--- > (In reply to comment #16) > > Thanks David. Asking Nish to take a look at this for you. > > Thanks for your attention to this issue. This has become an urgent > issue for our customer. If you can please provide an ETA when a fix > will be available. It will first need to get through the zesty queue (should only take a few hours) and then the SRU team will need to consider it: https://wiki.ubuntu.com/StableReleaseUpdates. Once they provide it in the appropriate -proposed pockets, it can take a week to make it to -updates, after verification. Thanks, Nish ** Description changed: [Impact] - * keepalived on ppc64el (due to a large page size) experiences + * keepalived on ppc64el (due to a large page size) experiences "Netlink: error: message truncated" messages. - * These Netlink truncations result in keepalived thinking that the the + * These Netlink truncations result in keepalived thinking that the the underlying device does not exist, even though it does. [Test Case] - * Creating 100 veth interfaces ppc64el and verify if "Netlink: error: + * Creating 100 veth interfaces ppc64el and verify if "Netlink: error: message truncated" errors are emitted. If so, the bug is present. If not, the bug is fixed. [Regression Potential] - * This is code issue, fixed upstream, in the keepalived code when the + * This is code issue, fixed upstream, in the keepalived code when the system page size exceeds 4096. The upstream fix was backported to all releases and should only properly limit the size of the buffer used for netlink to at most 8192 on systems with a page size greater than 8192. I believe risk of regression is very low. + + * Using the tests provided by David Wilder, I ran on both x86_64 and + ppc64el LXD containers. Without the backported changes, I saw no issues + on x86_64, and the reported issue on ppc64el (as expected, as a page + size greater than 4K is required to see the buffer size issue). With the + backported changes, both architectures show no issue with the provided + testcase. --- == Comment: #0 - Andrew Thorstensen - 2016-11-17 09:50:25 == ---Problem Description--- Using Ubuntu 16.04 on ppc64le, we are building a 'neutron network node' using the VRRP configuration (built on keepalived). Information on this OpenStack configuration can be found here: https://wiki.openstack.org/wiki/Neutron/L3_High_Availability_VRRP When we run, the configuration is failing to apply via keepalived. The logs post the following: Nov 17 02:58:31 p8test-lp1 Keepalived_vrrp[54542]: VRRP is trying to assign VIP to unknown qr-a5f5ba96-52 interface !!! go out and fix your conf !!! However, the device DOES exist. But the keepalived config just doesn't always deploy it. ii keepalived 1:1.2.19-1 ppc64el Failover and monitoring daemon for LVS clusters This configuration sometimes works, but does sometimes fail on Ubuntu 16.04.1 ---uname output--- Linux p8test-lp1 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:38:24 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux ---Additional Hardware Info--- This is a Power8 system with Ubuntu 16.04.1 installed. Though we see no indication that this is specific to Power. Machine Type = S822L Machine Type = 8286-42A ---Steps to Reproduce--- Install openstack. Run the network node in a VRRP HA configuration. Create a router and assign a global IP. == Comment: #5 - David J. Wilder - 2016-11-17 15:58:04 == The problem is fixed in this upstream commit: https://github.com/acassen/keepalived/commit/9f327bbf3e86def1055a106eda0633638bda0345 On systems with a page size larger than 4096 keepalived may report: "Netlink: error: message truncated" messages This error was reported on a ppc64le in an OpenStack/Nutron environment. Ppc64le is using a 64k pages size. I found that keepalived's netlink recvmsg buffer was too small causing messages to be truncated. The size of the read buffer for the netlink socket should be based on page size however, it should not exceed 8192. See the comment in the patch. I tested the fix by creating 100 veth interfaces and verifying the errors did not return. Signed-off-by: David WilderSigned-off-by: Quentin Armitage ... -- You received this bug notification because you are a member of Ubuntu High Availability Team, which is subscribed to keepalived in Ubuntu. https://bugs.launchpad.net/bugs/1642763 Title: keepalived raising VIP apply error Status in keepalived package in Ubuntu: Fix Committed Status in keepalived source package in Xenial: In Progress Status in keepalived source package in Yakkety: In Progress Bug description: [Impact] * keepalived on ppc64el (due to a
[Ubuntu-ha] [Bug 1642763] Re: keepalived raising VIP apply error
Test packages submitted at: https://launchpad.net/~nacc/+archive/ubuntu/lp1642763. -- You received this bug notification because you are a member of Ubuntu High Availability Team, which is subscribed to keepalived in Ubuntu. https://bugs.launchpad.net/bugs/1642763 Title: keepalived raising VIP apply error Status in keepalived package in Ubuntu: Fix Committed Status in keepalived source package in Xenial: In Progress Status in keepalived source package in Yakkety: In Progress Bug description: [Impact] * keepalived on ppc64el (due to a large page size) experiences "Netlink: error: message truncated" messages. * These Netlink truncations result in keepalived thinking that the the underlying device does not exist, even though it does. [Test Case] * Creating 100 veth interfaces ppc64el and verify if "Netlink: error: message truncated" errors are emitted. If so, the bug is present. If not, the bug is fixed. [Regression Potential] * This is code issue, fixed upstream, in the keepalived code when the system page size exceeds 4096. The upstream fix was backported to all releases and should only properly limit the size of the buffer used for netlink to at most 8192 on systems with a page size greater than 8192. I believe risk of regression is very low. --- == Comment: #0 - Andrew Thorstensen - 2016-11-17 09:50:25 == ---Problem Description--- Using Ubuntu 16.04 on ppc64le, we are building a 'neutron network node' using the VRRP configuration (built on keepalived). Information on this OpenStack configuration can be found here: https://wiki.openstack.org/wiki/Neutron/L3_High_Availability_VRRP When we run, the configuration is failing to apply via keepalived. The logs post the following: Nov 17 02:58:31 p8test-lp1 Keepalived_vrrp[54542]: VRRP is trying to assign VIP to unknown qr-a5f5ba96-52 interface !!! go out and fix your conf !!! However, the device DOES exist. But the keepalived config just doesn't always deploy it. ii keepalived 1:1.2.19-1 ppc64el Failover and monitoring daemon for LVS clusters This configuration sometimes works, but does sometimes fail on Ubuntu 16.04.1 ---uname output--- Linux p8test-lp1 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:38:24 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux ---Additional Hardware Info--- This is a Power8 system with Ubuntu 16.04.1 installed. Though we see no indication that this is specific to Power. Machine Type = S822L Machine Type = 8286-42A ---Steps to Reproduce--- Install openstack. Run the network node in a VRRP HA configuration. Create a router and assign a global IP. == Comment: #5 - David J. Wilder - 2016-11-17 15:58:04 == The problem is fixed in this upstream commit: https://github.com/acassen/keepalived/commit/9f327bbf3e86def1055a106eda0633638bda0345 On systems with a page size larger than 4096 keepalived may report: "Netlink: error: message truncated" messages This error was reported on a ppc64le in an OpenStack/Nutron environment. Ppc64le is using a 64k pages size. I found that keepalived's netlink recvmsg buffer was too small causing messages to be truncated. The size of the read buffer for the netlink socket should be based on page size however, it should not exceed 8192. See the comment in the patch. I tested the fix by creating 100 veth interfaces and verifying the errors did not return. Signed-off-by: David WilderSigned-off-by: Quentin Armitage ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1642763/+subscriptions ___ Mailing list: https://launchpad.net/~ubuntu-ha Post to : ubuntu-ha@lists.launchpad.net Unsubscribe : https://launchpad.net/~ubuntu-ha More help : https://help.launchpad.net/ListHelp
[Ubuntu-ha] [Bug 1642763] Re: keepalived raising VIP apply error
** Description changed: + [Impact] + + * keepalived on ppc64el (due to a large page size) experiences + "Netlink: error: message truncated" messages. + + * These Netlink truncations result in keepalived thinking that the the + underlying device does not exist, even though it does. + + [Test Case] + + * Creating 100 veth interfaces ppc64el and verify if "Netlink: error: + message truncated" errors are emitted. If so, the bug is present. If + not, the bug is fixed. + + [Regression Potential] + + * This is code issue, fixed upstream, in the keepalived code when the + system page size exceeds 4096. The upstream fix was backported to all + releases and should only properly limit the size of the buffer used for + netlink to at most 8192 on systems with a page size greater than 8192. I + believe risk of regression is very low. + + --- + == Comment: #0 - Andrew Thorstensen - 2016-11-17 09:50:25 == ---Problem Description--- Using Ubuntu 16.04 on ppc64le, we are building a 'neutron network node' using the VRRP configuration (built on keepalived). Information on this OpenStack configuration can be found here: https://wiki.openstack.org/wiki/Neutron/L3_High_Availability_VRRP When we run, the configuration is failing to apply via keepalived. The logs post the following: Nov 17 02:58:31 p8test-lp1 Keepalived_vrrp[54542]: VRRP is trying to assign VIP to unknown qr-a5f5ba96-52 interface !!! go out and fix your conf !!! - - However, the device DOES exist. But the keepalived config just doesn't always deploy it. + However, the device DOES exist. But the keepalived config just doesn't + always deploy it. ii keepalived 1:1.2.19-1 ppc64el Failover and monitoring daemon for LVS clusters - This configuration sometimes works, but does sometimes fail on Ubuntu 16.04.1 - + This configuration sometimes works, but does sometimes fail on Ubuntu + 16.04.1 + ---uname output--- Linux p8test-lp1 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:38:24 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux - + ---Additional Hardware Info--- This is a Power8 system with Ubuntu 16.04.1 installed. Though we see no indication that this is specific to Power. - Machine Type = S822L - - Machine Type = 8286-42A - + Machine Type = S822L + + Machine Type = 8286-42A + ---Steps to Reproduce--- - Install openstack. Run the network node in a VRRP HA configuration. Create a router and assign a global IP. + Install openstack. Run the network node in a VRRP HA configuration. Create a router and assign a global IP. == Comment: #5 - David J. Wilder - 2016-11-17 15:58:04 == The problem is fixed in this upstream commit: https://github.com/acassen/keepalived/commit/9f327bbf3e86def1055a106eda0633638bda0345 On systems with a page size larger than 4096 keepalived may report: "Netlink: error: message truncated" messages This error was reported on a ppc64le in an OpenStack/Nutron environment. Ppc64le is using a 64k pages size. I found that keepalived's netlink recvmsg buffer was too small causing messages to be truncated. The size of the read buffer for the netlink socket should be based on page size however, it should not exceed 8192. See the comment in the patch. I tested the fix by creating 100 veth interfaces and verifying the errors did not return. Signed-off-by: David WilderSigned-off-by: Quentin Armitage ... -- You received this bug notification because you are a member of Ubuntu High Availability Team, which is subscribed to keepalived in Ubuntu. https://bugs.launchpad.net/bugs/1642763 Title: keepalived raising VIP apply error Status in keepalived package in Ubuntu: Fix Committed Status in keepalived source package in Xenial: In Progress Status in keepalived source package in Yakkety: In Progress Bug description: [Impact] * keepalived on ppc64el (due to a large page size) experiences "Netlink: error: message truncated" messages. * These Netlink truncations result in keepalived thinking that the the underlying device does not exist, even though it does. [Test Case] * Creating 100 veth interfaces ppc64el and verify if "Netlink: error: message truncated" errors are emitted. If so, the bug is present. If not, the bug is fixed. [Regression Potential] * This is code issue, fixed upstream, in the keepalived code when the system page size exceeds 4096. The upstream fix was backported to all releases and should only properly limit the size of the buffer used for netlink to at most 8192 on systems with a page size greater than 8192. I believe risk of regression is very low. --- == Comment: #0 - Andrew Thorstensen - 2016-11-17 09:50:25 == ---Problem Description--- Using Ubuntu 16.04 on ppc64le, we are building a 'neutron network node' using the VRRP configuration (built on keepalived).
[Ubuntu-ha] [Bug 1642763] Re: keepalived raising VIP apply error
** Changed in: keepalived (Ubuntu) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu High Availability Team, which is subscribed to keepalived in Ubuntu. https://bugs.launchpad.net/bugs/1642763 Title: keepalived raising VIP apply error Status in keepalived package in Ubuntu: Fix Committed Status in keepalived source package in Xenial: In Progress Status in keepalived source package in Yakkety: In Progress Bug description: == Comment: #0 - Andrew Thorstensen - 2016-11-17 09:50:25 == ---Problem Description--- Using Ubuntu 16.04 on ppc64le, we are building a 'neutron network node' using the VRRP configuration (built on keepalived). Information on this OpenStack configuration can be found here: https://wiki.openstack.org/wiki/Neutron/L3_High_Availability_VRRP When we run, the configuration is failing to apply via keepalived. The logs post the following: Nov 17 02:58:31 p8test-lp1 Keepalived_vrrp[54542]: VRRP is trying to assign VIP to unknown qr-a5f5ba96-52 interface !!! go out and fix your conf !!! However, the device DOES exist. But the keepalived config just doesn't always deploy it. ii keepalived 1:1.2.19-1 ppc64el Failover and monitoring daemon for LVS clusters This configuration sometimes works, but does sometimes fail on Ubuntu 16.04.1 ---uname output--- Linux p8test-lp1 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:38:24 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux ---Additional Hardware Info--- This is a Power8 system with Ubuntu 16.04.1 installed. Though we see no indication that this is specific to Power. Machine Type = S822L Machine Type = 8286-42A ---Steps to Reproduce--- Install openstack. Run the network node in a VRRP HA configuration. Create a router and assign a global IP. == Comment: #5 - David J. Wilder - 2016-11-17 15:58:04 == The problem is fixed in this upstream commit: https://github.com/acassen/keepalived/commit/9f327bbf3e86def1055a106eda0633638bda0345 On systems with a page size larger than 4096 keepalived may report: "Netlink: error: message truncated" messages This error was reported on a ppc64le in an OpenStack/Nutron environment. Ppc64le is using a 64k pages size. I found that keepalived's netlink recvmsg buffer was too small causing messages to be truncated. The size of the read buffer for the netlink socket should be based on page size however, it should not exceed 8192. See the comment in the patch. I tested the fix by creating 100 veth interfaces and verifying the errors did not return. Signed-off-by: David WilderSigned-off-by: Quentin Armitage ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1642763/+subscriptions ___ Mailing list: https://launchpad.net/~ubuntu-ha Post to : ubuntu-ha@lists.launchpad.net Unsubscribe : https://launchpad.net/~ubuntu-ha More help : https://help.launchpad.net/ListHelp
[Ubuntu-ha] [Bug 1642763] Re: keepalived raising VIP apply error
** Also affects: keepalived (Ubuntu Yakkety) Importance: Undecided Status: New ** Also affects: keepalived (Ubuntu Xenial) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu High Availability Team, which is subscribed to keepalived in Ubuntu. https://bugs.launchpad.net/bugs/1642763 Title: keepalived raising VIP apply error Status in keepalived package in Ubuntu: Triaged Status in keepalived source package in Xenial: New Status in keepalived source package in Yakkety: New Bug description: == Comment: #0 - Andrew Thorstensen - 2016-11-17 09:50:25 == ---Problem Description--- Using Ubuntu 16.04 on ppc64le, we are building a 'neutron network node' using the VRRP configuration (built on keepalived). Information on this OpenStack configuration can be found here: https://wiki.openstack.org/wiki/Neutron/L3_High_Availability_VRRP When we run, the configuration is failing to apply via keepalived. The logs post the following: Nov 17 02:58:31 p8test-lp1 Keepalived_vrrp[54542]: VRRP is trying to assign VIP to unknown qr-a5f5ba96-52 interface !!! go out and fix your conf !!! However, the device DOES exist. But the keepalived config just doesn't always deploy it. ii keepalived 1:1.2.19-1 ppc64el Failover and monitoring daemon for LVS clusters This configuration sometimes works, but does sometimes fail on Ubuntu 16.04.1 ---uname output--- Linux p8test-lp1 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:38:24 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux ---Additional Hardware Info--- This is a Power8 system with Ubuntu 16.04.1 installed. Though we see no indication that this is specific to Power. Machine Type = S822L Machine Type = 8286-42A ---Steps to Reproduce--- Install openstack. Run the network node in a VRRP HA configuration. Create a router and assign a global IP. == Comment: #5 - David J. Wilder - 2016-11-17 15:58:04 == The problem is fixed in this upstream commit: https://github.com/acassen/keepalived/commit/9f327bbf3e86def1055a106eda0633638bda0345 On systems with a page size larger than 4096 keepalived may report: "Netlink: error: message truncated" messages This error was reported on a ppc64le in an OpenStack/Nutron environment. Ppc64le is using a 64k pages size. I found that keepalived's netlink recvmsg buffer was too small causing messages to be truncated. The size of the read buffer for the netlink socket should be based on page size however, it should not exceed 8192. See the comment in the patch. I tested the fix by creating 100 veth interfaces and verifying the errors did not return. Signed-off-by: David WilderSigned-off-by: Quentin Armitage ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1642763/+subscriptions ___ Mailing list: https://launchpad.net/~ubuntu-ha Post to : ubuntu-ha@lists.launchpad.net Unsubscribe : https://launchpad.net/~ubuntu-ha More help : https://help.launchpad.net/ListHelp
[Ubuntu-ha] [Bug 1642763] Re: keepalived raising VIP apply error
Thanks David. Asking Nish to take a look at this for you. ** Changed in: keepalived (Ubuntu) Assignee: Taco Screen team (taco-screen-team) => Nish Aravamudan (nacc) -- You received this bug notification because you are a member of Ubuntu High Availability Team, which is subscribed to keepalived in Ubuntu. https://bugs.launchpad.net/bugs/1642763 Title: keepalived raising VIP apply error Status in keepalived package in Ubuntu: Triaged Bug description: == Comment: #0 - Andrew Thorstensen - 2016-11-17 09:50:25 == ---Problem Description--- Using Ubuntu 16.04 on ppc64le, we are building a 'neutron network node' using the VRRP configuration (built on keepalived). Information on this OpenStack configuration can be found here: https://wiki.openstack.org/wiki/Neutron/L3_High_Availability_VRRP When we run, the configuration is failing to apply via keepalived. The logs post the following: Nov 17 02:58:31 p8test-lp1 Keepalived_vrrp[54542]: VRRP is trying to assign VIP to unknown qr-a5f5ba96-52 interface !!! go out and fix your conf !!! However, the device DOES exist. But the keepalived config just doesn't always deploy it. ii keepalived 1:1.2.19-1 ppc64el Failover and monitoring daemon for LVS clusters This configuration sometimes works, but does sometimes fail on Ubuntu 16.04.1 ---uname output--- Linux p8test-lp1 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:38:24 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux ---Additional Hardware Info--- This is a Power8 system with Ubuntu 16.04.1 installed. Though we see no indication that this is specific to Power. Machine Type = S822L Machine Type = 8286-42A ---Steps to Reproduce--- Install openstack. Run the network node in a VRRP HA configuration. Create a router and assign a global IP. == Comment: #5 - David J. Wilder - 2016-11-17 15:58:04 == The problem is fixed in this upstream commit: https://github.com/acassen/keepalived/commit/9f327bbf3e86def1055a106eda0633638bda0345 On systems with a page size larger than 4096 keepalived may report: "Netlink: error: message truncated" messages This error was reported on a ppc64le in an OpenStack/Nutron environment. Ppc64le is using a 64k pages size. I found that keepalived's netlink recvmsg buffer was too small causing messages to be truncated. The size of the read buffer for the netlink socket should be based on page size however, it should not exceed 8192. See the comment in the patch. I tested the fix by creating 100 veth interfaces and verifying the errors did not return. Signed-off-by: David WilderSigned-off-by: Quentin Armitage ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1642763/+subscriptions ___ Mailing list: https://launchpad.net/~ubuntu-ha Post to : ubuntu-ha@lists.launchpad.net Unsubscribe : https://launchpad.net/~ubuntu-ha More help : https://help.launchpad.net/ListHelp
[Ubuntu-ha] [Bug 1642763] Re: keepalived raising VIP apply error
Dave Wilder, do you have a way to trigger bug/test fix that wouldn't require a full openstack + HA-VRPP setup? -- You received this bug notification because you are a member of Ubuntu High Availability Team, which is subscribed to keepalived in Ubuntu. https://bugs.launchpad.net/bugs/1642763 Title: keepalived raising VIP apply error Status in keepalived package in Ubuntu: Triaged Bug description: == Comment: #0 - Andrew Thorstensen - 2016-11-17 09:50:25 == ---Problem Description--- Using Ubuntu 16.04 on ppc64le, we are building a 'neutron network node' using the VRRP configuration (built on keepalived). Information on this OpenStack configuration can be found here: https://wiki.openstack.org/wiki/Neutron/L3_High_Availability_VRRP When we run, the configuration is failing to apply via keepalived. The logs post the following: Nov 17 02:58:31 p8test-lp1 Keepalived_vrrp[54542]: VRRP is trying to assign VIP to unknown qr-a5f5ba96-52 interface !!! go out and fix your conf !!! However, the device DOES exist. But the keepalived config just doesn't always deploy it. ii keepalived 1:1.2.19-1 ppc64el Failover and monitoring daemon for LVS clusters This configuration sometimes works, but does sometimes fail on Ubuntu 16.04.1 ---uname output--- Linux p8test-lp1 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:38:24 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux ---Additional Hardware Info--- This is a Power8 system with Ubuntu 16.04.1 installed. Though we see no indication that this is specific to Power. Machine Type = S822L Machine Type = 8286-42A ---Steps to Reproduce--- Install openstack. Run the network node in a VRRP HA configuration. Create a router and assign a global IP. == Comment: #5 - David J. Wilder - 2016-11-17 15:58:04 == The problem is fixed in this upstream commit: https://github.com/acassen/keepalived/commit/9f327bbf3e86def1055a106eda0633638bda0345 On systems with a page size larger than 4096 keepalived may report: "Netlink: error: message truncated" messages This error was reported on a ppc64le in an OpenStack/Nutron environment. Ppc64le is using a 64k pages size. I found that keepalived's netlink recvmsg buffer was too small causing messages to be truncated. The size of the read buffer for the netlink socket should be based on page size however, it should not exceed 8192. See the comment in the patch. I tested the fix by creating 100 veth interfaces and verifying the errors did not return. Signed-off-by: David WilderSigned-off-by: Quentin Armitage ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1642763/+subscriptions ___ Mailing list: https://launchpad.net/~ubuntu-ha Post to : ubuntu-ha@lists.launchpad.net Unsubscribe : https://launchpad.net/~ubuntu-ha More help : https://help.launchpad.net/ListHelp
[Ubuntu-ha] [Bug 1642763] Re: keepalived raising VIP apply error
So I'd consider this will end up as Xenial SRU and depending on your counsel into cloud-archive as Jon asked you. Given the SRU policy (https://wiki.ubuntu.com/StableReleaseUpdates) a fix should go into the Development release first (currently zesty) and become SRUs from there. The fix upstream is just two days old and not released yet. Currently Ubuntu has no delta over the Debian version which means we should report the issue to Debian, fix it there and pick it up into Zesty on a merge/sync of their latest version. Your backported patch applies (almost) cleanly to the latest version in Debian/Ubuntu, so I'd ask you if you could report that to Debian as well and mention the created Debbug here. We could then sync the package afterwards and avoid having to maintain a Delta of some sorts. >From there the path to SRUs would be free then. ** Changed in: keepalived (Ubuntu) Importance: Undecided => Medium -- You received this bug notification because you are a member of Ubuntu High Availability Team, which is subscribed to keepalived in Ubuntu. https://bugs.launchpad.net/bugs/1642763 Title: keepalived raising VIP apply error Status in keepalived package in Ubuntu: Triaged Bug description: == Comment: #0 - Andrew Thorstensen - 2016-11-17 09:50:25 == ---Problem Description--- Using Ubuntu 16.04 on ppc64le, we are building a 'neutron network node' using the VRRP configuration (built on keepalived). Information on this OpenStack configuration can be found here: https://wiki.openstack.org/wiki/Neutron/L3_High_Availability_VRRP When we run, the configuration is failing to apply via keepalived. The logs post the following: Nov 17 02:58:31 p8test-lp1 Keepalived_vrrp[54542]: VRRP is trying to assign VIP to unknown qr-a5f5ba96-52 interface !!! go out and fix your conf !!! However, the device DOES exist. But the keepalived config just doesn't always deploy it. ii keepalived 1:1.2.19-1 ppc64el Failover and monitoring daemon for LVS clusters This configuration sometimes works, but does sometimes fail on Ubuntu 16.04.1 ---uname output--- Linux p8test-lp1 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:38:24 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux ---Additional Hardware Info--- This is a Power8 system with Ubuntu 16.04.1 installed. Though we see no indication that this is specific to Power. Machine Type = S822L Machine Type = 8286-42A ---Steps to Reproduce--- Install openstack. Run the network node in a VRRP HA configuration. Create a router and assign a global IP. == Comment: #5 - David J. Wilder - 2016-11-17 15:58:04 == The problem is fixed in this upstream commit: https://github.com/acassen/keepalived/commit/9f327bbf3e86def1055a106eda0633638bda0345 On systems with a page size larger than 4096 keepalived may report: "Netlink: error: message truncated" messages This error was reported on a ppc64le in an OpenStack/Nutron environment. Ppc64le is using a 64k pages size. I found that keepalived's netlink recvmsg buffer was too small causing messages to be truncated. The size of the read buffer for the netlink socket should be based on page size however, it should not exceed 8192. See the comment in the patch. I tested the fix by creating 100 veth interfaces and verifying the errors did not return. Signed-off-by: David WilderSigned-off-by: Quentin Armitage ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1642763/+subscriptions ___ Mailing list: https://launchpad.net/~ubuntu-ha Post to : ubuntu-ha@lists.launchpad.net Unsubscribe : https://launchpad.net/~ubuntu-ha More help : https://help.launchpad.net/ListHelp
[Ubuntu-ha] [Bug 1642763] Re: keepalived raising VIP apply error
Thank you for your bug report and even more so for an upstreamed patch. I'm thinking this will need pushed into the cloud-archive as well. I'll coordinate if so. ** Changed in: keepalived (Ubuntu) Status: New => Triaged -- You received this bug notification because you are a member of Ubuntu High Availability Team, which is subscribed to keepalived in Ubuntu. https://bugs.launchpad.net/bugs/1642763 Title: keepalived raising VIP apply error Status in keepalived package in Ubuntu: Triaged Bug description: == Comment: #0 - Andrew Thorstensen - 2016-11-17 09:50:25 == ---Problem Description--- Using Ubuntu 16.04 on ppc64le, we are building a 'neutron network node' using the VRRP configuration (built on keepalived). Information on this OpenStack configuration can be found here: https://wiki.openstack.org/wiki/Neutron/L3_High_Availability_VRRP When we run, the configuration is failing to apply via keepalived. The logs post the following: Nov 17 02:58:31 p8test-lp1 Keepalived_vrrp[54542]: VRRP is trying to assign VIP to unknown qr-a5f5ba96-52 interface !!! go out and fix your conf !!! However, the device DOES exist. But the keepalived config just doesn't always deploy it. ii keepalived 1:1.2.19-1 ppc64el Failover and monitoring daemon for LVS clusters This configuration sometimes works, but does sometimes fail on Ubuntu 16.04.1 ---uname output--- Linux p8test-lp1 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:38:24 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux ---Additional Hardware Info--- This is a Power8 system with Ubuntu 16.04.1 installed. Though we see no indication that this is specific to Power. Machine Type = S822L Machine Type = 8286-42A ---Steps to Reproduce--- Install openstack. Run the network node in a VRRP HA configuration. Create a router and assign a global IP. == Comment: #5 - David J. Wilder - 2016-11-17 15:58:04 == The problem is fixed in this upstream commit: https://github.com/acassen/keepalived/commit/9f327bbf3e86def1055a106eda0633638bda0345 On systems with a page size larger than 4096 keepalived may report: "Netlink: error: message truncated" messages This error was reported on a ppc64le in an OpenStack/Nutron environment. Ppc64le is using a 64k pages size. I found that keepalived's netlink recvmsg buffer was too small causing messages to be truncated. The size of the read buffer for the netlink socket should be based on page size however, it should not exceed 8192. See the comment in the patch. I tested the fix by creating 100 veth interfaces and verifying the errors did not return. Signed-off-by: David WilderSigned-off-by: Quentin Armitage ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1642763/+subscriptions ___ Mailing list: https://launchpad.net/~ubuntu-ha Post to : ubuntu-ha@lists.launchpad.net Unsubscribe : https://launchpad.net/~ubuntu-ha More help : https://help.launchpad.net/ListHelp