[Bug 1933092] Re: snat arp entry missing in qrouter namespace
** Changed in: cloud-archive/victoria Status: New => Fix Released ** Changed in: cloud-archive/wallaby Status: New => Fix Released ** Changed in: cloud-archive/xena Status: New => Fix Released ** Changed in: neutron (Ubuntu Groovy) Status: New => Fix Released ** Changed in: neutron (Ubuntu Hirsute) Status: New => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1933092 Title: snat arp entry missing in qrouter namespace To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1933092/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1933092] Re: snat arp entry missing in qrouter namespace
Reviewed: https://review.opendev.org/c/openstack/neutron/+/807244 Committed: https://opendev.org/openstack/neutron/commit/c689ac5a661a15ec7ad1098d21dc5affce97fb04 Submitter: "Zuul (22348)" Branch:stable/train commit c689ac5a661a15ec7ad1098d21dc5affce97fb04 Author: Hemanth Nakkina Date: Fri Jul 2 17:01:55 2021 +0530 Update arp entry of snat port on qrouter ns In some cases, the arp entry of snat port is not updated in qrouter namespace. l3-agent calls get_ports_by_subnet() while setting arps for the subnet. And the snat port is not returned if it is still unbound. One of the scenario this is observed is when router is created, external gateway set and internal subnet attached to router in quick succession. This patch retrieves snat port details from router info as well and updates arp entry for snat port. Conflicts: neutron/agent/l3/dvr_local_router.py Closes-Bug: #1933092 Change-Id: I7ee797b4b930306cf6360922d855f8b24f1b813d (cherry picked from commit be7d0bb6abc893e53dfc864c52506928b1d38fa3) (cherry picked from commit f1a9f4ed62fd2567cac174f80f87de53148ea7b9) ** Tags added: in-stable-train -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1933092 Title: snat arp entry missing in qrouter namespace To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1933092/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1933092] Re: snat arp entry missing in qrouter namespace
** Tags added: neutron-proactive-backport-potential -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1933092 Title: snat arp entry missing in qrouter namespace To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1933092/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1933092] Re: snat arp entry missing in qrouter namespace
** Changed in: neutron (Ubuntu Focal) Status: New => Fix Released ** Changed in: cloud-archive/ussuri Status: New => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1933092 Title: snat arp entry missing in qrouter namespace To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1933092/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1933092] Re: snat arp entry missing in qrouter namespace
This bug was fixed in the package neutron - 2:18.1.0+git2021072117.147830620f-0ubuntu1 --- neutron (2:18.1.0+git2021072117.147830620f-0ubuntu1) impish; urgency=medium [ Hemanth Nakkina ] * d/p/0001-Update-arp-entry-of-snat-port-on-qrouter-ns.patch: Fix to update arp entry of snat port on qrouter ns (LP: #1933092) [ Corey Bryant ] * New upstream snapshot for OpenStack Xena. * d/control: Align (Build-)Depends with upstream. * d/p/*: Rebased. * d/p/0001-Update-arp-entry-of-snat-port-on-qrouter-ns.patch: Dropped. Fixed in upstream snapshot. -- Corey Bryant Wed, 21 Jul 2021 17:22:28 -0400 ** Changed in: neutron (Ubuntu Impish) Status: New => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1933092 Title: snat arp entry missing in qrouter namespace To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1933092/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1933092] Re: snat arp entry missing in qrouter namespace
** Changed in: neutron (Ubuntu Focal) Importance: Undecided => Medium ** Changed in: neutron (Ubuntu Groovy) Importance: Undecided => Medium ** Changed in: neutron (Ubuntu Hirsute) Importance: Undecided => Medium ** Changed in: neutron (Ubuntu Impish) Importance: Undecided => Medium -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1933092 Title: snat arp entry missing in qrouter namespace To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1933092/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1933092] Re: snat arp entry missing in qrouter namespace
pushed to U/V/W and master branches: https://git.launchpad.net/~ubuntu-openstack- dev/ubuntu/+source/neutron/commit/?id=100e69d4593dde0363d8d368c836c2cc49dd2d7b https://git.launchpad.net/~ubuntu-openstack- dev/ubuntu/+source/neutron/commit/?id=6348afdd66039ca5365169e315872fb38ad64f3f https://git.launchpad.net/~ubuntu-openstack- dev/ubuntu/+source/neutron/commit/?id=5a27fae295fc475ea3d7105850066655c23c1269 https://git.launchpad.net/~ubuntu-openstack- dev/ubuntu/+source/neutron/commit/?id=22a4f5867c2a44d5ee6636af9fe2f79624a9cd6f chris plans to start new stable point releases tomorrow so this will get uploaded along with that -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1933092 Title: snat arp entry missing in qrouter namespace To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1933092/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1933092] Re: snat arp entry missing in qrouter namespace
The attachment "Debdiff for impish" seems to be a debdiff. The ubuntu- sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team. [This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.] ** Tags added: patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1933092 Title: snat arp entry missing in qrouter namespace To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1933092/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1933092] Re: snat arp entry missing in qrouter namespace
SRU team, All debdiffs for Ubuntu I/H/G/F and UCA X/W/V/U are uploaded. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1933092 Title: snat arp entry missing in qrouter namespace To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1933092/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1933092] Re: snat arp entry missing in qrouter namespace
** Patch added: "Debdiff for UCA xena" https://bugs.launchpad.net/cloud-archive/wallaby/+bug/1933092/+attachment/5509624/+files/lp1933092_xena.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1933092 Title: snat arp entry missing in qrouter namespace To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1933092/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1933092] Re: snat arp entry missing in qrouter namespace
** Patch added: "Debdiff for UCA ussuri" https://bugs.launchpad.net/cloud-archive/wallaby/+bug/1933092/+attachment/5509601/+files/lp1933092_ussuri.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1933092 Title: snat arp entry missing in qrouter namespace To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1933092/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1933092] Re: snat arp entry missing in qrouter namespace
** Patch added: "Debdiff for UCA victoria" https://bugs.launchpad.net/cloud-archive/wallaby/+bug/1933092/+attachment/5509600/+files/lp1933092_victoria.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1933092 Title: snat arp entry missing in qrouter namespace To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1933092/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1933092] Re: snat arp entry missing in qrouter namespace
** Patch added: "Debdiff for UCA wallaby" https://bugs.launchpad.net/cloud-archive/wallaby/+bug/1933092/+attachment/5509599/+files/lp1933092_wallaby.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1933092 Title: snat arp entry missing in qrouter namespace To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1933092/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1933092] Re: snat arp entry missing in qrouter namespace
** Patch added: "Debdiff for focal" https://bugs.launchpad.net/cloud-archive/wallaby/+bug/1933092/+attachment/5509598/+files/lp1933092_focal.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1933092 Title: snat arp entry missing in qrouter namespace To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1933092/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1933092] Re: snat arp entry missing in qrouter namespace
** Patch added: "Debdiff for groovy" https://bugs.launchpad.net/cloud-archive/wallaby/+bug/1933092/+attachment/5509597/+files/lp1933092_groovy.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1933092 Title: snat arp entry missing in qrouter namespace To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1933092/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1933092] Re: snat arp entry missing in qrouter namespace
** Patch added: "Debdiff for hirsute" https://bugs.launchpad.net/cloud-archive/wallaby/+bug/1933092/+attachment/5509596/+files/lp1933092_hirsute.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1933092 Title: snat arp entry missing in qrouter namespace To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1933092/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1933092] Re: snat arp entry missing in qrouter namespace
** Patch added: "Debdiff for impish" https://bugs.launchpad.net/cloud-archive/wallaby/+bug/1933092/+attachment/5509595/+files/lp1933092_impish.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1933092 Title: snat arp entry missing in qrouter namespace To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1933092/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1933092] Re: snat arp entry missing in qrouter namespace
** Tags added: sts sts-sru-needed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1933092 Title: snat arp entry missing in qrouter namespace To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1933092/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1933092] Re: snat arp entry missing in qrouter namespace
** Description changed: - In one of the cloud environment, the FIP attached to the Octavia - Loadbalancer VIP is not reachable. After analysis, we found the ARP - entry for SNAT IP is missing in the qrouter namespace where Amphora VM - is running. And so the return packets are not forwarded from qrouter to - snat on active l3-agent node. + [Impact] + Load Balancers deployed on the cloud are unreachable + + [Test Case] + 1. Deploy openstack with atleast 4 compute nodes with networking features DVR SNAT+L3HA + 2. Execute the script test_snat_arp_entry.sh + 3. The script loops for 20 times creating network, router and connecting router to external, internal network and checking if ARP entries are populated properly on qrouter namespaces + 4. The script stops if arp entries are missing. + 5. If the script runs for 20 loops, then there are no issues. + + [Regression Potential] + The issue only happens a few times when a router is created, external gateway set and internal subnet attached to router in quick succession. In other cases, the arp entry of snat is already added. + The fix just adds extra logic to add arp entry retrieving snat information from the router. In working cases, this extra logic will execute commands to add arp entry twice which should not cause further issues. + + [Original Bug Report] + In one of the cloud environment, the FIP attached to the Octavia Loadbalancer VIP is not reachable. After analysis, we found the ARP entry for SNAT IP is missing in the qrouter namespace where Amphora VM is running. And so the return packets are not forwarded from qrouter to snat on active l3-agent node. Version: Ubuntu Ussuri packages (16.3.2 point release) DVR+SNAT+L3HA enabled Expectation is to have PERMANENT arp entry for snat ip on qrouter namespace on all compute nodes 192.168.33.238 dev qr-4ee692e0-7a lladdr fa:16:3e:25:6a:73 used 38/38/38 probes 0 PERMANENT How to reproduce: Attaching a script to simulate the problem (without octavia) with following steps 1. network/subnet/router is created, network attached to router 2. verify if qrouter on all compute nodes has arp entries related to snat ip 3. if arp entries exists, delete network/subnet/router 4. Repeat steps 1,2,3 until missing arp entry is observed. I am able to reproduce missing arp entry sometimes in 3rd loop and sometimes in 6th loop. Observed arp entries for snat ip is updated at the following places [1] [2] but get_snat_interfaces() and get_ports_by_subnet() are not updated with snat ip in non-working cases. [1] https://opendev.org/openstack/neutron/src/commit/dfd04115b059c2263cdd8ac44ccc2ec47614bcc3/neutron/agent/l3/dvr_local_router.py#L570 [2] https://opendev.org/openstack/neutron/src/commit/dfd04115b059c2263cdd8ac44ccc2ec47614bcc3/neutron/agent/l3/dvr_local_router.py#L317 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1933092 Title: snat arp entry missing in qrouter namespace To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1933092/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1933092] Re: snat arp entry missing in qrouter namespace
** Also affects: neutron (Ubuntu) Importance: Undecided Status: New ** Also affects: neutron (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: neutron (Ubuntu Hirsute) Importance: Undecided Status: New ** Also affects: neutron (Ubuntu Groovy) Importance: Undecided Status: New ** Also affects: neutron (Ubuntu Impish) Importance: Undecided Status: New ** Also affects: cloud-archive Importance: Undecided Status: New ** Also affects: cloud-archive/victoria Importance: Undecided Status: New ** Also affects: cloud-archive/ussuri Importance: Undecided Status: New ** Also affects: cloud-archive/xena Importance: Undecided Status: New ** Also affects: cloud-archive/wallaby Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1933092 Title: snat arp entry missing in qrouter namespace To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1933092/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
