[Bug 1883089] Re: [L3] floating IP failed to bind due to no agent gateway port(fip-ns)
** Changed in: cloud-archive Status: New => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1883089 Title: [L3] floating IP failed to bind due to no agent gateway port(fip-ns) To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1883089/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1883089] Re: [L3] floating IP failed to bind due to no agent gateway port(fip-ns)
UCA Ussuri is released to ussuri-updates in package 2:16.3.2-0ubuntu3~cloud0, so marking the status as Fix released for UCA Ussuri ** Changed in: cloud-archive/ussuri Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1883089 Title: [L3] floating IP failed to bind due to no agent gateway port(fip-ns) To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1883089/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1883089] Re: [L3] floating IP failed to bind due to no agent gateway port(fip-ns)
Verified the test case on bionic-ussuri and the test case works with the package in cloud:archive:ussuri-proposed Deleted the floatin ip agent gateway on one of the compute node and launched a new VM on that compute and assigned FIP. Able to ping Floating IP. $ ping -c 1 10.5.153.114 PING 10.5.153.114 (10.5.153.114) 56(84) bytes of data. 64 bytes from 10.5.153.114: icmp_seq=1 ttl=62 time=3.66 ms --- 10.5.153.114 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 3.662/3.662/3.662/0.000 ms ** Tags removed: verification-needed verification-ussuri-needed ** Tags added: verification-done verification-ussuri-done -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1883089 Title: [L3] floating IP failed to bind due to no agent gateway port(fip-ns) To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1883089/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1883089] Re: [L3] floating IP failed to bind due to no agent gateway port(fip-ns)
This bug was fixed in the package neutron - 2:16.3.1-0ubuntu1.1 --- neutron (2:16.3.1-0ubuntu1.1) focal; urgency=medium [ Hemanth Nakkina ] * Fix to check L3 agent gateway port robustly (LP: #1883089) - d/p/0001-L3-Check-agent-gateway-port-robustly.patch. -- Chris MacNaughton Thu, 06 May 2021 10:39:19 + ** Changed in: neutron (Ubuntu Focal) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1883089 Title: [L3] floating IP failed to bind due to no agent gateway port(fip-ns) To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1883089/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1883089] Re: [L3] floating IP failed to bind due to no agent gateway port(fip-ns)
Tested on focal with neutron package 2:16.3.1-0ubuntu1.1 and the test case is successful * Deployed environment with dvr l3ha and centralised neutron snat gateways. * Floating IP agent gateway exists on all nova-compute nodes (4) and neutron-gateway nodes (3) after launching VMs on all compute nodes $ openstack port list --network ext_net -c id -c device_id -c binding_host_id -c device_owner -c fixed_ips | grep floatingip_agent_gateway | wc -l 7 * Deleted one of the Floating IP agent gateway port $ openstack port list --network ext_net -c id -c device_id -c binding_host_id -c device_owner -c fixed_ips | grep floatingip_agent_gateway | wc -l 6 * Launched VM on the node where gateway port is deleted. Floating IP agent gateway came back on the node $ openstack port list --network ext_net -c id -c device_id -c binding_host_id -c device_owner -c fixed_ips | grep floatingip_agent_gateway | wc -l 7 * ping to the floating ip successful $ ping -c 1 10.5.151.84 PING 10.5.151.84 (10.5.151.84) 56(84) bytes of data. 64 bytes from 10.5.151.84: icmp_seq=1 ttl=62 time=293 ms --- 10.5.151.84 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 292.825/292.825/292.825/0.000 ms ** Tags removed: verification-needed-focal ** Tags added: verification-done-focal -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1883089 Title: [L3] floating IP failed to bind due to no agent gateway port(fip-ns) To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1883089/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1883089] Re: [L3] floating IP failed to bind due to no agent gateway port(fip-ns)
Hello LIU, or anyone else affected, Accepted neutron into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:16.3.1-0ubuntu1.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed- focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification- failed-focal. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: neutron (Ubuntu Focal) Status: New => Fix Committed ** Tags added: verification-needed verification-needed-focal -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1883089 Title: [L3] floating IP failed to bind due to no agent gateway port(fip-ns) To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1883089/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1883089] Re: [L3] floating IP failed to bind due to no agent gateway port(fip-ns)
** Description changed:
In patch [1] it introduced a binding of DB uniq constraint for L3
agent gateway. In some extreme case the DvrFipGatewayPortAgentBinding
is in DB while the gateway port not. The current code path only checks
the binding existence which will pass a "None" port to the following
code path that results an AttributeError.
[1] https://review.opendev.org/#/c/702547/
Exception log:
2020-06-11 15:39:28.361 1285214 INFO neutron.db.l3_dvr_db [None
req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway
port for network 3fcb7702-ae0b-46b4-807f-8ae94d656dd3 does not exist on host
host-compute-1. Creating one.
2020-06-11 15:39:28.370 1285214 DEBUG neutron.db.l3_dvr_db [None
req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway
port for network 3fcb7702-ae0b-46b4-807f-8ae94d656dd3 already exists on host
host-compute-1. Probably it was just created by other worker.
create_fip_agent_gw_port_if_not_exists
/usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py:927
2020-06-11 15:39:28.390 1285214 DEBUG neutron.db.l3_dvr_db [None
req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway
port None found for the destination host: host-compute-1
create_fip_agent_gw_port_if_not_exists
/usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py:933
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server [None
req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Exception during message
handling: AttributeError: 'NoneType' object has no attribute 'get'
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server Traceback
(most recent call last):
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 170, in
_process_incoming
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220,
in dispatch
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190,
in _do_dispatch
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server result =
func(ctxt, **new_args)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
setattr(e, '_RETRY_EXCEEDED', True)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron/db/api.py", line 87, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return
f(*args, **kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
ectxt.value = e.inner_exc
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return
f(*args, **kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
LOG.debug("Retry wrapper got retriable exception: %s", e)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in
[Bug 1883089] Re: [L3] floating IP failed to bind due to no agent gateway port(fip-ns)
** Also affects: cloud-archive Importance: Undecided Status: New ** Also affects: cloud-archive/victoria Importance: Undecided Status: New ** Also affects: cloud-archive/ussuri Importance: Undecided Status: New ** Changed in: cloud-archive/victoria Status: New => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1883089 Title: [L3] floating IP failed to bind due to no agent gateway port(fip-ns) To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1883089/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1883089] Re: [L3] floating IP failed to bind due to no agent gateway port(fip-ns)
SRU Team, The fix has 2 commits (referred stable/ussuri below) https://review.opendev.org/c/openstack/neutron/+/779614 https://review.opendev.org/c/openstack/neutron/+/779613 779614 is already part of focal (latest ussuri stable point release on Apr 12) Uploaded debdiff with changes from 779613 ** Changed in: neutron (Ubuntu Focal) Assignee: (unassigned) => Hemanth Nakkina (hemanth-n) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1883089 Title: [L3] floating IP failed to bind due to no agent gateway port(fip-ns) To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1883089/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1883089] Re: [L3] floating IP failed to bind due to no agent gateway port(fip-ns)
** Description changed:
In patch [1] it introduced a binding of DB uniq constraint for L3
agent gateway. In some extreme case the DvrFipGatewayPortAgentBinding
is in DB while the gateway port not. The current code path only checks
the binding existence which will pass a "None" port to the following
code path that results an AttributeError.
[1] https://review.opendev.org/#/c/702547/
-
Exception log:
2020-06-11 15:39:28.361 1285214 INFO neutron.db.l3_dvr_db [None
req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway
port for network 3fcb7702-ae0b-46b4-807f-8ae94d656dd3 does not exist on host
host-compute-1. Creating one.
2020-06-11 15:39:28.370 1285214 DEBUG neutron.db.l3_dvr_db [None
req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway
port for network 3fcb7702-ae0b-46b4-807f-8ae94d656dd3 already exists on host
host-compute-1. Probably it was just created by other worker.
create_fip_agent_gw_port_if_not_exists
/usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py:927
2020-06-11 15:39:28.390 1285214 DEBUG neutron.db.l3_dvr_db [None
req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway
port None found for the destination host: host-compute-1
create_fip_agent_gw_port_if_not_exists
/usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py:933
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server [None
req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Exception during message
handling: AttributeError: 'NoneType' object has no attribute 'get'
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server Traceback
(most recent call last):
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 170, in
_process_incoming
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220,
in dispatch
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190,
in _do_dispatch
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server result =
func(ctxt, **new_args)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
setattr(e, '_RETRY_EXCEEDED', True)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron/db/api.py", line 87, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return
f(*args, **kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
ectxt.value = e.inner_exc
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return
f(*args, **kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
LOG.debug("Retry wrapper got retriable exception: %s", e)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220,
[Bug 1883089] Re: [L3] floating IP failed to bind due to no agent gateway port(fip-ns)
** Also affects: neutron (Ubuntu Impish) Importance: Undecided Status: New ** Also affects: neutron (Ubuntu Hirsute) Importance: Undecided Status: New ** Also affects: neutron (Ubuntu Groovy) Importance: Undecided Status: New ** Changed in: neutron (Ubuntu Impish) Status: New => Fix Released ** Changed in: neutron (Ubuntu Hirsute) Status: New => Fix Released ** Changed in: neutron (Ubuntu Groovy) Status: New => Fix Released ** Tags added: sts -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1883089 Title: [L3] floating IP failed to bind due to no agent gateway port(fip-ns) To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1883089/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1883089] Re: [L3] floating IP failed to bind due to no agent gateway port(fip-ns)
** Also affects: neutron (Ubuntu) Importance: Undecided Status: New ** Also affects: neutron (Ubuntu Focal) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1883089 Title: [L3] floating IP failed to bind due to no agent gateway port(fip-ns) To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1883089/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
