Public bug reported:
In patch https://review.opendev.org/#/c/702547/ new db level lock was
introduced to ensure that there is always max. 1 FIP gateway port for network
on the host.
But unfortunately I missed to clean entry in this new table when such gateway
port is removed. That causes issues with DVR routers when gateway is plug to
the router.
Steps to reproduce issue:
1. Create dvr router
2. Attach some network with vm to the router,
3. Attach external gateway network to the router,
4. It will create fip- namespace on compute,
5. remove this router, fip- namespace will be cleaned and agent's gateway port
will be removed from neutron db,
6. repeat steps 1-5 - it will now fail with error like:
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: ERROR
neutron.agent.l3.router_info [-] 'NoneType' object has no attribute 'get'
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: Traceback (most
recent call last):
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/server.py", line
165, in _process_incoming
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: res =
self.dispatcher.dispatch(message)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/dispatcher.py", line
276, in dispatch
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: return
self._do_dispatch(endpoint, method, ctxt, args)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/dispatcher.py", line
196, in _do_dispatch
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: result =
func(ctxt, **new_args)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 139, in
wrapped
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: setattr(e,
'_RETRY_EXCEEDED', True)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:
self.force_reraise()
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:
six.reraise(self.type_, self.value, self.tb)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: raise value
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 135, in
wrapped
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: return
f(*args, **kwargs)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/oslo_db/api.py", line 154, in wrapper
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: ectxt.value =
e.inner_exc
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:
self.force_reraise()
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:
six.reraise(self.type_, self.value, self.tb)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: raise value
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/oslo_db/api.py", line 142, in wrapper
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: return
f(*args, **kwargs)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 183, in
wrapped
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:
LOG.debug("Retry wrapper got retriable exception: %s", e)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:
self.force_reraise()
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:
six.reraise(self.type_, self.value, self.tb)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: raise value
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 179, in
wrapped
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: return
f(*dup_args, **dup_kwargs)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/opt/stack/neutron/neutron/api/rpc/handlers/l3_rpc.py", line 319, in
get_agent_gateway_port
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: admin_ctx,
network_id, host)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/opt/stack/neutron/neutron/db/l3_dvr_db.py", line 1078, in
create_fip_agent_gw_port_if_not_exists
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:
self._populate_mtu_and_subnets_for_ports(context, [agent_port])
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/opt/stack/neutron/neutron/db/l3_db.py", line 1762, in
_populate_mtu_and_subnets_for_ports
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: for p in
self._each_port_having_fixed_ips(ports)]
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/opt/stack/neutron/neutron/db/l3_db.py", line 1761, in <listcomp>
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: network_ids =
[p['network_id']
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/opt/stack/neutron/neutron/db/l3_db.py", line 1709, in
_each_port_having_fixed_ips
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: fixed_ips =
port.get('fixed_ips', [])
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: AttributeError:
'NoneType' object has no attribute 'get'
To solve this issue You need to manually remove corresponding row from
"dvr_fip_gateway_port_network" db table.
** Affects: neutron
Importance: High
Assignee: Slawek Kaplonski (slaweq)
Status: Confirmed
** Tags: l3-dvr-backlog
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1866336
Title:
Binding of floating ip agent gateway port and agent_id isn't removed
Status in neutron:
Confirmed
Bug description:
In patch https://review.opendev.org/#/c/702547/ new db level lock was
introduced to ensure that there is always max. 1 FIP gateway port for network
on the host.
But unfortunately I missed to clean entry in this new table when such gateway
port is removed. That causes issues with DVR routers when gateway is plug to
the router.
Steps to reproduce issue:
1. Create dvr router
2. Attach some network with vm to the router,
3. Attach external gateway network to the router,
4. It will create fip- namespace on compute,
5. remove this router, fip- namespace will be cleaned and agent's gateway
port will be removed from neutron db,
6. repeat steps 1-5 - it will now fail with error like:
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: ERROR
neutron.agent.l3.router_info [-] 'NoneType' object has no attribute 'get'
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: Traceback (most
recent call last):
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/server.py", line
165, in _process_incoming
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: res =
self.dispatcher.dispatch(message)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/dispatcher.py", line
276, in dispatch
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: return
self._do_dispatch(endpoint, method, ctxt, args)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/dispatcher.py", line
196, in _do_dispatch
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: result =
func(ctxt, **new_args)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 139, in
wrapped
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: setattr(e,
'_RETRY_EXCEEDED', True)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:
self.force_reraise()
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:
six.reraise(self.type_, self.value, self.tb)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: raise value
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 135, in
wrapped
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: return
f(*args, **kwargs)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/oslo_db/api.py", line 154, in wrapper
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: ectxt.value
= e.inner_exc
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:
self.force_reraise()
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:
six.reraise(self.type_, self.value, self.tb)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: raise value
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/oslo_db/api.py", line 142, in wrapper
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: return
f(*args, **kwargs)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 183, in
wrapped
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:
LOG.debug("Retry wrapper got retriable exception: %s", e)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:
self.force_reraise()
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:
six.reraise(self.type_, self.value, self.tb)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: raise value
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 179, in
wrapped
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: return
f(*dup_args, **dup_kwargs)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/opt/stack/neutron/neutron/api/rpc/handlers/l3_rpc.py", line 319, in
get_agent_gateway_port
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: admin_ctx,
network_id, host)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/opt/stack/neutron/neutron/db/l3_dvr_db.py", line 1078, in
create_fip_agent_gw_port_if_not_exists
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:
self._populate_mtu_and_subnets_for_ports(context, [agent_port])
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/opt/stack/neutron/neutron/db/l3_db.py", line 1762, in
_populate_mtu_and_subnets_for_ports
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: for p in
self._each_port_having_fixed_ips(ports)]
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/opt/stack/neutron/neutron/db/l3_db.py", line 1761, in <listcomp>
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: network_ids
= [p['network_id']
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: File
"/opt/stack/neutron/neutron/db/l3_db.py", line 1709, in
_each_port_having_fixed_ips
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: fixed_ips =
port.get('fixed_ips', [])
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: AttributeError:
'NoneType' object has no attribute 'get'
To solve this issue You need to manually remove corresponding row from
"dvr_fip_gateway_port_network" db table.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1866336/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
