Public bug reported:
We have seen this Traceback in Pike based installation, while trying to
cleanup a gateway with DVR routers.
2018-04-03 20:30:10.081 9672 DEBUG neutron.agent.l3.dvr_fip_ns [-] Delete FIP
link interfaces for router: e415276a-4f37-4ee0-ba48-12d3909153c7
delete_rtr_2_fip_link
/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-pac
kages/neutron/agent/l3/dvr_fip_ns.py:364
2018-04-03 20:30:10.082 9672 DEBUG neutron.agent.linux.utils [-] Running
command (rootwrap daemon): ['ip', 'netns', 'exec',
'qrouter-e415276a-4f37-4ee0-ba48-12d3909153c7', 'ip', '-o', 'link', 'show',
'rfp-e415276a-4'] execute_ro
otwrap_daemon
/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/linux/utils.py:108
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info [-]
u'e415276a-4f37-4ee0-ba48-12d3909153c7': KeyError:
u'e415276a-4f37-4ee0-ba48-12d3909153c7'
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info Traceback (most
recent call last):
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info File
"/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/common/utils.py",
line 186, in call
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info return
func(*args, **kwargs)
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info File
"/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/router_info.py",
line 1118, in process_delete
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info
self._process_external_on_delete()
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info File
"/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/router_info.py",
line 890, in _process_external_on_delete
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info
self._process_external_gateway(ex_gw_port)
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info File
"/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/router_info.py",
line 799, in _process_external_gateway
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info
self.external_gateway_removed(self.ex_gw_port, interface_name)
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info File
"/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py",
line 513, in external_gateway_removed
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info
self.fip_ns.delete_rtr_2_fip_link(self)
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info File
"/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_fip_ns.py",
line 402, in delete_rtr_2_fip_link
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info
self.local_subnets.release(ri.router_id)
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info File
"/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/item_allocator.py",
line 116, in release
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info
self.pool.add(self.allocations.pop(key))
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info KeyError:
u'e415276a-4f37-4ee0-ba48-12d3909153c7'
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info
Probably a check to make sure if the Key exists before release would be
a good idea.
We might also see if we can reproduce this in the master branch.
** Affects: neutron
Importance: Low
Status: New
** Tags: l3-dvr-backlog
** Changed in: neutron
Importance: Undecided => Critical
** Changed in: neutron
Importance: Critical => Low
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1761260
Title:
DVR: Add a check for the item_allocator IP before trying to release
it, since we see a KeyError sometimes, when the item is not there
anymore.
Status in neutron:
New
Bug description:
We have seen this Traceback in Pike based installation, while trying
to cleanup a gateway with DVR routers.
2018-04-03 20:30:10.081 9672 DEBUG neutron.agent.l3.dvr_fip_ns [-] Delete FIP
link interfaces for router: e415276a-4f37-4ee0-ba48-12d3909153c7
delete_rtr_2_fip_link
/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-pac
kages/neutron/agent/l3/dvr_fip_ns.py:364
2018-04-03 20:30:10.082 9672 DEBUG neutron.agent.linux.utils [-] Running
command (rootwrap daemon): ['ip', 'netns', 'exec',
'qrouter-e415276a-4f37-4ee0-ba48-12d3909153c7', 'ip', '-o', 'link', 'show',
'rfp-e415276a-4'] execute_ro
otwrap_daemon
/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/linux/utils.py:108
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info [-]
u'e415276a-4f37-4ee0-ba48-12d3909153c7': KeyError:
u'e415276a-4f37-4ee0-ba48-12d3909153c7'
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info Traceback
(most recent call last):
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info File
"/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/common/utils.py",
line 186, in call
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info return
func(*args, **kwargs)
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info File
"/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/router_info.py",
line 1118, in process_delete
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info
self._process_external_on_delete()
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info File
"/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/router_info.py",
line 890, in _process_external_on_delete
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info
self._process_external_gateway(ex_gw_port)
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info File
"/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/router_info.py",
line 799, in _process_external_gateway
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info
self.external_gateway_removed(self.ex_gw_port, interface_name)
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info File
"/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py",
line 513, in external_gateway_removed
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info
self.fip_ns.delete_rtr_2_fip_link(self)
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info File
"/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_fip_ns.py",
line 402, in delete_rtr_2_fip_link
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info
self.local_subnets.release(ri.router_id)
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info File
"/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/item_allocator.py",
line 116, in release
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info
self.pool.add(self.allocations.pop(key))
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info KeyError:
u'e415276a-4f37-4ee0-ba48-12d3909153c7'
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info
Probably a check to make sure if the Key exists before release would
be a good idea.
We might also see if we can reproduce this in the master branch.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1761260/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
