Public bug reported:
L3 agent can't handle properly the case when it has to remove HA router and for
some reason keepalived's config is already deleted.
It can happen e.g. in case when router is deleted, there is some error during
processing this deletion so agent tries to process it again but then
ha_conf/<router_id> is already removed.
There is error like:
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent [-] Error while deleting router
e41ab1d4-69cf-4dcb-877e-7d57fa90d3c0: FileNotFoundError: [Errno 2] No such file
or directory:
'/opt/stack/data/neutron/ha_confs/e41ab1d4-69cf-4dcb-877e-7d57fa90d3c0'
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent Traceback (most recent call last):
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py",
line 514, in _safe_router_removed
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent self._router_removed(ri, router_id)
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py",
line 551, in _router_removed
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent self.router_info[router_id] = ri
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent File
"/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent self.force_reraise()
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent File
"/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent File "/usr/local/lib/python3.6/dist-packages/six.py",
line 703, in reraise
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent raise value
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py",
line 547, in _router_removed
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent ri.delete()
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent File
"/opt/stack/neutron/neutron/agent/l3/ha_router.py", line 504, in delete
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent self.disable_keepalived()
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent File
"/opt/stack/neutron/neutron/agent/l3/ha_router.py", line 204, in
disable_keepalived
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent shutil.rmtree(conf_dir)
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent File "/usr/lib/python3.6/shutil.py", line 477, in
rmtree
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent onerror(os.lstat, path, sys.exc_info())
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent File "/usr/lib/python3.6/shutil.py", line 475, in
rmtree
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent orig_st = os.lstat(path)
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent FileNotFoundError: [Errno 2] No such file or directory:
'/opt/stack/data/neutron/ha_confs/e41ab1d4-69cf-4dcb-877e-7d57fa90d3c0'
in such case.
** Affects: neutron
Importance: Medium
Assignee: Slawek Kaplonski (slaweq)
Status: In Progress
** Tags: l3-ha
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1892866
Title:
FileNotFoundError not cached during cleaning of HA router
Status in neutron:
In Progress
Bug description:
L3 agent can't handle properly the case when it has to remove HA router and
for some reason keepalived's config is already deleted.
It can happen e.g. in case when router is deleted, there is some error during
processing this deletion so agent tries to process it again but then
ha_conf/<router_id> is already removed.
There is error like:
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent [-] Error while deleting router
e41ab1d4-69cf-4dcb-877e-7d57fa90d3c0: FileNotFoundError: [Errno 2] No such file
or directory:
'/opt/stack/data/neutron/ha_confs/e41ab1d4-69cf-4dcb-877e-7d57fa90d3c0'
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent Traceback (most recent call last):
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py",
line 514, in _safe_router_removed
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent self._router_removed(ri, router_id)
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py",
line 551, in _router_removed
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent self.router_info[router_id] = ri
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent File
"/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent self.force_reraise()
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent File
"/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent File "/usr/local/lib/python3.6/dist-packages/six.py",
line 703, in reraise
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent raise value
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py",
line 547, in _router_removed
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent ri.delete()
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent File
"/opt/stack/neutron/neutron/agent/l3/ha_router.py", line 504, in delete
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent self.disable_keepalived()
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent File
"/opt/stack/neutron/neutron/agent/l3/ha_router.py", line 204, in
disable_keepalived
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent shutil.rmtree(conf_dir)
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent File "/usr/lib/python3.6/shutil.py", line 477, in
rmtree
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent onerror(os.lstat, path, sys.exc_info())
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent File "/usr/lib/python3.6/shutil.py", line 475, in
rmtree
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent orig_st = os.lstat(path)
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR
neutron.agent.l3.agent FileNotFoundError: [Errno 2] No such file or directory:
'/opt/stack/data/neutron/ha_confs/e41ab1d4-69cf-4dcb-877e-7d57fa90d3c0'
in such case.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1892866/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
