[Yahoo-eng-team] [Bug 1573073] [NEW] When router has no ports _process_updated_router fails because the namespace does not exist

Saverio Proto Thu, 21 Apr 2016 07:36:06 -0700

Public bug reported:

Happens in Kilo. Cannot test on other releases.


Steps to reproduce, create a router and dont add any port.
The command 'neutron router-port-list' should return an empty set.
As soon as you create the router, the namespace qrouter-<uuid> is present on 
the network node, but after a while it is purged.
As soon as the namespace is not there anymore check the log file vpn-agent.log 
for the stacktrace.

When a router has no ports the namespace is deleted from the network
node. However this brakes the router updates and the file vpn-agent.log
is flooded with this traces:


2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Traceback 
(most recent call last):
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File 
"/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 343, in call
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     return 
func(*args, **kwargs)
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 628, 
in process
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     
self._process_internal_ports()
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 404, 
in _process_internal_ports
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     
existing_devices = self._get_existing_devices()
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 328, 
in _get_existing_devices
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     ip_devs = 
ip_wrapper.get_devices(exclude_loopback=True)
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 102, in 
get_devices
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     
log_fail_as_error=self.log_fail_as_error
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in 
execute
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     raise 
RuntimeError(m)
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info RuntimeError:
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Command: 
['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 
'exec', 'qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762', 'find', 
'/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f ']
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Exit code: 1
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Stdin:
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Stdout:
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Stderr: Cannot 
open network namespace "qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762": No such 
file or directory
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info
2016-04-21 16:22:17.774 23382 ERROR neutron.agent.l3.agent [-] Failed to 
process compatible router '8fc0f640-35bb-4d0b-bbbd-80c22be0e762'
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Traceback (most 
recent call last):
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 467, in 
_process_router_update
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     
self._process_router_if_compatible(router)
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 414, in 
_process_router_if_compatible
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     
self._process_updated_router(router)
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 428, in 
_process_updated_router
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     ri.process(self)
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 346, in call
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     self.logger(e)
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     
six.reraise(self.type_, self.value, self.tb)
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 343, in call
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     return 
func(*args, **kwargs)
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 628, 
in process
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     
self._process_internal_ports()
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 404, 
in _process_internal_ports
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     existing_devices 
= self._get_existing_devices()
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 328, 
in _get_existing_devices
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     ip_devs = 
ip_wrapper.get_devices(exclude_loopback=True)
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 102, in 
get_devices
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     
log_fail_as_error=self.log_fail_as_error
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in 
execute
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     raise 
RuntimeError(m)
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent RuntimeError:
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Command: ['sudo', 
'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 
'qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762', 'find', '/sys/class/net', 
'-maxdepth', '1', '-type', 'l', '-printf', '%f ']
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Exit code: 1
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Stdin:
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Stdout:
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Stderr: Cannot open 
network namespace "qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762": No such file 
or directory
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1573073

Title:
  When router has no ports _process_updated_router fails because the
  namespace does not exist

Status in neutron:
  New

Bug description:
  Happens in Kilo. Cannot test on other releases.

  Steps to reproduce, create a router and dont add any port.
  The command 'neutron router-port-list' should return an empty set.
  As soon as you create the router, the namespace qrouter-<uuid> is present on 
the network node, but after a while it is purged.
  As soon as the namespace is not there anymore check the log file 
vpn-agent.log for the stacktrace.

  When a router has no ports the namespace is deleted from the network
  node. However this brakes the router updates and the file vpn-
  agent.log is flooded with this traces:

  
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Traceback 
(most recent call last):
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File 
"/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 343, in call
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     return 
func(*args, **kwargs)
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 628, 
in process
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     
self._process_internal_ports()
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 404, 
in _process_internal_ports
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     
existing_devices = self._get_existing_devices()
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 328, 
in _get_existing_devices
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     ip_devs 
= ip_wrapper.get_devices(exclude_loopback=True)
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 102, in 
get_devices
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     
log_fail_as_error=self.log_fail_as_error
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in 
execute
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     raise 
RuntimeError(m)
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info RuntimeError:
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Command: 
['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 
'exec', 'qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762', 'find', 
'/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f ']
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Exit code: 1
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Stdin:
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Stdout:
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Stderr: 
Cannot open network namespace "qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762": 
No such file or directory
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info
  2016-04-21 16:22:17.774 23382 ERROR neutron.agent.l3.agent [-] Failed to 
process compatible router '8fc0f640-35bb-4d0b-bbbd-80c22be0e762'
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Traceback (most 
recent call last):
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 467, in 
_process_router_update
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     
self._process_router_if_compatible(router)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 414, in 
_process_router_if_compatible
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     
self._process_updated_router(router)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 428, in 
_process_updated_router
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     
ri.process(self)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 346, in call
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     self.logger(e)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     
six.reraise(self.type_, self.value, self.tb)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 343, in call
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     return 
func(*args, **kwargs)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 628, 
in process
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     
self._process_internal_ports()
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 404, 
in _process_internal_ports
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     
existing_devices = self._get_existing_devices()
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 328, 
in _get_existing_devices
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     ip_devs = 
ip_wrapper.get_devices(exclude_loopback=True)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 102, in 
get_devices
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     
log_fail_as_error=self.log_fail_as_error
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in 
execute
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     raise 
RuntimeError(m)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent RuntimeError:
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Command: ['sudo', 
'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 
'qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762', 'find', '/sys/class/net', 
'-maxdepth', '1', '-type', 'l', '-printf', '%f ']
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Exit code: 1
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Stdin:
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Stdout:
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Stderr: Cannot 
open network namespace "qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762": No such 
file or directory
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1573073/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to     : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1573073] [NEW] When router has no ports _process_updated_router fails because the namespace does not exist

Reply via email to