** Also affects: neutron/kilo Importance: Undecided Status: New ** Changed in: neutron/kilo Status: New => Fix Committed
** Changed in: neutron/kilo Milestone: None => 2015.1.3 -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1497309 Title: l3-agent unable to parse output from ip netns list (iproute2 >= 4.0) Status in neutron: Fix Released Status in neutron kilo series: Fix Committed Status in neutron package in Ubuntu: Fix Released Bug description: When run through sudo, ip netns has some extra output on Ubuntu wily: $ sudo ip netns qdhcp-35fc068a-750d-4add-b1d2-af392dbd8790 (id: 1) qrouter-49c6d7b1-8399-4944-81ad-093b6e786db0 (id: 0) and from l3-agent: 2015-09-18 14:15:47.889 26554 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is 59f744cafcab474baee68232c4cf70e9 _send /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:392 2015-09-18 14:15:47.895 26554 DEBUG neutron.agent.l3.agent [-] Starting _process_routers_loop _process_routers_loop /usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py:509 2015-09-18 14:15:47.896 26554 DEBUG oslo_service.loopingcall [-] Fixed interval looping call 'neutron.agent.l3.agent.L3NATAgentWithStateReport._report_state' sleeping for 29.93 seconds _run_loop /usr/lib/python2.7/dist-packages/oslo_service/loopingcall.py:121 2015-09-18 14:15:47.916 26554 DEBUG neutron.agent.l3.agent [-] Starting periodic_sync_routers_task - fullsync:True periodic_sync_routers_task /usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py:521 2015-09-18 14:15:47.953 26554 DEBUG neutron.agent.linux.utils [-] Command: ['ip', 'netns', 'list'] Exit code: 0 execute /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:151 2015-09-18 14:15:47.954 26554 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is d04be6386ef7495ebeb3cb656fb330a8 _send /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:392 2015-09-18 14:15:48.268 26554 DEBUG neutron.agent.l3.agent [-] Processing :[{u'status': u'ACTIVE', u'_interfaces': [{u'status': u'DOWN', u'subnets': [{u'ipv6_ra_mode': None, u'cidr': u'192.168.21.0/24', u'gateway_ip': u'192.168.21.1', u'id': u'242acaef-22b7-4044-ab1f-788bd31ad1da', u'subnetpool_id': None}], u'binding:host_id': u'juju-devel3-machine-12', u'name': u'', u'allowed_address_pairs': [], u'admin_state_up': True, u'network_id': u'35fc068a-750d-4add-b1d2-af392dbd8790', u'dns_name': u'', u'extra_dhcp_opts': [], u'mac_address': u'fa:16:3e:3f:4a:90', u'binding:vif_details': {u'port_filter': True, u'ovs_hybrid_plug': True}, u'binding:vif_type': u'ovs', u'device_owner': u'network:router_interface', u'tenant_id': u'85d6051d040347e5bbd689348405faf0', u'extra_subnets': [], u'binding:profile': {}, u'binding:vnic_type': u'normal', u'fixed_ips': [{u'subnet_id': u'242acaef-22b7-4044-ab1f-788bd31ad1da', u'prefixlen': 24, u'ip_address': u'192.168.21.1'}], u'id': u'bfc7b6e2-29a7-428b-b7fd-0675e9bf5df8', u'security_groups': [], u'device_id': u'49c6d7b1-8399-4944-81ad-093b6e786db0'}], u'enable_snat': True, u'ha_vr_id': 0, u'gw_port_host': None, u'gw_port_id': u'7cca3db9-5502-43be-b193-59d523e3c81b', u'admin_state_up': True, u'tenant_id': u'85d6051d040347e5bbd689348405faf0', u'gw_port': {u'status': u'DOWN', u'subnets': [{u'ipv6_ra_mode': None, u'cidr': u'10.5.0.0/16', u'gateway_ip': u'10.5.0.1', u'id': u'431e736d-04d1-4817-b3c1-c9579b4b51f0', u'subnetpool_id': None}], u'binding:host_id': u'juju-devel3-machine-12', u'name': u'', u'allowed_address_pairs': [], u'admin_state_up': True, u'network_id': u'00ef84fe-880a-45c6-ae3d-967089b161ef', u'dns_name': u'', u'extra_dhcp_opts': [], u'mac_address': u'fa:16:3e:6f:24:28', u'binding:vif_details': {u'port_filter': True, u'ovs_hybrid_plug': True}, u'binding:vif_type': u'ovs', u'device_owner': u'network:router_gateway', u'tenant_id': u'', u'extra_subnets': [], u'binding:profile': {}, u'binding:vnic_type': u'normal', u'fixed_ips': [{u'subnet_id': u'431e736d-04d1-4817-b3c1-c9579b4b51f0', u'prefixlen': 16, u'ip_address': u'10.5.150.0'}], u'id': u'7cca3db9-5502-43be-b193-59d523e3c81b', u'security_groups': [], u'device_id': u'49c6d7b1-8399-4944-81ad-093b6e786db0'}, u'distributed': False, u'_snat_router_interfaces': [], u'_floatingip_agent_interfaces': [], u'_floatingips': [{u'router_id': u'49c6d7b1-8399-4944-81ad-093b6e786db0', u'status': u'DOWN', u'tenant_id': u'85d6051d040347e5bbd689348405faf0', u'floating_network_id': u'00ef84fe-880a-45c6-ae3d-967089b161ef', u'fixed_ip_address': u'192.168.21.3', u'floating_ip_address': u'10.5.150.1', u'port_id': u'a5d45770-98ea-4a2b-b839-5274e783abca', u'id': u'7067f627-91ea-4917-ab31-48570d3c397b'}], u'routes': [], u'external_gateway_info': {u'network_id': u'00ef84fe-880a-45c6-ae3d-967089b161ef', u'enable_snat': True, u'external_fixed_ips': [{u'subnet_id': u'431e736d-04d1-4817-b3c1-c9579b4b51f0', u'ip_address': u'10.5.150.0'}]}, u'ha': False, u'id': u'49c6d7b1-8399-4944-81ad-093b6e786db0', u'name': u'provider-router'}] fetch_and_sync_all_routers /usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py:555 2015-09-18 14:15:48.272 26554 DEBUG neutron.agent.l3.agent [-] periodic_sync_routers_task successfully completed fetch_and_sync_all_routers /usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py:570 2015-09-18 14:15:48.273 26554 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/external/pids/49c6d7b1-8399-4944-81ad-093b6e786db0 (id: 0).pid get_value_from_file /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:240 2015-09-18 14:15:48.274 26554 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/external/pids/49c6d7b1-8399-4944-81ad-093b6e786db0 (id: 0).pid get_value_from_file /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:240 2015-09-18 14:15:48.276 26554 DEBUG neutron.agent.linux.external_process [-] No process started for 49c6d7b1-8399-4944-81ad-093b6e786db0 (id: 0) disable /usr/lib/python2.7/dist-packages/neutron/agent/linux/external_process.py:118 2015-09-18 14:15:48.287 26554 DEBUG neutron.agent.l3.agent [-] Starting router update for 49c6d7b1-8399-4944-81ad-093b6e786db0, action None, priority 1 _process_router_update /usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py:456 2015-09-18 14:15:48.326 26554 DEBUG neutron.agent.linux.utils [-] Command: ['ip', '-o', 'link', 'show', 'br-ex'] Exit code: 0 execute /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:151 2015-09-18 14:15:48.329 26554 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is 1502ca45d5e844e3937f095d9af27875 _send /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:392 2015-09-18 14:15:48.359 26554 ERROR neutron.agent.linux.utils [-] Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-49c6d7b1-8399-4944-81ad-093b6e786db0 (id: 0)', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f '] Exit code: 1 Stdin: Stdout: Stderr: Cannot open network namespace "qrouter-49c6d7b1-8399-4944-81ad-093b6e786db0 (id: 0)": No such file or directory 2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager [-] Failed to destroy stale namespace <neutron.agent.l3.namespaces.RouterNamespace object at 0x7f6d668b7110> 2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager Traceback (most recent call last): 2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/namespace_manager.py", line 143, in _cleanup 2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager ns.delete() 2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/namespaces.py", line 101, in delete 2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager for d in ns_ip.get_devices(exclude_loopback=True): 2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 112, in get_devices 2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager log_fail_as_error=self.log_fail_as_error 2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 154, in execute 2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager raise RuntimeError(m) 2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager RuntimeError: 2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-49c6d7b1-8399-4944-81ad-093b6e786db0 (id: 0)', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f '] 2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager Exit code: 1 2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager Stdin: 2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager Stdout: 2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager Stderr: Cannot open network namespace "qrouter-49c6d7b1-8399-4944-81ad-093b6e786db0 (id: 0)": No such file or directory 2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager 2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager ProblemType: Bug DistroRelease: Ubuntu 15.10 Package: neutron-l3-agent 2:7.0.0~b3-0ubuntu2 ProcVersionSignature: Ubuntu 4.2.0-7.7-generic 4.2.0 Uname: Linux 4.2.0-7-generic x86_64 ApportVersion: 2.18.1-0ubuntu1 Architecture: amd64 Date: Fri Sep 18 14:16:35 2015 JournalErrors: -- Logs begin at Fri 2015-09-18 14:01:29 UTC, end at Fri 2015-09-18 14:16:43 UTC. -- Sep 18 14:06:25 hostname kernel: conntrack: generic helper won't handle protocol 47. Please consider loading the specific helper module. PackageArchitecture: all SourcePackage: neutron UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1497309/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp