** Also affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1863982
Title:
Upgrade from Rocky to Stein, router namespace disappear
Status in kolla-ansible:
Confirmed
Status in neutron:
New
Bug description:
Upgrade All-in-one from Rocky to Stein.
Upgrading finished but the router namespace disappears.
============================================================
Before:
ip netns list
qrouter-79658dd5-e3b4-4b13-a361-16d696ed1d1c (id: 1)
qdhcp-4a183162-64f5-49f9-a615-7c0fd63cf2a8 (id: 0)
After:
ip netns list
============================================================
After about 1 minutes, dhcp ns has appeared and no error on dhcp-agent,
but qrouter ns is still missing, until manually restart the docker container
l3-agent.
l3-agent error after upgrade:
2020-02-20 02:57:07.306 12 INFO neutron.common.config [-] Logging enabled!
2020-02-20 02:57:07.308 12 INFO neutron.common.config [-]
/var/lib/kolla/venv/bin/neutron-l3-agent version 14.0.4
2020-02-20 02:57:08.616 12 INFO neutron.agent.l3.agent
[req-95654890-dab3-4106-b56d-c2685fb96f29 - - - - -] Agent HA routers count 0
2020-02-20 02:57:08.619 12 INFO neutron.agent.agent_extensions_manager
[req-95654890-dab3-4106-b56d-c2685fb96f29 - - - - -] Loaded agent extensions: []
2020-02-20 02:57:08.657 12 INFO eventlet.wsgi.server [-] (12) wsgi starting
up on http:/var/lib/neutron/keepalived-state-change
2020-02-20 02:57:08.710 12 INFO neutron.agent.l3.agent [-] L3 agent started
2020-02-20 02:57:10.716 12 INFO oslo.privsep.daemon
[req-681aad3f-ae14-4315-b96d-5e95225cdf92 - - - - -] Running privsep helper:
['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper',
'--privsep_context', 'neutron.privileged.default', '--privsep_sock_path',
'/tmp/tmpg8Ihqa/privsep.sock']
2020-02-20 02:57:11.750 12 INFO oslo.privsep.daemon
[req-681aad3f-ae14-4315-b96d-5e95225cdf92 - - - - -] Spawned new privsep daemon
via rootwrap
2020-02-20 02:57:11.614 29 INFO oslo.privsep.daemon [-] privsep daemon
starting
2020-02-20 02:57:11.622 29 INFO oslo.privsep.daemon [-] privsep process
running with uid/gid: 0/0
2020-02-20 02:57:11.627 29 INFO oslo.privsep.daemon [-] privsep process
running with capabilities (eff/prm/inh):
CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN/CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN/none
2020-02-20 02:57:11.628 29 INFO oslo.privsep.daemon [-] privsep daemon
running as pid 29
2020-02-20 02:57:14.449 12 INFO neutron.agent.l3.agent [-] Starting router
update for 79658dd5-e3b4-4b13-a361-16d696ed1d1c, action 3, priority 2,
update_id 49908db7-8a8c-410f-84a7-9e95a3dede16. Wait time elapsed: 0.000
2020-02-20 02:57:24.160 12 ERROR neutron.agent.linux.utils [-] Exit code: 4;
Stdin: # Generated by iptables_manager
2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info
self.process_floating_ip_address_scope_rules()
2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info
self.gen.next()
2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info File
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py",
line 438, in defer_apply
2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info raise
l3_exc.IpTablesApplyException(msg)
2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info
IpTablesApplyException: Failure applying iptables rules
2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent [-] Failed to process
compatible router: 79658dd5-e3b4-4b13-a361-16d696ed1d1c:
IpTablesApplyException: Failure applying iptables rules
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent Traceback (most
recent call last):
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py",
line 723, in _process_routers_if_compatible
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent
self._process_router_if_compatible(router)
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py",
line 567, in _process_router_if_compatible
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent
self._process_added_router(router)
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py",
line 575, in _process_added_router
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent ri.process()
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/common/utils.py", line
161, in call
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent self.logger(e)
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line
220, in __exit__
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent
self.force_reraise()
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line
196, in force_reraise
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent
six.reraise(self.type_, self.value, self.tb)
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/common/utils.py", line
158, in call
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent return
func(*args, **kwargs)
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py",
line 1189, in process
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent
self.process_address_scope()
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py",
line 1152, in process_address_scope
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent
self.process_floating_ip_address_scope_rules()
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent self.gen.next()
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py",
line 438, in defer_apply
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent raise
l3_exc.IpTablesApplyException(msg)
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent
IpTablesApplyException: Failure applying iptables rules
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent
2020-02-20 02:57:26.391 12 INFO neutron.agent.l3.agent [-] Starting router
update for 79658dd5-e3b4-4b13-a361-16d696ed1d1c, action 3, priority 2,
update_id 49908db7-8a8c-410f-84a7-9e95a3dede16. Wait time elapsed: 11.942
2020-02-20 02:57:27.878 12 INFO neutron.agent.linux.interface [-] Device
qg-a456d6e2-1d already exists
2020-02-20 02:57:32.276 12 ERROR neutron.agent.linux.utils [-] Exit code: 4;
Stdin: # Generated by iptables_manager
To manage notifications about this bug go to:
https://bugs.launchpad.net/kolla-ansible/+bug/1863982/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
