** Changed in: neutron
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1657981
Title:
FloatingIPs not reachable after restart of compute node (DVR)
Status in neutron:
Invalid
Bug description:
I am running OpenStack Newton on Ubuntu 16.04 using DVR. When I
restart a compute node, the FloatingIPs of those vms running on this
node are unreachable. A manual restart of the service
"neutron-l3-agent" or "neutron-vpn-agent" running in on node solves
the issue.
I think there must be a race condition at startup.
I get the following error in the neutron-vpn-agent.log:
2017-01-20 07:04:52.379 2541 INFO neutron.common.config [-] Logging enabled!
2017-01-20 07:04:52.379 2541 INFO neutron.common.config [-]
/usr/bin/neutron-vpn-agent version 9.0.0
2017-01-20 07:04:52.380 2541 WARNING stevedore.named [-] Could not load
neutron.agent.linux.interface.OVSInterfaceDriver
2017-01-20 07:04:53.112 2541 WARNING stevedore.named
[req-a9e10331-51ab-4c67-bfdd-0f6296510594 - - - - -] Could not load
neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas.IptablesFwaasDriver
2017-01-20 07:04:53.127 2541 INFO neutron.agent.agent_extensions_manager
[req-a9e10331-51ab-4c67-bfdd-0f6296510594 - - - - -] Loaded agent extensions:
['fwaas']
2017-01-20 07:04:53.128 2541 INFO neutron.agent.agent_extensions_manager
[req-a9e10331-51ab-4c67-bfdd-0f6296510594 - - - - -] Initializing agent
extension 'fwaas'
2017-01-20 07:04:53.163 2541 WARNING oslo_config.cfg
[req-bdd95fb9-bcd7-473e-a350-3bd8d6be8758 - - - - -] Option
"external_network_bridge" from group "DEFAULT" is deprecated for removal. Its
value may be silently ignored in the future.
2017-01-20 07:04:53.165 2541 WARNING stevedore.named
[req-bdd95fb9-bcd7-473e-a350-3bd8d6be8758 - - - - -] Could not load
neutron_vpnaas.services.vpn.device_drivers.strongswan_ipsec.StrongSwanDriver
2017-01-20 07:04:53.236 2541 INFO eventlet.wsgi.server [-] (2541) wsgi
starting up on http:/var/lib/neutron/keepalived-state-change
2017-01-20 07:04:53.261 2541 INFO neutron.agent.l3.agent [-] Agent has just
been revived. Doing a full sync.
2017-01-20 07:04:53.373 2541 INFO neutron.agent.l3.agent [-] L3 agent started
2017-01-20 07:05:22.832 2541 ERROR neutron.agent.linux.utils [-] Exit code:
1; Stdin: ; Stdout: ; Stderr: Cannot find device "fg-67afaa06-bb"
2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info [-] Exit
code: 1; Stdin: ; Stdout: ; Stderr: Cannot find device "fg-67afaa06-bb"
2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info Traceback
(most recent call last):
2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 239, in call
2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info return
func(*args, **kwargs)
2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 1062,
in process
2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info
self.process_external(agent)
2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_local_router.py", line
515, in process_external
2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info
self.create_dvr_fip_interfaces(ex_gw_port)
2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_local_router.py", line
546, in create_dvr_fip_interfaces
2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info
self.fip_ns.update_gateway_port(fip_agent_port)
2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_fip_ns.py", line 239, in
update_gateway_port
2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info
ipd.route.add_gateway(gw_ip)
2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 702, in
add_gateway
2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info
self._as_root([ip_version], tuple(args))
2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 373, in
_as_root
2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info
use_root_namespace=use_root_namespace)
2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 95, in
_as_root
2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info
log_fail_as_error=self.log_fail_as_error)
2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 104, in
_execute
2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info
log_fail_as_error=log_fail_as_error)
2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 138, in
execute
2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info raise
RuntimeError(msg)
2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info RuntimeError:
Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot find device "fg-67afaa06-bb"
2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info
2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent [-] Failed to
process compatible router: 552f5cfc-27c4-4a59-842a-b3e471a94ca1
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent Traceback (most
recent call last):
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 508, in
_process_router_update
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent
self._process_router_if_compatible(router)
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 443, in
_process_router_if_compatible
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent
self._process_added_router(router)
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 451, in
_process_added_router
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent ri.process(self)
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_local_router.py", line
580, in process
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent
super(DvrLocalRouter, self).process(agent)
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_router_base.py", line
31, in process
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent
super(DvrRouterBase, self).process(agent)
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 242, in call
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent self.logger(e)
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent
self.force_reraise()
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent
six.reraise(self.type_, self.value, self.tb)
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 239, in call
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent return
func(*args, **kwargs)
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 1062,
in process
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent
self.process_external(agent)
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_local_router.py", line
515, in process_external
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent
self.create_dvr_fip_interfaces(ex_gw_port)
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_local_router.py", line
546, in create_dvr_fip_interfaces
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent
self.fip_ns.update_gateway_port(fip_agent_port)
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_fip_ns.py", line 239, in
update_gateway_port
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent
ipd.route.add_gateway(gw_ip)
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 702, in
add_gateway
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent
self._as_root([ip_version], tuple(args))
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 373, in
_as_root
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent
use_root_namespace=use_root_namespace)
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 95, in
_as_root
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent
log_fail_as_error=self.log_fail_as_error)
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 104, in
_execute
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent
log_fail_as_error=log_fail_as_error)
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 138, in
execute
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent raise
RuntimeError(msg)
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent RuntimeError: Exit
code: 1; Stdin: ; Stdout: ; Stderr: Cannot find device "fg-67afaa06-bb"
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent
2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent
2017-01-20 07:05:22.978 2541 ERROR neutron.agent.linux.utils [-] Exit code:
2; Stdin: ; Stdout: ; Stderr: arping: Device fg-67afaa06-bb not available.
2017-01-20 07:05:22.980 2541 ERROR neutron.agent.linux.ip_lib [-] Failed
sending gratuitous ARP to 10.30.216.131 on fg-67afaa06-bb in namespace
fip-5d873120-a1f8-4b9d-83fb-96f20fdfa9bd
2017-01-20 07:05:22.980 2541 ERROR neutron.agent.linux.ip_lib Traceback (most
recent call last):
2017-01-20 07:05:22.980 2541 ERROR neutron.agent.linux.ip_lib File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 1007, in
_arping
2017-01-20 07:05:22.980 2541 ERROR neutron.agent.linux.ip_lib
ip_wrapper.netns.execute(arping_cmd, check_exit_code=True)
2017-01-20 07:05:22.980 2541 ERROR neutron.agent.linux.ip_lib File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 894, in
execute
2017-01-20 07:05:22.980 2541 ERROR neutron.agent.linux.ip_lib
log_fail_as_error=log_fail_as_error, **kwargs)
2017-01-20 07:05:22.980 2541 ERROR neutron.agent.linux.ip_lib File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 138, in
execute
2017-01-20 07:05:22.980 2541 ERROR neutron.agent.linux.ip_lib raise
RuntimeError(msg)
2017-01-20 07:05:22.980 2541 ERROR neutron.agent.linux.ip_lib RuntimeError:
Exit code: 2; Stdin: ; Stdout: ; Stderr: arping: Device fg-67afaa06-bb not
available.
2017-01-20 07:05:22.980 2541 ERROR neutron.agent.linux.ip_lib
2017-01-20 07:05:22.980 2541 ERROR neutron.agent.linux.ip_lib
After restart:
2017-01-20 07:15:10.579 5069 INFO neutron.common.config [-] Logging enabled!
2017-01-20 07:15:10.580 5069 INFO neutron.common.config [-]
/usr/bin/neutron-vpn-agent version 9.0.0
2017-01-20 07:15:10.581 5069 WARNING stevedore.named [-] Could not load
neutron.agent.linux.interface.OVSInterfaceDriver
2017-01-20 07:15:10.810 5069 WARNING stevedore.named
[req-99af950d-c09f-4f63-bfb1-5368d01f7a76 - - - - -] Could not load
neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas.IptablesFwaasDriver
2017-01-20 07:15:10.812 5069 INFO neutron.agent.agent_extensions_manager
[req-99af950d-c09f-4f63-bfb1-5368d01f7a76 - - - - -] Loaded agent extensions:
['fwaas']
2017-01-20 07:15:10.813 5069 INFO neutron.agent.agent_extensions_manager
[req-99af950d-c09f-4f63-bfb1-5368d01f7a76 - - - - -] Initializing agent
extension 'fwaas'
2017-01-20 07:15:10.820 5069 WARNING oslo_config.cfg
[req-016c1195-a7a2-4b94-b260-66c655ad4b2d - - - - -] Option
"external_network_bridge" from group "DEFAULT" is deprecated for removal. Its
value may be silently ignored in the future.
2017-01-20 07:15:10.821 5069 WARNING stevedore.named
[req-016c1195-a7a2-4b94-b260-66c655ad4b2d - - - - -] Could not load
neutron_vpnaas.services.vpn.device_drivers.strongswan_ipsec.StrongSwanDriver
2017-01-20 07:15:10.857 5069 INFO eventlet.wsgi.server [-] (5069) wsgi
starting up on http:/var/lib/neutron/keepalived-state-change
2017-01-20 07:15:10.950 5069 INFO neutron.agent.l3.agent [-] L3 agent started
Since now, the floating ips become reachable after a short period of
time.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1657981/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
