Public bug reported:
Hello,
We have a Mitaka deploy with several projects running, with 3 controller
nodes and 4 compute nodes. After restarting the L3 agent, for one
specific tenant the floating IPs that are attached to an instance are
not actually attached, L3 agent is throwing the following error:
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info [-] L3 agent
failure to setup floating IPs
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info Traceback
(most recent call last):
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 342,
in configure_fip_addresses
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info return
self.process_floating_ip_addresses(interface_name)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 314,
in process_floating_ip_addresses
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info fip,
interface_name, device)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_local_router.py", line
157, in add_floating_ip
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info
self.floating_ip_added_dist(fip, ip_cidr)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_local_router.py", line
88, in floating_ip_added_dist
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info
device.route.add_route(fip_cidr, str(rtr_2_fip.ip))
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 841, in
add_route
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info
self._run_as_root_detect_device_not_found([ip_version], tuple(args))
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 699, in
_run_as_root_detect_device_not_found
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info raise
exceptions.DeviceNotFoundError(device_name=self.name)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info
self.force_reraise()
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info
six.reraise(self.type_, self.value, self.tb)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 694, in
_run_as_root_detect_device_not_found
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info return
self._as_root(*args, **kwargs)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 361, in
_as_root
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info
use_root_namespace=use_root_namespace)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 94, in
_as_root
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info
log_fail_as_error=self.log_fail_as_error)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 103, in
_execute
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info
log_fail_as_error=log_fail_as_error)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 140, in
execute
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info raise
RuntimeError(msg)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info RuntimeError:
Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Network is
unreachable
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info [-] Failed to
process floating IPs.
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info Traceback
(most recent call last):
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 815,
in process_external
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info
fip_statuses = self.configure_fip_addresses(interface_name)
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 347,
in configure_fip_addresses
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info raise
n_exc.FloatingIpSetupException(msg)
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info
FloatingIpSetupException: L3 agent failure to setup floating IPs
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info
As of now, every time we try to attach/detach the floating IP on an
instance the L3 agent shows the same error. Neutron package versions
are:
ii neutron-common 2:8.2.0-0ubuntu1~cloud0
ii neutron-dhcp-agent 2:8.2.0-0ubuntu1~cloud0
ii neutron-l3-agent 2:8.2.0-0ubuntu1~cloud0
ii neutron-metadata-agent 2:8.2.0-0ubuntu1~cloud0
ii neutron-openvswitch-agent 2:8.2.0-0ubuntu1~cloud0
ii neutron-plugin-ml2 2:8.2.0-0ubuntu1~cloud0
OS: Ubuntu 14.04, uname: Linux compute01 3.16.0-77-generic
#99~14.04.1-Ubuntu SMP Tue Jun 28 19:17:10 UTC 2016 x86_64 x86_64 x86_64
GNU/Linux
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1640202
Title:
L3 agent failure to setup floating IPs
Status in neutron:
New
Bug description:
Hello,
We have a Mitaka deploy with several projects running, with 3
controller nodes and 4 compute nodes. After restarting the L3 agent,
for one specific tenant the floating IPs that are attached to an
instance are not actually attached, L3 agent is throwing the following
error:
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info [-] L3 agent
failure to setup floating IPs
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info Traceback
(most recent call last):
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 342,
in configure_fip_addresses
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info return
self.process_floating_ip_addresses(interface_name)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 314,
in process_floating_ip_addresses
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info fip,
interface_name, device)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_local_router.py", line
157, in add_floating_ip
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info
self.floating_ip_added_dist(fip, ip_cidr)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_local_router.py", line
88, in floating_ip_added_dist
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info
device.route.add_route(fip_cidr, str(rtr_2_fip.ip))
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 841, in
add_route
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info
self._run_as_root_detect_device_not_found([ip_version], tuple(args))
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 699, in
_run_as_root_detect_device_not_found
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info raise
exceptions.DeviceNotFoundError(device_name=self.name)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info
self.force_reraise()
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info
six.reraise(self.type_, self.value, self.tb)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 694, in
_run_as_root_detect_device_not_found
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info return
self._as_root(*args, **kwargs)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 361, in
_as_root
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info
use_root_namespace=use_root_namespace)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 94, in
_as_root
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info
log_fail_as_error=self.log_fail_as_error)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 103, in
_execute
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info
log_fail_as_error=log_fail_as_error)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 140, in
execute
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info raise
RuntimeError(msg)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info
RuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers:
Network is unreachable
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info [-] Failed
to process floating IPs.
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info Traceback
(most recent call last):
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 815,
in process_external
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info
fip_statuses = self.configure_fip_addresses(interface_name)
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info File
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 347,
in configure_fip_addresses
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info raise
n_exc.FloatingIpSetupException(msg)
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info
FloatingIpSetupException: L3 agent failure to setup floating IPs
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info
As of now, every time we try to attach/detach the floating IP on an
instance the L3 agent shows the same error. Neutron package versions
are:
ii neutron-common 2:8.2.0-0ubuntu1~cloud0
ii neutron-dhcp-agent 2:8.2.0-0ubuntu1~cloud0
ii neutron-l3-agent 2:8.2.0-0ubuntu1~cloud0
ii neutron-metadata-agent 2:8.2.0-0ubuntu1~cloud0
ii neutron-openvswitch-agent 2:8.2.0-0ubuntu1~cloud0
ii neutron-plugin-ml2 2:8.2.0-0ubuntu1~cloud0
OS: Ubuntu 14.04, uname: Linux compute01 3.16.0-77-generic
#99~14.04.1-Ubuntu SMP Tue Jun 28 19:17:10 UTC 2016 x86_64 x86_64
x86_64 GNU/Linux
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1640202/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
