** Changed in: nova
Status: Fix Committed => Fix Released
** Changed in: nova
Milestone: None => kilo-1
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1365606
Title:
Network deallocation can fail if a network has been deleted
Status in OpenStack Compute (Nova):
Fix Released
Bug description:
The call to get_instance_nw_info fails with an error if a network is
deleted during the deallocation. Networks are not supposed to be able
to be deleted if they have fixed ips in use, but there is a race where
a network can be deleted while an allocation is still in process. This
is make many times worse by the fact that get_instance_nw_info makes
3*networks + 1 (db + rpc) calls. This should be converted to a) get
everything in a single db request, b) handle networks not existing
gracefully.
The traceback from get_nw_info failing can look like:
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher Traceback
(most recent call last):
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133,
in _dispatch_and_reply
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher
incoming.message))
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176,
in _dispatch
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher return
self._do_dispatch(endpoint, method, ctxt, args)
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122,
in _do_dispatch
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher result
= getattr(endpoint, method)(ctxt, **new_args)
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 242, in
deallocate_fixed_ip
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher
address, instance=instance)
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 931, in
deallocate_fixed_ip
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher
instance_uuid)
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 395, in
_do_trigger_security_group_members_refresh_for_instance
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher None,
None)
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/server.py", line 139, in
inner
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher return
func(*args, **kwargs)
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 602, in
get_instance_nw_info
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher
rxtx_factor, host)
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 625, in
build_network_info_model
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher
instance_host)
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 701, in
_get_subnets_from_network
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher
network['project_id'], network['uuid'], vif.uuid)
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/network/nova_ipam_lib.py", line 46, in
get_subnets_by_net_id
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher n =
network_obj.Network.get_by_uuid(context.elevated(), net_id)
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 110, in wrapper
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher args,
kwargs)
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 425, in
object_class_action
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher
objver=objver, args=args, kwargs=kwargs)
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 150, in
call
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher
wait_for_reply=True, timeout=timeout)
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in
_send
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher
timeout=timeout)
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line
412, in send
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher return
self._send(target, ctxt, message, wait_for_reply, timeout)
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line
405, in _send
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher raise
result
2014-09-03 23:45:03.060 10725 TRACE oslo.messaging.rpc.dispatcher
NetworkNotFoundForUUID_Remote: Network could not be found for uuid
3af0a6e6-59f8-4d7e-90ec-b5b866f578f8
or:
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher Traceback
(most recent call last):
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133,
in _dispatch_and_reply
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher
incoming.message))
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176,
in _dispatch
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher return
self._do_dispatch(endpoint, method, ctxt, args)
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122,
in _do_dispatch
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher result
= getattr(endpoint, method)(ctxt, **new_args)
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 242, in
deallocate_fixed_ip
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher
address, instance=instance)
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 931, in
deallocate_fixed_ip
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher
instance_uuid)
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 395, in
_do_trigger_security_group_members_refresh_for_instance
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher None,
None)
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/server.py", line 139, in
inner
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher return
func(*args, **kwargs)
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 598, in
get_instance_nw_info
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher network
= self._get_network_by_id(context, vif.network_id)
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 1445, in
_get_network_by_id
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher
project_only='allow_none')
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 110, in wrapper
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher args,
kwargs)
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 425, in
object_class_action
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher
objver=objver, args=args, kwargs=kwargs)
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 150, in
call
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher
wait_for_reply=True, timeout=timeout)
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in
_send
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher
timeout=timeout)
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line
412, in send
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher return
self._send(target, ctxt, message, wait_for_reply, timeout)
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line
405, in _send
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher raise
result
2014-09-03 23:57:39.153 10725 TRACE oslo.messaging.rpc.dispatcher
NetworkNotFound_Remote: Network 23 could not be found.
depending on where in the call stack the race occurs.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1365606/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
