*** This bug is a duplicate of bug 1182131 ***
https://bugs.launchpad.net/bugs/1182131
I believe this was probably addressed with:
commit 6aa368b99249d01f8fd7183c15d11986ad6a6fb7
Author: Dan Smith <[email protected]>
Date: Thu Jul 3 08:09:39 2014 -0700
Avoid re-adding iptables rules for instances that have disappeared
The remove_filters_for_instance() method fails silently if the
instance's chain is gone (i.e. it's been deleted). If this
happens while we're refreshing security group rules, we will
not notice this case and re-add stale rules for an old instance,
breaking our firewall for new instances.
This adds a quick check after we've captured the lock to see if
the associated chain exists, and bails if it doesn't.
Change-Id: Ic75988939f82de49735d85fe99a9eecd4baf45c9
Related-bug: #1182131
** Changed in: nova
Status: Incomplete => Fix Committed
** This bug has been marked a duplicate of bug 1182131
nova-compute: instance created in self-referencing secgroup produces KeyError
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1202449
Title:
Race condition with spawn/delete of instance sharing same source group
security rule
Status in OpenStack Compute (Nova):
Fix Committed
Bug description:
Getting the below error when launching and instance at the same time as
deleting another instance.
These instances share a security group rule that has the same source group
(hope that makes sense)
Instance e5fc8a20-384a-4976-890d-54631962c9e2 was deleted at about the same
time as 98fd38d1-82cb-449e-92f9-1936d1e2628c was created.
98fd38d1-82cb-449e-92f9-1936d1e2628c failed when trying to build its
iptables because e5fc8a20-384a-4976-890d-54631962c9e2 no longer existed.
2013-07-17 15:30:35.754 ERROR nova.compute.manager
[req-6f82cdb7-db20-4f17-a344-edf6da70a871 XXXX XXXXX] [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] Instance failed to spawn
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] Traceback (most recent call last):
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1119, in spawn
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] block_device_info)
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1528, in
spawn
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] block_device_info)
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2443, in
_create_domain_and_network
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c]
self.firewall_driver.prepare_instance_filter(instance, network_info)
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] File
"/usr/lib/python2.7/dist-packages/nova/virt/firewall.py", line 193, in
prepare_instance_filter
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] ipv4_rules, ipv6_rules =
self.instance_rules(instance, network_info)
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] File
"/usr/lib/python2.7/dist-packages/nova/virt/firewall.py", line 424, in
instance_rules
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] conductor_api=capi)
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] File
"/usr/lib/python2.7/dist-packages/nova/network/api.py", line 102, in wrapped
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] return func(self, context, *args,
**kwargs)
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] File
"/usr/lib/python2.7/dist-packages/nova/network/api.py", line 375, in
get_instance_nw_info
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] result =
self._get_instance_nw_info(context, instance)
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] File
"/usr/lib/python2.7/dist-packages/nova/network/api.py", line 392, in
_get_instance_nw_info
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] nw_info =
self.network_rpcapi.get_instance_nw_info(context, **args)
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] File
"/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 207, in
get_instance_nw_info
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] project_id=project_id), version='1.9')
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/proxy.py", line 80,
in call
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] return rpc.call(context,
self._get_topic(topic), msg, timeout)
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/_init__.py", line
140, in call
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] return _get_impl().call(CONF, context,
topic, msg, timeout)
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
line 800, in call
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] rpc_amqp.get_connection_pool(conf,
Connection))
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 612,
in call
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] rv = list(rv)
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 561,
in iter
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] raise result
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] InstanceNotFound_Remote: Instance
e5fc8a20-384a-4976-890d-54631962c9e2 could not be found.
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] Traceback (most recent call last):
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c]
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 430,
in _process_data
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] rval = self.proxy.dispatch(ctxt, version,
method, **args)
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c]
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
line 133, in dispatch
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] return getattr(proxyobj, method)(ctxt,
**kwargs)
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c]
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] File
"/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 576, in
get_instance_nw_info
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] instance_uuid)
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c]
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] File
"/usr/lib/python2.7/dist-packages/nova/db/api.py", line 549, in
virtual_interface_get_by_instance
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] return
IMPL.virtual_interface_get_by_instance(context, instance_id)
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c]
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] File
"/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 114, in
wrapper
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] return f(*args, **kwargs)
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c]
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] File
"/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 126, in
wrapper
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] instance_get_by_uuid(context,
instance_uuid)
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c]
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] File
"/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 114, in
wrapper
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] return f(*args, **kwargs)
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c]
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] File
"/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1607, in
instance_get_by_uuid
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] return _instance_get_by_uuid(context,
uuid)
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c]
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] File
"/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 114, in
wrapper
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] return f(*args, **kwargs)
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c]
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] File
"/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1617, in
_instance_get_by_uuid
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] raise
exception.InstanceNotFound(instance_id=uuid)
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c]
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c] InstanceNotFound: Instance
e5fc8a20-384a-4976-890d-54631962c9e2 could not be found.
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c]
2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance:
98fd38d1-82cb-449e-92f9-1936d1e2628c]
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1202449/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
