** Changed in: nova
Status: Fix Committed => Fix Released
** Changed in: nova
Milestone: None => liberty-2
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1474550
Title:
network allocation randomly failing with InstanceUpdateConflict after
compare and swap was merged
Status in OpenStack Compute (nova):
Fix Released
Bug description:
Seeing this quite a bit:
http://logs.openstack.org/85/197185/10/gate/gate-tempest-dsvm-
cells/7ef1949/logs/screen-n-cpu.txt.gz?level=TRACE#_2015-07-14_16_39_06_862
2015-07-14 16:39:06.862 ERROR nova.compute.manager
[req-94a9749e-2aed-433c-8dea-f672b61b7fcf
tempest-DeleteServersTestJSON-193542712
tempest-DeleteServersTestJSON-409686918] [instance:
f143b952-9edd-4971-86fe-227d61294a76] Failed to allocate network(s)
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] Traceback (most recent call last):
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] File
"/opt/stack/new/nova/nova/compute/manager.py", line 2057, in _build_resources
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] requested_networks, security_groups)
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] File
"/opt/stack/new/nova/nova/compute/manager.py", line 1572, in
_build_networks_for_instance
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] requested_networks, macs,
security_groups, dhcp_options)
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] File
"/opt/stack/new/nova/nova/compute/manager.py", line 1606, in _allocate_network
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76]
instance.save(expected_task_state=[None])
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] File
"/opt/stack/new/nova/nova/objects/base.py", line 100, in wrapper
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] self._context, self, fn.__name__,
args, kwargs)
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] File
"/opt/stack/new/nova/nova/conductor/rpcapi.py", line 266, in object_action
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] objmethod=objmethod, args=args,
kwargs=kwargs)
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line
158, in call
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] retry=self.retry)
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90,
in _send
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] timeout=timeout, retry=retry)
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
line 361, in send
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] retry=retry)
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
line 352, in _send
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] raise result
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] InstanceUpdateConflict_Remote: Conflict
updating instance f143b952-9edd-4971-86fe-227d61294a76
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] Traceback (most recent call last):
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76]
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] File
"/opt/stack/new/nova/nova/conductor/manager.py", line 437, in _object_dispatch
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] return getattr(target, method)(*args,
**kwargs)
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76]
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] File
"/opt/stack/new/nova/nova/objects/base.py", line 116, in wrapper
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] return fn(self, *args, **kwargs)
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76]
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] File
"/opt/stack/new/nova/nova/objects/instance.py", line 862, in save
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76]
columns_to_join=_expected_cols(expected_attrs))
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76]
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] File
"/opt/stack/new/nova/nova/db/api.py", line 751, in
instance_update_and_get_original
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] columns_to_join=columns_to_join)
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76]
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] File
"/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 231, in wrapper
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] return f(*args, **kwargs)
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76]
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] File
"/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 146, in wrapper
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] ectxt.value = e.inner_exc
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76]
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 119, in
__exit__
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] six.reraise(self.type_, self.value,
self.tb)
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76]
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] File
"/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 136, in wrapper
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] return f(*args, **kwargs)
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76]
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] File
"/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 2452, in
instance_update_and_get_original
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] original=instance_ref))
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76]
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] File
"/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 2581, in _instance_update
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] raise
exception.InstanceUpdateConflict(instance_uuid=instance_uuid)
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76]
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76] InstanceUpdateConflict: Conflict updating
instance f143b952-9edd-4971-86fe-227d61294a76
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76]
2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance:
f143b952-9edd-4971-86fe-227d61294a76]
Ever since this was merged yesterday:
https://review.openstack.org/#/c/141115/
http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiSW5zdGFuY2VVcGRhdGVDb25mbGljdF9SZW1vdGVcIiBBTkQgdGFnczpcInNjcmVlbi1uLWNwdS50eHRcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQzNjkwNTIwMTM4NH0=
101 hits in 24 hours, check and gate, all failures.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1474550/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
