Public bug reported: Seeing this quite a bit:
http://logs.openstack.org/85/197185/10/gate/gate-tempest-dsvm- cells/7ef1949/logs/screen-n-cpu.txt.gz?level=TRACE#_2015-07-14_16_39_06_862 2015-07-14 16:39:06.862 ERROR nova.compute.manager [req-94a9749e-2aed-433c-8dea-f672b61b7fcf tempest-DeleteServersTestJSON-193542712 tempest-DeleteServersTestJSON-409686918] [instance: f143b952-9edd-4971-86fe-227d61294a76] Failed to allocate network(s) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] Traceback (most recent call last): 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/opt/stack/new/nova/nova/compute/manager.py", line 2057, in _build_resources 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] requested_networks, security_groups) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/opt/stack/new/nova/nova/compute/manager.py", line 1572, in _build_networks_for_instance 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] requested_networks, macs, security_groups, dhcp_options) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/opt/stack/new/nova/nova/compute/manager.py", line 1606, in _allocate_network 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] instance.save(expected_task_state=[None]) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/opt/stack/new/nova/nova/objects/base.py", line 100, in wrapper 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] self._context, self, fn.__name__, args, kwargs) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/opt/stack/new/nova/nova/conductor/rpcapi.py", line 266, in object_action 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] objmethod=objmethod, args=args, kwargs=kwargs) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] retry=self.retry) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] timeout=timeout, retry=retry) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 361, in send 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] retry=retry) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 352, in _send 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] raise result 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] InstanceUpdateConflict_Remote: Conflict updating instance f143b952-9edd-4971-86fe-227d61294a76 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] Traceback (most recent call last): 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/opt/stack/new/nova/nova/conductor/manager.py", line 437, in _object_dispatch 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] return getattr(target, method)(*args, **kwargs) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/opt/stack/new/nova/nova/objects/base.py", line 116, in wrapper 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] return fn(self, *args, **kwargs) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/opt/stack/new/nova/nova/objects/instance.py", line 862, in save 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] columns_to_join=_expected_cols(expected_attrs)) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/opt/stack/new/nova/nova/db/api.py", line 751, in instance_update_and_get_original 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] columns_to_join=columns_to_join) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 231, in wrapper 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] return f(*args, **kwargs) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 146, in wrapper 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] ectxt.value = e.inner_exc 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 119, in __exit__ 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] six.reraise(self.type_, self.value, self.tb) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 136, in wrapper 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] return f(*args, **kwargs) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 2452, in instance_update_and_get_original 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] original=instance_ref)) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 2581, in _instance_update 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] raise exception.InstanceUpdateConflict(instance_uuid=instance_uuid) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] InstanceUpdateConflict: Conflict updating instance f143b952-9edd-4971-86fe-227d61294a76 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] Ever since this was merged yesterday: https://review.openstack.org/#/c/141115/ http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiSW5zdGFuY2VVcGRhdGVDb25mbGljdF9SZW1vdGVcIiBBTkQgdGFnczpcInNjcmVlbi1uLWNwdS50eHRcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQzNjkwNTIwMTM4NH0= 101 hits in 24 hours, check and gate, all failures. ** Affects: nova Importance: High Assignee: Matt Riedemann (mriedem) Status: In Progress ** Tags: db ** Changed in: nova Status: New => Confirmed ** Changed in: nova Importance: Undecided => High -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1474550 Title: network allocation randomly failing with InstanceUpdateConflict after compare and swap was merged Status in OpenStack Compute (nova): In Progress Bug description: Seeing this quite a bit: http://logs.openstack.org/85/197185/10/gate/gate-tempest-dsvm- cells/7ef1949/logs/screen-n-cpu.txt.gz?level=TRACE#_2015-07-14_16_39_06_862 2015-07-14 16:39:06.862 ERROR nova.compute.manager [req-94a9749e-2aed-433c-8dea-f672b61b7fcf tempest-DeleteServersTestJSON-193542712 tempest-DeleteServersTestJSON-409686918] [instance: f143b952-9edd-4971-86fe-227d61294a76] Failed to allocate network(s) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] Traceback (most recent call last): 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/opt/stack/new/nova/nova/compute/manager.py", line 2057, in _build_resources 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] requested_networks, security_groups) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/opt/stack/new/nova/nova/compute/manager.py", line 1572, in _build_networks_for_instance 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] requested_networks, macs, security_groups, dhcp_options) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/opt/stack/new/nova/nova/compute/manager.py", line 1606, in _allocate_network 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] instance.save(expected_task_state=[None]) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/opt/stack/new/nova/nova/objects/base.py", line 100, in wrapper 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] self._context, self, fn.__name__, args, kwargs) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/opt/stack/new/nova/nova/conductor/rpcapi.py", line 266, in object_action 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] objmethod=objmethod, args=args, kwargs=kwargs) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] retry=self.retry) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] timeout=timeout, retry=retry) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 361, in send 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] retry=retry) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 352, in _send 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] raise result 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] InstanceUpdateConflict_Remote: Conflict updating instance f143b952-9edd-4971-86fe-227d61294a76 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] Traceback (most recent call last): 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/opt/stack/new/nova/nova/conductor/manager.py", line 437, in _object_dispatch 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] return getattr(target, method)(*args, **kwargs) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/opt/stack/new/nova/nova/objects/base.py", line 116, in wrapper 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] return fn(self, *args, **kwargs) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/opt/stack/new/nova/nova/objects/instance.py", line 862, in save 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] columns_to_join=_expected_cols(expected_attrs)) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/opt/stack/new/nova/nova/db/api.py", line 751, in instance_update_and_get_original 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] columns_to_join=columns_to_join) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 231, in wrapper 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] return f(*args, **kwargs) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 146, in wrapper 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] ectxt.value = e.inner_exc 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 119, in __exit__ 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] six.reraise(self.type_, self.value, self.tb) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 136, in wrapper 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] return f(*args, **kwargs) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 2452, in instance_update_and_get_original 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] original=instance_ref)) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 2581, in _instance_update 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] raise exception.InstanceUpdateConflict(instance_uuid=instance_uuid) 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] InstanceUpdateConflict: Conflict updating instance f143b952-9edd-4971-86fe-227d61294a76 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] Ever since this was merged yesterday: https://review.openstack.org/#/c/141115/ http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiSW5zdGFuY2VVcGRhdGVDb25mbGljdF9SZW1vdGVcIiBBTkQgdGFnczpcInNjcmVlbi1uLWNwdS50eHRcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQzNjkwNTIwMTM4NH0= 101 hits in 24 hours, check and gate, all failures. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1474550/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
