Public bug reported: When tried to run below rally scenario with concurrency 50, seeing issue with keystone. Can someone take a look? NOTE: Things will work fine with concurrency 10.
1. Create tenant, create network. 2. Create T1 router and set external network as gateway 3. Add network created in step 1 to T1 router 4. Launch instance(on kvm) in the private network and assign FIP. Ping FIP Setup: Single controller(32vCPU, 48GB RAM) 3 Network Nodes 100 ESX computes and 100 KVM computes Rally reports and logs attached to bug. Logs: 2016-03-01 01:26:34.699 DEBUG oslo_concurrency.lockutils [req-409c8595-d093-4cfe-8b98-b49d2c2accad ctx_rally_d6ed151ea67e4b78930c39c406fa64ed_user_0 ctx_rally_9526f233-a1b9-446b-beb6-d14dc678ff37_tenant_10] Releasing semaphore "refresh_cache-8c324106-c6dd-4b90-876d-e3cc33adfebf" from (pid=26585) lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:225 2016-03-01 01:26:34.704 ERROR nova.compute.manager [req-409c8595-d093-4cfe-8b98-b49d2c2accad ctx_rally_d6ed151ea67e4b78930c39c406fa64ed_user_0 ctx_rally_9526f233-a1b9-446b-beb6-d14dc678ff37_tenant_10] [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] Instance failed to spawn 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] Traceback (most recent call last): 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/compute/manager.py", line 2190, in _build_resources 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] yield resources 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/compute/manager.py", line 2036, in _build_and_run_instance 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] block_device_info=block_device_info) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2758, in spawn 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] admin_pass=admin_password) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3251, in _create_image 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] content=files, extra_md=extra_md, network_info=network_info) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/api/metadata/base.py", line 160, in __init__ 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] self.network_metadata = netutils.get_network_metadata(network_info) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/virt/netutils.py", line 185, in get_network_metadata 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] if not network_info: 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/network/model.py", line 526, in __len__ 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return self._sync_wrapper(fn, *args, **kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/network/model.py", line 513, in _sync_wrapper 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] self.wait() 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/network/model.py", line 545, in wait 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] self[:] = self._gt.wait() 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 175, in wait 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return self._exit_event.wait() 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return hubs.get_hub().switch() 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return self.greenlet.switch() 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] result = function(*args, **kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/utils.py", line 1160, in context_wrapper 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return func(*args, **kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/compute/manager.py", line 1583, in _allocate_network_async 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] six.reraise(*exc_info) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/compute/manager.py", line 1566, in _allocate_network_async 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] bind_host_id=bind_host_id) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 668, in allocate_for_instance 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] update_cells=True) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/network/base_api.py", line 246, in get_instance_nw_info 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] result = self._get_instance_nw_info(context, instance, **kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 911, in _get_instance_nw_info 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] preexisting_port_ids) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1713, in _build_network_info_model 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] current_neutron_port) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1565, in _nw_info_get_ips 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] client, fixed_ip['ip_address'], port['id']) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1496, in _get_floating_ips_by_fixed_and_port 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] port_id=port) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1467, in _safe_get_floating_ips 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return client.list_floatingips(**kwargs)['floatingips'] 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 100, in with_params 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] ret = self.function(instance, *args, **kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 699, in list_floatingips 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] **_params) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 280, in list 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] for r in self._pagination(collection, path, **params): 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 293, in _pagination 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] res = self.get(path, params=params) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 266, in get 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] headers=headers, params=params) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 243, in retry_request 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] headers=headers, params=params) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 195, in do_request 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] resp, replybody = self.httpclient.do_request(action, method, body=body) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 306, in do_request 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return self.request(url, method, **kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 294, in request 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] resp = super(SessionClient, self).request(*args, **kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 98, in request 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return self.session.request(url, method, **kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return func(*args, **kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 370, in request 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] auth_headers = self.get_auth_headers(auth) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 624, in get_auth_headers 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return auth.get_headers(self, **kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 84, in get_headers 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] token = self.get_token(session) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 90, in get_token 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return self.get_access(session).auth_token 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 136, in get_access 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] self.auth_ref = self.get_auth_ref(session) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 181, in get_auth_ref 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return self._plugin.get_auth_ref(session, **kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py", line 167, in get_auth_ref 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] authenticated=False, log=False, **rkwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 572, in post 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return self.request(url, 'POST', **kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return func(*args, **kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 452, in request 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] resp = send(**kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 496, in _send_request 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] raise exceptions.ConnectFailure(msg) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] ConnectFailure: Unable to establish connection to http://172.17.4.231:35357/v3/auth/tokens 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] 2016-03-01 01:26:34.707 INFO nova.compute.manager [req-409c8595-d093-4cfe-8b98-b49d2c2accad ctx_rally_d6ed151ea67e4b78930c39c406fa64ed_user_0 ctx_rally_9526f233-a1b9-446b-beb6-d14dc678ff37_tenant_10] [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] Terminating instance ** Affects: keystone Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1553149 Title: Instance in ERROR state due to ConnectFailure with keystone Status in OpenStack Identity (keystone): New Bug description: When tried to run below rally scenario with concurrency 50, seeing issue with keystone. Can someone take a look? NOTE: Things will work fine with concurrency 10. 1. Create tenant, create network. 2. Create T1 router and set external network as gateway 3. Add network created in step 1 to T1 router 4. Launch instance(on kvm) in the private network and assign FIP. Ping FIP Setup: Single controller(32vCPU, 48GB RAM) 3 Network Nodes 100 ESX computes and 100 KVM computes Rally reports and logs attached to bug. Logs: 2016-03-01 01:26:34.699 DEBUG oslo_concurrency.lockutils [req-409c8595-d093-4cfe-8b98-b49d2c2accad ctx_rally_d6ed151ea67e4b78930c39c406fa64ed_user_0 ctx_rally_9526f233-a1b9-446b-beb6-d14dc678ff37_tenant_10] Releasing semaphore "refresh_cache-8c324106-c6dd-4b90-876d-e3cc33adfebf" from (pid=26585) lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:225 2016-03-01 01:26:34.704 ERROR nova.compute.manager [req-409c8595-d093-4cfe-8b98-b49d2c2accad ctx_rally_d6ed151ea67e4b78930c39c406fa64ed_user_0 ctx_rally_9526f233-a1b9-446b-beb6-d14dc678ff37_tenant_10] [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] Instance failed to spawn 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] Traceback (most recent call last): 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/compute/manager.py", line 2190, in _build_resources 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] yield resources 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/compute/manager.py", line 2036, in _build_and_run_instance 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] block_device_info=block_device_info) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2758, in spawn 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] admin_pass=admin_password) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3251, in _create_image 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] content=files, extra_md=extra_md, network_info=network_info) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/api/metadata/base.py", line 160, in __init__ 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] self.network_metadata = netutils.get_network_metadata(network_info) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/virt/netutils.py", line 185, in get_network_metadata 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] if not network_info: 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/network/model.py", line 526, in __len__ 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return self._sync_wrapper(fn, *args, **kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/network/model.py", line 513, in _sync_wrapper 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] self.wait() 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/network/model.py", line 545, in wait 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] self[:] = self._gt.wait() 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 175, in wait 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return self._exit_event.wait() 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return hubs.get_hub().switch() 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return self.greenlet.switch() 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] result = function(*args, **kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/utils.py", line 1160, in context_wrapper 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return func(*args, **kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/compute/manager.py", line 1583, in _allocate_network_async 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] six.reraise(*exc_info) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/compute/manager.py", line 1566, in _allocate_network_async 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] bind_host_id=bind_host_id) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 668, in allocate_for_instance 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] update_cells=True) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/network/base_api.py", line 246, in get_instance_nw_info 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] result = self._get_instance_nw_info(context, instance, **kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 911, in _get_instance_nw_info 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] preexisting_port_ids) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1713, in _build_network_info_model 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] current_neutron_port) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1565, in _nw_info_get_ips 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] client, fixed_ip['ip_address'], port['id']) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1496, in _get_floating_ips_by_fixed_and_port 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] port_id=port) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1467, in _safe_get_floating_ips 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return client.list_floatingips(**kwargs)['floatingips'] 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 100, in with_params 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] ret = self.function(instance, *args, **kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 699, in list_floatingips 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] **_params) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 280, in list 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] for r in self._pagination(collection, path, **params): 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 293, in _pagination 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] res = self.get(path, params=params) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 266, in get 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] headers=headers, params=params) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 243, in retry_request 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] headers=headers, params=params) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 195, in do_request 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] resp, replybody = self.httpclient.do_request(action, method, body=body) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 306, in do_request 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return self.request(url, method, **kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 294, in request 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] resp = super(SessionClient, self).request(*args, **kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 98, in request 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return self.session.request(url, method, **kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return func(*args, **kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 370, in request 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] auth_headers = self.get_auth_headers(auth) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 624, in get_auth_headers 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return auth.get_headers(self, **kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 84, in get_headers 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] token = self.get_token(session) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 90, in get_token 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return self.get_access(session).auth_token 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 136, in get_access 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] self.auth_ref = self.get_auth_ref(session) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 181, in get_auth_ref 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return self._plugin.get_auth_ref(session, **kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py", line 167, in get_auth_ref 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] authenticated=False, log=False, **rkwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 572, in post 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return self.request(url, 'POST', **kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] return func(*args, **kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 452, in request 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] resp = send(**kwargs) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 496, in _send_request 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] raise exceptions.ConnectFailure(msg) 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] ConnectFailure: Unable to establish connection to http://172.17.4.231:35357/v3/auth/tokens 2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] 2016-03-01 01:26:34.707 INFO nova.compute.manager [req-409c8595-d093-4cfe-8b98-b49d2c2accad ctx_rally_d6ed151ea67e4b78930c39c406fa64ed_user_0 ctx_rally_9526f233-a1b9-446b-beb6-d14dc678ff37_tenant_10] [instance: 8c324106-c6dd-4b90-876d-e3cc33adfebf] Terminating instance To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1553149/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
