Apache will refuse connections that it cannot assign to threads once
MaxClients is exhausted, and if you're only running 10 threads, then I'm
also guessing that your MaxClients is set to be less than the number of
concurrent connections you're throwing at it.
I'm closing this because this is just an Apache tuning issue.
** Changed in: keystone
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1553149
Title:
Instance in ERROR state due to ConnectFailure with keystone
Status in OpenStack Identity (keystone):
Invalid
Bug description:
When tried to run below rally scenario with concurrency 50, seeing issue with
keystone. Can someone take a look?
NOTE: Things will work fine with concurrency 10.
1. Create tenant, create network.
2. Create T1 router and set external network as gateway
3. Add network created in step 1 to T1 router
4. Launch instance(on kvm) in the private network and assign FIP. Ping FIP
Setup:
Single controller(32vCPU, 48GB RAM)
3 Network Nodes
100 ESX computes and 100 KVM computes
Rally reports and logs attached to bug.
Logs:
2016-03-01 01:26:34.699 DEBUG oslo_concurrency.lockutils
[req-409c8595-d093-4cfe-8b98-b49d2c2accad
ctx_rally_d6ed151ea67e4b78930c39c406fa64ed_user_0
ctx_rally_9526f233-a1b9-446b-beb6-d14dc678ff37_tenant_10] Releasing semaphore
"refresh_cache-8c324106-c6dd-4b90-876d-e3cc33adfebf" from (pid=26585) lock
/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:225
2016-03-01 01:26:34.704 ERROR nova.compute.manager
[req-409c8595-d093-4cfe-8b98-b49d2c2accad
ctx_rally_d6ed151ea67e4b78930c39c406fa64ed_user_0
ctx_rally_9526f233-a1b9-446b-beb6-d14dc678ff37_tenant_10] [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] Instance failed to spawn
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] Traceback (most recent call last):
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/opt/stack/nova/nova/compute/manager.py", line 2190, in _build_resources
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] yield resources
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/opt/stack/nova/nova/compute/manager.py", line 2036, in _build_and_run_instance
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] block_device_info=block_device_info)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 2758, in spawn
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] admin_pass=admin_password)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 3251, in _create_image
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] content=files, extra_md=extra_md,
network_info=network_info)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/opt/stack/nova/nova/api/metadata/base.py", line 160, in __init__
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] self.network_metadata =
netutils.get_network_metadata(network_info)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/opt/stack/nova/nova/virt/netutils.py", line 185, in get_network_metadata
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] if not network_info:
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/opt/stack/nova/nova/network/model.py", line 526, in __len__
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] return self._sync_wrapper(fn, *args,
**kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/opt/stack/nova/nova/network/model.py", line 513, in _sync_wrapper
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] self.wait()
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/opt/stack/nova/nova/network/model.py", line 545, in wait
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] self[:] = self._gt.wait()
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 175, in
wait
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] return self._exit_event.wait()
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] return hubs.get_hub().switch()
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in
switch
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] return self.greenlet.switch()
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in
main
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] result = function(*args, **kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File "/opt/stack/nova/nova/utils.py",
line 1160, in context_wrapper
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] return func(*args, **kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/opt/stack/nova/nova/compute/manager.py", line 1583, in _allocate_network_async
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] six.reraise(*exc_info)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/opt/stack/nova/nova/compute/manager.py", line 1566, in _allocate_network_async
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] bind_host_id=bind_host_id)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/opt/stack/nova/nova/network/neutronv2/api.py", line 668, in
allocate_for_instance
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] update_cells=True)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/opt/stack/nova/nova/network/base_api.py", line 246, in get_instance_nw_info
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] result =
self._get_instance_nw_info(context, instance, **kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/opt/stack/nova/nova/network/neutronv2/api.py", line 911, in
_get_instance_nw_info
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] preexisting_port_ids)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/opt/stack/nova/nova/network/neutronv2/api.py", line 1713, in
_build_network_info_model
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] current_neutron_port)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/opt/stack/nova/nova/network/neutronv2/api.py", line 1565, in _nw_info_get_ips
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] client, fixed_ip['ip_address'],
port['id'])
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/opt/stack/nova/nova/network/neutronv2/api.py", line 1496, in
_get_floating_ips_by_fixed_and_port
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] port_id=port)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/opt/stack/nova/nova/network/neutronv2/api.py", line 1467, in
_safe_get_floating_ips
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] return
client.list_floatingips(**kwargs)['floatingips']
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
100, in with_params
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] ret = self.function(instance, *args,
**kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
699, in list_floatingips
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] **_params)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
280, in list
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] for r in self._pagination(collection,
path, **params):
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
293, in _pagination
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] res = self.get(path, params=params)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
266, in get
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] headers=headers, params=params)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
243, in retry_request
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] headers=headers, params=params)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
195, in do_request
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] resp, replybody =
self.httpclient.do_request(action, method, body=body)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 306, in
do_request
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] return self.request(url, method,
**kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 294, in
request
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] resp = super(SessionClient,
self).request(*args, **kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 98, in
request
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] return self.session.request(url,
method, **kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 94, in
inner
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] return func(*args, **kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 370, in
request
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] auth_headers =
self.get_auth_headers(auth)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 624, in
get_auth_headers
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] return auth.get_headers(self,
**kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 84, in
get_headers
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] token = self.get_token(session)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line
90, in get_token
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] return
self.get_access(session).auth_token
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line
136, in get_access
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] self.auth_ref =
self.get_auth_ref(session)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py",
line 181, in get_auth_ref
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] return
self._plugin.get_auth_ref(session, **kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py",
line 167, in get_auth_ref
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] authenticated=False, log=False,
**rkwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 572, in
post
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] return self.request(url, 'POST',
**kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 94, in
inner
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] return func(*args, **kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 452, in
request
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] resp = send(**kwargs)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 496, in
_send_request
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] raise exceptions.ConnectFailure(msg)
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] ConnectFailure: Unable to establish
connection to http://172.17.4.231:35357/v3/auth/tokens
2016-03-01 01:26:34.704 TRACE nova.compute.manager [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf]
2016-03-01 01:26:34.707 INFO nova.compute.manager
[req-409c8595-d093-4cfe-8b98-b49d2c2accad
ctx_rally_d6ed151ea67e4b78930c39c406fa64ed_user_0
ctx_rally_9526f233-a1b9-446b-beb6-d14dc678ff37_tenant_10] [instance:
8c324106-c6dd-4b90-876d-e3cc33adfebf] Terminating instance
To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1553149/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
