Hi Sujai, thanks for reporting this. The issue is not yet clear to me.
Can you please provide:
- max_pool_size, retry_interval, max_overflow settings of your first run
- the nova error message of the first and the second run
- the neutron error message of the first and second run
At the moment you provided a neutron message for the first run and a nova
message for the second. So I can't see what changed after updating the
settings. In general it looks like that for deleting 500 instances in parallel
a lot of sql queries are required. It seems like for such a use case you need
to tune your parameters..
But let's compare your error message of the first and second run first.
If they are equal, I tend to set this to invalid
** Also affects: nova
Importance: Undecided
Status: New
** Changed in: nova
Status: New => Incomplete
** Changed in: neutron
Status: New => Incomplete
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1640029
Title:
[stable/newton] Deleting heat stack failed due to error "QueuePool
limit of size 50 overflow 50 reached, connection timed out, timeout
30"
Status in neutron:
Incomplete
Status in OpenStack Compute (nova):
Incomplete
Bug description:
In my CH + stable/newton setup, I brought up 5 heat stacks each having 100
nova instances in the same /16 network.
Deleting those heat stacks failed due to the below error.
"
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions
TimeoutError: QueuePool limit of size 50 overflow 50 reached, connection timed
out, timeout 30
"
Because of this error, out of 500 instances, deletion of about 67 instances
got failed.
With default parameters in neutron.conf, I'm getting the below neutron error.
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource
[req-a0022887-cc01-4f2e-980d-490136524363 admin -] delete failed: No details.
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource Traceback (most
recent call last):
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/opt/stack/neutron/neutron/api/v2/resource.py", line 79, in resource
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource result =
method(request=request, **args)
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/opt/stack/neutron/neutron/api/v2/base.py", line 555, in delete
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource return
self._delete(request, id, **kwargs)
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/opt/stack/neutron/neutron/db/api.py", line 88, in wrapped
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource setattr(e,
'_RETRY_EXCEEDED', True)
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource
self.force_reraise()
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource
six.reraise(self.type_, self.value, self.tb)
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/opt/stack/neutron/neutron/db/api.py", line 84, in wrapped
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource return
f(*args, **kwargs)
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource ectxt.value =
e.inner_exc
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource
self.force_reraise()
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource
six.reraise(self.type_, self.value, self.tb)
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource return
f(*args, **kwargs)
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/opt/stack/neutron/neutron/db/api.py", line 124, in wrapped
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource
traceback.format_exc())
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource
self.force_reraise()
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource
six.reraise(self.type_, self.value, self.tb)
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/opt/stack/neutron/neutron/db/api.py", line 119, in wrapped
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource return
f(*dup_args, **dup_kwargs)
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/opt/stack/neutron/neutron/api/v2/base.py", line 577, in _delete
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource
obj_deleter(request.context, id, **kwargs)
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/opt/stack/vmware-nsx/vmware_nsx/plugins/nsx_v3/plugin.py", line 1814, in
delete_port
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource
self.disassociate_floatingips(context, port_id)
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/opt/stack/vmware-nsx/vmware_nsx/plugins/nsx_v3/plugin.py", line 2804, in
disassociate_floatingips
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource for fip_db in
fip_dbs:
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2761, in
__iter__
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource return
self._execute_and_instances(context)
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2774, in
_execute_and_instances
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource
close_with_result=True)
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2765, in
_connection_from_session
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource **kw)
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 893,
in connection
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource
execution_options=execution_options)
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 900,
in _connection_for_bind
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource conn =
engine.contextual_connect(**kw)
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039,
in contextual_connect
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource
self._wrap_pool_connect(self.pool.connect, None),
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074,
in _wrap_pool_connect
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource return fn()
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in
connect
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource return
_ConnectionFairy._checkout(self)
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in
_checkout
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource fairy =
_ConnectionRecord.checkout(pool)
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in
checkout
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource rec =
pool._do_get()
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1053, in
_do_get
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource (self.size(),
self.overflow(), self._timeout))
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource TimeoutError:
QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10
2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource
After changing the below parameters in /etc/neutron/neutron.conf and
max_pool_size = 50
retry_interval = 10
max_overflow = 50
pool_max_size = 50
pool_max_overflow = 50
pool_timeout = 30
below parameters in nova.conf and restarted the services and re-
executed the testcase.Still deleting heat stack is failing with the
below error
max_pool_size = 50
max_overflow = 50
n-api.log:
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions
[req-db3d6d66-9508-4eb8-be65-964f05ff50f8 admin admin] Unexpected exception in
API method
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions Traceback
(most recent call last):
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/opt/stack/nova/nova/api/openstack/extensions.py", line 338, in wrapped
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return
f(*args, **kwargs)
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/opt/stack/nova/nova/api/openstack/compute/servers.py", line 914, in delete
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions
self._delete(req.environ['nova.context'], req, id)
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/opt/stack/nova/nova/api/openstack/compute/servers.py", line 755, in _delete
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions instance
= self._get_server(context, req, instance_uuid)
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/opt/stack/nova/nova/api/openstack/compute/servers.py", line 382, in
_get_server
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions
expected_attrs=expected_attrs)
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/opt/stack/nova/nova/api/openstack/common.py", line 496, in get_instance
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions
expected_attrs=expected_attrs)
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/opt/stack/nova/nova/compute/api.py", line 2215, in get
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions
expected_attrs)
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/opt/stack/nova/nova/compute/api.py", line 2198, in _get_instance
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions context,
instance_uuid, expected_attrs=expected_attrs)
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
184, in wrapper
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions result =
fn(cls, context, *args, **kwargs)
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/opt/stack/nova/nova/objects/instance.py", line 444, in get_by_uuid
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions
use_slave=use_slave)
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/opt/stack/nova/nova/db/sqlalchemy/api.py", line 225, in wrapper
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return
f(*args, **kwargs)
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/opt/stack/nova/nova/objects/instance.py", line 436, in
_db_instance_get_by_uuid
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions
columns_to_join=columns_to_join)
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/opt/stack/nova/nova/db/api.py", line 725, in instance_get_by_uuid
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return
IMPL.instance_get_by_uuid(context, uuid, columns_to_join)
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/opt/stack/nova/nova/db/sqlalchemy/api.py", line 169, in wrapper
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return
f(*args, **kwargs)
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/opt/stack/nova/nova/db/sqlalchemy/api.py", line 270, in wrapped
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return
f(context, *args, **kwargs)
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1867, in instance_get_by_uuid
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions
columns_to_join=columns_to_join)
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1872, in _instance_get_by_uuid
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions
filter_by(uuid=uuid).\
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2659, in
first
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions ret =
list(self[0:1])
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2457, in
__getitem__
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return
list(res)
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2761, in
__iter__
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return
self._execute_and_instances(context)
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2774, in
_execute_and_instances
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions
close_with_result=True)
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2765, in
_connection_from_session
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions **kw)
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 893,
in connection
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions
execution_options=execution_options)
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 898,
in _connection_for_bind
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions engine,
execution_options)
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 334,
in _connection_for_bind
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions conn =
bind.contextual_connect()
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039,
in contextual_connect
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions
self._wrap_pool_connect(self.pool.connect, None),
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074,
in _wrap_pool_connect
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return
fn()
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in
connect
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return
_ConnectionFairy._checkout(self)
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in
_checkout
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions fairy =
_ConnectionRecord.checkout(pool)
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in
checkout
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions rec =
pool._do_get()
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1053, in
_do_get
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions
(self.size(), self.overflow(), self._timeout))
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions
TimeoutError: QueuePool limit of size 50 overflow 50 reached, connection timed
out, timeout 30
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions
2016-11-03 17:27:34.148 2399 INFO nova.api.openstack.wsgi
[req-db3d6d66-9508-4eb8-be65-964f05ff50f8 admin admin] HTTP exception thrown:
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and
attach the Nova API log if possible.
<class 'sqlalchemy.exc.TimeoutError'>
2016-11-03 17:27:34.148 2399 DEBUG nova.api.openstack.wsgi
[req-db3d6d66-9508-4eb8-be65-964f05ff50f8 admin admin] Returning 500 to user:
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and
attach the Nova API log if possible.
<class 'sqlalchemy.exc.TimeoutError'> __call__
/opt/stack/nova/nova/api/openstack/wsgi.py:1044
Please look into this.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1640029/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
