Public bug reported: In my CH + stable/newton setup, I brought up 5 heat stacks each having 100 nova instances in the same /16 network. Deleting those heat stacks failed due to the below error.
" 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions TimeoutError: QueuePool limit of size 50 overflow 50 reached, connection timed out, timeout 30 " Because of this error, out of 500 instances, deletion of about 67 instances got failed. With default parameters in neutron.conf, I'm getting the below neutron error. 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource [req-a0022887-cc01-4f2e-980d-490136524363 admin -] delete failed: No details. 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource Traceback (most recent call last): 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/resource.py", line 79, in resource 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource result = method(request=request, **args) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 555, in delete 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource return self._delete(request, id, **kwargs) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/api.py", line 88, in wrapped 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource self.force_reraise() 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/api.py", line 84, in wrapped 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource return f(*args, **kwargs) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource self.force_reraise() 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource return f(*args, **kwargs) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/api.py", line 124, in wrapped 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource traceback.format_exc()) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource self.force_reraise() 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/api.py", line 119, in wrapped 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 577, in _delete 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource obj_deleter(request.context, id, **kwargs) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/opt/stack/vmware-nsx/vmware_nsx/plugins/nsx_v3/plugin.py", line 1814, in delete_port 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource self.disassociate_floatingips(context, port_id) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/opt/stack/vmware-nsx/vmware_nsx/plugins/nsx_v3/plugin.py", line 2804, in disassociate_floatingips 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource for fip_db in fip_dbs: 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2761, in __iter__ 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource return self._execute_and_instances(context) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2774, in _execute_and_instances 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource close_with_result=True) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2765, in _connection_from_session 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource **kw) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 893, in connection 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource execution_options=execution_options) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 900, in _connection_for_bind 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource conn = engine.contextual_connect(**kw) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource self._wrap_pool_connect(self.pool.connect, None), 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource return fn() 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource return _ConnectionFairy._checkout(self) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource fairy = _ConnectionRecord.checkout(pool) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource rec = pool._do_get() 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1053, in _do_get 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource (self.size(), self.overflow(), self._timeout)) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource TimeoutError: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource After changing the below parameters in /etc/neutron/neutron.conf and max_pool_size = 50 retry_interval = 10 max_overflow = 50 pool_max_size = 50 pool_max_overflow = 50 pool_timeout = 30 below parameters in nova.conf and restarted the services and re-executed the testcase.Still deleting heat stack is failing with the below error max_pool_size = 50 max_overflow = 50 n-api.log: 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions [req-db3d6d66-9508-4eb8-be65-964f05ff50f8 admin admin] Unexpected exception in API method 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions Traceback (most recent call last): 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/extensions.py", line 338, in wrapped 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 914, in delete 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions self._delete(req.environ['nova.context'], req, id) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 755, in _delete 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions instance = self._get_server(context, req, instance_uuid) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 382, in _get_server 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions expected_attrs=expected_attrs) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/common.py", line 496, in get_instance 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions expected_attrs=expected_attrs) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 2215, in get 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions expected_attrs) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 2198, in _get_instance 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions context, instance_uuid, expected_attrs=expected_attrs) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions result = fn(cls, context, *args, **kwargs) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/objects/instance.py", line 444, in get_by_uuid 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions use_slave=use_slave) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 225, in wrapper 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/objects/instance.py", line 436, in _db_instance_get_by_uuid 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions columns_to_join=columns_to_join) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/db/api.py", line 725, in instance_get_by_uuid 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return IMPL.instance_get_by_uuid(context, uuid, columns_to_join) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 169, in wrapper 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 270, in wrapped 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return f(context, *args, **kwargs) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1867, in instance_get_by_uuid 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions columns_to_join=columns_to_join) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1872, in _instance_get_by_uuid 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions filter_by(uuid=uuid).\ 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2659, in first 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions ret = list(self[0:1]) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2457, in __getitem__ 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return list(res) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2761, in __iter__ 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return self._execute_and_instances(context) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2774, in _execute_and_instances 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions close_with_result=True) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2765, in _connection_from_session 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions **kw) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 893, in connection 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions execution_options=execution_options) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 898, in _connection_for_bind 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions engine, execution_options) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions conn = bind.contextual_connect() 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions self._wrap_pool_connect(self.pool.connect, None), 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return fn() 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return _ConnectionFairy._checkout(self) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions fairy = _ConnectionRecord.checkout(pool) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions rec = pool._do_get() 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1053, in _do_get 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions (self.size(), self.overflow(), self._timeout)) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions TimeoutError: QueuePool limit of size 50 overflow 50 reached, connection timed out, timeout 30 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions 2016-11-03 17:27:34.148 2399 INFO nova.api.openstack.wsgi [req-db3d6d66-9508-4eb8-be65-964f05ff50f8 admin admin] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'sqlalchemy.exc.TimeoutError'> 2016-11-03 17:27:34.148 2399 DEBUG nova.api.openstack.wsgi [req-db3d6d66-9508-4eb8-be65-964f05ff50f8 admin admin] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'sqlalchemy.exc.TimeoutError'> __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1044 Please look into this. ** Affects: neutron Importance: Undecided Status: New ** Summary changed: - [stable/newton] Deleting heat stack failed due to neutron error "QueuePool limit of size 50 overflow 50 reached, connection timed out, timeout 30" + [stable/newton] Deleting heat stack failed due to error "QueuePool limit of size 50 overflow 50 reached, connection timed out, timeout 30" -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1640029 Title: [stable/newton] Deleting heat stack failed due to error "QueuePool limit of size 50 overflow 50 reached, connection timed out, timeout 30" Status in neutron: New Bug description: In my CH + stable/newton setup, I brought up 5 heat stacks each having 100 nova instances in the same /16 network. Deleting those heat stacks failed due to the below error. " 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions TimeoutError: QueuePool limit of size 50 overflow 50 reached, connection timed out, timeout 30 " Because of this error, out of 500 instances, deletion of about 67 instances got failed. With default parameters in neutron.conf, I'm getting the below neutron error. 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource [req-a0022887-cc01-4f2e-980d-490136524363 admin -] delete failed: No details. 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource Traceback (most recent call last): 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/resource.py", line 79, in resource 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource result = method(request=request, **args) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 555, in delete 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource return self._delete(request, id, **kwargs) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/api.py", line 88, in wrapped 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource self.force_reraise() 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/api.py", line 84, in wrapped 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource return f(*args, **kwargs) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource self.force_reraise() 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource return f(*args, **kwargs) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/api.py", line 124, in wrapped 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource traceback.format_exc()) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource self.force_reraise() 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/api.py", line 119, in wrapped 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 577, in _delete 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource obj_deleter(request.context, id, **kwargs) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/opt/stack/vmware-nsx/vmware_nsx/plugins/nsx_v3/plugin.py", line 1814, in delete_port 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource self.disassociate_floatingips(context, port_id) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/opt/stack/vmware-nsx/vmware_nsx/plugins/nsx_v3/plugin.py", line 2804, in disassociate_floatingips 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource for fip_db in fip_dbs: 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2761, in __iter__ 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource return self._execute_and_instances(context) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2774, in _execute_and_instances 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource close_with_result=True) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2765, in _connection_from_session 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource **kw) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 893, in connection 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource execution_options=execution_options) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 900, in _connection_for_bind 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource conn = engine.contextual_connect(**kw) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource self._wrap_pool_connect(self.pool.connect, None), 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource return fn() 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource return _ConnectionFairy._checkout(self) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource fairy = _ConnectionRecord.checkout(pool) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource rec = pool._do_get() 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1053, in _do_get 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource (self.size(), self.overflow(), self._timeout)) 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource TimeoutError: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10 2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource After changing the below parameters in /etc/neutron/neutron.conf and max_pool_size = 50 retry_interval = 10 max_overflow = 50 pool_max_size = 50 pool_max_overflow = 50 pool_timeout = 30 below parameters in nova.conf and restarted the services and re- executed the testcase.Still deleting heat stack is failing with the below error max_pool_size = 50 max_overflow = 50 n-api.log: 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions [req-db3d6d66-9508-4eb8-be65-964f05ff50f8 admin admin] Unexpected exception in API method 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions Traceback (most recent call last): 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/extensions.py", line 338, in wrapped 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 914, in delete 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions self._delete(req.environ['nova.context'], req, id) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 755, in _delete 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions instance = self._get_server(context, req, instance_uuid) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 382, in _get_server 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions expected_attrs=expected_attrs) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/common.py", line 496, in get_instance 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions expected_attrs=expected_attrs) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 2215, in get 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions expected_attrs) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 2198, in _get_instance 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions context, instance_uuid, expected_attrs=expected_attrs) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions result = fn(cls, context, *args, **kwargs) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/objects/instance.py", line 444, in get_by_uuid 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions use_slave=use_slave) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 225, in wrapper 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/objects/instance.py", line 436, in _db_instance_get_by_uuid 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions columns_to_join=columns_to_join) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/db/api.py", line 725, in instance_get_by_uuid 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return IMPL.instance_get_by_uuid(context, uuid, columns_to_join) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 169, in wrapper 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 270, in wrapped 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return f(context, *args, **kwargs) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1867, in instance_get_by_uuid 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions columns_to_join=columns_to_join) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1872, in _instance_get_by_uuid 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions filter_by(uuid=uuid).\ 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2659, in first 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions ret = list(self[0:1]) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2457, in __getitem__ 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return list(res) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2761, in __iter__ 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return self._execute_and_instances(context) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2774, in _execute_and_instances 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions close_with_result=True) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2765, in _connection_from_session 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions **kw) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 893, in connection 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions execution_options=execution_options) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 898, in _connection_for_bind 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions engine, execution_options) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions conn = bind.contextual_connect() 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions self._wrap_pool_connect(self.pool.connect, None), 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return fn() 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return _ConnectionFairy._checkout(self) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions fairy = _ConnectionRecord.checkout(pool) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions rec = pool._do_get() 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1053, in _do_get 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions (self.size(), self.overflow(), self._timeout)) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions TimeoutError: QueuePool limit of size 50 overflow 50 reached, connection timed out, timeout 30 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions 2016-11-03 17:27:34.148 2399 INFO nova.api.openstack.wsgi [req-db3d6d66-9508-4eb8-be65-964f05ff50f8 admin admin] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'sqlalchemy.exc.TimeoutError'> 2016-11-03 17:27:34.148 2399 DEBUG nova.api.openstack.wsgi [req-db3d6d66-9508-4eb8-be65-964f05ff50f8 admin admin] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'sqlalchemy.exc.TimeoutError'> __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1044 Please look into this. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1640029/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
