[Bug 1384108] Re: Exception during message handling: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10
** Changed in: neutron (Ubuntu) Importance: Medium => Low ** Changed in: neutron (Ubuntu) Status: Confirmed => Incomplete -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1384108 Title: Exception during message handling: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1384108/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384108] Re: Exception during message handling: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10
** Changed in: neutron Importance: High => Medium -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1384108 Title: Exception during message handling: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1384108/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384108] Re: Exception during message handling: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10
** Changed in: neutron (Ubuntu) Importance: Undecided => Medium -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1384108 Title: Exception during message handling: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1384108/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384108] Re: Exception during message handling: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10
If this is happening on say a neutron API server, shouldn't an lsof against that pid show a pool-sized number of connections/file descriptors? Taking that a bit further, if lsof does not show a pool- sized number of connections, doesn't that suggest the pool code is getting a bit confused as to its inventory? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1384108 Title: Exception during message handling: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1384108/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384108] Re: Exception during message handling: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: neutron (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1384108 Title: Exception during message handling: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1384108/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384108] Re: Exception during message handling: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10
We saw the same error too:
2014-12-08 11:08:33.315 47268 INFO neutron.wsgi [-] (47268) accepted
('10.23.70.28', 48899)
2014-12-08 11:08:33.315 47268 INFO neutron.wsgi [-] 10.23.70.28 - -
[08/Dec/2014 11:08:33] "GET / HTTP/1.0" 200 262 0.000255
2014-12-08 11:08:33.985 47264 ERROR neutron.api.v2.resource
[req-98012a5a-2a0d-4052-a30f-59d0e99c5e39 None] index failed
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource Traceback (most
recent call last):
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/api/v2/resource.py",
line 87, in resource
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource result =
method(request=request, **args)
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/api/v2/base.py",
line 308, in index
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource return
self._items(request, True, parent_id)
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/api/v2/base.py",
line 242, in _items
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource obj_list =
obj_getter(request.context, **kwargs)
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py",
line 1419, in get_ports
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource items =
[self._make_port_dict(c, fields) for c in query]
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py",
line 2438, in __iter__
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource return
self._execute_and_instances(context)
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py",
line 2451, in _execute_and_instances
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource
close_with_result=True)
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py",
line 2442, in _connection_from_session
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource **kw)
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
line 854, in connection
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource
close_with_result=close_with_result)
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
line 860, in _connection_for_bind
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource return
engine.contextual_connect(**kwargs)
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1798, in contextual_connect
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource
self.pool.connect(),
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/pool.py",
line 338, in connect
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource return
_ConnectionFairy._checkout(self)
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/pool.py",
line 644, in _checkout
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource fairy =
_ConnectionRecord.checkout(pool)
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/pool.py",
line 440, in checkout
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource rec =
pool._do_get()
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/pool.py",
line 959, in _do_get
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource (self.size(),
self.overflow(), self._timeout))
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource TimeoutError:
QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10
2014-12-08 11:08:33.985 47264 TRACE neutron.api.v2.resource
With the error, we cannot create/delete any network. Once we reboot
neutron-server, then everything is fine.
For our case, we do have some other Exception before this:
2014-12-08 11:05:40.284 47019 DEBUG oslo.messaging._drivers.amqp [-] unpacked
context: {u'tenant': None, u'project_name': None, u'user_id': None, u'roles':
[u'admin'
[Bug 1384108] Re: Exception during message handling: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10
** Changed in: neutron Status: New => Confirmed ** Changed in: neutron Importance: Undecided => High ** Changed in: neutron Assignee: (unassigned) => Eugene Nikanorov (enikanorov) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1384108 Title: Exception during message handling: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1384108/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384108] Re: Exception during message handling: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10
Increasing the max pool size seems to resolve this particular issue; maybe having a default that's a multiple of the worker configuration would be more intelligent here? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1384108 Title: Exception during message handling: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1384108/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384108] Re: Exception during message handling: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10
I've bumped the max connections to 4 x the worker configuration just to be on the safe side. Retesting now to see if this helps. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1384108 Title: Exception during message handling: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1384108/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384108] Re: Exception during message handling: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10
Environment was being exercises using the boot-and-delete task/rally -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1384108 Title: Exception during message handling: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1384108/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
