Re: [Openstack] Keystone Too Many Connections Problem
I'm attempting to reproduce this now (haven't been successful with PKI + sqlite on disk), but in the meantime I opened a bug on this issue for tracking: https://bugs.launchpad.net/keystone/+bug/1162845 -Dolph On Fri, Mar 29, 2013 at 1:35 PM, Miller, Mark M (EB SW Cloud - RD - Corvallis) mark.m.mil...@hp.com wrote: Hi, ** ** I have a loop that gets PKI tokens from Keystone (RC2). After about 116 loops I get the following error and have to reset Keystone to continue. Has anyone else run into this? ** ** Thanks, ** ** Mark ** ** (keystone.auth.controllers): 2013-03-29 11:14:01,274 ERROR (OperationalError) (1040, 'Too many connections') None None Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/controllers.py, line 286, in authenticate_for_token self.authenticate(context, auth_info, auth_context) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/controllers.py, line 352, in authenticate auth_context) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/plugins/password.py, line 108, in authenticate user_info = UserAuthInfo(context, auth_payload) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/plugins/password.py, line 36, in __init__ self._validate_and_normalize_auth_data(auth_payload) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/plugins/password.py, line 87, in _validate_and_normalize_auth_data domain_ref = self._lookup_domain(user_info['domain']) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/plugins/password.py, line 60, in _lookup_domain context=self.context, domain_name=domain_name) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/common/manager.py, line 47, in _wrapper return f(*args, **kw) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/common/sql/core.py, line 272, in wrapper return method(*args, **kwargs) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/identity/backends/sql.py, line 589, in get_domain_by_name ref = session.query(Domain).filter_by(name=domain_name).one() File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py, line 2184, in one ret = list(self) File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py, line 2227, in __iter__ return self._execute_and_instances(context) File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py, line 2240, in _execute_and_instances close_with_result=True) File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py, line 2231, in _connection_from_session **kw) File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py, line 777, in connection close_with_result=close_with_result) File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py, line 783, in _connection_for_bind return engine.contextual_connect(**kwargs) File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py, line 2489, in contextual_connect self.pool.connect(), File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py, line 236, in connect return _ConnectionFairy(self).checkout() File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py, line 401, in __init__ rec = self._connection_record = pool._do_get() File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py, line 746, in _do_get con = self._create_connection() File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py, line 189, in _create_connection return _ConnectionRecord(self) File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py, line 282, in __init__ self.connection = self.__connect() File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py, line 344, in __connect connection = self.__pool._creator() File
Re: [Openstack] Keystone Too Many Connections Problem
I also opened bug 1162857, which produces a slightly different symptom with a different configuration (sqlite), but may share the same underlying cause. -Dolph On Mon, Apr 1, 2013 at 10:26 AM, Dolph Mathews dolph.math...@gmail.comwrote: I'm attempting to reproduce this now (haven't been successful with PKI + sqlite on disk), but in the meantime I opened a bug on this issue for tracking: https://bugs.launchpad.net/keystone/+bug/1162845 -Dolph On Fri, Mar 29, 2013 at 1:35 PM, Miller, Mark M (EB SW Cloud - RD - Corvallis) mark.m.mil...@hp.com wrote: Hi, ** ** I have a loop that gets PKI tokens from Keystone (RC2). After about 116 loops I get the following error and have to reset Keystone to continue. Has anyone else run into this? ** ** Thanks, ** ** Mark ** ** (keystone.auth.controllers): 2013-03-29 11:14:01,274 ERROR (OperationalError) (1040, 'Too many connections') None None Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/controllers.py, line 286, in authenticate_for_token self.authenticate(context, auth_info, auth_context) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/controllers.py, line 352, in authenticate auth_context) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/plugins/password.py, line 108, in authenticate user_info = UserAuthInfo(context, auth_payload) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/plugins/password.py, line 36, in __init__ self._validate_and_normalize_auth_data(auth_payload) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/plugins/password.py, line 87, in _validate_and_normalize_auth_data domain_ref = self._lookup_domain(user_info['domain']) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/plugins/password.py, line 60, in _lookup_domain context=self.context, domain_name=domain_name) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/common/manager.py, line 47, in _wrapper return f(*args, **kw) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/common/sql/core.py, line 272, in wrapper return method(*args, **kwargs) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/identity/backends/sql.py, line 589, in get_domain_by_name ref = session.query(Domain).filter_by(name=domain_name).one() File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py, line 2184, in one ret = list(self) File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py, line 2227, in __iter__ return self._execute_and_instances(context) File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py, line 2240, in _execute_and_instances close_with_result=True) File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py, line 2231, in _connection_from_session **kw) File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py, line 777, in connection close_with_result=close_with_result) File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py, line 783, in _connection_for_bind return engine.contextual_connect(**kwargs) File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py, line 2489, in contextual_connect self.pool.connect(), File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py, line 236, in connect return _ConnectionFairy(self).checkout() File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py, line 401, in __init__ rec = self._connection_record = pool._do_get() File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py, line 746, in _do_get con = self._create_connection() File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py, line 189, in _create_connection return _ConnectionRecord(self) File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py, line 282, in __init__ self.connection =
Re: [Openstack] Keystone Too Many Connections Problem
Mark, Which token driver are you using? -Dolph On Mon, Apr 1, 2013 at 10:49 AM, Dolph Mathews dolph.math...@gmail.comwrote: I also opened bug 1162857, which produces a slightly different symptom with a different configuration (sqlite), but may share the same underlying cause. -Dolph On Mon, Apr 1, 2013 at 10:26 AM, Dolph Mathews dolph.math...@gmail.comwrote: I'm attempting to reproduce this now (haven't been successful with PKI + sqlite on disk), but in the meantime I opened a bug on this issue for tracking: https://bugs.launchpad.net/keystone/+bug/1162845 -Dolph On Fri, Mar 29, 2013 at 1:35 PM, Miller, Mark M (EB SW Cloud - RD - Corvallis) mark.m.mil...@hp.com wrote: Hi, ** ** I have a loop that gets PKI tokens from Keystone (RC2). After about 116 loops I get the following error and have to reset Keystone to continue. Has anyone else run into this? ** ** Thanks, ** ** Mark ** ** (keystone.auth.controllers): 2013-03-29 11:14:01,274 ERROR (OperationalError) (1040, 'Too many connections') None None Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/controllers.py, line 286, in authenticate_for_token self.authenticate(context, auth_info, auth_context) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/controllers.py, line 352, in authenticate auth_context) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/plugins/password.py, line 108, in authenticate user_info = UserAuthInfo(context, auth_payload) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/plugins/password.py, line 36, in __init__ self._validate_and_normalize_auth_data(auth_payload) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/plugins/password.py, line 87, in _validate_and_normalize_auth_data domain_ref = self._lookup_domain(user_info['domain']) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/plugins/password.py, line 60, in _lookup_domain context=self.context, domain_name=domain_name) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/common/manager.py, line 47, in _wrapper return f(*args, **kw) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/common/sql/core.py, line 272, in wrapper return method(*args, **kwargs) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/identity/backends/sql.py, line 589, in get_domain_by_name ref = session.query(Domain).filter_by(name=domain_name).one() File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py, line 2184, in one ret = list(self) File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py, line 2227, in __iter__ return self._execute_and_instances(context) File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py, line 2240, in _execute_and_instances close_with_result=True) File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py, line 2231, in _connection_from_session **kw) File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py, line 777, in connection close_with_result=close_with_result) File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py, line 783, in _connection_for_bind return engine.contextual_connect(**kwargs) File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py, line 2489, in contextual_connect self.pool.connect(), File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py, line 236, in connect return _ConnectionFairy(self).checkout() File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py, line 401, in __init__ rec = self._connection_record = pool._do_get() File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py, line 746, in _do_get con = self._create_connection() File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py, line 189, in _create_connection return _ConnectionRecord(self) File
Re: [Openstack] Keystone Too Many Connections Problem
Thanks! I was able to reproduce and I've proposed a fix to both master and grizzly. master- https://review.openstack.org/#/c/25831/ grizzly- https://review.openstack.org/#/c/25833/ If you can verify this patch on your system, that would be really helpful. -Dolph On Mon, Apr 1, 2013 at 1:57 PM, Miller, Mark M (EB SW Cloud - RD - Corvallis) mark.m.mil...@hp.com wrote: Dolph, ** ** [token] driver = keystone.token.backends.sql.Token ** ** # Amount of time a token should remain valid (in seconds) expiration = 86400 ** ** I get the same results for either UUID or PKI tokens. ** ** Mark ** ** *From:* Dolph Mathews [mailto:dolph.math...@gmail.com] *Sent:* Monday, April 01, 2013 11:56 AM *To:* Miller, Mark M (EB SW Cloud - RD - Corvallis) *Cc:* openstack@lists.launchpad.net *Subject:* Re: [Openstack] Keystone Too Many Connections Problem ** ** Mark, ** ** Which token driver are you using? ** ** -Dolph ** ** On Mon, Apr 1, 2013 at 10:49 AM, Dolph Mathews dolph.math...@gmail.com wrote: I also opened bug 1162857, which produces a slightly different symptom with a different configuration (sqlite), but may share the same underlying cause. ** ** -Dolph ** ** On Mon, Apr 1, 2013 at 10:26 AM, Dolph Mathews dolph.math...@gmail.com wrote: I'm attempting to reproduce this now (haven't been successful with PKI + sqlite on disk), but in the meantime I opened a bug on this issue for tracking: ** ** https://bugs.launchpad.net/keystone/+bug/1162845 ** ** -Dolph ** ** On Fri, Mar 29, 2013 at 1:35 PM, Miller, Mark M (EB SW Cloud - RD - Corvallis) mark.m.mil...@hp.com wrote: Hi, I have a loop that gets PKI tokens from Keystone (RC2). After about 116 loops I get the following error and have to reset Keystone to continue. Has anyone else run into this? Thanks, Mark (keystone.auth.controllers): 2013-03-29 11:14:01,274 ERROR (OperationalError) (1040, 'Too many connections') None None Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/controllers.py, line 286, in authenticate_for_token self.authenticate(context, auth_info, auth_context) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/controllers.py, line 352, in authenticate auth_context) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/plugins/password.py, line 108, in authenticate user_info = UserAuthInfo(context, auth_payload) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/plugins/password.py, line 36, in __init__ self._validate_and_normalize_auth_data(auth_payload) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/plugins/password.py, line 87, in _validate_and_normalize_auth_data domain_ref = self._lookup_domain(user_info['domain']) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/plugins/password.py, line 60, in _lookup_domain context=self.context, domain_name=domain_name) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/common/manager.py, line 47, in _wrapper return f(*args, **kw) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/common/sql/core.py, line 272, in wrapper return method(*args, **kwargs) File /usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/identity/backends/sql.py, line 589, in get_domain_by_name ref = session.query(Domain).filter_by(name=domain_name).one() File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py, line 2184, in one ret = list(self) File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py, line 2227, in __iter__ return self._execute_and_instances(context) File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py, line 2240, in _execute_and_instances close_with_result=True) File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py, line 2231, in _connection_from_session **kw) File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py, line 777, in connection close_with_result=close_with_result) File /usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py, line 783
