Reviewed: https://review.openstack.org/344924 Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=e8022f3d3172f63eb11bbc3f39e0f595ecf3c7a3 Submitter: Jenkins Branch: master
commit e8022f3d3172f63eb11bbc3f39e0f595ecf3c7a3 Author: Mikhail Nikolaenko <[email protected]> Date: Wed Jul 20 15:56:53 2016 +0000 Retry revocation on MySQL deadlock Added wrapper for revoke method that makes additional attempts on deadlock. Change-Id: I506c6b432d4d23619237a32414d0039c749cd8b6 Closes-Bug: #1604862 ** Changed in: keystone Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1604862 Title: DBDeadlock: (_mysql_exceptions.OperationalError) while deleting user Status in OpenStack Identity (keystone): Fix Released Bug description: The error happened at the end of Rally test create-delete-node-group- templates.yaml when Rally started to remove users From Keystone admin log: 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi [req-23d801cf-baa4-410d-9d46-6eb6ca7dfa34 21fe895155b54960b09290ef179ce15e 56b54d0fb50e40b5bb7ccdb46c8d21a7 - default default] (_mysql_exceptions.OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') [SQL: u'DELETE FROM revocation_event WHERE revocation_event.revoked_at < %s'] [parameters: (datetime.datetime(2016, 5, 19, 6, 57, 44, 270231),)] 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi Traceback (most recent call last): 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 249, in __call__ 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi result = method(context, **params) 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/oslo_log/versionutils.py", line 165, in wrapped 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi return func_or_cls(*args, **kwargs) 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/controllers.py", line 174, in delete_user 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi self.identity_api.delete_user(user_id, initiator) 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/manager.py", line 124, in wrapped 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi __ret_val = __f(*args, **kwargs) 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 433, in wrapper 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi return f(self, *args, **kwargs) 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 443, in wrapper 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi return f(self, *args, **kwargs) 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 986, in delete_user 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi notifications.Audit.deleted(self._USER, user_id, initiator) 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/notifications.py", line 166, in deleted 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi public) 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/notifications.py", line 137, in _emit 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi public=public) 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/notifications.py", line 421, in _send_notification 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi notify_event_callbacks(SERVICE, resource_type, operation, payload) 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/notifications.py", line 316, in notify_event_callbacks 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi cb(service, resource_type, operation, payload) 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/revoke/core.py", line 84, in _user_callback 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi self.revoke_by_user(payload['resource_info']) 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/manager.py", line 124, in wrapped 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi __ret_val = __f(*args, **kwargs) 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/revoke/core.py", line 151, in revoke_by_user 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi return self.revoke(revoke_model.RevokeEvent(user_id=user_id)) 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/manager.py", line 124, in wrapped 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi __ret_val = __f(*args, **kwargs) 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/revoke/core.py", line 229, in revoke 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi self.driver.revoke(event) 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/revoke/backends/sql.py", line 100, in revoke 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi self._prune_expired_events() 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/revoke/backends/sql.py", line 77, in _prune_expired_events 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi query.delete(synchronize_session=False) 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 3048, in delete 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi delete_op.exec_() 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 1127, in exec_ 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi self._do_exec() 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 1311, in _do_exec 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi mapper=self.mapper) 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1034, in execute 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi bind, close_with_result=True).execute(clause, params or {}) 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi return meth(self, multiparams, params) 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi return connection._execute_clauseelement(self, multiparams, params) 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi compiled_sql, distilled_params 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi context) 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi util.raise_from_cause(newraise, exc_info) 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi reraise(type(exception), exception, tb=exc_tb, cause=cause) 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi context) 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi cursor.execute(statement, parameters) 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 219, in execute 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi self.errorhandler(self, exc, value) 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 38, in defaulterrorhandler 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi raise errorvalue 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi DBDeadlock: (_mysql_exceptions.OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') [SQL: u'DELETE FROM revocation_event WHERE revocation_event.revoked_at < %s'] [parameters: (datetime.datetime(2016, 5, 19, 6, 57, 44, 270231),)] 2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1604862/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
