Looking at this deeper, it appears neutron did properly retry this DB
action and the instance connection issue may be unrelated. Marking this
invalid.
** Changed in: neutron
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1822382
Title:
DBDeadlock for INSERT INTO resourcedeltas
Status in neutron:
Invalid
Bug description:
Recently we started seeing instances fail to become reachable in the
Octavia tempest jobs. This is intermittent, but recurring. This may be
related to other DBDeadlock bugs recently reported for quotas, but
since the SQL is different here I am reporting it.
This is on Master/Train.
Summary of the error in q-svc:
Mar 29 20:04:12.816598 ubuntu-xenial-rax-dfw-0004550340 neutron-
server[11470]: ERROR oslo_db.sqlalchemy.exc_filters
oslo_db.exception.DBDeadlock: (pymysql.err.InternalError) (1213,
'Deadlock found when trying to get lock; try restarting transaction')
[SQL: 'INSERT INTO resourcedeltas (resource, reservation_id, amount)
VALUES (%(resource)s, %(reservation_id)s, %(amount)s)'] [parameters:
{'reservation_id': '4f198b7d-ac31-42bb-98bd-686c830322ab', 'resource':
'port', 'amount': 1}] (Background on this error at:
http://sqlalche.me/e/2j85)
Full traceback from q-svc for once occurrence:
Mar 29 20:04:12.790909 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters
[req-56d6484e-b182-4dbd-8bb9-8db4ceb3c38a
req-ddb65494-cdaf-4dec-ab19-84efbede0da7 admin admin] DBAPIError exception
wrapped from (pymysql.err.InternalError) (1305, 'SAVEPOINT sa_savepoint_9 does
not exist') [SQL: 'ROLLBACK TO SAVEPOINT sa_savepoint_9'] (Background on this
error at: http://sqlalche.me/e/2j85): pymysql.err.InternalError: (1305,
'SAVEPOINT sa_savepoint_9 does not exist')
Mar 29 20:04:12.791441 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters Traceback (most
recent call last):
Mar 29 20:04:12.791889 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1193,
in _execute_context
Mar 29 20:04:12.792380 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters context)
Mar 29 20:04:12.792860 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/default.py", line
507, in do_execute
Mar 29 20:04:12.793296 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters
cursor.execute(statement, parameters)
Mar 29 20:04:12.793872 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py", line 165, in
execute
Mar 29 20:04:12.794320 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters result =
self._query(query)
Mar 29 20:04:12.794743 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py", line 321, in _query
Mar 29 20:04:12.795219 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters conn.query(q)
Mar 29 20:04:12.795668 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 860, in
query
Mar 29 20:04:12.796102 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
Mar 29 20:04:12.796505 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1061, in
_read_query_result
Mar 29 20:04:12.796904 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters result.read()
Mar 29 20:04:12.797336 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1349, in
read
Mar 29 20:04:12.797730 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters first_packet =
self.connection._read_packet()
Mar 29 20:04:12.798022 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1018, in
_read_packet
Mar 29 20:04:12.798305 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters
packet.check_error()
Mar 29 20:04:12.798600 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 384, in
check_error
Mar 29 20:04:12.798894 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters
err.raise_mysql_exception(self._data)
Mar 29 20:04:12.799224 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/pymysql/err.py", line 107, in
raise_mysql_exception
Mar 29 20:04:12.799516 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters raise
errorclass(errno, errval)
Mar 29 20:04:12.799802 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters
pymysql.err.InternalError: (1213, 'Deadlock found when trying to get lock; try
restarting transaction')
Mar 29 20:04:12.800094 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters
Mar 29 20:04:12.800509 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters The above exception
was the direct cause of the following exception:
Mar 29 20:04:12.800944 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters
Mar 29 20:04:12.801332 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters Traceback (most
recent call last):
Mar 29 20:04:12.801636 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py", line 2333,
in _flush
Mar 29 20:04:12.801922 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters
flush_context.execute()
Mar 29 20:04:12.802210 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/unitofwork.py", line
391, in execute
Mar 29 20:04:12.802555 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters
rec.execute(self)
Mar 29 20:04:12.802851 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/unitofwork.py", line
556, in execute
Mar 29 20:04:12.803240 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters uow
Mar 29 20:04:12.803792 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/persistence.py", line
181, in save_obj
Mar 29 20:04:12.804108 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters mapper, table,
insert)
Mar 29 20:04:12.804435 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/persistence.py", line
830, in _emit_insert_statements
Mar 29 20:04:12.804743 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters
execute(statement, multiparams)
Mar 29 20:04:12.805056 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 948,
in execute
Mar 29 20:04:12.805337 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters return
meth(self, multiparams, params)
Mar 29 20:04:12.805697 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/sqlalchemy/sql/elements.py", line 269,
in _execute_on_connection
Mar 29 20:04:12.806036 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters return
connection._execute_clauseelement(self, multiparams, params)
Mar 29 20:04:12.806449 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1060,
in _execute_clauseelement
Mar 29 20:04:12.806739 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters compiled_sql,
distilled_params
Mar 29 20:04:12.807148 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1200,
in _execute_context
Mar 29 20:04:12.807480 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters context)
Mar 29 20:04:12.807773 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1409,
in _handle_dbapi_exception
Mar 29 20:04:12.808062 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters
util.raise_from_cause(newraise, exc_info)
Mar 29 20:04:12.808388 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py", line 203,
in raise_from_cause
Mar 29 20:04:12.808732 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters
reraise(type(exception), exception, tb=exc_tb, cause=cause)
Mar 29 20:04:12.809104 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py", line 186,
in reraise
Mar 29 20:04:12.809390 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters raise
value.with_traceback(tb)
Mar 29 20:04:12.809730 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1193,
in _execute_context
Mar 29 20:04:12.810057 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters context)
Mar 29 20:04:12.810334 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/default.py", line
507, in do_execute
Mar 29 20:04:12.810632 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters
cursor.execute(statement, parameters)
Mar 29 20:04:12.811087 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py", line 165, in
execute
Mar 29 20:04:12.811477 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters result =
self._query(query)
Mar 29 20:04:12.811776 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py", line 321, in _query
Mar 29 20:04:12.812094 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters conn.query(q)
Mar 29 20:04:12.812378 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 860, in
query
Mar 29 20:04:12.812661 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
Mar 29 20:04:12.812937 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1061, in
_read_query_result
Mar 29 20:04:12.813324 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters result.read()
Mar 29 20:04:12.813722 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1349, in
read
Mar 29 20:04:12.814133 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters first_packet =
self.connection._read_packet()
Mar 29 20:04:12.814666 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1018, in
_read_packet
Mar 29 20:04:12.815045 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters
packet.check_error()
Mar 29 20:04:12.815417 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 384, in
check_error
Mar 29 20:04:12.815711 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters
err.raise_mysql_exception(self._data)
Mar 29 20:04:12.815990 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/local/lib/python3.5/dist-packages/pymysql/err.py", line 107, in
raise_mysql_exception
Mar 29 20:04:12.816272 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters raise
errorclass(errno, errval)
Mar 29 20:04:12.816598 ubuntu-xenial-rax-dfw-0004550340
neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters
oslo_db.exception.DBDeadlock: (pymysql.err.InternalError) (1213, 'Deadlock
found when trying to get lock; try restarting transaction') [SQL: 'INSERT INTO
resourcedeltas (resource, reservation_id, amount) VALUES (%(resource)s,
%(reservation_id)s, %(amount)s)'] [parameters: {'reservation_id':
'4f198b7d-ac31-42bb-98bd-686c830322ab', 'resource': 'port', 'amount': 1}]
(Background on this error at: http://sqlalche.me/e/2j85)
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1822382/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
