jledru opened a new issue #16453:
URL: https://github.com/apache/airflow/issues/16453
### Apache Airflow version: 1.10.12
#### Environment:
Cloud provider or hardware configuration: Cloud Composer 1.13.2
Kubernetes version :v1.18.17-gke.1900
Kernel (e.g. uname -a): x86_64
Install tools: pip
Others: mysql Ver 14.14 Distrib 5.7.32, for Linux (x86_64) using EditLine
wrapper, Python 3.6.10
#### What happened:
After activated DAG Serialization, we are facing from time to time :
```ERROR - (_mysql_exceptions.OperationalError) (1213, 'Deadlock found when
trying to get lock; try restarting transaction'
[SQL: DELETE FROM rendered_task_instance_fields WHERE
rendered_task_instance_fields.dag_id = %s AND
rendered_task_instance_fields.task_id = %s AND
(rendered_task_instance_fields.dag_id, rendered_task_instance_fields.task_id,
rendered_task_instance_fields.execution_date) NOT IN (SELECT subq1.dag_id,
subq1.task_id, subq1.execution_date
FROM (SELECT rendered_task_instance_fields.dag_id AS dag_id,
rendered_task_instance_fields.task_id AS task_id,
rendered_task_instance_fields.execution_date AS execution_date
FROM rendered_task_instance_fields
WHERE rendered_task_instance_fields.dag_id = %s AND
rendered_task_instance_fields.task_id = %s ORDER BY
rendered_task_instance_fields.execution_date DESC
LIMIT %s) AS subq1)
[parameters: ('PRODUCT_RANKING_V2', 'update_bq_table_RANKING_fr',
'PRODUCT_RANKING_V2', 'update_bq_table_RANKING_fr', 30)
(Background on this error at: http://sqlalche.me/e/13/e3q8)
Traceback (most recent call last)
File
"/opt/python3.6/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line
1277, in _execute_contex
cursor, statement, parameters, contex
File
"/opt/python3.6/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line
593, in do_execut
cursor.execute(statement, parameters
File "/opt/python3.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line
255, in execut
self.errorhandler(self, exc, value
File "/opt/python3.6/lib/python3.6/site-packages/MySQLdb/connections.py",
line 50, in defaulterrorhandle
raise errorvalu
File "/opt/python3.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line
252, in execut
res = self._query(query
File "/opt/python3.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line
378, in _quer
db.query(q
File "/opt/python3.6/lib/python3.6/site-packages/MySQLdb/connections.py",
line 280, in quer
_mysql.connection.query(self, query
_mysql_exceptions.OperationalError: (1213, 'Deadlock found when trying to
get lock; try restarting transaction'
The above exception was the direct cause of the following exception
Traceback (most recent call last)
File "/usr/local/lib/airflow/airflow/models/taskinstance.py", line 969, in
_run_raw_tas
RTIF.delete_old_records(self.task_id, self.dag_id, session=session
File "/usr/local/lib/airflow/airflow/utils/db.py", line 70, in wrappe
return func(*args, **kwargs
File "/usr/local/lib/airflow/airflow/models/renderedtifields.py", line
145, in delete_old_record
.delete(synchronize_session=False
File "/opt/python3.6/lib/python3.6/site-packages/sqlalchemy/orm/query.py",
line 3899, in delet
delete_op.exec_(
File
"/opt/python3.6/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py",
line 1697, in exec
self._do_exec(
File
"/opt/python3.6/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py",
line 1928, in _do_exe
self._execute_stmt(delete_stmt
File
"/opt/python3.6/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py",
line 1702, in _execute_stm
self.result = self.query._execute_crud(stmt, self.mapper
File "/opt/python3.6/lib/python3.6/site-packages/sqlalchemy/orm/query.py",
line 3541, in _execute_cru
return conn.execute(stmt, self._params
File
"/opt/python3.6/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line
1011, in execut
return meth(self, multiparams, params
File
"/opt/python3.6/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line
298, in _execute_on_connectio
return connection._execute_clauseelement(self, multiparams, params
File
"/opt/python3.6/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line
1130, in _execute_clauseelemen
distilled_params
File
"/opt/python3.6/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line
1317, in _execute_contex
e, statement, parameters, cursor, contex
File
"/opt/python3.6/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line
1511, in _handle_dbapi_exceptio
sqlalchemy_exception, with_traceback=exc_info[2], from_=
File
"/opt/python3.6/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line
182, in raise
raise exceptio
File
"/opt/python3.6/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line
1277, in _execute_contex
cursor, statement, parameters, contex
File
"/opt/python3.6/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line
593, in do_execut
cursor.execute(statement, parameters
File "/opt/python3.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line
255, in execut
self.errorhandler(self, exc, value
File "/opt/python3.6/lib/python3.6/site-packages/MySQLdb/connections.py",
line 50, in defaulterrorhandle
raise errorvalu
File "/opt/python3.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line
252, in execut
res = self._query(query
File "/opt/python3.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line
378, in _quer
db.query(q
File "/opt/python3.6/lib/python3.6/site-packages/MySQLdb/connections.py",
line 280, in quer
_mysql.connection.query(self, query
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1213,
'Deadlock found when trying to get lock; try restarting transaction'
[SQL: DELETE FROM rendered_task_instance_fields WHERE
rendered_task_instance_fields.dag_id = %s AND
rendered_task_instance_fields.task_id = %s AND
(rendered_task_instance_fields.dag_id, rendered_task_instance_fields.task_id,
rendered_task_instance_fields.execution_date) NOT IN (SELECT subq1.dag_id,
subq1.task_id, subq1.execution_date
FROM (SELECT rendered_task_instance_fields.dag_id AS dag_id,
rendered_task_instance_fields.task_id AS task_id,
rendered_task_instance_fields.execution_date AS execution_date
FROM rendered_task_instance_fields
WHERE rendered_task_instance_fields.dag_id = %s AND
rendered_task_instance_fields.task_id = %s ORDER BY
rendered_task_instance_fields.execution_date DESC
LIMIT %s) AS subq1)
[parameters: ('PRODUCT_RANKING_V2', 'update_bq_table_RANKING_fr',
'PRODUCT_RANKING_V2', 'update_bq_table_RANKING_fr', 30)
(Background on this error at: http://sqlalche.me/e/13/e3q8
```
It seems quite similar to this
[issue](https://github.com/apache/airflow/issues/8521), but not resolved in our
case in version 1.10.12. In the comments, seems that may be related to
concurrent dags, but in my case there is no concurrency, there is 1 dagrun at a
time.
The error seems transient as when we clear the task afterwards it is OK
#### What you expected to happen:
Not facing deadlock and mark task as success
#### How to reproduce it:
No pattern found except DAG Serialization, happen from time to time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
