honarkhah edited a comment on issue #8521:
URL: https://github.com/apache/airflow/issues/8521#issuecomment-674938047
In my case issue is about concurrent dags, the insertion issue is fixed in
airflow `1.10.11`
if you trigger more than 1 instance of a dag at the same time and you have
more than 1 worker you can reproduce this issue.
```
[2020-08-17 10:42:30,177] {taskinstance.py:882} INFO -
--------------------------------------------------------------------------------
[2020-08-17 10:42:30,187] {taskinstance.py:901} INFO - Executing
<Task(BranchPythonOperator): branch_task> on 2020-08-17T10:34:24.115521+00:00
[2020-08-17 10:42:30,190] {standard_task_runner.py:54} INFO - Started
process 553 to run task
[2020-08-17 10:42:30,212] {standard_task_runner.py:77} INFO - Running:
['airflow', 'run', 'logo_detection_webhook', 'branch_task',
'2020-08-17T10:34:24.115521+00:00', '--job_id', '169070', '--pool',
'default_pool', '--raw', '-sd',
'DAGS_FOLDER/model_training/logo_detection_webhook.py', '--cfg_path',
'/tmp/tmprvq51le4']
[2020-08-17 10:42:30,213] {standard_task_runner.py:78} INFO - Job 169070:
Subtask branch_task
[2020-08-17 10:42:30,251] {logging_mixin.py:112} INFO - Running %s on host
%s <TaskInstance: logo_detection_webhook.branch_task
2020-08-17T10:34:24.115521+00:00 [running]> 91a09a3997d3
[2020-08-17 10:43:21,389] {taskinstance.py:1150} ERROR -
(_mysql_exceptions.OperationalError) (1205, 'Lock wait timeout exceeded; try
restarting transaction')
[SQL: DELETE FROM rendered_task_instance_fields WHERE
rendered_task_instance_fields.dag_id = %s AND
rendered_task_instance_fields.task_id = %s AND
(rendered_task_instance_fields.dag_id, rendered_task_instance_fields.task_id,
rendered_task_instance_fields.execution_date) NOT IN (SELECT subq1.dag_id,
subq1.task_id, subq1.execution_date
FROM (SELECT rendered_task_instance_fields.dag_id AS dag_id,
rendered_task_instance_fields.task_id AS task_id,
rendered_task_instance_fields.execution_date AS execution_date
FROM rendered_task_instance_fields
WHERE rendered_task_instance_fields.dag_id = %s AND
rendered_task_instance_fields.task_id = %s ORDER BY
rendered_task_instance_fields.execution_date DESC
LIMIT %s) AS subq1)]
[parameters: ('logo_detection_webhook', 'branch_task',
'logo_detection_webhook', 'branch_task', 30)]
(Background on this error at: http://sqlalche.me/e/13/e3q8)
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py",
line 1278, in _execute_context
cursor, statement, parameters, context
File
"/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line
593, in do_execute
cursor.execute(statement, parameters)
File "/usr/local/lib/python3.7/site-packages/MySQLdb/cursors.py", line
255, in execute
self.errorhandler(self, exc, value)
File "/usr/local/lib/python3.7/site-packages/MySQLdb/connections.py", line
50, in defaulterrorhandler
raise errorvalue
File "/usr/local/lib/python3.7/site-packages/MySQLdb/cursors.py", line
252, in execute
res = self._query(query)
File "/usr/local/lib/python3.7/site-packages/MySQLdb/cursors.py", line
378, in _query
db.query(q)
File "/usr/local/lib/python3.7/site-packages/MySQLdb/connections.py", line
280, in query
_mysql.connection.query(self, query)
_mysql_exceptions.OperationalError: (1205, 'Lock wait timeout exceeded; try
restarting transaction')
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
