abhishekbhakat commented on issue #23361:
URL: https://github.com/apache/airflow/issues/23361#issuecomment-1138646896
> The deadlock issue is not the cause of the tasks being set to skipped --
the deadlock occurs when trying to.
>
> @abhishekbhakat what error do you get with that? The log you included
there doesn't show any error.
@ashb The log file provided has that error in around line no. 5810
Attaching the error message below:
```
[[34m2022-04-29 12:25:18,883[0m] {[34mscheduler_job.py:[0m753} ERROR[0m
- Exception when executing SchedulerJob._run_scheduler_loop[0m
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 1685, in _execute_context
self.dialect.do_executemany(
File
"/usr/local/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py",
line 917, in do_executemany
self._psycopg2_extras().execute_batch(
File "/usr/local/lib/python3.9/site-packages/psycopg2/extras.py", line
1187, in execute_batch
cur.execute(b";".join(sqls))
psycopg2.errors.DeadlockDetected: deadlock detected
DETAIL: Process 29749 waits for ShareLock on transaction 62344; blocked by
process 31075.
Process 31075 waits for ShareLock on transaction 62338; blocked by process
29749.
HINT: See server log for query details.
CONTEXT: while updating tuple (4,23) in relation "task_instance"
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File
"/usr/local/lib/python3.9/site-packages/airflow/jobs/scheduler_job.py", line
736, in _execute
self._run_scheduler_loop()
File
"/usr/local/lib/python3.9/site-packages/airflow/jobs/scheduler_job.py", line
824, in _run_scheduler_loop
num_queued_tis = self._do_scheduling(session)
File
"/usr/local/lib/python3.9/site-packages/airflow/jobs/scheduler_job.py", line
906, in _do_scheduling
callback_to_run = self._schedule_dag_run(dag_run, session)
File
"/usr/local/lib/python3.9/site-packages/airflow/jobs/scheduler_job.py", line
1146, in _schedule_dag_run
self._verify_integrity_if_dag_changed(dag_run=dag_run, session=session)
File "/usr/local/lib/python3.9/site-packages/airflow/utils/session.py",
line 68, in wrapper
return func(*args, **kwargs)
File
"/usr/local/lib/python3.9/site-packages/airflow/jobs/scheduler_job.py", line
1176, in _verify_integrity_if_dag_changed
dag_run.verify_integrity(session=session)
File "/usr/local/lib/python3.9/site-packages/airflow/utils/session.py",
line 68, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/airflow/models/dagrun.py",
line 944, in verify_integrity
session.flush()
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
line 3255, in flush
self._flush(objects)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
line 3395, in _flush
transaction.rollback(_capture_exception=True)
File
"/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line
70, in __exit__
compat.raise_(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py",
line 211, in raise_
raise exception
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
line 3355, in _flush
flush_context.execute()
File
"/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line
453, in execute
rec.execute(self)
File
"/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line
627, in execute
util.preloaded.orm_persistence.save_obj(
File
"/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/persistence.py", line
234, in save_obj
_emit_update_statements(
File
"/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/persistence.py", line
998, in _emit_update_statements
c = connection._execute_20(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 1520, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py",
line 313, in _execute_on_connection
return connection._execute_clauseelement(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 1389, in _execute_clauseelement
ret = self._execute_context(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 1748, in _execute_context
self._handle_dbapi_exception(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 1929, in _handle_dbapi_exception
util.raise_(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py",
line 211, in raise_
raise exception
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 1685, in _execute_context
self.dialect.do_executemany(
File
"/usr/local/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py",
line 917, in do_executemany
self._psycopg2_extras().execute_batch(
File "/usr/local/lib/python3.9/site-packages/psycopg2/extras.py", line
1187, in execute_batch
cur.execute(b";".join(sqls))
sqlalchemy.exc.OperationalError: (psycopg2.errors.DeadlockDetected) deadlock
detected
DETAIL: Process 29749 waits for ShareLock on transaction 62344; blocked by
process 31075.
Process 31075 waits for ShareLock on transaction 62338; blocked by process
29749.
HINT: See server log for query details.
CONTEXT: while updating tuple (4,23) in relation "task_instance"
[SQL: UPDATE task_instance SET state=%(state)s WHERE task_instance.task_id =
%(task_instance_task_id)s AND task_instance.dag_id = %(task_instance_dag_id)s
AND task_instance.run_id = %(task_instance_run_id)s AND task_instance.map_index
= %(task_instance_map_index)s]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
