GitHub user enchant3dmango added a comment to the discussion: The scheduler
stopped running due to a deadlock problem
The error indicates a database deadlock issue, specifically when updating the
DAG scheduling information. Error:
```bash
sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled
back due to a previous exception during flush.
```
```bash
(pymysql.err.OperationalError) (1213, 'Deadlock found when trying to get lock;
try restarting transaction')
```
And, it happens when the Airflow tries to execute this query:
```sql
UPDATE dag SET next_dagrun=%(next_dagrun)s,
next_dagrun_data_interval_start=%(next_dagrun_data_interval_start)s,
next_dagrun_data_interval_end=%(next_dagrun_data_interval_end)s,
next_dagrun_create_after=%(next_dagrun_create_after)s WHERE dag.dag_id =
%(dag_dag_id)s
```
The issue occurs because concurrent transactions are trying to update the same
DAG record. The very short schedule interval (30 seconds) might contribute to
this by creating many concurrent operations. You may try to update the schedule
interval to minutely and also try to add and set `max_active_runs_per_dag=1` to
your DAG.
Your DAG will be like this:
```python
with DAG(
dag_id=task_name,
schedule=timedelta(seconds=60),
start_date=datetime(2024, 1, 1),
dagrun_timeout=timedelta(seconds=300),
catchup=False,
max_active_runs_per_dag=1
):
sub_hook = SubprocessHook()
@task
def run_command(execute, arguments, capture, work_dir, env):
command = [execute] + arguments.split()
result = sub_hook.run_command(command, env=env, cwd=work_dir)
if result.exit_code is not 0:
raise ValueError(result.output)
return result.output
run_command(dag_config['exec'], dag_config['arg'],
dag_config.get("capture", False),
dag_config.get('work_dir', XXXXX_PATH /
"config"), dag_config.get('env', None))
```
GitHub link:
https://github.com/apache/airflow/discussions/45275#discussioncomment-11694121
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
