MatthewRBruce commented on issue #13685:
URL: https://github.com/apache/airflow/issues/13685#issuecomment-767719815
Sam and I have been looking at this, and so far here’s my theory:
It appears this happens when the Scheduler is restarted/killed during an
active DagRun. Based on looking at the scheduling code, and the logs, If a
DagRun is at it’s max_active_runs (indicated by a log line like this:
{scheduler_job.py:1598} INFO - DAG airflow-utils.send-airflow-heartbeat is at
(or above) max_active_runs (1 of 1), not creating any more runs ) then the
next_dagrun_create_after field for that DagRun will be set to None/NULL here:
https://github.com/apache/airflow/blob/2.0.0/airflow/jobs/scheduler_job.py#L1604
Now if the scheduler is restarted/killed before this Dag Finishes and it
starts back up, the DagModel will be returned by
`DagModel.dags_needing_dagrun(session)` here:
https://github.com/apache/airflow/blob/2.0.0/airflow/jobs/scheduler_job.py#L1473-L1474
and then a new DagRun will be created for it, resulting in a violation of
the Unique Key Constraint
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
