kaxil commented on code in PR #77157:
URL: https://github.com/apache/airflow/pull/77157#discussion_r4607081101
##########
airflow-core/src/airflow/models/backfill.py:
##########
@@ -315,7 +315,10 @@ def _get_latest_dag_run_row_query(*, dag_id: str, info:
DagRunInfo):
stmt = stmt.where(DagRun.partition_date == info.partition_date)
if info.logical_date is not None:
stmt = stmt.where(DagRun.logical_date == info.logical_date)
- stmt = stmt.order_by(DagRun.start_date.is_(None), DagRun.start_date.desc())
+ # A requeued run has no start_date until it starts again, so fall back to
queued_at;
+ # otherwise an older finished run of the same partition is taken as the
latest.
+ latest_attempt = func.coalesce(DagRun.start_date, DagRun.queued_at)
Review Comment:
A run that goes from QUEUED straight to SUCCESS or FAILED (marked from the
UI or API) keeps `start_date` NULL, since `DagRun.set_state` only sets
`end_date` on that transition. With the coalesce, its newer `queued_at` now
outranks a RUNNING run of the same partition, so `--reprocess-behavior
completed` creates a duplicate run while that one is still running. On `main`
the running run sorts first and the backfill records `in flight`. Could the
in-flight decision be an `exists()` over `State.unfinished_dr_states` for the
partition, with this lookup only choosing the run for the reprocess decision?
That would also cover an older run still queued behind a newer run that started
and finished, which neither ordering handles. (The comment above
`create_dagrun` further down still says this lookup picks the newest run by
`start_date`.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]