yuseok89 opened a new issue, #74482:
URL: https://github.com/apache/airflow/issues/74482
### Description
The latest run of a Dag is picked in three different ways across these views.
| Where | How the latest run is picked |
|---|---|
| Dashboard Failed Dags count (`ui/dashboard.py` `dag_stats`) |
`logical_date DESC`, runs without a logical date skipped |
| Dags list `last_dag_run_state` filter and `last_run_*` sorting, and `GET
/api/v2/dags` (`common/db/dags.py` `generate_dag_with_latest_run_query`) |
`max(id)` |
| Dags list latest run shown on cards and in the table, Dag page header
(`latest_dag_runs`, `latest_run`) | `run_after DESC` |
I'd like to propose using `run_after DESC, id DESC` in all three, in two
steps.
**Step 1. Add a `(dag_id, run_after)` index on `dag_run`** (#74481)
Picking the latest run by `run_after` needs an index in that order.
`dag_run` has indexes on `dag_id` and on `run_after`, but none that keeps the
runs of one Dag in `run_after` order, so a per-Dag lookup either sorts all runs
of the Dag or scans `idx_dag_run_run_after` and filters on `dag_id`. #74401 ran
into the same thing for the Dags list recent runs. This step changes no
behavior and already speeds up `GET /ui/dags/{dag_id}/latest_run` for Dags
whose runs are old.
**Step 2. Pick the latest run by `run_after DESC, id DESC` in all three**
- `run_after` is never null, unlike `logical_date`, which is null for manual
runs without a logical date and for asset-triggered runs.
- Most of what users see already uses it. The places that would change are
the dashboard count and the `max(id)` filter and sort.
- `_check_last_n_dagruns_failed`, which decides whether to pause a Dag after
consecutive failures, already orders by `run_after DESC, id DESC`.
This changes what `GET /api/v2/dags` returns for `last_dag_run_state`,
`dag_run_state`, `dag_run_start_date_*` and `dag_run_end_date_*`, and for
`last_run_state` and `last_run_start_date` sorting, whenever the newest-created
run is not the latest by `run_after`. The main case is a backfill over a past
interval.
The Grid orders runs by the timetable's `run_ordering`, which is a separate,
pluggable concept and is out of scope here.
### Use case/motivation
"Latest run" is one concept, but each view defines it differently, so a user
can see one run as the latest on a card, filter Dags by the state of another,
and count failed Dags from a third. The dashboard Failed Dags count and the
Dags list it links to can disagree, and the list can show one latest run while
filtering and sorting by another. Backfills over a past interval and manual
runs without a logical date make this visible.
Can we settle on `run_after` as the definition of a Dag's latest run here?
### Related issues
#57492, #74401, #74481
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
