yuseok89 commented on code in PR #70470:
URL: https://github.com/apache/airflow/pull/70470#discussion_r3744342019
##########
airflow-core/src/airflow/api_fastapi/common/parameters.py:
##########
@@ -1473,19 +1473,29 @@ def depends(cls, has_pending_actions: bool | None =
Query(None)) -> _PendingActi
QueryPendingActionsFilter = Annotated[_PendingActionsFilter,
Depends(_PendingActionsFilter.depends)]
+# A lookback this large is effectively unbounded (users omit the param for
"any time"); capping it
+# also keeps utcnow() - timedelta(hours=...) from overflowing on an absurdly
large value.
+_MAX_DAG_RUN_STATE_WINDOW_HOURS = 24 * 366 * 100 # ~100 years
+
+
class _AnyDagRunStateFilter(BaseParam[DagRunState | None]):
"""Filter Dags that have any DagRun in the given state, not only the
latest one."""
+ def __init__(self, value: DagRunState | None = None, skip_none: bool =
True) -> None:
+ super().__init__(value, skip_none)
+ self.within_hours: int | None = None
+
def to_orm(self, select: Select) -> Select:
if self.value is None and self.skip_none:
return select
- # EXISTS resolves each Dag via the (dag_id, state) index instead of
scanning every run in the state.
- has_run_in_state = (
- sql_select(DagRun.dag_id)
- .where(DagRun.dag_id == DagModel.dag_id, DagRun.state ==
self.value)
- .exists()
- )
+ # EXISTS seeks the (dag_id, state) index per Dag rather than scanning
the whole table; the
+ # optional run_after bound is not covered by that index, so it is
filtered within each Dag's
+ # matching rows (still bounded per Dag, not a full scan).
+ conditions = [DagRun.dag_id == DagModel.dag_id, DagRun.state ==
self.value]
+ if self.within_hours is not None:
+ conditions.append(DagRun.run_after >= timezone.utcnow() -
timedelta(hours=self.within_hours))
+ has_run_in_state =
sql_select(DagRun.dag_id).where(*conditions).exists()
Review Comment:
Thanks.
I did a quick benchmark of the inverted approach on Postgres (10M `dag_run`
rows, warm cache). Rough numbers:
| within_hours | current | inverted (run_after-first) |
|---|---:|---:|
| none | < 50 ms | < 50 ms |
| 7 days | ~1 s | ~1 s |
| 30 days | < 100 ms | ~4 s |
| 90 days | < 50 ms | ~7 s |
| 365 days | < 50 ms | ~5 s |
At least in this setup, the inverted version scans the whole `run_after`
slice, so it seemed to get worse as the window widens.
The one thing that held up across every window (tens of ms) was a covering
index `(dag_id, state, run_after)`.
I know you wanted to avoid a new index.
Just flagging this in case it changes the trade-off.
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
