uranusjr commented on code in PR #71072:
URL: https://github.com/apache/airflow/pull/71072#discussion_r3793566842
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -2292,20 +2292,34 @@ def _create_dagruns_for_partitioned_asset_dags(self,
session: Session) -> set[st
)
or 0
)
+ # Distinct dag_ids across the *whole* backlog, not just this
tick's oldest-cap
+ # slice (`pending_apdrs`) — a Dag whose partitions haven't reached
the front of
+ # the FIFO queue yet would otherwise be missing from the log/audit
row until its
+ # turn comes up. Same WHERE clause as the count query above so the
two never drift.
+ backlog_dag_ids = set(
+ session.scalars(
+ select(AssetPartitionDagRun.target_dag_id)
+ .join(DagModel, DagModel.dag_id ==
AssetPartitionDagRun.target_dag_id)
+ .where(
+ AssetPartitionDagRun.created_dag_run_id.is_(None),
+ DagModel.is_stale.is_(False),
+ )
+ .distinct()
+ )
+ )
Review Comment:
I think we can merge this with the above (for `count()`) into one query,
something like
```python
backlogs_per_dag = dict(
session.execute(
select(AssetPartitionDagRun.target_dag_id, func.count())
.select_from(AssetPartitionDagRun)
.join(DagModel, DagModel.dag_id ==
AssetPartitionDagRun.target_dag_id)
.where(
AssetPartitionDagRun.created_dag_run_id.is_(None),
DagModel.is_stale.is_(False),
)
.group_by(AssetPartitionDagRun.target_dag_id)
.order_by(func.count().desc())
)
)
backlog_total = sum(n for _, n in backlogs_per_dag.items())
```
And since `backlogs_per_dag` is `dict[str, int]`, it can double as
`Collection[str]` and replace `backlog_dag_ids`.
Also, we probably should add an index on APDR to improve performance in the
query. This is probably fine for now, and the change can be a separate PR,
since the lack of an index didn’t seem to cause serious issues before this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
