uranusjr commented on code in PR #71072:
URL: https://github.com/apache/airflow/pull/71072#discussion_r4582164436


##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -400,6 +424,12 @@ def __init__(
         self._multi_team = conf.getboolean("core", "multi_team")
         self._dag_tags_in_metrics = conf.getboolean("metrics", 
"dag_tags_in_metrics", fallback=False)
         self._max_partition_dag_runs_per_loop = MAX_PARTITION_DAG_RUNS_PER_LOOP
+        # Edge-triggers the "partition Dag run cap reached" audit Log rows: 
the dag_ids whose row
+        # was committed during the current backlog episode. Cleared by
+        # `_report_partition_cap_backlog` when the backlog is at or below the 
cap.
+        # Process-local: each scheduler in an HA deployment writes its own 
audit row per Dag per
+        # episode; cross-process de-duplication would need a DB read per 
callback run.

Review Comment:
   Every scheduler runs this callback on its own timer. In an HA deployment 
with N schedulers, each affected Dag now always gets N rows per episode. The 
comment's reason for not deduplicating is weaker now that this runs once a 
minute rather than every loop.
   
   One option is to skip any dag_id that already has a “partition Dag run cap” 
reached row within the last few intervals. OK if duplicates are acceptable, but 
the comment should say each scheduler writes its own row.



##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -2521,28 +2552,22 @@ def _create_dagruns_for_partitioned_asset_dags(self, 
session: Session) -> set[st
         meantime, in which case the stale-fingerprint cleanup below drops it 
instead.
         """
         # Cap per-tick work so the scheduler transaction stays bounded and 
other
-        # scheduling work isn't starved. Remaining APDRs drain across 
subsequent ticks.
-        # FIFO is intentional: the oldest pending APDR fires first. A 
persistently
-        # unsatisfiable APDR at the head (e.g. broken mapper, upstream that 
will
-        # never arrive) blocks newer ones until an operator removes it or fixes
-        # the underlying mapper. We surface the stuck state rather than 
silently
-        # rotating past it.
-        # `with_row_locks(skip_locked=True)` mirrors the sibling ADRQ claim 
path:
-        # in HA two schedulers can otherwise both grab the same satisfied APDR
-        # and race the `created_dag_run_id` UPDATE, orphaning whichever DagRun
-        # loses. The `id` tiebreaker on `order_by` keeps LIMIT deterministic 
when
-        # two APDRs share a `created_at` under bulk asset-event ingestion.
-        # SQLite is single-writer and silently drops `FOR UPDATE`, which is 
fine.
+        # scheduling work isn't starved; the remainder drains over subsequent 
ticks.
+        # FIFO is intentional: a persistently unsatisfiable APDR at the head 
(e.g.
+        # broken mapper, upstream that will never arrive) blocks newer ones 
until an
+        # operator removes it or fixes the mapper, surfacing the stuck state 
rather
+        # than silently rotating past it.
+        # `with_row_locks(skip_locked=True)` mirrors the sibling ADRQ claim 
path: in
+        # HA two schedulers can otherwise both grab the same satisfied APDR 
and race
+        # the `created_dag_run_id` UPDATE, orphaning whichever DagRun loses. 
The `id`
+        # tiebreaker on `order_by` keeps LIMIT deterministic when two APDRs 
share a
+        # `created_at` under bulk asset-event ingestion. SQLite is 
single-writer and
+        # silently drops `FOR UPDATE`, which is fine.

Review Comment:
   This only changes wrapping now. Revert if it’s not intended.



##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -4332,6 +4357,61 @@ def _remove_unreferenced_triggers(self, *, session: 
Session = NEW_SESSION) -> No
             .execution_options(synchronize_session="fetch")
         )
 
+    def _report_partition_cap_backlog(self) -> None:
+        """Log and audit the Dags in the pending partitioned Dag run backlog 
above the per-tick cap."""
+        with create_session() as session:
+            backlogs_per_dag = {
+                dag_id: count
+                for dag_id, count in session.execute(
+                    select(AssetPartitionDagRun.target_dag_id, func.count())
+                    .select_from(AssetPartitionDagRun)
+                    .join(DagModel, DagModel.dag_id == 
AssetPartitionDagRun.target_dag_id)
+                    .where(*_build_pending_partitioned_apdr_filters())
+                    .group_by(AssetPartitionDagRun.target_dag_id)
+                )
+            }
+            backlog_total = sum(backlogs_per_dag.values())
+            if backlog_total <= self._max_partition_dag_runs_per_loop:
+                self._partition_cap_reported_dag_ids.clear()
+                return
+
+            # No SQL-side ORDER BY: it would have no dag_id tie-break, and 
relying on dict
+            # insertion order mirroring DB row order across backends (SQLite 
in tests vs
+            # Postgres/MySQL in production) isn't a contract worth trusting — 
sort explicitly.

Review Comment:
   The "No SQL-side ORDER BY" note just explains the `sorted()` call on the 
next line and is very AI-toned. Good candidate for human rewrite.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to