seanmuth opened a new pull request, #72730: URL: https://github.com/apache/airflow/pull/72730
\`DagRun.get_running_dag_runs_to_examine\` orders RUNNING DagRuns by \`BackfillDagRun.sort_ordinal\` (nulls first) ahead of everything else. Since every non-backfill RUNNING DagRun has no matching \`BackfillDagRun\` row (and therefore a NULL \`sort_ordinal\`), it sorts strictly ahead of every backfill DagRun regardless of how long that backfill run has been waiting. Combined with the per-loop \`LIMIT max_dagruns_per_loop_to_schedule\` (default 20), this means a backfill DagRun's task instances can never be created or scheduled at all once the number of concurrently RUNNING non-backfill DagRuns meets or exceeds that limit. This isn't just deprioritization, it's permanent starvation for as long as that condition holds, since fresh non-backfill DagRuns keep arriving to refill the tier ahead of it on every loop. Reported independently multiple times as backfill task instances stuck permanently in \`None\`/\`no_status\` with \`last_scheduling_decision\` also stuck at \`NULL\` (#58554), and reproduced deterministically on a production deployment with a few hundred concurrently running DagRuns. Deprioritizing backfills behind currently-scheduled work is intentional (a large backfill shouldn't starve time-sensitive scheduled runs), but that's only meant to apply while a backfill DagRun is still \`QUEUED\` waiting for a \`max_active_runs\` slot — that ordering is handled separately by \`get_queued_dag_runs_to_set_running\`, via the backfill's own \`sort_ordinal\`/\`max_active_runs\`. Once a DagRun is actually \`RUNNING\`, its backfill's internal ordering has no bearing on how the scheduler should prioritize examining it against other RUNNING DagRuns. This removes \`sort_ordinal\` from this query entirely, so a RUNNING backfill DagRun competes purely on \`last_scheduling_decision\` (nulls first) like every other RUNNING DagRun — meaning a freshly-promoted backfill run, which has never been scheduled and so has a NULL \`last_scheduling_decision\`, sorts to the front rather than being starved at the back. Thanks to @zh-jn for independently diagnosing this exact root cause on #58554. Added a regression test that reproduces the starvation directly: without this fix it fails (a backfill DagRun's \`dag_id\` never appears in the examine batch when enough ordinary DagRuns are running to fill the limit ahead of it); with the fix it passes. Confirmed the rest of the existing backfill test suite (including the tests covering the *intentional* \`sort_ordinal\` ordering in the separate queued→running promotion path) still passes unchanged. closes: #58554 ##### Was generative AI tooling used to co-author this PR? - [X] Yes (Claude, Sonnet 5) <!-- Generated-by: Claude Sonnet 5 following https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
