HydroYann opened a new issue, #76040:
URL: https://github.com/apache/airflow/issues/76040
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
3.3.2
### What happened and how to reproduce it?
Two scheduler instances can promote a later backfill DagRun before earlier
queued
runs even when the backfill's `max_active_runs=1` and its `sort_ordinal`
specifies
forward ordering. Exactly one run is running: the concurrency cap holds, but
the
requested order does not. The same inversion occurs with reverse sort
ordinals.
### Minimal executed result
One daily DAG, one backfill, three queued runs, and
`[scheduler] max_dagruns_per_loop_to_schedule=2`:
| Logical date (UTC) | Backfill sort ordinal | Final committed state |
| --- | --- | --- |
| 2025-08-13 | 1 | queued |
| 2025-08-14 | 2 | queued |
| 2025-08-15 | 3 | running |
Both DAG-level and backfill-level `max_active_runs` are 1. The DAG is
unpaused and
not stale. All three runs are eligible, and none is already running
initially.
### Reproducer
Attach `airflow-backfill-order-reproducer.zip` from this workspace when
submitting
this report. After extracting it, with Docker Compose, Make, and Python 3
installed:
```sh
cd airflow-backfill-order-reproducer
make reproduce
```
The Compose file pins both official images by digest. It creates an isolated
PostgreSQL database, publishes no ports, and does not use an existing Airflow
installation. Output is written under `evidence/batch-2`,
`evidence/batch-20`,
and `evidence/batch-46`. Exit 0 means all controls and assertions of the
reported
race matched; it does NOT mean Airflow's ordering is correct.
The deterministic schedule is:
1. Seed an actual serialized DAG/version and a backfill with ordered DagRun
associations.
2. Scheduler A enters the released `_start_queued_dagruns()` implementation.
Its real candidate query locks the first two DagRuns.
3. Pause A at entry to `_lock_backfills()`, before it acquires the Backfill
lock.
4. Scheduler B enters the same upstream method using a separate PostgreSQL
connection. Its query skips A's locked candidates and selects the third
run.
5. B acquires the Backfill lock, observes zero running runs, promotes the
third
run, and commits.
6. Resume A. It acquires the Backfill lock, observes the running third run,
and
leaves its earlier candidates queued because the concurrency cap is full.
7. Read final committed state with an independent session.
In the minimal forward run, A used PostgreSQL backend PID
`135` and B used `136`.
The candidate sets were disjoint. Independent snapshots show that every run
is
still queued after A's candidate locking, and the third run is already
running
after B commits while A remains paused.
The only scheduler instrumentation is a subclass that records the candidates
and
waits on a `threading.Event` immediately before calling the real
`super()._lock_backfills()`. There are no mocked queries, counts, lock
results,
DAG loads, or state updates, and the installed Airflow source is unchanged.
**Scope:** this is a focused queued-to-running scheduler/database
reproduction,
using two real SchedulerJobRunner objects and independent database sessions
in
one Python process. It does not launch two scheduler daemons, a webserver, or
workers. DAG/backfill metadata is arranged using upstream ORM models, not the
public backfill endpoint. Reverse ordering is represented by reverse
`BackfillDagRun.sort_ordinal` associations, not a CLI invocation. The DAG
contains
an EmptyOperator, but task execution is not needed or exercised.
### Controls and additional executed cases
Each case uses its own DAG, which is paused afterward to prevent
interference.
The serial controls run the same upstream promotion method without the forced
interleaving. All cases finish with exactly one running DagRun.
| Candidate batch limit | Case | Running logical date (UTC) | Running sort
ordinal | Ordering |
| --- | --- | --- | --- | --- |
| 2 | serial_forward | 2025-08-13 | 1 | preserved |
| 2 | race_forward | 2025-08-15 | 3 | violated |
| 2 | serial_reverse | 2025-08-15 | 1 | preserved |
| 2 | race_reverse | 2025-08-13 | 3 | violated |
| 20 | serial_forward | 2025-08-13 | 1 | preserved |
| 20 | race_forward | 2025-09-02 | 21 | violated |
| 20 | serial_reverse | 2025-09-02 | 1 | preserved |
| 20 | race_reverse | 2025-08-13 | 21 | violated |
| 46 | serial_forward | 2025-08-13 | 1 | preserved |
| 46 | race_forward | 2025-09-28 | 47 | violated |
| 46 | serial_reverse | 2025-09-28 | 1 | preserved |
| 46 | race_reverse | 2025-08-13 | 47 | violated |
The installed upstream default candidate batch limit is
`20`. Thus this does not depend on an
artificially small limit: the 20-row batch reproduces it too. With batch 46
and
47 daily runs, August 13 through September 27 remain queued while September
28
runs. Each case has one more queued candidate than the configured batch
limit.
[airflow-backfill-order-reproducer.zip](https://github.com/user-attachments/files/34187979/airflow-backfill-order-reproducer.zip)
### What you think should happen instead?
For a backfill with `max_active_runs=1`, the next eligible DagRun should
follow
the backfill's requested sort order. Temporary candidate row locks held by
another scheduler should not let a later ordinal bypass earlier queued runs.
The serial controls demonstrate this expected order in both directions.
If ordering is intentionally best-effort under HA, please clarify that
contract:
the backfill documentation exposes run ordering and the candidate query's
comments state that sort ordinals ensure the right order.
## Relevant implementation
The executed SQL locks candidates before the Backfill:
```sql
-- Candidate query, after ordering and LIMIT:
FOR NO KEY UPDATE OF dag_run SKIP LOCKED
-- Subsequent Backfill query:
SELECT ... FROM backfill
WHERE backfill.id IN (...) FOR UPDATE SKIP LOCKED;
```
Full executed SQL and parameters are in each `sql.jsonl`.
-
[`DagRun.get_queued_dag_runs_to_set_running()`](https://github.com/apache/airflow/blob/3.3.2/airflow-core/src/airflow/models/dagrun.py#L690-L773)
- [`SchedulerJobRunner._lock_backfills()` and
`_start_queued_dagruns()`](https://github.com/apache/airflow/blob/3.3.2/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L2689-L2808)
Related: #58807 introduced Backfill locking to protect `max_active_runs`.
This report concerns ordering, not exceeding the concurrency limit; that
limit
remains respected in every executed case. No fix is proposed in this report.
### Operating System
Debian GNU/Linux 12 (bookworm), Linux/aarch64 container
### Deployment
None
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
## Environment
- Airflow: 3.3.2, official `apache/airflow:3.3.2-python3.12` image.
- Python: 3.12.14.
- OS: Debian GNU/Linux 12 (bookworm), Linux/aarch64 container.
- Database: PostgreSQL 17.9 (Debian 17.9-1.pgdg12+1) on
aarch64-unknown-linux-gnu, compiled by gcc (Debian 12.2.0-14+deb12u1) 12.2.0,
64-bit.
- Transaction isolation: `read committed`.
- Deployment: disposable Docker Compose, no image customizations or extra
packages.
- Executor configured as LocalExecutor; not started or used by this
promotion-only test.
- Evidence recorded at: `2026-10-30T14:07:58.239979+00:00` (minimal batch).
- Source hashes, image digests, logs and committed row snapshots are
included in the attachment.
### Additional notes
The harness forces a valid interleaving to make the failure deterministic. It
does not measure how frequently the race occurs in a deployed HA cluster.
Other databases and unreleased Airflow versions were not tested.
The reproduction harness and this draft were prepared with generative-AI
assistance. Results above come from running the attached harness against the
released Airflow image and PostgreSQL, not from source analysis alone.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
