hussein-awala opened a new pull request, #72873: URL: https://github.com/apache/airflow/pull/72873
An expensive mapped-task expansion currently keeps scheduling locks on every Dag run selected in the same batch. For example, an update to run A can wait for a 10,000-instance expansion in run B even after the scheduler has finished examining A. Select scheduling candidates without locking the whole batch, then claim, revalidate, schedule, and commit each run separately. A scheduler skips candidates that another scheduler has locked or examined in the meantime. Database retries apply only to the current run, and callbacks for committed runs are delivered before scheduling the next candidate. The intended benefit is shorter lock waits across Dag runs. This adds a claim query and a transaction boundary per candidate, and prevents some updates from being batched together. It can increase database round trips and commit overhead. Multiple schedulers can also reach other candidates sooner, potentially increasing concurrent expansion work. This does not bound aggregate database load or claim improved scheduler throughput. Task-instance creation remains eager, including full expansion of a mapped task. This PR does **not** defer creation across scheduler loops or impose a creation budget. Scheduling changes for each running Dag run become visible when that run commits, before the scheduler finishes the remaining candidates. Existing mapped-task semantics and `task_instance_mutation_hook` behavior are preserved. Dag-run creation and the pool-lock critical section for queuing tasks retain their existing transaction boundaries. ### PostgreSQL 14 contention experiment Local PostgreSQL 14.24, 20 running Dags, a real 10,000-instance expansion in the middle of the batch, and a concurrent connection updating a previously examined run. Three repetitions per variant; no artificial delay holds locks open. | Measurement | Batch transaction | Transaction per run | | --- | ---: | ---: | | Conflicting update latency, median | 2.431 s | 1.251 ms | | Conflicting update latency, range | 2.234–2.829 s | 1.172–2.600 ms | | Workload duration, median | 2.430 s | 2.409 s | | Scheduler commits | 1 | 20 | | Scheduler driver SELECT calls | 6 | 26 | | Scheduler driver UPDATE calls | 3 | 21 | | Scheduler driver INSERT calls | 1 | 1 | The baseline writer waited on `Lock/transactionid`, with the scheduler PID reported by `pg_blocking_pids`. With separate transactions, that update finished during the expansion and had no blocker. Cluster WAL deltas were approximately 11 MB for both variants; these measurements do not establish reduced WAL production. This is a controlled transaction-boundary experiment using the existing `TaskMap.expand_mapped_task`, not a full scheduler throughput benchmark. Other scheduling work is replaced by a timestamp update. Driver calls were counted with SQLAlchemy cursor events; one `executemany` call may represent multiple server operations. Broader workload measurements are still needed to assess the extra query/commit cost under multiple schedulers. ### Validation - PostgreSQL 14: 42 transaction, query-count, and scheduler regressions passed; the final future-run fixture and six contention runs also passed. - MySQL 8.0: all 15 targeted regressions passed after correcting the future-run test fixture's timestamp range and UTC handling. - SQLite: 13,039 passed across API (3,949), Other (2,016), CLI (930), Serialization (593), Always (2,105), models (1,464), triggerer/dependencies/utilities (1,088), and the scheduler/Core/executor subset (894). - The combined Core run was killed with exit 137 on the local 1.9 GB VM. All Core tests subsequently completed in smaller processes. Standard backend-specific, system, and long-running exclusions still apply. - Fast static checks, including mypy, commit hooks, manual checks, and CI test selection completed. The full local suites ran before the final rebase onto current `main`. Upstream changes did not touch the six files in this PR; the rebased patch is identical. Static checks were repeated after rebasing. Related: #71967. This change starts from `main` and has no dependency on that PR. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Codex (GPT-6) Generated-by: Codex (GPT-6) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- Drafted-by: Codex (GPT-6) (no human review before posting) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
