dor-bernstein opened a new issue, #70614:
URL: https://github.com/apache/airflow/issues/70614
### Apache Airflow version
3.2.0+ (observed on 3.2.2)
### What happened
The scheduler intermittently stalls: `scheduler.critical_section_duration`
climbs while scheduler CPU drops and task scheduling starves. Inspecting the
metadata DB during a stall shows scheduler statements of the form `UPDATE dag
SET exceeds_max_non_backfill = … WHERE dag.dag_id = …` blocked with
`wait_event_type = Lock` / `wait_event = transactionid`, waiting on the DAG
processor's parse transaction.
### Root cause
Before 3.2, the scheduler evaluated `max_active_runs` in memory and did not
write the `dag` row during dagrun scheduling. **#60006 ("Separate 'next dag
run' from 'max active runs'", merge commit
`32bc0119b408c9a37841edd021b646324a02745e`, shipped in 3.2.0)** added the
`dag.exceeds_max_non_backfill` column (migration
`0097_3_2_0_add_exceeds_max_runs_flag_to_dag_model`) and made the scheduler
**write it to the `dag` row** during dagrun scheduling:
- Write:
`airflow-core/src/airflow/jobs/scheduler_job_runner.py::_set_exceeds_max_active_runs`
(called from `_create_dag_runs` and `_start_queued_dagruns`).
That write now competes with a pre-existing lock held by the DAG processor
for the entire per-file parse transaction:
- Lock:
`airflow-core/src/airflow/dag_processing/collection.py::DagModelOperation.find_orm_dags`
→ `with_row_locks(select(DagModel) …)` i.e. `SELECT … FOR UPDATE` on every
`dag` row for a parsed file, held through `update_dags` + serialization +
permission sync until the transaction commits.
So the scheduler's `UPDATE dag SET exceeds_max_non_backfill = …` blocks on
the DAG processor's `FOR UPDATE`. This is amplified when a single parsed file
defines many DAGs (e.g. dynamically generated DAGs), because that parse
transaction can run for minutes — holding the `dag`-row locks the whole time
and stalling the scheduler loop.
### How to reproduce it (conceptually)
1. Use the FAB auth manager and generate many DAGs from a single DAG file.
2. While the DAG processor is parsing that file (holding `SELECT … FOR
UPDATE` on its `dag` rows), have the scheduler attempt to update
`exceeds_max_non_backfill` for one of those DAGs.
3. Observe the scheduler statement blocked on `Lock`/`transactionid`, and
`critical_section_duration` rising while scheduler CPU falls.
### Suggested direction
Reduce the parse transaction's `dag`-row lock hold and/or the scheduler's
contention on it. As a partial step, per-DAG permission sync (another writer in
the same parse transaction) can be made to honor `[fab] update_fab_perms` —
proposed in #70589. A more complete fix would shorten/scope the `find_orm_dags`
lock or the scheduler's `dag`-row write.
### Anything else
Related: #23232 (pluggable appless security manager).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
