safaehar opened a new issue, #77201:
URL: https://github.com/apache/airflow/issues/77201

   ### Description
   
   Could the scheduler reduce how long it holds locks on RUNNING DagRuns, so 
clearing tasks in one run is less dependent on the scheduling cost of other 
runs in the same batch?
   
   On Airflow 3.3.2 with PostgreSQL, task-clear API requests have returned HTTP 
500 when an update to `dag_run` reached our configured 10-second 
`statement_timeout`. PostgreSQL showed the update waiting on another 
transaction's lock. This was a statement timeout during a lock wait, not a 
confirmed deadlock. The blocking backend was not conclusively mapped to a 
scheduler in that capture; the scheduler code below identifies a concrete 
contention path to investigate.
   
   Our scheduler batch limit was `scheduler.max_dagruns_per_loop_to_schedule = 
120` (above the default of 20). This request concerns the lifetime of those 
locks, including smaller batches with expensive scheduling decisions.
   
   ### Current mechanism
   
   In 3.3.2:
   
   1. 
[`DagRun.get_running_dag_runs_to_examine`](https://github.com/apache/airflow/blob/3.3.2/airflow-core/src/airflow/models/dagrun.py#L654)
 selects and locks the batch with `SKIP LOCKED`.
   2. 
[`SchedulerJobRunner._schedule_all_dag_runs`](https://github.com/apache/airflow/blob/3.3.2/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L2811)
 checks the selected runs sequentially and commits after the loop.
   3. 
[`clear_task_instances`](https://github.com/apache/airflow/blob/3.3.2/airflow-core/src/airflow/models/taskinstance.py#L357)
 updates DagRun fields such as `clear_number` and `queued_at`, including for 
runs that are already RUNNING.
   
   Consequently, a clear targeting run A can wait for the remainder of the 
batch even after the scheduler has finished checking A. `SKIP LOCKED` lets 
other schedulers skip those rows; it does not prevent a clear's update from 
waiting.
   
   Source inspection of `main` at `3b35135afe9549e99088c4d557c62e5deafb125b` 
still shows the [batch lock 
acquisition](https://github.com/apache/airflow/blob/3b35135afe9549e99088c4d557c62e5deafb125b/airflow-core/src/airflow/models/dagrun.py#L815)
 and [commit after the scheduling 
loop](https://github.com/apache/airflow/blob/3b35135afe9549e99088c4d557c62e5deafb125b/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L3368).
 We have not reproduced the API failure on that revision.
   
   ### Use case/motivation
   
   Keep interactive task clears responsive while the scheduler processes 
expensive DagRuns, without compromising HA scheduler coordination or 
transactional consistency.
   
   Would smaller independently acquired/committed batches, or another approach 
to bounding lock lifetime, be viable? Simply committing after each run while 
retaining a preselected batch would release locks on all remaining rows, so 
that alone would not preserve the current safety guarantees.
   
   Reducing the configured batch limit is a possible mitigation, but not a 
bound on transaction duration. We do not yet have a controlled before/after 
result proving that it resolves these timeouts.
   
   ### Suggested regression test
   
   Proposed PostgreSQL concurrency test, not an executed reproduction:
   
   1. Have the scheduler acquire two RUNNING runs A and B through its normal 
selection path.
   2. Allow its scheduling decision for A to finish, then pause processing B at 
a test barrier.
   3. From a separate session, clear a failed task in A using the normal clear 
path and a short statement timeout.
   4. Check whether the clear waits until B is released and the scheduler batch 
commits.
   
   An improved mechanism should bound this wait while ensuring another 
scheduler cannot process unowned or stale preselected runs.
   
   ### Related issues
   
   - #72730 fixes starvation when selecting RUNNING backfills; it does not 
change batch-level lock lifetime.
   - #66817 concerns stale session objects and cross-scheduler deadlocks; this 
request concerns lock duration and interactive clear latency.
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to