anmolxlight opened a new pull request, #74588: URL: https://github.com/apache/airflow/pull/74588
Fixes #74575. ## Problem `LocalExecutor` gave all workers one `SimpleQueue` as their input. In CPython, `SimpleQueue.get()` holds a shared read lock while it blocks on the pipe. If the idle worker holding that lock is SIGKILLed (for example by the OOM killer), nothing releases the lock. Every other worker then waits on it forever, and the executor keeps accepting work it can never run. Tasks stay `queued` while the scheduler reports healthy. ## Fix Give each worker its own input queue and dispatch each workload to a specific worker: - `_dispatch` picks an idle worker, spawns one under `parallelism`, or queues behind a busy one. - A dead worker is reaped together with its own queue, so it cannot block the others. - Dispatched-but-not-started keys are tracked separately (`_dispatched`), so a worker that dies before START still leaves the key for the scheduler's stuck-in-queued revoke, as before. - Terminal results are honored even for keys no longer in `running`, so a worker is not left excluded. ## Tests - New regression test `test_workload_runs_after_worker_holding_read_lock_is_killed`: kills all idle workers, then requires a later task to succeed. On `main` it fails with the symptom from the issue (task never leaves the queue, 30s timeout). With this fix it passes (~3.6s). - Rewrote `test_local_executor_check_workers.py` for the new reap contract. - `test_local_executor.py` and `test_local_executor_check_workers.py`: 65 passed, 1 skipped. - `test_base_executor.py` has 3 `test_debug_dump*` errors that also occur on unmodified `main`; unrelated. - `ruff check`, `ruff format --check`, and `mypy` on `local_executor.py` are clean. ## Notes This does not change behavior when workers stay alive. The per-worker queue is also what makes a killed worker harmless to the rest of the pool. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
