potiuk opened a new pull request, #75086: URL: https://github.com/apache/airflow/pull/75086
related: #70526, #70545 #70545 drains the result queue once before each `activity_queue.put()`, but the put itself can still block forever. A worker can write a result between the drain and the put; if the activity pipe is full at that moment the put blocks, and the worker keeps reading workloads and writing results until the result pipe is full too, so each side waits on the other. Linux pipes free space only a whole page at a time, so this shows on loaded Linux CI runners (the new `test_process_workloads_progresses_under_concurrent_result_refill` hung at its 30s limit in job 117757985706 on an unrelated PR) and not on macOS. The put now runs on a short-lived helper thread while the main thread keeps reading results until it completes, so the result pipe cannot fill up behind a blocked put. The workload is registered as running before it is sent, so a "running" result read while the put is in flight is not dropped. If the dispatch is interrupted (SIGTERM raises `SystemExit`), draining continues until the pending put completes before re-raising, so `end()` does not queue its shutdown messages behind an abandoned writer. If no worker is alive the interruption is re-raised at once, since nothing could take the pending write and `end()` sends nothing. Trade-offs worth a look from executor maintainers: - one thread start per dispatched workload; the worst case for dispatching 200 workloads went from about 8 ms to about 50 ms, - `end()` still sends its shutdown messages with a plain blocking put; left as is to keep this change focused, - an `mp.Queue` (background writer thread, risky to fork workers under) or non-blocking pipe writes (multiprocessing internals) were considered and rejected. This could not be reproduced on macOS, where pipes free space byte by byte; the Linux behaviour was checked against a model of the kernel's pipe rules, where the previous code still deadlocked in 50-97% of interleavings with a lagging worker and the new code in none. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5.5) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
