potiuk commented on PR #75086: URL: https://github.com/apache/airflow/pull/75086#issuecomment-6265468756
A question for executor maintainers before this leaves draft: this starts a short-lived thread for every workload put, and the edge cases around it (interruption, no live workers, failed puts) needed handling of their own. An alternative would be one long-running feeder thread per executor that owns all writes to the activity queue: `_process_workloads` would hand workloads to it through an in-process queue and keep draining results while it is busy. That removes the per-put thread start and the 50 ms polling, and most of the interruption handling moves into one place (stopping the feeder in `end()`). The cost is a thread living for the executor's whole lifetime, which needs care around forking workers while it runs. Would the long-running thread be the better approach here, or is the per-put thread acceptable for the volumes LocalExecutor handles? --- Drafted-by: Claude Code (Opus 5.5); reviewed by @potiuk before posting -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
