garvit-arora commented on issue #74544: URL: https://github.com/apache/airflow/issues/74544#issuecomment-6122218969
Thanks, this helps clarify the tradeoff. #39235 addressed scheduler hangs with CeleryExecutor and HTTP emission, so switching the listener to threads everywhere could reintroduce that problem. I have a local, unsubmitted patch that keeps `ProcessPoolExecutor` for the scheduler and uses a single `ThreadPoolExecutor` only in API-server workers. I’m checking the API-server detection and worker-lifecycle behavior before proposing it upstream. With your thread-pool workaround, have the orphaned processes and missed OpenLineage events both stopped during API-server worker refresh? Any observations from that run would help establish the regression case for a focused PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
