shivaam commented on PR #64110: URL: https://github.com/apache/airflow/pull/64110#issuecomment-4115204774
Thanks for looking into this! I've been investigating this issue as well and wanted to share some findings that might be relevant. The root cause appears to be the `inspect()` call itself (line 224) rather than the hostname matching logic. When `celery_app.control.inspect().active_queues()` runs before `worker_main()`, it opens broker connections and initializes internal connection pools (`kombu.pools`). When `worker_main()` subsequently forks prefork pool workers, this pre-initialized state interferes with the worker's internal task dispatch — causing tasks to get stuck in RESERVED. I verified this on a live Airflow 3.2 instance: removing the `inspect()` call entirely or cleaning up global pool state with `kombu.pools.reset()` afterward both resolve the issue. More details in my comment on the issue: https://github.com/apache/airflow/issues/59707#issuecomment-4108337693 Could you share more about your testing setup? I want to make sure I'm not missing something — if the matching logic change alone fixed it in your environment, that would be an important data point. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
