seanghaeli opened a new pull request, #54250: URL: https://github.com/apache/airflow/pull/54250
(partial) fix for https://github.com/apache/airflow/issues/53359. When a new trigger gets started, we see messages like: ``` trigger **** starting triggerer's async thread was blocked for 2.41 seconds ``` A proposed fix in https://github.com/apache/airflow/pull/53454 is to sharing an async client across multiple connections. As summarized in https://github.com/aio-libs/aiobotocore/issues/1088, asyncio loops are not thread safe and each client runs a thread, so we cannot share connections like this. Due to this aiobotocore limitation, I don't think a complete fix is possible from our end. However, I do think we can at least minimize the amount of event loop blocking time by running long-lived clients. With this approach, we should only see event loop blocking once per unique `aws_conn_id`, rather than on every waiter call. I would appreciate feedback and verification on the following: (1) clients are only shared between triggers using the same aws_conn_id, (2) no client sharing occurs across different event loops, (3) credential isolation is maintained, and (4) comments on the memory tradeoff in storing potentially many clients. It is my understanding that if (1) and (2) are satisfied, then thread safety is maintained within aiobotocore. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
