jason810496 opened a new issue, #76417: URL: https://github.com/apache/airflow/issues/76417
### Body - Caught by @Andrushika in https://github.com/apache/airflow/issues/75865#issuecomment-6622866800 ## What happens - When many write requests arrive at once (for example `POST /api/v2/variables` or `POST /api/v2/assets/events`), the API server stalls for 30s at a time. - Requests caught in a stall fail with 500 (`QueuePool limit ... TimeoutError`). - The Execution API runs on the same event loop, so running tasks can miss heartbeats while it is stalled. ## How to reproduce - Postgres 16, default DB pool (5 + 10 overflow), `[api] workers = 1`. - Send 600 `POST /api/v2/variables`, 24 at a time. - 3.3.2: 600/600 in 1.6s, no errors. - 3.4.0b2: intermittent. One run took 31.8s with 7 x 500; another took 1.8s with no errors. - 3.4.0rc1 (from the report): 146/600 done after 180s, 19 x 500, max latency 90s. - 3.4.0rc2: 403/600 done after 180s, 16 x 500, max latency 90s. - 3.4.0rc2 with #76416: 600/600 in 3.8s, no errors. ## Cause - The audit log dependency (`action_logging`) is `async`, but it commits the audit row through the route's sync session, so the commit runs on the event loop. - When the pool is full, that commit waits for a connection and blocks the whole event loop. - The connections are held by sync endpoints running in the threadpool. Each one is released only when its session is closed, and closing it needs the event loop. - So nothing is released until `pool_timeout` (30s) expires. ## When it was introduced - The sync commit inside the async dependency has been there since 3.2.0 (#58524). In 3.0 and 3.1 the dependency only added the row; it was committed when the session closed in the threadpool. - 3.3.2 has the same code but did not stall in these tests. - 3.4.0 increases load on the pool enough to hit it: sometimes in b2, on every run in rc1 and rc2. - The specific 3.4 change that raised the pool load is not isolated. Undoing only the `is_revoked` part of #74856 (back on the event loop) in rc2 halved the stall but did not remove it. ## Fix - #76416 writes the audit row on its own async session, so the audit log dependency makes no blocking database call on the event loop ### Committer - [x] I acknowledge that I am a maintainer/committer of the Apache Airflow project. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
