Andrushika commented on issue #75865:
URL: https://github.com/apache/airflow/issues/75865#issuecomment-6622866800
Seems like I found another regression in 3.4.0rc2... Might be worth another
look before release.Seems like I found another regression in 3.4.0rc2... Might
worth another look before release.
Issue: the API server stalls when it gets many write requests at the same
time. 3.3.2 does not have this problem.
Setup: `airflow standalone`, Postgres 16, default DB pool (5 + 10 overflow),
`[api] workers = 1`. I sent 600 `POST /api/v2/assets/events`, with 24 requests
at the same time.
| version | finished | time | 500 errors | max latency |
|---|---|---|---|---|
| 3.3.2 | 600 / 600 | 4.9 s | 0 | 0.54 s |
| 3.4.0b2 | 600 / 600 | 15 s | 0 | 1.3 s |
| 3.4.0rc1 | 146 / 600 (stopped at 180 s) | - | 19 | 90 s |
| 3.4.0rc2 | 600 / 600 | 136 s | 10 | 60.7 s |
`POST /api/v2/variables` gives the same result on rc2. During the stall, one
running task also failed because its heartbeat timed out.
It looks like the audit log commit in `log_action` blocks the event loop
when the DB pool is busy. Moving it to a thread (`await
run_in_threadpool(session.commit)`) fixed it in my test.
I think it's worth taking a look before the release, but it needs many write
requests at the same time, so I am not quite sure if this is enough for -1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
