tirkarthi commented on PR #61623:
URL: https://github.com/apache/airflow/pull/61623#issuecomment-3866315992
As per my understanding there are only 5 connections in the pool by default
and with concurrent requests and db calls being made synchronously in
requires_access_dag and in the endpoint other requests might need to wait based
on max_overflow.
```
Prompt : If I have connection pool as 5 in sqlalchemy and create 10 sessions
concurrently then do the last 5 sessions wait for the initial 5 sessions to be
returned back to the pool
Gemini output :
The short answer is: not necessarily. It depends entirely on your
max_overflow setting. By default,
SQLAlchemy doesn't just make those extra 5 sessions sit in a queue; it often
creates "overflow" connections
to handle the spike.
Here is the breakdown of how SQLAlchemy manages this scenario:
1. The pool_size vs. max_overflow
When you define a pool, SQLAlchemy uses two primary parameters to determine
how to handle concurrent requests:
pool_size (5 in your case): The number of "permanent" connections kept
open in the pool.
max_overflow (Default is 10): The number of additional connections the
pool is allowed to create if the pool_size is exhausted.
2. What happens to your 10 sessions?
If you are using the default settings (pool_size=5, max_overflow=10):
Sessions 1–5: Grab the 5 "checked-in" connections from the pool
immediately.
Sessions 6–10: Since the pool is full, the engine looks at max_overflow.
Because 5 is less than the default limit of 10, it creates 5 new temporary
connections.
Result: All 10 sessions run concurrently. No one waits.
3. When do sessions actually wait?
The "waiting" behavior only kicks in if you exceed pool_size + max_overflow.
If you set max_overflow=0 or if you tried to open 16 sessions with default
settings:
The 16th session would hit the limit.
It would wait for a connection to be returned to the pool.
This wait is governed by pool_timeout (default is 30 seconds). If no
connection becomes free in that time, SQLAlchemy throws a TimeoutError
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
