#35672: Using database connection pooling, threads do not return the connection
-------------------------------------+-------------------------------------
Reporter: André S. Hansen | Owner: (none)
Type: Bug | Status: new
Component: Database layer | Version: 5.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Description changed by André S. Hansen:
Old description:
> I am testing the new db connection pooling, which increased the
> performance greatly, typically lowering api response times from 30ms ->
> 20ms on my dev env.
>
> **However, after a while i started getting**
> {{{
> psycopg_pool.PoolTimeout : couldn't get a connection after 30.00 sec
> }}}
>
> Further investigation shows the reason was a Threaded Middleware orm
> interaction, like this simplified code.
>
> {{{
> class TimeMonitorMiddleware(object):
> batched_logs: List[RequestLog] = []
>
> def __call__(self, request: WSGIRequest) -> HttpResponse:
> # ...
>
> # Save batched logs
> if len(self.batched_logs) >= 50:
> Thread(target=RequestLog.objects.bulk_create,
> args=(batched_logs,)).start()
>
> return response
> }}}
>
>
>
> **Using settings**
> {{{
> {
> "ENGINE": os.environ.get("DB_ENGINE",
> "django.db.backends.postgresql"),
> "OPTIONS": {"pool": True}
> }
> }}}
>
> **Extra info:**
> - Python 3.12.1
> - Django 5.1
> - psycopg 3.2.1
> - psycopg-pool 3.2.2
> - Local docker image hosted postgresql database using image "postgres"
New description:
I am testing the new db connection pooling, which increased the
performance greatly, typically lowering api response times from 30ms ->
20ms on my dev env.
**However, after a while i started getting**
{{{
psycopg_pool.PoolTimeout : couldn't get a connection after 30.00 sec
}}}
Further investigation shows the reason was a Threaded Middleware orm
interaction, like this simplified code. My theory is that the thread
request and receives a connection from the pool, but do not return it.
{{{
class TimeMonitorMiddleware(object):
batched_logs: List[RequestLog] = []
def __call__(self, request: WSGIRequest) -> HttpResponse:
# ...
# Save batched logs
if len(self.batched_logs) >= 50:
Thread(target=RequestLog.objects.bulk_create,
args=(batched_logs,)).start()
return response
}}}
**Using settings**
{{{
{
"ENGINE": os.environ.get("DB_ENGINE", "django.db.backends.postgresql"),
"OPTIONS": {"pool": True}
}
}}}
**Extra info:**
- Python 3.12.1
- Django 5.1
- psycopg 3.2.1
- psycopg-pool 3.2.2
- Local docker image hosted postgresql database using image "postgres"
--
--
Ticket URL: <https://code.djangoproject.com/ticket/35672#comment:11>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/01070191653fb5e6-746b0f7a-f66f-42c9-a3e0-059132bb3cbf-000000%40eu-central-1.amazonses.com.
