#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------
Reporter: Marcin Nowak | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
Description changed by Marcin Nowak:
Old description:
> Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
> connections.
> ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0
> UWSGi threads are enabled (`--enable-threads`)
> Python 3.8.1 and 3.7.3
>
> After few hours all posgtress connections are used (too many connections
> error).
> pg_stat_activity is reporting idle connections with part of queries
> visible.
>
> I would try to deliver more details later.
>
> [[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]
>
> Facts:
> - setting up CONN_MAX_AGE > 0 does not help
> - CONN_MAX_AGE = 30, starting `runserver` and making many calls with `ab
> -c 10 -n 100 http://127.0.0.1:8000/` uses all available connections and
> ends with error 500 (due to no available connections)
> - CONN_MAX_AGE = 30, starting `runserver --nothreading` and making calls
> with `ab -c 100 -n 100 http://127.0.0.1:8000/` works fine
> - CONN_MAX_AGE=30 + runserver + ab scenario is reproducible on clean
> install. Called view must access database.
> - CONN_MAX_AGE=0 + runserver + ab -> cannot reproduce.
> - Django 2.x has similar issue for CONN_MAX_AGE=30, but can reuse
> connections (still there are several errors on high traffic, but app
> server can "recover" and handle bunch of incoming requests)
> - Looks like Django 3.x cannot reuse connections
>
> `ab -c 10 -n 200 ` Django 3.0.4:
> Complete requests: 200
> Failed requests: 108
>
> `ab -c 10 -n 1000` Django 2.2.11:
> Complete requests: 1000
> Failed requests: 37
New description:
Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
connections.
ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0
UWSGi threads are enabled (`--enable-threads`)
Python 3.8.1 and 3.7.3
After few hours all posgtress connections are used (too many connections
error).
pg_stat_activity is reporting idle connections with part of queries
visible.
I would try to deliver more details later.
[[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]
----
Facts:
- setting up CONN_MAX_AGE > 0 does not help
- CONN_MAX_AGE = 30, starting `runserver` and making many calls with `ab
-c 10 -n 100 http://127.0.0.1:8000/` uses all available connections and
ends with error 500 (due to no available connections)
- CONN_MAX_AGE = 30, starting `runserver --nothreading` and making calls
with `ab -c 100 -n 100 http://127.0.0.1:8000/` works fine
- CONN_MAX_AGE=30 + runserver + ab scenario is reproducible on clean
install. Called view must access database.
- CONN_MAX_AGE=0 + runserver + ab -> cannot reproduce.
- Django 2.x has similar issue for CONN_MAX_AGE=30, but can reuse
connections (still there are several errors on high traffic, but app
server can "recover" and handle bunch of incoming requests)
- Looks like Django 3.x cannot reuse connections
----
Runserver (with threads) test results:
`ab -c 10 -n 200 ` Django 3.0.4:
Complete requests: 200
Failed requests: 108
`ab -c 10 -n 1000` Django 2.2.11:
Complete requests: 1000
Failed requests: 37
----
There may be two separate cases:
1. not reusing connection when CONN_MAX_AGE is greater than 0
2. leaking (not closing?) connections on uwsgi with enabled threads and
CONN_MAX_AGE set to 0
--
--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:7>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/065.719111c7a477342a64cbb8afd2409889%40djangoproject.com.
