#28062: Using iterators behind pgBouncer leads to non-existent cursor erros
-------------------------------------+-------------------------------------
Reporter: Sergey | Owner: nobody
Fursov |
Type: | Status: new
Uncategorized |
Component: Database | Version: 1.11
layer (models, ORM) |
Severity: Normal | Keywords:
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
Hi!
In our application stack we connect from our web/background workers to
postgres database through dedicated pgBouncer service.
pgBouncer is running in transaction pooling mode and django connections
work in autocommit mode.
After upgrading to 1.11 version with new server-side cursors for queryset
iterators, many of our DB requests were started failing with "Cursor
_django_curs_<id> does not exist'.
According to pgBouncer docs (https://wiki.postgresql.org/wiki/PgBouncer),
it doesn't support using WITH HOLD cursors with transaction pooling mode.
But it doesn't seems reasonable to run pgBouncer in session pooling mode.
We can just switch to direct persistent connections to postgres DB, but in
this case we have to increase minimum number of DB connection to number of
our workers (from 5 to ~30-40) to prevent blocking requests and this will
violate recommendation to have (core_count * 2) + effective_spindle_count
max connections to DB
(https://wiki.postgresql.org/wiki/Number_Of_Database_Connections).
Introducing server-side cursors seems like breaking change in our case
(and i guess many django apps use pgBouncer behind Postgres DB)
So is there some recommendations how efficiently work with postgres DB in
django 1.11?
Looks like another possible option is wrapping all iterator() calls in
their own transactions, but django internally use cursors in several
places (e.g. in ModelChoiceIterator).
I think this possible problem should be documented in some way, and it
would be great to describe some options for projects to migrate to new
version.
Thanks,
Sergey
--
Ticket URL: <https://code.djangoproject.com/ticket/28062>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/049.1cca1d2b367f3b14e87529cb020923c6%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
