Re: [Django] #37000: cursor_iter relies on GC for server-side cursor cleanup, causing transaction abort after savepoint rollback

[Django]

#37000: cursor_iter relies on GC for server-side cursor cleanup, causing
transaction abort after savepoint rollback
-+-
 Reporter:  Ratskó László|Owner:  Vishy
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  iterator, server-| Triage Stage:  Accepted
  side-cursor, savepoint, psycopg3   |
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Vishy):

 * owner:  Vidhi Singh => Vishy

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/django-updates/0107019eb17a5286-3f88dd62-9283-4dc1-a475-7bb4468f4055-00%40eu-central-1.amazonses.com.

Re: [Django] #37000: cursor_iter relies on GC for server-side cursor cleanup, causing transaction abort after savepoint rollback

[Django]

#37000: cursor_iter relies on GC for server-side cursor cleanup, causing
transaction abort after savepoint rollback
-+-
 Reporter:  Ratskó László|Owner:  Vidhi
 |  Singh
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  iterator, server-| Triage Stage:  Accepted
  side-cursor, savepoint, psycopg3   |
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Comment (by Vidhi Singh):

 I’ve worked on this issue and opened a PR to make server-side cursor
 closing more
 resilient when iterator() is interrupted inside transaction.atomic().

 The fix ensures that delayed cursor.close() calls do not raise
 InvalidCursorName
 and break the transaction.

 PR: https://github.com/django/django/pull/
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/django-updates/0107019d2592e44a-39de1fb6-211a-4f0e-92fa-401712ce4b65-00%40eu-central-1.amazonses.com.

Re: [Django] #37000: cursor_iter relies on GC for server-side cursor cleanup, causing transaction abort after savepoint rollback

[Django]

#37000: cursor_iter relies on GC for server-side cursor cleanup, causing
transaction abort after savepoint rollback
-+-
 Reporter:  Ratskó László|Owner:  Vidhi
 |  Singh
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  iterator, server-| Triage Stage:  Accepted
  side-cursor, savepoint, psycopg3   |
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Vidhi Singh):

 * owner:  (none) => Vidhi Singh
 * status:  new => assigned

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/django-updates/0107019d248925ef-3c49502f-47d7-4c20-8fdc-b151f1d436de-00%40eu-central-1.amazonses.com.

Re: [Django] #37000: cursor_iter relies on GC for server-side cursor cleanup, causing transaction abort after savepoint rollback

[Django]

#37000: cursor_iter relies on GC for server-side cursor cleanup, causing
transaction abort after savepoint rollback
-+-
 Reporter:  Ratskó László|Owner:  (none)
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  iterator, server-| Triage Stage:  Accepted
  side-cursor, savepoint, psycopg3   |
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Comment (by Vidhi Singh):

 Can I work on this?
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/django-updates/0107019d1ebb5f4a-ca35e322-1d69-4d4e-a791-e06d9f52dde1-00%40eu-central-1.amazonses.com.

Re: [Django] #37000: cursor_iter relies on GC for server-side cursor cleanup, causing transaction abort after savepoint rollback

[Django]

#37000: cursor_iter relies on GC for server-side cursor cleanup, causing
transaction abort after savepoint rollback
-+-
 Reporter:  Ratskó László|Owner:  (none)
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  iterator, server-| Triage Stage:  Accepted
  side-cursor, savepoint, psycopg3   |
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Comment (by Simon Charette):

 I was able to reproduce with the following test

 {{{#!diff
 diff --git a/tests/backends/postgresql/test_server_side_cursors.py
 b/tests/backends/postgresql/test_server_side_cursors.py
 index 9a6457cce6..900a2dc229 100644
 --- a/tests/backends/postgresql/test_server_side_cursors.py
 +++ b/tests/backends/postgresql/test_server_side_cursors.py
 @@ -1,9 +1,10 @@
  import operator
 +import gc
  import unittest
  from collections import namedtuple
  from contextlib import contextmanager

 -from django.db import connection, models
 +from django.db import connection, models, transaction
  from django.db.utils import ProgrammingError
  from django.test import TestCase
  from django.test.utils import garbage_collect
 @@ -154,3 +155,12 @@ class ServerSideCursorsPostgres(TestCase):
  # most likely need to be adapted.
  with self.assertRaises(ProgrammingError):
  perform_query()
 +
 +def test_transaction_cursor_closing(self):
 +with self.assertRaises(ValueError), transaction.atomic():
 +persons = Person.objects.iterator()
 +next(persons)
 +raise ValueError
 +del persons
 +gc.collect()
 +list(Person.objects.all())
 }}}

 Without the last query we still get presented with a resource warning

 {{{#!python
 Exception ignored while closing generator :
 Traceback (most recent call last):
   File "/django/source/django/db/models/sql/compiler.py", line 2266, in
 cursor_iter
 cursor.close()
   File "/usr/local/lib/python3.14/site-
 packages/psycopg/_server_cursor.py", line 73, in close
 self._conn.wait(self._close_gen())
   File "/usr/local/lib/python3.14/site-packages/psycopg/connection.py",
 line 484, in wait
 return waiting.wait(gen, self.pgconn.socket, interval=interval)
   File "psycopg_binary/_psycopg/waiting.pyx", line 241, in
 psycopg_binary._psycopg.wait_c
   File "/usr/local/lib/python3.14/site-
 packages/psycopg/_server_cursor_base.py", line 163, in _close_gen
 yield from self._conn._exec_command(query)
   File "/usr/local/lib/python3.14/site-
 packages/psycopg/_connection_base.py", line 483, in _exec_command
 raise e.error_from_result(result, encoding=self.pgconn._encoding)
 psycopg.errors.InvalidCursorName: cursor
 "_django_curs_140716552528704_sync_1" does not exist
 Exception ignored while calling deallocator :
 Traceback (most recent call last):
   File "/usr/local/lib/python3.14/site-
 packages/psycopg/_server_cursor_base.py", line 55, in __del__
 __warn(
 ResourceWarning:  was deleted while still open. Please use 'with'
 or '.close()' to close the cursor properly
 }}}

 The approach in [https://github.com/django/django/pull/20975/ the Claude
 generated MR] has merit but it's too naive in the sense that it tries to
 close all server side cursors on each savepoint rollbacks but these can be
 nested and we likely want to keep weakrefs to cursors as to avoid memory
 leaks.

 An ideal solution would weak track cursors by savepoint ID and only do so
 in nested transactions. I'll note that even if we do so failing to consume
 iterators returned by `QuerySet.iterator` entirely will always incur a
 `ResourceWarning` (transaction or not) so maybe we should also (or
 instead) adjust
 
[https://docs.djangoproject.com/en/6.0/ref/models/querysets/#django.db.models.query.QuerySet.iterator
 the documentation] to mention that the following pattern should be used
 instead?

 {{{#!python
 import contextlib

 def export_items():
 try:
 with (
 transaction.atomic(),
 contextlib.closing(Item.objects.iterator()) as items
  ):
 for item in items:
 if item.value == "bad":
 raise ValueError("Export failed")
 process(item)
 except ValueError:
 Item.objects.create(value="error logged")
 }}}

 to enforce explicit closing of iterators? Maybe we should even have the
 object returned by `QuerySet.iterator` be a context closing of itself so
 we can document

 {{{#!python
 de

Re: [Django] #37000: cursor_iter relies on GC for server-side cursor cleanup, causing transaction abort after savepoint rollback

[Django]

#37000: cursor_iter relies on GC for server-side cursor cleanup, causing
transaction abort after savepoint rollback
-+-
 Reporter:  Ratskó László|Owner:  (none)
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  iterator, server-| Triage Stage:  Accepted
  side-cursor, savepoint, psycopg3   |
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Simon Charette):

 * cc: Simon Charette (added)
 * owner:  Huwaiza => (none)
 * stage:  Ready for checkin => Accepted
 * status:  assigned => new

Comment:

 Huwaiza please refrain from assigning ticket to yourself mid conversation
 with reports and changing ticket status -- RFC is a status meant that the
 patch is ready to be checked in and there's no patch here.

 Refer to [https://docs.djangoproject.com/en/6.0/internals/contributing
 /triaging-tickets/#:~:text=Ready%20for%20checkin%E2%80%9D.%20You the
 triaging ticket documentation].

 -

 Thanks for confirming Ratskó, I'll see if I can write a regression test
 that reproduces the issue as I've been unable to do so yet. #28062 is a
 related ticket, savepoint rollback might have been a cause all along.
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/django-updates/0107019d1ce67266-9d2bd1af-4b18-4a97-aa84-224e3c06d9bc-00%40eu-central-1.amazonses.com.

Re: [Django] #37000: cursor_iter relies on GC for server-side cursor cleanup, causing transaction abort after savepoint rollback

[Django]

#37000: cursor_iter relies on GC for server-side cursor cleanup, causing
transaction abort after savepoint rollback
-+-
 Reporter:  Ratskó László|Owner:  Huwaiza
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  iterator, server-| Triage Stage:  Ready for
  side-cursor, savepoint, psycopg3   |  checkin
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Huwaiza):

 * cc: Huwaiza (added)
 * owner:  (none) => Huwaiza
 * stage:  Accepted => Ready for checkin
 * status:  new => assigned

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/django-updates/0107019d1c9ed276-070ab3d2-6a6e-4e3a-bc7a-740fe963bb47-00%40eu-central-1.amazonses.com.

Re: [Django] #37000: cursor_iter relies on GC for server-side cursor cleanup, causing transaction abort after savepoint rollback

[Django]

#37000: cursor_iter relies on GC for server-side cursor cleanup, causing
transaction abort after savepoint rollback
-+-
 Reporter:  Ratskó László|Owner:  (none)
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  iterator, server-| Triage Stage:  Accepted
  side-cursor, savepoint, psycopg3   |
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Comment (by Ratskó László):

 Thanks for accepting the ticket!

 To clarify: yes, this is specifically a nested transaction.atomic() issue
 (savepoints). We have ATOMIC_REQUESTS = True, so every request is already
 in a transaction, and any transaction.atomic() in our code creates a
 savepoint. Sorry for not mentioning that in the report.

 Our pattern follows the documented approach — catching exceptions outside
 the atomic block:


 {{{
 # ATOMIC_REQUESTS = True → request is already in a transaction

 def export_view(request):
 try:
 with transaction.atomic():  # this creates a savepoint (nested)
 for item in queryset.iterator():  # server-side cursor
 process(item)  # may raise
 except SomeError:
 save_error_status()  # ← fails with InFailedSqlTransaction
 }}}

 This matches the pattern from the docs:



 {{{
 @transaction.atomic
 def viewfunc(request):
 create_parent()

 try:
 with transaction.atomic():
 generate_relationships()
 except IntegrityError:
 handle_exception()

 add_children()
 }}}

 We tested both scenarios:

 Nested (savepoint) — FAILS:

 {{{

   with transaction.atomic():  # outer (ATOMIC_REQUESTS)
   try:
   with transaction.atomic():  # inner → savepoint
   for item in queryset.iterator():
   raise ValueError()
   except ValueError:
   Model.objects.create(...)# ← InFailedSqlTransaction
 }}}

 Non-nested (full rollback) — WORKS:


 {{{

   try:
   with transaction.atomic():  # not nested → full transaction
   for item in queryset.iterator():
   raise ValueError()
   except ValueError:
   Model.objects.create(...)# ← OK, transaction is IDLE after
 full rollback
 }}}

 Let me know if we can help somehow by providing more information. Thanks
 again!
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/django-updates/0107019d1c306938-e860aa63-0332-4087-bc06-45df7cdee8d3-00%40eu-central-1.amazonses.com.

Re: [Django] #37000: cursor_iter relies on GC for server-side cursor cleanup, causing transaction abort after savepoint rollback

[Django]

#37000: cursor_iter relies on GC for server-side cursor cleanup, causing
transaction abort after savepoint rollback
-+-
 Reporter:  Ratskó László|Owner:  (none)
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  iterator, server-| Triage Stage:  Accepted
  side-cursor, savepoint, psycopg3   |
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Simon Charette):

 * stage:  Unreviewed => Accepted

Comment:

 Accepting but I'd like to clarify a specific point.

 In your report you mention that entering the `transaction.atomic` context
 will create a ''savepoint'' but in practice that only happens if the
 `atomic` block is nested which is not included in your example.

 Can you also reproduce without using a nested transaction as the example
 you provided catches an exception inside `atomic` which
 
[https://docs.djangoproject.com/en/6.0/topics/db/transactions/#django.db.transaction.atomic
 is a documented anti-pattern]?

 We should make named cursor closing more resilient but it'd be good to
 know if this can be triggered with proper usage of `transaction.atomic`.
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/django-updates/0107019d1b29d8df-c92754b1-a92d-4a2a-98a7-77228a15b4c5-00%40eu-central-1.amazonses.com.