Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  Priyank
 |  Panchal
 Type:  Bug  |   Status:  closed
Component:  Database layer   |  Version:  dev
  (models, ORM)  |
 Severity:  Normal   |   Resolution:  fixed
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by nessita <124304+nessita@…>):

 * resolution:   => fixed
 * status:  assigned => closed

Comment:

 In [changeset:"6a9db1e626952ffb37633586e6647319ea53467d" 6a9db1e]:
 {{{#!CommitTicketReference repository=""
 revision="6a9db1e626952ffb37633586e6647319ea53467d"
 Fixed #34865 -- Released memory earlier than garbage collection on
 database wrapping layers.

 Thank you Florian Apolloner, Jake Howard and Patryk Zawadzki for
 the clarifying comments and reviews.
 }}}
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/django-updates/01070195a601ffdb-ea9470cb-0cec-4cf4-a165-10398f2bef3f-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  Priyank
 |  Panchal
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  dev
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Natalia Bidart):

 * needs_better_patch:  1 => 0
 * needs_tests:  1 => 0
 * stage:  Accepted => Ready for checkin
 * version:  4.2 => dev

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/django-updates/01070195a59420e0-f1f70138-8e5f-4004-bb9c-ef89480ca7e5-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  Priyank
 |  Panchal
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  1|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Natalia Bidart):

 * needs_better_patch:  0 => 1
 * needs_tests:  0 => 1
 * stage:  Ready for checkin => Accepted

Comment:

 I have updated the ticker per my review. I'm undecided whether we should
 add a small release note highlighting this in case users are doing custom
 resource cleanup and this gets in the way somehow...
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/django-updates/010701958c4a0d57-85c9ad63-7c25-4868-b773-95575163bdb5-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  Priyank
 |  Panchal
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Florian Apolloner):

 * needs_better_patch:  1 => 0
 * needs_tests:  1 => 0
 * stage:  Accepted => Ready for checkin

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/django-updates/01070195007c2e91-cb0a8848-aac5-4eb6-89a4-4fd259bb6d7e-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  Priyank
 |  Panchal
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  1|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Comment (by Patryk Zawadzki):

 Yeah, using a (cappable!) thread pool would be amazing, especially if the
 async ORM API was also able to delegate to a request-specific DB thread
 instead of mixing everything in a single shared thread.
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/django-updates/01070194f5685589-560f0874-3376-4e41-a8bf-2a90f91ae7f3-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  Priyank
 |  Panchal
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  1|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Comment (by Florian Apolloner):

 Mhm yeah I recommended to switch to a fixed thread pool at some point iirc
 :D Though even with a fixed thread pool I think that the "thread/asgi"
 local might be per request then (ie async context id) which would show
 this problem. Thank you for confirming that you are using ASGI.
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/django-updates/01070194f4c1e72b-e13f2088-fa7d-491e-9fe7-49f7595cb035-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  Priyank
 |  Panchal
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  1|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Comment (by Patryk Zawadzki):

 Florian, we're using ASGI, so I believe it's very much spawning and
 collecting threads left and right. The database wrapper-related objects
 come up in garbage lists every few requests with `uvicorn` in production
 mode.
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/django-updates/01070194f4a1542a-430770b5-6417-416e-8f27-c6f94b2cf374-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  Priyank
 |  Panchal
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  1|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Comment (by Florian Apolloner):

 Thinking more about this ticket (please note that I am thinking out loud
 here, so let me know if take a wrong turn somewhere): `DatabaseWrapper` is
 in a standard setup *not* supposed to get garbage collected at all. Each
 thread will have one wrapper per defined database stored in a thread local
 (`ConnectionHandler`/`BaseConnectionHandler`). So unless the "web"server
 is spawning a new thread per connection this should not be a problem (I
 could see it being a problem for runserver but that is for dev only
 anyways).

 So to the people having issues here: Where exactly are you seeing this, in
 job queues (celery etc) or in your webservers (which exactly). Fábio
 mentioned seeing it on uvicorn / daphne, is this also a problem for
 gunicorn (I know that one usually has a fixed worker & thread count
 there).

 Please note that I am not saying that there is no problem hidden here, the
 cyclic reference is certainly ugly. But if the object isn't ment to get
 garbage collected, it is not the end of the world. That said I would
 certainly prefer a solution that doesn't use a weakref proxy (we would
 need to do some profiling on the overhead).
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/django-updates/01070194e5e3c494-74194af1-ec8b-46ef-8afd-7220daa1675f-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  Priyank
 |  Panchal
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  1|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Andreas Pelme):

 * cc: Andreas Pelme (removed)

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/django-updates/01070194e5c89c56-33931ff5-b978-418a-af9f-532f5e966689-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  Priyank
 |  Panchal
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  1|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Comment (by Filip Owczarek):

 Hi,
 I just submitted my proposed solution.

 https://github.com/django/django/pull/19153
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/django-updates/01070194e5c5f46a-e5e7ec54-9a3c-4f00-9868-1d0d9c2bc3e0-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  Priyank
 |  Panchal
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  1|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Comment (by Florian Apolloner):

 Hi Patryk, I agree which is why this ticket is accepted. It just needs a
 PR with tests (or an explanation why tests make no sense, are not
 possible) and then we can see what we can do about that.

 > What makes it worse is that in this case, it holds a reference to a
 `psycopg` connection object, holding all of the psycopg internals hostage
 and preventing proper finalization of the connection (and, I assume, in
 Django 5.x also preventing the connection from being reclaimed by the
 connection pool? I did not verify that).

 I don't think so, the connection is not returned to the pool on GC but
 rather when `.close` is called on it at the end of the request:
 
https://github.com/django/django/blob/5da3ad7bf90fba7321f4c2834db44aa920c70bc7/django/db/backends/postgresql/base.py#L391
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/django-updates/01070194d5ffc443-6fb0a624-d837-42aa-a835-05bc866609c0-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  Priyank
 |  Panchal
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  1|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Comment (by Patryk Zawadzki):

 Florian, it does clean up ''eventually'' thanks to the garbage collection,
 but garbage collection is a last resort method of freeing unused memory.

 There are two important problems with relyin on GC:

 1. It happens very rarely (the default thresholds are `(700, 10 10)`,
 which means the deepest collection cycle—generation 2—only runs every
 70,000 dereferenced objects. That's very rare; in a simple Django
 application, that can mean a hundred requests.
 2. It also locks the global interpreter lock and effectively freezes the
 entire Python process (and all its threads) until it goes through all
 uncollectable objects. The more objects it finds (not just the ones with
 reference cycles but also everything else they hold a reference to), the
 longer the freeze. In extreme cases, it can randomly add seconds to your
 response times.

 Aggregation of garbage leads to memory fragmentation as unused memory
 cannot be reused until the garbage collection thresholds are met, which
 leads to the overall memory use slowly creeping up.

 What makes it worse is that in this case, it holds a reference to a
 `psycopg` connection object, holding all of the psycopg internals hostage
 and preventing proper finalization of the connection (and, I assume, in
 Django 5.x also preventing the connection from being reclaimed by the
 connection pool? I did not verify that).
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/django-updates/01070194d5c1e87c-0c570d5f-9130-4b02-aaf5-c22b707388c9-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  Priyank
 |  Panchal
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  1|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Florian Apolloner):

 * cc: Florian Apolloner (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018bd1dc90f3-03ef4b90-e96e-4c87-a28b-c3b7f87b6b42-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  Priyank
 |  Panchal
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  1|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak):

 * needs_better_patch:  0 => 1
 * needs_tests:  0 => 1


Comment:

 Marking as needs improvement per Florian's comment.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018b68615a6d-8dbbe1d2-5152-4fac-8392-9b4c1a1a716e-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  Priyank
 |  Panchal
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Sarah Boyce):

 * has_patch:  0 => 1


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018b67973918-aca90daa-1e90-47a6-825a-3aff53cc28ce-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  Priyank
 |  Panchal
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Florian Apolloner):

 > Throwing gc.collect() actually get it, but it take some milliseconds,
 even if the underlying database connection is already closed.

 with "actually get it" do you mean that it cleans it up properly? If yes
 there does not seem to be real leak here since garbage collection usually
 runs every now and then (or do I miss something). It might still be
 beneficial to improve the performance here but that might require a test
 or two :D

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018b00ebeb6e-065266a0-bfa6-4baf-a25b-b0d30635761e-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  Priyank
 |  Panchal
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Priyank Panchal):

 * owner:  nobody => Priyank Panchal
 * status:  new => assigned


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018af1a7a1e8-63df2ad0-be1a-4b4d-8779-430a3dc7fcf4-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by David Wobrock):

 * cc: David Wobrock (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018aec4b39da-4358b82a-65b0-4d26-a787-2ecb8ffd93ee-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Fábio Domingues):

 * Attachment "fix_34865.patch" removed.

 Possible approach.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018abdb7f564-31a53745-452e-45b5-addb-5977df3bb0c2-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Fábio Domingues):

 * Attachment "fix_34865.patch" added.

 Possible approach.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018abdb7f598-3b53abf6-6fc8-450b-a1cd-19905966adbd-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Fábio Domingues):

 * Attachment "fix_34865.patch" added.

 Possible approach.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018abdb4b401-429640f1-e00c-42b3-9472-5e3dd21ebb00-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Fábio Domingues):

 * Attachment "fix_34865.patch" removed.

 Possible approach

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018abdb4b3b5-ac21e14b-8cef-4a03-af4d-37f1f8fd90c8-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Fábio Domingues):

 One more detail. Using `objgraph.show_growth` without doing `gc.collect`
 give us this on each new request:
 {{{
 list   1990   +10
 dict   6798+3
 deque20+2
 lock 43+2
 OrderedDict  11+2
 partial  24+2
 ReferenceType  3848+1
 DatabaseWrapper   6+1
 DatabaseClient6+1
 DatabaseCreation  6+1
 DatabaseFeatures  6+1
 DatabaseIntrospection 6+1
 DatabaseOperations6+1
 BaseDatabaseValidation6+1
 DatabaseErrorWrapper  5+1
 PGconn5+1
 Connection5+1
 PrepareManager5+1
 AdaptersMap   7+1
 TypesRegistry 7+1
 }}}
 with the previous patch the output of `show_growth` is empty.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018abd500b05-ea715257-fc17-49d5-8f78-fcd45f9590fc-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Fábio Domingues):

 * Attachment "fix_34865.patch" added.

 Possible approach

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018abd253692-52e9e721-cb24-42a7-956b-36f6bf259fec-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Fábio Domingues):

 Tested using Uvicorn and Daphne, the database engine should not matter, I
 tested with PostgreSQL, `CONN_MAX_AGE` could be any value and the effect
 is the same using ''async'' or ''sync'' views.

 {{{#!python

 get_refs = lambda: [o for o in gc.get_objects() if isinstance(o,
 BaseDatabaseWrapper)]
 plot_refs = lambda: objgraph.show_backrefs(get_refs(), filename="sample-
 graph.png")

 def sync_view(request):
 MyModel.objects.count()
 # gc.collect()
 # plot_refs()
 return HttpResponse(f"Number of BaseDatabaseWrapper instances:
 {len(get_refs())}")

 async def async_view(request):
 await MyModel.objects.acount()
 # gc.collect()
 # plot_refs()
 return HttpResponse(f"Number of BaseDatabaseWrapper instances:
 {len(get_refs())}")
 }}}

 On each refresh you should see the number of instances increase. If you
 plot the references you would see many diagrams like the one in the
 attachment. Throwing `gc.collect()` actually get it, but it take some
 milliseconds, even if the underlying database connection is already
 closed.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018abd249ab6-09f65099-17cf-417b-a230-55b822ecd29c-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Carlton Gibson):

 > ...I'd like to check out the patch proposal.

 Yes, please.

 Any chance you could give a minimal example showing the behaviour here, to
 look at?
 Various folks have reported ''leaky'' behaviour on Channels/Daphne, but
 we've not been able to pin it down.

 > … ends up in a circular reference ... and the GC could not cleanup the
 objects.

 Just out of interest, could you try throwing in a `gc.collect()` — that
 **should** get it (but `WeakRef` is hopefully right in this case.)

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018abccd24cc-a453c7c0-c88a-4f5c-9121-d4ac3341b64b-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak):

 * cc: Carlton Gibson, Andrew Godwin, Andreas Pelme (added)
 * status:  closed => new
 * resolution:  duplicate =>
 * stage:  Unreviewed => Accepted


Comment:

 We can keep this separately.

 Tentatively accepted, I'd like to check out the patch proposal.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018abcb67305-02273a49-00f2-450e-ab55-cd0ed1fe6b6e-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  nobody
 Type:  Bug  |   Status:  closed
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:  duplicate
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Fábio Domingues):

 Replying to [comment:1 Mariusz Felisiak]:
 > Fabio, thanks for the report. Is there a reason to open a new ticket and
 don't add your findings in the #33497? As far as I'm aware, this is a
 duplicate.
 Fixing this memory leak will not solve the problem of the persistent
 connections not been reuse, only the fact that they will not pill up (and
 starve the DB) if inaccessible.
 But I'm new here, still learning. Should I move the findings to #33497?

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018abc70950d-0f49ad69-8984-4fae-a458-517ae1b8623e-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  nobody
 Type:  Bug  |   Status:  closed
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:  duplicate
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak):

 * status:  new => closed
 * resolution:   => duplicate


Comment:

 Fabio, thanks for the report. Is there a reason to open a new ticket and
 don't add your findings in the #33497? As far as I'm aware, this is a
 duplicate.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018abc6a6f87-c7d64c06-bf34-4407-bb29-eab1c03a1c1e-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Fábio Domingues):

 * Attachment "databasewrapper.png" added.


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018abc524418-5e95a1a9-24de-460d-8b70-5ce508bc2bdd-00%40eu-central-1.amazonses.com.