Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  Priyank
 |  Panchal
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  1|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Florian Apolloner):

 * cc: Florian Apolloner (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018bd1dc90f3-03ef4b90-e96e-4c87-a28b-c3b7f87b6b42-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  Priyank
 |  Panchal
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  1|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak):

 * needs_better_patch:  0 => 1
 * needs_tests:  0 => 1


Comment:

 Marking as needs improvement per Florian's comment.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018b68615a6d-8dbbe1d2-5152-4fac-8392-9b4c1a1a716e-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  Priyank
 |  Panchal
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Sarah Boyce):

 * has_patch:  0 => 1


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018b67973918-aca90daa-1e90-47a6-825a-3aff53cc28ce-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  Priyank
 |  Panchal
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Florian Apolloner):

 > Throwing gc.collect() actually get it, but it take some milliseconds,
 even if the underlying database connection is already closed.

 with "actually get it" do you mean that it cleans it up properly? If yes
 there does not seem to be real leak here since garbage collection usually
 runs every now and then (or do I miss something). It might still be
 beneficial to improve the performance here but that might require a test
 or two :D

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018b00ebeb6e-065266a0-bfa6-4baf-a25b-b0d30635761e-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  Priyank
 |  Panchal
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Priyank Panchal):

 * owner:  nobody => Priyank Panchal
 * status:  new => assigned


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018af1a7a1e8-63df2ad0-be1a-4b4d-8779-430a3dc7fcf4-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by David Wobrock):

 * cc: David Wobrock (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018aec4b39da-4358b82a-65b0-4d26-a787-2ecb8ffd93ee-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Fábio Domingues):

 * Attachment "fix_34865.patch" removed.

 Possible approach.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018abdb7f564-31a53745-452e-45b5-addb-5977df3bb0c2-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Fábio Domingues):

 * Attachment "fix_34865.patch" added.

 Possible approach.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018abdb7f598-3b53abf6-6fc8-450b-a1cd-19905966adbd-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Fábio Domingues):

 * Attachment "fix_34865.patch" added.

 Possible approach.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018abdb4b401-429640f1-e00c-42b3-9472-5e3dd21ebb00-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Fábio Domingues):

 * Attachment "fix_34865.patch" removed.

 Possible approach

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018abdb4b3b5-ac21e14b-8cef-4a03-af4d-37f1f8fd90c8-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Fábio Domingues):

 One more detail. Using `objgraph.show_growth` without doing `gc.collect`
 give us this on each new request:
 {{{
 list   1990   +10
 dict   6798+3
 deque20+2
 lock 43+2
 OrderedDict  11+2
 partial  24+2
 ReferenceType  3848+1
 DatabaseWrapper   6+1
 DatabaseClient6+1
 DatabaseCreation  6+1
 DatabaseFeatures  6+1
 DatabaseIntrospection 6+1
 DatabaseOperations6+1
 BaseDatabaseValidation6+1
 DatabaseErrorWrapper  5+1
 PGconn5+1
 Connection5+1
 PrepareManager5+1
 AdaptersMap   7+1
 TypesRegistry 7+1
 }}}
 with the previous patch the output of `show_growth` is empty.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018abd500b05-ea715257-fc17-49d5-8f78-fcd45f9590fc-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Fábio Domingues):

 * Attachment "fix_34865.patch" added.

 Possible approach

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018abd253692-52e9e721-cb24-42a7-956b-36f6bf259fec-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Fábio Domingues):

 Tested using Uvicorn and Daphne, the database engine should not matter, I
 tested with PostgreSQL, `CONN_MAX_AGE` could be any value and the effect
 is the same using ''async'' or ''sync'' views.

 {{{#!python

 get_refs = lambda: [o for o in gc.get_objects() if isinstance(o,
 BaseDatabaseWrapper)]
 plot_refs = lambda: objgraph.show_backrefs(get_refs(), filename="sample-
 graph.png")

 def sync_view(request):
 MyModel.objects.count()
 # gc.collect()
 # plot_refs()
 return HttpResponse(f"Number of BaseDatabaseWrapper instances:
 {len(get_refs())}")

 async def async_view(request):
 await MyModel.objects.acount()
 # gc.collect()
 # plot_refs()
 return HttpResponse(f"Number of BaseDatabaseWrapper instances:
 {len(get_refs())}")
 }}}

 On each refresh you should see the number of instances increase. If you
 plot the references you would see many diagrams like the one in the
 attachment. Throwing `gc.collect()` actually get it, but it take some
 milliseconds, even if the underlying database connection is already
 closed.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018abd249ab6-09f65099-17cf-417b-a230-55b822ecd29c-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Carlton Gibson):

 > ...I'd like to check out the patch proposal.

 Yes, please.

 Any chance you could give a minimal example showing the behaviour here, to
 look at?
 Various folks have reported ''leaky'' behaviour on Channels/Daphne, but
 we've not been able to pin it down.

 > … ends up in a circular reference ... and the GC could not cleanup the
 objects.

 Just out of interest, could you try throwing in a `gc.collect()` — that
 **should** get it (but `WeakRef` is hopefully right in this case.)

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018abccd24cc-a453c7c0-c88a-4f5c-9121-d4ac3341b64b-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak):

 * cc: Carlton Gibson, Andrew Godwin, Andreas Pelme (added)
 * status:  closed => new
 * resolution:  duplicate =>
 * stage:  Unreviewed => Accepted


Comment:

 We can keep this separately.

 Tentatively accepted, I'd like to check out the patch proposal.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018abcb67305-02273a49-00f2-450e-ab55-cd0ed1fe6b6e-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  nobody
 Type:  Bug  |   Status:  closed
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:  duplicate
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Fábio Domingues):

 Replying to [comment:1 Mariusz Felisiak]:
 > Fabio, thanks for the report. Is there a reason to open a new ticket and
 don't add your findings in the #33497? As far as I'm aware, this is a
 duplicate.
 Fixing this memory leak will not solve the problem of the persistent
 connections not been reuse, only the fact that they will not pill up (and
 starve the DB) if inaccessible.
 But I'm new here, still learning. Should I move the findings to #33497?

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018abc70950d-0f49ad69-8984-4fae-a458-517ae1b8623e-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  nobody
 Type:  Bug  |   Status:  closed
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:  duplicate
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak):

 * status:  new => closed
 * resolution:   => duplicate


Comment:

 Fabio, thanks for the report. Is there a reason to open a new ticket and
 don't add your findings in the #33497? As far as I'm aware, this is a
 duplicate.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018abc6a6f87-c7d64c06-bf34-4407-bb29-eab1c03a1c1e-00%40eu-central-1.amazonses.com.

Re: [Django] #34865: DatabaseWrapper are not GC and connections are not closed

[Django]

#34865: DatabaseWrapper are not GC and connections are not closed
-+-
 Reporter:  Fábio Domingues  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Fábio Domingues):

 * Attachment "databasewrapper.png" added.


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018abc524418-5e95a1a9-24de-460d-8b70-5ce508bc2bdd-00%40eu-central-1.amazonses.com.