Re: [Django] #34870: Memory leak when using psycopg-c with django hstore

[Django]

#34870: Memory leak when using psycopg-c with django hstore
-+-
 Reporter:  David Burke  |Owner:  nobody
 Type:  Bug  |   Status:  closed
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:  invalid
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Florian Apolloner):

 Ok, raising an exception could cause problems, but as far as that
 traceback is concerned I am reading the following out of it:

  * Django registers the hstore type once per connection. This makes sense
 since connections to multiple databases might have different hstore oids
 (In older versions of Django we might have registered types globally for
 psycopg once, but I think with psycopg3 this got improved).
  * Django only queries the database once per alias since `get_hstore_oids`
 is cached.

 So from the looks of it it seems as if celery opens a new connection per
 task which is kinda (?) wasteful. But it is definitely not a bug in
 Django.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018ad1d2486d-9e6972ab-2ee3-40c6-81cf-dc00ef66fcb2-00%40eu-central-1.amazonses.com.

Re: [Django] #34870: Memory leak when using psycopg-c with django hstore

[Django]

#34870: Memory leak when using psycopg-c with django hstore
-+-
 Reporter:  David Burke  |Owner:  nobody
 Type:  Bug  |   Status:  closed
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:  invalid
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by David Burke):

 Right. It seems odd to run the register function more than once.
 Stacktrace as requested.

 {{{
   File "/env/lib/python3.10/site-packages/celery/app/trace.py", line 477,
 in trace_task
 R = retval = fun(*args, **kwargs)
   File "/env/lib/python3.10/site-packages/celery/app/trace.py", line 760,
 in __protected_call__
 return self.run(*args, **kwargs)
   File "/memory_leak/tasks.py", line 9, in count_it
 User.objects.count()
   File "/env/lib/python3.10/site-packages/django/db/models/manager.py",
 line 87, in manager_method
 return getattr(self.get_queryset(), name)(*args, **kwargs)
   File "/env/lib/python3.10/site-packages/django/db/models/query.py", line
 608, in count
 return self.query.get_count(using=self.db)
   File "/env/lib/python3.10/site-packages/django/db/models/sql/query.py",
 line 568, in get_count
 return obj.get_aggregation(using, {"__count": Count("*")})["__count"]
   File "/env/lib/python3.10/site-packages/django/db/models/sql/query.py",
 line 554, in get_aggregation
 result = compiler.execute_sql(SINGLE)
   File "/env/lib/python3.10/site-
 packages/django/db/models/sql/compiler.py", line 1560, in execute_sql
 cursor = self.connection.cursor()
   File "/env/lib/python3.10/site-packages/django/utils/asyncio.py", line
 26, in inner
 return func(*args, **kwargs)
   File "/env/lib/python3.10/site-
 packages/django/db/backends/base/base.py", line 330, in cursor
 return self._cursor()
   File "/env/lib/python3.10/site-
 packages/django/db/backends/base/base.py", line 306, in _cursor
 self.ensure_connection()
   File "/env/lib/python3.10/site-packages/django/utils/asyncio.py", line
 26, in inner
 return func(*args, **kwargs)
   File "/env/lib/python3.10/site-
 packages/django/db/backends/base/base.py", line 289, in ensure_connection
 self.connect()
   File "/env/lib/python3.10/site-packages/django/utils/asyncio.py", line
 26, in inner
 return func(*args, **kwargs)
   File "/env/lib/python3.10/site-
 packages/django/db/backends/base/base.py", line 273, in connect
 connection_created.send(sender=self.__class__, connection=self)
   File "/env/lib/python3.10/site-packages/django/dispatch/dispatcher.py",
 line 176, in send
 return [
   File "/env/lib/python3.10/site-packages/django/dispatch/dispatcher.py",
 line 177, in 
 (receiver, receiver(signal=self, sender=sender, **named))
   File "/env/lib/python3.10/site-
 packages/django/contrib/postgres/signals.py", line 43, in
 register_type_handlers
 raise Exception()
 }}}

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018ad1cae2f2-a4c77d93-c92b-40fa-9f1f-3a31723e6e88-00%40eu-central-1.amazonses.com.

Re: [Django] #34870: Memory leak when using psycopg-c with django hstore

[Django]

#34870: Memory leak when using psycopg-c with django hstore
-+-
 Reporter:  David Burke  |Owner:  nobody
 Type:  Bug  |   Status:  closed
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:  invalid
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Florian Apolloner):

 * cc: Florian Apolloner (added)


Comment:

 Aside from the leak I wouldn't be surprised if we have a bug in Django or
 celery here as well. Mind printing the traceback around registering the
 hstore so we see where is calling us and why?

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018ad119337c-c673c80c-2826-415d-bf6a-d24e413b1007-00%40eu-central-1.amazonses.com.

Re: [Django] #34870: Memory leak when using psycopg-c with django hstore

[Django]

#34870: Memory leak when using psycopg-c with django hstore
-+-
 Reporter:  David Burke  |Owner:  nobody
 Type:  Bug  |   Status:  closed
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:  invalid
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak):

 * status:  new => closed
 * resolution:   => invalid


Comment:

 David, thanks for the investigation. It seems to be fixed in the `psycopg`
 (Thanks Daniele!) so I'm closing as "invalid".

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018acfa154b7-938b4215-4dc5-47cd-a595-b43e9b43382e-00%40eu-central-1.amazonses.com.

Re: [Django] #34870: Memory leak when using psycopg-c with django hstore

[Django]

#34870: Memory leak when using psycopg-c with django hstore
-+-
 Reporter:  David Burke  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by David Burke):

 https://github.com/psycopg/psycopg/issues/647 has been reported and looks
 to have a resolution. I'm unsure if it makes sense for Celery to call
 these Django contrib.postgres signal functions for each task, but at least
 there is no longer a memory leak once this patch is applied.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018aced58e3f-c7650def-7a39-4c65-863b-3e30e6aa4de5-00%40eu-central-1.amazonses.com.

Re: [Django] #34870: Memory leak when using psycopg-c with django hstore

[Django]

#34870: Memory leak when using psycopg-c with django hstore
-+-
 Reporter:  David Burke  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by David Burke):

 Removing the line `hstore.register_hstore(ti, connection.connection)` in
 django/contrib/postgres/signals.py also "fixes" the leak. It appears that
 it's being registered for each celery task.
 So we have:

 - Why does celery result in calling this django signal so many times?
 - Why does a psycopg connection string parsing function leak memory when
 called many times.
 - Why does Django register it so many times.

 Personally I could use pure python psycopg and move on with life. But I
 have to think it's a common set up to use Postgres, Celery, and Django
 together. psycopg-c is what that project recommends for production usage.
 I presume HStore is just incidental and that any Django contrib Postgres
 feature would be affected.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018ac9ee95c8-278fd893-4cdc-4d8c-8a3a-8c4387180f46-00%40eu-central-1.amazonses.com.

Re: [Django] #34870: Memory leak when using psycopg-c with django hstore

[Django]

#34870: Memory leak when using psycopg-c with django hstore
-+-
 Reporter:  David Burke  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by David Burke):

 I'm unsure which project is at fault. The reverse could be said as - not
 using Django's "django.contrib.postgres" fixes the issue, does it follow
 that django is at fault. Is there something that makes you suspect psycopg
 is at fault or a better place to submit an issue? In the linked sample
 repo, just commenting out that one line in INSTALLED_APPS fixes it. But
 then the HStore feature is no longer available from Django.

 If I can strip out more code away for a more minimalist reproduction repo,
 I will follow up here. It would also be more ideal to remove Celery to
 rule that out.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018ac9caf271-4c51cae6-e105-4e1e-a1b5-ef0c51c1c808-00%40eu-central-1.amazonses.com.

Re: [Django] #34870: Memory leak when using psycopg-c with django hstore

[Django]

#34870: Memory leak when using psycopg-c with django hstore
-+-
 Reporter:  David Burke  |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Tim Graham):

 If the pure Python psycopg fixes the issue, does it follow that psycopg-c
 is at fault? Can you explain why Django is at fault? If not, I'm not sure
 this ticket is actionable.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018ac9853493-9b46e304-6120-4bea-94bb-287d8349569e-00%40eu-central-1.amazonses.com.