#35757: memcached.PyMemcacheCache reentrancy problem with ASGI-based runserver
-------------------------------------+-------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Bug | Status: closed
Component: Core (Cache system) | Version: 5.1
Severity: Normal | Resolution: duplicate
Keywords: asgi, async | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Harm Verhagen):
I don't think its a duplicate of #33625 (although it seems related)
I reopened for reconsideration.
=== Why this is different?
#33625 is an optimization problem. All works fine, but could be optimized
for performance. The connection is opened/closed which adds (a bit of)
latency to each request. (which might be acceptable, or not depending on
your usecase)
This problem (#35757) however is about that the system actually breaks.
When running the test above. runserver stops working completely, until
restarted.
If you are developing a javascript app, that issues multiple requests
simultaniously, then runserver (in this configuration) is unusabe.
Another reason why its no duplicate. I can reproduce the problem also with
`django.core.cache.backends.memcached.PyLibMCCache` instead of
`'....memcached.PyMemcacheCache`.
PyLibMCCache already does provide connection pooling, so the optimization
requested in #33625 is not applicable for PyLibMCCache.
=== Reproducibility
Where you able to reproduce the problem with the example attached? (the
problem that runserver actually gives 500's and eventually stops working
completely)
=== Results with PyLibMCCache
NB: this might be a different problem, triggered by the same testcase
==== Result (PyLibMCCache)
When running requests in parallel, some of the requests get a 500.
{{{
Internal Server Error: /polls/
Traceback (most recent call last):
File "venv/lib/python3.12/site-packages/asgiref/sync.py", line 518, in
thread_handler
raise exc_info[1]
File "venv/lib/python3.12/site-
packages/django/core/handlers/exception.py", line 42, in inner
response = await get_response(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "venv/lib/python3.12/site-packages/asgiref/sync.py", line 518, in
thread_handler
raise exc_info[1]
File "venv/lib/python3.12/site-packages/django/core/handlers/base.py",
line 253, in _get_response_async
response = await wrapped_callback(
^^^^^^^^^^^^^^^^^^^^^^^
File "venv/lib/python3.12/site-packages/asgiref/sync.py", line 468, in
__call__
ret = await asyncio.shield(exec_coro)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/homebrew/Cellar/[email protected]/3.12.6/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/thread.py",
line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "venv/lib/python3.12/site-packages/asgiref/sync.py", line 522, in
thread_handler
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "django-35757/mysite/polls/views.py", line 22, in index
v = incr()
^^^^^^
File "/tmp/dbg/django-35757/mysite/polls/views.py", line 11, in incr
val = cache.incr(key, 1)
^^^^^^^^^^^^^^^^^^
File
"/Users/harmverhagen/Documents/symeon/intemo/sensoren/cityznz/cityzenz/api/venv/lib/python3.12
/site-packages/django/core/cache/backends/memcached.py", line 110, in incr
val = self._cache.incr(key, delta)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pylibmc.Error: 1 keys failed
HTTP GET /polls/ 500 [5.13, 127.0.0.1:63568]
}}}
NB: Sometimes I even noticed a python crash !
{{{
HTTP GET /polls/ 200 [0.11, 127.0.0.1:63819]
Python(51613,0x17d037000) malloc: double free for ptr 0x13980a400
Python(51613,0x17d037000) malloc: *** set a breakpoint in
malloc_error_break to debug
}}}
So, maybe the `PyLibMCCache` issue is something different
--
Ticket URL: <https://code.djangoproject.com/ticket/35757#comment:15>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/010701924d32efa7-338c52b7-7454-4952-8410-5d4603be57fb-000000%40eu-central-1.amazonses.com.
