#35757: memcached.PyMemcacheCache reentrancy problem with ASGI-based runserver
-------------------------------+--------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
Description changed by Harm Verhagen:
Old description:
> I found that using `memcached.PyMemcacheCache` with runserver has
> reentrancy problems.
>
> I implemented a json api in django, when issuing multiple request
> simultaneously from javascript, the requests sometimes crash.
>
> I have urls that do something like this.
>
> {{{
> from django.core.cache import cache
>
> def example_url(request):
> cache.incr("some key", 1) # count requests
>
> return JsonResponse(data=data, status=201)
> }}}
>
> When issuing 5-10 requests simultaneously on a runserver, it often (not
> always) get this crash
>
> {{{
>
> File "..../views/api.py", line 2512, in example_url
> cache.incr("some key", 1)
> File "./venv/lib/python3.12/site-
> packages/django/core/cache/backends/memcached.py", line 110, in incr
> val = self._cache.incr(key, delta)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "./venv/lib/python3.12/site-packages/pymemcache/client/hash.py",
> line 350, in incr
> return self._run_cmd("incr", key, False, *args, **kwargs)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "./venv/lib/python3.12/site-packages/pymemcache/client/hash.py",
> line 322, in _run_cmd
> return self._safely_run_func(client, func, default_val, *args,
> **kwargs)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "./venv/lib/python3.12/site-packages/pymemcache/client/hash.py",
> line 211, in _safely_run_func
> result = func(*args, **kwargs)
> ^^^^^^^^^^^^^^^^^^^^^
> File "./venv/lib/python3.12/site-packages/pymemcache/client/base.py",
> line 827, in incr
> results = self._misc_cmd([cmd], b"incr", noreply)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "./venv/lib/python3.12/site-packages/pymemcache/client/base.py",
> line 1291, in _misc_cmd
> buf, line = _reader(self.sock, buf)
> ^^^^^^^^^^^^^^^^^^^^^^^
> File "./venv/lib/python3.12/site-packages/pymemcache/client/base.py",
> line 1658, in _readline
> buf = _recv(sock, RECV_SIZE)
> ^^^^^^^^^^^^^^^^^^^^^^
> File "./venv/lib/python3.12/site-packages/pymemcache/client/base.py",
> line 1750, in _recv
> return sock.recv(size)
>
> OSError: [Errno 9] Bad file descriptor
> }}}
>
>
> The following code looked a bit suspicious.
>
> `cache.backends.memcached.py:99 BaseMemcachedCache`
>
> {{{
> def close(self, **kwargs):
> # Many clients don't clean up connections properly.
> self._cache.disconnect_all()
> }}}
>
> Doesn't this race? closing a all connections, might close a connection
> of a parallel request that was just opened but not yet finished.
>
> == Notes
> I'm not putting a heavy load on runserver, its a simple javascript app,
> that happens to do 10 requests on a single page. Django should be able
> to handle that, also in a dev environment
>
> == Workaround
> Adding the `--noasgi` option seems to help.
> {{{
> ./manage.py runserver --noasgi
> }}}
>
> Weird enough the move obvious, `--nothreading` option does _not_ help.
> That crashes in the same way.
>
>
> == Version info
>
> {{{
> Django==5.1.1
> pymemcache==4.0.0
> python 3.12.6
> }}}
New description:
I found that using `memcached.PyMemcacheCache` with runserver has
reentrancy problems.
I implemented a json api in django, when issuing multiple request
simultaneously from javascript, the requests sometimes crash.
I have urls that do something like this.
{{{
from django.core.cache import cache
def example_url(request):
cache.incr("some key", 1) # count requests
return JsonResponse(data=data, status=201)
}}}
When issuing 5-10 requests simultaneously on a runserver, it often (not
always) get this crash
{{{
File "..../views/api.py", line 2512, in example_url
cache.incr("some key", 1)
File "./venv/lib/python3.12/site-
packages/django/core/cache/backends/memcached.py", line 110, in incr
val = self._cache.incr(key, delta)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "./venv/lib/python3.12/site-packages/pymemcache/client/hash.py",
line 350, in incr
return self._run_cmd("incr", key, False, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "./venv/lib/python3.12/site-packages/pymemcache/client/hash.py",
line 322, in _run_cmd
return self._safely_run_func(client, func, default_val, *args,
**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "./venv/lib/python3.12/site-packages/pymemcache/client/hash.py",
line 211, in _safely_run_func
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "./venv/lib/python3.12/site-packages/pymemcache/client/base.py",
line 827, in incr
results = self._misc_cmd([cmd], b"incr", noreply)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "./venv/lib/python3.12/site-packages/pymemcache/client/base.py",
line 1291, in _misc_cmd
buf, line = _reader(self.sock, buf)
^^^^^^^^^^^^^^^^^^^^^^^
File "./venv/lib/python3.12/site-packages/pymemcache/client/base.py",
line 1658, in _readline
buf = _recv(sock, RECV_SIZE)
^^^^^^^^^^^^^^^^^^^^^^
File "./venv/lib/python3.12/site-packages/pymemcache/client/base.py",
line 1750, in _recv
return sock.recv(size)
OSError: [Errno 9] Bad file descriptor
}}}
The following code looked a bit suspicious.
`cache.backends.memcached.py:99 BaseMemcachedCache`
{{{
def close(self, **kwargs):
# Many clients don't clean up connections properly.
self._cache.disconnect_all()
}}}
Doesn't this race? closing a all connections, might close a connection
of a parallel request that was just opened but not yet finished.
== Notes
I'm not putting a heavy load on runserver, its a simple javascript app,
that happens to do 10 requests on a single page. Django should be able
to handle that, also in a dev environment
== Workaround
Adding the `--noasgi` option seems to help.
{{{
./manage.py runserver --noasgi
}}}
Weird enough the (to me) more obvious, `--nothreading` option does _not_
help. That crashes in the same way.
== Version info
{{{
Django==5.1.1
pymemcache==4.0.0
python 3.12.6
}}}
--
--
Ticket URL: <https://code.djangoproject.com/ticket/35757#comment:5>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/01070191e69bb8b1-e337f732-9689-4685-9957-a6e0c0d88f8b-000000%40eu-central-1.amazonses.com.
