#33092: Add note regarding thread-safety when using PyMemcacheCache.
--------------------------------------+------------------------------------
Reporter: Martijn van der Blom | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Core (Cache system) | Version: 3.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
--------------------------------------+------------------------------------
Changes (by Nick Pope):
* type: Bug => Cleanup/optimization
* stage: Unreviewed => Accepted
Old description:
> In our application we were using the MemcachedCache backend to connect to
> a memcached server. Since this backend will be removed in Django 4.1 we
> thought we'd migrate to the alternative PyMemcacheCache backend as
> suggested in the Django documentation at:
> https://docs.djangoproject.com/en/3.2/topics/cache/
>
> After upgrading we encountered several errors when running the
> application in gunicorn with the gevent worker class.
>
> Summary of errors:
> - gevent._socketcommon.cancel_wait_ex: [Errno 9] File descriptor was
> closed in another greenlet
> - gevent.exceptions.ConcurrentObjectUseError: This socket is already
> used by another greenlet: <bound method Waiter.switch of
> <gevent._gevent_c_waiter.Waiter object at 0x7f8e77b00a40>>
> - OSError: [Errno 9] Bad file descriptor
>
> These errors seem to be related to either the Django backend
> implementation or Pymemcache not handling multi-threading/thread-safety
> properly.
> There is a related bug for the Pymemcache library where a member of that
> team states that is up the application using Pymemcache to handle thread-
> safety
> (https://github.com/pinterest/pymemcache/issues/195#issuecomment-452523524).
> In this case the Django framework. This comment in Django's
> BaseMemcachedCache implementation indicates that that was the original
> intent:
> https://github.com/django/django/blob/main/django/core/cache/backends/memcached.py#L38
> so i think PyMemcacheCache should handle this.
>
> Example project that reproduces the error:
> https://github.com/mvanderblom/django-memcached-bugreport
>
> For us, this error prevents us from using the PyMemcacheCache backend and
> thus from upgrading to Django 4.1 when it gets released.
New description:
For thread-safety when using `pymemcache` the option `'use_pooling': True`
can be passed via `OPTIONS` which will make `pymemcache.HashClient` use
`pymemcache.PooledClient` instead of `pymemcache.Client` internally.
Some documentation should be added or improved.
----
In our application we were using the MemcachedCache backend to connect to
a memcached server. Since this backend will be removed in Django 4.1 we
thought we'd migrate to the alternative PyMemcacheCache backend as
suggested in the Django documentation at:
https://docs.djangoproject.com/en/3.2/topics/cache/
After upgrading we encountered several errors when running the application
in gunicorn with the gevent worker class.
Summary of errors:
- gevent._socketcommon.cancel_wait_ex: [Errno 9] File descriptor was
closed in another greenlet
- gevent.exceptions.ConcurrentObjectUseError: This socket is already used
by another greenlet: <bound method Waiter.switch of
<gevent._gevent_c_waiter.Waiter object at 0x7f8e77b00a40>>
- OSError: [Errno 9] Bad file descriptor
These errors seem to be related to either the Django backend
implementation or Pymemcache not handling multi-threading/thread-safety
properly.
There is a related bug for the Pymemcache library where a member of that
team states that is up the application using Pymemcache to handle thread-
safety
(https://github.com/pinterest/pymemcache/issues/195#issuecomment-452523524).
In this case the Django framework. This comment in Django's
BaseMemcachedCache implementation indicates that that was the original
intent:
https://github.com/django/django/blob/main/django/core/cache/backends/memcached.py#L38
so i think PyMemcacheCache should handle this.
Example project that reproduces the error:
https://github.com/mvanderblom/django-memcached-bugreport
For us, this error prevents us from using the PyMemcacheCache backend and
thus from upgrading to Django 4.1 when it gets released.
--
Comment:
So, yes, `pymemcache.Client` is not thread-safe. We are using
`pymemcache.HashClient` so that we can support connections to multiple
servers.
I note that `pymemcache.PooledClient` is thread-safe according to the
[https://pymemcache.readthedocs.io/en/latest/getting_started.html?highlight=thread#using-a
-client-pool documentation]. We can pass the `use_pooling` flag to
`HashClient`. Unfortunately `pymemcache`'s documentation is a little
sparse!
If I add `'OPTIONS': {'use_pooling': True}` to your `CACHES` configuration
in your reproducer the problem goes away for me.
Would you be prepared to open a PR with a tweak to the documentation? I
already mentioned `use_pooling` in at the end of the
[https://docs.djangoproject.com/en/3.2/topics/cache/#cache-arguments cache
arguments] section, so maybe we just need to amend the sentence before?
--
Ticket URL: <https://code.djangoproject.com/ticket/33092#comment:2>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/069.9a48941c8446c663eee4f39fa2a9fab7%40djangoproject.com.
