Can you check what the cache configuration is at /admin/settings/general/ ?
David
On Thu, Mar 14, 2024 at 11:34 AM Szymon Niemiec <szymon.niem...@husarion.com>
wrote:
> Hello everyone,
> We are using latest reviewboard and we've recently migrated to Docker
> setup. Problem is, reviewboard randomly spits error 500 while browsing the
> site. Looking into `docker compose logs -f`, we can see that sometimes (not
> always) it tries to connect to memcached server on localhost:11211:
>
> reviewboard-1 | 2024-03-14 10:36:23,523 - ERROR - None - AnonymousUser -
> /health/ - djblets.util.views - Health check: unable to connect to cache
> "default": All servers seem to be down right now
> reviewboard-1 | 2024-03-14 10:36:23,524 - ERROR - None - AnonymousUser -
> /health/ - django.request - Service Unavailable: /health/
> reviewboard-1 | 2024-03-14 10:36:23,525 - DEBUG - -
> pymemcache.client.hash - retrying failed server: ('127.0.0.1', 11211)
> reviewboard-1 | [2024-03-14 10:36:23 +0000] [17] [DEBUG] Ignoring
> connection reset
> reviewboard-1 | [2024-03-14 10:36:27 +0000] [20] [DEBUG] GET /
> reviewboard-1 | 2024-03-14 10:36:27,126 - DEBUG - -
> pymemcache.client.hash - retrying failed server: ('127.0.0.1', 11211)
> reviewboard-1 | 2024-03-14 10:36:27,127 - ERROR - -
> djblets.cache.synchronizer - Unexpected error checking for expiration in
> cached synchronization state key
> "{OUR_DOMAIN}:/:{OUR_DOMAIN}:siteconfig:1:generation". Is the cache server
> down? Error = [Errno 111] Connection refused
> reviewboard-1 | Traceback (most recent call last):
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line
> 82, in is_expired
> reviewboard-1 | sync_gen = self._get_latest_sync_gen()
> reviewboard-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line
> 162, in _get_latest_sync_gen
> reviewboard-1 | return cache.get(self.cache_key)
> reviewboard-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/django/core/cache/backends/memcached.py",
> line 77, in get
> reviewboard-1 | return self._cache.get(key, default)
> reviewboard-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 347,
> in get
> reviewboard-1 | return self._run_cmd("get", key, default,
> default=default, **kwargs)
> reviewboard-1 |
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 322,
> in _run_cmd
> reviewboard-1 | return self._safely_run_func(client, func,
> default_val, *args, **kwargs)
> reviewboard-1 |
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 199,
> in _safely_run_func
> reviewboard-1 | result = func(*args, **kwargs)
> reviewboard-1 | ^^^^^^^^^^^^^^^^^^^^^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 687,
> in get
> reviewboard-1 | return self._fetch_cmd(b"get", [key], False,
> key_prefix=self.key_prefix).get(
> reviewboard-1 |
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 1133,
> in _fetch_cmd
> reviewboard-1 | self._connect()
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 424,
> in _connect
> reviewboard-1 | sock.connect(sockaddr)
> reviewboard-1 | ConnectionRefusedError: [Errno 111] Connection refused
> reviewboard-1 | 2024-03-14 10:36:27,128 - DEBUG - -
> pymemcache.client.hash - marking server as dead: ('127.0.0.1', 11211)
> reviewboard-1 | 2024-03-14 10:36:27,129 - ERROR - -
> djblets.cache.synchronizer - Unexpected error checking for expiration in
> cached synchronization state key
> "{OUR_DOMAIN}:/:extensionmgr:reviewboard.extensions:gen". Is the cache
> server down? Error = [Errno 111] Connection refused
> reviewboard-1 | Traceback (most recent call last):
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line
> 82, in is_expired
> reviewboard-1 | sync_gen = self._get_latest_sync_gen()
> reviewboard-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line
> 162, in _get_latest_sync_gen
> reviewboard-1 | return cache.get(self.cache_key)
> reviewboard-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/django/core/cache/backends/memcached.py",
> line 77, in get
> reviewboard-1 | return self._cache.get(key, default)
> reviewboard-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 347,
> in get
> reviewboard-1 | return self._run_cmd("get", key, default,
> default=default, **kwargs)
> reviewboard-1 |
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 322,
> in _run_cmd
> reviewboard-1 | return self._safely_run_func(client, func,
> default_val, *args, **kwargs)
> reviewboard-1 |
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 211,
> in _safely_run_func
> reviewboard-1 | result = func(*args, **kwargs)
> reviewboard-1 | ^^^^^^^^^^^^^^^^^^^^^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 687,
> in get
> reviewboard-1 | return self._fetch_cmd(b"get", [key], False,
> key_prefix=self.key_prefix).get(
> reviewboard-1 |
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 1133,
> in _fetch_cmd
> reviewboard-1 | self._connect()
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 424,
> in _connect
> reviewboard-1 | sock.connect(sockaddr)
> reviewboard-1 | ConnectionRefusedError: [Errno 111] Connection refused
> reviewboard-1 | 2024-03-14 10:36:27,131 - ERROR - -
> djblets.cache.synchronizer - Unexpected error checking for expiration in
> cached synchronization state key
> "{OUR_DOMAIN}:/:integrationmgr:djblets.integrations.manager.IntegrationManager:gen".
> Is the cache server down? Error = All servers seem to be down right now
> reviewboard-1 | Traceback (most recent call last):
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line
> 82, in is_expired
> reviewboard-1 | sync_gen = self._get_latest_sync_gen()
> reviewboard-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line
> 162, in _get_latest_sync_gen
> reviewboard-1 | return cache.get(self.cache_key)
> reviewboard-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/django/core/cache/backends/memcached.py",
> line 77, in get
> reviewboard-1 | return self._cache.get(key, default)
> reviewboard-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 347,
> in get
> reviewboard-1 | return self._run_cmd("get", key, default,
> default=default, **kwargs)
> reviewboard-1 |
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 314,
> in _run_cmd
> reviewboard-1 | client = self._get_client(key)
> reviewboard-1 | ^^^^^^^^^^^^^^^^^^^^^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 182,
> in _get_client
> reviewboard-1 | raise MemcacheError("All servers seem to be down
> right now")'
>
>
>
> This won't work, because memcached is running in separate container in
> compose setup. But in settings-local.py the address is correct:
>
> # cat sitedir/conf/settings_local.py
> ...
>
> # Cache backend settings.
> CACHES = {
> "default": {
> "BACKEND": "django.core.cache.backends.memcached.PyMemcacheCache",
> "LOCATION": "memcached:11211"
> }
> }
> ...
>
> And there is no 127.0.0.1:11211 anywhere in settings, so I assume this is
> default value that somehow ended up in the cache server pool.
>
> How to fix this?
> The problem is frequent, basically 50% page loads fail due to this.
>
> Thanks in advance,
> Best regards,
> Szymon
>
> --
> Supercharge your Review Board with Power Pack:
> https://www.reviewboard.org/powerpack/
> Want us to host Review Board for you? Check out RBCommons:
> https://rbcommons.com/
> Happy user? Let us know! https://www.reviewboard.org/users/
> ---
> You received this message because you are subscribed to the Google Groups
> "Review Board Community" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to reviewboard+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/reviewboard/f083ab4c-cea6-4eda-aa17-0aa9e28b6239n%40googlegroups.com
> <https://groups.google.com/d/msgid/reviewboard/f083ab4c-cea6-4eda-aa17-0aa9e28b6239n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
Supercharge your Review Board with Power Pack:
https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons:
https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups
"Review Board Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to reviewboard+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/reviewboard/CAFS3VNVPtWEO49UE6AS%3D%2BSAD2Uj0QBvJaUtBsOnpjPvO%3D1%3DMYw%40mail.gmail.com.
