Hello everyone,
We are using latest reviewboard and we've recently migrated to Docker
setup. Problem is, reviewboard randomly spits error 500 while browsing the
site. Looking into `docker compose logs -f`, we can see that sometimes (not
always) it tries to connect to memcached server on localhost:11211:
reviewboard-1 | 2024-03-14 10:36:23,523 - ERROR - None - AnonymousUser -
/health/ - djblets.util.views - Health check: unable to connect to cache
"default": All servers seem to be down right now
reviewboard-1 | 2024-03-14 10:36:23,524 - ERROR - None - AnonymousUser -
/health/ - django.request - Service Unavailable: /health/
reviewboard-1 | 2024-03-14 10:36:23,525 - DEBUG - -
pymemcache.client.hash - retrying failed server: ('127.0.0.1', 11211)
reviewboard-1 | [2024-03-14 10:36:23 +] [17] [DEBUG] Ignoring
connection reset
reviewboard-1 | [2024-03-14 10:36:27 +] [20] [DEBUG] GET /
reviewboard-1 | 2024-03-14 10:36:27,126 - DEBUG - -
pymemcache.client.hash - retrying failed server: ('127.0.0.1', 11211)
reviewboard-1 | 2024-03-14 10:36:27,127 - ERROR - -
djblets.cache.synchronizer - Unexpected error checking for expiration in
cached synchronization state key
"{OUR_DOMAIN}:/:{OUR_DOMAIN}:siteconfig:1:generation". Is the cache server
down? Error = [Errno 111] Connection refused
reviewboard-1 | Traceback (most recent call last):
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line
82, in is_expired
reviewboard-1 | sync_gen = self._get_latest_sync_gen()
reviewboard-1 |^^^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line
162, in _get_latest_sync_gen
reviewboard-1 | return cache.get(self.cache_key)
reviewboard-1 |^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/django/core/cache/backends/memcached.py",
line 77, in get
reviewboard-1 | return self._cache.get(key, default)
reviewboard-1 |^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 347,
in get
reviewboard-1 | return self._run_cmd("get", key, default,
default=default, **kwargs)
reviewboard-1 |
^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 322,
in _run_cmd
reviewboard-1 | return self._safely_run_func(client, func,
default_val, *args, **kwargs)
reviewboard-1 |
^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 199,
in _safely_run_func
reviewboard-1 | result = func(*args, **kwargs)
reviewboard-1 | ^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 687,
in get
reviewboard-1 | return self._fetch_cmd(b"get", [key], False,
key_prefix=self.key_prefix).get(
reviewboard-1 |
^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 1133,
in _fetch_cmd
reviewboard-1 | self._connect()
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 424,
in _connect
reviewboard-1 | sock.connect(sockaddr)
reviewboard-1 | ConnectionRefusedError: [Errno 111] Connection refused
reviewboard-1 | 2024-03-14 10:36:27,128 - DEBUG - -
pymemcache.client.hash - marking server as dead: ('127.0.0.1', 11211)
reviewboard-1 | 2024-03-14 10:36:27,129 - ERROR - -
djblets.cache.synchronizer - Unexpected error checking for expiration in
cached synchronization state key
"{OUR_DOMAIN}:/:extensionmgr:reviewboard.extensions:gen". Is the cache
server down? Error = [Errno 111] Connection refused
reviewboard-1 | Traceback (most recent call last):
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line
82, in is_expired
reviewboard-1 | sync_gen = self._get_latest_sync_gen()
reviewboard-1 |^^^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line
162, in _get_latest_sync_gen
reviewboard-1 | return cache.get(self.cache_key)
reviewboard-1 |^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/django/core/cache/backends/memcached.py",
line 77, in get
reviewboard-1 | return self._cache.get(key, default)
reviewboard-1 |^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 347,
in get
reviewboard-1 | return self._run_cmd("get", key, default,
default=default, **kwargs)
reviewboard-1 |