Re: Reviewboard 6.0.2 in Docker connects to the wrong memcached server 50% of the time
Hello, you are right, the setting was wrong, as seen from the screenshot
below.
Putting proper URL fixes the problem.
Sorry for bothering, then. I was handling the whole migration process from
the command line and didn't occur to me to check the admin area, I assume
what's in the DB overrides the setting in settings-local.py
Thank you for help!
BR
Szymon
On Thursday, March 14, 2024 at 4:35:40 PM UTC+1 David Trowbridge wrote:
> Can you check what the cache configuration is at /admin/settings/general/ ?
>
> David
>
> On Thu, Mar 14, 2024 at 11:34 AM Szymon Niemiec
> wrote:
>
>> Hello everyone,
>> We are using latest reviewboard and we've recently migrated to Docker
>> setup. Problem is, reviewboard randomly spits error 500 while browsing the
>> site. Looking into `docker compose logs -f`, we can see that sometimes (not
>> always) it tries to connect to memcached server on localhost:11211:
>>
>> reviewboard-1 | 2024-03-14 10:36:23,523 - ERROR - None - AnonymousUser -
>> /health/ - djblets.util.views - Health check: unable to connect to cache
>> "default": All servers seem to be down right now
>> reviewboard-1 | 2024-03-14 10:36:23,524 - ERROR - None - AnonymousUser -
>> /health/ - django.request - Service Unavailable: /health/
>> reviewboard-1 | 2024-03-14 10:36:23,525 - DEBUG - -
>> pymemcache.client.hash - retrying failed server: ('127.0.0.1', 11211)
>> reviewboard-1 | [2024-03-14 10:36:23 +] [17] [DEBUG] Ignoring
>> connection reset
>> reviewboard-1 | [2024-03-14 10:36:27 +] [20] [DEBUG] GET /
>> reviewboard-1 | 2024-03-14 10:36:27,126 - DEBUG - -
>> pymemcache.client.hash - retrying failed server: ('127.0.0.1', 11211)
>> reviewboard-1 | 2024-03-14 10:36:27,127 - ERROR - -
>> djblets.cache.synchronizer - Unexpected error checking for expiration in
>> cached synchronization state key
>> "{OUR_DOMAIN}:/:{OUR_DOMAIN}:siteconfig:1:generation". Is the cache server
>> down? Error = [Errno 111] Connection refused
>> reviewboard-1 | Traceback (most recent call last):
>> reviewboard-1 | File
>> "/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line
>> 82, in is_expired
>> reviewboard-1 | sync_gen = self._get_latest_sync_gen()
>> reviewboard-1 |^^^
>> reviewboard-1 | File
>> "/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line
>> 162, in _get_latest_sync_gen
>> reviewboard-1 | return cache.get(self.cache_key)
>> reviewboard-1 |^
>> reviewboard-1 | File
>> "/venv/lib/python3.11/site-packages/django/core/cache/backends/memcached.py",
>>
>> line 77, in get
>> reviewboard-1 | return self._cache.get(key, default)
>> reviewboard-1 |^
>> reviewboard-1 | File
>> "/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 347,
>> in get
>> reviewboard-1 | return self._run_cmd("get", key, default,
>> default=default, **kwargs)
>> reviewboard-1 |
>> ^
>> reviewboard-1 | File
>> "/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 322,
>> in _run_cmd
>> reviewboard-1 | return self._safely_run_func(client, func,
>> default_val, *args, **kwargs)
>> reviewboard-1 |
>> ^
>> reviewboard-1 | File
>> "/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 199,
>> in _safely_run_func
>> reviewboard-1 | result = func(*args, **kwargs)
>> reviewboard-1 | ^
>> reviewboard-1 | File
>> "/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 687,
>> in get
>> reviewboard-1 | return self._fetch_cmd(b"get", [key], False,
>> key_prefix=self.key_prefix).get(
>> reviewboard-1 |
>> ^
>> reviewboard-1 | File
>> "/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 1133,
>> in _fetch_cmd
>> reviewboard-1 | self._connect()
>> reviewboard-1 | File
>> "/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 424,
>> in _connect
>> reviewboard-1 | sock.connect(sockaddr)
>> reviewboard-1 | ConnectionRefusedError: [Errno 111] Connection refused
>> reviewboard-1 | 2024-03-14 10:36:27,128 - DEBUG - -
>> pymemcache.client.hash - marking server as dead: ('127.0.0.1', 11211)
>> reviewboard-1 | 2024-03-14 10:36:27,129 - ERROR - -
>> djblets.cache.synchronizer - Unexpected error checking for expiration in
>> cached synchronization state key
>> "{OUR_DOMAIN}:/:extensionmgr:reviewboard.extensions:gen". Is the cache
>> server down? Error = [Errno 111] Connection refused
>> reviewboard-1 | Traceback (most recent call last):
>> reviewboard-1 | File
>> "/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line
>> 82,
Re: Reviewboard 6.0.2 in Docker connects to the wrong memcached server 50% of the time
Can you check what the cache configuration is at /admin/settings/general/ ?
David
On Thu, Mar 14, 2024 at 11:34 AM Szymon Niemiec
wrote:
> Hello everyone,
> We are using latest reviewboard and we've recently migrated to Docker
> setup. Problem is, reviewboard randomly spits error 500 while browsing the
> site. Looking into `docker compose logs -f`, we can see that sometimes (not
> always) it tries to connect to memcached server on localhost:11211:
>
> reviewboard-1 | 2024-03-14 10:36:23,523 - ERROR - None - AnonymousUser -
> /health/ - djblets.util.views - Health check: unable to connect to cache
> "default": All servers seem to be down right now
> reviewboard-1 | 2024-03-14 10:36:23,524 - ERROR - None - AnonymousUser -
> /health/ - django.request - Service Unavailable: /health/
> reviewboard-1 | 2024-03-14 10:36:23,525 - DEBUG - -
> pymemcache.client.hash - retrying failed server: ('127.0.0.1', 11211)
> reviewboard-1 | [2024-03-14 10:36:23 +] [17] [DEBUG] Ignoring
> connection reset
> reviewboard-1 | [2024-03-14 10:36:27 +] [20] [DEBUG] GET /
> reviewboard-1 | 2024-03-14 10:36:27,126 - DEBUG - -
> pymemcache.client.hash - retrying failed server: ('127.0.0.1', 11211)
> reviewboard-1 | 2024-03-14 10:36:27,127 - ERROR - -
> djblets.cache.synchronizer - Unexpected error checking for expiration in
> cached synchronization state key
> "{OUR_DOMAIN}:/:{OUR_DOMAIN}:siteconfig:1:generation". Is the cache server
> down? Error = [Errno 111] Connection refused
> reviewboard-1 | Traceback (most recent call last):
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line
> 82, in is_expired
> reviewboard-1 | sync_gen = self._get_latest_sync_gen()
> reviewboard-1 |^^^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line
> 162, in _get_latest_sync_gen
> reviewboard-1 | return cache.get(self.cache_key)
> reviewboard-1 |^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/django/core/cache/backends/memcached.py",
> line 77, in get
> reviewboard-1 | return self._cache.get(key, default)
> reviewboard-1 |^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 347,
> in get
> reviewboard-1 | return self._run_cmd("get", key, default,
> default=default, **kwargs)
> reviewboard-1 |
> ^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 322,
> in _run_cmd
> reviewboard-1 | return self._safely_run_func(client, func,
> default_val, *args, **kwargs)
> reviewboard-1 |
> ^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 199,
> in _safely_run_func
> reviewboard-1 | result = func(*args, **kwargs)
> reviewboard-1 | ^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 687,
> in get
> reviewboard-1 | return self._fetch_cmd(b"get", [key], False,
> key_prefix=self.key_prefix).get(
> reviewboard-1 |
> ^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 1133,
> in _fetch_cmd
> reviewboard-1 | self._connect()
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 424,
> in _connect
> reviewboard-1 | sock.connect(sockaddr)
> reviewboard-1 | ConnectionRefusedError: [Errno 111] Connection refused
> reviewboard-1 | 2024-03-14 10:36:27,128 - DEBUG - -
> pymemcache.client.hash - marking server as dead: ('127.0.0.1', 11211)
> reviewboard-1 | 2024-03-14 10:36:27,129 - ERROR - -
> djblets.cache.synchronizer - Unexpected error checking for expiration in
> cached synchronization state key
> "{OUR_DOMAIN}:/:extensionmgr:reviewboard.extensions:gen". Is the cache
> server down? Error = [Errno 111] Connection refused
> reviewboard-1 | Traceback (most recent call last):
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line
> 82, in is_expired
> reviewboard-1 | sync_gen = self._get_latest_sync_gen()
> reviewboard-1 |^^^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line
> 162, in _get_latest_sync_gen
> reviewboard-1 | return cache.get(self.cache_key)
> reviewboard-1 |^
> reviewboard-1 | File
> "/venv/lib/python3.11/site-packages/django/core/cache/backends/memcached.py",
> line 77, in get
> reviewboard-1 | return self._cache.get(key, default)
> reviewboard-1 |
Reviewboard 6.0.2 in Docker connects to the wrong memcached server 50% of the time
Hello everyone,
We are using latest reviewboard and we've recently migrated to Docker
setup. Problem is, reviewboard randomly spits error 500 while browsing the
site. Looking into `docker compose logs -f`, we can see that sometimes (not
always) it tries to connect to memcached server on localhost:11211:
reviewboard-1 | 2024-03-14 10:36:23,523 - ERROR - None - AnonymousUser -
/health/ - djblets.util.views - Health check: unable to connect to cache
"default": All servers seem to be down right now
reviewboard-1 | 2024-03-14 10:36:23,524 - ERROR - None - AnonymousUser -
/health/ - django.request - Service Unavailable: /health/
reviewboard-1 | 2024-03-14 10:36:23,525 - DEBUG - -
pymemcache.client.hash - retrying failed server: ('127.0.0.1', 11211)
reviewboard-1 | [2024-03-14 10:36:23 +] [17] [DEBUG] Ignoring
connection reset
reviewboard-1 | [2024-03-14 10:36:27 +] [20] [DEBUG] GET /
reviewboard-1 | 2024-03-14 10:36:27,126 - DEBUG - -
pymemcache.client.hash - retrying failed server: ('127.0.0.1', 11211)
reviewboard-1 | 2024-03-14 10:36:27,127 - ERROR - -
djblets.cache.synchronizer - Unexpected error checking for expiration in
cached synchronization state key
"{OUR_DOMAIN}:/:{OUR_DOMAIN}:siteconfig:1:generation". Is the cache server
down? Error = [Errno 111] Connection refused
reviewboard-1 | Traceback (most recent call last):
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line
82, in is_expired
reviewboard-1 | sync_gen = self._get_latest_sync_gen()
reviewboard-1 |^^^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line
162, in _get_latest_sync_gen
reviewboard-1 | return cache.get(self.cache_key)
reviewboard-1 |^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/django/core/cache/backends/memcached.py",
line 77, in get
reviewboard-1 | return self._cache.get(key, default)
reviewboard-1 |^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 347,
in get
reviewboard-1 | return self._run_cmd("get", key, default,
default=default, **kwargs)
reviewboard-1 |
^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 322,
in _run_cmd
reviewboard-1 | return self._safely_run_func(client, func,
default_val, *args, **kwargs)
reviewboard-1 |
^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 199,
in _safely_run_func
reviewboard-1 | result = func(*args, **kwargs)
reviewboard-1 | ^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 687,
in get
reviewboard-1 | return self._fetch_cmd(b"get", [key], False,
key_prefix=self.key_prefix).get(
reviewboard-1 |
^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 1133,
in _fetch_cmd
reviewboard-1 | self._connect()
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 424,
in _connect
reviewboard-1 | sock.connect(sockaddr)
reviewboard-1 | ConnectionRefusedError: [Errno 111] Connection refused
reviewboard-1 | 2024-03-14 10:36:27,128 - DEBUG - -
pymemcache.client.hash - marking server as dead: ('127.0.0.1', 11211)
reviewboard-1 | 2024-03-14 10:36:27,129 - ERROR - -
djblets.cache.synchronizer - Unexpected error checking for expiration in
cached synchronization state key
"{OUR_DOMAIN}:/:extensionmgr:reviewboard.extensions:gen". Is the cache
server down? Error = [Errno 111] Connection refused
reviewboard-1 | Traceback (most recent call last):
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line
82, in is_expired
reviewboard-1 | sync_gen = self._get_latest_sync_gen()
reviewboard-1 |^^^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line
162, in _get_latest_sync_gen
reviewboard-1 | return cache.get(self.cache_key)
reviewboard-1 |^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/django/core/cache/backends/memcached.py",
line 77, in get
reviewboard-1 | return self._cache.get(key, default)
reviewboard-1 |^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 347,
in get
reviewboard-1 | return self._run_cmd("get", key, default,
default=default, **kwargs)
reviewboard-1 |
