Your message dated Tue, 31 Oct 2023 15:49:17 +0000
with message-id <[email protected]>
and subject line Bug#1055039: fixed in redis 5:7.0.14-2
has caused the Debian Bug report #1055039,
regarding redis-server: Crash every two hours (oom), seemingly due to systemd's
ProcSubset=pid
to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact [email protected]
immediately.)
--
1055039: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1055039
Debian Bug Tracking System
Contact [email protected] with problems
--- Begin Message ---
Package: redis-server
Version: 5:7.0.11-1
Severity: important
User: [email protected]
Usertags: origin-kali
Dear Maintainer,
After migrating an instance from bullseye to bookworm, redis started to
crash every 2 hours. I tracked it to a change in the system unit file,
the value ProcSubset=pid is what causes the issue.
Long story below.
Part of the Kali Linux infrastructure, we have a host that runs
mirrorbits, a geo redirector. Mirrorbits stores its data in a redis
database.
Some quick numbers:
```
# free -h
total used free shared buff/cache available
Mem: 61Gi 24Gi 13Gi 612Ki 22Gi 37Gi
Swap: 0B 0B 0B
# redis-cli info | grep -E '(used_memory_(peak_)?human)'
used_memory_human:22.94G
used_memory_peak_human:23.24G
```
This instance is managed with Ansible. Not in producion yet.
It was running fine with Debian bullseye, then we re-deployed it on a
bookworm VM. On this new host, Redis crashes every two hours roughly:
```
# journalctl | grep redis | grep code=killed | tail
Oct 28 14:58:30 host systemd[1]: redis.service: Main process exited,
code=killed, status=11/SEGV
Oct 28 16:44:24 host systemd[1]: redis.service: Main process exited,
code=killed, status=11/SEGV
Oct 28 18:49:49 host systemd[1]: redis.service: Main process exited,
code=killed, status=11/SEGV
Oct 28 21:07:28 host systemd[1]: redis.service: Main process exited,
code=killed, status=11/SEGV
Oct 28 22:54:32 host systemd[1]: redis.service: Main process exited,
code=killed, status=11/SEGV
Oct 29 00:39:06 host systemd[1]: redis.service: Main process exited,
code=killed, status=11/SEGV
Oct 29 02:43:30 host systemd[1]: redis.service: Main process exited,
code=killed, status=11/SEGV
```
Looking at the Redis log files, we see this kind of line, repeated
hundreds of times within a few seconds, before Redis finally crashes:
```
85555:M 15 Oct 2023 01:55:55.811 # Out Of Memory allocating 24576 bytes!
```
First thing I did was to disable the RDB snapshot (set every hours in
our config), just to make sure it was not related. It was not, Redis
kept crashing.
Redis RAM usage is rather constant for us (from 22G to 24G), and on this
machine there's only mirrorbits+redis running. There's plenty of RAM
available, I monitored the RAM during a crash, and `free` reports around
37G of RAM available. So I don't think we're running out of RAM.
I checked what changed in the Redis package, between bullseye and
bookworm, and this commit stands out:
d/redis-server.service: harden systemd service file
https://salsa.debian.org/lamby/pkg-redis/-/commit/8fec88c1
I tried to revert the systemd unit file to the bullseye version, and
Redis worked again, no more crash. From there I re-enabled the changes
one by one, until I found the setting that causes the crash:
ProcSubset=pid
I'm not really knowledgeable regarding systemd hardening, but after
reading the doc, it seems pretty clear that this setting is
questionable, and probably shouldn't be enabled.
Quoting systemd.exec(5)`:
If "pid", all files and directories not directly associated with process
management and introspection are made invisible in the /proc/ file
system configured for the unit's processes. [...] Note that Linux
exposes various kernel APIs via /proc/, which are made unavailable with
this setting. Since these APIs are used frequently this option is useful
only in a few, specific cases, and is not suitable for most non-trivial
programs.
At this point I think there's enough information to support disabling
ProcSubset=pid. Please tell me if you need more information, since the
issue is reproducible, it's easy for me to provide more logs.
Thanks in advance!
Arnaud
--- End Message ---
--- Begin Message ---
Source: redis
Source-Version: 5:7.0.14-2
Done: Chris Lamb <[email protected]>
We believe that the bug you reported is fixed in the latest version of
redis, which is due to be installed in the Debian FTP archive.
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to [email protected],
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
Chris Lamb <[email protected]> (supplier of updated redis package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing [email protected])
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Format: 1.8
Date: Tue, 31 Oct 2023 16:34:25 +0100
Source: redis
Built-For-Profiles: nocheck
Architecture: source
Version: 5:7.0.14-2
Distribution: unstable
Urgency: medium
Maintainer: Chris Lamb <[email protected]>
Changed-By: Chris Lamb <[email protected]>
Closes: 1055039
Changes:
redis (5:7.0.14-2) unstable; urgency=medium
.
* Drop ProcSubset=pid hardening flag from the systemd unit files it appears
to cause crashes with memory allocation errors. A huge thanks to Arnaud
Rebillout <[email protected]> for the extensive investigation.
(Closes: #1055039)
Checksums-Sha1:
3ad7c933011e5c5a1ac2e787a9fd5c237cdfaa8d 2273 redis_7.0.14-2.dsc
5397887fb464e6bd677084ed5194c78d1ef84416 29012 redis_7.0.14-2.debian.tar.xz
fc426950ff11d8b4280e526aefefe6b2922d3fde 7503 redis_7.0.14-2_amd64.buildinfo
Checksums-Sha256:
732325c41b50c24460e0744ad518abe54f44ae256d13adba1c02bd87de6e446d 2273
redis_7.0.14-2.dsc
22e09a7544f0311207d053761d352c15321b1f3c9ddb0500f1dd2fe6d549be10 29012
redis_7.0.14-2.debian.tar.xz
3a009b2b46323a44a6cd821c17405d01331dd5d9caae61930c56445ceb02ba50 7503
redis_7.0.14-2_amd64.buildinfo
Files:
402fec4ecec9ae1a5a08750647e5327a 2273 database optional redis_7.0.14-2.dsc
e5128948271053c3a797777663a212e1 29012 database optional
redis_7.0.14-2.debian.tar.xz
113384b3357543bc0da3f2e327a22e06 7503 database optional
redis_7.0.14-2_amd64.buildinfo
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCAAdFiEEwv5L0nHBObhsUz5GHpU+J9QxHlgFAmVBHycACgkQHpU+J9Qx
HliAlQ//R2eMn7tGmGj9twmKsCCQMqoefS9U4cgf5rQMBGLbNfgqCicpGaNoef2D
oVnUc3BYyTfls7d5cutvnUT7XnwoJIepBHo4qDlQutIYTrlLmlEdr6YWQMZHY3l+
u1t8DaD4G8eTGlX0iw06HS2pfY8EFKKESPsUf42qSEULfAvqpGbJM38cX9rOTUTt
ca6jIkGhhEC5VKxh49U0v+rEctRGDooCSr692BRcNcptKHsumo8ajfZQ65oT+Zyd
wfPnwiIuWK/txNv3TP114RAGg30SS2iOhL0RmCBPRTUAW74pBVlsQRFjLVimDzFF
QUFVkCEOU4VNqK/xNzkRZXRGuhLWLNVCeIgj7DfHXaBpy4A9JiFIvWWimc4qGtWH
UMZSAyv/Al9hMncbyhC4gsz48yo+wNrDj1S3nL/lnjEYZOSuDdog9nsj/Dpva/Ek
ttQl4F9japNreS0gtqpfr3Fj15Eah+XkneWgl4OyBUHipGgdGm46bCnq24ZOYvH4
pk3FyceEvDOkxbrxFo9pFfV15CWv9erZgBEkwDkZhw9mynM4siWyUxewb6jA2pDc
59b7DI45A37OkRwdszH2bFPoYfDEjWzqRjE07TwzyD44zTSQsk8MFUsj9EgEdQEu
F7E6Zu6sD0oixC+Iflyxjo5MZNywEPCn6wRKCc8MihaElHKJRkQ=
=TCon
-----END PGP SIGNATURE-----
--- End Message ---
