Your message dated Tue, 31 Oct 2023 16:04:53 +0000
with message-id <[email protected]>
and subject line Bug#1055039: fixed in redis 5:7.2.2-2
has caused the Debian Bug report #1055039,
regarding redis-server: Crash every two hours (oom), seemingly due to systemd's
ProcSubset=pid
to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact [email protected]
immediately.)
--
1055039: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1055039
Debian Bug Tracking System
Contact [email protected] with problems
--- Begin Message ---
Package: redis-server
Version: 5:7.0.11-1
Severity: important
User: [email protected]
Usertags: origin-kali
Dear Maintainer,
After migrating an instance from bullseye to bookworm, redis started to
crash every 2 hours. I tracked it to a change in the system unit file,
the value ProcSubset=pid is what causes the issue.
Long story below.
Part of the Kali Linux infrastructure, we have a host that runs
mirrorbits, a geo redirector. Mirrorbits stores its data in a redis
database.
Some quick numbers:
```
# free -h
total used free shared buff/cache available
Mem: 61Gi 24Gi 13Gi 612Ki 22Gi 37Gi
Swap: 0B 0B 0B
# redis-cli info | grep -E '(used_memory_(peak_)?human)'
used_memory_human:22.94G
used_memory_peak_human:23.24G
```
This instance is managed with Ansible. Not in producion yet.
It was running fine with Debian bullseye, then we re-deployed it on a
bookworm VM. On this new host, Redis crashes every two hours roughly:
```
# journalctl | grep redis | grep code=killed | tail
Oct 28 14:58:30 host systemd[1]: redis.service: Main process exited,
code=killed, status=11/SEGV
Oct 28 16:44:24 host systemd[1]: redis.service: Main process exited,
code=killed, status=11/SEGV
Oct 28 18:49:49 host systemd[1]: redis.service: Main process exited,
code=killed, status=11/SEGV
Oct 28 21:07:28 host systemd[1]: redis.service: Main process exited,
code=killed, status=11/SEGV
Oct 28 22:54:32 host systemd[1]: redis.service: Main process exited,
code=killed, status=11/SEGV
Oct 29 00:39:06 host systemd[1]: redis.service: Main process exited,
code=killed, status=11/SEGV
Oct 29 02:43:30 host systemd[1]: redis.service: Main process exited,
code=killed, status=11/SEGV
```
Looking at the Redis log files, we see this kind of line, repeated
hundreds of times within a few seconds, before Redis finally crashes:
```
85555:M 15 Oct 2023 01:55:55.811 # Out Of Memory allocating 24576 bytes!
```
First thing I did was to disable the RDB snapshot (set every hours in
our config), just to make sure it was not related. It was not, Redis
kept crashing.
Redis RAM usage is rather constant for us (from 22G to 24G), and on this
machine there's only mirrorbits+redis running. There's plenty of RAM
available, I monitored the RAM during a crash, and `free` reports around
37G of RAM available. So I don't think we're running out of RAM.
I checked what changed in the Redis package, between bullseye and
bookworm, and this commit stands out:
d/redis-server.service: harden systemd service file
https://salsa.debian.org/lamby/pkg-redis/-/commit/8fec88c1
I tried to revert the systemd unit file to the bullseye version, and
Redis worked again, no more crash. From there I re-enabled the changes
one by one, until I found the setting that causes the crash:
ProcSubset=pid
I'm not really knowledgeable regarding systemd hardening, but after
reading the doc, it seems pretty clear that this setting is
questionable, and probably shouldn't be enabled.
Quoting systemd.exec(5)`:
If "pid", all files and directories not directly associated with process
management and introspection are made invisible in the /proc/ file
system configured for the unit's processes. [...] Note that Linux
exposes various kernel APIs via /proc/, which are made unavailable with
this setting. Since these APIs are used frequently this option is useful
only in a few, specific cases, and is not suitable for most non-trivial
programs.
At this point I think there's enough information to support disabling
ProcSubset=pid. Please tell me if you need more information, since the
issue is reproducible, it's easy for me to provide more logs.
Thanks in advance!
Arnaud
--- End Message ---
--- Begin Message ---
Source: redis
Source-Version: 5:7.2.2-2
Done: Chris Lamb <[email protected]>
We believe that the bug you reported is fixed in the latest version of
redis, which is due to be installed in the Debian FTP archive.
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to [email protected],
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
Chris Lamb <[email protected]> (supplier of updated redis package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing [email protected])
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Format: 1.8
Date: Tue, 31 Oct 2023 16:44:01 +0100
Source: redis
Built-For-Profiles: nocheck
Architecture: source
Version: 5:7.2.2-2
Distribution: experimental
Urgency: medium
Maintainer: Chris Lamb <[email protected]>
Changed-By: Chris Lamb <[email protected]>
Closes: 1055039
Changes:
redis (5:7.2.2-2) experimental; urgency=medium
.
* Drop ProcSubset=pid hardening flag from the systemd unit files it appears
to cause crashes with memory allocation errors. A huge thanks to Arnaud
Rebillout <[email protected]> for the extensive investigation.
(Closes: #1055039)
Checksums-Sha1:
6eaaede5fdd11bfb06b4de3afcb107913bb3ad26 2231 redis_7.2.2-2.dsc
b28d2b0a953a9b13f9f7b1e1893c0c520139a795 28892 redis_7.2.2-2.debian.tar.xz
457e0a9d4eb06b6747e2595ea37dcf4a30dfb879 7487 redis_7.2.2-2_amd64.buildinfo
Checksums-Sha256:
0bc89ec1701a5a32b5eb779a4878d763803e6718d4a439406a1e87c435c014cf 2231
redis_7.2.2-2.dsc
14b8ad8f28219555238032ebdb1494f59fcbc22a27f7c1562868005a950597cb 28892
redis_7.2.2-2.debian.tar.xz
c759b755b6c5c3be5bed5bc134d653a64ad4b3c46ca90f15a6663bdcc99db84b 7487
redis_7.2.2-2_amd64.buildinfo
Files:
4de7023f8eeaeb14adb79c3335d76fb8 2231 database optional redis_7.2.2-2.dsc
5992fb3a9068e1694d9cca3bda18ee2d 28892 database optional
redis_7.2.2-2.debian.tar.xz
b09e84d02424ec3dc6fbf737ce973a9d 7487 database optional
redis_7.2.2-2_amd64.buildinfo
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCAAdFiEEwv5L0nHBObhsUz5GHpU+J9QxHlgFAmVBIWgACgkQHpU+J9Qx
HlgPcA//ZBYwdCN2yxd14H+GoRcz1C6cx9gNYh6meAVIEM0+vW1kQvAwls47FNi0
hWFhEubfM92LQef4+CUia7KAt2nBq/QvX+nk+Drfos650yEbxtE81VLZlx7rf+v9
/PpG2D5inCZhQjU2S5FFl3vkV2fpiJxfhdDAtYrP/097R9LXAAPlGtsJ5FIK3lmh
B9UBmeYyvcFo3fCQCRAvCzsI8aKsuE+wKs0pgxW17e8vgg99JoAI+hrAxl0mATww
THGJvk4VKeAlT48ye80iwn9O6FdzoraAw9O+x8CFebcQvLG2bAKoBCzk1cDBx154
5zXNnWgZr1TJWJGTBx7uhX53ubsTgvcICE5BrlMnoqtkmDKO/tenPtysydhaslce
Rcymt75zBpqJ4i/ml5zKOXecyQZlg3lPSTX1dYrCsxyLZII4i/KCzZNSpvIWF7Fw
UA+OmK/Q1bgVn6rgnRmfdgaWEVS2w+6Gf5PP//Svv4KHJlxla04Hx9s8Mbce4R36
S0JDxMlSVWFuiVX7hSvM7GKgR2EY6R40SK0YIVrjYGZdA9dEWwuon3OyhO7FS/pX
nxQeWYqumDg8RHv7jp8Pek97C9lfLxCrr6jCdEvlA+NIOeahyE2zTXg94RPNJj+c
AMrcGp1eJVt6qKxm1iU8xldigho9jZ9FYAhh2YwtUW8qfUYjOD0=
=GzHg
-----END PGP SIGNATURE-----
--- End Message ---
