Bug#996177: cryptsetup: please report fatal errors without having to use -v
On Thu, 14 Oct 2021 at 20:48:51 +0200, Marc Lehmann wrote: > I reported this from another system, but both were recently upgraded to > bullseye. > > I know because I use kvm to see if the machine will actually boot (Cthus > the different memory setup) and the kvm in bullseye has a bug that makes > this very hard (remote display makes it freeze randomly), and I had to > work around this bug, so I know it was not buster. Could still be an older initramfs image though. If you're able to reproduce this please run `cryptsetup --version` directly afterwards (i.e., at initramfs stage if that's where the issue appears). >> Looking at the upstream git log, I found >> 206b70c837f29c8b34cb0d80ae496870550ec50c >> which fixes https://gitlab.com/cryptsetup/cryptsetup/-/issues/488 which looks >> really familiar :-) > > It looks very similar. It is not the message I got with -v, which > specifically had the error number (3) in it somewhere, but maybe thats > because it ran out of memory in a different place. My reproducer (with cryptsetup 2.1.0) does have “Command failed with code -3 (out of memory)” with ‘-v’: (initramfs) free totalusedfree shared buff/cache available Mem: 493060 29808 363896 40 99356 364040 Swap: 0 0 0 (initramfs) cryptsetup luksDump /dev/vda5 […] Keyslots: 0: luks2 Key:512 bits Priority: normal Cipher: aes-xts-plain64 Cipher key: 512 bits PBKDF: argon2i Time cost: 4 Memory: 605915 Threads:2 […] (initramfs) cryptsetup luksOpen /dev/vda5 --keyfile-size=32 --key-file=/dev/urandom --test-passphrase (initramfs) echo $? 3 (initramfs) cryptsetup luksOpen -v /dev/vda5 --keyfile-size=32 --key-file=/dev/urandom --test-passphrase Command failed with code -3 (out of memory). (initramfs) cryptsetup luksOpen --debug /dev/vda5 --keyfile-size=32 --key-file=/dev/urandom --test-passphrase # cryptsetup 2.1.0 processing "cryptsetup luksOpen --debug /dev/vda5 --keyfile-size=32 --key-file=/dev/urandom --test-passphrase" # Running command open. # Locking memory. # Installing SIGINT/SIGTERM handler. # Unblocking interruption on signal. # Allocating context for crypt device /dev/vda5. # Trying to open and read device /dev/vda5 with direct-io. # Initialising device-mapper backend library. # Trying to load any crypt type from device /dev/vda5. # Crypto backend (OpenSSL 1.1.1d 10 Sep 2019) initialized in cryptsetup library version 2.1.0. # Detected kernel Linux 4.19.0-18-amd64 x86_64. # Loading LUKS2 header (repair disabled). # Opening lock resource file /run/cryptsetup/L_254:5 # Acquiring read lock for device /dev/vda5. # Verifying read lock handle for device /dev/vda5. # Device /dev/vda5 READ lock taken. # Trying to read primary LUKS2 header at offset 0x0. # Opening locked device /dev/vda5 # Veryfing locked device handle (bdev) # LUKS2 header version 2 of size 16384 bytes, checksum sha256. # Checksum:e3d5da875cd56c8d48144ec6ef85229a8bdf52ad42a6c8b98a3b72ad32ece6de (on-disk) # Checksum:e3d5da875cd56c8d48144ec6ef85229a8bdf52ad42a6c8b98a3b72ad32ece6de (in-memory) # Trying to read secondary LUKS2 header at offset 0x4000. # Opening locked device /dev/vda5 # Veryfing locked device handle (bdev) # LUKS2 header version 2 of size 16384 bytes, checksum sha256. # Checksum:af4ba03f7cdd87c763d505ae21b76c475fb072428949c8a87e94e15bbe54339b (on-disk) # Checksum:af4ba03f7cdd87c763d505ae21b76c475fb072428949c8a87e94e15bbe54339b (in-memory) # Device size 3781165056, offset 16777216. # Device /dev/vda5 READ lock released. # Only 2 active CPUs detected, PBKDF threads decreased from 4 to 2. # Not enough physical memory detected, PBKDF max memory decreased from 1048576kB to 246530kB. # PBKDF argon2i, hash sha256, time_ms 2000 (iterations 0), max_memory_kb 246530, parallel_threads 2. # Checking volume passphrase using token -1. # File descriptor passphrase entry requested. # Checking volume passphrase [keyslot -1] using passphrase. # Keyslot 0 priority 1 != 2 (required), skipped. # Trying to open LUKS2 keyslot 0. # Keyslot 0 (luks2) open failed with -12. # Releasing crypt device /dev/vda5 context. # Releasing device-mapper backend. # Unlocking memory. Command failed with code -3 (out of memory). However, after upgrading (and rebuilding the initramfs) I get “Not enough available memory to open a keyslot.” instead of having to pass ‘-v’, ‘--debug’ or inspect the return code. -- Guilhem. signature.asc Description: PGP signature
Bug#996177: cryptsetup: please report fatal errors without having to use -v
On Tue, Oct 12, 2021 at 02:25:31AM +0200, Guilhem Moulin wrote: > > Also did you run luksFormat on that same machine (and didn't remove > > memory sticks afterwards)? > > I was actually able to reproduce this with buster's 2.1.0, but doing the > same thing after upgrading to bullseye or sid yields > which sounds alright. Did you run reportbug(1) from the affected system > or from an older one? I reported this from another system, but both were recently upgraded to bullseye. I know because I use kvm to see if the machine will actually boot (Cthus the different memory setup) and the kvm in bullseye has a bug that makes this very hard (remote display makes it freeze randomly), and I had to work around this bug, so I know it was not buster. > Looking at the upstream git log, I found > 206b70c837f29c8b34cb0d80ae496870550ec50c > which fixes https://gitlab.com/cryptsetup/cryptsetup/-/issues/488 which looks > really familiar :-) It looks very similar. It is not the message I got with -v, which specifically had the error number (3) in it somewhere, but maybe thats because it ran out of memory in a different place. I tried to recreate the conditions again (which was basically runnign kvm without -m option), but I can't get the kernel to boot with the default memory, and the minimum amount I got it to boot with was 256MB, at which cryptsetup was able to open the volume. -- The choice of a Deliantra, the free code+content MORPG -==- _GNU_ http://www.deliantra.net ==-- _ generation ---==---(_)__ __ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / schm...@schmorp.de -=/_/_//_/\_,_/ /_/\_\
Bug#996177: cryptsetup: please report fatal errors without having to use -v
On Thu, 14 Oct 2021 at 19:53:14 +0200, Marc Lehmann wrote: > On Tue, Oct 12, 2021 at 12:33:32AM +0200, Guilhem Moulin > wrote: >> Could you please share the memory cost of the PBKDF, > > I wouldn't know how to do that. `cryptsetup luksDump` >> of `free` just before running cryptsetup? That would help us trying to >> reproduce this and forward the issue upstream. I tried to reproduce >> this but only managed to trigger the OOM killer. > > Not sure what this extra information has to do with the problem I reported Which is why I'm saying, I tried to reproduce this and failed. >> Also did you run luksFormat on that same machine (and didn't remove >> memory sticks afterwards)? > > There was _significantly_ less memory in the machine at the time of the > problem. Unrelated to this issue, it's best to run `cryptsetup luksChangeKey` after hardware changes to use adequate PBKDF parameters. > I think you are likely confused about the nature of the problem I reported > - the problem is not that cryptsetup used too much memory. The problem > is that it only reported it when using an extra option, as opposed to > reporting it when it hit the issue. > > cryptsetup was not killed by the oom killer, so it should have had no > problems reporting the result of some failed allocation. Got that. And again I believe this is https://gitlab.com/cryptsetup/cryptsetup/-/issues/488 . Are you really running 2:2.3.5-1 on the affected system? -- Guilhem. signature.asc Description: PGP signature
Bug#996177: cryptsetup: please report fatal errors without having to use -v
On Tue, Oct 12, 2021 at 12:33:32AM +0200, Guilhem Moulin wrote: > Could you please share the memory cost of the PBKDF, I wouldn't know how to do that. > of `free` just before running cryptsetup? That would help us trying to > reproduce this and forward the issue upstream. I tried to reproduce > this but only managed to trigger the OOM killer. Not sure what this extra information has to do with the problem I reported - the problem is that the oom error is not reported unless -v is used, which has nothing to do with the oom killer or how much memory was needed - cryptsetup surely had reasons to use a lot of memory, but it should report an error if it can, as opposed to silently fail to do its job. > Also did you run luksFormat on that same machine (and didn't remove > memory sticks afterwards)? There was _significantly_ less memory in the machine at the time of the problem. I think you are likely confused about the nature of the problem I reported - the problem is not that cryptsetup used too much memory. The problem is that it only reported it when using an extra option, as opposed to reporting it when it hit the issue. cryptsetup was not killed by the oom killer, so it should have had no problems reporting the result of some failed allocation. -- The choice of a Deliantra, the free code+content MORPG -==- _GNU_ http://www.deliantra.net ==-- _ generation ---==---(_)__ __ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / schm...@schmorp.de -=/_/_//_/\_,_/ /_/\_\
Bug#996177: cryptsetup: please report fatal errors without having to use -v
Control: found -1 2:2.1.0-5+deb10u2 On Tue, 12 Oct 2021 at 00:33:32 +0200, Guilhem Moulin wrote: > On Mon, 11 Oct 2021 at 22:09:07 +0200, Marc Lehmann wrote: >> Specifically, the machine didn't have enough ram, probably because the >> default algorithm (argon) requires more ram than the machine had. > > Could you please share the memory cost of the PBKDF, and also the output > of `free` just before running cryptsetup? That would help us trying to > reproduce this and forward the issue upstream. I tried to reproduce > this but only managed to trigger the OOM killer. > > Also did you run luksFormat on that same machine (and didn't remove > memory sticks afterwards)? I was actually able to reproduce this with buster's 2.1.0, but doing the same thing after upgrading to bullseye or sid yields (initramfs) cryptsetup luksOpen /dev/vda5 test_crypt Enter passphrase for /dev/vda5: [5.236310] device-mapper: uevent: version 1.0.3 [5.240230] device-mapper: ioctl: 4.43.0-ioctl (2020-10-01) initialised: dm-de...@redhat.com Not enough available memory to open a keyslot. which sounds alright. Did you run reportbug(1) from the affected system or from an older one? Looking at the upstream git log, I found 206b70c837f29c8b34cb0d80ae496870550ec50c which fixes https://gitlab.com/cryptsetup/cryptsetup/-/issues/488 which looks really familiar :-) -- Guilhem. signature.asc Description: PGP signature
Bug#996177: cryptsetup: please report fatal errors without having to use -v
Control: tag -1 moreinfo Hi, On Mon, 11 Oct 2021 at 22:09:07 +0200, Marc Lehmann wrote: > Specifically, the machine didn't have enough ram, probably because the > default algorithm (argon) requires more ram than the machine had. Could you please share the memory cost of the PBKDF, and also the output of `free` just before running cryptsetup? That would help us trying to reproduce this and forward the issue upstream. I tried to reproduce this but only managed to trigger the OOM killer. Also did you run luksFormat on that same machine (and didn't remove memory sticks afterwards)? -- Guilhem. signature.asc Description: PGP signature
Bug#996177: cryptsetup: please report fatal errors without having to use -v
On 11/10/2021 22:09, Marc Lehmann wrote: > > Specifically, the machine didn't have enough ram, probably because the > default algorithm (argon) requires more ram than the machine had. > > Unfortunately, cryptsetup silently fails - you get a password prompt, youc > an enter any password, your shell prompt appears without any message but > nothing happened. If the process was killed by OOM (out of memory daemon), then it cannot print anything - it is simply stopped by the kernel. But if it really fails with return code 3 (out of memory), then it should print a visible error - if not, we have to fix it. (Verbose option only translates return code to something readable but there should be an error message much earlier.) Is it simple cryptsetup open activation (no systemd-cryptsetup magic)? (If so, please add new issue upstream and link this bug, we will fix that.) Thanks, Milan
Bug#996177: cryptsetup: please report fatal errors without having to use -v
Package: cryptsetup Version: 2:2.3.5-1 Severity: wishlist Dear Maintainer, I just had a somewhat lengthy initramfs debug issue caused by some suboptimal cryptsetup behaviour. Specifically, the machine didn't have enough ram, probably because the default algorithm (argon) requires more ram than the machine had. Unfortunately, cryptsetup silently fails - you get a password prompt, youc an enter any password, your shell prompt appears without any message but nothing happened. Only when running cryptsetup with -v did it output an error message akin to (repeated from memory): error -3 not enough memory I think requiring users to use --verbose to see fatal error messages is highly suboptimal behaviour. It would be nice if future versions of cryptsetup would output error messages by default, at least when they are fatal, i.e. the action could not be executed. Thanks! -- Package-specific info: -- System Information: Debian Release: 11.1 APT prefers stable-updates APT policy: (990, 'stable-updates'), (990, 'stable-security'), (990, 'stable'), (500, 'unstable-debug'), (500, 'testing-debug'), (500, 'stable-debug'), (500, 'unstable'), (500, 'testing'), (500, 'oldstable'), (1, 'experimental-debug'), (1, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: i386, x32 Kernel: Linux 5.10.0-8-amd64 (SMP w/8 CPU threads) Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE Locale: LANG=en_IE.UTF-8, LC_CTYPE=en_IE.UTF-8 (charmap=UTF-8), LANGUAGE not set Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages cryptsetup depends on: ii cryptsetup-bin 2:2.3.5-1 ii debconf [debconf-2.0] 1.5.77 ii dmsetup2:1.02.175-2.1 ii libc6 2.31-13+deb11u2 Versions of packages cryptsetup recommends: ii cryptsetup-initramfs 2:2.3.5-1 ii cryptsetup-run2:2.3.5-1 Versions of packages cryptsetup suggests: ii dosfstools 4.2-1 ii keyutils1.6.1-2 ii liblocale-gettext-perl 1.07-4+b1 -- debconf information excluded
