Package: libpam-fscrypt
Version: 0.3.7-1
Severity: important
X-Debbugs-Cc: [email protected]
Dear Maintainer,
I could describe the bug, but since it is a complex topic for me, I
prefer to
share the analysis performed using local AI; it goes into detail and
provides
an alternative solution as well as guidance on how to fix the issue.
For now, to resolve the issue, I removed the libpam-fscrypt package.
Thanks,
Antonio
-- Description: --
`pam_fscrypt.so`, when enabled in the PAM session stack (`/etc/pam.d/common-
session`, as shipped by the Debian packaging of `libpam-fscrypt`),
enumerates
mount points taken from `/proc/self/mountinfo` and calls `stat()` on them
during `pam_sm_open_session()`.
If any of those mount points is a FUSE mount whose daemon is alive but not
serving requests, that `stat()` never returns: the calling thread goes into
uninterruptible sleep (`D`, `wchan=fuse_get_req`). Because PAM session
opening
is *in-line* with the SSH connection setup in `sshd-session`, this
blocks the
SSH connection itself — and, from then on, every subsequent SSH login on
that
host.
The trigger is completely ordinary: mounting an `sshfs` share whose SSH
server
runs on the same host (or in the same mount namespace) as the FUSE
client. In
that situation `sshfs` creates the mount point *before* its SFTP
handshake has
completed, so the new mount point exists but cannot be served; meanwhile the
daemon is waiting for data on the very SSH connection whose session setup is
now blocked in `stat()` on that mount point.
Neither side can progress and there is no timeout anywhere in the path: a
textbook circular wait.
Note that this is *not* an sshfs bug and not an SFTP problem: plain `ssh`,
`sftp` and a manually invoked `sftp-server` all work correctly on the same
machine.
-- Steps to reproduce --
Prerequisites: Debian forky/sid, `libpam-fscrypt 0.3.7-1`, `openssh-server
1:10.5p1-1`, `sshfs 3.7.3-1.2`, `fuse3 3.18.3-1`; sshd listening on port 22
(any port); key-based auth for root; `session optional pam_fscrypt.so`
enabled
in `/etc/pam.d/common-session`.
1. As root: `sshfs -p 22 root@localhost:/ /mnt`
-> the command never returns and prints nothing (expected: instant
mount).
2. In another terminal, as root or any user:
`timeout -s KILL 15 ssh -p 22 -o BatchMode=yes root@localhost
'echo hi'`
-> "hi" is never printed; the command is killed by the timeout
(rc=137).
(expected: instant output)
3. `ps -eo pid,state,comm,wchan:30 | awk '$2=="D"'`
-> one or more `sshd-session` processes stuck in `D`,
`wchan=fuse_get_req`.
Recovery requires killing the FUSE daemon from a local console (or an
already-
open session); SIGTERM/SIGKILL do not remove the tasks while they are in
`D`:
# kill -9 <sshfs pid> # requests then fail immediately with
ENOTCONN
# umount -l /mnt
-- Actual result --
`sshfs` hangs indefinitely; every subsequent SSH login hangs in PAM session
open; the hung tasks are unkillable until the FUSE daemon disappears.
-- Expected result --
PAM session opening must never block on a filesystem it does not need.
`fscrypt` has no reason to `stat()` arbitrary mount points synchronously
inside
sshd's session phase, and certainly not without any protection against non-
responding filesystems.
-- Impact --
* Any local/loopback `sshfs` (or other FUSE) usage can silently render
the SSH
service of a host unusable: logins hang after authentication, with no
error
message anywhere.
* Because PAM session open is in-line, sshd's connection slots and
`MaxStartups` fill up with hung sessions.
* The condition is self-perpetuating: each failed attempt leaves another
dead
mount point in `/proc/self/mountinfo`, which hangs the next login as well.
* Recovery needs local console access; on a headless box this is a hard
lockout
(hence severity important rather than normal).
-- Workaround used --
Commenting out the module in `/etc/pam.d/common-session`:
-session optional pam_fscrypt.so
+#session optional pam_fscrypt.so
No sshd restart is needed (PAM configuration is read per
authentication). After
this, `sshfs -p 22 root@localhost:/ /mnt` mounts instantly and unmounts
cleanly. This is safe on the affected machine because fscrypt was never
provisioned there:
$ sudo fscrypt status
filesystems supporting encryption: 0
filesystems with fscrypt metadata: 0
[ERROR] fscrypt status: filesystem / is not setup for use with fscrypt
i.e. there are no fscrypt policies to unlock at login, so the module was
doing
nothing except hanging logins.
-- Proposed fix --
In `pam_sm_open_session()` (and in any mount-table walk in the library):
* Do not `stat()` mount points of filesystem types that cannot carry an
fscrypt policy (`fuse`, `fuse.sshfs`, network filesystems, ...) — the
information needed is already available as text in `/proc/self/mountinfo`.
* If a stat is unavoidable, make it non-blocking / interruptible:
`openat(..., O_PATH | O_NONBLOCK)` or `statx(..., AT_STATX_DONT_SYNC)`,
and/or
run the enumeration in a child process with a deadline so that PAM
session open
can never block indefinitely.
Additionally (separate, lower priority; sshfs side): creating the FUSE mount
before the SFTP handshake succeeds exposes a mount point that cannot be
served
for an unbounded time window. Mounting only after the handshake
completes — or
tearing the mount down immediately when the handshake fails/times out —
would
remove this class of deadlock and the classic "orphaned sshfs mount"
problem in
general.
-- End of report --
-- System Information: --
Debian Release: forky/sid
APT prefers unstable
Architecture: amd64 (x86_64)
Kernel: Linux 7.2.6-1-liquorix-amd64 (SMP w/32 CPU cores)
Locale: LANG=it_IT.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8), LANGUAGE=it
(charmap=UTF-8)
Shell: /bin/sh linked to: dash
Versions of packages related to libpam-fscrypt:
ii fscrypt 0.3.7-1
ii libpam-fscrypt 0.3.7-1
ii fuse3 3.18.3-1
ii openssh-server 1:10.5p1-1
ii sshfs 3.7.3-1.2
