Package: fai-server
Version: 6.0.3+deb12u1
Severity: important
Dear Maintainer,
In bin/fai-diskimage, cleanup() runs "rm -rf $mnt" unconditionally after the
umount retry loop:
for dir in $(mount | awk '{print $3}'| grep $mnt | sort -r); do
for i in {1..15}; do
umount $dir >&/dev/null && break
...
done
echo "ERROR: fai-diskimage cannot umount $dir" >&2
done
...
rm -rf $mnt
If a umount under $mnt fails, that mount is still live when rm -rf runs, and rm
recurses through it. When the failing mount is $mnt/dev -- which lib/updatebase
bind-mounts from the build host:
if [ -f /etc/init.d/udev ]; then
mount --bind /dev $FAI_ROOT/dev
mount --make-private $FAI_ROOT/dev
-- the deletion lands in the build host's real /dev.
Observed on a Debian 12 build host running sysvinit (systemd not installed),
building an image with a large package set:
Still cannot umount /tmp/fai-diskimage.XXXXXX/dev
<http://fai-diskimage.xxxxxx/dev>. Will try again.
ERROR: fai-diskimage cannot umount /tmp/fai-diskimage.XXXXXX/dev
<http://fai-diskimage.xxxxxx/dev>
rm: cannot remove '/tmp/fai-diskimage.XXXXXX/dev/shm
<http://fai-diskimage.xxxxxx/dev/shm>': Device or resource busy
rm: cannot remove '/tmp/fai-diskimage.XXXXXX/dev/pts
<http://fai-diskimage.xxxxxx/dev/pts>': Device or resource busy
Afterwards the host's /dev held only three entries and /dev/vda* was gone. The
root filesystem stayed mounted, so the machine kept running, but it was left
with essentially no device nodes. The image itself had built correctly; the
damage was done entirely in cleanup.
I suspect this is rarely seen because the bind-mount is conditional on
/etc/init.d/udev existing. On a systemd build host that file is absent, the
host's /dev is never placed under $mnt, and so rm -rf $mnt cannot reach it. It
becomes reachable on a sysvinit build host, where the file is present.
The umount itself can fail for exactly the reason the loop's own comment
anticipates -- a package postinst leaving something using the filesystem beyond
the retry budget. I did not identify the specific holder in my case.
A minimal fix would be to stop rm crossing the mount boundary:
rm -rf --one-file-system $mnt
This is a no-op on the normal path, where everything under $mnt has already
been unmounted, and prevents the traversal when a umount has failed. Skipping
the rm altogether when any umount failed would be stronger still.
Note the "ERROR: fai-diskimage cannot umount $dir" message is currently printed
unconditionally after the inner loop, including when the umount succeeded, so
it is not on its own a reliable signal of this condition.
The same unguarded "rm -rf $mnt" is present in 6.4.3 (trixie) and 6.6 (sid).
Thanks for FAI.
Thanks
--MKL
