Bug#993738: udev: symlink creation race condition can cause log delays
Am 05.09.21 um 21:02 schrieb Lukas Schwaighofer: Hi, Thanks for finding that! I'll report back once I've had a chance to try this. The attached PR is quite large and I wasn't able to apply it against the version now in bullseye, and I don't want to change to a different systemd/udev altogether. Carefully reading the descriptions I'm confident that this is indeed the same issue. I have a workaround in place now that doesn't have any drawbacks for me. I'll just stick with that for now. Hm, an alternative could be to revert the offending commits, i.e. the ones listed in https://github.com/systemd/systemd/issues/20212#issue-942476183 This is apparently what was done for RHEL8 (by the author of those patches). Michael OpenPGP_signature Description: OpenPGP digital signature
Bug#993738: udev: symlink creation race condition can cause log delays
Hi, > Thanks for finding that! I'll report back once I've had a chance to > try this. The attached PR is quite large and I wasn't able to apply it against the version now in bullseye, and I don't want to change to a different systemd/udev altogether. Carefully reading the descriptions I'm confident that this is indeed the same issue. I have a workaround in place now that doesn't have any drawbacks for me. I'll just stick with that for now. Thanks for your help Lukas pgpki4K5ItXCE.pgp Description: OpenPGP digital signature
Bug#993738: udev: symlink creation race condition can cause log delays
Hi Michael, thanks a lot for your quick response. On Sun, 5 Sep 2021 19:44:15 +0200 Michael Biebl wrote: > So you create a lot of partitions which all have the same label? > > Isn't this asking for trouble anyway as you can't be certain, which > device the /dev/disk/by-label/dummy will point to? I'm using this in a virtualization setup. The LVs I use have a label of "root", "var", etc. Within the VMs those are unique and are used in /etc/fstab.. I don't care about these labels on the host system but I'd like my system to not break if possible :) . > Sounds like > https://github.com/systemd/systemd/issues/20212 > > You might give the linked PR > https://github.com/systemd/systemd/pull/20603 > a try Thanks for finding that! I'll report back once I've had a chance to try this. Regards Lukas pgpXz6VYZdwMP.pgp Description: OpenPGP digital signature
Bug#993738: udev: symlink creation race condition can cause log delays
Am 05.09.21 um 19:34 schrieb Michael Biebl:
Am 05.09.21 um 19:24 schrieb Lukas Schwaighofer:
for i in `seq 0 49` ; do
lvcreate -L 4M -n dummy-${i} vgtest
mkfs.ext4 -L dummy /dev/vgtest/dummy-${i}
So you create a lot of partitions which all have the same label?
Isn't this asking for trouble anyway as you can't be certain, which
device the /dev/disk/by-label/dummy will point to?
Sounds like
https://github.com/systemd/systemd/issues/20212
You might give the linked PR
https://github.com/systemd/systemd/pull/20603
a try
OpenPGP_signature
Description: OpenPGP digital signature
Bug#993738: udev: symlink creation race condition can cause log delays
Am 05.09.21 um 19:24 schrieb Lukas Schwaighofer:
Package: udev
Version: 247.3-6
Severity: important
Dear systemd/udev Maintainers,
I've noticed a race condition with udev symlink creation, that can
cause long delays when activating logical volumes in LVM. Steps to
reproduce the problem (needs root):
# prepare
truncate -s 1G test.img
loop="$(losetup --show -f test.img)"
echo "Loop device used: $loop"
pvcreate "$loop"
vgcreate vgtest "$loop"
# create 50 LVs with an ext4 fs, all with same label
# (increase the number to make the problem more sever)
for i in `seq 0 49` ; do
lvcreate -L 4M -n dummy-${i} vgtest
mkfs.ext4 -L dummy /dev/vgtest/dummy-${i}
So you create a lot of partitions which all have the same label?
Isn't this asking for trouble anyway as you can't be certain, which
device the /dev/disk/by-label/dummy will point to?
OpenPGP_signature
Description: OpenPGP digital signature
Bug#993738: udev: symlink creation race condition can cause log delays
Package: udev
Version: 247.3-6
Severity: important
Dear systemd/udev Maintainers,
I've noticed a race condition with udev symlink creation, that can
cause long delays when activating logical volumes in LVM. Steps to
reproduce the problem (needs root):
# prepare
truncate -s 1G test.img
loop="$(losetup --show -f test.img)"
echo "Loop device used: $loop"
pvcreate "$loop"
vgcreate vgtest "$loop"
# create 50 LVs with an ext4 fs, all with same label
# (increase the number to make the problem more sever)
for i in `seq 0 49` ; do
lvcreate -L 4M -n dummy-${i} vgtest
mkfs.ext4 -L dummy /dev/vgtest/dummy-${i}
done
# set all LVs in vgtest as inactive
vgchange -an vgtest
# trigger the problem by activating all lvs with the same label at
# once (run `journalctl -f` in a different shell to see log
# messages)
vgchange -ay vgtest
# can be repeated by setting as inactive and then active again
# cleanup
vgchange -an vgtest
losetup -d "$loop"
rm test.img
Observe that the activation takes a very long time. The log is flodded
with messages like
dm-??: Failed to update device symlinks: Too many levels of symbolic links
The devices are opened only very slowly (observe by running
`ls /dev/vgtest` while the activation is in progress). This has
rendered one of my systems unbootable (wait for one of the necessary
volumes times out during boot).
I've temporarily worked around the issue by disabling the "by-label"
symlinks in udev for device mapper devices:
sed '/by-label/ s/^/#/' \
/lib/udev/rules.d/60-persistent-storage-dm.rules \
> /etc/udev/rules.d/60-persistent-storage-dm.rules
udevadm control --reload
Notice how the activation now is very fast. To undo this workaround
remove /etc/udev/rules.d/60-persistent-storage-dm.rules and reload udev
rules again.
The same issue is *not* present in the udev version from buster
(241-7~deb10u8).
Thank you
Lukas Schwaighofer
pgp7lJETh0lyP.pgp
Description: OpenPGP digital signature
