Public bug reported:

This might instead be a bug in util-linux's fsck, but Karol Zak tells me
it's systemd's or udevd's problem:

https://github.com/util-linux/util-linux/issues/4477

I disagree, but what do I know?  I'm coming to you because it needs a
person who can consider how these programs work together as a system.
I'd report to systemd upstream, but their bug reporting guidelines
suggest to start with my distro, and you also have a relevant patch in
play not accepted by upstream (lp2090972-libblkid-fix-spurious-ext-
superblock-checksum-mismat.patch in util-linux 2.39.3-9ubuntu6.4.

At my work, we've been seeing mount failures, very intermittently. I
have a reproducible test case in my local vm where I can cause the mount
failure within 1000 reboots or so in a certain configuration of Noble
(with some patches of our own applied, but nothing I think is relevant
to this). Debugging into it some I've narrowed it down to happen in a
2nd run of udevd against an add uevent for an ext4 filesystem. The
initramfs phase run of udevd over this event seems to be okay, adding
the by-uuid symlink and udev database entry. I've added debugging to
show processing of the following 60-persistent-storage.rules rule
returning from in udev_rule_apply_line_to_event() in udev-rules.c
without a uuid symlink (devlink) struct object being added to the in the
dev sd_device struct object for the event.

KERNEL!="sr*|mmcblk[0-9]boot[0-9]", IMPORT{builtin}="blkid"

With that missing, at the end of rule processing in
udev_event_execute_rules() the call to udev_node_update() calls
udev_node_update() with a sd_device dev argument that lacks the devlink
for the by-uuid symlink for the filesystem that will fail to mount.
udev_node_update() looks for devlinks present in event->dev_db_clone (a
representation of the udev database before processing this event?) to
see which are present under dev's devlink collection. It finds it
missing in dev's collection and calls link_update which removes the
corresponding /dev/disk/by-uuid symlink on the actual drive. When that
gets removed the systemd device unit for that symlink gets deactivated
(state to "inactive (dead)" as shown by systemctl status).  systemd-
[email protected] has a binds to dependency so also gets deactivated to the
same inactive (dead) state. The mount unit has a requires/after
dependency on the systemd-fsck service unit for that uuid (my fstab uses
uuids to identify filesystems). So that never gets activated, and the
mount never happens.

I reproduce this by adding the following unit to a VM running Noble
(with our company's patches):

        [Unit]
        Description=Reboot if no mounts fail
        SuccessAction=reboot
        DefaultDependencies=no
        Requires=local-fs.target
        After=local-fs.target

        [Install]
        WantedBy=sysinit.target

I then set the grub.cfg timeout value to zero (instead of 10 seconds).
This way I get the problem within three hours on my QEMU VM, or a little
over 1000 reboots.

Some other info showing the state of the failure. sc below is an eshell
alias for systemctl. Log messages not corresponding to messages in your
source would be from "printf debugging" statements I added myself:

/scp:qvm:/tmp # lsblk -f
NAME   FSTYPE FSVER LABEL UUID                                 FSAVAIL FSUSE% 
MOUNTPOINTS
sda                                                                           
├─sda1 ext4   1.0         75d03876-a9c1-4506-938a-934acf9d72bf    8.2G    12% /
├─sda2 ext4   1.0         1cc3b564-f846-4ad4-9c8b-eae6988f8416    5.2G     5% 
/var
├─sda3 swap   1           5e7bfad3-00ed-4fb2-b021-70645395daa1                
[SWAP]
├─sda4 ext4   1.0         bdc8fdc1-ce85-4d3c-b2be-c2ca96cbf4e7   27.4G     0% 
/cache
└─sda5 vfat   FAT32       2F9C-D226                             501.4M     0% 
/boot/efi
sdb                                                                           
├─sdb1 ext4   1.0         738a91e7-dbd7-4b3d-bff8-846272eb91af   22.2G     0% 
/usr/local/acme
├─sdb2 ext4   1.0         d08ef5fe-6e72-4035-bd9d-ce85238142bd    2.7G     0% 
/tmp
└─sdb3 ext4   1.0         4e53b0ae-1b9d-4731-80aa-9eeb03ee9c3f   21.3G     0% 
/cache2
sdc                                                                           
├─sdc1                                                                        
└─sdc2 ext4   1.0         601f92dd-f6f8-411e-8d81-6f3e1f61e530    5.5G     0% 
/usr/local/acme/coredumps
sdd                                                                           
├─sdd1 ext4   1.0         329989e2-e86d-4114-b8b7-4376f7ea13e5   29.7G     0% 
/cache4
└─sdd2 ext4   1.0         2eed8f14-506d-42e0-a866-2c08f6e9c462   16.6G     0% 
/usr/local/acme/logs
sde                                                                           
└─sde1 ext4   1.0         211a269b-c0ae-4b5a-a440-ed36cc8c9352   46.4G     0% 
/cache5

/scp:qvm:/tmp # sc list-dependencies --all cache3.mount
cache3.mount
● ├─-.mount
● │ └─-.slice
○ ├─dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.device
● ├─system.slice
● │ └─-.slice
○ 
└─systemd-fsck@dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.service
○   
├─dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.device
●   ├─system-systemd\x2dfsck.slice
●   │ └─system.slice
●   │   └─-.slice
●   └─systemd-fsckd.socket
●     ├─-.mount
●     │ └─-.slice
●     └─system.slice
●       └─-.slice


/scp:qvm:/tmp # sc status -l 
'dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.device'
○ dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.device - 
QEMU_HARDDISK primary
     Loaded: loaded
     Active: inactive (dead) since Tue 2026-06-16 20:46:24 UTC; 3min 53s ago
   Duration: 493ms

Jun 16 20:49:42 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.device: 
unit_can_stop_refuse_manual:../src/core/unit.c:6213: can refuse manual
Jun 16 20:49:42 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.device: 
unit_can_stop:../src/core/unit.c:2054: return !!stop() through vtable
Jun 16 20:49:42 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.device: 
unit_can_stop_refuse_manual:../src/core/unit.c:6213: can refuse manual
Jun 16 20:49:42 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.device: 
unit_can_stop:../src/core/unit.c:2054: return !!stop() through vtable
Jun 16 20:50:17 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.device: 
unit_can_stop_refuse_manual:../src/core/unit.c:6213: can refuse manual
Jun 16 20:50:17 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.device: 
unit_can_stop:../src/core/unit.c:2054: return !!stop() through vtable
Notice: journal has been rotated since unit was started, output may be 
incomplete.
/scp:qvm:/tmp [3] # sc status -l 
'systemd-fsck@dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.service'
○ 
systemd-fsck@dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.service
 - File System Check on /dev/disk/by-uuid/bde994ca-3285-47d8-9780-fdddc29b81eb
     Loaded: loaded (/usr/lib/systemd/system/[email protected]; static)
     Active: inactive (dead) since Tue 2026-06-16 20:46:24 UTC; 4min 6s ago
   Duration: 204ms
       Docs: man:[email protected](8)
    Process: 1633 ExecStart=/usr/lib/systemd/systemd-fsck 
/dev/disk/by-uuid/bde994ca-3285-47d8-9780-fdddc29b81eb (code=exited, 
status=0/SUCCESS)
   Main PID: 1633 (code=exited, status=0/SUCCESS)
        CPU: 7ms

Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
systemd-fsck@dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.service:
 service.c: service_enter_stop_post() final else, calling service_enter_signal()
Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
systemd-fsck@dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.service:
 service.c: SERVICE_FINAL_... recursive call to service_enter_signal()
Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
systemd-fsck@dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.service:
 service.c: service_enter_signal() r == 0  final else not fail label, calling 
service_enter_dead()
Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
systemd-fsck@dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.service:
 service.c: service_enter_dead(),  s->result == SERVICE_SUCCESS, calling 
unit_log_success()
Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
systemd-fsck@dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.service:
 Deactivated successfully.
Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: Stopped 
systemd-fsck@dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.service
 - File System Check on /dev/disk/by-uuid/bde994ca-3285-47d8-9780-fdddc29b81eb.
Jun 16 20:49:42 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
systemd-fsck@dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.service:
 unit_can_stop_refuse_manual:../src/core/unit.c:6213: can refuse manual
Jun 16 20:49:42 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
systemd-fsck@dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.service:
 unit_can_stop:../src/core/unit.c:2054: return !!stop() through vtable
Jun 16 20:50:30 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
systemd-fsck@dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.service:
 unit_can_stop_refuse_manual:../src/core/unit.c:6213: can refuse manual
Jun 16 20:50:30 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
systemd-fsck@dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.service:
 unit_can_stop:../src/core/unit.c:2054: return !!stop() through vtable
/scp:qvm:/tmp [3] # *cat /etc/fstab
UUID=75d03876-a9c1-4506-938a-934acf9d72bf       /       ext4    defaults 1 1
UUID=2F9C-D226  /boot/efi       vfat    noatime 0 0
UUID=bdc8fdc1-ce85-4d3c-b2be-c2ca96cbf4e7       /cache  ext4    
inode_readahead_blks=0,noatime 1 4
UUID=4e53b0ae-1b9d-4731-80aa-9eeb03ee9c3f       /cache2 ext4    
inode_readahead_blks=0,noatime 1 4
UUID=bde994ca-3285-47d8-9780-fdddc29b81eb       /cache3 ext4    
inode_readahead_blks=0,noatime 1 4
UUID=329989e2-e86d-4114-b8b7-4376f7ea13e5       /cache4 ext4    
inode_readahead_blks=0,noatime 1 4
UUID=211a269b-c0ae-4b5a-a440-ed36cc8c9352       /cache5 ext4    
inode_readahead_blks=0,noatime 1 4
proc        /proc             proc     defaults 0 0
UUID=d08ef5fe-6e72-4035-bd9d-ce85238142bd       /tmp    ext4    defaults 1 1
UUID=738a91e7-dbd7-4b3d-bff8-846272eb91af       /usr/local/acme ext4    
noatime,inode_readahead_blks=0 1 2
UUID=601f92dd-f6f8-411e-8d81-6f3e1f61e530       /usr/local/acme/coredumps       
ext4    noatime,inode_readahead_blks=0 1 2
UUID=2eed8f14-506d-42e0-a866-2c08f6e9c462       /usr/local/acme/logs    ext4    
noatime,inode_readahead_blks=0 1 2
UUID=1cc3b564-f846-4ad4-9c8b-eae6988f8416       /var    ext4    defaults 1 2
UUID=5e7bfad3-00ed-4fb2-b021-70645395daa1       swap    swap    sw 0 0
/scp:qvm:/tmp # *ls -l /dev/disk/by-uuid/
total 0
lrwxrwxrwx 1 root root 10 Jun 16 20:46 1cc3b564-f846-4ad4-9c8b-eae6988f8416 -> 
../../sda2
lrwxrwxrwx 1 root root 10 Jun 16 20:46 211a269b-c0ae-4b5a-a440-ed36cc8c9352 -> 
../../sde1
lrwxrwxrwx 1 root root 10 Jun 16 20:46 2F9C-D226 -> ../../sda5
lrwxrwxrwx 1 root root 10 Jun 16 20:46 2eed8f14-506d-42e0-a866-2c08f6e9c462 -> 
../../sdd2
lrwxrwxrwx 1 root root 10 Jun 16 20:46 329989e2-e86d-4114-b8b7-4376f7ea13e5 -> 
../../sdd1
lrwxrwxrwx 1 root root 10 Jun 16 20:46 4e53b0ae-1b9d-4731-80aa-9eeb03ee9c3f -> 
../../sdb3
lrwxrwxrwx 1 root root 10 Jun 16 20:46 5e7bfad3-00ed-4fb2-b021-70645395daa1 -> 
../../sda3
lrwxrwxrwx 1 root root 10 Jun 16 20:46 601f92dd-f6f8-411e-8d81-6f3e1f61e530 -> 
../../sdc2
lrwxrwxrwx 1 root root 10 Jun 16 20:46 738a91e7-dbd7-4b3d-bff8-846272eb91af -> 
../../sdb1
lrwxrwxrwx 1 root root 10 Jun 16 20:46 75d03876-a9c1-4506-938a-934acf9d72bf -> 
../../sda1
lrwxrwxrwx 1 root root 10 Jun 16 20:46 bdc8fdc1-ce85-4d3c-b2be-c2ca96cbf4e7 -> 
../../sda4
lrwxrwxrwx 1 root root 10 Jun 16 20:46 d08ef5fe-6e72-4035-bd9d-ce85238142bd -> 
../../sdb2
/scp:qvm:/tmp #

journal entries (I've added logging statements of my own):

45 matches in 33 lines for "sdc1" in buffer: 
debug-udev-update_devnode-tue.jctl-b.out.lz
    438:Jun 16 20:46:23 a192-168-100-2.deploy.acmetechnologies.com kernel:  
sdc: sdc1 sdc2
    980:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: The log level is changed to 'debug' while processing 
device (SEQNUM=2470, ACTION=add)
    982:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: /usr/lib/udev/rules.d/60-persistent-storage.rules:80 
Added SYMLINK 'disk/by-id/scsi-0QEMU_QEMU_HARDDISK_3-part1'
    983:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: 
/usr/lib/udev/rules.d/60-persistent-storage.rules:116 Added SYMLINK 
'disk/by-path/pci-0000:00:03.0-scsi-0:0:2:0-part1'
    984:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: 
/usr/lib/udev/rules.d/60-persistent-storage.rules:135 Importing properties from 
results of builtin command 'blkid'
    985:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: Probe /dev/sdc1 with raid and offset=0
   1001:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
systemd-fsck[1700]: /dev/sdc1: clean, 11/2883584 files, 259873/11533824 blocks
   1004:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: 
/usr/lib/udev/rules.d/60-persistent-storage.rules:150 Added SYMLINK 
'disk/by-partuuid/437f26c8-e793-4f9d-9d3a-94117b15a8f5'
   1005:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: 
/usr/lib/udev/rules.d/60-persistent-storage.rules:151 Added SYMLINK 
'disk/by-partlabel/primary'
   1006:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: 
/usr/lib/udev/rules.d/60-persistent-storage.rules:156 Added SYMLINK 
'disk/by-diskseq/27-part1'
   1007:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: /usr/lib/udev/rules.d/69-vdev.rules:6 Importing 
properties from results of '/lib/udev/vdev_id -d sdc1'
   1008:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: Starting '/lib/udev/vdev_id -d sdc1'
   1083:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: '/lib/udev/vdev_id -d sdc1'(out) 'Error: Config file 
"/etc/zfs/vdev_id.conf" not found'
   1085:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: Process '/lib/udev/vdev_id -d sdc1' failed with exit 
code 1.
   1086:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: /usr/lib/udev/rules.d/69-vdev.rules:6 Command 
"/lib/udev/vdev_id -d sdc1" returned 1 (error), ignoring
   1087:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: Setting permissions /dev/sdc1, uid=0, gid=6, 
mode=0660
   1088:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: Removing/updating old device symlink 
'/dev/disk/by-uuid/bde994ca-3285-47d8-9780-fdddc29b81eb', which is no longer 
belonging to this device.
   1089:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: Links that do belong to this device...
   1090:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: 
------------------------------------------------------------------------
   1091:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: 0: 
/dev/disk/by-partuuid/437f26c8-e793-4f9d-9d3a-94117b15a8f5
   1092:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: 1: /dev/disk/by-diskseq/27-part1
   1093:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: 2: 
/dev/disk/by-path/pci-0000:00:03.0-scsi-0:0:2:0-part1
   1094:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: 3: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_3-part1
   1095:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: 4: /dev/disk/by-partlabel/primary
   1096:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: 
------------------------------------------------------------------------
   1097:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: No reference left for 
'/dev/disk/by-uuid/bde994ca-3285-47d8-9780-fdddc29b81eb', removing
   1098:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: Successfully created symlink 
'/dev/disk/by-diskseq/27-part1' to '/dev/sdc1'
   1099:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: Successfully created symlink 
'/dev/disk/by-partlabel/primary' to '/dev/sdc1'
   1100:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: Successfully created symlink '/dev/block/8:33' to 
'/dev/sdc1'
   1101:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: sd-device: Created db file '/run/udev/data/b8:33' 
for 
'/devices/pci0000:00/0000:00:03.0/virtio0/host2/target2:0:2/2:0:2:0/block/sdc/sdc1'
   1102:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: Adding watch on '/dev/sdc1'
   1103:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: Device processed (SEQNUM=2470, ACTION=add)
   1104:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: sd-device-monitor(worker): Passed 1268 byte to 
netlink monitor.


I also see how fsck (or the program it forks) has changed the ext4 fs 
superblock and has a last written timestamp with the same second as the journal 
entry where the builtin blkid (libblkid) rule processing starts.

Thinking that the problem comes from udevd and util-linux fsck since
2014 no longer both using /dev/sdX as the  file for their uses of flock,
I tried adding code to udevd to take a shared lock on
/run/fsck/sdX.lock, the lockfile fsck now uses. When I used a blocking
call (with some narrowing by udev rule line number to avoid the bug
going away simply by perturbing the timing) the problem went away. When
I changed that to a non-blocking call, logged the errno, and allowed
udevd to proceed normally, I got the problem back with errno after flock
being 11 (EAGAIN or EWOULDBLOCK), which convinced me that fsck was
running and holding an exclusive lock when the problem happened. With
those patches reverted, I instead reverted fsck to its pre-2014 locking
behaviour, where it took its exclusive lock on /dev/sdX again. That also
prevented the problem from happening. "Preventing" here means running
for 7+ hours without seeing it, or roughly more than 3000 boots.

Note that I am running with lp2090972-libblkid-fix-spurious-ext-
superblock-checksum-mismat.patch from ubuntu bug report
https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/2090972 but
that does not avoid this problem.

Here's the patch I revert from Karol Zak:
https://github.com/util-linux/util-linux/commit/3bbdae633f4a1dda5f95ee6c61f18a1c8ef12250
But he doesn't view his lock as one to take the other side of udevd's read lock 
only to protect fsck from other instances of itself (see my util-linux bug 
report above). Still, I was thinking I'd suggest reverting that a local patch 
if my bosses like that idea. Yet, I'm worried about the systemd bug report 
leading up to Karol Zak's change:

https://bugs.freedesktop.org/show_bug.cgi?id=79576

Is there some other solution? Do you acknowledge this as a real bug?

Version info:

/scp:qvm:~ # *cat /etc/os-release 
PRETTY_NAME="Ubuntu 24.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.3 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/";
SUPPORT_URL="https://help.ubuntu.com/";
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/";
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy";
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

/scp:qvm:~ # dpkg -s systemd | grep ^Ver
(standard input):9:Version: 255.4-1ubuntu8.12acme24.0.2

/scp:qvm:~ # dpkg -s util-linux | grep ^Ver
(standard input):10:Version: 2.39.3-9ubuntu6.4acme24.0.1

Expected to happen: all auto mounts happen every boot
What happened: a mount, for me a non-root, non-boot one but a varying one, 
failed every 1000 or so boots.

** Affects: util-linux (Ubuntu)
     Importance: Undecided
         Status: New


** Tags: fsck locking systemd udevd util-linux

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2160558

Title:
  udevd and fsck contend causing the prior to miss fs uuids

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/2160558/+subscriptions


-- 
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to