On Fri, 2026-08-28 at 11:25 +0200, Antonio Alvarez Feijoo wrote: > The transient unit cancel-multipath-wait-$kernel creates a timer and > service with "Conflicts=initrd-cleanup.service": > > - sda: /usr/lib/udev/rules.d/56-multipath.rules ... > --on-active=1.000000 ... > - sda: ... 'Running timer as unit: cancel-multipath-wait-sda.timer' > - sda: ... 'Will run service as unit: > cancel-multipath-wait-sda.service' > > This introduces a race not always reproducible: there is a 1 second > window where this transient unit can start after > initrd-cleanup.service/start is enqueued but is still held back. When > that happens, starting cancel-multipath-wait-$kernel.service pulls in > a stop job for initrd-cleanup.service and the queued start job is > cancelled, the initrd is left with an empty job queue and hangs > forever, before switch-root: > > - initrd-parse-etc.service: Triggering OnSuccess= dependencies. > - initrd-cleanup.service: Installed new job > initrd-cleanup.service/start as 115 > - initrd-cleanup.service: starting held back, waiting for: > dracut-pre-pivot.service > - initrd-cleanup.service: Job 115 initrd-cleanup.service/start > finished, result=canceled > - initrd-cleanup.service: Installed new job > initrd-cleanup.service/stop as 133 > - initrd-cleanup.service: Job 133 initrd-cleanup.service/stop > finished, result=done > > "Conflicts=" is symmetric: if a unit has a "Conflicts=" requirement > on a set of other units, then starting it will stop all of them and > starting any of them will stop it. This is safe against > shutdown.target, whose jobs are enqueued irreversibly: a unit > starting during shutdown fails to start rather than cancelling the > shutdown. It is not safe against initrd-cleanup.service, whose start > job is an ordinary job enqueued by initrd-parse-etc.service via > "OnSuccess=". > > Ordering "Before=initrd-cleanup.service" is enough and is not > destructive: a trigger already in flight completes before cleanup > runs, and a unit with no job does not delay it. Tearing the > transient units down before switch-root is already handled by the > isolate to initrd-switch-root.target, which stops everything that > is not "IgnoreOnIsolate=yes". > > Signed-off-by: Antonio Alvarez Feijoo <[email protected]>
Reviewed-by: Martin Wilck <[email protected]> @Ben and others, I wasn't certain about this at first. Below is my flow of thoughts from the SUSE bugzilla, through which I came to think that it's correct. But this is quite subtle, so please give it some own thoughts ;-) Martin The general idea of the "cancel" timer is this: 1. uevent for a block device arrives, the WWID is yet unknown to multipath 2. multipathd sees the event and temporarily sets SYSTEMD_READY=0, avoiding other subsystems like MD or LVM grabbing it, and waiting for potential other paths to appear 3. udev starts the transient timer unit 3a. some other path to the same volume appears, and the timer is cancelled (we are not discussing this case here) 4. when the timer expires, it starts "udevadm trigger" 5. the kernel generates a new uevent for the same block device 6. while handling the uevent, "multipath -u" is run. 7. realizes that the wait for other paths has timed out, and remove SYSTEMD_READY, opening it for other layers to grab. There's a race window between 4 and 6 which systemd cannot handle. systemd does not "know" that a uevent has been triggered, so even if 3) or 4) happen before the start of initrd-cleanup.service, 5, 6, 7 may happen afterwards. In earlier versions of multipathd and dracut "multipath -u" would attempt to start multipathd.service via socket activation (multipathd.socket), which would then stop initrd-cleanup.service, as it Conflicts: it. Since 0.9.8, multipathd.socket is disabled by default, and we haven't included it in the initrd any more for a while, too [*]. It's important to understand that the "cancel" timer+service are just a safeguard for the case that no new uevent for the device in question occurs. If we're in the initrd, we can be sure that every uevent is going to be replayed in the udev coldplug phase. We still need to start the transient timer in the initrd for handling the case that the device in question is needed for the root file system, and is not multipathed. But when initrd-cleanup.service is started, the root device has been discovered already, and we don't need to bother about the "cancel" timer and service. It doesn't matter if they run or not. In the worst case, Antonio's patch will cause these units to be started after initrd-cleanup.service is started, which may cause an extra "add" uevent for the block device in question. But that shouldn't be a problem because such an event is going to be replayed after switching root, anyway. IIUC, if the "Conflicts" is removed, the timer unit will continue running through the switch root phase, and will be taken up by systemd through serialization [*] upstream dracut version history related to multipathd.socket: 247d2b ("fix(multipathd.service): adapt to upstream multipath-tools unit file") (dracut 056) added socket dep to the multipathd.service unit file. 02e646f ("fix(multipath): install multipathd.socket") (dracut 058) made sure the socket unit file was installed. 297525c ("fix(multipath): remove dependency on multipathd.socket") (dracut 060) removed both 5e87b68 ("refactor(multipath): remove custom multipathd.service") (dracut 107) removed the .service file. > --- > multipath/multipath.rules.in | 14 ++++++++++++-- > 1 file changed, 12 insertions(+), 2 deletions(-) > > diff --git a/multipath/multipath.rules.in > b/multipath/multipath.rules.in > index 2ac1972f..7b605dd4 100644 > --- a/multipath/multipath.rules.in > +++ b/multipath/multipath.rules.in > @@ -68,8 +68,18 @@ ENV{.SAVED_FM_WAIT_UNTIL}=="?*", > GOTO="pretend_mpath" > # the --on-active timeout. > # > # We must trigger an "add" event because LVM2 will only act on > those. > - > -RUN+="@SYSDIR_BIN@/systemd-run --unit=cancel-multipath-wait-$kernel > --description 'cancel waiting for multipath siblings of $kernel' -- > no-block --timer-property DefaultDependencies=no --timer-property > Conflicts=shutdown.target --timer-property Before=shutdown.target -- > timer-property Conflicts=initrd-cleanup.service --timer-property > Before=initrd-cleanup.service --timer-property AccuracySec=500ms -- > property DefaultDependencies=no --property Conflicts=shutdown.target > --property Before=shutdown.target --property Conflicts=initrd- > cleanup.service --property Before=initrd-cleanup.service --on- > active=$env{FIND_MULTIPATHS_WAIT_UNTIL} @SYSDIR_BIN@/udevadm trigger > --action=add $sys$devpath" > +# > +# These units are ordered Before=initrd-cleanup.service so that a > trigger that > +# is already in flight completes before the initrd switches root. > They must > +# *not* use Conflicts=initrd-cleanup.service: unlike > shutdown.target, whose > +# jobs are enqueued irreversibly, initrd-cleanup.service/start is a > plain job > +# enqueued by initrd-parse-etc.service via OnSuccess=. Starting a > unit that > +# conflicts with it therefore cancels that job instead of failing to > start, > +# and OnSuccess= is never retried, so the initrd would hang forever > with an > +# empty job queue. Stopping these units before switch-root is > already > +# guaranteed by the isolate to initrd-switch-root.target. > + > +RUN+="@SYSDIR_BIN@/systemd-run --unit=cancel-multipath-wait-$kernel > --description 'cancel waiting for multipath siblings of $kernel' -- > no-block --timer-property DefaultDependencies=no --timer-property > Conflicts=shutdown.target --timer-property Before=shutdown.target -- > timer-property Before=initrd-cleanup.service --timer-property > AccuracySec=500ms --property DefaultDependencies=no --property > Conflicts=shutdown.target --property Before=shutdown.target -- > property Before=initrd-cleanup.service --on- > active=$env{FIND_MULTIPATHS_WAIT_UNTIL} @SYSDIR_BIN@/udevadm trigger > --action=add $sys$devpath" > > LABEL="pretend_mpath" > ENV{DM_MULTIPATH_DEVICE_PATH}="1" -- Dr. Martin Wilck <[email protected]> SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg, Germany Geschäftsführer: Jochen Jaser, Andrew McDonald (HRB 36809,AG Nürnberg)
