Hi all, Back in March, Daniel Wagner posted here about improving NVMe-over-Fabrics (NVMe-oF) autoconnect by representing each fabric connection as a systemd unit:
https://lists.freedesktop.org/archives/systemd-devel/2026-March/052085.html That thread did not get a reply on-list, but the idea has developed since -- we now have a working proof of concept -- so we would like a sanity check from the systemd community, plus a question about one behavior that surprised us. What changed since the original post ------------------------------------ The first prototype had nvme-cli *generate unit files* on disk (under /run/systemd/system), write matching udev rules, then daemon-reload and start them. It worked, but that is a lot of moving parts (writing files, reloading udev, daemon-reload) for a set of connections that changes at runtime as nvme controllers come and go. We have since moved to a small daemon (nvme-discoverd, shipping with nvme-cli) that creates one *transient* unit per connection via org.freedesktop.systemd1.Manager.StartTransientUnit -- no files on disk, no daemon-reload. Each unit is essentially: Type=oneshot RemainAfterExit=yes ExecStart=/usr/sbin/nvme connect ... # writes /dev/nvme-fabrics, then exits ExecStop=/usr/sbin/nvme disconnect ... After=network.target # TCP/RDMA; omitted for Fibre Channel Before=nvme-discoverd.service TimeoutStopSec=... CollectMode=inactive-or-failed Why this shape: - The connect process writes to /dev/nvme-fabrics (a blocking, uninterruptible operation) and exits; the kernel then owns the connection. Type=oneshot + RemainAfterExit=yes keeps the unit "active (exited)" to model the live connection and to anchor the ExecStop= teardown. Doing the blocking write in a child unit keeps it off the daemon's event loop. - After=network.target (reversed at shutdown) tears the controller down before the network goes away, avoiding I/O errors on in-flight requests. Before=nvme-discoverd.service stops the daemon first so it does not observe the shutdown-time device removals and try to reconnect. - Restart (RestartUnit), garbage collection (CollectMode=inactive-or- failed), supervision and ordering come for free. The PoC connects discovery and I/O controllers, each unit's ExecStop= cleanly disconnects, and reconnect/adoption work. (The "disconnect-all should skip in-use connections" question from the original post we have addressed separately, with an ownership registry in nvme-cli, so this note is only about the unit mechanics.) What we would like feedback on ------------------------------ 1. Is modelling a long-lived, kernel-maintained resource as a oneshot + RemainAfterExit transient unit -- where the "process" exits immediately and the unit exists mainly to anchor ordering and the ExecStop= teardown -- an idiomatic use of systemd, or are we abusing it? Is there a primitive we should be using instead? 2. For shutdown ordering, is leaning on the After=network.target reversal (plus Before=<daemon>.service) the right way to sequence per-connection teardown before the network goes away? The original file-based prototype used Wants=/After=network-online.target, which is about boot readiness; our daemon handles connect retry itself, so we use network.target purely for teardown ordering. Are we thinking about this correctly? 3. A behavior that surprised us at first: systemd does not expand specifiers (%t, %N, %n) in the Exec* arguments of a transient unit created through the StartTransientUnit D-Bus API -- the arguments reach the program verbatim. For example a path passed as %t/nvme/discoverd/units/%N.devid arrives literally instead of becoming /run/nvme/discoverd/units/<unit-name>.devid This is by design (we confirmed it in the v261 source), but the documentation does not mention it. We worked around it by substituting the real runtime directory and unit name ourselves before the call. Two small asks: (a) would a one-line note help the next person who trips on this -- in the StartTransientUnit / ExecStart property description in org.freedesktop.systemd1(5), and/or in the Specifiers section of systemd.unit(5)? and (b) is substituting the values ourselves the intended approach, or is there a systemd-blessed pattern we are missing? Thanks for any guidance -- we would much rather hear "that is an abuse, do X instead" now than after this ships. Regards, Martin Belanger
