On Fri, Sep 18, 2026 at 12:48:22AM +0000, David Matlack wrote: > On 2026-09-16 06:50 PM, Bjorn Helgaas wrote: > > On Fri, Sep 11, 2026 at 04:44:02PM +0000, David Matlack wrote: > > > On 2026-09-10 06:48 PM, Bjorn Helgaas wrote: > > > > On Tue, Jul 28, 2026 at 10:09:55PM +0000, David Matlack wrote: > > > > > Set up a File-Lifecycle-Bound (FLB) handler for the PCI core to enable > > > > > it to participate in the preservation of PCI devices across Live > > > > > Update. > > > > > Essentially, this commit enables the PCI core to allocate a struct > > > > > (struct pci_ser) and preserve it across a Live Update whenever at > > > > > least > > > > > one device is preserved. > > > > > > > > I assume pci_ser is the state the PCI core needs to preserve across > > > > kexec so the new kernel's enumeration doesn't interrupt the device > > > > operation. And that whatever state the endpoint drivers need to > > > > adopt/inherit the device in the new kernel is managed without any help > > > > from the PCI core? > > > > > > Yes. > > > > > > > > Preserving PCI devices across Live Update is built on top of the Live > > > > > Update Orchestrator's (LUO) support for file preservation. Drivers are > > > > > expected to expose a file to userspace to represent a single PCI > > > > > device > > > > > and support preservation of that file. This is intended primarily to > > > > > support preservation of PCI devices bound to VFIO drivers. > > > > > > > > Where do drivers expose this file? sysfs? I guess it's a file per > > > > preserved device? Thinking like a driver writer, I'm expecting a hint > > > > about how to expose this file (should also be in the file doc somehere > > > > if it's not already). > > > > > > There is no requirement about how drivers do this from the PCI core > > > perspective. For all intents and purposes, the VFIO PCI variant drivers > > > are the only drivers that are going to be supported in the next 1-2 > > > years. They expose a misc character device for each file. > > > > If this file isn't relevant to the PCI core, maybe we don't need to > > mention it here. It doesn't seem like it motivates this patch. > > > > > > > This commit enables drivers to register their liveupdate_file_handler > > > > > with the PCI core so that the PCI core can do its own tracking and > > > > > enforcement of which devices are preserved. > > > > > > > > > > pci_liveupdate_register_flb(driver_file_handler); > > > > > pci_liveupdate_unregister_flb(driver_file_handler); > > > > > > > > So a driver calls pci_liveupdate_register_flb() once, then > > > > pci_liveupdate_preserve() once for each device it wants preserved? > > > > > > Yes > > > > > > > > When the first file (with a handler registered with the PCI core) is > > > > > preserved, the PCI core will be notified to allocate its tracking > > > > > struct > > > > > (pci_ser). > > > > > > > > The passive voice here makes the actors a bit obscure. I guess a > > > > LIVEUPDATE_SESSION_PRESERVE_FD ioctl on some per-device file kicks > > > > this off? > > > > > > Yes. (And I will reduce the passive voice in the next version.) > > > > > > > I guess the pci_ser allocation is in > > > > pci_liveupdate_flb_ops.preserve(), i.e., pci_flb_preserve()? > > > > > > Yes. > > > > > > > So the PCI core tracker (pci_ser) isn't actually allocated at the time > > > > of pci_liveupdate_register_flb(); it's allocated on the first > > > > LIVEUPDATE_SESSION_PRESERVE_FD ioctl for a driver that has called > > > > pci_liveupdate_register_flb()? > > > > > > Yes. The first device that gets preserved triggers the allocation of > > > struct pci_ser. And the last device that gets unpreserved (preservation > > > cancelled) triggers the freeing of struct pci_ser. > > > > > > > IIUC the call tree for that ioctl looks something like this: > > > > > > > > <driver> > > > > pci_liveupdate_register_flb > > > > liveupdate_register_flb(fh, &pci_liveupdate_flb) > > > > > > > > luo_session_ioctl > > > > op = &luo_session_ioctl_ops[...] > > > > op->execute # eg > > > > luo_session_preserve_fd() > > > > luo_session_preserve_fd > > > > luo_preserve_file > > > > luo_flb_file_preserve > > > > luo_flb_file_preserve_one > > > > if (outgoing_count == 0) # only for first FLB > > > > device > > > > flb->ops->preserve # eg pci_flb_preserve() > > > > pci_flb_preserve > > > > ser = kho_alloc_preserve <-- alloc pci_ser > > > > outgoing.count = 1 > > > > fh->ops->preserve # something not included > > > > here > > > > So IIUC this part of the path looks like this, which answers my > > question below about ordering of pci_ser and > > pci_liveupdate_preserve(): > > > > fh->ops->preserve # eg > > vfio_pci_liveupdate_preserve() > > vfio_pci_liveupdate_preserve > > pci_liveupdate_preserve > > pci_liveupdate_preserve_device > > dev_ser = pci_flb_alloc_dev_ser <-- alloc per-dev > > PCI core serialized state > > dev_ser->bdf = pci_dev_id(dev) > > > > > > Seems like kind of an awkward way to allocate pci_ser. Couldn't it be > > > > allocated on the first call to pci_liveupdate_register_flb()? That > > > > would be a lot easier for driver writers to trace through. > > > > > > I agree the LUO FLB API is a bit awkward, but this is how it works. > > > > > > If we allocated it during pci_liveupdate_register_flb() we would then > > > need to stash it in a global variable to hand-off the LUO later. Despite > > > the awkwardness of FLBs, it is useful to avoid globals and have LUO > > > management the lifetime. > > > > It seems like pci_ser is a singleton by design, so a global variable > > doesn't sound like it would be terrible to me. > > > > > > > When the last file is unpreserved (i.e. preservation > > > > > cancelled) the PCI core will be notified to free struct pci_ser. > > > > > > > > There's a lot going on behind "PCI core will be notified". I assume > > > > these refer to the first-time behavior of luo_flb_file_preserve_one() > > > > and last-time behavior of liveupdate_flb_put_outgoing(), which is > > > > honestly kind of hard to suss out. > > > > Could we say something specific and PCI-related here, to help connect > > the dots? Most of these paths are outside the PCI core. > > > > IIUC luo_session essentially has a refcount (outgoing.count) > > incremented by each LIVEUPDATE_SESSION_PRESERVE_FD ioctl, and the 0->1 > > transition in luo_flb_file_preserve_one() ends up calling > > pci_flb_preserve(), where pci_ser is allocated. > > > > And the refcount is decremented by luo_flb_file_unpreserve() (in a > > luo_session .release() function), where the 1->0 transition in > > liveupdate_flb_put_outgoing() calls pci_flb_unpreserve() where pci_ser > > is deallocated. > > > > That gets into a lot of detail, probably too much for a commit log. > > Maybe mentioning the function names by which the PCI core is notified > > to alloc/free pci_ser would be enough of a bread crumb. > > > > > > This series doesn't include a caller of pci_liveupdate_preserve() (or > > > > pci_liveupdate_register_flb()), so I can't figure out the ordering. > > > > Obviously pci_liveupdate_register_flb() must be first. > > > > > > In every version of this patch series I have sent I included a link to > > > the vfio-pci driver changes that build on top of this, rebased that > > > series on top of this one, uploaded it to my GitHub, and included a link > > > in the cover letter. Here is the relevant section from the v8 cover > > > letter: > > > > > > . This series was tested in conjunction with v5 of the VFIO PCI driver > > > . series: > > > . > > > . > > > https://lore.kernel.org/kvm/[email protected]/ > > > . > > > . The full set of patches used for testing can be found on GitHub. > > > . > > > . > > > https://github.com/dmatlack/linux/tree/liveupdate/pci/base/v8-with-vfio > > > > > > > > > > I first thought pci_liveupdate_preserve() would be called via the > > > > fh->ops->preserve() in the luo_session_preserve_fd() ioctl path, but > > > > it's not. pci_liveupdate_preserve() is intended for the driver to > > > > call it directly. But it looks like it has to be called *after* the > > > > ioctl? Obviously I'm confused :) > > > > > > It is called by the driver during it's fh->ops->preserve() callback. In > > > other words, it is called during the ioctl by the driver. > > > > I think the updated call tree above shows the connection? > > Yes the call tree you added above is correct. Here is an attempt at the > complete picture that I plan to include in the kernel-doc in v9: > > * Call Flow > * --------- > * > * :: > * > * # Driver initialization > * pci_liveupdate_register_flb(fh) > * > * # Userspace: ioctl(LIVEUPDATE_SESSION_PRESERVE_FD, devfd) > * luo_preserve_file() > * luo_flb_file_preserve() > * luo_flb_file_preserve_one() # first preserved file only > * pci_flb_preserve() # alloc and preserve struct pci_ser > * fh->ops->preserve() # driver callback > * pci_liveupdate_preserve(dev) # record this device in struct > pci_ser > * > * # Userspace: preservation cancelled or session torn down > * luo_file_unpreserve_files() > * luo_flb_file_unpreserve() > * liveupdate_flb_put_outgoing() # last unpreserved file only > * pci_flb_unpreserve() # free struct pci_ser > * > * # ---------------- kexec ---------------- > * > * # New kernel: PCI enumeration > * pci_setup_device() > * pci_liveupdate_setup_device() > * liveupdate_flb_get_incoming() > * luo_flb_retrieve_one() # first request only > * pci_flb_retrieve() # previous kernel's struct pci_ser > * > * # Userspace: ioctl(LIVEUPDATE_SESSION_FINISH) > * luo_file_finish_one() > * fh->ops->finish() # driver callback > * pci_liveupdate_finish(dev) # release this device's pci_dev_ser > * luo_flb_file_finish() > * liveupdate_flb_put_incoming() # last incoming file only > * pci_flb_finish() # free struct pci_ser > * > > And here is an updated commit message that I hope explains everything more > clearly: > > PCI: liveupdate: Set up FLB handler for the PCI core > > Set up a File-Lifecycle-Bound (FLB) handler so that the PCI core can > preserve its own state across a Live Update kexec. > > Preserving a PCI device across kexec requires preserving two independent > sets of state: > > - Driver state, e.g. everything vfio-pci needs so that userspace can > keep using the device in the new kernel. The driver preserves this > itself and the PCI core is not involved. > > - PCI core state, e.g. which devices are preserved, so that the new > kernel knows not to disturb them while they are still running and > doing DMA. That is what this commit adds, serialized into struct > pci_ser. > > Userspace, not the kernel, decides which devices are preserved, and it > does so through the Live Update Orchestrator's (LUO) support for file > preservation: a driver exposes a file that represents a single PCI > device, and userspace preserves that device with > ioctl(LIVEUPDATE_SESSION_PRESERVE_FD) on that file. Binding preservation > to a file gives it proper lifecycle management, e.g. the preservation is > undone if userspace cancels it or goes away. How a driver exposes that > file is up to the driver and invisible to the PCI core (vfio-pci variant > drivers, the first intended use-case, use their per-device cdev). > > LUO only knows that a file was preserved; it does not know that the
s/the/it/ > represents a PCI device, or which one. Bridging that gap, drivers > register their liveupdate_file_handler with the PCI core: > > pci_liveupdate_register_flb(driver_file_handler); > pci_liveupdate_unregister_flb(driver_file_handler); > > LUO then refcounts the PCI core's FLB against the files preserved by > that handler, and that refcount drives the lifetime of struct pci_ser: > > - On the first preserved file, luo_flb_file_preserve_one() calls > pci_flb_preserve(), which allocates struct pci_ser and preserves it > with KHO. > > - On the last unpreserved file (i.e. preservation cancelled), > liveupdate_flb_put_outgoing() calls pci_flb_unpreserve(), which > unpreserves and frees struct pci_ser. > > - In the next kernel, pci_flb_retrieve() hands the PCI core the struct > pci_ser built by the previous kernel, whenever the PCI core asks for > it (e.g. during enumeration), and pci_flb_finish() frees it once the > PCI core is done with it. > > So the flow for preserving a device, once a driver has registered, looks > like this: > > ioctl(LIVEUPDATE_SESSION_PRESERVE_FD) > luo_session_preserve_fd() > luo_preserve_file() > luo_flb_file_preserve() > luo_flb_file_preserve_one() # only on the first preserved file > pci_flb_preserve() # alloc + KHO-preserve pci_ser > fh->ops->preserve() # driver callback, e.g. vfio-pci > > Note that struct pci_ser is deliberately not allocated when a driver > calls pci_liveupdate_register_flb(). A driver can be loaded for the > lifetime of the machine without ever preserving a device, and there is > no reason to allocate memory and hand it to the next kernel in that > case. Letting LUO own the lifetime also means the PCI core does not have > to duplicate LUO's refcounting and unwind logic for preservation > failures, session aborts and fd close, and the incoming side > (retrieve/finish) comes from the same object rather than requiring a > separate KHO FDT entry owned by the PCI core. > > Note: This commit only allocates struct pci_ser and preserves it across > Live Update. A subsequent commit adds pci_liveupdate_preserve(), the API > drivers call from their fh->ops->preserve() callback to tell the PCI > core exactly which devices are being preserved. > > Note: There is no reason to check for kho_is_enabled() since it can be > assumed to return true. If KHO was not enabled then Live Update would > not be enabled and these routines would never run. Beautiful, thanks for your patience :)
