Add documentation for preserving VFIO device files across a Live Update, as well as some generic file preservation documentation. This documentation will be extended in the future as new types of files are supported and new dependency/ordering requirements are added.
Signed-off-by: David Matlack <[email protected]> --- Documentation/userspace-api/liveupdate.rst | 144 +++++++++++++++++++++ 1 file changed, 144 insertions(+) diff --git a/Documentation/userspace-api/liveupdate.rst b/Documentation/userspace-api/liveupdate.rst index 41c0473e4f16..dbf1e4aeddd7 100644 --- a/Documentation/userspace-api/liveupdate.rst +++ b/Documentation/userspace-api/liveupdate.rst @@ -14,6 +14,150 @@ ioctl uAPI =========== .. kernel-doc:: include/uapi/linux/liveupdate.h +File Preservation +================= + +Files can be preserved across Live Update in sessions. Since only one process +can open /dev/liveupdate, sessions must be created by a centralized process +(e.g. "luod") and then passed via UDS to lower privilege processes (e.g. VMMs) +for them to preserve their own files. + +luod:: + + luo_fd = open("/dev/liveupdate", ...); + + ... + + // Create a new session with the given name. + struct liveupdate_ioctl_create_session arg = { + .size = sizeof(arg), + .name = SESSION_NAME, + }; + ioctl(luo_fd, LIVEUPDATE_IOCTL_CREATE_SESSION, &arg); + + // Send session_fd to the VMM over UDS. + send_session_fd(..., arg.fd); + +VMM:: + + // Receive the newly created session from luod over UDS + session_fd = create_session(SESSION_NAME); + + ... + + // Preserve a file with a unique token value in the session. + struct liveupdate_session_preserve_fd arg = { + .size = sizeof(arg), + .fd = fd, + .token = TOKEN, + } + ioctl(session_fd, LIVEUPDATE_SESSION_PRESERVE_FD, &arg); + +Files can be unpreserved with the LIVEUPDATE_SESSION_UNPRESERVE_FD ioctl. They +are also unpreserved once the last reference to the session is dropped. To +carry preserved files across a Live Update, references must be kept on the +session files through the reboot(LINUX_REBOOT_CMD_KEXEC) syscall. + +While a file is preserved in a session, the kernel holds an extra reference +to it to prevent it from being destroyed. + +Only the following types of files support LIVEUPDATE_SESSION_PRESERVE_FD. More +types of files are expected to be added in the future. + + - memfd + - VFIO character device files (vfio-pci only) + +File Retrieval +============== + +Files that are preserved in a session retrieved after +reboot(LINUX_REBOOT_CMD_KEXEC). + +luod:: + + luo_fd = open("/dev/liveupdate", ...); + + ... + + struct liveupdate_ioctl_retrieve_session arg = { + .size = sizeof(arg), + .name = SESSION_NAME, + }; + ioctl(luo_fd, LIVEUPDATE_IOCTL_RETRIEVE_SESSION, &arg); + + // Send session_fd to VMM over UDS. + send_session_fd(..., arg.fd); + +VMM:: + + // Receive the retrieved session from luod over UDS + session_fd = retrieve_session(SESSION_NAME); + + ... + + // Retrieve the file associated with the token from the session. + struct liveupdate_session_retrieve_fd arg = { + .size = sizeof(arg), + .token = TOKEN, + }; + ioctl(session_fd, LIVEUPDATE_SESSION_RETRIEVE_FD, &arg); + + ... + + ioctl(session_fd, LIVEUPDATE_SESSION_FINISH, ...); + +A session can only be finished once all of the files within it have been +retrieved, and are fully restored from the kernel's perspective. The exact +requirements will vary by file type. + +VFIO Character Device (cdev) Files +================================== + +The kernel supports preserving VFIO character device files across Live Update +within a session:: + + device_fd = open("/dev/vfio/devices/X"); + + ... + + ioctl(session_fd, LIVEUPDATE_SESSION_PRESERVE_FD, { ..., device_fd, ...}); + +Attempting to preserve files acquired via VFIO_GROUP_GET_DEVICE_FD will fail. + +Since the kernel holds an extra reference to files preserved in sessions, there +is no way for the underlying PCI device to be unbound from vfio-pci while it +is being preserved. + +When a VFIO device file is preserved in a session, interrupts must be disabled +on the device prior to reboot(LINUX_REBOOT_CMD_KEXEC), or the kexec will fail. + +Preserved VFIO device files can be retrieved after a Live Update just like any +other preserved file:: + + ioctl(session_fd, LIVEUPDATE_SESSION_RETRIEVE_FD, &arg); + device_fd = arg.fd; + + ... + + ioctl(session_fd, LIVEUPDATE_SESSION_FINISH, ...); + +Prior to LIVEUPDATE_SESSION_FINISH, preserved devices must be retrieved from +the session and bound to an iommufd. Attempting to open the device through +its character device (/dev/vfio/devices/X) or VFIO_GROUP_GET_DEVICE_FD will +fail with -EBUSY. + +The eventual goal of these support is to preserve devices running uninterrupted +across a Live Update. However there are many steps still needed to achieve this +(see Future Work below). So for now, VFIO will reset and restore the device +back into an idle state during reboot(LINUX_REBOOT_CMD_KEXEC). + +Future work: + + - Preservation of iommufd files + - Preservation of IOMMU driver state + - Preservation of PCI state (BAR resources, device state, bridge state, ...) + - Preservation of vfio-pci driver state + See Also ======== -- 2.53.0.rc1.225.gd81095ad13-goog
