On Thu, 11 Mar 2021 00:50:09 +0530 Tarun Gupta <targu...@nvidia.com> wrote:
> Document interfaces used for VFIO device migration. Added flow of state > changes > during live migration with VFIO device. Tested by building docs with the new > vfio-migration.rst file. > > v2: > - Included the new vfio-migration.rst file in index.rst > - Updated dirty page tracking section, also added details about > 'pre-copy-dirty-page-tracking' opt-out option. > - Incorporated comments around wording of doc. > > Signed-off-by: Tarun Gupta <targu...@nvidia.com> > Signed-off-by: Kirti Wankhede <kwankh...@nvidia.com> > --- > MAINTAINERS | 1 + > docs/devel/index.rst | 1 + > docs/devel/vfio-migration.rst | 135 ++++++++++++++++++++++++++++++++++ > 3 files changed, 137 insertions(+) > create mode 100644 docs/devel/vfio-migration.rst > (...) > diff --git a/docs/devel/vfio-migration.rst b/docs/devel/vfio-migration.rst > new file mode 100644 > index 0000000000..6196fb132c > --- /dev/null > +++ b/docs/devel/vfio-migration.rst > @@ -0,0 +1,135 @@ > +===================== > +VFIO device Migration > +===================== Maybe add an introductory sentence or two describing the general approach? I.e. we have a general framework, and specific support for devices needs to be hooked up. > + > +VFIO devices use an iterative approach for migration because certain VFIO > +devices (e.g. GPU) have large amount of data to be transfered. The iterative > +pre-copy phase of migration allows for the guest to continue whilst the VFIO > +device state is transferred to the destination, this helps to reduce the > total > +downtime of the VM. VFIO devices can choose to skip the pre-copy phase of > +migration by returning pending_bytes as zero during the pre-copy phase. What about something like: "Migration of VFIO devices consists of two phases: the optional pre-copy phase, and the stop-and-copy phase. The pre-copy phase is iterative and allows to accommodate VFIO devices that have a large amount of data that needs to be transferred. The iterative pre-copy phase..." > + > +A detailed description of the UAPI for VFIO device migration can be found in > +the comment for the ``vfio_device_migration_info`` structure in the header > +file linux-headers/linux/vfio.h. > + > +VFIO device hooks for iterative approach: > + > +* A ``save_setup`` function that sets up the migration region, sets _SAVING > + flag in the VFIO device state and informs the VFIO IOMMU module to start > + dirty page tracking. > + > +* A ``load_setup`` function that sets up the migration region on the > + destination and sets _RESUMING flag in the VFIO device state. > + > +* A ``save_live_pending`` function that reads pending_bytes from the vendor > + driver, which indicates the amount of data that the vendor driver has yet > to > + save for the VFIO device. > + > +* A ``save_live_iterate`` function that reads the VFIO device's data from the > + vendor driver through the migration region during iterative phase. > + > +* A ``save_live_complete_precopy`` function that resets _RUNNING flag from > the > + VFIO device state, saves the device config space, if any, and iteratively > + copies the remaining data for the VFIO device untill the vendor driver s/untill/until/ > + indicates that no data remains (pending bytes is zero). > + > +* A ``load_state`` function that loads the config section and the data > + sections that are generated by the save functions above > + > +* ``cleanup`` functions for both save and load that perform any migration > + related cleanup, including unmapping the migration region > + > +A VM state change handler is registered to change the VFIO device state when > +the VM state changes. > + > +Similarly, a migration state change notifier is registered to get a > +notification on migration state change. These states are translated to VFIO s/to/to the corresponding/ > +device state and conveyed to vendor driver. s/to/to the/ (...) > +Postcopy > +======== > + > +Postcopy migration is not supported for VFIO devices. s/is not/is currently not/ ?
