Re: [libvirt] Matching the type of mediated devices in the migration
Hi: Thanks for unfolding your idea. The picture is clearer to me now. I didn't realize that you also want to support cross hardware migration. Well, I thought for a while, the cross hardware migration might be not popular in vGPU case but could be quite popular in other mdev cases. Let me continue my summary: Mdev dev type has already included a parent driver name/a group name/physical device version/configuration type. For example i915-GVTg_V5_4. The driver name and the group name could already distinguish the vendor and the product between different mdevs, e.g. between Intel and Nvidia, between vGPU or vOther. Each device provides a collection of the version of device state of data stream in a preferred order in a mdev type, as newer version of device state might contains more information which might help on performances. Let's say a new device N and an old device O, they both support mdev_type M. For example: Device N is newer and supports the versions of device state: [ 6.3 6.2 .6.1 ] in mdev type M Device O is older and supports the versions of device state: [ 5.3 5.2 5.1 ] in mdev type M - Version scheme of device state in backwards compatibility case: Migrate a VM from a VM with device O to a VM with device N, the mdev type is M. Device N: [ 6.3 6.2 6.1 5.3 ] in M Device O: [ 5.3 5.2 5.1 ] in M Version used in migration: 5.3 The new device directly supports mdev_type M with the preferred version on Device O. Good, best situation. Device N: [ 6.3 6.2 6.1 5.2 ] in M Device O: [ 5.3 5.2 5.1 ] in M Version used in migration: 5.2 The new device supports mdev_type M, but not the preferred version. After the migration, the vendor driver might have to disable some features which is not mentioned in 5.2 device state. But this totally depends on the vendor driver. If user wish to achieve the best experience, he should update the vendor driver in device N, which supports the preferred version on device O. Device N: [ 6.3 6.2 6.1 ] in M Device O: [ 5.3 5.2 5.1 ] in M Version used in migration: None No version is matched. Migration would fail. User should update the vendor driver on device N and device O. - Version scheme of device state in forwards compatibility case: Migrate a VM from a VM with N to a VM with device O, the mdev type is M. Device N: [ 6.3 6.2 .6.1 ] in M Device O: [ 5.3 5.2 5.1 ] in M, but the user updates the vendor driver on device O. Now device O could support [ 5.3 5.2 5.1 6.1 ] (As an old device, the Device O still prefers version 5.3) Version used in migration: 6.1 As the new device states is going to migrate to an old device, the vendor driver on old device might have to specially dealing with the new version of device state. It depends on the vendor driver. - QEMU has to figure out and choose the version of device states before reading device state from the region. (Perhaps we can put the option of selection in the control part of the region as well) - Libvirt will check if there is any match of the version in the collection in device O and device N before migration. - Each mdev_type has its own collection of versions. (Device can support different versions in different types) - Better the collection is not a range, better they could be a collection of the version strings. (The vendor driver might drop some versions during the upgrade since they are not ideal) That's the picture so far in my mind. Thanks, Zhi. -Original Message- From: Alex Williamson [mailto:alex.william...@redhat.com] Sent: Wednesday, August 1, 2018 8:19 PM To: Wang, Zhi A Cc: libvir-list@redhat.com; kwankh...@nvidia.com Subject: Re: Matching the type of mediated devices in the migration On Wed, 1 Aug 2018 10:22:39 +0000 "Wang, Zhi A" wrote: > Hi: > > Let me summarize the understanding so far I got from the discussions since I > am new to this discussion. > > The mdev_type would be a generic stuff since we don't want userspace > application to be confused. The example of mdev_type is: I don't think 'generic' is the right term here. An mdev_type is a specific thing with a defined interface, we just don't define what that interface is. > There are several pre-defined mdev_types with different configurations, let's > say MDEV_TYPE A/B/C. The HW 1.0 might only support MDEV_TYPE A, the HW 2.0 > might support both MDEV_TYPE A and B, but due to HW difference, we cannot > migrate MDEV_TYPE A with HW 1.0 to MDEV_TYPE A with HW 2.0 even they have the > same MDEV_TYPE. So we need a device version either in the existing MDEV_TYPE > or a new sysfs entry. This is correct, if a foo_type_a is exposed by the same vendor driver on different hardware, then the vendor driver is guaranteeing those mdev devices are software compatible to the user. Whether the vendor driver is willing or able to support migration across the underlying hardware is a separate question.
Re: [libvirt] Matching the type of mediated devices in the migration
Hi: Let me summarize the understanding so far I got from the discussions since I am new to this discussion. The mdev_type would be a generic stuff since we don't want userspace application to be confused. The example of mdev_type is: There are several pre-defined mdev_types with different configurations, let's say MDEV_TYPE A/B/C. The HW 1.0 might only support MDEV_TYPE A, the HW 2.0 might support both MDEV_TYPE A and B, but due to HW difference, we cannot migrate MDEV_TYPE A with HW 1.0 to MDEV_TYPE A with HW 2.0 even they have the same MDEV_TYPE. So we need a device version either in the existing MDEV_TYPE or a new sysfs entry. Libvirt would have to check MDEV_TYPE match between source machine and destination machine, then the device version. If any of them is different, then it fails the migration. If my above understanding is correct, for VFIO part, we could define the device version as string or a magic number. For example, the vendor mdev driver could pass the vendor/device id and a version to VFIO and VFIO could expose them in the UUID sysfs no matter through a new sysfs entry or through existing MDEV_TYPE. I prefer to expose it in the mdev_supported_types, since the libvirt node device list could extract the device version when it enumerating the host PCI devices or other devices, which supports mdev. We can also put it into UUID sysfs, but the user might have to first logon the target machine and then check the UUID and the device version by themselves, based on current code of libvirty. I suppose all the host device management would be in node device in libvirt, which provides remotely management of the host devices. For the format of a device version, an example would be: Vendor ID(16bit)Device ID(16bit)Class ID(16bit)Version(16bit) For string version of the device version, I guess we have to define the max string length, which is hard to say yet. Also, a magic number is easier to be put into the state data header during the migration. Thanks, Zhi. -Original Message- From: Alex Williamson [mailto:alex.william...@redhat.com] Sent: Tuesday, July 31, 2018 12:49 AM To: Wang, Zhi A Cc: libvir-list@redhat.com; kwankh...@nvidia.com Subject: Re: Matching the type of mediated devices in the migration On Tue, 31 Jul 2018 04:05:11 +0800 Zhi Wang wrote: > On 07/30/18 23:56, Alex Williamson wrote: > > On Sun, 29 Jul 2018 21:19:41 +0000 > > "Wang, Zhi A" wrote: > > > >> BACKGROUND > >> > >> As the live migration of mdev is going to be supported in VFIO, a scheme > >> of deciding if a mdev could be migratable between the source machine and > >> the destination machine is needed. Mostly, this email is going to discuss > >> a possible solution which needs fewer modifications of libvirt/VFIO. > >> > >> The configuration of a mdev is located in the domain XML, which guides > >> libvirt how to find the mdev and generating the command line for QEMU. It > >> basically only includes the UUID of a mdev. The domain XML of the source > >> machine and destination machine are going to be compared before the > >> migration really happens. Each configuration item would be compared and > >> checked by libvirt. If one item of the source machine is different from > >> the item of destination machine, the migration fails. For mdev, there is > >> no any check/match before the migration happens yet. > >> > >> The user could use the node device list of libvirt to list the host > >> devices and see the capabilities of those devices. The current node device > >> code of libvirt has already been able to extract the supported mdev types > >> from a host PCI device, plus some basic information, like max supported > >> mdev instance of a host PCI device. > >> > >> THE SOLUTION > >> > >> To strictly check the mdev type and make sure the migration happens > >> between the compatible mediated devices, three new mandatory elements in > >> the domain XML below the hostdev element would be introduced: > >> > >> vendorid: The vendor ID of the mdev, which comes from the host PCI device. > >> A user could obtain this information from the host PCI device which > >> supports mdev in the node device list. > >> productid: The product ID of the mdev, which also comes from the host PCI > >> device. A user could obtain this information from the same approach above. > >> > > > > The parent of an mdev device is not necessarily a PCI device. > Good point. I didn't get that. > > > >> mdevtype: The type of the mdev. As the creation of the mdev is managed by > >> the user, the user knows
[libvirt] Matching the type of mediated devices in the migration
BACKGROUND As the live migration of mdev is going to be supported in VFIO, a scheme of deciding if a mdev could be migratable between the source machine and the destination machine is needed. Mostly, this email is going to discuss a possible solution which needs fewer modifications of libvirt/VFIO. The configuration of a mdev is located in the domain XML, which guides libvirt how to find the mdev and generating the command line for QEMU. It basically only includes the UUID of a mdev. The domain XML of the source machine and destination machine are going to be compared before the migration really happens. Each configuration item would be compared and checked by libvirt. If one item of the source machine is different from the item of destination machine, the migration fails. For mdev, there is no any check/match before the migration happens yet. The user could use the node device list of libvirt to list the host devices and see the capabilities of those devices. The current node device code of libvirt has already been able to extract the supported mdev types from a host PCI device, plus some basic information, like max supported mdev instance of a host PCI device. THE SOLUTION To strictly check the mdev type and make sure the migration happens between the compatible mediated devices, three new mandatory elements in the domain XML below the hostdev element would be introduced: vendorid: The vendor ID of the mdev, which comes from the host PCI device. A user could obtain this information from the host PCI device which supports mdev in the node device list. productid: The product ID of the mdev, which also comes from the host PCI device. A user could obtain this information from the same approach above. mdevtype: The type of the mdev. As the creation of the mdev is managed by the user, the user knows the type of the mdev and would be responsible for filling out this information. These three elements are only needed when the device API of a mdev is "vfio-PCI". Take the example of mdev configuration from https://libvirt.org/formatdomain.html to illustrate the modification: 0xdead 0xbeef type With the newly introduced elements above, the flow of the creation of a domain XML with mdev will be like: 1. The user obtains the vendorid/productid from node device list 2. The user fills the vendorid/productid/mdevtype in the domain XML 3. When a migration happens, libvirt check these elements. If one item is different between two domain XML, then migration fails. POSSIBLE MODIFICATION OF LIBVIRT 1) Introduce three new elements in domain XML parsing and processing functions. 2) Extend the function virDomainDeviceInfoCheckABIStability() which is going to check the host dev part of the domain XMLs between the source machine and the destination machine. So it could fail the migration when it finds out the IDs and the mdev type are different between domain XMLs. PROS Minor changes in libvirt could achieve the mdev type match in the migration. Modifying VFIO and other mdev components is not necessary. Thanks, Zhi. -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
