Re: [Qemu-devel] [PATCH v6 4/4] docs: Add Documentation for Mediated devices

[Daniel P. Berrange]

On Thu, Aug 04, 2016 at 12:33:54AM +0530, Kirti Wankhede wrote:
> diff --git a/Documentation/vfio-mediated-device.txt 
> b/Documentation/vfio-mediated-device.txt
> new file mode 100644
> index ..029152670141
> --- /dev/null
> +++ b/Documentation/vfio-mediated-device.txt
> @@ -0,0 +1,235 @@
> +Mediated device management interface via sysfs
> +---
> +This is the interface that allows user space software, like libvirt, to query
> +and configure mediated device in a HW agnostic fashion. This management
> +interface provide flexibility to underlying physical device's driver to 
> support
> +mediated device hotplug, multiple mediated devices per virtual machine, 
> multiple
> +mediated devices from different physical devices, etc.

A key point from the libvirt POV is that we want to be able to use the
sysfs interfaces without having to write vendor specific custom code for
each vendor's hardware.

> +Under per-physical device sysfs:
> +
> +
> +* mdev_supported_types: (read only)
> +List the current supported mediated device types and its details.

This really ought to describe the data format that is to be reported,
as from libvirt POV we don't want to see every vendor's driver reporting
arbitrarily different information here.

> +* mdev_create: (write only)
> + Create a mediated device on target physical device.
> + Input syntax: 
> + where,
> + UUID: mediated device's UUID
> + idx: mediated device index inside a VM
> + params: extra parameters required by driver

There's no specification about what 'params' is - it just looks like
an arbitrary vendor specific blob, which is not something that's
particularly pleasant to use. How would a userspace application
discover what parameters exist, and whether they are required to be
passed, vs optional, and standardization of those parameters across
different vendors's vGPU drivers so we don't have each vendor doing
something different.

> + Example:
> + # echo "12345678-1234-1234-1234-123456789abc:0:0" >
> +  /sys/bus/pci/devices/\:05\:00.0/mdev_create
> +
> +* mdev_destroy: (write only)
> + Destroy a mediated device on a target physical device.
> + Input syntax: 
> + where,
> + UUID: mediated device's UUID
> + idx: mediated device index inside a VM
> + Example:
> + # echo "12345678-1234-1234-1234-123456789abc:0" >
> +/sys/bus/pci/devices/\:05\:00.0/mdev_destroy

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH v6 4/4] docs: Add Documentation for Mediated devices

[Kirti Wankhede]

On 8/4/2016 1:01 PM, Tian, Kevin wrote:
>> From: Kirti Wankhede [mailto:kwankh...@nvidia.com]
>> Sent: Thursday, August 04, 2016 3:04 AM
>>
>> +
>> +* mdev_supported_types: (read only)
>> +List the current supported mediated device types and its details.
>> +
>> +* mdev_create: (write only)
>> +Create a mediated device on target physical device.
>> +Input syntax: 
>> +where,
>> +UUID: mediated device's UUID
>> +idx: mediated device index inside a VM
> 
> Is above description too specific to VM usage? mediated device can
> be used by other user components too, e.g. an user space driver.
> Better to make the description general (you can list above as one
> example).
>
Ok. I'll change it to VM or user space component.

> Also I think calling it idx a bit limited, which means only numbers
> possible. Is it more flexible to call it 'handle' and then any string
> can be used here?
> 

Index is integer, it is to keep track of mediated device instance number
created for a user space component or VM.

>> +params: extra parameters required by driver
>> +Example:
>> +# echo "12345678-1234-1234-1234-123456789abc:0:0" >
>> + /sys/bus/pci/devices/\:05\:00.0/mdev_create
>> +
>> +* mdev_destroy: (write only)
>> +Destroy a mediated device on a target physical device.
>> +Input syntax: 
>> +where,
>> +UUID: mediated device's UUID
>> +idx: mediated device index inside a VM
>> +Example:
>> +# echo "12345678-1234-1234-1234-123456789abc:0" >
>> +   /sys/bus/pci/devices/\:05\:00.0/mdev_destroy
>> +
>> +Under mdev class sysfs /sys/class/mdev/:
>> +
>> +
>> +* mdev_start: (write only)
>> +This trigger the registration interface to notify the driver to
>> +commit mediated device resource for target VM.
>> +The mdev_start function is a synchronized call, successful return of
>> +this call will indicate all the requested mdev resource has been fully
>> +committed, the VMM should continue.
>> +Input syntax: 
>> +Example:
>> +# echo "12345678-1234-1234-1234-123456789abc" >
>> +/sys/class/mdev/mdev_start
>> +
>> +* mdev_stop: (write only)
>> +This trigger the registration interface to notify the driver to
>> +release resources of mediated device of target VM.
>> +Input syntax: 
>> +Example:
>> +# echo "12345678-1234-1234-1234-123456789abc" >
>> + /sys/class/mdev/mdev_stop
> 
> I think it's clearer to create a node per mdev under /sys/class/mdev,
> and then move start/stop as attributes under each mdev node, e.g:
> 
> echo "0/1" > /sys/class/mdev/12345678-1234-1234-1234-123456789abc/start
> 

To support multiple mdev devices in one VM or user space driver, process
is to create or configure all mdev devices for that VM or user space
driver and then have a single 'start' which means all requested mdev
resources are committed.

> Doing this way is more extensible to add more capabilities under
> each mdev node, and different capability set may be implemented
> for them.
> 

You can add extra capabilities for each mdev device node using
'mdev_attr_groups' of 'struct parent_ops' from vendor driver.


>> +
>> +Mediated device Hotplug:
>> +---
>> +
>> +To support mediated device hotplug,  and  can be
>> +accessed during VM runtime, and the corresponding registration callback is
>> +invoked to allow driver to support hotplug.
> 
> 'hotplug' is an action on the mdev user (e.g. the VM), not on mdev itself.
> You can always create a mdev as long as physical device has enough
> available resource to support requested config. Destroying a mdev 
> may fail if there is still user on target mdev.
>

Here point is: user need to pass UUID to mdev_create and device will be
created even if VM or user space driver is running.

Thanks,
Kirti

> Thanks
> Kevin
> 

Re: [Qemu-devel] [PATCH v6 4/4] docs: Add Documentation for Mediated devices

[Tian, Kevin]

> From: Kirti Wankhede [mailto:kwankh...@nvidia.com]
> Sent: Thursday, August 04, 2016 3:04 AM
> 
> +
> +* mdev_supported_types: (read only)
> +List the current supported mediated device types and its details.
> +
> +* mdev_create: (write only)
> + Create a mediated device on target physical device.
> + Input syntax: 
> + where,
> + UUID: mediated device's UUID
> + idx: mediated device index inside a VM

Is above description too specific to VM usage? mediated device can
be used by other user components too, e.g. an user space driver.
Better to make the description general (you can list above as one
example).

Also I think calling it idx a bit limited, which means only numbers
possible. Is it more flexible to call it 'handle' and then any string
can be used here?

> + params: extra parameters required by driver
> + Example:
> + # echo "12345678-1234-1234-1234-123456789abc:0:0" >
> +  /sys/bus/pci/devices/\:05\:00.0/mdev_create
> +
> +* mdev_destroy: (write only)
> + Destroy a mediated device on a target physical device.
> + Input syntax: 
> + where,
> + UUID: mediated device's UUID
> + idx: mediated device index inside a VM
> + Example:
> + # echo "12345678-1234-1234-1234-123456789abc:0" >
> +/sys/bus/pci/devices/\:05\:00.0/mdev_destroy
> +
> +Under mdev class sysfs /sys/class/mdev/:
> +
> +
> +* mdev_start: (write only)
> + This trigger the registration interface to notify the driver to
> + commit mediated device resource for target VM.
> + The mdev_start function is a synchronized call, successful return of
> + this call will indicate all the requested mdev resource has been fully
> + committed, the VMM should continue.
> + Input syntax: 
> + Example:
> + # echo "12345678-1234-1234-1234-123456789abc" >
> + /sys/class/mdev/mdev_start
> +
> +* mdev_stop: (write only)
> + This trigger the registration interface to notify the driver to
> + release resources of mediated device of target VM.
> + Input syntax: 
> + Example:
> + # echo "12345678-1234-1234-1234-123456789abc" >
> +  /sys/class/mdev/mdev_stop

I think it's clearer to create a node per mdev under /sys/class/mdev,
and then move start/stop as attributes under each mdev node, e.g:

echo "0/1" > /sys/class/mdev/12345678-1234-1234-1234-123456789abc/start

Doing this way is more extensible to add more capabilities under
each mdev node, and different capability set may be implemented
for them.

> +
> +Mediated device Hotplug:
> +---
> +
> +To support mediated device hotplug,  and  can be
> +accessed during VM runtime, and the corresponding registration callback is
> +invoked to allow driver to support hotplug.

'hotplug' is an action on the mdev user (e.g. the VM), not on mdev itself.
You can always create a mdev as long as physical device has enough
available resource to support requested config. Destroying a mdev 
may fail if there is still user on target mdev.

Thanks
Kevin

[Qemu-devel] [PATCH v6 4/4] docs: Add Documentation for Mediated devices

[Kirti Wankhede]

Add file Documentation/vfio-mediated-device.txt that include details of
mediated device framework.

Signed-off-by: Kirti Wankhede 
Signed-off-by: Neo Jia 
Change-Id: I137dd646442936090d92008b115908b7b2c7bc5d
---
 Documentation/vfio-mediated-device.txt | 235 +
 1 file changed, 235 insertions(+)
 create mode 100644 Documentation/vfio-mediated-device.txt

diff --git a/Documentation/vfio-mediated-device.txt 
b/Documentation/vfio-mediated-device.txt
new file mode 100644
index ..029152670141
--- /dev/null
+++ b/Documentation/vfio-mediated-device.txt
@@ -0,0 +1,235 @@
+VFIO Mediated devices [1]
+---
+
+There are more and more use cases/demands to virtualize the DMA devices which
+doesn't have SR_IOV capability built-in. To do this, drivers of different
+devices had to develop their own management interface and set of APIs and then
+integrate it to user space software. We've identified common requirements and
+unified management interface for such devices to make user space software
+integration easier.
+
+The VFIO driver framework provides unified APIs for direct device access. It is
+an IOMMU/device agnostic framework for exposing direct device access to
+user space, in a secure, IOMMU protected environment. This framework is
+used for multiple devices like GPUs, network adapters and compute accelerators.
+With direct device access, virtual machines or user space applications have
+direct access of physical device. This framework is reused for mediated 
devices.
+
+Mediated core driver provides a common interface for mediated device management
+that can be used by drivers of different devices. This module provides a 
generic
+interface to create/destroy mediated device, add/remove it to mediated bus
+driver, add/remove device to IOMMU group. It also provides an interface to
+register different types of bus drivers, for example, Mediated VFIO PCI driver
+is designed for mediated PCI devices and supports VFIO APIs. Similarly, driver
+can be designed to support any type of mediated device and added to this
+framework. Mediated bus driver add/delete mediated device to VFIO Group.
+
+Below is the high Level block diagram, with NVIDIA, Intel and IBM devices
+as example, since these are the devices which are going to actively use
+this module as of now. NVIDIA and Intel uses vfio_mpci.ko module for their GPUs
+which are PCI devices. There has to be different bus driver for Channel I/O
+devices, vfio_mccw.ko.
+
+
+ +---+
+ |   |
+ | +---+ |  mdev_register_driver() +--+
+ | |   | +<+  |
+ | |   | | |  |
+ | |  mdev | +>+ vfio_mpci.ko |<-> VFIO user
+ | |  bus  | | probe()/remove()|  |APIs
+ | |  driver   | | |  |
+ | |   | | +--+
+ | |   | |  mdev_register_driver() +--+
+ | |   | +<+  |
+ | |   | | |  |
+ | |   | +>+ vfio_mccw.ko |<-> VFIO user
+ | +---+ | probe()/remove()|  |APIs
+ |   | |  |
+ |  MDEV CORE| +--+
+ |   MODULE  |
+ |   mdev.ko |
+ | +---+ |  mdev_register_device() +--+
+ | |   | +<+  |
+ | |   | | |  nvidia.ko   |<-> physical
+ | |   | +>+  |device
+ | |   | |callbacks+--+
+ | | Physical  | |
+ | |  device   | |  mdev_register_device() +--+
+ | | interface | |<+  |
+ | |   | | |  i915.ko |<-> physical
+ | |   | +>+  |device
+ | |   | |callbacks+--+
+ | |   | |
+ | |   | |  mdev_register_device() +--+
+ | |   | +<+  |
+ | |   | | | ccw_device.ko|<-> physical
+ | |   | +>+  |device
+ | |   | |callbacks+--+
+ | +---+ |
+ +---+
+
+
+Registration Interfaces
+---
+
+Mediated core driver provides two types of registration interfaces:
+
+1. Registration interface