[ovirt-devel] Re: device compatibility interface for live migration with assigned devices
On Thu, 2020-08-20 at 12:01 +0800, Yan Zhao wrote: > On Thu, Aug 20, 2020 at 02:29:07AM +0100, Sean Mooney wrote: > > On Thu, 2020-08-20 at 08:39 +0800, Yan Zhao wrote: > > > On Tue, Aug 18, 2020 at 11:36:52AM +0200, Cornelia Huck wrote: > > > > On Tue, 18 Aug 2020 10:16:28 +0100 > > > > Daniel P. Berrangé wrote: > > > > > > > > > On Tue, Aug 18, 2020 at 05:01:51PM +0800, Jason Wang wrote: > > > > > >On 2020/8/18 下午4:55, Daniel P. Berrangé wrote: > > > > > > > > > > > > On Tue, Aug 18, 2020 at 11:24:30AM +0800, Jason Wang wrote: > > > > > > > > > > > > On 2020/8/14 下午1:16, Yan Zhao wrote: > > > > > > > > > > > > On Thu, Aug 13, 2020 at 12:24:50PM +0800, Jason Wang wrote: > > > > > > > > > > > > On 2020/8/10 下午3:46, Yan Zhao wrote: > > > > > > we actually can also retrieve the same information through sysfs, > > > > > > .e.g > > > > > > > > > > > > |- [path to device] > > > > > > |--- migration > > > > > > | |--- self > > > > > > | | |---device_api > > > > > > || |---mdev_type > > > > > > || |---software_version > > > > > > || |---device_id > > > > > > || |---aggregator > > > > > > | |--- compatible > > > > > > | | |---device_api > > > > > > || |---mdev_type > > > > > > || |---software_version > > > > > > || |---device_id > > > > > > || |---aggregator > > > > > > > > > > > > > > > > > > Yes but: > > > > > > > > > > > > - You need one file per attribute (one syscall for one attribute) > > > > > > - Attribute is coupled with kobject > > > > > > > > Is that really that bad? You have the device with an embedded kobject > > > > anyway, and you can just put things into an attribute group? > > > > > > > > [Also, I think that self/compatible split in the example makes things > > > > needlessly complex. Shouldn't semantic versioning and matching already > > > > cover nearly everything? I would expect very few cases that are more > > > > complex than that. Maybe the aggregation stuff, but I don't think we > > > > need that self/compatible split for that, either.] > > > > > > Hi Cornelia, > > > > > > The reason I want to declare compatible list of attributes is that > > > sometimes it's not a simple 1:1 matching of source attributes and target > > > attributes > > > as I demonstrated below, > > > source mdev of (mdev_type i915-GVTg_V5_2 + aggregator 1) is compatible to > > > target mdev of (mdev_type i915-GVTg_V5_4 + aggregator 2), > > >(mdev_type i915-GVTg_V5_8 + aggregator 4) > > > > the way you are doing the nameing is till really confusing by the way > > if this has not already been merged in the kernel can you chagne the mdev > > so that mdev_type i915-GVTg_V5_2 is 2 of mdev_type i915-GVTg_V5_1 instead > > of half the device > > > > currently you need to deived the aggratod by the number at the end of the > > mdev type to figure out > > how much of the phsicial device is being used with is a very unfridly api > > convention > > > > the way aggrator are being proposed in general is not really someting i > > like but i thin this at least > > is something that should be able to correct. > > > > with the complexity in the mdev type name + aggrator i suspect that this > > will never be support > > in openstack nova directly requireing integration via cyborg unless we can > > pre partion the > > device in to mdevs staicaly and just ignore this. > > > > this is way to vendor sepecif to integrate into something like openstack in > > nova unless we can guarentee > > taht how aggreator work will be portable across vendors genericly. > > > > > > > > and aggragator may be just one of such examples that 1:1 matching does not > > > fit. > > > > for openstack nova i dont see us support anything beyond the 1:1 case where > > the mdev type does not change. > > > > hi Sean, > I understand it's hard for openstack. but 1:N is always meaningful. > e.g. > if source device 1 has cap A, it is compatible to > device 2: cap A, > device 3: cap A+B, > device 4: cap A+B+C > > to allow openstack to detect it correctly, in compatible list of > device 2, we would say compatible cap is A; > device 3, compatible cap is A or A+B; > device 4, compatible cap is A or A+B, or A+B+C; > > then if openstack finds device A's self cap A is contained in compatible > cap of device 2/3/4, it can migrate device 1 to device 2,3,4. > > conversely, device 1's compatible cap is only A, > so it is able to migrate device 2 to device 1, and it is not able to > migrate device 3/4 to device 1. yes we build the palcement servce aroudn the idea of capablites as traits on resocue providres. which is why i originally asked if we coudl model compatibality with feature flags we can seaislyt model deivce as aupport A, A+B or A+B+C and then select hosts and evice based on that but the list of compatable deivce you are propsoeing hide this feature infomation which whould be what
[ovirt-devel] Re: device compatibility interface for live migration with assigned devices
On Thu, Aug 20, 2020 at 02:29:07AM +0100, Sean Mooney wrote: > On Thu, 2020-08-20 at 08:39 +0800, Yan Zhao wrote: > > On Tue, Aug 18, 2020 at 11:36:52AM +0200, Cornelia Huck wrote: > > > On Tue, 18 Aug 2020 10:16:28 +0100 > > > Daniel P. Berrangé wrote: > > > > > > > On Tue, Aug 18, 2020 at 05:01:51PM +0800, Jason Wang wrote: > > > > >On 2020/8/18 下午4:55, Daniel P. Berrangé wrote: > > > > > > > > > > On Tue, Aug 18, 2020 at 11:24:30AM +0800, Jason Wang wrote: > > > > > > > > > > On 2020/8/14 下午1:16, Yan Zhao wrote: > > > > > > > > > > On Thu, Aug 13, 2020 at 12:24:50PM +0800, Jason Wang wrote: > > > > > > > > > > On 2020/8/10 下午3:46, Yan Zhao wrote: > > > > > we actually can also retrieve the same information through sysfs, > > > > > .e.g > > > > > > > > > > |- [path to device] > > > > > |--- migration > > > > > | |--- self > > > > > | | |---device_api > > > > > || |---mdev_type > > > > > || |---software_version > > > > > || |---device_id > > > > > || |---aggregator > > > > > | |--- compatible > > > > > | | |---device_api > > > > > || |---mdev_type > > > > > || |---software_version > > > > > || |---device_id > > > > > || |---aggregator > > > > > > > > > > > > > > > Yes but: > > > > > > > > > > - You need one file per attribute (one syscall for one attribute) > > > > > - Attribute is coupled with kobject > > > > > > Is that really that bad? You have the device with an embedded kobject > > > anyway, and you can just put things into an attribute group? > > > > > > [Also, I think that self/compatible split in the example makes things > > > needlessly complex. Shouldn't semantic versioning and matching already > > > cover nearly everything? I would expect very few cases that are more > > > complex than that. Maybe the aggregation stuff, but I don't think we > > > need that self/compatible split for that, either.] > > > > Hi Cornelia, > > > > The reason I want to declare compatible list of attributes is that > > sometimes it's not a simple 1:1 matching of source attributes and target > > attributes > > as I demonstrated below, > > source mdev of (mdev_type i915-GVTg_V5_2 + aggregator 1) is compatible to > > target mdev of (mdev_type i915-GVTg_V5_4 + aggregator 2), > >(mdev_type i915-GVTg_V5_8 + aggregator 4) > the way you are doing the nameing is till really confusing by the way > if this has not already been merged in the kernel can you chagne the mdev > so that mdev_type i915-GVTg_V5_2 is 2 of mdev_type i915-GVTg_V5_1 instead of > half the device > > currently you need to deived the aggratod by the number at the end of the > mdev type to figure out > how much of the phsicial device is being used with is a very unfridly api > convention > > the way aggrator are being proposed in general is not really someting i like > but i thin this at least > is something that should be able to correct. > > with the complexity in the mdev type name + aggrator i suspect that this will > never be support > in openstack nova directly requireing integration via cyborg unless we can > pre partion the > device in to mdevs staicaly and just ignore this. > > this is way to vendor sepecif to integrate into something like openstack in > nova unless we can guarentee > taht how aggreator work will be portable across vendors genericly. > > > > > and aggragator may be just one of such examples that 1:1 matching does not > > fit. > for openstack nova i dont see us support anything beyond the 1:1 case where > the mdev type does not change. > hi Sean, I understand it's hard for openstack. but 1:N is always meaningful. e.g. if source device 1 has cap A, it is compatible to device 2: cap A, device 3: cap A+B, device 4: cap A+B+C to allow openstack to detect it correctly, in compatible list of device 2, we would say compatible cap is A; device 3, compatible cap is A or A+B; device 4, compatible cap is A or A+B, or A+B+C; then if openstack finds device A's self cap A is contained in compatible cap of device 2/3/4, it can migrate device 1 to device 2,3,4. conversely, device 1's compatible cap is only A, so it is able to migrate device 2 to device 1, and it is not able to migrate device 3/4 to device 1. Thanks Yan > i woudl really prefer if there was just one mdev type that repsented the > minimal allcatable unit and the > aggragaotr where used to create compostions of that. i.e instad of > i915-GVTg_V5_2 beign half the device, > have 1 mdev type i915-GVTg and if the device support 8 of them then we can > aggrate 4 of i915-GVTg > > if you want to have muplie mdev type to model the different amoutn of the > resouce e.g. i915-GVTg_small i915-GVTg_large > that is totlaly fine too or even i915-GVTg_4 indcating it sis 4 of i915-GVTg > > failing that i would just expose an mdev type per composable resouce and > allow us to compose them a the user level with >
[ovirt-devel] Re: device compatibility interface for live migration with assigned devices
On Wed, Aug 19, 2020 at 09:22:34PM -0600, Alex Williamson wrote:
> On Thu, 20 Aug 2020 08:39:22 +0800
> Yan Zhao wrote:
>
> > On Tue, Aug 18, 2020 at 11:36:52AM +0200, Cornelia Huck wrote:
> > > On Tue, 18 Aug 2020 10:16:28 +0100
> > > Daniel P. Berrangé wrote:
> > >
> > > > On Tue, Aug 18, 2020 at 05:01:51PM +0800, Jason Wang wrote:
> > > > >On 2020/8/18 下午4:55, Daniel P. Berrangé wrote:
> > > > >
> > > > > On Tue, Aug 18, 2020 at 11:24:30AM +0800, Jason Wang wrote:
> > > > >
> > > > > On 2020/8/14 下午1:16, Yan Zhao wrote:
> > > > >
> > > > > On Thu, Aug 13, 2020 at 12:24:50PM +0800, Jason Wang wrote:
> > > > >
> > > > > On 2020/8/10 下午3:46, Yan Zhao wrote:
> > > >
> > > > > we actually can also retrieve the same information through sysfs,
> > > > > .e.g
> > > > >
> > > > > |- [path to device]
> > > > > |--- migration
> > > > > | |--- self
> > > > > | | |---device_api
> > > > > || |---mdev_type
> > > > > || |---software_version
> > > > > || |---device_id
> > > > > || |---aggregator
> > > > > | |--- compatible
> > > > > | | |---device_api
> > > > > || |---mdev_type
> > > > > || |---software_version
> > > > > || |---device_id
> > > > > || |---aggregator
> > > > >
> > > > >
> > > > > Yes but:
> > > > >
> > > > > - You need one file per attribute (one syscall for one attribute)
> > > > > - Attribute is coupled with kobject
> > >
> > > Is that really that bad? You have the device with an embedded kobject
> > > anyway, and you can just put things into an attribute group?
> > >
> > > [Also, I think that self/compatible split in the example makes things
> > > needlessly complex. Shouldn't semantic versioning and matching already
> > > cover nearly everything? I would expect very few cases that are more
> > > complex than that. Maybe the aggregation stuff, but I don't think we
> > > need that self/compatible split for that, either.]
> > Hi Cornelia,
> >
> > The reason I want to declare compatible list of attributes is that
> > sometimes it's not a simple 1:1 matching of source attributes and target
> > attributes
> > as I demonstrated below,
> > source mdev of (mdev_type i915-GVTg_V5_2 + aggregator 1) is compatible to
> > target mdev of (mdev_type i915-GVTg_V5_4 + aggregator 2),
> >(mdev_type i915-GVTg_V5_8 + aggregator 4)
> >
> > and aggragator may be just one of such examples that 1:1 matching does not
> > fit.
>
> If you're suggesting that we need a new 'compatible' set for every
> aggregation, haven't we lost the purpose of aggregation? For example,
> rather than having N mdev types to represent all the possible
> aggregation values, we have a single mdev type with N compatible
> migration entries, one for each possible aggregation value. BTW, how do
> we have multiple compatible directories? compatible0001,
> compatible0002? Thanks,
>
do you think the bin_attribute I proposed yesterday good?
Then we can have a single compatible with a variable in the mdev_type and
aggregator.
mdev_type=i915-GVTg_V5_{val1:int:2,4,8}
aggregator={val1}/2
Thanks
Yan
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/5D57XTSYLP3FTAOZS5R3GRZ2EXLSY5MZ/
[ovirt-devel] Re: device compatibility interface for live migration with assigned devices
On Wed, Aug 19, 2020 at 09:13:45PM -0600, Alex Williamson wrote: > On Thu, 20 Aug 2020 08:18:10 +0800 > Yan Zhao wrote: > > > On Wed, Aug 19, 2020 at 11:50:21AM -0600, Alex Williamson wrote: > > <...> > > > > > > > What I care about is that we have a *standard* userspace API for > > > > > > > performing device compatibility checking / state migration, for > > > > > > > use by > > > > > > > QEMU/libvirt/ OpenStack, such that we can write code without > > > > > > > countless > > > > > > > vendor specific code paths. > > > > > > > > > > > > > > If there is vendor specific stuff on the side, that's fine as we > > > > > > > can > > > > > > > ignore that, but the core functionality for device compat / > > > > > > > migration > > > > > > > needs to be standardized. > > > > > > > > > > > > To summarize: > > > > > > - choose one of sysfs or devlink > > > > > > - have a common interface, with a standardized way to add > > > > > > vendor-specific attributes > > > > > > ? > > > > > > > > > > Please refer to my previous email which has more example and details. > > > > > > > > > hi Parav, > > > > the example is based on a new vdpa tool running over netlink, not based > > > > on devlink, right? > > > > For vfio migration compatibility, we have to deal with both mdev and > > > > physical > > > > pci devices, I don't think it's a good idea to write a new tool for it, > > > > given > > > > we are able to retrieve the same info from sysfs and there's already an > > > > mdevctl from Alex (https://github.com/mdevctl/mdevctl). > > > > > > > > hi All, > > > > could we decide that sysfs is the interface that every VFIO vendor > > > > driver > > > > needs to provide in order to support vfio live migration, otherwise the > > > > userspace management tool would not list the device into the compatible > > > > list? > > > > > > > > if that's true, let's move to the standardizing of the sysfs interface. > > > > (1) content > > > > common part: (must) > > > >- software_version: (in major.minor.bugfix scheme) > > > >- device_api: vfio-pci or vfio-ccw ... > > > >- type: mdev type for mdev device or > > > >a signature for physical device which is a counterpart for > > > >mdev type. > > > > > > > > device api specific part: (must) > > > > - pci id: pci id of mdev parent device or pci id of physical pci > > > > device (device_api is vfio-pci) > > > > > > As noted previously, the parent PCI ID should not matter for an mdev > > > device, if a vendor has a dependency on matching the parent device PCI > > > ID, that's a vendor specific restriction. An mdev device can also > > > expose a vfio-pci device API without the parent device being PCI. For > > > a physical PCI device, shouldn't the PCI ID be encompassed in the > > > signature? Thanks, > > > > > you are right. I need to put the PCI ID as a vendor specific field. > > I didn't do that because I wanted all fields in vendor specific to be > > configurable by management tools, so they can configure the target device > > according to the value of a vendor specific field even they don't know > > the meaning of the field. > > But maybe they can just ignore the field when they can't find a matching > > writable field to configure the target. > > > If fields can be ignored, what's the point of reporting them? Seems > it's no longer a requirement. Thanks, > sorry about the confusion. I mean this condition: about to migrate, openstack searches if there are existing matching MDEVs, if yes, i.e. all common/vendor specific fields match, then just create a VM with the matching target MDEV. (in this condition, the PCI ID field is not ignored); if not, openstack tries to create one MDEV according to mdev_type, and configures MDEV according to the vendor specific attributes. as PCI ID is not a configurable field, it just ignore the field. Thanks Yan ___ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/NCJLF7GXLGXKJNQJZHDDFY3RTA7NJJQY/
[ovirt-devel] Re: device compatibility interface for live migration with assigned devices
On Thu, 20 Aug 2020 08:39:22 +0800 Yan Zhao wrote: > On Tue, Aug 18, 2020 at 11:36:52AM +0200, Cornelia Huck wrote: > > On Tue, 18 Aug 2020 10:16:28 +0100 > > Daniel P. Berrangé wrote: > > > > > On Tue, Aug 18, 2020 at 05:01:51PM +0800, Jason Wang wrote: > > > >On 2020/8/18 下午4:55, Daniel P. Berrangé wrote: > > > > > > > > On Tue, Aug 18, 2020 at 11:24:30AM +0800, Jason Wang wrote: > > > > > > > > On 2020/8/14 下午1:16, Yan Zhao wrote: > > > > > > > > On Thu, Aug 13, 2020 at 12:24:50PM +0800, Jason Wang wrote: > > > > > > > > On 2020/8/10 下午3:46, Yan Zhao wrote: > > > > > > > we actually can also retrieve the same information through sysfs, .e.g > > > > > > > > |- [path to device] > > > > |--- migration > > > > | |--- self > > > > | | |---device_api > > > > || |---mdev_type > > > > || |---software_version > > > > || |---device_id > > > > || |---aggregator > > > > | |--- compatible > > > > | | |---device_api > > > > || |---mdev_type > > > > || |---software_version > > > > || |---device_id > > > > || |---aggregator > > > > > > > > > > > > Yes but: > > > > > > > > - You need one file per attribute (one syscall for one attribute) > > > > - Attribute is coupled with kobject > > > > Is that really that bad? You have the device with an embedded kobject > > anyway, and you can just put things into an attribute group? > > > > [Also, I think that self/compatible split in the example makes things > > needlessly complex. Shouldn't semantic versioning and matching already > > cover nearly everything? I would expect very few cases that are more > > complex than that. Maybe the aggregation stuff, but I don't think we > > need that self/compatible split for that, either.] > Hi Cornelia, > > The reason I want to declare compatible list of attributes is that > sometimes it's not a simple 1:1 matching of source attributes and target > attributes > as I demonstrated below, > source mdev of (mdev_type i915-GVTg_V5_2 + aggregator 1) is compatible to > target mdev of (mdev_type i915-GVTg_V5_4 + aggregator 2), >(mdev_type i915-GVTg_V5_8 + aggregator 4) > > and aggragator may be just one of such examples that 1:1 matching does not > fit. If you're suggesting that we need a new 'compatible' set for every aggregation, haven't we lost the purpose of aggregation? For example, rather than having N mdev types to represent all the possible aggregation values, we have a single mdev type with N compatible migration entries, one for each possible aggregation value. BTW, how do we have multiple compatible directories? compatible0001, compatible0002? Thanks, Alex ___ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/64OQSLXMX2I2ZZ7TTFLW5ENA4KHPWOLZ/
[ovirt-devel] Re: device compatibility interface for live migration with assigned devices
On Thu, 20 Aug 2020 08:18:10 +0800
Yan Zhao wrote:
> On Wed, Aug 19, 2020 at 11:50:21AM -0600, Alex Williamson wrote:
> <...>
> > > > > > What I care about is that we have a *standard* userspace API for
> > > > > > performing device compatibility checking / state migration, for use
> > > > > > by
> > > > > > QEMU/libvirt/ OpenStack, such that we can write code without
> > > > > > countless
> > > > > > vendor specific code paths.
> > > > > >
> > > > > > If there is vendor specific stuff on the side, that's fine as we can
> > > > > > ignore that, but the core functionality for device compat /
> > > > > > migration
> > > > > > needs to be standardized.
> > > > >
> > > > > To summarize:
> > > > > - choose one of sysfs or devlink
> > > > > - have a common interface, with a standardized way to add
> > > > > vendor-specific attributes
> > > > > ?
> > > >
> > > > Please refer to my previous email which has more example and details.
> > > >
> > > hi Parav,
> > > the example is based on a new vdpa tool running over netlink, not based
> > > on devlink, right?
> > > For vfio migration compatibility, we have to deal with both mdev and
> > > physical
> > > pci devices, I don't think it's a good idea to write a new tool for it,
> > > given
> > > we are able to retrieve the same info from sysfs and there's already an
> > > mdevctl from Alex (https://github.com/mdevctl/mdevctl).
> > >
> > > hi All,
> > > could we decide that sysfs is the interface that every VFIO vendor driver
> > > needs to provide in order to support vfio live migration, otherwise the
> > > userspace management tool would not list the device into the compatible
> > > list?
> > >
> > > if that's true, let's move to the standardizing of the sysfs interface.
> > > (1) content
> > > common part: (must)
> > >- software_version: (in major.minor.bugfix scheme)
> > >- device_api: vfio-pci or vfio-ccw ...
> > >- type: mdev type for mdev device or
> > >a signature for physical device which is a counterpart for
> > > mdev type.
> > >
> > > device api specific part: (must)
> > > - pci id: pci id of mdev parent device or pci id of physical pci
> > > device (device_api is vfio-pci)
> >
> > As noted previously, the parent PCI ID should not matter for an mdev
> > device, if a vendor has a dependency on matching the parent device PCI
> > ID, that's a vendor specific restriction. An mdev device can also
> > expose a vfio-pci device API without the parent device being PCI. For
> > a physical PCI device, shouldn't the PCI ID be encompassed in the
> > signature? Thanks,
> >
> you are right. I need to put the PCI ID as a vendor specific field.
> I didn't do that because I wanted all fields in vendor specific to be
> configurable by management tools, so they can configure the target device
> according to the value of a vendor specific field even they don't know
> the meaning of the field.
> But maybe they can just ignore the field when they can't find a matching
> writable field to configure the target.
If fields can be ignored, what's the point of reporting them? Seems
it's no longer a requirement. Thanks,
Alex
> > > - subchannel_type (device_api is vfio-ccw)
> > >
> > > vendor driver specific part: (optional)
> > > - aggregator
> > > - chpid_type
> > > - remote_url
> > >
> > > NOTE: vendors are free to add attributes in this part with a
> > > restriction that this attribute is able to be configured with the same
> > > name in sysfs too. e.g.
> > > for aggregator, there must be a sysfs attribute in device node
> > > /sys/devices/pci:00/:00:02.0/882cc4da-dede-11e7-9180-078a62063ab1/intel_vgpu/aggregator,
> > > so that the userspace tool is able to configure the target device
> > > according to source device's aggregator attribute.
> > >
> > >
> > > (2) where and structure
> > > proposal 1:
> > > |- [path to device]
> > > |--- migration
> > > | |--- self
> > > | ||-software_version
> > > | ||-device_api
> > > | ||-type
> > > | ||-[pci_id or subchannel_type]
> > > | ||-
> > > | |--- compatible
> > > | ||-software_version
> > > | ||-device_api
> > > | ||-type
> > > | ||-[pci_id or subchannel_type]
> > > | ||-
> > > multiple compatible is allowed.
> > > attributes should be ASCII text files, preferably with only one value
> > > per file.
> > >
> > >
> > > proposal 2: use bin_attribute.
> > > |- [path to device]
> > > |--- migration
> > > | |--- self
> > > | |--- compatible
> > >
> > > so we can continue use multiline format. e.g.
> > > cat compatible
> > > software_version=0.1.0
> > > device_api=vfio_pci
> > > type=i915-GVTg_V5_{val1:int:1,2,4,8}
> > > pci_id=80865963
> > > aggregator={val1}/2
> > >
> > > Thanks
> > > Yan
> > >
> >
>
___
Devel mailing list -- devel@ovirt.org
To unsubscribe
[ovirt-devel] Re: device compatibility interface for live migration with assigned devices
On Thu, 2020-08-20 at 08:39 +0800, Yan Zhao wrote: > On Tue, Aug 18, 2020 at 11:36:52AM +0200, Cornelia Huck wrote: > > On Tue, 18 Aug 2020 10:16:28 +0100 > > Daniel P. Berrangé wrote: > > > > > On Tue, Aug 18, 2020 at 05:01:51PM +0800, Jason Wang wrote: > > > >On 2020/8/18 下午4:55, Daniel P. Berrangé wrote: > > > > > > > > On Tue, Aug 18, 2020 at 11:24:30AM +0800, Jason Wang wrote: > > > > > > > > On 2020/8/14 下午1:16, Yan Zhao wrote: > > > > > > > > On Thu, Aug 13, 2020 at 12:24:50PM +0800, Jason Wang wrote: > > > > > > > > On 2020/8/10 下午3:46, Yan Zhao wrote: > > > > we actually can also retrieve the same information through sysfs, .e.g > > > > > > > > |- [path to device] > > > > |--- migration > > > > | |--- self > > > > | | |---device_api > > > > || |---mdev_type > > > > || |---software_version > > > > || |---device_id > > > > || |---aggregator > > > > | |--- compatible > > > > | | |---device_api > > > > || |---mdev_type > > > > || |---software_version > > > > || |---device_id > > > > || |---aggregator > > > > > > > > > > > > Yes but: > > > > > > > > - You need one file per attribute (one syscall for one attribute) > > > > - Attribute is coupled with kobject > > > > Is that really that bad? You have the device with an embedded kobject > > anyway, and you can just put things into an attribute group? > > > > [Also, I think that self/compatible split in the example makes things > > needlessly complex. Shouldn't semantic versioning and matching already > > cover nearly everything? I would expect very few cases that are more > > complex than that. Maybe the aggregation stuff, but I don't think we > > need that self/compatible split for that, either.] > > Hi Cornelia, > > The reason I want to declare compatible list of attributes is that > sometimes it's not a simple 1:1 matching of source attributes and target > attributes > as I demonstrated below, > source mdev of (mdev_type i915-GVTg_V5_2 + aggregator 1) is compatible to > target mdev of (mdev_type i915-GVTg_V5_4 + aggregator 2), >(mdev_type i915-GVTg_V5_8 + aggregator 4) the way you are doing the nameing is till really confusing by the way if this has not already been merged in the kernel can you chagne the mdev so that mdev_type i915-GVTg_V5_2 is 2 of mdev_type i915-GVTg_V5_1 instead of half the device currently you need to deived the aggratod by the number at the end of the mdev type to figure out how much of the phsicial device is being used with is a very unfridly api convention the way aggrator are being proposed in general is not really someting i like but i thin this at least is something that should be able to correct. with the complexity in the mdev type name + aggrator i suspect that this will never be support in openstack nova directly requireing integration via cyborg unless we can pre partion the device in to mdevs staicaly and just ignore this. this is way to vendor sepecif to integrate into something like openstack in nova unless we can guarentee taht how aggreator work will be portable across vendors genericly. > > and aggragator may be just one of such examples that 1:1 matching does not > fit. for openstack nova i dont see us support anything beyond the 1:1 case where the mdev type does not change. i woudl really prefer if there was just one mdev type that repsented the minimal allcatable unit and the aggragaotr where used to create compostions of that. i.e instad of i915-GVTg_V5_2 beign half the device, have 1 mdev type i915-GVTg and if the device support 8 of them then we can aggrate 4 of i915-GVTg if you want to have muplie mdev type to model the different amoutn of the resouce e.g. i915-GVTg_small i915-GVTg_large that is totlaly fine too or even i915-GVTg_4 indcating it sis 4 of i915-GVTg failing that i would just expose an mdev type per composable resouce and allow us to compose them a the user level with some other construct mudeling a attament to the device. e.g. create composed mdev or somethig that is an aggreateion of multiple sub resouces each of which is an mdev. so kind of like how bond port work. we would create an mdev for each of the sub resouces and then create a bond or aggrated mdev by reference the other mdevs by uuid then attach only the aggreated mdev to the instance. the current aggrator syntax and sematic however make me rather uncofrotable when i think about orchestating vms on top of it even to boot them let alone migrate them. > > So, we explicitly list out self/compatible attributes, and management > tools only need to check if self attributes is contained compatible > attributes. > > or do you mean only compatible list is enough, and the management tools > need to find out self list by themselves? > But I think provide a self list is easier for management tools. > > Thanks > Yan >
[ovirt-devel] Re: device compatibility interface for live migration with assigned devices
On Tue, Aug 18, 2020 at 11:36:52AM +0200, Cornelia Huck wrote: > On Tue, 18 Aug 2020 10:16:28 +0100 > Daniel P. Berrangé wrote: > > > On Tue, Aug 18, 2020 at 05:01:51PM +0800, Jason Wang wrote: > > >On 2020/8/18 下午4:55, Daniel P. Berrangé wrote: > > > > > > On Tue, Aug 18, 2020 at 11:24:30AM +0800, Jason Wang wrote: > > > > > > On 2020/8/14 下午1:16, Yan Zhao wrote: > > > > > > On Thu, Aug 13, 2020 at 12:24:50PM +0800, Jason Wang wrote: > > > > > > On 2020/8/10 下午3:46, Yan Zhao wrote: > > > > > we actually can also retrieve the same information through sysfs, .e.g > > > > > > |- [path to device] > > > |--- migration > > > | |--- self > > > | | |---device_api > > > || |---mdev_type > > > || |---software_version > > > || |---device_id > > > || |---aggregator > > > | |--- compatible > > > | | |---device_api > > > || |---mdev_type > > > || |---software_version > > > || |---device_id > > > || |---aggregator > > > > > > > > > Yes but: > > > > > > - You need one file per attribute (one syscall for one attribute) > > > - Attribute is coupled with kobject > > Is that really that bad? You have the device with an embedded kobject > anyway, and you can just put things into an attribute group? > > [Also, I think that self/compatible split in the example makes things > needlessly complex. Shouldn't semantic versioning and matching already > cover nearly everything? I would expect very few cases that are more > complex than that. Maybe the aggregation stuff, but I don't think we > need that self/compatible split for that, either.] Hi Cornelia, The reason I want to declare compatible list of attributes is that sometimes it's not a simple 1:1 matching of source attributes and target attributes as I demonstrated below, source mdev of (mdev_type i915-GVTg_V5_2 + aggregator 1) is compatible to target mdev of (mdev_type i915-GVTg_V5_4 + aggregator 2), (mdev_type i915-GVTg_V5_8 + aggregator 4) and aggragator may be just one of such examples that 1:1 matching does not fit. So, we explicitly list out self/compatible attributes, and management tools only need to check if self attributes is contained compatible attributes. or do you mean only compatible list is enough, and the management tools need to find out self list by themselves? But I think provide a self list is easier for management tools. Thanks Yan ___ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CKSO5JRQOY2WJGLARGYE6OO5ZF6SZYDB/
[ovirt-devel] Re: device compatibility interface for live migration with assigned devices
On Wed, Aug 19, 2020 at 11:50:21AM -0600, Alex Williamson wrote:
<...>
> > > > > What I care about is that we have a *standard* userspace API for
> > > > > performing device compatibility checking / state migration, for use by
> > > > > QEMU/libvirt/ OpenStack, such that we can write code without countless
> > > > > vendor specific code paths.
> > > > >
> > > > > If there is vendor specific stuff on the side, that's fine as we can
> > > > > ignore that, but the core functionality for device compat / migration
> > > > > needs to be standardized.
> > > >
> > > > To summarize:
> > > > - choose one of sysfs or devlink
> > > > - have a common interface, with a standardized way to add
> > > > vendor-specific attributes
> > > > ?
> > >
> > > Please refer to my previous email which has more example and details.
> > hi Parav,
> > the example is based on a new vdpa tool running over netlink, not based
> > on devlink, right?
> > For vfio migration compatibility, we have to deal with both mdev and
> > physical
> > pci devices, I don't think it's a good idea to write a new tool for it,
> > given
> > we are able to retrieve the same info from sysfs and there's already an
> > mdevctl from Alex (https://github.com/mdevctl/mdevctl).
> >
> > hi All,
> > could we decide that sysfs is the interface that every VFIO vendor driver
> > needs to provide in order to support vfio live migration, otherwise the
> > userspace management tool would not list the device into the compatible
> > list?
> >
> > if that's true, let's move to the standardizing of the sysfs interface.
> > (1) content
> > common part: (must)
> >- software_version: (in major.minor.bugfix scheme)
> >- device_api: vfio-pci or vfio-ccw ...
> >- type: mdev type for mdev device or
> >a signature for physical device which is a counterpart for
> >mdev type.
> >
> > device api specific part: (must)
> > - pci id: pci id of mdev parent device or pci id of physical pci
> > device (device_api is vfio-pci)
>
> As noted previously, the parent PCI ID should not matter for an mdev
> device, if a vendor has a dependency on matching the parent device PCI
> ID, that's a vendor specific restriction. An mdev device can also
> expose a vfio-pci device API without the parent device being PCI. For
> a physical PCI device, shouldn't the PCI ID be encompassed in the
> signature? Thanks,
>
you are right. I need to put the PCI ID as a vendor specific field.
I didn't do that because I wanted all fields in vendor specific to be
configurable by management tools, so they can configure the target device
according to the value of a vendor specific field even they don't know
the meaning of the field.
But maybe they can just ignore the field when they can't find a matching
writable field to configure the target.
Thanks
Yan
> > - subchannel_type (device_api is vfio-ccw)
> >
> > vendor driver specific part: (optional)
> > - aggregator
> > - chpid_type
> > - remote_url
> >
> > NOTE: vendors are free to add attributes in this part with a
> > restriction that this attribute is able to be configured with the same
> > name in sysfs too. e.g.
> > for aggregator, there must be a sysfs attribute in device node
> > /sys/devices/pci:00/:00:02.0/882cc4da-dede-11e7-9180-078a62063ab1/intel_vgpu/aggregator,
> > so that the userspace tool is able to configure the target device
> > according to source device's aggregator attribute.
> >
> >
> > (2) where and structure
> > proposal 1:
> > |- [path to device]
> > |--- migration
> > | |--- self
> > | ||-software_version
> > | ||-device_api
> > | ||-type
> > | ||-[pci_id or subchannel_type]
> > | ||-
> > | |--- compatible
> > | ||-software_version
> > | ||-device_api
> > | ||-type
> > | ||-[pci_id or subchannel_type]
> > | ||-
> > multiple compatible is allowed.
> > attributes should be ASCII text files, preferably with only one value
> > per file.
> >
> >
> > proposal 2: use bin_attribute.
> > |- [path to device]
> > |--- migration
> > | |--- self
> > | |--- compatible
> >
> > so we can continue use multiline format. e.g.
> > cat compatible
> > software_version=0.1.0
> > device_api=vfio_pci
> > type=i915-GVTg_V5_{val1:int:1,2,4,8}
> > pci_id=80865963
> > aggregator={val1}/2
> >
> > Thanks
> > Yan
> >
>
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/AYS5JBKQLPFBE3MQTDTNK52MFMFSHSI7/
[ovirt-devel] Re: device compatibility interface for live migration with assigned devices
On Wed, 19 Aug 2020 11:30:35 +0800 Yan Zhao wrote: > On Tue, Aug 18, 2020 at 09:39:24AM +, Parav Pandit wrote: > > Hi Cornelia, > > > > > From: Cornelia Huck > > > Sent: Tuesday, August 18, 2020 3:07 PM > > > To: Daniel P. Berrangé > > > Cc: Jason Wang ; Yan Zhao > > > ; k...@vger.kernel.org; libvir-l...@redhat.com; > > > qemu-de...@nongnu.org; Kirti Wankhede ; > > > eau...@redhat.com; xin-ran.w...@intel.com; cor...@lwn.net; openstack- > > > disc...@lists.openstack.org; shaohe.f...@intel.com; kevin.t...@intel.com; > > > Parav Pandit ; jian-feng.d...@intel.com; > > > dgilb...@redhat.com; zhen...@linux.intel.com; hejie...@intel.com; > > > bao.yum...@zte.com.cn; Alex Williamson ; > > > eskul...@redhat.com; smoo...@redhat.com; intel-gvt- > > > d...@lists.freedesktop.org; Jiri Pirko ; > > > dinec...@redhat.com; devel@ovirt.org > > > Subject: Re: device compatibility interface for live migration with > > > assigned > > > devices > > > > > > On Tue, 18 Aug 2020 10:16:28 +0100 > > > Daniel P. Berrangé wrote: > > > > > > > On Tue, Aug 18, 2020 at 05:01:51PM +0800, Jason Wang wrote: > > > > >On 2020/8/18 下午4:55, Daniel P. Berrangé wrote: > > > > > > > > > > On Tue, Aug 18, 2020 at 11:24:30AM +0800, Jason Wang wrote: > > > > > > > > > > On 2020/8/14 下午1:16, Yan Zhao wrote: > > > > > > > > > > On Thu, Aug 13, 2020 at 12:24:50PM +0800, Jason Wang wrote: > > > > > > > > > > On 2020/8/10 下午3:46, Yan Zhao wrote: > > > > > > > > > we actually can also retrieve the same information through sysfs, > > > > > .e.g > > > > > > > > > > |- [path to device] > > > > > |--- migration > > > > > | |--- self > > > > > | | |---device_api > > > > > || |---mdev_type > > > > > || |---software_version > > > > > || |---device_id > > > > > || |---aggregator > > > > > | |--- compatible > > > > > | | |---device_api > > > > > || |---mdev_type > > > > > || |---software_version > > > > > || |---device_id > > > > > || |---aggregator > > > > > > > > > > > > > > > Yes but: > > > > > > > > > > - You need one file per attribute (one syscall for one attribute) > > > > > - Attribute is coupled with kobject > > > > > > Is that really that bad? You have the device with an embedded kobject > > > anyway, and you can just put things into an attribute group? > > > > > > [Also, I think that self/compatible split in the example makes things > > > needlessly complex. Shouldn't semantic versioning and matching already > > > cover nearly everything? I would expect very few cases that are more > > > complex than that. Maybe the aggregation stuff, but I don't think we need > > > that self/compatible split for that, either.] > > > > > > > > > > > > > All of above seems unnecessary. > > > > > > > > > > Another point, as we discussed in another thread, it's really hard > > > > > to make sure the above API work for all types of devices and > > > > > frameworks. So having a vendor specific API looks much better. > > > > > > > > > > From the POV of userspace mgmt apps doing device compat checking / > > > > > migration, we certainly do NOT want to use different vendor > > > > > specific APIs. We want to have an API that can be used / controlled > > > > > in a > > > standard manner across vendors. > > > > > > > > > >Yes, but it could be hard. E.g vDPA will chose to use devlink > > > > > (there's a > > > > >long debate on sysfs vs devlink). So if we go with sysfs, at least > > > > > two > > > > >APIs needs to be supported ... > > > > > > > > NB, I was not questioning devlink vs sysfs directly. If devlink is > > > > related to netlink, I can't say I'm enthusiastic as IMKE sysfs is > > > > easier to deal with. I don't know enough about devlink to have much of > > > > an > > > opinion though. > > > > The key point was that I don't want the userspace APIs we need to deal > > > > with to be vendor specific. > > > > > > From what I've seen of devlink, it seems quite nice; but I understand why > > > sysfs might be easier to deal with (especially as there's likely already > > > a lot of > > > code using it.) > > > > > > I understand that some users would like devlink because it is already > > > widely > > > used for network drivers (and some others), but I don't think the > > > majority of > > > devices used with vfio are network (although certainly a lot of them are.) > > > > > > > > > > > What I care about is that we have a *standard* userspace API for > > > > performing device compatibility checking / state migration, for use by > > > > QEMU/libvirt/ OpenStack, such that we can write code without countless > > > > vendor specific code paths. > > > > > > > > If there is vendor specific stuff on the side, that's fine as we can > > > > ignore that, but the core functionality for device compat / migration > > > > needs to be standardized. > > > > > > To summarize: > > > - choose
[ovirt-devel] Vdsm: ovirt-4.4.2 branch created
Hi, ovirt-4.4.2 branch has been created in Vdsm repository. Any future changes for 4.4.2 must be backported there. You can merge non-4.4.2 stuff to master now. Regards, Milan ___ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/3DUB42LF3ZYBZTU3ONVZQS5VJW6QLWOW/
[ovirt-devel] Re: device compatibility interface for live migration with assigned devices
On 2020/8/19 下午1:58, Parav Pandit wrote:
From: Yan Zhao
Sent: Wednesday, August 19, 2020 9:01 AM
On Tue, Aug 18, 2020 at 09:39:24AM +, Parav Pandit wrote:
Please refer to my previous email which has more example and details.
hi Parav,
the example is based on a new vdpa tool running over netlink, not based on
devlink, right?
Right.
For vfio migration compatibility, we have to deal with both mdev and physical
pci devices, I don't think it's a good idea to write a new tool for it, given
we are
able to retrieve the same info from sysfs and there's already an mdevctl from
mdev attribute should be visible in the mdev's sysfs tree.
I do not propose to write a new mdev tool over netlink. I am sorry if I implied
that with my suggestion of vdpa tool.
If underlying device is vdpa, mdev might be able to understand vdpa device and
query from it and populate in mdev sysfs tree.
Note that vdpa is bus independent so it can't work now and the support
of mdev on top of vDPA have been rejected (and duplicated with vhost-vDPA).
Thanks
The vdpa tool I propose is usable even without mdevs.
vdpa tool's role is to create one or more vdpa devices and place on the "vdpa"
bus which is the lowest layer here.
Additionally this tool let user query virtqueue stats, db stats.
When a user creates vdpa net device, user may need to configure features of the
vdpa device such as VIRTIO_NET_F_MAC, default VIRTIO_NET_F_MTU.
These are vdpa level features, attributes. Mdev is layer above it.
Alex
(https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.
com%2Fmdevctl%2Fmdevctldata=02%7C01%7Cparav%40nvidia.com%7C
0c2691d430304f5ea11308d843f2d84e%7C43083d15727340c1b7db39efd9ccc17
a%7C0%7C0%7C637334057571911357sdata=KxH7PwxmKyy9JODut8BWr
LQyOBylW00%2Fyzc4rEvjUvA%3Dreserved=0).
Sorry for above link mangling. Our mail server is still transitioning due to
company acquisition.
I am less familiar on below points to comment.
hi All,
could we decide that sysfs is the interface that every VFIO vendor driver needs
to provide in order to support vfio live migration, otherwise the userspace
management tool would not list the device into the compatible list?
if that's true, let's move to the standardizing of the sysfs interface.
(1) content
common part: (must)
- software_version: (in major.minor.bugfix scheme)
- device_api: vfio-pci or vfio-ccw ...
- type: mdev type for mdev device or
a signature for physical device which is a counterpart for
mdev type.
device api specific part: (must)
- pci id: pci id of mdev parent device or pci id of physical pci
device (device_api is vfio-pci)
- subchannel_type (device_api is vfio-ccw)
vendor driver specific part: (optional)
- aggregator
- chpid_type
- remote_url
NOTE: vendors are free to add attributes in this part with a restriction that
this
attribute is able to be configured with the same name in sysfs too. e.g.
for aggregator, there must be a sysfs attribute in device node
/sys/devices/pci:00/:00:02.0/882cc4da-dede-11e7-9180-
078a62063ab1/intel_vgpu/aggregator,
so that the userspace tool is able to configure the target device according to
source device's aggregator attribute.
(2) where and structure
proposal 1:
|- [path to device]
|--- migration
| |--- self
| ||-software_version
| ||-device_api
| ||-type
| ||-[pci_id or subchannel_type]
| ||-
| |--- compatible
| ||-software_version
| ||-device_api
| ||-type
| ||-[pci_id or subchannel_type]
| ||-
multiple compatible is allowed.
attributes should be ASCII text files, preferably with only one value per file.
proposal 2: use bin_attribute.
|- [path to device]
|--- migration
| |--- self
| |--- compatible
so we can continue use multiline format. e.g.
cat compatible
software_version=0.1.0
device_api=vfio_pci
type=i915-GVTg_V5_{val1:int:1,2,4,8}
pci_id=80865963
aggregator={val1}/2
Thanks
Yan
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/YUCNHOYJ46WJVRCKER2ISRZ3VH2P3L4O/
[ovirt-devel] Re: device compatibility interface for live migration with assigned devices
On 2020/8/19 下午4:13, Yan Zhao wrote: On Wed, Aug 19, 2020 at 03:39:50PM +0800, Jason Wang wrote: On 2020/8/19 下午2:59, Yan Zhao wrote: On Wed, Aug 19, 2020 at 02:57:34PM +0800, Jason Wang wrote: On 2020/8/19 上午11:30, Yan Zhao wrote: hi All, could we decide that sysfs is the interface that every VFIO vendor driver needs to provide in order to support vfio live migration, otherwise the userspace management tool would not list the device into the compatible list? if that's true, let's move to the standardizing of the sysfs interface. (1) content common part: (must) - software_version: (in major.minor.bugfix scheme) This can not work for devices whose features can be negotiated/advertised independently. (E.g virtio devices) sorry, I don't understand here, why virtio devices need to use vfio interface? I don't see any reason that virtio devices can't be used by VFIO. Do you? Actually, virtio devices have been used by VFIO for many years: - passthrough a hardware virtio devices to userspace(VM) drivers - using virtio PMD inside guest So, what's different for it vs passing through a physical hardware via VFIO? The difference is in the guest, the device could be either real hardware or emulated ones. even though the features are negotiated dynamically, could you explain why it would cause software_version not work? Virtio device 1 supports feature A, B, C Virtio device 2 supports feature B, C, D So you can't migrate a guest from device 1 to device 2. And it's impossible to model the features with versions. I think this thread is discussing about vfio related devices. - device_api: vfio-pci or vfio-ccw ... - type: mdev type for mdev device or a signature for physical device which is a counterpart for mdev type. device api specific part: (must) - pci id: pci id of mdev parent device or pci id of physical pci device (device_api is vfio-pci)API here. So this assumes a PCI device which is probably not true. for device_api of vfio-pci, why it's not true? for vfio-ccw, it's subchannel_type. Ok but having two different attributes for the same file is not good idea. How mgmt know there will be a 3rd type? that's why some attributes need to be common. e.g. device_api: it's common because mgmt need to know it's a pci device or a ccw device. and the api type is already defined vfio.h. (The field is agreed by and actually suggested by Alex in previous mail) type: mdev_type for mdev. if mgmt does not understand it, it would not be able to create one compatible mdev device. software_version: mgmt can compare the major and minor if it understands this fields. I think it would be helpful if you can describe how mgmt is expected to work step by step with the proposed sysfs API. This can help people to understand. Thanks for the patience. Since sysfs is uABI, when accepted, we need support it forever. That's why we need to be careful. - subchannel_type (device_api is vfio-ccw) vendor driver specific part: (optional) - aggregator - chpid_type - remote_url For "remote_url", just wonder if it's better to integrate or reuse the existing NVME management interface instead of duplicating it here. Otherwise it could be a burden for mgmt to learn. E.g vendor A may use "remote_url" but vendor B may use a different attribute. it's vendor driver specific. vendor specific attributes are inevitable, and that's why we are discussing here of a way to standardizing of it. Well, then you will end up with a very long list to discuss. E.g for networking devices, you will have "mac", "v(x)lan" and a lot of other. Note that "remote_url" is not vendor specific but NVME (class/subsystem) specific. yes, it's just NVMe specific. I added it as an example to show what is vendor specific. if one attribute is vendor specific across all vendors, then it's not vendor specific, it's already common attribute, right? It's common but the issue is about naming and mgmt overhead. Unless you have a unified API per class (NVME, ethernet, etc), you can't prevent vendor from using another name instead of "remote_url". The point is that if vendor/class specific part is unavoidable, why not making all of the attributes vendor specific? some parts need to be common, as I listed above. This is hard, unless VFIO knows the type of device (e.g it's a NVME or networking device). our goal is that mgmt can use it without understanding the meaning of vendor specific attributes. I'm not sure this is the correct design of uAPI. Is there something similar in the existing uAPIs? And it might be hard to work for virtio devices. NOTE: vendors are free to add attributes in this part with a restriction that this attribute is able to be configured with the same name in sysfs too. e.g. Sysfs works well for common attributes belongs to a class, but I'm not sure it can work well for
[ovirt-devel] Max running ansible-playbook
Hi to all. on our angine we always see max 1 ansible-playbook running on the same time. We use OVA export to make some backup, but the whole task takes a lot of time because only one ansible-playbook "/usr/share/ovirt-engine/playbooks/ovirt-ova-export.yml" run at the same time. How can we ingrease this value? -- -- Shellrent - Il primo hosting italiano Security First *Tommaso De Marchi* /COO - Chief Operating Officer/ Shellrent Srl Via dell'Edilizia, 19 - 36100 Vicenza Tel. 0444321155 | Fax 04441492177 ___ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7HYOAIQUAMLTSLKT3LZO3WZXQZRSMSBW/
[ovirt-devel] Re: device compatibility interface for live migration with assigned devices
On Wed, Aug 19, 2020 at 03:39:50PM +0800, Jason Wang wrote: > > On 2020/8/19 下午2:59, Yan Zhao wrote: > > On Wed, Aug 19, 2020 at 02:57:34PM +0800, Jason Wang wrote: > > > On 2020/8/19 上午11:30, Yan Zhao wrote: > > > > hi All, > > > > could we decide that sysfs is the interface that every VFIO vendor > > > > driver > > > > needs to provide in order to support vfio live migration, otherwise the > > > > userspace management tool would not list the device into the compatible > > > > list? > > > > > > > > if that's true, let's move to the standardizing of the sysfs interface. > > > > (1) content > > > > common part: (must) > > > > - software_version: (in major.minor.bugfix scheme) > > > > > > This can not work for devices whose features can be negotiated/advertised > > > independently. (E.g virtio devices) > > > > > sorry, I don't understand here, why virtio devices need to use vfio > > interface? > > > I don't see any reason that virtio devices can't be used by VFIO. Do you? > > Actually, virtio devices have been used by VFIO for many years: > > - passthrough a hardware virtio devices to userspace(VM) drivers > - using virtio PMD inside guest > So, what's different for it vs passing through a physical hardware via VFIO? even though the features are negotiated dynamically, could you explain why it would cause software_version not work? > > > I think this thread is discussing about vfio related devices. > > > > > > - device_api: vfio-pci or vfio-ccw ... > > > > - type: mdev type for mdev device or > > > > a signature for physical device which is a counterpart for > > > >mdev type. > > > > > > > > device api specific part: (must) > > > > - pci id: pci id of mdev parent device or pci id of physical pci > > > > device (device_api is vfio-pci)API here. > > > > > > So this assumes a PCI device which is probably not true. > > > > > for device_api of vfio-pci, why it's not true? > > > > for vfio-ccw, it's subchannel_type. > > > Ok but having two different attributes for the same file is not good idea. > How mgmt know there will be a 3rd type? that's why some attributes need to be common. e.g. device_api: it's common because mgmt need to know it's a pci device or a ccw device. and the api type is already defined vfio.h. (The field is agreed by and actually suggested by Alex in previous mail) type: mdev_type for mdev. if mgmt does not understand it, it would not be able to create one compatible mdev device. software_version: mgmt can compare the major and minor if it understands this fields. > > > > > > > > - subchannel_type (device_api is vfio-ccw) > > > > vendor driver specific part: (optional) > > > > - aggregator > > > > - chpid_type > > > > - remote_url > > > > > > For "remote_url", just wonder if it's better to integrate or reuse the > > > existing NVME management interface instead of duplicating it here. > > > Otherwise > > > it could be a burden for mgmt to learn. E.g vendor A may use "remote_url" > > > but vendor B may use a different attribute. > > > > > it's vendor driver specific. > > vendor specific attributes are inevitable, and that's why we are > > discussing here of a way to standardizing of it. > > > Well, then you will end up with a very long list to discuss. E.g for > networking devices, you will have "mac", "v(x)lan" and a lot of other. > > Note that "remote_url" is not vendor specific but NVME (class/subsystem) > specific. > yes, it's just NVMe specific. I added it as an example to show what is vendor specific. if one attribute is vendor specific across all vendors, then it's not vendor specific, it's already common attribute, right? > The point is that if vendor/class specific part is unavoidable, why not > making all of the attributes vendor specific? > some parts need to be common, as I listed above. > > > our goal is that mgmt can use it without understanding the meaning of vendor > > specific attributes. > > > I'm not sure this is the correct design of uAPI. Is there something similar > in the existing uAPIs? > > And it might be hard to work for virtio devices. > > > > > > > > NOTE: vendors are free to add attributes in this part with a > > > > restriction that this attribute is able to be configured with the same > > > > name in sysfs too. e.g. > > > > > > Sysfs works well for common attributes belongs to a class, but I'm not > > > sure > > > it can work well for device/vendor specific attributes. Does this mean > > > mgmt > > > need to iterate all the attributes in both src and dst? > > > > > no. just attributes under migration directory. > > > > > > for aggregator, there must be a sysfs attribute in device node > > > > /sys/devices/pci:00/:00:02.0/882cc4da-dede-11e7-9180-078a62063ab1/intel_vgpu/aggregator, > > > > so that the userspace tool is able to configure the target device > > > > according to source device's aggregator attribute. >
[ovirt-devel] Re: device compatibility interface for live migration with assigned devices
On 2020/8/19 下午2:59, Yan Zhao wrote:
On Wed, Aug 19, 2020 at 02:57:34PM +0800, Jason Wang wrote:
On 2020/8/19 上午11:30, Yan Zhao wrote:
hi All,
could we decide that sysfs is the interface that every VFIO vendor driver
needs to provide in order to support vfio live migration, otherwise the
userspace management tool would not list the device into the compatible
list?
if that's true, let's move to the standardizing of the sysfs interface.
(1) content
common part: (must)
- software_version: (in major.minor.bugfix scheme)
This can not work for devices whose features can be negotiated/advertised
independently. (E.g virtio devices)
sorry, I don't understand here, why virtio devices need to use vfio interface?
I don't see any reason that virtio devices can't be used by VFIO. Do you?
Actually, virtio devices have been used by VFIO for many years:
- passthrough a hardware virtio devices to userspace(VM) drivers
- using virtio PMD inside guest
I think this thread is discussing about vfio related devices.
- device_api: vfio-pci or vfio-ccw ...
- type: mdev type for mdev device or
a signature for physical device which is a counterpart for
mdev type.
device api specific part: (must)
- pci id: pci id of mdev parent device or pci id of physical pci
device (device_api is vfio-pci)API here.
So this assumes a PCI device which is probably not true.
for device_api of vfio-pci, why it's not true?
for vfio-ccw, it's subchannel_type.
Ok but having two different attributes for the same file is not good
idea. How mgmt know there will be a 3rd type?
- subchannel_type (device_api is vfio-ccw)
vendor driver specific part: (optional)
- aggregator
- chpid_type
- remote_url
For "remote_url", just wonder if it's better to integrate or reuse the
existing NVME management interface instead of duplicating it here. Otherwise
it could be a burden for mgmt to learn. E.g vendor A may use "remote_url"
but vendor B may use a different attribute.
it's vendor driver specific.
vendor specific attributes are inevitable, and that's why we are
discussing here of a way to standardizing of it.
Well, then you will end up with a very long list to discuss. E.g for
networking devices, you will have "mac", "v(x)lan" and a lot of other.
Note that "remote_url" is not vendor specific but NVME (class/subsystem)
specific.
The point is that if vendor/class specific part is unavoidable, why not
making all of the attributes vendor specific?
our goal is that mgmt can use it without understanding the meaning of vendor
specific attributes.
I'm not sure this is the correct design of uAPI. Is there something
similar in the existing uAPIs?
And it might be hard to work for virtio devices.
NOTE: vendors are free to add attributes in this part with a
restriction that this attribute is able to be configured with the same
name in sysfs too. e.g.
Sysfs works well for common attributes belongs to a class, but I'm not sure
it can work well for device/vendor specific attributes. Does this mean mgmt
need to iterate all the attributes in both src and dst?
no. just attributes under migration directory.
for aggregator, there must be a sysfs attribute in device node
/sys/devices/pci:00/:00:02.0/882cc4da-dede-11e7-9180-078a62063ab1/intel_vgpu/aggregator,
so that the userspace tool is able to configure the target device
according to source device's aggregator attribute.
(2) where and structure
proposal 1:
|- [path to device]
|--- migration
| |--- self
| ||-software_version
| ||-device_api
| ||-type
| ||-[pci_id or subchannel_type]
| ||-
| |--- compatible
| ||-software_version
| ||-device_api
| ||-type
| ||-[pci_id or subchannel_type]
| ||-
multiple compatible is allowed.
attributes should be ASCII text files, preferably with only one value
per file.
proposal 2: use bin_attribute.
|- [path to device]
|--- migration
| |--- self
| |--- compatible
so we can continue use multiline format. e.g.
cat compatible
software_version=0.1.0
device_api=vfio_pci
type=i915-GVTg_V5_{val1:int:1,2,4,8}
pci_id=80865963
aggregator={val1}/2
So basically two questions:
- how hard to standardize sysfs API for dealing with compatibility check (to
make it work for most types of devices)
sorry, I just know we are in the process of standardizing of it :)
It's not easy. As I said, the current design can't work for virtio
devices and it's not hard to find other examples. I remember some Intel
devices have bitmask based capability registers.
- how hard for the mgmt to learn with a vendor specific attributes (vs
existing management API)
what is existing management API?
It depends on the type of devices. E.g for NVME, we've already had one
(/sys/kernel/config/nvme)?
Thanks
[ovirt-devel] Re: device compatibility interface for live migration with assigned devices
On Wed, Aug 19, 2020 at 02:57:34PM +0800, Jason Wang wrote:
>
> On 2020/8/19 上午11:30, Yan Zhao wrote:
> > hi All,
> > could we decide that sysfs is the interface that every VFIO vendor driver
> > needs to provide in order to support vfio live migration, otherwise the
> > userspace management tool would not list the device into the compatible
> > list?
> >
> > if that's true, let's move to the standardizing of the sysfs interface.
> > (1) content
> > common part: (must)
> > - software_version: (in major.minor.bugfix scheme)
>
>
> This can not work for devices whose features can be negotiated/advertised
> independently. (E.g virtio devices)
>
sorry, I don't understand here, why virtio devices need to use vfio interface?
I think this thread is discussing about vfio related devices.
>
> > - device_api: vfio-pci or vfio-ccw ...
> > - type: mdev type for mdev device or
> > a signature for physical device which is a counterpart for
> >mdev type.
> >
> > device api specific part: (must)
> >- pci id: pci id of mdev parent device or pci id of physical pci
> > device (device_api is vfio-pci)API here.
>
>
> So this assumes a PCI device which is probably not true.
>
for device_api of vfio-pci, why it's not true?
for vfio-ccw, it's subchannel_type.
>
> >- subchannel_type (device_api is vfio-ccw)
> > vendor driver specific part: (optional)
> >- aggregator
> >- chpid_type
> >- remote_url
>
>
> For "remote_url", just wonder if it's better to integrate or reuse the
> existing NVME management interface instead of duplicating it here. Otherwise
> it could be a burden for mgmt to learn. E.g vendor A may use "remote_url"
> but vendor B may use a different attribute.
>
it's vendor driver specific.
vendor specific attributes are inevitable, and that's why we are
discussing here of a way to standardizing of it.
our goal is that mgmt can use it without understanding the meaning of vendor
specific attributes.
>
> >
> > NOTE: vendors are free to add attributes in this part with a
> > restriction that this attribute is able to be configured with the same
> > name in sysfs too. e.g.
>
>
> Sysfs works well for common attributes belongs to a class, but I'm not sure
> it can work well for device/vendor specific attributes. Does this mean mgmt
> need to iterate all the attributes in both src and dst?
>
no. just attributes under migration directory.
>
> > for aggregator, there must be a sysfs attribute in device node
> > /sys/devices/pci:00/:00:02.0/882cc4da-dede-11e7-9180-078a62063ab1/intel_vgpu/aggregator,
> > so that the userspace tool is able to configure the target device
> > according to source device's aggregator attribute.
> >
> >
> > (2) where and structure
> > proposal 1:
> > |- [path to device]
> >|--- migration
> >| |--- self
> >| ||-software_version
> >| ||-device_api
> >| ||-type
> >| ||-[pci_id or subchannel_type]
> >| ||-
> >| |--- compatible
> >| ||-software_version
> >| ||-device_api
> >| ||-type
> >| ||-[pci_id or subchannel_type]
> >| ||-
> > multiple compatible is allowed.
> > attributes should be ASCII text files, preferably with only one value
> > per file.
> >
> >
> > proposal 2: use bin_attribute.
> > |- [path to device]
> >|--- migration
> >| |--- self
> >| |--- compatible
> >
> > so we can continue use multiline format. e.g.
> > cat compatible
> >software_version=0.1.0
> >device_api=vfio_pci
> >type=i915-GVTg_V5_{val1:int:1,2,4,8}
> >pci_id=80865963
> >aggregator={val1}/2
>
>
> So basically two questions:
>
> - how hard to standardize sysfs API for dealing with compatibility check (to
> make it work for most types of devices)
sorry, I just know we are in the process of standardizing of it :)
> - how hard for the mgmt to learn with a vendor specific attributes (vs
> existing management API)
what is existing management API?
Thanks
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/QPOL2YB5QS3QDJ442JHSWWM4P7PTIUC3/
[ovirt-devel] Re: device compatibility interface for live migration with assigned devices
On 2020/8/19 上午11:30, Yan Zhao wrote:
hi All,
could we decide that sysfs is the interface that every VFIO vendor driver
needs to provide in order to support vfio live migration, otherwise the
userspace management tool would not list the device into the compatible
list?
if that's true, let's move to the standardizing of the sysfs interface.
(1) content
common part: (must)
- software_version: (in major.minor.bugfix scheme)
This can not work for devices whose features can be
negotiated/advertised independently. (E.g virtio devices)
- device_api: vfio-pci or vfio-ccw ...
- type: mdev type for mdev device or
a signature for physical device which is a counterpart for
mdev type.
device api specific part: (must)
- pci id: pci id of mdev parent device or pci id of physical pci
device (device_api is vfio-pci)API here.
So this assumes a PCI device which is probably not true.
- subchannel_type (device_api is vfio-ccw)
vendor driver specific part: (optional)
- aggregator
- chpid_type
- remote_url
For "remote_url", just wonder if it's better to integrate or reuse the
existing NVME management interface instead of duplicating it here.
Otherwise it could be a burden for mgmt to learn. E.g vendor A may use
"remote_url" but vendor B may use a different attribute.
NOTE: vendors are free to add attributes in this part with a
restriction that this attribute is able to be configured with the same
name in sysfs too. e.g.
Sysfs works well for common attributes belongs to a class, but I'm not
sure it can work well for device/vendor specific attributes. Does this
mean mgmt need to iterate all the attributes in both src and dst?
for aggregator, there must be a sysfs attribute in device node
/sys/devices/pci:00/:00:02.0/882cc4da-dede-11e7-9180-078a62063ab1/intel_vgpu/aggregator,
so that the userspace tool is able to configure the target device
according to source device's aggregator attribute.
(2) where and structure
proposal 1:
|- [path to device]
|--- migration
| |--- self
| ||-software_version
| ||-device_api
| ||-type
| ||-[pci_id or subchannel_type]
| ||-
| |--- compatible
| ||-software_version
| ||-device_api
| ||-type
| ||-[pci_id or subchannel_type]
| ||-
multiple compatible is allowed.
attributes should be ASCII text files, preferably with only one value
per file.
proposal 2: use bin_attribute.
|- [path to device]
|--- migration
| |--- self
| |--- compatible
so we can continue use multiline format. e.g.
cat compatible
software_version=0.1.0
device_api=vfio_pci
type=i915-GVTg_V5_{val1:int:1,2,4,8}
pci_id=80865963
aggregator={val1}/2
So basically two questions:
- how hard to standardize sysfs API for dealing with compatibility check
(to make it work for most types of devices)
- how hard for the mgmt to learn with a vendor specific attributes (vs
existing management API)
Thanks
Thanks
Yan
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/WOIFM623L2PI5B3IG3IQWUS2DQ7HL2OP/
[ovirt-devel] Re: device compatibility interface for live migration with assigned devices
> From: Jason Wang
> Sent: Wednesday, August 19, 2020 12:19 PM
>
>
> On 2020/8/19 下午1:26, Parav Pandit wrote:
> >
> >> From: Jason Wang
> >> Sent: Wednesday, August 19, 2020 8:16 AM
> >
> >> On 2020/8/18 下午5:32, Parav Pandit wrote:
> >>> Hi Jason,
> >>>
> >>> From: Jason Wang
> >>> Sent: Tuesday, August 18, 2020 2:32 PM
> >>>
> >>>
> >>> On 2020/8/18 下午4:55, Daniel P. Berrangé wrote:
> >>> On Tue, Aug 18, 2020 at 11:24:30AM +0800, Jason Wang wrote:
> >>> On 2020/8/14 下午1:16, Yan Zhao wrote:
> >>> On Thu, Aug 13, 2020 at 12:24:50PM +0800, Jason Wang wrote:
> >>> On 2020/8/10 下午3:46, Yan Zhao wrote:
> >>> driver is it handled by?
> >>> It looks that the devlink is for network device specific, and in
> >>> devlink.h, it says include/uapi/linux/devlink.h - Network physical
> >>> device Netlink interface, Actually not, I think there used to have
> >>> some discussion last year and the conclusion is to remove this
> >>> comment.
> >>>
> >>> [...]
> >>>
> Yes, but it could be hard. E.g vDPA will chose to use devlink
> (there's a long
> >> debate on sysfs vs devlink). So if we go with sysfs, at least two
> >> APIs needs to be supported ...
> >>> We had internal discussion and proposal on this topic.
> >>> I wanted Eli Cohen to be back from vacation on Wed 8/19, but since
> >>> this is
> >> active discussion right now, I will share the thoughts anyway.
> >>> Here are the initial round of thoughts and proposal.
> >>>
> >>> User requirements:
> >>> ---
> >>> 1. User might want to create one or more vdpa devices per PCI PF/VF/SF.
> >>> 2. User might want to create one or more vdpa devices of type
> >>> net/blk or
> >> other type.
> >>> 3. User needs to look and dump at the health of the queues for debug
> purpose.
> >>> 4. During vdpa net device creation time, user may have to provide a
> >>> MAC
> >> address and/or VLAN.
> >>> 5. User should be able to set/query some of the attributes for
> >>> debug/compatibility check 6. When user wants to create vdpa device,
> >>> it needs
> >> to know which device supports creation.
> >>> 7. User should be able to see the queue statistics of doorbells,
> >>> wqes etc regardless of class type
> >>
> >> Note that wqes is probably not something common in all of the vendors.
> > Yes. I virtq descriptors stats is better to monitor the virtqueues.
> >
> >>
> >>> To address above requirements, there is a need of vendor agnostic
> >>> tool, so
> >> that user can create/config/delete vdpa device(s) regardless of the vendor.
> >>> Hence,
> >>> We should have a tool that lets user do it.
> >>>
> >>> Examples:
> >>> -
> >>> (a) List parent devices which supports creating vdpa devices.
> >>> It also shows which class types supported by this parent device.
> >>> In below command two parent devices support vdpa device creation.
> >>> First is PCI VF whose bdf is 03.00:5.
> >>> Second is PCI SF whose name is mlx5_sf.1
> >>>
> >>> $ vdpa list pd
> >>
> >> What did "pd" mean?
> >>
> > Parent device which support creation of one or more vdpa devices.
> > In a system there can be multiple parent devices which may be support vdpa
> creation.
> > User should be able to know which devices support it, and when user creates
> > a
> vdpa device, it tells which parent device to use for creation as done in below
> vdpa dev add example.
> >>> pci/:03.00:5
> >>> class_supports
> >>> net vdpa
> >>> virtbus/mlx5_sf.1
> >>
> >> So creating mlx5_sf.1 is the charge of devlink?
> >>
> > Yes.
> > But here vdpa tool is working at the parent device identifier {bus+name}
> instead of devlink identifier.
> >
> >
> >>> class_supports
> >>> net
> >>>
> >>> (b) Now add a vdpa device and show the device.
> >>> $ vdpa dev add pci/:03.00:5 type net
> >>
> >> So if you want to create devices types other than vdpa on
> >> pci/:03.00:5 it needs some synchronization with devlink?
> > Please refer to FAQ-1, a new tool is not linked to devlink because vdpa
> > will
> evolve with time and devlink will fall short.
> > So no, it doesn't need any synchronization with devlink.
> > As long as parent device exist, user can create it.
> > All synchronization will be within drivers/vdpa/vdpa.c This user
> > interface is exposed via new netlink family by doing genl_register_family()
> > with
> new name "vdpa" in drivers/vdpa/vdpa.c.
>
>
> Just to make sure I understand here.
>
> Consider we had virtbus/mlx5_sf.1. Process A want to create a vDPA instance on
> top of it but Process B want to create a IB instance. Then I think some
> synchronization is needed at at least parent device level?
Likely but rdma device will be created either through
$ rdma link add command.
Or auto created by driver because there is only one without much configuration.
While vdpa device(s) for virtbus/mlx5_sf.1 will be created through vdpa
subsystem.
And vdpa's synchronization will be contained within drivers/vdpa/vdpa.c
>
>
> >
> >>
> >>> $ vdpa dev show
> >>>
[ovirt-devel] Re: device compatibility interface for live migration with assigned devices
On 2020/8/19 下午1:26, Parav Pandit wrote:
From: Jason Wang
Sent: Wednesday, August 19, 2020 8:16 AM
On 2020/8/18 下午5:32, Parav Pandit wrote:
Hi Jason,
From: Jason Wang
Sent: Tuesday, August 18, 2020 2:32 PM
On 2020/8/18 下午4:55, Daniel P. Berrangé wrote:
On Tue, Aug 18, 2020 at 11:24:30AM +0800, Jason Wang wrote:
On 2020/8/14 下午1:16, Yan Zhao wrote:
On Thu, Aug 13, 2020 at 12:24:50PM +0800, Jason Wang wrote:
On 2020/8/10 下午3:46, Yan Zhao wrote:
driver is it handled by?
It looks that the devlink is for network device specific, and in
devlink.h, it says include/uapi/linux/devlink.h - Network physical
device Netlink interface, Actually not, I think there used to have
some discussion last year and the conclusion is to remove this
comment.
[...]
Yes, but it could be hard. E.g vDPA will chose to use devlink (there's a long
debate on sysfs vs devlink). So if we go with sysfs, at least two APIs needs to
be
supported ...
We had internal discussion and proposal on this topic.
I wanted Eli Cohen to be back from vacation on Wed 8/19, but since this is
active discussion right now, I will share the thoughts anyway.
Here are the initial round of thoughts and proposal.
User requirements:
---
1. User might want to create one or more vdpa devices per PCI PF/VF/SF.
2. User might want to create one or more vdpa devices of type net/blk or
other type.
3. User needs to look and dump at the health of the queues for debug purpose.
4. During vdpa net device creation time, user may have to provide a MAC
address and/or VLAN.
5. User should be able to set/query some of the attributes for
debug/compatibility check 6. When user wants to create vdpa device, it needs
to know which device supports creation.
7. User should be able to see the queue statistics of doorbells, wqes
etc regardless of class type
Note that wqes is probably not something common in all of the vendors.
Yes. I virtq descriptors stats is better to monitor the virtqueues.
To address above requirements, there is a need of vendor agnostic tool, so
that user can create/config/delete vdpa device(s) regardless of the vendor.
Hence,
We should have a tool that lets user do it.
Examples:
-
(a) List parent devices which supports creating vdpa devices.
It also shows which class types supported by this parent device.
In below command two parent devices support vdpa device creation.
First is PCI VF whose bdf is 03.00:5.
Second is PCI SF whose name is mlx5_sf.1
$ vdpa list pd
What did "pd" mean?
Parent device which support creation of one or more vdpa devices.
In a system there can be multiple parent devices which may be support vdpa
creation.
User should be able to know which devices support it, and when user creates a
vdpa device, it tells which parent device to use for creation as done in below
vdpa dev add example.
pci/:03.00:5
class_supports
net vdpa
virtbus/mlx5_sf.1
So creating mlx5_sf.1 is the charge of devlink?
Yes.
But here vdpa tool is working at the parent device identifier {bus+name}
instead of devlink identifier.
class_supports
net
(b) Now add a vdpa device and show the device.
$ vdpa dev add pci/:03.00:5 type net
So if you want to create devices types other than vdpa on
pci/:03.00:5 it needs some synchronization with devlink?
Please refer to FAQ-1, a new tool is not linked to devlink because vdpa will
evolve with time and devlink will fall short.
So no, it doesn't need any synchronization with devlink.
As long as parent device exist, user can create it.
All synchronization will be within drivers/vdpa/vdpa.c
This user interface is exposed via new netlink family by doing genl_register_family()
with new name "vdpa" in drivers/vdpa/vdpa.c.
Just to make sure I understand here.
Consider we had virtbus/mlx5_sf.1. Process A want to create a vDPA
instance on top of it but Process B want to create a IB instance. Then I
think some synchronization is needed at at least parent device level?
$ vdpa dev show
vdpa0@pci/:03.00:5 type net state inactive maxqueues 8 curqueues 4
(c) vdpa dev show features vdpa0
iommu platform
version 1
(d) dump vdpa statistics
$ vdpa dev stats show vdpa0
kickdoorbells 10
wqes 100
(e) Now delete a vdpa device previously created.
$ vdpa dev del vdpa0
Design overview:
---
1. Above example tool runs over netlink socket interface.
2. This enables users to return meaningful error strings in addition to code so
that user can be more informed.
Often this is missing in ioctl()/configfs/sysfs interfaces.
3. This tool over netlink enables syscaller tests to be more usable like other
subsystems to keep kernel robust
4. This provides vendor agnostic view of all vdpa capable parent and vdpa
devices.
5. Each driver which supports vdpa device creation, registers the parent device
along with supported classes.
FAQs:
1. Why not using devlink?
Ans: Because as vdpa echo system grows,
[ovirt-devel] Re: device compatibility interface for live migration with assigned devices
> From: Yan Zhao
> Sent: Wednesday, August 19, 2020 9:01 AM
> On Tue, Aug 18, 2020 at 09:39:24AM +, Parav Pandit wrote:
> > Please refer to my previous email which has more example and details.
> hi Parav,
> the example is based on a new vdpa tool running over netlink, not based on
> devlink, right?
Right.
> For vfio migration compatibility, we have to deal with both mdev and physical
> pci devices, I don't think it's a good idea to write a new tool for it, given
> we are
> able to retrieve the same info from sysfs and there's already an mdevctl from
mdev attribute should be visible in the mdev's sysfs tree.
I do not propose to write a new mdev tool over netlink. I am sorry if I implied
that with my suggestion of vdpa tool.
If underlying device is vdpa, mdev might be able to understand vdpa device and
query from it and populate in mdev sysfs tree.
The vdpa tool I propose is usable even without mdevs.
vdpa tool's role is to create one or more vdpa devices and place on the "vdpa"
bus which is the lowest layer here.
Additionally this tool let user query virtqueue stats, db stats.
When a user creates vdpa net device, user may need to configure features of the
vdpa device such as VIRTIO_NET_F_MAC, default VIRTIO_NET_F_MTU.
These are vdpa level features, attributes. Mdev is layer above it.
> Alex
> (https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.
> com%2Fmdevctl%2Fmdevctldata=02%7C01%7Cparav%40nvidia.com%7C
> 0c2691d430304f5ea11308d843f2d84e%7C43083d15727340c1b7db39efd9ccc17
> a%7C0%7C0%7C637334057571911357sdata=KxH7PwxmKyy9JODut8BWr
> LQyOBylW00%2Fyzc4rEvjUvA%3Dreserved=0).
>
Sorry for above link mangling. Our mail server is still transitioning due to
company acquisition.
I am less familiar on below points to comment.
> hi All,
> could we decide that sysfs is the interface that every VFIO vendor driver
> needs
> to provide in order to support vfio live migration, otherwise the userspace
> management tool would not list the device into the compatible list?
>
> if that's true, let's move to the standardizing of the sysfs interface.
> (1) content
> common part: (must)
>- software_version: (in major.minor.bugfix scheme)
>- device_api: vfio-pci or vfio-ccw ...
>- type: mdev type for mdev device or
>a signature for physical device which is a counterpart for
> mdev type.
>
> device api specific part: (must)
> - pci id: pci id of mdev parent device or pci id of physical pci
> device (device_api is vfio-pci)
> - subchannel_type (device_api is vfio-ccw)
>
> vendor driver specific part: (optional)
> - aggregator
> - chpid_type
> - remote_url
>
> NOTE: vendors are free to add attributes in this part with a restriction that
> this
> attribute is able to be configured with the same name in sysfs too. e.g.
> for aggregator, there must be a sysfs attribute in device node
> /sys/devices/pci:00/:00:02.0/882cc4da-dede-11e7-9180-
> 078a62063ab1/intel_vgpu/aggregator,
> so that the userspace tool is able to configure the target device according to
> source device's aggregator attribute.
>
>
> (2) where and structure
> proposal 1:
> |- [path to device]
> |--- migration
> | |--- self
> | ||-software_version
> | ||-device_api
> | ||-type
> | ||-[pci_id or subchannel_type]
> | ||-
> | |--- compatible
> | ||-software_version
> | ||-device_api
> | ||-type
> | ||-[pci_id or subchannel_type]
> | ||-
> multiple compatible is allowed.
> attributes should be ASCII text files, preferably with only one value per
> file.
>
>
> proposal 2: use bin_attribute.
> |- [path to device]
> |--- migration
> | |--- self
> | |--- compatible
>
> so we can continue use multiline format. e.g.
> cat compatible
> software_version=0.1.0
> device_api=vfio_pci
> type=i915-GVTg_V5_{val1:int:1,2,4,8}
> pci_id=80865963
> aggregator={val1}/2
>
> Thanks
> Yan
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/ORWTLWUD77ZKBHZ55AAX7KZDP2U6PPU2/
