[ovirt-devel] Re: device compatibility interface for live migration with assigned devices

[Alex Williamson]

On Thu, 30 Jul 2020 11:41:04 +0800
Yan Zhao  wrote:

> On Wed, Jul 29, 2020 at 01:12:55PM -0600, Alex Williamson wrote:
> > On Wed, 29 Jul 2020 12:28:46 +0100
> > Sean Mooney  wrote:
> >   
> > > On Wed, 2020-07-29 at 16:05 +0800, Yan Zhao wrote:  
> > > > On Mon, Jul 27, 2020 at 04:23:21PM -0600, Alex Williamson wrote:
> > > > > On Mon, 27 Jul 2020 15:24:40 +0800
> > > > > Yan Zhao  wrote:
> > > > > 
> > > > > > > > As you indicate, the vendor driver is responsible for checking 
> > > > > > > > version
> > > > > > > > information embedded within the migration stream.  Therefore a
> > > > > > > > migration should fail early if the devices are incompatible.  
> > > > > > > > Is it  
> > > > > > > 
> > > > > > > but as I know, currently in VFIO migration protocol, we have no 
> > > > > > > way to
> > > > > > > get vendor specific compatibility checking string in migration 
> > > > > > > setup stage
> > > > > > > (i.e. .save_setup stage) before the device is set to _SAVING 
> > > > > > > state.
> > > > > > > In this way, for devices who does not save device data in precopy 
> > > > > > > stage,
> > > > > > > the migration compatibility checking is as late as in 
> > > > > > > stop-and-copy
> > > > > > > stage, which is too late.
> > > > > > > do you think we need to add the getting/checking of vendor 
> > > > > > > specific
> > > > > > > compatibility string early in save_setup stage?
> > > > > > >  
> > > > > > 
> > > > > > hi Alex,
> > > > > > after an offline discussion with Kevin, I realized that it may not 
> > > > > > be a
> > > > > > problem if migration compatibility check in vendor driver occurs 
> > > > > > late in
> > > > > > stop-and-copy phase for some devices, because if we report device
> > > > > > compatibility attributes clearly in an interface, the chances for
> > > > > > libvirt/openstack to make a wrong decision is little.
> > > > > 
> > > > > I think it would be wise for a vendor driver to implement a pre-copy
> > > > > phase, even if only to send version information and verify it at the
> > > > > target.  Deciding you have no device state to send during pre-copy 
> > > > > does
> > > > > not mean your vendor driver needs to opt-out of the pre-copy phase
> > > > > entirely.  Please also note that pre-copy is at the user's discretion,
> > > > > we've defined that we can enter stop-and-copy at any point, including
> > > > > without a pre-copy phase, so I would recommend that vendor drivers
> > > > > validate compatibility at the start of both the pre-copy and the
> > > > > stop-and-copy phases.
> > > > > 
> > > > 
> > > > ok. got it!
> > > > 
> > > > > > so, do you think we are now arriving at an agreement that we'll 
> > > > > > give up
> > > > > > the read-and-test scheme and start to defining one interface 
> > > > > > (perhaps in
> > > > > > json format), from which libvirt/openstack is able to parse and 
> > > > > > find out
> > > > > > compatibility list of a source mdev/physical device?
> > > > > 
> > > > > Based on the feedback we've received, the previously proposed 
> > > > > interface
> > > > > is not viable.  I think there's agreement that the user needs to be
> > > > > able to parse and interpret the version information.  Using json seems
> > > > > viable, but I don't know if it's the best option.  Is there any
> > > > > precedent of markup strings returned via sysfs we could follow?
> > > > 
> > > > I found some examples of using formatted string under /sys, mostly under
> > > > tracing. maybe we can do a similar implementation.
> > > > 
> > > > #cat /sys/kernel/debug/tracing/events/kvm/kvm_mmio/format
> > > > 
> > > > name: kvm_mmio
> > > > ID: 32
> > > > format:
> > > > field:unsigned short common_type;   offset:0;   size:2; 
> > > > signed:0;
> > > > field:unsigned char common_flags;   offset:2;   size:1; 
> > > > signed:0;
> > > > field:unsigned char common_preempt_count;   offset:3;   
> > > > size:1; signed:0;
> > > > field:int common_pid;   offset:4;   size:4; signed:1;
> > > > 
> > > > field:u32 type; offset:8;   size:4; signed:0;
> > > > field:u32 len;  offset:12;  size:4; signed:0;
> > > > field:u64 gpa;  offset:16;  size:8; signed:0;
> > > > field:u64 val;  offset:24;  size:8; signed:0;
> > > > 
> > > > print fmt: "mmio %s len %u gpa 0x%llx val 0x%llx", 
> > > > __print_symbolic(REC->type, { 0, "unsatisfied-read" }, { 1, "read"
> > > > }, { 2, "write" }), REC->len, REC->gpa, REC->val
> > > > 
> > > this is not json fromat and its not supper frendly to parse.  
> > > > 
> > > > #cat /sys/devices/pci:00/:00:02.0/uevent
> > > > DRIVER=vfio-pci
> > > > PCI_CLASS=3
> > > > PCI_ID=8086:591D
> > > > PCI_SUBSYS_ID=8086:2212
> > > > PCI_SLOT_NAME=:00:02.0
> > > > MODALIAS=pci:v8086d591Dsv8086sd2212bc03sc00i00
> > > > 
> > > this is ini format or conf formant 
> > > this is pretty simple to 

[ovirt-devel] Re: device compatibility interface for live migration with assigned devices

[Sean Mooney]

On Thu, 2020-07-30 at 11:41 +0800, Yan Zhao wrote:
> > > >interface_version=3
> > 
> > Not much granularity here, I prefer Sean's previous
> > .[.bugfix] scheme.
> > 
> 
> yes, .[.bugfix] scheme may be better, but I'm not sure if
> it works for a complicated scenario.
> e.g for pv_mode,
> (1) initially,  pv_mode is not supported, so it's pv_mode=none, it's 0.0.0,
> (2) then, pv_mode=ppgtt is supported, pv_mode="none+ppgtt", it's 0.1.0,
> indicating pv_mode=none can migrate to pv_mode="none+ppgtt", but not vice 
> versa.
> (3) later, pv_mode=context is also supported,
> pv_mode="none+ppgtt+context", so it's 0.2.0.
> 
> But if later, pv_mode=ppgtt is removed. pv_mode="none+context", how to
> name its version?
it would become 1.0.0
addtion of a feature is a minor version bump as its backwards compatiable.
if you dont request the new feature you dont need to use it and it can continue 
to behave like
a 0.0.0 device evne if its capably of acting as a 0.1.0 device.
when you remove a feature that is backward incompatable as any isnstance that 
was prevously not
using it would nolonger work so you have to bump the major version.
>  "none+ppgtt" (0.1.0) is not compatible to
> "none+context", but "none+ppgtt+context" (0.2.0) is compatible to
> "none+context".
> 
> Maintain such scheme is painful to vendor driver.
not really its how most software libs are version today. some use other schemes
but semantic versioning is don right is a concies and easy to consume set of 
rules
https://semver.org/ however you are right that it forcnes vendor to think about 
backwards
and forwards compatiablty with each change which for the most part is a good 
thing.
it goes hand in hand with have stable abi and api definitons to ensuring 
firmware updates and driver chagnes
dont break userspace that depend on the kernel interfaces they expose.

___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/QAV6MRSFA3S2GY4JUOXKPGVMSKRWVNUU/

[ovirt-devel] Re: device compatibility interface for live migration with assigned devices

[Sean Mooney]

On Thu, 2020-07-30 at 09:56 +0800, Yan Zhao wrote:
> On Wed, Jul 29, 2020 at 12:28:46PM +0100, Sean Mooney wrote:
> > On Wed, 2020-07-29 at 16:05 +0800, Yan Zhao wrote:
> > > On Mon, Jul 27, 2020 at 04:23:21PM -0600, Alex Williamson wrote:
> > > > On Mon, 27 Jul 2020 15:24:40 +0800
> > > > Yan Zhao  wrote:
> > > > 
> > > > > > > As you indicate, the vendor driver is responsible for checking 
> > > > > > > version
> > > > > > > information embedded within the migration stream.  Therefore a
> > > > > > > migration should fail early if the devices are incompatible.  Is 
> > > > > > > it  
> > > > > > 
> > > > > > but as I know, currently in VFIO migration protocol, we have no way 
> > > > > > to
> > > > > > get vendor specific compatibility checking string in migration 
> > > > > > setup stage
> > > > > > (i.e. .save_setup stage) before the device is set to _SAVING state.
> > > > > > In this way, for devices who does not save device data in precopy 
> > > > > > stage,
> > > > > > the migration compatibility checking is as late as in stop-and-copy
> > > > > > stage, which is too late.
> > > > > > do you think we need to add the getting/checking of vendor specific
> > > > > > compatibility string early in save_setup stage?
> > > > > >  
> > > > > 
> > > > > hi Alex,
> > > > > after an offline discussion with Kevin, I realized that it may not be 
> > > > > a
> > > > > problem if migration compatibility check in vendor driver occurs late 
> > > > > in
> > > > > stop-and-copy phase for some devices, because if we report device
> > > > > compatibility attributes clearly in an interface, the chances for
> > > > > libvirt/openstack to make a wrong decision is little.
> > > > 
> > > > I think it would be wise for a vendor driver to implement a pre-copy
> > > > phase, even if only to send version information and verify it at the
> > > > target.  Deciding you have no device state to send during pre-copy does
> > > > not mean your vendor driver needs to opt-out of the pre-copy phase
> > > > entirely.  Please also note that pre-copy is at the user's discretion,
> > > > we've defined that we can enter stop-and-copy at any point, including
> > > > without a pre-copy phase, so I would recommend that vendor drivers
> > > > validate compatibility at the start of both the pre-copy and the
> > > > stop-and-copy phases.
> > > > 
> > > 
> > > ok. got it!
> > > 
> > > > > so, do you think we are now arriving at an agreement that we'll give 
> > > > > up
> > > > > the read-and-test scheme and start to defining one interface (perhaps 
> > > > > in
> > > > > json format), from which libvirt/openstack is able to parse and find 
> > > > > out
> > > > > compatibility list of a source mdev/physical device?
> > > > 
> > > > Based on the feedback we've received, the previously proposed interface
> > > > is not viable.  I think there's agreement that the user needs to be
> > > > able to parse and interpret the version information.  Using json seems
> > > > viable, but I don't know if it's the best option.  Is there any
> > > > precedent of markup strings returned via sysfs we could follow?
> > > 
> > > I found some examples of using formatted string under /sys, mostly under
> > > tracing. maybe we can do a similar implementation.
> > > 
> > > #cat /sys/kernel/debug/tracing/events/kvm/kvm_mmio/format
> > > 
> > > name: kvm_mmio
> > > ID: 32
> > > format:
> > > field:unsigned short common_type;   offset:0;   size:2; 
> > > signed:0;
> > > field:unsigned char common_flags;   offset:2;   size:1; 
> > > signed:0;
> > > field:unsigned char common_preempt_count;   offset:3;   
> > > size:1; signed:0;
> > > field:int common_pid;   offset:4;   size:4; signed:1;
> > > 
> > > field:u32 type; offset:8;   size:4; signed:0;
> > > field:u32 len;  offset:12;  size:4; signed:0;
> > > field:u64 gpa;  offset:16;  size:8; signed:0;
> > > field:u64 val;  offset:24;  size:8; signed:0;
> > > 
> > > print fmt: "mmio %s len %u gpa 0x%llx val 0x%llx", 
> > > __print_symbolic(REC->type, { 0, "unsatisfied-read" }, { 1,
> > > "read"
> > > }, { 2, "write" }), REC->len, REC->gpa, REC->val
> > > 
> > 
> > this is not json fromat and its not supper frendly to parse.
> 
> yes, it's just an example. It's exported to be used by userspace perf &
> trace_cmd.
> 
> > > 
> > > #cat /sys/devices/pci:00/:00:02.0/uevent
> > > DRIVER=vfio-pci
> > > PCI_CLASS=3
> > > PCI_ID=8086:591D
> > > PCI_SUBSYS_ID=8086:2212
> > > PCI_SLOT_NAME=:00:02.0
> > > MODALIAS=pci:v8086d591Dsv8086sd2212bc03sc00i00
> > > 
> > 
> > this is ini format or conf formant 
> > this is pretty simple to parse whichi would be fine.
> > that said you could also have a version or capablitiy directory with a file
> > for each key and a singel value.
> > 
> 
> if this is easy for openstack, maybe we can organize the data like below way?
>  
>  |- [device]
> |- migration
>

[ovirt-devel] Re: please i'm facing a with making el8

[Sandro Bonazzola]

Hi,
nice to see students working on oVirt for their final college project!
May I kindly ask more details about your project and what are you trying to
achieve by rebuilding oVirt Node?

Il giorno mer 29 lug 2020 alle ore 10:50 Amit Chaudhary <
amit.chaudha...@galgotiasuniversity.edu.in> ha scritto:

>
>  ovirt.zip
> 
> i have attached my ovirt zip. the issue im facing is that the kickstart is
> not generated for el8 and im not able to figure out how to use .j2 file
> that is being generated. i've manage to frankenstein a kickstart file that
> is in the zip with all the settings but its also giving error while
> compressing (dracut module 'livenet' cannot be found or installed) while if
> i search installed modules of dracut i can see livenet is installed. see
> see my zip it have logs of last compilation also.
>
>
> sir/mam it would be a great help for my college final year project project
> on hci and my research paper.
>
> please sir/mam please help me
>
> Amit Chaudhary
> +919650596355
>
> *CONFIDENTIALITY NOTICE*
>
>This communication (including any attachment/s) is intended only
> for the use of the addressee(s) and contains information that is PRIVILEGED
> AND CONFIDENTIAL. Unauthorized reading, dissemination, distribution, or
> copying of this communication is prohibited.
>
>   * DISCLAIMER*
>
>Any views or opinions presented in this email are solely those of
> the author and do not necessarily represent those of the University.
>
> WARNING:-
>
>This communication, including any attachments, may not be free of
> viruses, interceptions or interference, and may not be compatible with your
> systems. You should carry out your own virus checks before opening any
> attachment to this e-mail. The sender of this e-mail and Galgotias
> University shall not be liable for any damage that you may sustain as a
> result of viruses, incompleteness of this message, a delay in receipt of
> this message or computer problems experienced.
> ___
> Devel mailing list -- devel@ovirt.org
> To unsubscribe send an email to devel-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/3Y2FTAC6WBMQKKKZBAC6UEG5CIOLUBNT/
>


-- 

Sandro Bonazzola

MANAGER, SOFTWARE ENGINEERING, EMEA R RHV

Red Hat EMEA 

sbona...@redhat.com


*Red Hat respects your work life balance. Therefore there is no need to
answer this email out of your office hours.
*
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/E5TST2SEYQE4HSIRDJQEI4ELJRAVFYNV/

[ovirt-devel] Re: OST switching back to CirrOS

[Michal Skrivanek]

> On 30 Jul 2020, at 12:13, Yedidyah Bar David  wrote:
> 
> On Wed, Jul 29, 2020 at 7:09 PM Michal Skrivanek
>  wrote:
>> 
>> Hi,
>> FYI we’ve just switched back from CentOS to CirrOS VMs in OST basic suite, 
>> saving 900MB of data transferred all the time (and copied back and forth 
>> during test) as well as lower memory requirements (down to 128MB from 384MB, 
>> times 3 VMs).
>> For a good measure a memory hot unplug test was added too.
>> 
>> Let me know if you see anything unusual, other than faster run times.
> 
> Several suites are broken right now, e.g.:
> 
> https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1691/testReport/junit/(root)/004_basic_sanity/vm_run/
> 
> Due to missing function get_vm0_ip_address, which you removed in your
> patch. Is there a problem to restore this function (even if basic
> suite does not use it anymore), or it won't work as-is? Perhaps it

it will work, it’s just the convoluted linking of files between suites…

> won't work if moving to cirros also other suites? Trying, anyway:
> 
> https://gerrit.ovirt.org/110559

I noticed too, sorry, forgot to add you to review of 
https://gerrit.ovirt.org/#/c/110554/
i’ll merge it now

> 
> Best regards,
> -- 
> Didi
> 
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/LEIP73YZIHH5MS6O4ATPFRVOXBAUJDBW/

[ovirt-devel] Re: OST switching back to CirrOS

[Yedidyah Bar David]

On Wed, Jul 29, 2020 at 7:09 PM Michal Skrivanek
 wrote:
>
> Hi,
> FYI we’ve just switched back from CentOS to CirrOS VMs in OST basic suite, 
> saving 900MB of data transferred all the time (and copied back and forth 
> during test) as well as lower memory requirements (down to 128MB from 384MB, 
> times 3 VMs).
> For a good measure a memory hot unplug test was added too.
>
> Let me know if you see anything unusual, other than faster run times.

Several suites are broken right now, e.g.:

https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1691/testReport/junit/(root)/004_basic_sanity/vm_run/

Due to missing function get_vm0_ip_address, which you removed in your
patch. Is there a problem to restore this function (even if basic
suite does not use it anymore), or it won't work as-is? Perhaps it
won't work if moving to cirros also other suites? Trying, anyway:

https://gerrit.ovirt.org/110559

Best regards,
-- 
Didi
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/V4O6DP3QGVF3T3W35TRGAKKVAFQ4JXWX/

[ovirt-devel] Re: Error while executing action New NFS Storage Domain: Unexpected exception

[Nir Soffer]

On Thu, Jul 30, 2020, 10:57 Dana Elfassy  wrote:

> Thanks Nir,
> I just checked again, it still happens
> This is the original error:
>
> 2020-07-30 03:53:26,071-0400 ERROR (jsonrpc/4) [storage.TaskManager.Task]
> (Task='2904a936-d6ca-464e-9c86-973c9fbb9b78') Unexpected error (task:880)
> Traceback (most recent call last):
>   File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py",
> line 250, in initSANLock
> lockspace_name, idsPath, align=alignment, sector=block_size)
> sanlock.SanlockException: (-202, 'Sanlock lockspace write failure', 'IO
> timeout')
>

So your storage was too slow, and sanlock could not write 1MiB to storage
in 10 seconds.

You can delete the partly created domain and try again.

In 8.3 sanlock will use 60 seconds timeout for this kind of operations, so
such failure will be less likely.

But this storage failure means your storage is not healthy and you will
have more trouble.


> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
>   File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 887,
> in _run
> return fn(*args, **kargs)
>   File "", line 2, in createStorageDomain
>   File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in
> method
> ret = func(*args, **kwargs)
>   File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2644,
> in createStorageDomain
> max_hosts=max_hosts)
>   File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 120,
> in create
> fsd.initSPMlease()
>   File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 999, in
> initSPMlease
> return self._manifest.initDomainLock()
>   File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 627, in
> initDomainLock
> self._domainLock.initLock(self.getDomainLease())
>   File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py",
> line 308, in initLock
> block_size=self._block_size)
>   File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py",
> line 260, in initSANLock
> raise se.ClusterLockInitError()
> vdsm.storage.exception.ClusterLockInitError: Could not initialize cluster
> lock: ()
> 2020-07-30 03:53:26,071-0400 INFO  (jsonrpc/4) [storage.TaskManager.Task]
> (Task='2904a936-d6ca-464e-9c86-973c9fbb9b78') aborting: Task is aborted:
> 'value=Could not initialize cluster lock: () abortedcode=701' (task:1190)
> 2020-07-30 03:53:26,071-0400 ERROR (jsonrpc/4) [storage.Dispatcher] FINISH
> createStorageDomain error=Could not initialize cluster lock: ()
> (dispatcher:83)
>
> [delfassy@delfassy mnt]$ ls -lhZ
> drwxrwxrwx. 3 vdsm kvm  system_u:object_r:nfs_t:s0 4.0K Jul 30
> 10:53 new_mnt
>
> On Wed, Jul 29, 2020 at 12:30 PM Nir Soffer  wrote:
>
>>
>>
>> On Wed, Jul 29, 2020, 12:09 Dana Elfassy  wrote:
>>
>>> Hi,
>>> When I'm trying to add a storage domain via the webadmin I'm getting the
>>> error message that's on the subject.
>>> in the vdsm.log I see this error:
>>> 2020-07-29 04:53:48,045-0400 ERROR (check/loop) [storage.Monitor] Error
>>> checking path 
>>> /rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata
>>> (monitor:525)
>>> Traceback (most recent call last):
>>>   File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line
>>> 523, in _pathChecked
>>> delay = result.delay()
>>>   File "/usr/lib/python3.6/site-packages/vdsm/storage/check.py", line
>>> 391, in delay
>>> raise exception.MiscFileReadException(self.path, self.rc, self.err)
>>> vdsm.storage.exception.MiscFileReadException: Internal file read
>>> failure: 
>>> ('/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata',
>>> 1, bytearray(b"/usr/bin/dd: failed to open
>>> \'/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata\':
>>> No such file or directory\n"))
>>>
>>
>> This means the metadata file was not accessible when the monitor tried to
>> read.
>>
>> Does it happen now?
>>
>> Anyway this is not the unexpected error that failed the operation.
>>
>> Can you share vdsm log showing the original error?
>>
>>
>>> when looking at the directory I do see that metadata was created under
>>> this directory, with the following content:
>>> [delfassy@delfassy mnt]$ cd
>>> new_nfs/cddff7bb-bb0a-4fc8-8ba4-023be91d822d/dom_md/
>>> [delfassy@delfassy dom_md]$ less metadata
>>> CLASS=Data
>>> DESCRIPTION=spider
>>> IOOPTIMEOUTSEC=10
>>> LEASERETRIES=3
>>> LEASETIMESEC=60
>>> LOCKPOLICY=
>>> LOCKRENEWALINTERVALSEC=5
>>> POOL_UUID=
>>> REMOTE_PATH=vserver-spider.eng.lab.tlv.redhat.com:/pub/delfassy/new_nfs
>>> ROLE=Regular
>>> SDUUID=cddff7bb-bb0a-4fc8-8ba4-023be91d822d
>>> TYPE=NFS
>>> VERSION=4
>>> _SHA_CKSUM=f4953016c94f205127b00a5b10f0e7d5d9cf24f7
>>>
>>
>> Looks correct
>>
>>
>>> The permissions on 

[ovirt-devel] Re: Error while executing action New NFS Storage Domain: Unexpected exception

[Dana Elfassy]

Thanks Nir,
I just checked again, it still happens
This is the original error:

2020-07-30 03:53:26,071-0400 ERROR (jsonrpc/4) [storage.TaskManager.Task]
(Task='2904a936-d6ca-464e-9c86-973c9fbb9b78') Unexpected error (task:880)
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line
250, in initSANLock
lockspace_name, idsPath, align=alignment, sector=block_size)
sanlock.SanlockException: (-202, 'Sanlock lockspace write failure', 'IO
timeout')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 887,
in _run
return fn(*args, **kargs)
  File "", line 2, in createStorageDomain
  File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in
method
ret = func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2644,
in createStorageDomain
max_hosts=max_hosts)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 120,
in create
fsd.initSPMlease()
  File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 999, in
initSPMlease
return self._manifest.initDomainLock()
  File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 627, in
initDomainLock
self._domainLock.initLock(self.getDomainLease())
  File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line
308, in initLock
block_size=self._block_size)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line
260, in initSANLock
raise se.ClusterLockInitError()
vdsm.storage.exception.ClusterLockInitError: Could not initialize cluster
lock: ()
2020-07-30 03:53:26,071-0400 INFO  (jsonrpc/4) [storage.TaskManager.Task]
(Task='2904a936-d6ca-464e-9c86-973c9fbb9b78') aborting: Task is aborted:
'value=Could not initialize cluster lock: () abortedcode=701' (task:1190)
2020-07-30 03:53:26,071-0400 ERROR (jsonrpc/4) [storage.Dispatcher] FINISH
createStorageDomain error=Could not initialize cluster lock: ()
(dispatcher:83)

[delfassy@delfassy mnt]$ ls -lhZ
drwxrwxrwx. 3 vdsm kvm  system_u:object_r:nfs_t:s0 4.0K Jul 30
10:53 new_mnt

On Wed, Jul 29, 2020 at 12:30 PM Nir Soffer  wrote:

>
>
> On Wed, Jul 29, 2020, 12:09 Dana Elfassy  wrote:
>
>> Hi,
>> When I'm trying to add a storage domain via the webadmin I'm getting the
>> error message that's on the subject.
>> in the vdsm.log I see this error:
>> 2020-07-29 04:53:48,045-0400 ERROR (check/loop) [storage.Monitor] Error
>> checking path 
>> /rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata
>> (monitor:525)
>> Traceback (most recent call last):
>>   File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line
>> 523, in _pathChecked
>> delay = result.delay()
>>   File "/usr/lib/python3.6/site-packages/vdsm/storage/check.py", line
>> 391, in delay
>> raise exception.MiscFileReadException(self.path, self.rc, self.err)
>> vdsm.storage.exception.MiscFileReadException: Internal file read failure:
>> ('/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata',
>> 1, bytearray(b"/usr/bin/dd: failed to open
>> \'/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata\':
>> No such file or directory\n"))
>>
>
> This means the metadata file was not accessible when the monitor tried to
> read.
>
> Does it happen now?
>
> Anyway this is not the unexpected error that failed the operation.
>
> Can you share vdsm log showing the original error?
>
>
>> when looking at the directory I do see that metadata was created under
>> this directory, with the following content:
>> [delfassy@delfassy mnt]$ cd
>> new_nfs/cddff7bb-bb0a-4fc8-8ba4-023be91d822d/dom_md/
>> [delfassy@delfassy dom_md]$ less metadata
>> CLASS=Data
>> DESCRIPTION=spider
>> IOOPTIMEOUTSEC=10
>> LEASERETRIES=3
>> LEASETIMESEC=60
>> LOCKPOLICY=
>> LOCKRENEWALINTERVALSEC=5
>> POOL_UUID=
>> REMOTE_PATH=vserver-spider.eng.lab.tlv.redhat.com:/pub/delfassy/new_nfs
>> ROLE=Regular
>> SDUUID=cddff7bb-bb0a-4fc8-8ba4-023be91d822d
>> TYPE=NFS
>> VERSION=4
>> _SHA_CKSUM=f4953016c94f205127b00a5b10f0e7d5d9cf24f7
>>
>
> Looks correct
>
>
>> The permissions on the directory are:
>> drwxrwxrwx. 3 vdsm kvm  4096 Jul 29 11:49 new_mnt
>>
>
> You don't need 0777, 0750 should work.
>
> ls -lhZ?
>
>
>> Are there any other configurations I'm missing? Any idea how this issue
>> can be resolved?
>> Thanks,
>> Dana
>> ___
>> Devel mailing list -- devel@ovirt.org
>> To unsubscribe send an email to devel-le...@ovirt.org
>> Privacy Statement: https://www.ovirt.org/privacy-policy.html
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> List 

[ovirt-devel] Re: device compatibility interface for live migration with assigned devices

[Yan Zhao]

On Wed, Jul 29, 2020 at 01:12:55PM -0600, Alex Williamson wrote:
> On Wed, 29 Jul 2020 12:28:46 +0100
> Sean Mooney  wrote:
> 
> > On Wed, 2020-07-29 at 16:05 +0800, Yan Zhao wrote:
> > > On Mon, Jul 27, 2020 at 04:23:21PM -0600, Alex Williamson wrote:  
> > > > On Mon, 27 Jul 2020 15:24:40 +0800
> > > > Yan Zhao  wrote:
> > > >   
> > > > > > > As you indicate, the vendor driver is responsible for checking 
> > > > > > > version
> > > > > > > information embedded within the migration stream.  Therefore a
> > > > > > > migration should fail early if the devices are incompatible.  Is 
> > > > > > > it
> > > > > > 
> > > > > > but as I know, currently in VFIO migration protocol, we have no way 
> > > > > > to
> > > > > > get vendor specific compatibility checking string in migration 
> > > > > > setup stage
> > > > > > (i.e. .save_setup stage) before the device is set to _SAVING state.
> > > > > > In this way, for devices who does not save device data in precopy 
> > > > > > stage,
> > > > > > the migration compatibility checking is as late as in stop-and-copy
> > > > > > stage, which is too late.
> > > > > > do you think we need to add the getting/checking of vendor specific
> > > > > > compatibility string early in save_setup stage?
> > > > > >
> > > > > 
> > > > > hi Alex,
> > > > > after an offline discussion with Kevin, I realized that it may not be 
> > > > > a
> > > > > problem if migration compatibility check in vendor driver occurs late 
> > > > > in
> > > > > stop-and-copy phase for some devices, because if we report device
> > > > > compatibility attributes clearly in an interface, the chances for
> > > > > libvirt/openstack to make a wrong decision is little.  
> > > > 
> > > > I think it would be wise for a vendor driver to implement a pre-copy
> > > > phase, even if only to send version information and verify it at the
> > > > target.  Deciding you have no device state to send during pre-copy does
> > > > not mean your vendor driver needs to opt-out of the pre-copy phase
> > > > entirely.  Please also note that pre-copy is at the user's discretion,
> > > > we've defined that we can enter stop-and-copy at any point, including
> > > > without a pre-copy phase, so I would recommend that vendor drivers
> > > > validate compatibility at the start of both the pre-copy and the
> > > > stop-and-copy phases.
> > > >   
> > > 
> > > ok. got it!
> > >   
> > > > > so, do you think we are now arriving at an agreement that we'll give 
> > > > > up
> > > > > the read-and-test scheme and start to defining one interface (perhaps 
> > > > > in
> > > > > json format), from which libvirt/openstack is able to parse and find 
> > > > > out
> > > > > compatibility list of a source mdev/physical device?  
> > > > 
> > > > Based on the feedback we've received, the previously proposed interface
> > > > is not viable.  I think there's agreement that the user needs to be
> > > > able to parse and interpret the version information.  Using json seems
> > > > viable, but I don't know if it's the best option.  Is there any
> > > > precedent of markup strings returned via sysfs we could follow?  
> > > 
> > > I found some examples of using formatted string under /sys, mostly under
> > > tracing. maybe we can do a similar implementation.
> > > 
> > > #cat /sys/kernel/debug/tracing/events/kvm/kvm_mmio/format
> > > 
> > > name: kvm_mmio
> > > ID: 32
> > > format:
> > > field:unsigned short common_type;   offset:0;   size:2; 
> > > signed:0;
> > > field:unsigned char common_flags;   offset:2;   size:1; 
> > > signed:0;
> > > field:unsigned char common_preempt_count;   offset:3;   
> > > size:1; signed:0;
> > > field:int common_pid;   offset:4;   size:4; signed:1;
> > > 
> > > field:u32 type; offset:8;   size:4; signed:0;
> > > field:u32 len;  offset:12;  size:4; signed:0;
> > > field:u64 gpa;  offset:16;  size:8; signed:0;
> > > field:u64 val;  offset:24;  size:8; signed:0;
> > > 
> > > print fmt: "mmio %s len %u gpa 0x%llx val 0x%llx", 
> > > __print_symbolic(REC->type, { 0, "unsatisfied-read" }, { 1, "read"
> > > }, { 2, "write" }), REC->len, REC->gpa, REC->val
> > >   
> > this is not json fromat and its not supper frendly to parse.
> > > 
> > > #cat /sys/devices/pci:00/:00:02.0/uevent
> > > DRIVER=vfio-pci
> > > PCI_CLASS=3
> > > PCI_ID=8086:591D
> > > PCI_SUBSYS_ID=8086:2212
> > > PCI_SLOT_NAME=:00:02.0
> > > MODALIAS=pci:v8086d591Dsv8086sd2212bc03sc00i00
> > >   
> > this is ini format or conf formant 
> > this is pretty simple to parse whichi would be fine.
> > that said you could also have a version or capablitiy directory with a file
> > for each key and a singel value.
> > 
> > i would prefer to only have to do one read personally the list the files in
> > directory and then read tehm all ot build the datastucture myself but that 
> > is
> > doable though the simple ini 

[ovirt-devel] Re: device compatibility interface for live migration with assigned devices

[Yan Zhao]

On Wed, Jul 29, 2020 at 12:28:46PM +0100, Sean Mooney wrote:
> On Wed, 2020-07-29 at 16:05 +0800, Yan Zhao wrote:
> > On Mon, Jul 27, 2020 at 04:23:21PM -0600, Alex Williamson wrote:
> > > On Mon, 27 Jul 2020 15:24:40 +0800
> > > Yan Zhao  wrote:
> > > 
> > > > > > As you indicate, the vendor driver is responsible for checking 
> > > > > > version
> > > > > > information embedded within the migration stream.  Therefore a
> > > > > > migration should fail early if the devices are incompatible.  Is it 
> > > > > >  
> > > > > 
> > > > > but as I know, currently in VFIO migration protocol, we have no way to
> > > > > get vendor specific compatibility checking string in migration setup 
> > > > > stage
> > > > > (i.e. .save_setup stage) before the device is set to _SAVING state.
> > > > > In this way, for devices who does not save device data in precopy 
> > > > > stage,
> > > > > the migration compatibility checking is as late as in stop-and-copy
> > > > > stage, which is too late.
> > > > > do you think we need to add the getting/checking of vendor specific
> > > > > compatibility string early in save_setup stage?
> > > > >  
> > > > 
> > > > hi Alex,
> > > > after an offline discussion with Kevin, I realized that it may not be a
> > > > problem if migration compatibility check in vendor driver occurs late in
> > > > stop-and-copy phase for some devices, because if we report device
> > > > compatibility attributes clearly in an interface, the chances for
> > > > libvirt/openstack to make a wrong decision is little.
> > > 
> > > I think it would be wise for a vendor driver to implement a pre-copy
> > > phase, even if only to send version information and verify it at the
> > > target.  Deciding you have no device state to send during pre-copy does
> > > not mean your vendor driver needs to opt-out of the pre-copy phase
> > > entirely.  Please also note that pre-copy is at the user's discretion,
> > > we've defined that we can enter stop-and-copy at any point, including
> > > without a pre-copy phase, so I would recommend that vendor drivers
> > > validate compatibility at the start of both the pre-copy and the
> > > stop-and-copy phases.
> > > 
> > 
> > ok. got it!
> > 
> > > > so, do you think we are now arriving at an agreement that we'll give up
> > > > the read-and-test scheme and start to defining one interface (perhaps in
> > > > json format), from which libvirt/openstack is able to parse and find out
> > > > compatibility list of a source mdev/physical device?
> > > 
> > > Based on the feedback we've received, the previously proposed interface
> > > is not viable.  I think there's agreement that the user needs to be
> > > able to parse and interpret the version information.  Using json seems
> > > viable, but I don't know if it's the best option.  Is there any
> > > precedent of markup strings returned via sysfs we could follow?
> > 
> > I found some examples of using formatted string under /sys, mostly under
> > tracing. maybe we can do a similar implementation.
> > 
> > #cat /sys/kernel/debug/tracing/events/kvm/kvm_mmio/format
> > 
> > name: kvm_mmio
> > ID: 32
> > format:
> > field:unsigned short common_type;   offset:0;   size:2; 
> > signed:0;
> > field:unsigned char common_flags;   offset:2;   size:1; 
> > signed:0;
> > field:unsigned char common_preempt_count;   offset:3;   
> > size:1; signed:0;
> > field:int common_pid;   offset:4;   size:4; signed:1;
> > 
> > field:u32 type; offset:8;   size:4; signed:0;
> > field:u32 len;  offset:12;  size:4; signed:0;
> > field:u64 gpa;  offset:16;  size:8; signed:0;
> > field:u64 val;  offset:24;  size:8; signed:0;
> > 
> > print fmt: "mmio %s len %u gpa 0x%llx val 0x%llx", 
> > __print_symbolic(REC->type, { 0, "unsatisfied-read" }, { 1, "read"
> > }, { 2, "write" }), REC->len, REC->gpa, REC->val
> > 
> this is not json fromat and its not supper frendly to parse.
yes, it's just an example. It's exported to be used by userspace perf &
trace_cmd.

> > 
> > #cat /sys/devices/pci:00/:00:02.0/uevent
> > DRIVER=vfio-pci
> > PCI_CLASS=3
> > PCI_ID=8086:591D
> > PCI_SUBSYS_ID=8086:2212
> > PCI_SLOT_NAME=:00:02.0
> > MODALIAS=pci:v8086d591Dsv8086sd2212bc03sc00i00
> > 
> this is ini format or conf formant 
> this is pretty simple to parse whichi would be fine.
> that said you could also have a version or capablitiy directory with a file
> for each key and a singel value.
> 
if this is easy for openstack, maybe we can organize the data like below way?
 
 |- [device]
|- migration
|-self
|-compatible1
|-compatible2

e.g. 
#cat /sys/bus/pci/devices/:00:02.0/UUID1/migration/self
filed1=xxx
filed2=xxx
filed3=xxx
filed3=xxx
#cat /sys/bus/pci/devices/:00:02.0/UUID1/migration/compatible
filed1=xxx
filed2=xxx
filed3=xxx
filed3=xxx

or in a 

[ovirt-devel] Re: device compatibility interface for live migration with assigned devices

[Alex Williamson]

On Wed, 29 Jul 2020 12:28:46 +0100
Sean Mooney  wrote:

> On Wed, 2020-07-29 at 16:05 +0800, Yan Zhao wrote:
> > On Mon, Jul 27, 2020 at 04:23:21PM -0600, Alex Williamson wrote:  
> > > On Mon, 27 Jul 2020 15:24:40 +0800
> > > Yan Zhao  wrote:
> > >   
> > > > > > As you indicate, the vendor driver is responsible for checking 
> > > > > > version
> > > > > > information embedded within the migration stream.  Therefore a
> > > > > > migration should fail early if the devices are incompatible.  Is it 
> > > > > >
> > > > > 
> > > > > but as I know, currently in VFIO migration protocol, we have no way to
> > > > > get vendor specific compatibility checking string in migration setup 
> > > > > stage
> > > > > (i.e. .save_setup stage) before the device is set to _SAVING state.
> > > > > In this way, for devices who does not save device data in precopy 
> > > > > stage,
> > > > > the migration compatibility checking is as late as in stop-and-copy
> > > > > stage, which is too late.
> > > > > do you think we need to add the getting/checking of vendor specific
> > > > > compatibility string early in save_setup stage?
> > > > >
> > > > 
> > > > hi Alex,
> > > > after an offline discussion with Kevin, I realized that it may not be a
> > > > problem if migration compatibility check in vendor driver occurs late in
> > > > stop-and-copy phase for some devices, because if we report device
> > > > compatibility attributes clearly in an interface, the chances for
> > > > libvirt/openstack to make a wrong decision is little.  
> > > 
> > > I think it would be wise for a vendor driver to implement a pre-copy
> > > phase, even if only to send version information and verify it at the
> > > target.  Deciding you have no device state to send during pre-copy does
> > > not mean your vendor driver needs to opt-out of the pre-copy phase
> > > entirely.  Please also note that pre-copy is at the user's discretion,
> > > we've defined that we can enter stop-and-copy at any point, including
> > > without a pre-copy phase, so I would recommend that vendor drivers
> > > validate compatibility at the start of both the pre-copy and the
> > > stop-and-copy phases.
> > >   
> > 
> > ok. got it!
> >   
> > > > so, do you think we are now arriving at an agreement that we'll give up
> > > > the read-and-test scheme and start to defining one interface (perhaps in
> > > > json format), from which libvirt/openstack is able to parse and find out
> > > > compatibility list of a source mdev/physical device?  
> > > 
> > > Based on the feedback we've received, the previously proposed interface
> > > is not viable.  I think there's agreement that the user needs to be
> > > able to parse and interpret the version information.  Using json seems
> > > viable, but I don't know if it's the best option.  Is there any
> > > precedent of markup strings returned via sysfs we could follow?  
> > 
> > I found some examples of using formatted string under /sys, mostly under
> > tracing. maybe we can do a similar implementation.
> > 
> > #cat /sys/kernel/debug/tracing/events/kvm/kvm_mmio/format
> > 
> > name: kvm_mmio
> > ID: 32
> > format:
> > field:unsigned short common_type;   offset:0;   size:2; 
> > signed:0;
> > field:unsigned char common_flags;   offset:2;   size:1; 
> > signed:0;
> > field:unsigned char common_preempt_count;   offset:3;   
> > size:1; signed:0;
> > field:int common_pid;   offset:4;   size:4; signed:1;
> > 
> > field:u32 type; offset:8;   size:4; signed:0;
> > field:u32 len;  offset:12;  size:4; signed:0;
> > field:u64 gpa;  offset:16;  size:8; signed:0;
> > field:u64 val;  offset:24;  size:8; signed:0;
> > 
> > print fmt: "mmio %s len %u gpa 0x%llx val 0x%llx", 
> > __print_symbolic(REC->type, { 0, "unsatisfied-read" }, { 1, "read"
> > }, { 2, "write" }), REC->len, REC->gpa, REC->val
> >   
> this is not json fromat and its not supper frendly to parse.
> > 
> > #cat /sys/devices/pci:00/:00:02.0/uevent
> > DRIVER=vfio-pci
> > PCI_CLASS=3
> > PCI_ID=8086:591D
> > PCI_SUBSYS_ID=8086:2212
> > PCI_SLOT_NAME=:00:02.0
> > MODALIAS=pci:v8086d591Dsv8086sd2212bc03sc00i00
> >   
> this is ini format or conf formant 
> this is pretty simple to parse whichi would be fine.
> that said you could also have a version or capablitiy directory with a file
> for each key and a singel value.
> 
> i would prefer to only have to do one read personally the list the files in
> directory and then read tehm all ot build the datastucture myself but that is
> doable though the simple ini format use d for uevent seams the best of 3 
> options
> provided above.
> > > 
> > > Your idea of having both a "self" object and an array of "compatible"
> > > objects is perhaps something we can build on, but we must not assume
> > > PCI devices at the root level of the object.  Providing both the
> > > mdev-type and the driver is a bit 

[ovirt-devel] Re: device compatibility interface for live migration with assigned devices

[Dr. David Alan Gilbert]

* Alex Williamson (alex.william...@redhat.com) wrote:
> On Mon, 27 Jul 2020 15:24:40 +0800
> Yan Zhao  wrote:
> 
> > > > As you indicate, the vendor driver is responsible for checking version
> > > > information embedded within the migration stream.  Therefore a
> > > > migration should fail early if the devices are incompatible.  Is it  
> > > but as I know, currently in VFIO migration protocol, we have no way to
> > > get vendor specific compatibility checking string in migration setup stage
> > > (i.e. .save_setup stage) before the device is set to _SAVING state.
> > > In this way, for devices who does not save device data in precopy stage,
> > > the migration compatibility checking is as late as in stop-and-copy
> > > stage, which is too late.
> > > do you think we need to add the getting/checking of vendor specific
> > > compatibility string early in save_setup stage?
> > >  
> > hi Alex,
> > after an offline discussion with Kevin, I realized that it may not be a
> > problem if migration compatibility check in vendor driver occurs late in
> > stop-and-copy phase for some devices, because if we report device
> > compatibility attributes clearly in an interface, the chances for
> > libvirt/openstack to make a wrong decision is little.
> 
> I think it would be wise for a vendor driver to implement a pre-copy
> phase, even if only to send version information and verify it at the
> target.  Deciding you have no device state to send during pre-copy does
> not mean your vendor driver needs to opt-out of the pre-copy phase
> entirely.  Please also note that pre-copy is at the user's discretion,
> we've defined that we can enter stop-and-copy at any point, including
> without a pre-copy phase, so I would recommend that vendor drivers
> validate compatibility at the start of both the pre-copy and the
> stop-and-copy phases.

That's quite curious; from a migration point of view I'd expect if you
did want to skip pre-copy, that you'd go through the motions of entering
it and then not saving any data and then going to stop-and-copy,
rather than having two flows.

Note that failing at a late stage of stop-and-copy is a pain; if you've
just spent an hour migrating your huge busy VM over, you're going to be
pretty annoyed when it goes pop near the end.

Dave

> > so, do you think we are now arriving at an agreement that we'll give up
> > the read-and-test scheme and start to defining one interface (perhaps in
> > json format), from which libvirt/openstack is able to parse and find out
> > compatibility list of a source mdev/physical device?
> 
> Based on the feedback we've received, the previously proposed interface
> is not viable.  I think there's agreement that the user needs to be
> able to parse and interpret the version information.  Using json seems
> viable, but I don't know if it's the best option.  Is there any
> precedent of markup strings returned via sysfs we could follow?
> 
> Your idea of having both a "self" object and an array of "compatible"
> objects is perhaps something we can build on, but we must not assume
> PCI devices at the root level of the object.  Providing both the
> mdev-type and the driver is a bit redundant, since the former includes
> the latter.  We can't have vendor specific versioning schemes though,
> ie. gvt-version. We need to agree on a common scheme and decide which
> fields the version is relative to, ex. just the mdev type?
> 
> I had also proposed fields that provide information to create a
> compatible type, for example to create a type_x2 device from a type_x1
> mdev type, they need to know to apply an aggregation attribute.  If we
> need to explicitly list every aggregation value and the resulting type,
> I think we run aground of what aggregation was trying to avoid anyway,
> so we might need to pick a language that defines variable substitution
> or some kind of tagging.  For example if we could define ${aggr} as an
> integer within a specified range, then we might be able to define a type
> relative to that value (type_x${aggr}) which requires an aggregation
> attribute using the same value.  I dunno, just spit balling.  Thanks,
> 
> Alex
--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/NVBPQWOBRFERSVPFWJN6NYCHB2OLYU3C/

[ovirt-devel] Re: device compatibility interface for live migration with assigned devices

[Sean Mooney]

On Wed, 2020-07-29 at 16:05 +0800, Yan Zhao wrote:
> On Mon, Jul 27, 2020 at 04:23:21PM -0600, Alex Williamson wrote:
> > On Mon, 27 Jul 2020 15:24:40 +0800
> > Yan Zhao  wrote:
> > 
> > > > > As you indicate, the vendor driver is responsible for checking version
> > > > > information embedded within the migration stream.  Therefore a
> > > > > migration should fail early if the devices are incompatible.  Is it  
> > > > 
> > > > but as I know, currently in VFIO migration protocol, we have no way to
> > > > get vendor specific compatibility checking string in migration setup 
> > > > stage
> > > > (i.e. .save_setup stage) before the device is set to _SAVING state.
> > > > In this way, for devices who does not save device data in precopy stage,
> > > > the migration compatibility checking is as late as in stop-and-copy
> > > > stage, which is too late.
> > > > do you think we need to add the getting/checking of vendor specific
> > > > compatibility string early in save_setup stage?
> > > >  
> > > 
> > > hi Alex,
> > > after an offline discussion with Kevin, I realized that it may not be a
> > > problem if migration compatibility check in vendor driver occurs late in
> > > stop-and-copy phase for some devices, because if we report device
> > > compatibility attributes clearly in an interface, the chances for
> > > libvirt/openstack to make a wrong decision is little.
> > 
> > I think it would be wise for a vendor driver to implement a pre-copy
> > phase, even if only to send version information and verify it at the
> > target.  Deciding you have no device state to send during pre-copy does
> > not mean your vendor driver needs to opt-out of the pre-copy phase
> > entirely.  Please also note that pre-copy is at the user's discretion,
> > we've defined that we can enter stop-and-copy at any point, including
> > without a pre-copy phase, so I would recommend that vendor drivers
> > validate compatibility at the start of both the pre-copy and the
> > stop-and-copy phases.
> > 
> 
> ok. got it!
> 
> > > so, do you think we are now arriving at an agreement that we'll give up
> > > the read-and-test scheme and start to defining one interface (perhaps in
> > > json format), from which libvirt/openstack is able to parse and find out
> > > compatibility list of a source mdev/physical device?
> > 
> > Based on the feedback we've received, the previously proposed interface
> > is not viable.  I think there's agreement that the user needs to be
> > able to parse and interpret the version information.  Using json seems
> > viable, but I don't know if it's the best option.  Is there any
> > precedent of markup strings returned via sysfs we could follow?
> 
> I found some examples of using formatted string under /sys, mostly under
> tracing. maybe we can do a similar implementation.
> 
> #cat /sys/kernel/debug/tracing/events/kvm/kvm_mmio/format
> 
> name: kvm_mmio
> ID: 32
> format:
> field:unsigned short common_type;   offset:0;   size:2; 
> signed:0;
> field:unsigned char common_flags;   offset:2;   size:1; 
> signed:0;
> field:unsigned char common_preempt_count;   offset:3;   
> size:1; signed:0;
> field:int common_pid;   offset:4;   size:4; signed:1;
> 
> field:u32 type; offset:8;   size:4; signed:0;
> field:u32 len;  offset:12;  size:4; signed:0;
> field:u64 gpa;  offset:16;  size:8; signed:0;
> field:u64 val;  offset:24;  size:8; signed:0;
> 
> print fmt: "mmio %s len %u gpa 0x%llx val 0x%llx", 
> __print_symbolic(REC->type, { 0, "unsatisfied-read" }, { 1, "read"
> }, { 2, "write" }), REC->len, REC->gpa, REC->val
> 
this is not json fromat and its not supper frendly to parse.
> 
> #cat /sys/devices/pci:00/:00:02.0/uevent
> DRIVER=vfio-pci
> PCI_CLASS=3
> PCI_ID=8086:591D
> PCI_SUBSYS_ID=8086:2212
> PCI_SLOT_NAME=:00:02.0
> MODALIAS=pci:v8086d591Dsv8086sd2212bc03sc00i00
> 
this is ini format or conf formant 
this is pretty simple to parse whichi would be fine.
that said you could also have a version or capablitiy directory with a file
for each key and a singel value.

i would prefer to only have to do one read personally the list the files in
directory and then read tehm all ot build the datastucture myself but that is
doable though the simple ini format use d for uevent seams the best of 3 options
provided above.
> > 
> > Your idea of having both a "self" object and an array of "compatible"
> > objects is perhaps something we can build on, but we must not assume
> > PCI devices at the root level of the object.  Providing both the
> > mdev-type and the driver is a bit redundant, since the former includes
> > the latter.  We can't have vendor specific versioning schemes though,
> > ie. gvt-version. We need to agree on a common scheme and decide which
> > fields the version is relative to, ex. just the mdev type?
> 
> what about making all comparing fields vendor specific?
>