[ovirt-devel] Re: ovirt-engine-sdk repository changes
On Mon, Aug 9, 2021 at 3:52 PM Nir Soffer wrote: > On Mon, Aug 9, 2021 at 4:05 PM Martin Necas wrote: > > just bit summary of few changes in ovirt-engine-sdk repo [1]. There was > the decision to split the SDK generator and the python SDK itself. So we > created a new repository on GitHub called python-ovirt-engine-sdk4 [2]. > > > > We switched from Travis CI to GitHub Actions, because Travis no longer > supported .org domain where we had our tests. The GitHub Actions > automatically builds the SDK and pushes it to the new repo similar to the > Go SDK. > > > > There were some issues with the documentation generator [3] so we > switched to the new version of pdoc [4]. > > > > We migrated the SDK examples because the examples have nothing to do > with the generator itself moved it to the new repository [6]. > > The examples and the generator are different parts of the python sdk. Why > do we need to add another repository? We already have too many > repositories. > We needed to split the current SDK to allow easier build and packaging of the Python part. Because the generator part depends on many Java packages, building it for different platforms (for example for CentOS Stream 9) requires huge amounts of resources. Separating Python part allows us easy Python build for different Python version and probably it will allow us to add SDK into EPEL, which would simplify using it as a dependency for other projects (for example oVirt Ansible Collection) > With this change we lose all the history from the sdk examples, and we > break all the links to the example scripts. Try this search: > https://www.google.co.il/search?q=upload_disk.py If really needed we can add some README to current generator repo, but the most important link, which is official documentation, hasn't changed: http://ovirt.github.io/ovirt-engine-sdk/4.4/ Not to mention that finally after solving lots of issues with documentation generation it's up-to-date and aligned with SDK releases. And I really need to raise it again: EXAMPLES IN PYTHON SDK SHOULD NOT BE USED AS PRODUCTION READY TOOLS AND THAT'S WHY THERE IS NO HARM IN MOVING THEM TO DIFFERENT REPOSITORY. > > Why was this not discussed here before the change? > It was discussed between all maintainers of SDK. > > Nir > > -- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o. ___ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/EJKQJJR6RJQZNQPEU2PQ6SBWFSRIMU3P/
[ovirt-devel] Re: ovirt-engine-sdk repository changes
On Mon, Aug 9, 2021 at 4:05 PM Martin Necas wrote: > just bit summary of few changes in ovirt-engine-sdk repo [1]. There was the > decision to split the SDK generator and the python SDK itself. So we created > a new repository on GitHub called python-ovirt-engine-sdk4 [2]. > > We switched from Travis CI to GitHub Actions, because Travis no longer > supported .org domain where we had our tests. The GitHub Actions > automatically builds the SDK and pushes it to the new repo similar to the Go > SDK. > > There were some issues with the documentation generator [3] so we switched to > the new version of pdoc [4]. > > We migrated the SDK examples because the examples have nothing to do with the > generator itself moved it to the new repository [6]. The examples and the generator are different parts of the python sdk. Why do we need to add another repository? We already have too many repositories. With this change we lose all the history from the sdk examples, and we break all the links to the example scripts. Try this search: https://www.google.co.il/search?q=upload_disk.py Why was this not discussed here before the change? Nir ___ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/AYDA3XK6OJWN6S4MERGEUZLDOL3HDMR2/
[ovirt-devel] Re: Migration without shared storage is unsafe (was: Change in ovirt-system-tests[master]: HE: Use node image)
On Mon, Aug 9, 2021 at 4:29 PM Nir Soffer wrote: > > On Mon, Aug 9, 2021 at 4:01 PM Nir Soffer wrote: > > > > On Mon, Aug 9, 2021 at 2:42 PM Yedidyah Bar David wrote: > > > > > > On Mon, Aug 9, 2021 at 1:43 PM Nir Soffer wrote: > > > > > > > > On Mon, Aug 9, 2021 at 10:35 AM Yedidyah Bar David > > > > wrote: > > > > > > > > > > On Sun, Aug 8, 2021 at 5:42 PM Code Review wrote: > > > > > > > > > > > > From Jenkins CI : > > > > > > > > > > > > Jenkins CI has posted comments on this change. ( > > > > > > https://gerrit.ovirt.org/c/ovirt-system-tests/+/115392 ) > > > > > > > > > > > > Change subject: HE: Use node image > > > > > > .. > > > > > > > > > > > > > > > > > > Patch Set 13: Continuous-Integration-1 > > > > > > > > > > > > Build Failed > > > > > > > > > > While trying to deactivate a host, the engine wanted to migrate a VM > > > > > (vm0) from host-0 to host-1. vdsm log of host-0 says: > > > > > > > > > > 2021-08-08 14:31:10,076+ ERROR (migsrc/cde311f9) [virt.vm] > > > > > (vmId='cde311f9-9a33-4eb9-8338-fa22ff49edc2') Failed to migrate > > > > > (migration:503) > > > > > Traceback (most recent call last): > > > > > File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line > > > > > 477, in _regular_run > > > > > time.time(), machineParams > > > > > File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line > > > > > 578, in _startUnderlyingMigration > > > > > self._perform_with_conv_schedule(duri, muri) > > > > > File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line > > > > > 667, in _perform_with_conv_schedule > > > > > self._perform_migration(duri, muri) > > > > > File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line > > > > > 596, in _perform_migration > > > > > self._migration_flags) > > > > > File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line > > > > > 159, in call > > > > > return getattr(self._vm._dom, name)(*a, **kw) > > > > > File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", > > > > > line 101, in f > > > > > ret = attr(*args, **kwargs) > > > > > File > > > > > "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", > > > > > line 131, in wrapper > > > > > ret = f(*args, **kwargs) > > > > > File "/usr/lib/python3.6/site-packages/vdsm/common/function.py", > > > > > line 94, in wrapper > > > > > return func(inst, *args, **kwargs) > > > > > File "/usr/lib64/python3.6/site-packages/libvirt.py", line 2126, in > > > > > migrateToURI3 > > > > > raise libvirtError('virDomainMigrateToURI3() failed') > > > > > libvirt.libvirtError: Unsafe migration: Migration without shared > > > > > storage is unsafe > > > > > > > > Please share the vm xml: > > > > > > > > sudo virsh -r dumpxl vm-name > > > > > > I think you should be able to find a dump of it in vdsm.log: > > > > > > https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/18650/artifact/check-patch.he-basic_suite_master.el8.x86_64/test_logs/ost-he-basic-suite-master-host-0/var/log/vdsm/vdsm.log > > > > > > I think the first line of starting a migration is: > > > > > > 2021-08-08 14:31:08,350+ DEBUG (jsonrpc/4) [jsonrpc.JsonRpcServer] > > > Calling 'VM.migrate' in bridge with {'vmID': > > > 'cde311f9-9a33-4eb9-8338-fa22ff49edc2', 'params': > > > > > > A few lines later: > > > > > > 2021-08-08 14:31:08,387+ DEBUG (migsrc/cde311f9) > > > [virt.metadata.Descriptor] dumped metadata for > > > cde311f9-9a33-4eb9-8338-fa22ff49edc2: > > encoding='utf-8'?> > > > > > > 98304 > > > > This is not the vm xml but the metadata xml. > > > > Looking at the logs on both hosts: > > > > [nsoffer@sparse ost]$ head -1 *vdsm.log > > ==> host0-vdsm.log <== > > 2021-08-08 13:16:04,676+ INFO (MainThread) [vds] (PID: 65169) I > > am the actual vdsm 4.40.80.3.12.git6d67b935b > > ost-he-basic-suite-master-host-0 (4.18.0-326.el8.x86_64) (vdsmd:162) > > > > ==> host1-vdsm.log <== > > 2021-08-08 15:40:54,367+0200 INFO (MainThread) [vds] (PID: 23005) I > > am the actual vdsm 4.40.80.4.5.git4309a3949 > > ost-he-basic-suite-master-host-1 (4.18.0-326.el8.x86_64) (vdsmd:162) > > > > - The hosts clocks are using different time zones (+ vs +0200) is > > this intended? > > - You are testing different versions of vdsm - is this intended? > > > > We have about 60 errors: > > $ grep 'Migration without shared storage is unsafe' host0-vdsm.log | wc -l > > 60 > > > > Looking at the first migration that failed, the vm xml is here: > > > > 2021-08-08 14:20:34,127+ INFO (vm/cde311f9) [virt.vm] > > (vmId='cde311f9-9a33-4eb9-8338-fa22ff49edc2') > ncoding='utf-8'?> > > http://libvirt.org/schemas/domain/qemu/1.0; > > xmlns:ovirt-vm="http://ovirt.org/vm/1.0; type="kvm"> > > vm0 > > cde311f9-9a33-4eb9-8338-fa22ff49edc2 > > ... > > > > The relevant parts for storage are: > > > > > > > iothread="1"
[ovirt-devel] Re: Migration without shared storage is unsafe (was: Change in ovirt-system-tests[master]: HE: Use node image)
On Mon, Aug 9, 2021 at 4:01 PM Nir Soffer wrote: > > On Mon, Aug 9, 2021 at 2:42 PM Yedidyah Bar David wrote: > > > > On Mon, Aug 9, 2021 at 1:43 PM Nir Soffer wrote: > > > > > > On Mon, Aug 9, 2021 at 10:35 AM Yedidyah Bar David > > > wrote: > > > > > > > > On Sun, Aug 8, 2021 at 5:42 PM Code Review wrote: > > > > > > > > > > From Jenkins CI : > > > > > > > > > > Jenkins CI has posted comments on this change. ( > > > > > https://gerrit.ovirt.org/c/ovirt-system-tests/+/115392 ) > > > > > > > > > > Change subject: HE: Use node image > > > > > .. > > > > > > > > > > > > > > > Patch Set 13: Continuous-Integration-1 > > > > > > > > > > Build Failed > > > > > > > > While trying to deactivate a host, the engine wanted to migrate a VM > > > > (vm0) from host-0 to host-1. vdsm log of host-0 says: > > > > > > > > 2021-08-08 14:31:10,076+ ERROR (migsrc/cde311f9) [virt.vm] > > > > (vmId='cde311f9-9a33-4eb9-8338-fa22ff49edc2') Failed to migrate > > > > (migration:503) > > > > Traceback (most recent call last): > > > > File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line > > > > 477, in _regular_run > > > > time.time(), machineParams > > > > File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line > > > > 578, in _startUnderlyingMigration > > > > self._perform_with_conv_schedule(duri, muri) > > > > File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line > > > > 667, in _perform_with_conv_schedule > > > > self._perform_migration(duri, muri) > > > > File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line > > > > 596, in _perform_migration > > > > self._migration_flags) > > > > File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line > > > > 159, in call > > > > return getattr(self._vm._dom, name)(*a, **kw) > > > > File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line > > > > 101, in f > > > > ret = attr(*args, **kwargs) > > > > File > > > > "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", > > > > line 131, in wrapper > > > > ret = f(*args, **kwargs) > > > > File "/usr/lib/python3.6/site-packages/vdsm/common/function.py", > > > > line 94, in wrapper > > > > return func(inst, *args, **kwargs) > > > > File "/usr/lib64/python3.6/site-packages/libvirt.py", line 2126, in > > > > migrateToURI3 > > > > raise libvirtError('virDomainMigrateToURI3() failed') > > > > libvirt.libvirtError: Unsafe migration: Migration without shared > > > > storage is unsafe > > > > > > Please share the vm xml: > > > > > > sudo virsh -r dumpxl vm-name > > > > I think you should be able to find a dump of it in vdsm.log: > > > > https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/18650/artifact/check-patch.he-basic_suite_master.el8.x86_64/test_logs/ost-he-basic-suite-master-host-0/var/log/vdsm/vdsm.log > > > > I think the first line of starting a migration is: > > > > 2021-08-08 14:31:08,350+ DEBUG (jsonrpc/4) [jsonrpc.JsonRpcServer] > > Calling 'VM.migrate' in bridge with {'vmID': > > 'cde311f9-9a33-4eb9-8338-fa22ff49edc2', 'params': > > > > A few lines later: > > > > 2021-08-08 14:31:08,387+ DEBUG (migsrc/cde311f9) > > [virt.metadata.Descriptor] dumped metadata for > > cde311f9-9a33-4eb9-8338-fa22ff49edc2: > encoding='utf-8'?> > > > > 98304 > > This is not the vm xml but the metadata xml. > > Looking at the logs on both hosts: > > [nsoffer@sparse ost]$ head -1 *vdsm.log > ==> host0-vdsm.log <== > 2021-08-08 13:16:04,676+ INFO (MainThread) [vds] (PID: 65169) I > am the actual vdsm 4.40.80.3.12.git6d67b935b > ost-he-basic-suite-master-host-0 (4.18.0-326.el8.x86_64) (vdsmd:162) > > ==> host1-vdsm.log <== > 2021-08-08 15:40:54,367+0200 INFO (MainThread) [vds] (PID: 23005) I > am the actual vdsm 4.40.80.4.5.git4309a3949 > ost-he-basic-suite-master-host-1 (4.18.0-326.el8.x86_64) (vdsmd:162) > > - The hosts clocks are using different time zones (+ vs +0200) is > this intended? > - You are testing different versions of vdsm - is this intended? > > We have about 60 errors: > $ grep 'Migration without shared storage is unsafe' host0-vdsm.log | wc -l > 60 > > Looking at the first migration that failed, the vm xml is here: > > 2021-08-08 14:20:34,127+ INFO (vm/cde311f9) [virt.vm] > (vmId='cde311f9-9a33-4eb9-8338-fa22ff49edc2') ncoding='utf-8'?> > http://libvirt.org/schemas/domain/qemu/1.0; > xmlns:ovirt-vm="http://ovirt.org/vm/1.0; type="kvm"> > vm0 > cde311f9-9a33-4eb9-8338-fa22ff49edc2 > ... > > The relevant parts for storage are: > > > iothread="1" name="qemu" type="qcow2" /> > file="/rhev/data-center/7d97ea80-f849-11eb-ac79-5452d501341a/46fa5761-bb9e-46be-8f1c-35f4b03d0203/images/20002ad2-4a97-4d2f-b3fc-c103477b5b91/614abd56-4d4f-4412-aa2a-3f7bad2f3a87"> > > > >
[ovirt-devel] Re: Migration without shared storage is unsafe (was: Change in ovirt-system-tests[master]: HE: Use node image)
On Mon, Aug 9, 2021 at 4:01 PM Nir Soffer wrote: > > On Mon, Aug 9, 2021 at 2:42 PM Yedidyah Bar David wrote: > > > > On Mon, Aug 9, 2021 at 1:43 PM Nir Soffer wrote: > > > > > > On Mon, Aug 9, 2021 at 10:35 AM Yedidyah Bar David > > > wrote: > > > > > > > > On Sun, Aug 8, 2021 at 5:42 PM Code Review wrote: > > > > > > > > > > From Jenkins CI : > > > > > > > > > > Jenkins CI has posted comments on this change. ( > > > > > https://gerrit.ovirt.org/c/ovirt-system-tests/+/115392 ) > > > > > > > > > > Change subject: HE: Use node image > > > > > .. > > > > > > > > > > > > > > > Patch Set 13: Continuous-Integration-1 > > > > > > > > > > Build Failed > > > > > > > > While trying to deactivate a host, the engine wanted to migrate a VM > > > > (vm0) from host-0 to host-1. vdsm log of host-0 says: > > > > > > > > 2021-08-08 14:31:10,076+ ERROR (migsrc/cde311f9) [virt.vm] > > > > (vmId='cde311f9-9a33-4eb9-8338-fa22ff49edc2') Failed to migrate > > > > (migration:503) > > > > Traceback (most recent call last): > > > > File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line > > > > 477, in _regular_run > > > > time.time(), machineParams > > > > File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line > > > > 578, in _startUnderlyingMigration > > > > self._perform_with_conv_schedule(duri, muri) > > > > File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line > > > > 667, in _perform_with_conv_schedule > > > > self._perform_migration(duri, muri) > > > > File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line > > > > 596, in _perform_migration > > > > self._migration_flags) > > > > File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line > > > > 159, in call > > > > return getattr(self._vm._dom, name)(*a, **kw) > > > > File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line > > > > 101, in f > > > > ret = attr(*args, **kwargs) > > > > File > > > > "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", > > > > line 131, in wrapper > > > > ret = f(*args, **kwargs) > > > > File "/usr/lib/python3.6/site-packages/vdsm/common/function.py", > > > > line 94, in wrapper > > > > return func(inst, *args, **kwargs) > > > > File "/usr/lib64/python3.6/site-packages/libvirt.py", line 2126, in > > > > migrateToURI3 > > > > raise libvirtError('virDomainMigrateToURI3() failed') > > > > libvirt.libvirtError: Unsafe migration: Migration without shared > > > > storage is unsafe > > > > > > Please share the vm xml: > > > > > > sudo virsh -r dumpxl vm-name > > > > I think you should be able to find a dump of it in vdsm.log: > > > > https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/18650/artifact/check-patch.he-basic_suite_master.el8.x86_64/test_logs/ost-he-basic-suite-master-host-0/var/log/vdsm/vdsm.log > > > > I think the first line of starting a migration is: > > > > 2021-08-08 14:31:08,350+ DEBUG (jsonrpc/4) [jsonrpc.JsonRpcServer] > > Calling 'VM.migrate' in bridge with {'vmID': > > 'cde311f9-9a33-4eb9-8338-fa22ff49edc2', 'params': > > > > A few lines later: > > > > 2021-08-08 14:31:08,387+ DEBUG (migsrc/cde311f9) > > [virt.metadata.Descriptor] dumped metadata for > > cde311f9-9a33-4eb9-8338-fa22ff49edc2: > encoding='utf-8'?> > > > > 98304 > > This is not the vm xml but the metadata xml. OK > > Looking at the logs on both hosts: > > [nsoffer@sparse ost]$ head -1 *vdsm.log > ==> host0-vdsm.log <== > 2021-08-08 13:16:04,676+ INFO (MainThread) [vds] (PID: 65169) I > am the actual vdsm 4.40.80.3.12.git6d67b935b > ost-he-basic-suite-master-host-0 (4.18.0-326.el8.x86_64) (vdsmd:162) > > ==> host1-vdsm.log <== > 2021-08-08 15:40:54,367+0200 INFO (MainThread) [vds] (PID: 23005) I > am the actual vdsm 4.40.80.4.5.git4309a3949 > ost-he-basic-suite-master-host-1 (4.18.0-326.el8.x86_64) (vdsmd:162) > > - The hosts clocks are using different time zones (+ vs +0200) is > this intended? > - You are testing different versions of vdsm - is this intended? Both of these are a result of the patch this was ran for - which makes host-0 use ovirt-node, for the he-basic suite. > > We have about 60 errors: > $ grep 'Migration without shared storage is unsafe' host0-vdsm.log | wc -l > 60 > > Looking at the first migration that failed, the vm xml is here: > > 2021-08-08 14:20:34,127+ INFO (vm/cde311f9) [virt.vm] > (vmId='cde311f9-9a33-4eb9-8338-fa22ff49edc2') ncoding='utf-8'?> > http://libvirt.org/schemas/domain/qemu/1.0; > xmlns:ovirt-vm="http://ovirt.org/vm/1.0; type="kvm"> > vm0 > cde311f9-9a33-4eb9-8338-fa22ff49edc2 > ... > > The relevant parts for storage are: > > > iothread="1" name="qemu" type="qcow2" /> >
[ovirt-devel] ovirt-engine-sdk repository changes
Hi, just bit summary of few changes in ovirt-engine-sdk repo [1]. There was the decision to split the SDK generator and the python SDK itself. So we created a new repository on GitHub called python-ovirt-engine-sdk4 [2]. We switched from Travis CI to GitHub Actions, because Travis no longer supported .org domain where we had our tests. The GitHub Actions automatically builds the SDK and pushes it to the new repo similar to the Go SDK. There were some issues with the documentation generator [3] so we switched to the new version of pdoc [4]. We migrated the SDK examples because the examples have nothing to do with the generator itself moved it to the new repository [6]. Martin Necas [1] https://github.com/ovirt/ovirt-engine-sdk [2] https://github.com/ovirt/python-ovirt-engine-sdk4 [3] http://ovirt.github.io/ovirt-engine-sdk/master/ [4] https://pdoc.dev/docs/pdoc.html [5] https://gerrit.ovirt.org/#/c/ovirt-engine-sdk/+/116066/ [6] https://github.com/oVirt/python-ovirt-engine-sdk4/tree/main/examples ___ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/KZT2H3VWUFU2CPQEBTJPA3S7LJ7CMPDK/
[ovirt-devel] Re: Migration without shared storage is unsafe (was: Change in ovirt-system-tests[master]: HE: Use node image)
On Mon, Aug 9, 2021 at 2:42 PM Yedidyah Bar David wrote: > > On Mon, Aug 9, 2021 at 1:43 PM Nir Soffer wrote: > > > > On Mon, Aug 9, 2021 at 10:35 AM Yedidyah Bar David wrote: > > > > > > On Sun, Aug 8, 2021 at 5:42 PM Code Review wrote: > > > > > > > > From Jenkins CI : > > > > > > > > Jenkins CI has posted comments on this change. ( > > > > https://gerrit.ovirt.org/c/ovirt-system-tests/+/115392 ) > > > > > > > > Change subject: HE: Use node image > > > > .. > > > > > > > > > > > > Patch Set 13: Continuous-Integration-1 > > > > > > > > Build Failed > > > > > > While trying to deactivate a host, the engine wanted to migrate a VM > > > (vm0) from host-0 to host-1. vdsm log of host-0 says: > > > > > > 2021-08-08 14:31:10,076+ ERROR (migsrc/cde311f9) [virt.vm] > > > (vmId='cde311f9-9a33-4eb9-8338-fa22ff49edc2') Failed to migrate > > > (migration:503) > > > Traceback (most recent call last): > > > File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line > > > 477, in _regular_run > > > time.time(), machineParams > > > File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line > > > 578, in _startUnderlyingMigration > > > self._perform_with_conv_schedule(duri, muri) > > > File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line > > > 667, in _perform_with_conv_schedule > > > self._perform_migration(duri, muri) > > > File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line > > > 596, in _perform_migration > > > self._migration_flags) > > > File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line > > > 159, in call > > > return getattr(self._vm._dom, name)(*a, **kw) > > > File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line > > > 101, in f > > > ret = attr(*args, **kwargs) > > > File > > > "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", > > > line 131, in wrapper > > > ret = f(*args, **kwargs) > > > File "/usr/lib/python3.6/site-packages/vdsm/common/function.py", > > > line 94, in wrapper > > > return func(inst, *args, **kwargs) > > > File "/usr/lib64/python3.6/site-packages/libvirt.py", line 2126, in > > > migrateToURI3 > > > raise libvirtError('virDomainMigrateToURI3() failed') > > > libvirt.libvirtError: Unsafe migration: Migration without shared > > > storage is unsafe > > > > Please share the vm xml: > > > > sudo virsh -r dumpxl vm-name > > I think you should be able to find a dump of it in vdsm.log: > > https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/18650/artifact/check-patch.he-basic_suite_master.el8.x86_64/test_logs/ost-he-basic-suite-master-host-0/var/log/vdsm/vdsm.log > > I think the first line of starting a migration is: > > 2021-08-08 14:31:08,350+ DEBUG (jsonrpc/4) [jsonrpc.JsonRpcServer] > Calling 'VM.migrate' in bridge with {'vmID': > 'cde311f9-9a33-4eb9-8338-fa22ff49edc2', 'params': > > A few lines later: > > 2021-08-08 14:31:08,387+ DEBUG (migsrc/cde311f9) > [virt.metadata.Descriptor] dumped metadata for > cde311f9-9a33-4eb9-8338-fa22ff49edc2: encoding='utf-8'?> > > 98304 This is not the vm xml but the metadata xml. Looking at the logs on both hosts: [nsoffer@sparse ost]$ head -1 *vdsm.log ==> host0-vdsm.log <== 2021-08-08 13:16:04,676+ INFO (MainThread) [vds] (PID: 65169) I am the actual vdsm 4.40.80.3.12.git6d67b935b ost-he-basic-suite-master-host-0 (4.18.0-326.el8.x86_64) (vdsmd:162) ==> host1-vdsm.log <== 2021-08-08 15:40:54,367+0200 INFO (MainThread) [vds] (PID: 23005) I am the actual vdsm 4.40.80.4.5.git4309a3949 ost-he-basic-suite-master-host-1 (4.18.0-326.el8.x86_64) (vdsmd:162) - The hosts clocks are using different time zones (+ vs +0200) is this intended? - You are testing different versions of vdsm - is this intended? We have about 60 errors: $ grep 'Migration without shared storage is unsafe' host0-vdsm.log | wc -l 60 Looking at the first migration that failed, the vm xml is here: 2021-08-08 14:20:34,127+ INFO (vm/cde311f9) [virt.vm] (vmId='cde311f9-9a33-4eb9-8338-fa22ff49edc2') http://libvirt.org/schemas/domain/qemu/1.0; xmlns:ovirt-vm="http://ovirt.org/vm/1.0; type="kvm"> vm0 cde311f9-9a33-4eb9-8338-fa22ff49edc2 ... The relevant parts for storage are: 20002ad2-4a97-4d2f-b3fc-c103477b5b91 738c8486-8929-44ec-9083-69327bde9c65 So we have one qcow2 disk on file storage, and one direct lun. On the destination, the first migration attempt is here: 2021-08-08 16:31:08,437+0200 DEBUG (jsonrpc/2) [jsonrpc.JsonRpcServer] Calling 'VM.migrationCreate' in bridge with {'vmID': 'c de311f9-9a33-4eb9-8338-fa22ff49edc2', 'params':
[ovirt-devel] Re: Migration without shared storage is unsafe (was: Change in ovirt-system-tests[master]: HE: Use node image)
On Mon, Aug 9, 2021 at 1:43 PM Nir Soffer wrote: > > On Mon, Aug 9, 2021 at 10:35 AM Yedidyah Bar David wrote: > > > > On Sun, Aug 8, 2021 at 5:42 PM Code Review wrote: > > > > > > From Jenkins CI : > > > > > > Jenkins CI has posted comments on this change. ( > > > https://gerrit.ovirt.org/c/ovirt-system-tests/+/115392 ) > > > > > > Change subject: HE: Use node image > > > .. > > > > > > > > > Patch Set 13: Continuous-Integration-1 > > > > > > Build Failed > > > > While trying to deactivate a host, the engine wanted to migrate a VM > > (vm0) from host-0 to host-1. vdsm log of host-0 says: > > > > 2021-08-08 14:31:10,076+ ERROR (migsrc/cde311f9) [virt.vm] > > (vmId='cde311f9-9a33-4eb9-8338-fa22ff49edc2') Failed to migrate > > (migration:503) > > Traceback (most recent call last): > > File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line > > 477, in _regular_run > > time.time(), machineParams > > File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line > > 578, in _startUnderlyingMigration > > self._perform_with_conv_schedule(duri, muri) > > File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line > > 667, in _perform_with_conv_schedule > > self._perform_migration(duri, muri) > > File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line > > 596, in _perform_migration > > self._migration_flags) > > File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line > > 159, in call > > return getattr(self._vm._dom, name)(*a, **kw) > > File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line 101, > > in f > > ret = attr(*args, **kwargs) > > File "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", > > line 131, in wrapper > > ret = f(*args, **kwargs) > > File "/usr/lib/python3.6/site-packages/vdsm/common/function.py", > > line 94, in wrapper > > return func(inst, *args, **kwargs) > > File "/usr/lib64/python3.6/site-packages/libvirt.py", line 2126, in > > migrateToURI3 > > raise libvirtError('virDomainMigrateToURI3() failed') > > libvirt.libvirtError: Unsafe migration: Migration without shared > > storage is unsafe > > Please share the vm xml: > > sudo virsh -r dumpxl vm-name I think you should be able to find a dump of it in vdsm.log: https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/18650/artifact/check-patch.he-basic_suite_master.el8.x86_64/test_logs/ost-he-basic-suite-master-host-0/var/log/vdsm/vdsm.log I think the first line of starting a migration is: 2021-08-08 14:31:08,350+ DEBUG (jsonrpc/4) [jsonrpc.JsonRpcServer] Calling 'VM.migrate' in bridge with {'vmID': 'cde311f9-9a33-4eb9-8338-fa22ff49edc2', 'params': A few lines later: 2021-08-08 14:31:08,387+ DEBUG (migsrc/cde311f9) [virt.metadata.Descriptor] dumped metadata for cde311f9-9a33-4eb9-8338-fa22ff49edc2: 98304 true 4.6 False 0 {} false 96 96 auto_resume 1628431993.720967 ovirtmgmt ;vdsmdummy; 36001405bc9d94e4419b4b80a2f702e2f 36001405bc9d94e4419b4b80a2f702e2f False 46fa5761-bb9e-46be-8f1c-35f4b03d0203 20002ad2-4a97-4d2f-b3fc-c103477b5b91 False 7d97ea80-f849-11eb-ac79-5452d501341a 614abd56-4d4f-4412-aa2a-3f7bad2f3a87 1 46fa5761-bb9e-46be-8f1c-35f4b03d0203 20002ad2-4a97-4d2f-b3fc-c103477b5b91 0 /rhev/data-center/mnt/192.168.200.2:_exports_nfs_share1/46fa5761-bb9e-46be-8f1c-35f4b03d0203/images/20002ad2-4a97-4d2f-b3fc-c103477b5b91/1d3f07dc-b481-492f-a2a6-7c46689d82ba.lease /rhev/data-center/mnt/192.168.200.2:_exports_nfs_share1/46fa5761-bb9e-46be-8f1c-35f4b03d0203/images/20002ad2-4a97-4d2f-b3fc-c103477b5b91/1d3f07dc-b481-492f-a2a6-7c46689d82ba 1d3f07dc-b481-492f-a2a6-7c46689d82ba 46fa5761-bb9e-46be-8f1c-35f4b03d0203 20002ad2-4a97-4d2f-b3fc-c103477b5b91 0 /rhev/data-center/mnt/192.168.200.2:_exports_nfs_share1/46fa5761-bb9e-46be-8f1c-35f4b03d0203/images/20002ad2-4a97-4d2f-b3fc-c103477b5b91/614abd56-4d4f-4412-aa2a-3f7bad2f3a87.lease /rhev/data-center/mnt/192.168.200.2:_exports_nfs_share1/46fa5761-bb9e-46be-8f1c-35f4b03d0203/images/20002ad2-4a97-4d2f-b3fc-c103477b5b91/614abd56-4d4f-4412-aa2a-3f7bad2f3a87 614abd56-4d4f-4412-aa2a-3f7bad2f3a87 46fa5761-bb9e-46be-8f1c-35f4b03d0203 20002ad2-4a97-4d2f-b3fc-c103477b5b91 0 /rhev/data-center/mnt/192.168.200.2:_exports_nfs_share1/46fa5761-bb9e-46be-8f1c-35f4b03d0203/images/20002ad2-4a97-4d2f-b3fc-c103477b5b91/a4309ef3-01bb-45db-8bf7-0f9498a7feeb.lease
[ovirt-devel] Re: OST HE: Engine VM went down due to cpu load (was: [oVirt Jenkins] ovirt-system-tests_he-basic-suite-master - Build # 2126 - Failure!)
On Mon, Aug 9, 2021 at 1:39 PM Nir Soffer wrote: > > On Sun, Aug 8, 2021 at 10:14 AM Yedidyah Bar David wrote: > > > > On Thu, Aug 5, 2021 at 9:31 AM Yedidyah Bar David wrote: > > > > > > On Wed, Aug 4, 2021 at 1:56 PM Michal Skrivanek > > > wrote: > > > > I don’t really know for sure, but AFAICT it should be real data from > > > > the start. > > > > Maybe for the first interval, but afterwards it’s always a libvirt > > > > reported value > > > > > > Adding Nir. Not sure who else... sorry. > > > > > > This now happened again: > > > > > > https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/2129/ > > > > > > Console has: > > > > > > 06:25:25 2021-08-05 03:25:25+,873 INFO[root] Starting the > > > engine VM... (test_008_restart_he_vm:96) > > > > > > broker.log has (I think it only logs once a minute): > > > > > > Thread-4::INFO::2021-08-05 > > > 05:25:31,995::cpu_load_no_engine::126::cpu_load_no_engine.CpuLoadNoEngine::(calculate_load) > > > System load total=0.8164, engine=0., non-engine=0.8164 > > > Thread-4::INFO::2021-08-05 > > > 05:26:32,072::cpu_load_no_engine::126::cpu_load_no_engine.CpuLoadNoEngine::(calculate_load) > > > System load total=0.8480, engine=0., non-engine=0.8480 > > > Thread-4::INFO::2021-08-05 > > > 05:27:32,175::cpu_load_no_engine::126::cpu_load_no_engine.CpuLoadNoEngine::(calculate_load) > > > System load total=0.7572, engine=0.2656, non-engine=0.4916 > > > > > > vdsm.log [1] has: > > > > > > 2021-08-05 05:25:29,017+0200 DEBUG (jsonrpc/4) [jsonrpc.JsonRpcServer] > > > Calling 'VM.create' in bridge... > > > > > > 2021-08-05 05:25:31,991+0200 DEBUG (jsonrpc/7) [api] FINISH getStats > > > response={'status': {'code': 0, 'message': 'Done'}, 'statsList': > > > [{'statusTime': '2152587436', 'status': 'WaitForLaunch', 'vmId': > > > '230ea8e8-e365-46cd-98fa-e9d6a653306f', 'vmName': 'HostedEngine', > > > 'vmType': 'kvm', 'kvmEnable': 'true', 'acpiEnable': 'true', > > > 'elapsedTime': '2', 'monitorResponse': '0', 'clientIp': '', > > > 'timeOffset': '0', 'cpuUser': '0.00', 'cpuSys': '0.00',... > > > > > > and 17 more such [2] lines. Line 11 is the first one with cpuUser != > > > 0.00, at '2021-08-05 05:27:02', 92 seconds later. Incidentally (or > > > not), this is also the first line with 'network' in it. There are > > > other differences along the way - e.g. I see status moving from > > > WaitForLaunch to 'Powering up' and to 'Up', but the first 'Up' line is > > > number 7 - 40 seconds before cpuUser>0. > > Milan should be able to help with this. Thanks. > > In storage monitoring we avoid this issue by reporting actual=False > before we got the first monitoring results, so engine can wait for the actual > results. > https://github.com/oVirt/vdsm/blob/4309a39492040300e1b983eb583e8847f5cc7538/lib/vdsm/storage/monitor.py#L297 Makes sense. That's indeed what I was looking for, for VM cpu usage. > > > > I'd like to clarify that I do not see this mainly as an OST issue, but > > > more as a general HE HA issue - if users start global maint, then > > > restart the engine vm, then exit global maint too quickly, the > > > reported high cpu load might make the machine go down. In OST, I can > > > easily just add another 60 seconds or so delay after the engine is up. > > > Of course we can do the same also in HA, and I'd be for doing this, if > > > we do not get any more information (or find out that this is a > > > recently-introduced bug and fix it). > > If this is a real issue you should be able to reproduce this on a real system. In "real", you might refer to two different things: 1. OST is a different environment - has ridiculously little memory/cpu, etc., or something else that is not expected or not recommended for a real system. 2. The _flow_ is not real. As in, it's unlikely that a real user will exit global maintenance so quickly after starting the engine VM, without looking around a bit more. I agree with both - and even if it's eventually considered a real bug, I'd not consider it severe. But just saying "OST is not a real system" is not something I can completely agree with. We have a balance/tradeoff here between trying to imitate "real systems" as accurately as possible and between doing this efficiently/effectively. I do not think there is a deliberate design choice to make it arbitrarily different from real systems. Best regards, -- Didi ___ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/BTR3FHQP6LZNTLVOC6PHMBIV5WH27YUG/
[ovirt-devel] Re: Migration without shared storage is unsafe (was: Change in ovirt-system-tests[master]: HE: Use node image)
On Mon, Aug 9, 2021 at 10:35 AM Yedidyah Bar David wrote: > > On Sun, Aug 8, 2021 at 5:42 PM Code Review wrote: > > > > From Jenkins CI : > > > > Jenkins CI has posted comments on this change. ( > > https://gerrit.ovirt.org/c/ovirt-system-tests/+/115392 ) > > > > Change subject: HE: Use node image > > .. > > > > > > Patch Set 13: Continuous-Integration-1 > > > > Build Failed > > While trying to deactivate a host, the engine wanted to migrate a VM > (vm0) from host-0 to host-1. vdsm log of host-0 says: > > 2021-08-08 14:31:10,076+ ERROR (migsrc/cde311f9) [virt.vm] > (vmId='cde311f9-9a33-4eb9-8338-fa22ff49edc2') Failed to migrate > (migration:503) > Traceback (most recent call last): > File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line > 477, in _regular_run > time.time(), machineParams > File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line > 578, in _startUnderlyingMigration > self._perform_with_conv_schedule(duri, muri) > File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line > 667, in _perform_with_conv_schedule > self._perform_migration(duri, muri) > File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line > 596, in _perform_migration > self._migration_flags) > File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line > 159, in call > return getattr(self._vm._dom, name)(*a, **kw) > File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line 101, > in f > ret = attr(*args, **kwargs) > File "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", > line 131, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib/python3.6/site-packages/vdsm/common/function.py", > line 94, in wrapper > return func(inst, *args, **kwargs) > File "/usr/lib64/python3.6/site-packages/libvirt.py", line 2126, in > migrateToURI3 > raise libvirtError('virDomainMigrateToURI3() failed') > libvirt.libvirtError: Unsafe migration: Migration without shared > storage is unsafe Please share the vm xml: sudo virsh -r dumpxl vm-name Nir ___ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/PWYKPD4AJ7GD6QNA5XOLVJC6CMOYEHDW/
[ovirt-devel] Re: OST HE: Engine VM went down due to cpu load (was: [oVirt Jenkins] ovirt-system-tests_he-basic-suite-master - Build # 2126 - Failure!)
On Sun, Aug 8, 2021 at 10:14 AM Yedidyah Bar David wrote: > > On Thu, Aug 5, 2021 at 9:31 AM Yedidyah Bar David wrote: > > > > On Wed, Aug 4, 2021 at 1:56 PM Michal Skrivanek > > wrote: > > > I don’t really know for sure, but AFAICT it should be real data from the > > > start. > > > Maybe for the first interval, but afterwards it’s always a libvirt > > > reported value > > > > Adding Nir. Not sure who else... sorry. > > > > This now happened again: > > > > https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/2129/ > > > > Console has: > > > > 06:25:25 2021-08-05 03:25:25+,873 INFO[root] Starting the > > engine VM... (test_008_restart_he_vm:96) > > > > broker.log has (I think it only logs once a minute): > > > > Thread-4::INFO::2021-08-05 > > 05:25:31,995::cpu_load_no_engine::126::cpu_load_no_engine.CpuLoadNoEngine::(calculate_load) > > System load total=0.8164, engine=0., non-engine=0.8164 > > Thread-4::INFO::2021-08-05 > > 05:26:32,072::cpu_load_no_engine::126::cpu_load_no_engine.CpuLoadNoEngine::(calculate_load) > > System load total=0.8480, engine=0., non-engine=0.8480 > > Thread-4::INFO::2021-08-05 > > 05:27:32,175::cpu_load_no_engine::126::cpu_load_no_engine.CpuLoadNoEngine::(calculate_load) > > System load total=0.7572, engine=0.2656, non-engine=0.4916 > > > > vdsm.log [1] has: > > > > 2021-08-05 05:25:29,017+0200 DEBUG (jsonrpc/4) [jsonrpc.JsonRpcServer] > > Calling 'VM.create' in bridge... > > > > 2021-08-05 05:25:31,991+0200 DEBUG (jsonrpc/7) [api] FINISH getStats > > response={'status': {'code': 0, 'message': 'Done'}, 'statsList': > > [{'statusTime': '2152587436', 'status': 'WaitForLaunch', 'vmId': > > '230ea8e8-e365-46cd-98fa-e9d6a653306f', 'vmName': 'HostedEngine', > > 'vmType': 'kvm', 'kvmEnable': 'true', 'acpiEnable': 'true', > > 'elapsedTime': '2', 'monitorResponse': '0', 'clientIp': '', > > 'timeOffset': '0', 'cpuUser': '0.00', 'cpuSys': '0.00',... > > > > and 17 more such [2] lines. Line 11 is the first one with cpuUser != > > 0.00, at '2021-08-05 05:27:02', 92 seconds later. Incidentally (or > > not), this is also the first line with 'network' in it. There are > > other differences along the way - e.g. I see status moving from > > WaitForLaunch to 'Powering up' and to 'Up', but the first 'Up' line is > > number 7 - 40 seconds before cpuUser>0. Milan should be able to help with this. In storage monitoring we avoid this issue by reporting actual=False before we got the first monitoring results, so engine can wait for the actual results. https://github.com/oVirt/vdsm/blob/4309a39492040300e1b983eb583e8847f5cc7538/lib/vdsm/storage/monitor.py#L297 > > I'd like to clarify that I do not see this mainly as an OST issue, but > > more as a general HE HA issue - if users start global maint, then > > restart the engine vm, then exit global maint too quickly, the > > reported high cpu load might make the machine go down. In OST, I can > > easily just add another 60 seconds or so delay after the engine is up. > > Of course we can do the same also in HA, and I'd be for doing this, if > > we do not get any more information (or find out that this is a > > recently-introduced bug and fix it). If this is a real issue you should be able to reproduce this on a real system. Nir ___ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/UZPQE6TUC433FRNMXO6LENYDNRJ7EE5L/
[ovirt-devel] Migration without shared storage is unsafe (was: Change in ovirt-system-tests[master]: HE: Use node image)
On Sun, Aug 8, 2021 at 5:42 PM Code Review wrote: > > From Jenkins CI : > > Jenkins CI has posted comments on this change. ( > https://gerrit.ovirt.org/c/ovirt-system-tests/+/115392 ) > > Change subject: HE: Use node image > .. > > > Patch Set 13: Continuous-Integration-1 > > Build Failed While trying to deactivate a host, the engine wanted to migrate a VM (vm0) from host-0 to host-1. vdsm log of host-0 says: 2021-08-08 14:31:10,076+ ERROR (migsrc/cde311f9) [virt.vm] (vmId='cde311f9-9a33-4eb9-8338-fa22ff49edc2') Failed to migrate (migration:503) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 477, in _regular_run time.time(), machineParams File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 578, in _startUnderlyingMigration self._perform_with_conv_schedule(duri, muri) File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 667, in _perform_with_conv_schedule self._perform_migration(duri, muri) File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 596, in _perform_migration self._migration_flags) File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line 159, in call return getattr(self._vm._dom, name)(*a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line 101, in f ret = attr(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/common/function.py", line 94, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python3.6/site-packages/libvirt.py", line 2126, in migrateToURI3 raise libvirtError('virDomainMigrateToURI3() failed') libvirt.libvirtError: Unsafe migration: Migration without shared storage is unsafe Any idea? -- Didi ___ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7UK4BDLCD6PR6DPYL3G6UMMYH3NIEX36/