TASK [ovirt.hosted_engine_setup : Activate storage domain] ****************************************** ... Error: Fault reason is "Operation Failed". Fault detail is "[]". HTTP response code is 400.
usually means that the engine failed to activate that storage domain; unfortunately engine error messages are not always that clear (see https://bugzilla.redhat.com/1554922 <https://bugzilla.redhat.com/show_bug.cgi?id=1554922> ) but this is often due to fact the the NFS share or the iSCSI lun or whatever you used wasn't really clean. Are you manually cleaning it between one attempt and the next one? On Tue, Apr 2, 2019 at 10:50 AM Callum Smith <[email protected]> wrote: > Dear Simone, > > With no changes, we're now seeing this baffling error: > > TASK [ovirt.hosted_engine_setup : Parse OVF] > ******************************************************** > task path: > /etc/ansible/playbook/ovirt-ansible-hosted-engine-setup/tasks/create_storage_domain.yml:120 > <virthyp04.virt.in.bmrc.ox.ac.uk> ESTABLISH SSH CONNECTION FOR USER: root > <virthyp04.virt.in.bmrc.ox.ac.uk> SSH: EXEC ssh -C -o ControlMaster=auto > -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o > PreferredAuthentications= > gssapi-with-mic,gssapi-keyex,hostbased,publickey -o > PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o > ControlPath=/etc/ansible/.ansible/cp/2c1e73 > 363c virthyp04.virt.in.bmrc.ox.ac.uk '/bin/sh -c '"'"'echo ~root && sleep > 0'"'"'' > <virthyp04.virt.in.bmrc.ox.ac.uk> (0, '/root\n', '') > <virthyp04.virt.in.bmrc.ox.ac.uk> ESTABLISH SSH CONNECTION FOR USER: root > <virthyp04.virt.in.bmrc.ox.ac.uk> SSH: EXEC ssh -C -o ControlMaster=auto > -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o > PreferredAuthentications= > gssapi-with-mic,gssapi-keyex,hostbased,publickey -o > PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o > ControlPath=/etc/ansible/.ansible/cp/2c1e73 > 363c virthyp04.virt.in.bmrc.ox.ac.uk '/bin/sh -c '"'"'( umask 77 && mkdir > -p "` echo /root/.ansible/tmp/ansible-tmp-1553937522.31-129798476242320 `" > && echo a > nsible-tmp-1553937522.31-129798476242320="` echo > /root/.ansible/tmp/ansible-tmp-1553937522.31-129798476242320 `" ) && sleep > 0'"'"'' > <virthyp04.virt.in.bmrc.ox.ac.uk> (0, > 'ansible-tmp-1553937522.31-129798476242320=/root/.ansible/tmp/ansible-tmp-1553937522.31-129798476242320\n', > '') > Using module file /opt/ansible/lib/ansible/modules/files/xml.py > <virthyp04.virt.in.bmrc.ox.ac.uk> PUT > /etc/ansible/.ansible/tmp/ansible-local-32213KmUe6/tmp8wMU8o TO > /root/.ansible/tmp/ansible-tmp-1553937522.31-12979847624 > 2320/AnsiballZ_xml.py > <virthyp04.virt.in.bmrc.ox.ac.uk> SSH: EXEC sftp -b - -C -o > ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no > -o PreferredAuthentica > tions=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o > PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o > ControlPath=/etc/ansible/.ansible/cp/ > 2c1e73363c '[virthyp04.virt.in.bmrc.ox.ac.uk]' > <virthyp04.virt.in.bmrc.ox.ac.uk> (0, 'sftp> put > /etc/ansible/.ansible/tmp/ansible-local-32213KmUe6/tmp8wMU8o > /root/.ansible/tmp/ansible-tmp-1553937522.31-129 > 798476242320/AnsiballZ_xml.py\n', '') > <virthyp04.virt.in.bmrc.ox.ac.uk> ESTABLISH SSH CONNECTION FOR USER: root > <virthyp04.virt.in.bmrc.ox.ac.uk> SSH: EXEC ssh -C -o ControlMaster=auto > -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o > PreferredAuthentications= > gssapi-with-mic,gssapi-keyex,hostbased,publickey -o > PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o > ControlPath=/etc/ansible/.ansible/cp/2c1e73 > 363c virthyp04.virt.in.bmrc.ox.ac.uk '/bin/sh -c '"'"'chmod u+x > /root/.ansible/tmp/ansible-tmp-1553937522.31-129798476242320/ > /root/.ansible/tmp/ansible-tmp-1 > 553937522.31-129798476242320/AnsiballZ_xml.py && sleep 0'"'"'' > <virthyp04.virt.in.bmrc.ox.ac.uk> (0, '', '') > <virthyp04.virt.in.bmrc.ox.ac.uk> ESTABLISH SSH CONNECTION FOR USER: root > <virthyp04.virt.in.bmrc.ox.ac.uk> SSH: EXEC ssh -C -o ControlMaster=auto > -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o > PreferredAuthentications= > gssapi-with-mic,gssapi-keyex,hostbased,publickey -o > PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o > ControlPath=/etc/ansible/.ansible/cp/2c1e73 > 363c -tt virthyp04.virt.in.bmrc.ox.ac.uk '/bin/sh -c '"'"'/usr/bin/python > /root/.ansible/tmp/ansible-tmp-1553937522.31-129798476242320/AnsiballZ_xml.py > && sle > ep 0'"'"'' > <virthyp04.virt.in.bmrc.ox.ac.uk> (0, '\r\n{"count": 1, "matches": > [{"Disk": {"{http://schemas.dmtf.org/ovf/envelope/1/}wipe-after-delete": > "false", "{http:// > schemas.dmtf.org/ovf/envelope/1/}format": " > http://www.vmware.com/specifications/vmdk.html#sparse";, "{ > http://schemas.dmtf.org/ovf/envelope/1/}vm_snapshot_id": > "5f2be758-82d7-4c07-a220-9060e782dc7a", "{ > http://schemas.dmtf.org/ovf/envelope/1/}parentRef": "", "{ > http://schemas.dmtf.org/ovf/envelope/1/}fileRef": "6f76686 > b-199c-4cb3-bbbe-86fc34365745/72bc3948-5d8d-4877-bac8-7db4995045b5", "{ > http://schemas.dmtf.org/ovf/envelope/1/}actual_size": "51", "{ > http://schemas.dmtf.org/o > vf/envelope/1/}volume-format": "COW", "{ > http://schemas.dmtf.org/ovf/envelope/1/}boot": "true", "{ > http://schemas.dmtf.org/ovf/envelope/1/}size": "51", "{http:/ > /schemas.dmtf.org/ovf/envelope/1/}volume-type": "Sparse", "{ > http://schemas.dmtf.org/ovf/envelope/1/}disk-type": "System", "{ > http://schemas.dmtf.org/ovf/envelo > pe/1/}diskId": "72bc3948-5d8d-4877-bac8-7db4995045b5", "{ > http://schemas.dmtf.org/ovf/envelope/1/}disk-interface": "VirtIO"}}], > "changed": false, "actions": {" > xpath": "/ovf:Envelope/Section/Disk", "state": "present", "namespaces": > {"vssd": " > http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_VirtualSystemSettingDa > ta", "rasd": " > http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData";, > "xsi": "http://www.w3.org/2001/XMLSchema-instance";, "ovf": " > http://schemas.dmtf.org/ovf/envelope/1/"}}, "msg": 1, "invocation": > {"module_args": {"xpath": "/ovf:Envelope/Section/Disk", "count": false, > "set_children": null, "xmlstring": null, "strip_cdata_tags": false, > "attribute": "ovf:size", "pretty_print": false, "add_children": null, > "value": null, "content": "attribute", "state": "present", "namespaces": > {"vssd": " > http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_VirtualSystemSettingData";, > "rasd": " > http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData";, > "xsi": "http://www.w3.org/2001/XMLSchema-instance";, "ovf": " > http://schemas.dmtf.org/ovf/envelope/1/"}, "input_type": "yaml", > "print_match": false, "path": > "/var/tmp/localvmMNxnwL/master/vms/074a62d4-44f9-4ffe-a172-2702a9fe96df/074a62d4-44f9-4ffe-a172-2702a9fe96df.ovf", > "backup": false}}}\r\n', 'Shared connection to > virthyp04.virt.in.bmrc.ox.ac.uk closed.\r\n') > <virthyp04.virt.in.bmrc.ox.ac.uk> ESTABLISH SSH CONNECTION FOR USER: root > <virthyp04.virt.in.bmrc.ox.ac.uk> SSH: EXEC ssh -C -o ControlMaster=auto > -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o > PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey > -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o > ControlPath=/etc/ansible/.ansible/cp/2c1e73363c > virthyp04.virt.in.bmrc.ox.ac.uk '/bin/sh -c '"'"'rm -f -r > /root/.ansible/tmp/ansible-tmp-1553937522.31-129798476242320/ > /dev/null > 2>&1 && sleep 0'"'"'' > <virthyp04.virt.in.bmrc.ox.ac.uk> (0, '', '') > ok: [virthyp04.virt.in.bmrc.ox.ac.uk] => { > "actions": { > "namespaces": { > "ovf": "http://schemas.dmtf.org/ovf/envelope/1/";, > "rasd": " > http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData > ", > "vssd": " > http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_VirtualSystemSettingData > ", > "xsi": "http://www.w3.org/2001/XMLSchema-instance"; > }, > "state": "present", > "xpath": "/ovf:Envelope/Section/Disk" > }, > "changed": false, > "count": 1, > "invocation": { > "module_args": { > "add_children": null, > "attribute": "ovf:size", > "backup": false, > "content": "attribute", > "count": false, > "input_type": "yaml", > "namespaces": { > "ovf": "http://schemas.dmtf.org/ovf/envelope/1/";, > "rasd": " > http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData > ", > "vssd": " > http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_VirtualSystemSettingData > ", > "xsi": "http://www.w3.org/2001/XMLSchema-instance"; > }, > "path": > "/var/tmp/localvmMNxnwL/master/vms/074a62d4-44f9-4ffe-a172-2702a9fe96df/074a62d4-44f9-4ffe-a172-2702a9fe96df.ovf", > "pretty_print": false, > "print_match": false, > "set_children": null, > "state": "present", > "strip_cdata_tags": false, > "value": null, > "xmlstring": null, > "xpath": "/ovf:Envelope/Section/Disk" > } > }, > "matches": [ > { > "Disk": { > "{http://schemas.dmtf.org/ovf/envelope/1/}actual_size": > "51", > "{http://schemas.dmtf.org/ovf/envelope/1/}boot": "true", > "{http://schemas.dmtf.org/ovf/envelope/1/}disk-interface": > "VirtIO", > "{http://schemas.dmtf.org/ovf/envelope/1/}disk-type": > "System", > "{http://schemas.dmtf.org/ovf/envelope/1/}diskId": > "72bc3948-5d8d-4877-bac8-7db4995045b5", > "{http://schemas.dmtf.org/ovf/envelope/1/}fileRef": > "6f76686b-199c-4cb3-bbbe-86fc34365745/72bc3948-5d8d-4877-bac8-7db4995045b5", > "{http://schemas.dmtf.org/ovf/envelope/1/}format": " > http://www.vmware.com/specifications/vmdk.html#sparse";, > "{http://schemas.dmtf.org/ovf/envelope/1/}parentRef": "", > "{http://schemas.dmtf.org/ovf/envelope/1/}size": "51", > "{http://schemas.dmtf.org/ovf/envelope/1/}vm_snapshot_id": > "5f2be758-82d7-4c07-a220-9060e782dc7a", > "{http://schemas.dmtf.org/ovf/envelope/1/}volume-format": > "COW", > "{http://schemas.dmtf.org/ovf/envelope/1/}volume-type": > "Sparse", > "{http://schemas.dmtf.org/ovf/envelope/1/}wipe-after-delete": > "false" > } > } > ], > "msg": 1 > } > > TASK [ovirt.hosted_engine_setup : Get required size] > ************************************************ > task path: > /etc/ansible/playbook/ovirt-ansible-hosted-engine-setup/tasks/create_storage_domain.yml:132 > ok: [virthyp04.virt.in.bmrc.ox.ac.uk] => { > "ansible_facts": { > "required_size": "65498251264" > }, > "changed": false > } > > TASK [ovirt.hosted_engine_setup : debug] > ************************************************************ > task path: > /etc/ansible/playbook/ovirt-ansible-hosted-engine-setup/tasks/create_storage_domain.yml:139 > ok: [virthyp04.virt.in.bmrc.ox.ac.uk] => { > "required_size": "65498251264" > } > > TASK [ovirt.hosted_engine_setup : Remove unsuitable storage domain] > ********************************* > task path: > /etc/ansible/playbook/ovirt-ansible-hosted-engine-setup/tasks/create_storage_domain.yml:140 > skipping: [virthyp04.virt.in.bmrc.ox.ac.uk] => { > "changed": false, > "skip_reason": "Conditional result was False" > } > > TASK [ovirt.hosted_engine_setup : debug] > ************************************************************ > task path: > /etc/ansible/playbook/ovirt-ansible-hosted-engine-setup/tasks/create_storage_domain.yml:151 > ok: [virthyp04.virt.in.bmrc.ox.ac.uk] => { > "remove_storage_domain_details": { > "changed": false, > "skip_reason": "Conditional result was False", > "skipped": true > } > } > > TASK [ovirt.hosted_engine_setup : Check storage domain free space] > ********************************** > task path: > /etc/ansible/playbook/ovirt-ansible-hosted-engine-setup/tasks/create_storage_domain.yml:152 > skipping: [virthyp04.virt.in.bmrc.ox.ac.uk] => { > "changed": false, > "skip_reason": "Conditional result was False" > } > > TASK [ovirt.hosted_engine_setup : Activate storage domain] > ****************************************** > task path: > /etc/ansible/playbook/ovirt-ansible-hosted-engine-setup/tasks/create_storage_domain.yml:161 > <virthyp04.virt.in.bmrc.ox.ac.uk> ESTABLISH SSH CONNECTION FOR USER: root > <virthyp04.virt.in.bmrc.ox.ac.uk> SSH: EXEC ssh -C -o ControlMaster=auto > -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o > PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey > -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o > ControlPath=/etc/ansible/.ansible/cp/2c1e73363c > virthyp04.virt.in.bmrc.ox.ac.uk '/bin/sh -c '"'"'echo ~root && sleep > 0'"'"'' > <virthyp04.virt.in.bmrc.ox.ac.uk> (0, '/root\n', '') > <virthyp04.virt.in.bmrc.ox.ac.uk> ESTABLISH SSH CONNECTION FOR USER: root > <virthyp04.virt.in.bmrc.ox.ac.uk> SSH: EXEC ssh -C -o ControlMaster=auto > -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o > PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey > -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o > ControlPath=/etc/ansible/.ansible/cp/2c1e73363c > virthyp04.virt.in.bmrc.ox.ac.uk '/bin/sh -c '"'"'( umask 77 && mkdir -p > "` echo /root/.ansible/tmp/ansible-tmp-1553937525.89-247224387363848 `" && > echo ansible-tmp-1553937525.89-247224387363848="` echo > /root/.ansible/tmp/ansible-tmp-1553937525.89-247224387363848 `" ) && sleep > 0'"'"'' > <virthyp04.virt.in.bmrc.ox.ac.uk> (0, > 'ansible-tmp-1553937525.89-247224387363848=/root/.ansible/tmp/ansible-tmp-1553937525.89-247224387363848\n', > '') > Using module file > /opt/ansible/lib/ansible/modules/cloud/ovirt/ovirt_storage_domain.py > <virthyp04.virt.in.bmrc.ox.ac.uk> PUT > /etc/ansible/.ansible/tmp/ansible-local-32213KmUe6/tmpQtVJtM TO > /root/.ansible/tmp/ansible-tmp-1553937525.89-247224387363848/AnsiballZ_ovirt_storage_domain.py > <virthyp04.virt.in.bmrc.ox.ac.uk> SSH: EXEC sftp -b - -C -o > ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no > -o > PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey > -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o > ControlPath=/etc/ansible/.ansible/cp/2c1e73363c '[ > virthyp04.virt.in.bmrc.ox.ac.uk]' > <virthyp04.virt.in.bmrc.ox.ac.uk> (0, 'sftp> put > /etc/ansible/.ansible/tmp/ansible-local-32213KmUe6/tmpQtVJtM > /root/.ansible/tmp/ansible-tmp-1553937525.89-247224387363848/AnsiballZ_ovirt_storage_domain.py\n', > '') > <virthyp04.virt.in.bmrc.ox.ac.uk> ESTABLISH SSH CONNECTION FOR USER: root > <virthyp04.virt.in.bmrc.ox.ac.uk> SSH: EXEC ssh -C -o ControlMaster=auto > -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o > PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey > -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o > ControlPath=/etc/ansible/.ansible/cp/2c1e73363c > virthyp04.virt.in.bmrc.ox.ac.uk '/bin/sh -c '"'"'chmod u+x > /root/.ansible/tmp/ansible-tmp-1553937525.89-247224387363848/ > /root/.ansible/tmp/ansible-tmp-1553937525.89-247224387363848/AnsiballZ_ovirt_storage_domain.py > && sleep 0'"'"'' > <virthyp04.virt.in.bmrc.ox.ac.uk> (0, '', '') > <virthyp04.virt.in.bmrc.ox.ac.uk> ESTABLISH SSH CONNECTION FOR USER: root > <virthyp04.virt.in.bmrc.ox.ac.uk> SSH: EXEC ssh -C -o ControlMaster=auto > -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o > PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey > -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o > ControlPath=/etc/ansible/.ansible/cp/2c1e73363c -tt > virthyp04.virt.in.bmrc.ox.ac.uk '/bin/sh -c '"'"'/usr/bin/python > /root/.ansible/tmp/ansible-tmp-1553937525.89-247224387363848/AnsiballZ_ovirt_storage_domain.py > && sleep 0'"'"'' > <virthyp04.virt.in.bmrc.ox.ac.uk> (1, '\r\n{"msg": "Fault reason is > \\"Operation Failed\\". Fault detail is \\"[]\\". HTTP response code is > 400.", "failed": true, "exception": "Traceback (most recent call last):\\n > File \\"/tmp/ansible_ovirt_storage_domain_payload_w8oO0Y/__main__.py\\", > line 664, in main\\n storage_domains_module.post_create_check(sd_id)\\n > File \\"/tmp/ansible_ovirt_storage_domain_payload_w8oO0Y/__main__.py\\", > line 526, in post_create_check\\n id=storage_domain.id,\\n File > \\"/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py\\", line 3053, > in add\\n return self._internal_add(storage_domain, headers, query, > wait)\\n File > \\"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\\", line 232, in > _internal_add\\n return future.wait() if wait else future\\n File > \\"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\\", line 55, in > wait\\n return self._code(response)\\n File > \\"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\\", line 229, in > callback\\n self._check_fault(response)\\n File > \\"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\\", line 132, in > _check_fault\\n self._raise_error(response, body)\\n File > \\"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\\", line 118, in > _raise_error\\n raise error\\nError: Fault reason is \\"Operation > Failed\\". Fault detail is \\"[]\\". HTTP response code is 400.\\n", > "invocation": {"module_args": {"comment": null, "warning_low_space": null, > "fetch_nested": false, "localfs": null, "data_center": "Default", "id": > null, "iscsi": null, "state": "present", "wipe_after_delete": null, > "destroy": null, "fcp": null, "description": null, "format": null, > "nested_attributes": [], "host": "virthyp04.virt.in.bmrc.ox.ac.uk", > "discard_after_delete": null, "wait": true, "domain_function": "data", > "name": "hosted_storage", "critical_space_action_blocker": null, "posixfs": > null, "poll_interval": 3, "glusterfs": null, "nfs": null, "timeout": 180, > "backup": null}}}\r\n', 'Shared connection to > virthyp04.virt.in.bmrc.ox.ac.uk closed.\r\n') > <virthyp04.virt.in.bmrc.ox.ac.uk> Failed to connect to the host via ssh: > Shared connection to virthyp04.virt.in.bmrc.ox.ac.uk closed. > <virthyp04.virt.in.bmrc.ox.ac.uk> ESTABLISH SSH CONNECTION FOR USER: root > <virthyp04.virt.in.bmrc.ox.ac.uk> SSH: EXEC ssh -C -o ControlMaster=auto > -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o > PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey > -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o > ControlPath=/etc/ansible/.ansible/cp/2c1e73363c > virthyp04.virt.in.bmrc.ox.ac.uk '/bin/sh -c '"'"'rm -f -r > /root/.ansible/tmp/ansible-tmp-1553937525.89-247224387363848/ > /dev/null > 2>&1 && sleep 0'"'"'' > <virthyp04.virt.in.bmrc.ox.ac.uk> (0, '', '') > The full traceback is: > Traceback (most recent call last): > File "/tmp/ansible_ovirt_storage_domain_payload_w8oO0Y/__main__.py", > line 664, in main > storage_domains_module.post_create_check(sd_id) > File "/tmp/ansible_ovirt_storage_domain_payload_w8oO0Y/__main__.py", > line 526, in post_create_check > id=storage_domain.id, > File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line > 3053, in add > return self._internal_add(storage_domain, headers, query, wait) > File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line > 232, in _internal_add > return future.wait() if wait else future > File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, > in wait > return self._code(response) > File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line > 229, in callback > self._check_fault(response) > File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line > 132, in _check_fault > self._raise_error(response, body) > File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line > 118, in _raise_error > raise error > Error: Fault reason is "Operation Failed". Fault detail is "[]". HTTP > response code is 400. > fatal: [virthyp04.virt.in.bmrc.ox.ac.uk]: FAILED! => { > "changed": false, > "invocation": { > "module_args": { > "backup": null, > "comment": null, > "critical_space_action_blocker": null, > "data_center": "Default", > "description": null, > "destroy": null, > "discard_after_delete": null, > "domain_function": "data", > "fcp": null, > "fetch_nested": false, > "format": null, > "glusterfs": null, > "host": "virthyp04.virt.in.bmrc.ox.ac.uk", > "id": null, > "iscsi": null, > "localfs": null, > "name": "hosted_storage", > "nested_attributes": [], > "nfs": null, > "poll_interval": 3, > "posixfs": null, > "state": "present", > "timeout": 180, > "wait": true, > "warning_low_space": null, > "wipe_after_delete": null > } > }, > "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[]\". > HTTP response code is 400." > } > to retry, use: --limit @/etc/ansible/playbook/ovirt.retry > > PLAY RECAP > ****************************************************************************************** > virthyp04.virt.in.bmrc.ox.ac.uk : ok=216 changed=69 unreachable=0 > failed=1 > > Regards, > Callum > > -- > > Callum Smith > Research Computing Core > Wellcome Trust Centre for Human Genetics > University of Oxford > e. [email protected] > > On 29 Mar 2019, at 17:48, Simone Tiraboschi <[email protected]> wrote: > > > > On Fri, Mar 29, 2019 at 6:14 PM Callum Smith <[email protected]> wrote: > >> So close now: >> >> fatal: [virthyp04.virt.in.bmrc.ox.ac.uk]: FAILED! => {"changed": true, >> "cmd": ["dd", "bs=20480", "count=1", "oflag=direct", >> "if=/var/tmp/localvmDBMVgn/e208e0f9-0f4d-4d0d-9104-10d8a26bfab6", >> "of=/rhev/data-center/mnt/10.141.15.248:_export_virtman_hosted__storage/b4f93b28-1497-44b0-9eaf-5e5e2b71bce8/images/645c4286-71e4-4cce-9049-345903929e1b/e208e0f9-0f4d-4d0d-9104-10d8a26bfab6"], >> "delta": "0:00:00.005134", "end": "2019-03-29 17:04:27.952367", "msg": >> "non-zero return code", "rc": 1, "start": "2019-03-29 17:04:27.947233", >> "stderr": "dd: failed to open >> ‘/rhev/data-center/mnt/10.141.15.248:_export_virtman_hosted__storage/b4f93b28-1497-44b0-9eaf-5e5e2b71bce8/images/645c4286-71e4-4cce-9049-345903929e1b/e208e0f9-0f4d-4d0d-9104-10d8a26bfab6’: >> Permission denied", "stderr_lines": ["dd: failed to open >> ‘/rhev/data-center/mnt/10.141.15.248:_export_virtman_hosted__storage/b4f93b28-1497-44b0-9eaf-5e5e2b71bce8/images/645c4286-71e4-4cce-9049-345903929e1b/e208e0f9-0f4d-4d0d-9104-10d8a26bfab6’: >> Permission denied"], "stdout": "", "stdout_lines": []} >> to retry, use: --limit @/etc/ansible/playbook/ovirt.retry >> >> ls -laZ /rhev/data-center/mnt/10.141.15.248: >> _export_virtman_hosted__storage/b4f93b28-1497-44b0-9eaf-5e5e2b71bce8/images/645c4286-71e4-4cce-9049-345903929e1b/e208e0f9-0f4d-4d0d-9104-10d8a26bfab6 >> -rw-rw----. vdsm kvm system_u:object_r:nfs_t:s0 >> /rhev/data-center/mnt/10.141.15.248: >> _export_virtman_hosted__storage/b4f93b28-1497-44b0-9eaf-5e5e2b71bce8/images/645c4286-71e4-4cce-9049-345903929e1b/e208e0f9-0f4d-4d0d-9104-10d8a26bfab6 >> >> Any ideas on this one? I can almost touch this deployment now... Looking >> at the command it should run as `vdsm` so should work fine, could this be >> SELinux? >> > > Yes, it should be executed as vdsm user: > > https://github.com/oVirt/ovirt-ansible-hosted-engine-setup/blob/master/tasks/create_target_vm/03_hosted_engine_final_tasks.yml#L156 > > Did you tried executing it with -vvv ? > Can you please share the playbook you are using to trigger that role? > > > >> >> Regards, >> Callum >> >> -- >> >> Callum Smith >> Research Computing Core >> Wellcome Trust Centre for Human Genetics >> University of Oxford >> e. [email protected] >> >> On 29 Mar 2019, at 15:50, Callum Smith <[email protected]> wrote: >> >> Guilty, will roll back and try again! >> >> Regards, >> Callum >> >> -- >> >> Callum Smith >> Research Computing Core >> Wellcome Trust Centre for Human Genetics >> University of Oxford >> e. [email protected] >> >> On 29 Mar 2019, at 15:35, Simone Tiraboschi <[email protected]> wrote: >> >> The error comes from here: >> >> TASK [ovirt.hosted_engine_setup : Parse OVF] >> *************************************************************************************************************************************** >> fatal: [virthyp04.virt.in.bmrc.ox.ac.uk]: FAILED! => {"changed": false, >> "msg": "missing parameter(s) required by 'attribute': value"} >> >> but are you really using it with ansible 2.8 alpha 1? >> I'd strongly suggest to switch back to a stable release of ansible which >> is currently 2.7.9. >> >> That one was due to: https://github.com/ansible/ansible/issues/53459 >> In the next ansible build it will be just a warning as for: >> https://github.com/ansible/ansible/pull/54336 >> >> https://github.com/oVirt/ovirt-ansible-hosted-engine-setup/pull/150/files >> already address this on ovirt-ansible-hosted-engine-setup to be >> compatible with future ansible releases. >> >> >> On Fri, Mar 29, 2019 at 3:53 PM Callum Smith <[email protected]> >> wrote: >> >>> The OVF in question is here: >>> >>> <ovf:Envelope ovf:version="0.9" xmlns:ovf=" >>> http://schemas.dmtf.org/ovf/envelope/1/"; xmlns:rasd=" >>> http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData"; >>> xmlns:vssd=" >>> http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_VirtualSystemSettingData"; >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";><References><File >>> ovf:description="074a62d4-44f9-4ffe-a172-2702a9fe96df" >>> ovf:href="6f76686b-199c-4cb3-bbbe-86fc34365745/72bc3948-5d8d-4877-bac8-7db4995045b5" >>> ovf:id="72bc3948-5d8d-4877-bac8-7db4995045b5" ovf:size="54760833024" >>> /></References><Section xsi:type="ovf:NetworkSection_Type"><Info>List of >>> Networks</Info></Section><Section xsi:type="ovf:DiskSection_Type"><Disk >>> ovf:actual_size="51" ovf:boot="true" ovf:disk-interface="VirtIO" >>> ovf:disk-type="System" ovf:diskId="72bc3948-5d8d-4877-bac8-7db4995045b5" >>> ovf:fileRef="6f76686b-199c-4cb3-bbbe-86fc34365745/72bc3948-5d8d-4877-bac8-7db4995045b5" >>> ovf:format="http://www.vmware.com/specifications/vmdk.html#sparse"; >>> ovf:parentRef="" ovf:size="51" >>> ovf:vm_snapshot_id="5f2be758-82d7-4c07-a220-9060e782dc7a" >>> ovf:volume-format="COW" ovf:volume-type="Sparse" >>> ovf:wipe-after-delete="false" /></Section><Content ovf:id="out" >>> xsi:type="ovf:VirtualSystem_Type"><Name>074a62d4-44f9-4ffe-a172-2702a9fe96df</Name><TemplateId>074a62d4-44f9-4ffe-a172-2702a9fe96df</TemplateId><Description>Created >>> by OVABuilder</Description><Domain /><CreationDate>2019/03/19 >>> 08:35:09</CreationDate><TimeZone >>> /><IsAutoSuspend>false</IsAutoSuspend><VmType>1</VmType><default_display_type>0</default_display_type><default_boot_sequence>1</default_boot_sequence><Section >>> ovf:id="074a62d4-44f9-4ffe-a172-2702a9fe96df" ovf:required="false" >>> xsi:type="ovf:OperatingSystemSection_Type"><Info>Guest >>> OS</Info><Description>OtherLinux</Description></Section><Section >>> xsi:type="ovf:VirtualHardwareSection_Type"><Info>4 CPU, 16384 >>> Memory</Info><System><vssd:VirtualSystemType>RHEVM >>> 4.6.0.163</vssd:VirtualSystemType></System><Item><rasd:Caption>4 virtual >>> CPU</rasd:Caption><rasd:Description>Number of virtual >>> CPU</rasd:Description><rasd:InstanceId>1</rasd:InstanceId><rasd:ResourceType>3</rasd:ResourceType><rasd:num_of_sockets>1</rasd:num_of_sockets><rasd:cpu_per_socket>4</rasd:cpu_per_socket></Item><Item><rasd:Caption>16384 >>> MB of memory</rasd:Caption><rasd:Description>Memory >>> Size</rasd:Description><rasd:InstanceId>2</rasd:InstanceId><rasd:ResourceType>4</rasd:ResourceType><rasd:AllocationUnits>MegaBytes</rasd:AllocationUnits><rasd:VirtualQuantity>16384</rasd:VirtualQuantity></Item><Item><rasd:Caption>Drive >>> 1</rasd:Caption><rasd:InstanceId>72bc3948-5d8d-4877-bac8-7db4995045b5</rasd:InstanceId><rasd:ResourceType>17</rasd:ResourceType><rasd:HostResource>6f76686b-199c-4cb3-bbbe-86fc34365745/72bc3948-5d8d-4877-bac8-7db4995045b5</rasd:HostResource><rasd:Parent>00000000-0000-0000-0000-000000000000</rasd:Parent><rasd:Template>00000000-0000-0000-0000-000000000000</rasd:Template><rasd:ApplicationList >>> /><rasd:StorageId>00000000-0000-0000-0000-000000000000</rasd:StorageId><rasd:StoragePoolId>00000000-0000-0000-0000-000000000000</rasd:StoragePoolId><rasd:CreationDate>2019/03/19 >>> 08:35:09</rasd:CreationDate><rasd:LastModified>2019/03/19 >>> 08:35:09</rasd:LastModified></Item><Item><rasd:Caption>Ethernet 0 >>> rhevm</rasd:Caption><rasd:InstanceId>3</rasd:InstanceId><rasd:ResourceType>10</rasd:ResourceType><rasd:ResourceSubType>3</rasd:ResourceSubType><rasd:Connection>rhevm</rasd:Connection><rasd:Name>eth0</rasd:Name><rasd:speed>1000</rasd:speed></Item><Item><rasd:Caption>Graphics</rasd:Caption><rasd:InstanceId>5</rasd:InstanceId><rasd:ResourceType>20</rasd:ResourceType><rasd:VirtualQuantity>1</rasd:VirtualQuantity></Item></Section></Content></ovf:Envelope> >>> >>> Regards, >>> Callum >>> >>> -- >>> >>> Callum Smith >>> Research Computing Core >>> Wellcome Trust Centre for Human Genetics >>> University of Oxford >>> e. [email protected] >>> >>> On 29 Mar 2019, at 14:42, Callum Smith <[email protected]> wrote: >>> >>> Ok so we're getting very close now, weird OVF error: >>> >>> Full ansible log attached >>> >>> Only error in the engine.log looks normal/expected to me: >>> 2019-03-29 14:32:44,370Z ERROR >>> [org.ovirt.engine.core.bll.pm.FenceProxyLocator] >>> (EE-ManagedThreadFactory-engineScheduled-Thread-71) [4405d6db] Can not run >>> fence action on host 'vir >>> thyp04.virt.in.bmrc.ox.ac.uk', no suitable proxy host was found. >>> >>> <ovirt.20190329133000.ansible.log> >>> >>> Feeling damn close to success here, but have managed to replicate this >>> issue twice re-running the installer. >>> >>> Regards, >>> Callum >>> >>> -- >>> >>> Callum Smith >>> Research Computing Core >>> Wellcome Trust Centre for Human Genetics >>> University of Oxford >>> e. [email protected] >>> >>> On 29 Mar 2019, at 11:50, Simone Tiraboschi <[email protected]> wrote: >>> >>> >>> >>> On Fri, Mar 29, 2019 at 12:36 PM Callum Smith <[email protected]> >>> wrote: >>> >>>> ip link del ovirtmgmt has done the job.... >>>> >>>> Another issue, but this is likely due to randomised MAC addresses: >>>> fatal: [virthyp04.virt.in.bmrc.ox.ac.uk]: FAILED! => {"changed": true, >>>> "cmd": ["virt-install", "-n", "HostedEngineLocal", "--os-variant", "rhel7", >>>> "--virt-type", "kvm", "--memory", "4096", "--vcpus", "64", "--network", >>>> "network=default,mac=fe:58:6c:da:1e:cc,model=virtio", "--disk", >>>> "/var/tmp/localvmOCYiyF/images/6f76686b-199c-4cb3-bbbe-86fc34365745/72bc3948-5d8d-4877-bac8-7db4995045b5", >>>> "--import", "--disk", "path=/var/tmp/localvmOCYiyF/seed.iso,device=cdrom", >>>> "--noautoconsole", "--rng", "/dev/random", "--graphics", "vnc", "--video", >>>> "vga", "--sound", "none", "--controller", "usb,model=none", "--memballoon", >>>> "none", "--boot", "hd,menu=off", "--clock", "kvmclock_present=yes"], >>>> "delta": "0:00:01.355834", "end": "2019-03-29 11:31:02.100143", "msg": >>>> "non-zero return code", "rc": 1, "start": "2019-03-29 11:31:00.744309", >>>> "stderr": "ERROR unsupported configuration: Unable to use MAC address >>>> starting with reserved value 0xFE - 'fe:58:6c:da:1e:cc' - \nDomain >>>> installation does not appear to have been successful.\nIf it was, you can >>>> restart your domain by running:\n virsh --connect qemu:///system start >>>> HostedEngineLocal\notherwise, please restart your installation.", >>>> "stderr_lines": ["ERROR unsupported configuration: Unable to use MAC >>>> address starting with reserved value 0xFE - 'fe:58:6c:da:1e:cc' - ", >>>> "Domain installation does not appear to have been successful.", "If it was, >>>> you can restart your domain by running:", " virsh --connect >>>> qemu:///system start HostedEngineLocal", "otherwise, please restart >>>> your installation."], "stdout": "\nStarting install...", "stdout_lines": >>>> ["", "Starting install..."]} >>>> >>>> Seems it doesn't take into account reserved values when generating. >>>> >>> >>> If not specified by the user, >>> a random unicast MAC address is randomly generated here: >>> >>> https://github.com/oVirt/ovirt-ansible-hosted-engine-setup/blob/master/tasks/pre_checks/validate_mac_address.yml#L4 >>> >>> for sure it's unicast but maybe we should make it more robust on >>> reserved values. >>> Simply try again for now; I'll open an issue to track it, thanks! >>> >>> >>>> >>>> I hope this feedback is valuable - I have a good feeling about the >>>> current deploy otherwise though. >>>> >>>> Regards, >>>> Callum >>>> >>>> -- >>>> >>>> Callum Smith >>>> Research Computing Core >>>> Wellcome Trust Centre for Human Genetics >>>> University of Oxford >>>> e. [email protected] >>>> >>>> On 29 Mar 2019, at 11:01, Simone Tiraboschi <[email protected]> >>>> wrote: >>>> >>>> >>>> >>>> On Fri, Mar 29, 2019 at 11:56 AM Callum Smith <[email protected]> >>>> wrote: >>>> >>>>> Dear Simone, >>>>> >>>>> It doesn't seem to want to work: >>>>> >>>>> # Settings >>>>> he_fqdn: "he.virt.in.bmrc.ox.ac.uk" >>>>> he_ansible_host_name: "virthyp04.virt.in.bmrc.ox.ac.uk" >>>>> he_admin_password: <snip> >>>>> he_appliance_password: <snip> >>>>> >>>>> # Resources >>>>> he_mem_size_MB: "4096" >>>>> >>>>> # Storage >>>>> he_domain_type: "nfs" >>>>> he_storage_domain_addr: <snip> >>>>> he_storage_domain_path: <snip> >>>>> >>>>> # Network >>>>> he_vm_ip_addr: "10.141.31.240" >>>>> he_vm_ip_prefix: "20" >>>>> he_dns_addr: ["10.141.31.251","10.141.31.252","10.141.31.253"] >>>>> he_default_gateway_4: "10.141.31.254" >>>>> he_gateway: he_default_gateway_4 >>>>> he_force_ip4: true >>>>> he_bridge_if: bond0.910 >>>>> #he_just_collect_network_interfaces: true >>>>> >>>>> # Email >>>>> he_smtp_port: 25 >>>>> he_smtp_server: smtp.ox.ac.uk >>>>> he_dest_email: [email protected] >>>>> he_source_email: [email protected] >>>>> >>>>> # Ansible Stuff >>>>> ansible_ssh_user: root >>>>> ansible_become: false >>>>> >>>>> host_key_checking: false >>>>> >>>>> I've attached the output of the ansible command as a log file, this is >>>>> what happens when the IF bond0.910 is assigned the IP and `ovirtmgmt` is >>>>> not defined on the host. >>>>> >>>> >>>> >>>> TASK [ovirt.hosted_engine_setup : debug] >>>> ******************************************************************************************************************************************* >>>> ok: [virthyp04.virt.in.bmrc.ox.ac.uk] => { >>>> "target_address_v4": { >>>> "changed": true, >>>> "cmd": "ip addr show ovirtmgmt | grep 'inet ' | cut -d' ' -f6 | >>>> cut -d'/' -f1", >>>> "delta": "0:00:00.008744", >>>> "end": "2019-03-29 10:26:07.510481", >>>> "failed": false, >>>> "rc": 0, >>>> "start": "2019-03-29 10:26:07.501737", >>>> "stderr": "", >>>> "stderr_lines": [], >>>> "stdout": "", >>>> "stdout_lines": [] >>>> } >>>> } >>>> >>>> according to the logs ovirtmgmt is still there. >>>> >>>> can you please share the output of 'ip a'? >>>> >>>> >>>>> >>>>> Regards, >>>>> Callum >>>>> >>>>> -- >>>>> >>>>> Callum Smith >>>>> Research Computing Core >>>>> Wellcome Trust Centre for Human Genetics >>>>> University of Oxford >>>>> e. [email protected] >>>>> >>>>> On 28 Mar 2019, at 16:23, Simone Tiraboschi <[email protected]> >>>>> wrote: >>>>> >>>>> >>>>> >>>>> On Thu, Mar 28, 2019 at 1:44 PM Callum Smith <[email protected]> >>>>> wrote: >>>>> >>>>> Dear Simone, >>>>> >>>>> This is my experience too, but I'm now hitting this error on the >>>>> hosted-engine install at the part where it registers the hypervisor as the >>>>> first host in the engine: >>>>> >>>>> 2019-03-28 12:40:50,025Z INFO >>>>> [org.ovirt.engine.core.bll.host.HostConnectivityChecker] >>>>> (EE-ManagedThreadFactory-engine-Thread-1) [49f371c1] Engine managed to >>>>> communicate with VDSM >>>>> agent on host 'virthyp04.virt.in.bmrc.ox.ac.uk' with address ' >>>>> virthyp04.virt.in.bmrc.ox.ac.uk' >>>>> ('db571f8a-fc85-40d3-b86f-c0038e3cd7e7') >>>>> 2019-03-28 12:40:53,111Z WARN >>>>> [org.ovirt.engine.core.bll.network.NetworkConfigurator] >>>>> (EE-ManagedThreadFactory-engine-Thread-1) [49f371c1] Failed to find a >>>>> valid >>>>> interface for the >>>>> management network of host virthyp04.virt.in.bmrc.ox.ac.uk. If the >>>>> interface ovirtmgmt is a bridge, it should be torn-down manually. >>>>> 2019-03-28 12:40:53,111Z ERROR >>>>> [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] >>>>> (EE-ManagedThreadFactory-engine-Thread-1) [49f371c1] Exception: >>>>> org.ovirt.engine.cor >>>>> e.bll.network.NetworkConfigurator$NetworkConfiguratorException: >>>>> Interface ovirtmgmt is invalid for management network >>>>> >>>>> The host's ovirtmgmt network connection is a statically assigned IP on >>>>> a VLAN on a bond, how should I be configuring this if not manually? >>>>> >>>>> >>>>> If you need to deploy over vlan 123 over bond0 simply configure a >>>>> device exactly called bond0.123 and statically configure your IP address >>>>> there. >>>>> Choose it for hosted-engine deployment, nothing more: ovirtmgmt will >>>>> be automatically created over that device and the vlan ID will be set at >>>>> engine level for the whole management network. >>>>> >>>>> >>>>> >>>>> Regards, >>>>> Callum >>>>> >>>>> -- >>>>> >>>>> Callum Smith >>>>> Research Computing Core >>>>> Wellcome Trust Centre for Human Genetics >>>>> University of Oxford >>>>> e. [email protected] >>>>> >>>>> On 27 Mar 2019, at 17:09, Simone Tiraboschi <[email protected]> >>>>> wrote: >>>>> >>>>> >>>>> >>>>> On Wed, Mar 27, 2019 at 4:27 PM Callum Smith <[email protected]> >>>>> wrote: >>>>> >>>>> It's ok, migrating to 4.3.2 on the oVirt node (from 4.3.0) did the job >>>>> of fixing it. >>>>> >>>>> It is a bug if you intend on using the ovirtmgmt network to deploy >>>>> your ansible from >>>>> >>>>> >>>>> This is a bit tricky: when the engine brings up the host it also >>>>> creates the management bridge and this could lead to a temporary network >>>>> down on the selected interface for the bridge creation time (a couple of >>>>> seconds?) >>>>> I tried it on a LAN and ansible ssh connection always survived but I >>>>> 'm not sure it's always true. >>>>> >>>>> >>>>> , and you need it to have an IP address already on that range! But - >>>>> it works as expected with the ovirtmgmt bridge setup so nothing to worry >>>>> about. >>>>> >>>>> Regards, >>>>> Callum >>>>> >>>>> -- >>>>> >>>>> Callum Smith >>>>> Research Computing Core >>>>> Wellcome Trust Centre for Human Genetics >>>>> University of Oxford >>>>> e. [email protected] >>>>> >>>>> On 27 Mar 2019, at 14:57, Simone Tiraboschi <[email protected]> >>>>> wrote: >>>>> >>>>> >>>>> >>>>> On Wed, Mar 27, 2019 at 3:24 PM Callum Smith <[email protected]> >>>>> wrote: >>>>> >>>>> Dear All, >>>>> >>>>> We're trying to deploy our hosted engine remotely using the ansible >>>>> hosted engine playbook, which has been a rocky road but we're now at the >>>>> point where it's installing, and failing. We've got a pre-defined >>>>> bond/VLAN >>>>> setup for our interface which has the correct bond0 bond0.123 and >>>>> ovirtmgmt >>>>> bridge on top but we're hitting the classic error: >>>>> >>>>> Failed to find a valid in >>>>> terface for the management network of host >>>>> virthyp04.virt.in.bmrc.ox.ac.uk. If the interface ovirtmgmt is a >>>>> bridge, it should be torn-down manually. >>>>> >>>>> Does this bug still exist in the latest (4.3) version, and is >>>>> installing using ansible with this network configuration impossible? >>>>> >>>>> >>>>> I don't think it's a bug; please avoid manually creating ovirtmgmt and >>>>> simply set: >>>>> he_bridge_if: "bond0.123", in ansible variable file and the management >>>>> bridge will be created for you at host-deploy time. >>>>> >>>>> >>>>> >>>>> Regards, >>>>> Callum >>>>> >>>>> -- >>>>> >>>>> Callum Smith >>>>> Research Computing Core >>>>> Wellcome Trust Centre for Human Genetics >>>>> University of Oxford >>>>> e. [email protected] >>>>> >>>>> _______________________________________________ >>>>> Users mailing list -- [email protected] >>>>> To unsubscribe send an email to [email protected] >>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>> oVirt Code of Conduct: >>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>> List Archives: >>>>> https://lists.ovirt.org/archives/list/[email protected]/message/SBOZ6FRBRQKE7FMTEHT6FBKHOW3PIPHK/ >>>>> >>>>> >>> >>> >> _______________________________________________ >> Users mailing list -- [email protected] >> To unsubscribe send an email to [email protected] >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/[email protected]/message/QOGFRHXDVOIM4WA5N4JT7RBSW7PJIYJQ/ >> >> >
_______________________________________________ Users mailing list -- [email protected] To unsubscribe send an email to [email protected] Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/[email protected]/message/UUMPPWKBDSHKFRAM2B2OY6T552AFZVE3/
