Hi! This is from ansible logs: 2021-04-27 22:20:38,286+0300 ERROR ansible failed { "ansible_host": "localhost", "ansible_playbook": "/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml", "ansible_result": { "_ansible_no_log": false, "changed": false, "exception": "Traceback (most recent call last):\n File \"/tmp/ansible_ovirt_storage_domain_payload_hlz7j9sk/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_storage_domain.py\", line 804, in main\n File \"/tmp/ansible_ovirt_storage_domain_payload_hlz7j9sk/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/module_utils/ovirt.py\", line 651, in create\n **kwargs\n File \"/usr/lib64/python3.6/site-packages/ovirtsdk4/services.py\", line 26182, in add\n return self._internal_add(storage_domain, headers, query, wait)\n File \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 232, in _internal_add\n return future.wait() if wait else future\n File \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 55, in wait\n return self._code(response)\n File \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 229, in callback\n self._check_fault(response)\n File \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 132, in _check_fault\n self._raise_error(response, body)\n File \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 118, in _raise_error\n raise error\novirtsdk4.Error: Fault reason is \"Operation Failed\". Fault detail is \"[Physical device initialization failed. Please check that the device is empty and accessible by the host.]\". HTTP response code is 400.\n", "invocation": { "module_args": { "backup": null, "comment": null, "critical_space_action_blocker": null, "data_center": "Default", "description": null, "destroy": null, "discard_after_delete": true, "domain_function": "data", "fcp": { "lun_id": "36000d31005b4f6000000000000000029" }, "fetch_nested": false, "format": null, "glusterfs": null, "host": "ovirt02.virt.local", "id": null, "iscsi": null, "localfs": null, "managed_block_storage": null, "name": "hosted_storage", "nested_attributes": [], "nfs": null, "poll_interval": 3, "posixfs": null, "state": "unattached", "timeout": 180, "wait": true, "warning_low_space": null, "wipe_after_delete": null } }, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Physical device initialization failed. Please check that the device is empty and accessible by the host.]\". HTTP response code is 400." }, "ansible_task": "Add Fibre Channel storage domain", "ansible_type": "task", "status": "FAILED",
This seem to happen same time: 2021-04-27 22:20:37,576+0300 INFO (jsonrpc/5) [vdsm.api] FINISH getDeviceList return={'devList': [{'GUID': '36000d31005b4f6000000000000000029', 'capacity': '536870912000', 'devtype': 'FCP', 'fwrev': '0704', 'logicalblocksize': '512', 'pathlist': [], 'pathstatus': [{'physdev': 'sdu', 'state': 'active', 'capacity': '536870912000', 'lun': '3', 'type': 'FCP'}, {'physdev': 'sds', 'state': 'active', 'capacity': '536870912000', 'lun': '3', 'type': 'FCP'}, {'physdev': 'sdv', 'state': 'active', 'capacity': '536870912000', 'lun': '3', 'type': 'FCP'}, {'physdev': 'sdt', 'state': 'active', 'capacity': '536870912000', 'lun': '3', 'type': 'FCP'}], 'physicalblocksize': '4096', 'productID': 'Compellent Vol', 'pvUUID': '', 'pvsize': '', 'serial': 'SCOMPELNTCompellent_Vol_0005b4f6-00000029', 'vendorID': 'COMPELNT', 'vgUUID': '', 'discard_max_bytes': 268435456, 'discard_zeroes_data': 0, 'status': 'unknown'}]} from=::ffff:192.168.222.89,57472, flow_id=676e5ced-7a4f-4a36-8842-cb98a38f5f4b, task_id=792a93ab-5129-4bf2-8ff0-70b5cdf19221 (api:54) 2021-04-27 22:20:37,641+0300 INFO (jsonrpc/4) [vdsm.api] START createVG(vgname='bd3efbef-0178-4a21-9a13-2a293482fdd5', devlist=['36000d31005b4f6000000000000000029'], force=False, options=None) from=::ffff:192.168.222.89,57472, flow_id=6dce1780, task_id=4b269432-9f27-406b-a03b-6dfaaa2d1f67 (api:48) 2021-04-27 22:20:37,729+0300 WARN (jsonrpc/4) [storage.LVM] Command with specific filter failed or returned no data, retrying with a wider filter, cmd=['/sbin/lvm', 'pvcreate', '--config', 'devices { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter=["a|^/dev/mapper/36000d31005b4f6000000000000000029$|", "r|.*|"] hints="none" obtain_device_list_from_udev=0 } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min=50 retain_days=0 }', '--metadatasize', '128m', '--metadatacopies', '2', '--metadataignore', 'y', '/dev/mapper/36000d31005b4f6000000000000000029'] rc=5 out=[] err=[" Can't open /dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted filesystem?", " Can't open /dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted filesystem?"] (lvm:511) 2021-04-27 22:20:37,830+0300 WARN (jsonrpc/4) [storage.LVM] All 2 tries have failed: cmd=['/sbin/lvm', 'pvcreate', '--config', 'devices { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter=["a|^/dev/mapper/360002ac00000000000000261000057b9$|^/dev/mapper/36000d31005b4f600000000000000001a$|^/dev/mapper/36000d31005b4f600000000000000001d$|^/dev/mapper/36000d31005b4f6000000000000000029$|", "r|.*|"] hints="none" obtain_device_list_from_udev=0 } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min=50 retain_days=0 }', '--metadatasize', '128m', '--metadatacopies', '2', '--metadataignore', 'y', '/dev/mapper/36000d31005b4f6000000000000000029'] rc=5 err=[" Can't open /dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted filesystem?", " Can't open /dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted filesystem?"] (lvm:538) 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.LVM] pvcreate failed with rc=5 (lvm:1142) 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.LVM] [], [" Can't open /dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted filesystem?", " Can't open /dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted filesystem?"] (lvm:1143) 2021-04-27 22:20:37,830+0300 INFO (jsonrpc/4) [vdsm.api] FINISH createVG error=Failed to initialize physical device: ("['/dev/mapper/36000d31005b4f6000000000000000029']",) from=::ffff:192.168.222.89,57472, flow_id=6dce1780, task_id=4b269432-9f27-406b-a03b-6dfaaa2d1f67 (api:52) 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.TaskManager.Task] (Task='4b269432-9f27-406b-a03b-6dfaaa2d1f67') Unexpected error (task:880) 2021-04-27 22:20:37,830+0300 INFO (jsonrpc/4) [storage.TaskManager.Task] (Task='4b269432-9f27-406b-a03b-6dfaaa2d1f67') aborting: Task is aborted: 'value=Failed to initialize physical device: ("[\'/dev/mapper/36000d31005b4f6000000000000000029\']",) abortedcode=601' (task:1190) 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.Dispatcher] FINISH createVG error=Failed to initialize physical device: ("['/dev/mapper/36000d31005b4f6000000000000000029']",) (dispatcher:83) I tried clearing the disk but from above it seems that something else is wonky here. Thanks, Juhani On Wed, Apr 28, 2021 at 8:30 AM Yedidyah Bar David <d...@redhat.com> wrote: > > On Tue, Apr 27, 2021 at 10:59 PM Juhani Rautiainen > <juhani.rautiai...@gmail.com> wrote: > > > > Story of the problems continues. Finally shut everything down, got > > storage domains to maintenance and then this happens: > > > > ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is > > "[Physical device initialization failed. Please check that the device > > is empty and accessible by the host.]". HTTP response code is 400. > > [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": > > "Fault reason is \"Operation Failed\". Fault detail is \"[Physical > > device initialization failed. Please check that the device is empty > > and accessible by the host.]\". HTTP response code is 400."} > > Can you please check/share relevant parts of vdsm.log when this happens? > Thanks. > > > > > No amount zeroing device helped. > > > > Next plan: Find out if I can restore backup to standalone server. If > > that fails that's the end of oVirt for me. It worked fine for a couple > > of years but this update hassle is too much. Should have gone until > > the end with 4.3. > > > > Thanks, > > Juhani > > > > On Tue, Apr 27, 2021 at 11:47 AM Juhani Rautiainen > > <juhani.rautiai...@gmail.com> wrote: > > > > > > Hmm. Is it possible that when the other node is still running v4.3 > > > this operation can't be completed as it doesn't know how to do it. > > > > > > Thanks, > > > Juhani > > > > > > On Tue, Apr 27, 2021 at 11:07 AM Juhani Rautiainen > > > <juhani.rautiai...@gmail.com> wrote: > > > > > > > > It seems that it is not supported in oVirt yet? I got this response > > > > when I tried to change master with those storage domain that I have: > > > > > > > > <?xml version="1.0" encoding="UTF-8" standalone="yes"?> > > > > <fault> > > > > <detail>[Cannot switch master storage domain. Switch master > > > > storage domain operation is not supported.]</detail> > > > > <reason>Operation Failed</reason> > > > > </fault> > > > > > > > > So is this really the only way to do this: shutdown everything and put > > > > other storage domains into maintenance? It would have been nice if > > > > this information was in the upgrade guide. It made this seem so easy > > > > and simple... > > > > > > > > Thanks, > > > > Juhani > > > > > > > > On Tue, Apr 27, 2021 at 10:47 AM Juhani Rautiainen > > > > <juhani.rautiai...@gmail.com> wrote: > > > > > > > > > > Thanks this looks like what I'm looking for. I'm still wondering how > > > > > to use this. I have LUN just for new hosted storage. Ansible created > > > > > storage domain to it correctly but just can't activate it. So is the > > > > > idea that I activate this unattached hosted_storage domain and try to > > > > > use API to make it master? I attached a screen shot how it looks > > > > > currently. > > > > > > > > > > > > > > > > > > > > On Tue, Apr 27, 2021 at 10:41 AM Yedidyah Bar David <d...@redhat.com> > > > > > wrote: > > > > > > > > > > > > On Tue, Apr 27, 2021 at 10:15 AM Juhani Rautiainen > > > > > > <juhani.rautiai...@gmail.com> wrote: > > > > > > > > > > > > > > To continue. I noticed that another storage domain took the data > > > > > > > (master) now. I saw one advice that you can force change by > > > > > > > putting > > > > > > > the storage domain to maintenance mode. Problem is that there are > > > > > > > VM's > > > > > > > running on these domains. How is this supposed to work during the > > > > > > > restore? > > > > > > > > > > > > There is a recent change [1] by Shani (Cced) that should allow you > > > > > > to choose another storage domain as master. So you can create a new > > > > > > (temporary?) SD with the correct compatibility level and then set it > > > > > > to master. > > > > > > > > > > > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1576923 > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > Juhani > > > > > > > > > > > > > > On Tue, Apr 27, 2021 at 9:58 AM Juhani Rautiainen > > > > > > > <juhani.rautiai...@gmail.com> wrote: > > > > > > > > > > > > > > > > Hi! > > > > > > > > > > > > > > > > I started the upgrade from 4.3->4.4. Now I'm stuck as restoring > > > > > > > > the > > > > > > > > backup fails to create the correct storage domain for Hosted > > > > > > > > Engine. > > > > > > > > How can I create one? Error from ansible task is: > > > > > > > > > > > > > > > > [ ERROR ] ovirtsdk4.Error: Fault reason is "Operation Failed". > > > > > > > > Fault > > > > > > > > detail is "[Domain format is different from master storage > > > > > > > > domain > > > > > > > > format]". HTTP response code is 400. > > > > > > > > [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, > > > > > > > > "msg": > > > > > > > > "Fault reason is \"Operation Failed\". Fault detail is \"[Domain > > > > > > > > format is different from master storage domain format]\". HTTP > > > > > > > > response code is 400."} > > > > > > > > > > > > > > > > And from UI I can see that what has been created is a data > > > > > > > > domain. Not > > > > > > > > data (master) domain as old one was. Now I'm stuck here. This > > > > > > > > is a > > > > > > > > fibre channel system where I'm trying to do this if it is > > > > > > > > relevant. > > > > > > > > > > > > > > > > What I could find from the logs is this: > > > > > > > > 2021-04-27 09:36:06,925+0300 DEBUG > > > > > > > > otopi.ovirt_hosted_engine_setup.ansible_utils > > > > > > > > ansible_utils._process_output:105 storage_domain_details: > > > > > > > > {'changed': > > > > > > > > False, 'ovirt_storage_domains': [{'href': > > > > > > > > '/ovirt-engine/api/storagedomains/dd52022b-7616-47f6-9534-6f1a4084fdf4', > > > > > > > > 'comment': '', 'description': '', 'id': > > > > > > > > 'dd52022b-7616-47f6-9534-6f1a4084fdf4', 'name': > > > > > > > > 'hosted_storage', > > > > > > > > 'available': 531502202880, 'backup': False, 'block_size': 512, > > > > > > > > 'committed': 0, 'critical_space_action_blocker': 5, > > > > > > > > 'discard_after_delete': True, 'disk_profiles': [], > > > > > > > > 'disk_snapshots': > > > > > > > > [], 'disks': [], 'external_status': 'ok', 'master': False, > > > > > > > > 'permissions': [], 'status': 'unattached', 'storage': {'type': > > > > > > > > 'fcp', > > > > > > > > 'volume_group': {'id': 'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit', > > > > > > > > 'logical_units': [{'discard_max_size': 268435456, > > > > > > > > 'discard_zeroes_data': False, 'id': > > > > > > > > '36000d31005b4f6000000000000000029', 'lun_mapping': 3, 'paths': > > > > > > > > 0, > > > > > > > > 'product_id': 'Compellent Vol', 'serial': > > > > > > > > 'SCOMPELNTCompellent_Vol_0005b4f6-00000029', 'size': > > > > > > > > 536870912000, > > > > > > > > 'storage_domain_id': 'dd52022b-7616-47f6-9534-6f1a4084fdf4', > > > > > > > > 'vendor_id': 'COMPELNT', 'volume_group_id': > > > > > > > > 'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit'}]}}, > > > > > > > > 'storage_connections': > > > > > > > > [], 'storage_format': 'v5', 'supports_discard': True, > > > > > > > > 'supports_discard_zeroes_data': False, 'templates': [], 'type': > > > > > > > > 'data', 'used': 4294967296, 'vms': [], > > > > > > > > 'warning_low_space_indicator': > > > > > > > > 10, 'wipe_after_delete': False}], 'failed': False} > > > > > > > > > > > > > > > > 'master': False? I'm not sure if this creation or check. I > > > > > > > > tried this > > > > > > > > opration twice. I removed the new hosted_storage and remove > > > > > > > > also old > > > > > > > > hosted_engine domain on second try to make sure that it doesn't > > > > > > > > prevent creation of another master. No luck with that. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Juhani > > > > > > > _______________________________________________ > > > > > > > Users mailing list -- users@ovirt.org > > > > > > > To unsubscribe send an email to users-le...@ovirt.org > > > > > > > Privacy Statement: https://www.ovirt.org/privacy-policy.html > > > > > > > oVirt Code of Conduct: > > > > > > > https://www.ovirt.org/community/about/community-guidelines/ > > > > > > > List Archives: > > > > > > > https://lists.ovirt.org/archives/list/users@ovirt.org/message/TZHLFDIYRBQFVDLEVG4SWVZPXSVO7SKK/ > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Didi > > > > > > > > > > > -- > Didi > _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/53Q5AR3BLX6XUKN7D76ITIH65DLSFVAO/