On Wednesday, 28 April 2021 08:57:32 CEST Yedidyah Bar David wrote:
> On Wed, Apr 28, 2021 at 9:29 AM Juhani Rautiainen
> <juhani.rautiai...@gmail.com> wrote:
>
> >
> >
> > Hi!
> >
> >
> >
> > This is from ansible logs:
> > 2021-04-27 22:20:38,286+0300 ERROR ansible failed {
> >
> > "ansible_host": "localhost",
> > "ansible_playbook":
> >
> > "/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml",
> >
> > "ansible_result": {
> >
> > "_ansible_no_log": false,
> > "changed": false,
> > "exception": "Traceback (most recent call last):\n File
> >
> > \"/tmp/ansible_ovirt_storage_domain_payload_hlz7j9sk/ansible_ovirt_storage
> > _domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_
> > storage_domain.py\",
line 804, in main\n File
> > \"/tmp/ansible_ovirt_storage_domain_payload_hlz7j9sk/ansible_ovirt_storage
> > _domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/module_utils/o
> > virt.py\",
line 651, in create\n **kwargs\n File
> > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/services.py\", line
> > 26182, in add\n return self._internal_add(storage_domain, headers,
> > query, wait)\n File
> > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 232,
> > in _internal_add\n return future.wait() if wait else future\n File
> > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 55,
> > in wait\n return self._code(response)\n File
> > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 229,
> > in callback\n self._check_fault(response)\n File
> > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 132,
> > in _check_fault\n self._raise_error(response, body)\n File
> > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 118,
> > in _raise_error\n raise error\novirtsdk4.Error: Fault reason is
> > \"Operation Failed\". Fault detail is \"[Physical device
> > initialization failed. Please check that the device is empty and
> > accessible by the host.]\". HTTP response code is 400.\n",
> >
> > "invocation": {
> >
> > "module_args": {
> >
> > "backup": null,
> > "comment": null,
> > "critical_space_action_blocker": null,
> > "data_center": "Default",
> > "description": null,
> > "destroy": null,
> > "discard_after_delete": true,
> > "domain_function": "data",
> > "fcp": {
> >
> > "lun_id": "36000d31005b4f6000000000000000029"
> >
> > },
> > "fetch_nested": false,
> > "format": null,
> > "glusterfs": null,
> > "host": "ovirt02.virt.local",
> > "id": null,
> > "iscsi": null,
> > "localfs": null,
> > "managed_block_storage": null,
> > "name": "hosted_storage",
> > "nested_attributes": [],
> > "nfs": null,
> > "poll_interval": 3,
> > "posixfs": null,
> > "state": "unattached",
> > "timeout": 180,
> > "wait": true,
> > "warning_low_space": null,
> > "wipe_after_delete": null
> >
> > }
> >
> > },
> > "msg": "Fault reason is \"Operation Failed\". Fault detail is
> >
> > \"[Physical device initialization failed. Please check that the device
> > is empty and accessible by the host.]\". HTTP response code is 400."
> >
> > },
> > "ansible_task": "Add Fibre Channel storage domain",
> > "ansible_type": "task",
> > "status": "FAILED",
> >
> >
> >
> > This seem to happen same time:
> >
> >
> >
> > 2021-04-27 22:20:37,576+0300 INFO (jsonrpc/5) [vdsm.api] FINISH
> > getDeviceList return={'devList': [{'GUID':
> > '36000d31005b4f6000000000000000029', 'capacity': '536870912000',
> > 'devtype': 'FCP', 'fwrev': '0704', 'logicalblocksize': '512',
> > 'pathlist': [], 'pathstatus': [{'physdev': 'sdu', 'state': 'active',
> > 'capacity': '536870912000', 'lun': '3', 'type': 'FCP'}, {'physdev':
> > 'sds', 'state': 'active', 'capacity': '536870912000', 'lun': '3',
> > 'type': 'FCP'}, {'physdev': 'sdv', 'state': 'active', 'capacity':
> > '536870912000', 'lun': '3', 'type': 'FCP'}, {'physdev': 'sdt',
> > 'state': 'active', 'capacity': '536870912000', 'lun': '3', 'type':
> > 'FCP'}], 'physicalblocksize': '4096', 'productID': 'Compellent Vol',
> > 'pvUUID': '', 'pvsize': '', 'serial':
> > 'SCOMPELNTCompellent_Vol_0005b4f6-00000029', 'vendorID': 'COMPELNT',
> > 'vgUUID': '', 'discard_max_bytes': 268435456, 'discard_zeroes_data':
> > 0, 'status': 'unknown'}]} from=::ffff:192.168.222.89,57472,
> > flow_id=676e5ced-7a4f-4a36-8842-cb98a38f5f4b,
> > task_id=792a93ab-5129-4bf2-8ff0-70b5cdf19221 (api:54)
> > 2021-04-27 22:20:37,641+0300 INFO (jsonrpc/4) [vdsm.api] START
> > createVG(vgname='bd3efbef-0178-4a21-9a13-2a293482fdd5',
> > devlist=['36000d31005b4f6000000000000000029'], force=False,
> > options=None) from=::ffff:192.168.222.89,57472, flow_id=6dce1780,
> > task_id=4b269432-9f27-406b-a03b-6dfaaa2d1f67 (api:48)
> > 2021-04-27 22:20:37,729+0300 WARN (jsonrpc/4) [storage.LVM] Command
> > with specific filter failed or returned no data, retrying with a wider
> > filter, cmd=['/sbin/lvm', 'pvcreate', '--config', 'devices {
> > preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
> > write_cache_state=0 disable_after_error_count=3
> > filter=["a|^/dev/mapper/36000d31005b4f6000000000000000029$|", "r|.*|"]
> >
> > hints="none" obtain_device_list_from_udev=0 } global {
> >
> > locking_type=1 prioritise_write_locks=1 wait_for_locks=1
> > use_lvmetad=0 } backup { retain_min=50 retain_days=0 }',
> > '--metadatasize', '128m', '--metadatacopies', '2', '--metadataignore',
> > 'y', '/dev/mapper/36000d31005b4f6000000000000000029'] rc=5 out=[]
> > err=[" Can't open /dev/mapper/36000d31005b4f6000000000000000029
> > exclusively. Mounted filesystem?", " Can't open
> > /dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted
> > filesystem?"] (lvm:511)
>
>
> It seems like this ^^^ is the problem. I suggest to check around e.g.
> with lsof etc., and/or search the net for something like
This is more likely caused by old signatures on the device. You should try to
wipe all the signatures with
wipefs -a /dev/mapper/36000d31005b4f6000000000000000029
of with
kpartx -d /dev/mapper/36000d31005b4f6000000000000000029
and try again
> "pvcreate" "Can't open" "exclusively. Mounted filesystem?"
>
> Best regards,
>
>
> > 2021-04-27 22:20:37,830+0300 WARN (jsonrpc/4) [storage.LVM] All 2
> > tries have failed: cmd=['/sbin/lvm', 'pvcreate', '--config', 'devices
> > { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
> > write_cache_state=0 disable_after_error_count=3
> > filter=["a|^/dev/mapper/360002ac00000000000000261000057b9$|^/dev/mapper/36
> > 000d31005b4f600000000000000001a$|^/dev/mapper/36000d31005b4f60000000000000
> > 0001d$|^/dev/mapper/36000d31005b4f6000000000000000029$|",
"r|.*|"]
> > hints="none" obtain_device_list_from_udev=0 } global { locking_type=1
> > prioritise_write_locks=1 wait_for_locks=1
> > use_lvmetad=0 } backup { retain_min=50 retain_days=0 }',
> > '--metadatasize', '128m', '--metadatacopies', '2', '--metadataignore',
> > 'y', '/dev/mapper/36000d31005b4f6000000000000000029'] rc=5 err=["
> > Can't open /dev/mapper/36000d31005b4f6000000000000000029 exclusively.
> > Mounted filesystem?", " Can't open
> > /dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted
> > filesystem?"] (lvm:538)
> > 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.LVM] pvcreate
> > failed with rc=5 (lvm:1142)
> > 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.LVM] [], ["
> > Can't open /dev/mapper/36000d31005b4f6000000000000000029 exclusively.
> > Mounted filesystem?", " Can't open
> > /dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted
> > filesystem?"] (lvm:1143)
> > 2021-04-27 22:20:37,830+0300 INFO (jsonrpc/4) [vdsm.api] FINISH
> > createVG error=Failed to initialize physical device:
> > ("['/dev/mapper/36000d31005b4f6000000000000000029']",)
> > from=::ffff:192.168.222.89,57472, flow_id=6dce1780,
> > task_id=4b269432-9f27-406b-a03b-6dfaaa2d1f67 (api:52)
> > 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4)
> > [storage.TaskManager.Task]
> > (Task='4b269432-9f27-406b-a03b-6dfaaa2d1f67') Unexpected error
> > (task:880)
> > 2021-04-27 22:20:37,830+0300 INFO (jsonrpc/4)
> > [storage.TaskManager.Task]
> > (Task='4b269432-9f27-406b-a03b-6dfaaa2d1f67') aborting: Task is
> > aborted: 'value=Failed to initialize physical device:
> > ("[\'/dev/mapper/36000d31005b4f6000000000000000029\']",)
> > abortedcode=601' (task:1190)
> > 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.Dispatcher]
> > FINISH createVG error=Failed to initialize physical device:
> > ("['/dev/mapper/36000d31005b4f6000000000000000029']",) (dispatcher:83)
> >
> >
> >
> > I tried clearing the disk but from above it seems that something else
> > is wonky here.
> >
> >
> >
> > Thanks,
> > Juhani
> >
> >
> >
> > On Wed, Apr 28, 2021 at 8:30 AM Yedidyah Bar David <d...@redhat.com>
> > wrote:
>
> > >
> > >
> > > On Tue, Apr 27, 2021 at 10:59 PM Juhani Rautiainen
> > > <juhani.rautiai...@gmail.com> wrote:
> > >
> > > >
> > > >
> > > > Story of the problems continues. Finally shut everything down, got
> > > > storage domains to maintenance and then this happens:
> > > >
> > > >
> > > >
> > > > ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is
> > > > "[Physical device initialization failed. Please check that the device
> > > > is empty and accessible by the host.]". HTTP response code is 400.
> > > > [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg":
> > > > "Fault reason is \"Operation Failed\". Fault detail is \"[Physical
> > > > device initialization failed. Please check that the device is empty
> > > > and accessible by the host.]\". HTTP response code is 400."}
> > >
> > >
> > >
> > > Can you please check/share relevant parts of vdsm.log when this
> > > happens?
> > > Thanks.
> > >
> > >
> > >
> > > >
> > > >
> > > > No amount zeroing device helped.
> > > >
> > > >
> > > >
> > > > Next plan: Find out if I can restore backup to standalone server. If
> > > > that fails that's the end of oVirt for me. It worked fine for a
> > > > couple
> > > > of years but this update hassle is too much. Should have gone until
> > > > the end with 4.3.
> > > >
> > > >
> > > >
> > > > Thanks,
> > > > Juhani
> > > >
> > > >
> > > >
> > > > On Tue, Apr 27, 2021 at 11:47 AM Juhani Rautiainen
> > > > <juhani.rautiai...@gmail.com> wrote:
> > > >
> > > > >
> > > > >
> > > > > Hmm. Is it possible that when the other node is still running v4.3
> > > > > this operation can't be completed as it doesn't know how to do it.
> > > > >
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Juhani
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Apr 27, 2021 at 11:07 AM Juhani Rautiainen
> > > > > <juhani.rautiai...@gmail.com> wrote:
> > > > >
> > > > > >
> > > > > >
> > > > > > It seems that it is not supported in oVirt yet? I got this
> > > > > > response
> > > > > > when I tried to change master with those storage domain that I
> > > > > > have:
> > > > > >
> > > > > >
> > > > > >
> > > > > > <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> > > > > > <fault>
> > > > > >
> > > > > > <detail>[Cannot switch master storage domain. Switch master
> > > > > >
> > > > > > storage domain operation is not supported.]</detail>
> > > > > >
> > > > > > <reason>Operation Failed</reason>
> > > > > >
> > > > > > </fault>
> > > > > >
> > > > > >
> > > > > >
> > > > > > So is this really the only way to do this: shutdown everything and
> > > > > > put
> > > > > > other storage domains into maintenance? It would have been nice
> > > > > > if
> > > > > > this information was in the upgrade guide. It made this seem so
> > > > > > easy
> > > > > > and simple...
> > > > > >
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > Juhani
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Apr 27, 2021 at 10:47 AM Juhani Rautiainen
> > > > > > <juhani.rautiai...@gmail.com> wrote:
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Thanks this looks like what I'm looking for. I'm still wondering
> > > > > > > how
> > > > > > > to use this. I have LUN just for new hosted storage. Ansible
> > > > > > > created
> > > > > > > storage domain to it correctly but just can't activate it. So is
> > > > > > > the
> > > > > > > idea that I activate this unattached hosted_storage domain and
> > > > > > > try to
> > > > > > > use API to make it master? I attached a screen shot how it
> > > > > > > looks
> > > > > > > currently.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Apr 27, 2021 at 10:41 AM Yedidyah Bar David
> > > > > > > <d...@redhat.com> wrote:
> > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Apr 27, 2021 at 10:15 AM Juhani Rautiainen
> > > > > > > > <juhani.rautiai...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > To continue. I noticed that another storage domain took the
> > > > > > > > > data
> > > > > > > > > (master) now. I saw one advice that you can force change by
> > > > > > > > > putting
> > > > > > > > > the storage domain to maintenance mode. Problem is that
> > > > > > > > > there are VM's
> > > > > > > > > running on these domains. How is this supposed to work
> > > > > > > > > during the
> > > > > > > > > restore?
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > There is a recent change [1] by Shani (Cced) that should allow
> > > > > > > > you
> > > > > > > > to choose another storage domain as master. So you can create
> > > > > > > > a new
> > > > > > > > (temporary?) SD with the correct compatibility level and then
> > > > > > > > set it
> > > > > > > > to master.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1576923
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Juhani
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Apr 27, 2021 at 9:58 AM Juhani Rautiainen
> > > > > > > > > <juhani.rautiai...@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hi!
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I started the upgrade from 4.3->4.4. Now I'm stuck as
> > > > > > > > > > restoring the
> > > > > > > > > > backup fails to create the correct storage domain for
> > > > > > > > > > Hosted Engine.
> > > > > > > > > > How can I create one? Error from ansible task is:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > [ ERROR ] ovirtsdk4.Error: Fault reason is "Operation
> > > > > > > > > > Failed". Fault
> > > > > > > > > > detail is "[Domain format is different from master storage
> > > > > > > > > > domain
> > > > > > > > > > format]". HTTP response code is 400.
> > > > > > > > > > [ ERROR ] fatal: [localhost]: FAILED! => {"changed":
> > > > > > > > > > false, "msg":
> > > > > > > > > > "Fault reason is \"Operation Failed\". Fault detail is
> > > > > > > > > > \"[Domain
> > > > > > > > > > format is different from master storage domain format]\".
> > > > > > > > > > HTTP
> > > > > > > > > > response code is 400."}
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > And from UI I can see that what has been created is a data
> > > > > > > > > > domain. Not
> > > > > > > > > > data (master) domain as old one was. Now I'm stuck here.
> > > > > > > > > > This is a
> > > > > > > > > > fibre channel system where I'm trying to do this if it is
> > > > > > > > > > relevant.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > What I could find from the logs is this:
> > > > > > > > > > 2021-04-27 09:36:06,925+0300 DEBUG
> > > > > > > > > > otopi.ovirt_hosted_engine_setup.ansible_utils
> > > > > > > > > > ansible_utils._process_output:105 storage_domain_details:
> > > > > > > > > > {'changed':
> > > > > > > > > > False, 'ovirt_storage_domains': [{'href':
> > > > > > > > > > '/ovirt-engine/api/storagedomains/dd52022b-7616-47f6-9534-
> > > > > > > > > > 6f1a4084fdf4',
> > > > > > > > > > 'comment': '', 'description': '', 'id':
> > > > > > > > > > 'dd52022b-7616-47f6-9534-6f1a4084fdf4', 'name':
> > > > > > > > > > 'hosted_storage',
> > > > > > > > > > 'available': 531502202880, 'backup': False, 'block_size':
> > > > > > > > > > 512,
> > > > > > > > > > 'committed': 0, 'critical_space_action_blocker': 5,
> > > > > > > > > > 'discard_after_delete': True, 'disk_profiles': [],
> > > > > > > > > > 'disk_snapshots':
> > > > > > > > > > [], 'disks': [], 'external_status': 'ok', 'master':
> > > > > > > > > > False,
> > > > > > > > > > 'permissions': [], 'status': 'unattached', 'storage':
> > > > > > > > > > {'type': 'fcp',
> > > > > > > > > > 'volume_group': {'id':
> > > > > > > > > > 'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit',
> > > > > > > > > > 'logical_units': [{'discard_max_size': 268435456,
> > > > > > > > > > 'discard_zeroes_data': False, 'id':
> > > > > > > > > > '36000d31005b4f6000000000000000029', 'lun_mapping': 3,
> > > > > > > > > > 'paths': 0,
> > > > > > > > > > 'product_id': 'Compellent Vol', 'serial':
> > > > > > > > > > 'SCOMPELNTCompellent_Vol_0005b4f6-00000029', 'size':
> > > > > > > > > > 536870912000,
> > > > > > > > > > 'storage_domain_id':
> > > > > > > > > > 'dd52022b-7616-47f6-9534-6f1a4084fdf4',
> > > > > > > > > > 'vendor_id': 'COMPELNT', 'volume_group_id':
> > > > > > > > > > 'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit'}]}},
> > > > > > > > > > 'storage_connections':
> > > > > > > > > > [], 'storage_format': 'v5', 'supports_discard': True,
> > > > > > > > > > 'supports_discard_zeroes_data': False, 'templates': [],
> > > > > > > > > > 'type':
> > > > > > > > > > 'data', 'used': 4294967296, 'vms': [],
> > > > > > > > > > 'warning_low_space_indicator':
> > > > > > > > > > 10, 'wipe_after_delete': False}], 'failed': False}
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 'master': False? I'm not sure if this creation or check. I
> > > > > > > > > > tried this
> > > > > > > > > > opration twice. I removed the new hosted_storage and
> > > > > > > > > > remove also old
> > > > > > > > > > hosted_engine domain on second try to make sure that it
> > > > > > > > > > doesn't
> > > > > > > > > > prevent creation of another master. No luck with that.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Juhani
> > > > > > > > >
> > > > > > > > > _______________________________________________
> > > > > > > > > Users mailing list -- users@ovirt.org
> > > > > > > > > To unsubscribe send an email to users-le...@ovirt.org
> > > > > > > > > Privacy Statement:
> > > > > > > > > https://www.ovirt.org/privacy-policy.html
> > > > > > > > > oVirt Code of Conduct:
> > > > > > > > > https://www.ovirt.org/community/about/community-guidelines/
> > > > > > > > >
List Archives:
> > > > > > > > > https://lists.ovirt.org/archives/list/users@ovirt.org/messa
> > > > > > > > > ge/TZHLFDIYRBQFVDLEVG4SWVZPXSVO7SKK/> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Didi
> > > > > > > >
> > > > > > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > > --
> > > Didi
> > >
> > >
> >
> >
>
>
>
> --
> Didi
> _______________________________________________
> Users mailing list -- users@ovirt.org
> To unsubscribe send an email to users-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/ List Archives:
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/2R53V3IMD5JKK
> B22A3G6OWME3GXDDUFW/
signature.asc
Description: This is a digitally signed message part.
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ATGU42QKVE3WWS3WRFZHZ6SNZBLYJSPF/
