[ovirt-users] Recovering from corrupted hosted-engine
Hi Guys,
We had a 2-node self-supported RHEV cluster that we used for our development
environment. We're in the middle of a migration from RHEV over to VMware - one
host has already been converted to VMware and we have actively been running VM
migrations over to VMware. During the migration, we had an extended power
outage and had to improperly shut down the RHEV host. Since bringing it back up
we've not been able to start the hosted-engine. Each time we try to start the
hosted engine we're getting the following messages:
## START
[root@bmrhev01 ~]# hosted-engine --vm-start
VM exists and is down, cleaning up and restarting
VM in WaitForLaunch
[root@bmrhev01 ~]# hosted-engine --vm-status
--== Host bmrhev01 (id: 1) status ==--
conf_on_shared_storage : True
Status up-to-date : True
Hostname : bmrhev01
Host ID: 1
Engine status : {"reason": "vm not running on this host",
"health": "bad", "vm": "down_unexpected", "detail": "unknown"}
Score : 3400
stopped: False
Local maintenance : False
crc32 : 325ff4b3
local_conf_timestamp : 7920
Host timestamp : 7920
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=7920 (Tue Jun 15 14:59:32 2021)
host-id=1
score=3400
vm_conf_refresh_time=7920 (Tue Jun 15 14:59:32 2021)
conf_on_shared_storage=True
maintenance=False
state=EngineStarting
stopped=False
END ##
We've tried rebooting a number of times. Restarting various oVirt services.
Nothing appears to be resolving the issue. At this point, the only thing I
truly care about is migrating the guest VMs over to VMware. There's nothing
else in the RHEV environment I care about. I'm happy to settle with any of the
following, providing they can salvage the guest VMs:
a) Deleting and redeploying the hosted-engine.
b) Abandoning the hosted-engine and some how converting the guest VMs over to
VMware.
I don't have enough experience to know if any of the above is possible or the
repercussions of either of them. Any help from anyone would be very helpful.
I'm sorry to be a leach, rather than a contributor. I'm seriously in trouble
here and a kind heart would be much appreciated.
Thanks, Tim.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/GUFN2F6WRLH772PAGSNRIZDD3ATA7QCC/
[ovirt-users] Re: hosted-engine cannot can't communicate with vm
Hi Harry, In case you're in a similar position to me, and simply looking to stand up your hosted-engine just to migrate to a different hypervisor, I'm going through a similar situation where I wasn't able to get my hosted-engine to boot. Below is my thread: https://lists.ovirt.org/archives/list/users@ovirt.org/thread/GUFN2F6WRLH772PAGSNRIZDD3ATA7QCC/ I've made some progress with exporting the VMs using the method I've linked in my responses, albeit the process has been slow and painful. ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/N6DIKXSDV6ALBD4ZU4KOZPR3XD4M2PYS/
[ovirt-users] Re: Recovering from corrupted hosted-engine
If anyone else runs into this issue, I've been able to recover one of the VMs using the "qemu-img convert -O vmdk vmdiskfile vmdiskfile.vmdk" command outlined in Howard Johnson's response linked below: https://access.redhat.com/discussions/881153 Tried the same actions on a Linux machine, but CentOS is not loading, throwing errors saying it can't find the /root or /boot partitions. I'm hopeful that I can at least salvage a few of the Windows VMs currently held hostage in RHEV. The UUID folder structure is hard to navigate for someone that doesn't know what he's looking at (i.e. me) so I'm likely to have mixed results. Other articles I've come across suggest that it should be possible to redeploy the hosted-engine and import the existing Storage Domain, and the VMs should reappear. I've not had success with this because I haven't been able to make "IP.Address:/path/to/storage/domain" accessible, which is likely what faulted the original hosted-engine. Now that I've nuked the original hosted-engine, I'm stuck down the path I've created for myself, either a) continuing with "qemu-img convert", or b) figuring how to to make "cd IP.Address:/path/to/storage/domain" stop saying: "No such file or directory". If anyone has any tips, that would be much appreciated. ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZQUBTEKGV3VHPYZ47KQ5HSPB4P5C4C53/
[ovirt-users] Re: Recovering from corrupted hosted-engine
OK. To bring this to a close... There were some casualties, but ultimately I was able to salvage a dozen or so virtual machines using the "qemu-img convert -O vmdk vmdiskfile vmdiskfile.vmdk" command. In case anyone else finds themself in a similar situation, this involved the following steps: 1) Give up on the hosted-engine. 2) SSH into RHEV and navigate to the VM storage domain. 3) Go into each UUID folder and run # qemu-img convert -O vmdk vmdiskfile vmdiskfile.vmdk 4) Secure-copy vmdiskfile_converted.vmdk to VMware. 5) SSH onto the VMware host. 6) Run # vmkfstool -i vmdiskfile.vmdk new_file_name.vmdk. 7) Attach new_file_name.vmdk to a new VM in VMware. Make sure "Firmware" is set to "BIOS", under Edit Settings > VM Options > Boot Options. 8) Boot the VMware VM, and Windows should automatically load and you should be good to start removing the RHEV-related VM tools and install the VMware tools. ** Note: Remember my situation involved migrating away from RHEV over to VMware. So, I really wasn't concerned about recovering the hosted-engine. And, prior the hosted-engine going down, in preparation to migration to VMware I had already removed all snapshots from all RHEV VMs. This all made what's below possible, and a lot easier. Also, these steps didn't always work. I was able to recover 75% of Windows VMs. Unfortunately, I experienced a 100% lost for Linux VMs. ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/YY7DIWFCOGKSOD2RLXR2AHAB5OMZGCBJ/
