On Sun, Oct 18, 2015 at 10:14 PM, Nir Soffer <nsof...@redhat.com> wrote: > On Sun, Oct 18, 2015 at 7:00 PM, Jaret Garcia <jaret.gar...@packet.mx> wrote: >> Hi everyone, >> >> Afew weeks ago we had a problem with the SPM and all host in the cluster got >> stocked in contending, we restarted hosts one by one, and the issue was >> solved. Howerver we didn't notice that one server even it never stop >> running, it changed its state some way and then no changes could be done to >> the VM, we tried to add more RAM and we saw the message "Cannot run VM. This >> VM is not managed by the engine", > > I would open a bug about this, and attach engine and vdsm logs showing the > timeframe of this event. > >> so we ssh the VM an send it to reboot, and >> once we did that the VM never came back > > Sure, if engine does not know this vm, it will never restart it. The > libvirt vm is not > persistent, engine is keeping the vm info in the engine database, and keeps > the > vm up on some host. > >> , we still see the VM in the engine >> administration but it does not show any information regarding to network, >> disk, and so. > > Please attach engine db dump to the bug, to understand what is "does not show > any information" > >> We created another VM to replace the services in the one we >> lost, however we need to recover the files in the lost VM, we believe the >> image should be in the storage but we haven't found a way to recover it, >> some time ago we came across a similar situation but at that time it was a >> NFS data domain, so it was easier for us to go inside the storage server an >> search for the VM ID to scp the image and mount it somewhere else, this time >> the storage is iscsi and even we found that the hosts mount the target in >> /rhev/data-center/mnt/blockSD/ we only see there the active images for the >> cluster, can anyone point us how we can recover the lost image? We know the >> VM ID and the Disk ID from Ovirt. > > To recover the images, you need the image id. If you don't see it in the > engine > ui, you can try to search in the engine database. > (Adding Maor to help with finding the image id in the database) > > The pool id can be found on the host in /rhev/data-center - there > should be one directory, > its name is the pool id. If you have more than one, use the one which > is not empty. > > # Assuming this value (taken from my test setup) > > pool_id = 591475db-6fa9-455d-9c05-7f6e30fb06d5 > image_id = 5b10b1b9-ee82-46ee-9f3d-3659d37e4851 > > Once you found the image id, do: > > # Update lvm metadata daemon > > pvscan --cache > > # Find the volumes > > # lvs -o lv_name,vg_name,tags | awk '/IU_<image_id>/ {print $1,$2}' > 2782e797-e49a-4364-99d7-d7544a42e939 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 > 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 > > Now we know that: > domain_id = 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 > > # Activate the lvs > > lvchange -ay > 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/2782e797-e49a-4364-99d7-d7544a42e939 > lvchange -ay > 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 > > # Find the top volume by running qemu-img info on all the lvs > > # qemu-img info > /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/2782e797-e49a-4364-99d7-d7544a42e939 > image: > /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/2782e797-e49a-4364-99d7-d7544a42e939 > file format: qcow2 > virtual size: 8.0G (8589934592 bytes) > disk size: 0 > cluster_size: 65536 > Format specific information: > compat: 0.10 > > # qemu-img info > /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 > image: > /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 > file format: qcow2 > virtual size: 8.0G (8589934592 bytes) > disk size: 0 > cluster_size: 65536 > backing file: > ../5b10b1b9-ee82-46ee-9f3d-3659d37e4851/2782e797-e49a-4364-99d7-d7544a42e939 > (actual path: > /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/../5b10b1b9-ee82-46ee-9f3d-3659d37e4851/2782e797-e49a-4364-99d7-d7544a42e939) > backing file format: qcow2 > Format specific information: > compat: 0.10 > > The top volume is the one with the largest number of items in the > "backing file" value.
Correction: using the backing file, you can see the parent of each volume. The volume without the backing file is the base volume. The top volume is the volume which is not parent of any other volume. Here is an example with 3 volumes: # qemu-img info /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/2782e797-e49a-4364-99d7-d7544a42e939 image: /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/2782e797-e49a-4364-99d7-d7544a42e939 file format: qcow2 virtual size: 8.0G (8589934592 bytes) disk size: 0 cluster_size: 65536 Format specific information: compat: 0.10 This is the base volume. # qemu-img info /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 image: /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 file format: qcow2 virtual size: 8.0G (8589934592 bytes) disk size: 0 cluster_size: 65536 backing file: ../5b10b1b9-ee82-46ee-9f3d-3659d37e4851/2782e797-e49a-4364-99d7-d7544a42e939 (actual path: /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/../5b10b1b9-ee82-46ee-9f3d-3659d37e4851/2782e797-e49a-4364-99d7-d7544a42e939) backing file format: qcow2 Format specific information: compat: 0.10 This volume parent is 2782e797-e49a-4364-99d7-d7544a42e939 (the base volume) # qemu-img info /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/9de6a73e-49a6-45e6-b1aa-bc85e630bf39 image: /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/9de6a73e-49a6-45e6-b1aa-bc85e630bf39 file format: qcow2 virtual size: 8.0G (8589934592 bytes) disk size: 0 cluster_size: 65536 backing file: ../5b10b1b9-ee82-46ee-9f3d-3659d37e4851/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 (actual path: /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/../5b10b1b9-ee82-46ee-9f3d-3659d37e4851/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12) backing file format: qcow2 Format specific information: compat: 0.10 This volume parent is 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 (the volume above) So this is the top volume, which can be used to copy the volume data with qemu-img convert. Another way to find this info is using the volume metadata in /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/metadata but it may be stale; the canonical source of information is the qcow image reported by qemu-img. An easier way to get the information, is using getVolumesList - but this uses vdsm metadata, which may be stale (in disaster recovery context). # vdsClient -s 0 getVolumesList 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 591475db-6fa9-455d-9c05-7f6e30fb06d5 5b10b1b9-ee82-46ee-9f3d-3659d37e4851 9de6a73e-49a6-45e6-b1aa-bc85e630bf39 : Parent is 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 2782e797-e49a-4364-99d7-d7544a42e939 : {"DiskAlias":"test_Disk1","DiskDescription":""}. 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 : Parent is 2782e797-e49a-4364-99d7-d7544a42e939 In ovirt 3.6, there is even easier way, using vdsm-tool dump-volumes-chains. This is based on getVolumesList, so it vdsm metadata is broken, you should use the lower level qemu-img info. # vdsm-tool dump-volume-chains 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 Images volume chains (base volume first) image: 55b41fbd-5e22-4f9d-b72f-aa7af9d7ccb8 - 78f22775-916c-4e72-8c5b-9917734b26da status: OK, voltype: SHARED, format: COW, legality: LEGAL, type: SPARSE image: 5b10b1b9-ee82-46ee-9f3d-3659d37e4851 - 2782e797-e49a-4364-99d7-d7544a42e939 status: OK, voltype: INTERNAL, format: COW, legality: LEGAL, type: SPARSE - 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 status: OK, voltype: INTERNAL, format: COW, legality: LEGAL, type: SPARSE - 9de6a73e-49a6-45e6-b1aa-bc85e630bf39 status: OK, voltype: LEAF, format: COW, legality: LEGAL, type: SPARSE image: 7ea9086c-c82d-405b-aabc-2d66f2106f6d - 3f400d56-4412-439c-af43-f379bb5160af status: OK, voltype: LEAF, format: RAW, legality: LEGAL, type: PREALLOCATED image: 8cd92346-555f-4a87-8415-ed681dc7a0a7 - cc56eca9-26c5-4428-8797-b3e7fa7a0c89 status: OK, voltype: LEAF, format: RAW, legality: LEGAL, type: PREALLOCATED > In this case, it is > /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 > > So: > volume_id = 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 > > # Prepare the image to create the links in /rhev/data-center > > In a perfect wold, we could use the path to the lv /dev/vgname/lvname, > but the relative path > used by qemu is based on the directories and symbolic links created > inside /rhev/data-center > the easier way to created them is by preparing the image. > > # vdsClient -s 0 prepareImage 591475db-6fa9-455d-9c05-7f6e30fb06d5 > 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 > 5b10b1b9-ee82-46ee-9f3d-3659d37e4851 > 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 > {'domainID': '6c77adb1-74fc-4fa9-a0ac-3b5a4b789318', > 'imageID': '5b10b1b9-ee82-46ee-9f3d-3659d37e4851', > 'leaseOffset': 113246208, > 'leasePath': '/dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/leases', > 'path': > '/rhev/data-center/mnt/blockSD/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/images/5b10b1b9-ee82-46ee-9f3d-3659d37e4851/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12', > 'volType': 'path', > 'volumeID': '4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12'} > > # Copy the volume data to some file system > > I'm using raw, you may like to use qcow2 > > cd <some mountpoint> > qemu-img convert -p -O raw > /rhev/data-center/mnt/blockSD/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/images/5b10b1b9-ee82-46ee-9f3d-3659d37e4851/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 > saved-disk.img > > # Teardown the image > > # vdsClient -s 0 teardownImage 591475db-6fa9-455d-9c05-7f6e30fb06d5 > 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 > 5b10b1b9-ee82-46ee-9f3d-3659d37e4851 > 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 > OK > > # Check the saved image > > # qemu-img info saved-disk.img > image: saved-disk.img > file format: raw > virtual size: 8.0G (8589934592 bytes) > disk size: 1.2G > > You can mount this image and copy files, or copy the data to an empty > disk you created for the new vm. > > Nir > >> >> Our Setup >> ovirt version: 3.5.4 hosted engine >> 4 supermicro hosts running centos 7.1 >> 1 iscsi storage server running Open-E DSS v7 Lite >> >> Thanks in advance >> >> Jaret >> Email sent using Packet Mail - Email, Groupware and Calendaring for the >> cloud! >> >> _______________________________________________ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users