Re: [ovirt-devel] Issue: Device path changed after adding disks to guest VM
On Wed, Jan 06, 2021 at 04:36:46PM +0100, Peter Krempa wrote: > On Wed, Jan 06, 2021 at 17:16:24 +0200, Nir Soffer wrote: > > On Wed, Dec 2, 2020 at 4:57 PM Joy Li wrote: > > [...] > > > Comparing to state before reboot: > > > > # virsh -r domblklist disk-mapping > > Target Source > > --- > > sdc - > > sda > > /rhev/data-center/mnt/blockSD/84dc4e3c-00fd-4263-84e8-fc2466e9/images/40018b33-2b11-4d10-82e4-604a5b135fb2/40f455c4-8c92-4f8f-91c2-991b0ddfc2f5 > > vda /dev/mapper/3600140594af345ed76d42058f2b1a454 > > vdb /dev/mapper/360014050058f2f8a0474dc7a8a7cc6a5 > > vdc /dev/mapper/36001405b4d0c0b7544d47438b21296ef > > > > # ls -lh /dev/disk/by-id/virtio-* > > lrwxrwxrwx. 1 root root 9 Jan 6 09:42 > > /dev/disk/by-id/virtio-b97e68b2-87ea-45ca-9 -> ../../vda > > lrwxrwxrwx. 1 root root 9 Jan 6 09:42 > > /dev/disk/by-id/virtio-d9a29187-f492-4a0d-a -> ../../vdb > > lrwxrwxrwx. 1 root root 9 Jan 6 09:51 > > /dev/disk/by-id/virtio-e801c2e4-dc2e-4c53-b -> ../../vdc > > > > In the guest disks are mapped to the same device name. > > > > It looks like libivrt domblklist is not correct - vdb and vdc are switched. > > Peter, this expected? > > The names in 'virsh domblklist' are unfortunately and confusingly chosen > to match the expected /dev/ device node name, but it's at kernel's > discretion to name /dev/ nodes. > > This means that it's not guaranteed that what you see in 'virsh > domblklist' will match the state in the guest. Essentially the only thing the disk device name is used for is sorting the elements within the XML document. This in turn affects what order PCI addresses (virtio-blk) or SCSI LUNS (virtio-scsi) are assigned in. This influences/hints as to what order the guest OS *might* assign device names in. The device name from the XML is not exposed to the guest directly though. Certainly when hotplugging/unplugging is involved all bets are off wrt what disk names you'll see in the guest vs the XML. Dont expect them to match except by luck. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Re: [ovirt-devel] Issue: Device path changed after adding disks to guest VM
On Wed, Jan 06, 2021 at 17:16:24 +0200, Nir Soffer wrote: > On Wed, Dec 2, 2020 at 4:57 PM Joy Li wrote: [...] > Comparing to state before reboot: > > # virsh -r domblklist disk-mapping > Target Source > --- > sdc - > sda > /rhev/data-center/mnt/blockSD/84dc4e3c-00fd-4263-84e8-fc2466e9/images/40018b33-2b11-4d10-82e4-604a5b135fb2/40f455c4-8c92-4f8f-91c2-991b0ddfc2f5 > vda /dev/mapper/3600140594af345ed76d42058f2b1a454 > vdb /dev/mapper/360014050058f2f8a0474dc7a8a7cc6a5 > vdc /dev/mapper/36001405b4d0c0b7544d47438b21296ef > > # ls -lh /dev/disk/by-id/virtio-* > lrwxrwxrwx. 1 root root 9 Jan 6 09:42 > /dev/disk/by-id/virtio-b97e68b2-87ea-45ca-9 -> ../../vda > lrwxrwxrwx. 1 root root 9 Jan 6 09:42 > /dev/disk/by-id/virtio-d9a29187-f492-4a0d-a -> ../../vdb > lrwxrwxrwx. 1 root root 9 Jan 6 09:51 > /dev/disk/by-id/virtio-e801c2e4-dc2e-4c53-b -> ../../vdc > > In the guest disks are mapped to the same device name. > > It looks like libivrt domblklist is not correct - vdb and vdc are switched. > Peter, this expected? The names in 'virsh domblklist' are unfortunately and confusingly chosen to match the expected /dev/ device node name, but it's at kernel's discretion to name /dev/ nodes. This means that it's not guaranteed that what you see in 'virsh domblklist' will match the state in the guest. In this case I think the reorder happens as the PCI address of vdb is larger than the address of vdc. A partial workaround can be to use the data provided by the qemu guest agent, for exampe via 'virsh domfsinfo': $ virsh domfsinfo fedora32 Mountpoint Name Type Target /dm-0 xfsvda /bootvda1 xfsvda Here the guest-host matching is established via the PCI address so the 'Target' field accurately refers to the target in the VM XML. Similarly the linux kernel recently changed enumeration of SCSI devices so they are not guaranteed to match either. Libguestfs for example needed a workaround. https://github.com/libguestfs/libguestfs/commit/bca9b94fc593771b3801b09b95e477f160517909
Re: [ovirt-devel] Issue: Device path changed after adding disks to guest VM
On Wed, Dec 2, 2020 at 4:57 PM Joy Li wrote: > > Thanks a lot Nir! Good to know that oVirt cannot guarantee the disk names so > that I don't need to spend more time trying to enable such a feature. > > I can always reproduce the problem via my application, basically, the > procedure is as following: > > 1. create a VM Which guest OS? Can you share the guest disk image or ISO image used to instal it? > 2. add disks to the VM (e.g.: disk names: disk1, disk3) > 3. check the disk mappings via `virsh domblklist ` Please shared the libvirt domain xml (virsh dumpxml vm-name) > 4. add another disk (let's say, disk2, give a name alphabetically before some > existing disks) Add disk while the vm is running (hotplug)? > 5. shutdown the VM via hypervisor and start it again (reboot won't work) What do you mean by "reboot does not work?" > 6. `virsh domblklist` again, then you might see the problem I mentioned before Mapping is different compared with state before the reboot? > There are no virtio devices inside /dev/disk/by-id/xxx of my guest VM. Maybe you don't have systemd-udev installed? The links in /dev/disk/... are created by udev during startup, and when detecting a new disk. > And I just noticed that the disks mapping information given by hypervisor > (from VM configuration or virsh command) is different from the reality inside > the VM. The disk name inside the VM was actually not changed. > > So now my issue is that given a disk name (/dev/vdb) of a VM, how can I get > its wwid? Before I got it from the hypervisor, but now the hypervisor's > information is not reliable, and since the disk is unformatted, I cannot use > UUID. You can use: # udevadm info /dev/vda P: /devices/pci:00/:00:02.5/:06:00.0/virtio4/block/vda N: vda L: 0 S: disk/by-path/pci-:06:00.0 S: disk/by-id/virtio-b97e68b2-87ea-45ca-9 S: disk/by-path/virtio-pci-:06:00.0 E: DEVPATH=/devices/pci:00/:00:02.5/:06:00.0/virtio4/block/vda E: DEVNAME=/dev/vda E: DEVTYPE=disk E: MAJOR=252 E: MINOR=0 E: SUBSYSTEM=block E: USEC_INITIALIZED=10518442 E: ID_SERIAL=b97e68b2-87ea-45ca-9 E: ID_PATH=pci-:06:00.0 E: ID_PATH_TAG=pci-_06_00_0 E: DEVLINKS=/dev/disk/by-path/pci-:06:00.0 /dev/disk/by-id/virtio-b97e68b2-87ea-45ca-9 /dev/disk/by-path/virtio-pci-:06:00.0 E: TAGS=:systemd: I tried to reproduce you issue 4.4.5 development build: Starting vm with 2 direct LUN disks: # virsh -r dumpxml disk-mapping ... 40018b33-2b11-4d10-82e4-604a5b135fb2 b97e68b2-87ea-45ca-94fb-277d5b30baa2 d9a29187-f492-4a0d-aea2-7d5216c957d7 ... # virsh -r domblklist disk-mapping Target Source --- sdc - sda /rhev/data-center/mnt/blockSD/84dc4e3c-00fd-4263-84e8-fc2466e9/images/40018b33-2b11-4d10-82e4-604a5b135fb2/40f455c4-8c92-4f8f-91c2-991b0ddfc2f5 vda /dev/mapper/3600140594af345ed76d42058f2b1a454 vdb /dev/mapper/360014050058f2f8a0474dc7a8a7cc6a5 In guest: # ls -lh /dev/disk/by-id/virtio-* lrwxrwxrwx. 1 root root 9 Jan 6 09:42 /dev/disk/by-id/virtio-b97e68b2-87ea-45ca-9 -> ../../vda lrwxrwxrwx. 1 root root 9 Jan 6 09:42 /dev/disk/by-id/virtio-d9a29187-f492-4a0d-a -> ../../vdb NOTE: "d9a29187-f492-4a0d-a" are the first characters of the disk id: "d9a29187-f492-4a0d-aea2-7d5216c957d7" seen in oVirt: https://my-engine/ovirt-engine/webadmin/?locale=en_US#disks-general;id=d9a29187-f492-4a0d-aea2-7d5216c957d7 Adding another disk in sorted in the middle (while the vm is running): # virsh -r dumpxml disk-mapping ... 40018b33-2b11-4d10-82e4-604a5b135fb2 b97e68b2-87ea-45ca-94fb-277d5b30baa2 d9a29187-f492-4a0d-aea2-7d5216c957d7 e801c2e4-dc2e-4c53-b17b-bf6de99f16ed ... # virsh -r domblklist disk-mapping Target Source --- sdc - sda /rhev/data-center/mnt/blockSD/84dc4e3c-00fd-4263-84e8-fc2466e9/images/40018b33-2b11-4d10-82e4-604a5b135fb2/40f455c4-8c92-4f8f-91c2-991b0ddfc2f5 vda /dev/mapper/3600140594af345ed76d42058f2b1a454 vdb /dev/mapper/360014050058f2f8a0474dc7a8a7cc6a5 vdc
Re: Migration via qemu+ssh using a given private ssh key possible ?
On Tue, Jan 05, 2021 at 07:39:49PM +0100, Oliver Dzombic wrote: > Hi Peter, > > thank you very much for this hint. Seems to work! :) > > But i also tried another switch: no_verify=1 > > The whole call: > > virsh -K0 -k0 migrate --copy-storage-inc --verbose --persistent --live > testInstance > qemu+ssh://testnode4:22/system?keyfile=/tmp/key-5ff4b02ca966c?no_verify=1?no_tty=1 > > So as you can see > > no_verify=1 > no_tty=1 > > has been added. You have incorrect URL syntax here. You need to use "&" to separate parameters, not "?". The "?" is only valid to separate the path from the start of the parameter list. qemu+ssh://testnode4:22/system?keyfile=/tmp/key-5ff4b02ca966c_verify=1_tty=1 Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Re: Migration via qemu+ssh using a given private ssh key possible ?
On Tue, Jan 05, 2021 at 19:39:49 +0100, Oliver Dzombic wrote: > Hi Peter, [...] > So as you can see > > no_verify=1 > no_tty=1 > > has been added. > > But still i receive > > > The authenticity of host '[testnode4]:22 ([10.0.1.4]:22)' can't be > established. > ECDSA key fingerprint is SHA256:tcF31bWN6Gg8O5bMTkkusbcariPBWjGdLAP7WnfdqsM. > Are you sure you want to continue connecting (yes/no/[fingerprint])? I've looked at the code briefly and it seems to be implemented by adding: -o StrictHostKeyChecking=no to the command line of the 'ssh' binary, so it should work, but maybe there's some snag. Could you please file this as a bug/issue: https://gitlab.com/libvirt/libvirt/-/issues/new
