Re: [okeanos-dev] Re: KVM versions, machine types and failed migrations
On Wed, Jan 09, 2013 at 03:27:53PM +0200, Vangelis Koukis wrote: On Wed, Jan 09, 2013 at 01:10:45pm +, Daniel P. Berrange wrote: When doing migration, the fundamental requirement is that the guest OS visible machine ABI must not change. Thus there are three key things to take care of when launching QEMU on the migration target host. - The device PCI/USB addresses must be identical to the source - The machine type must be identical to the source - The CPU model must be identical to the source Thanks for the detailed list of requirements, we'll take it into account for the relevant Ganeti patch. If you don't follow those requirements, either QEMU or the guest OS or both will crash burn during migration you get to keep both pieces :-) My point is, are these requirements left up to the caller of kvm -incoming to satisfy? Since the migration will most probably break, wouldn't it be best for QEMU to detect this and complain loudly, instead of continuing with the migration, failing silently and destroying the VM? Sure there could be some yes, do it, I know it is going to break option, which will make QEMU proceed with the migration. However, in 99% of the cases this is just user error, e.g. the user has upgraded the version on the other end and has not specified -M explicitly. It would be best if QEMU was able to detect and warn the user about what is going to happen, because it does lead to the VM dying. What you describe is certainly desirable, but it is quite hard to achieve with current QEMU. Much of the work with moving to the new QEMU object model configuration descriptions has been motivated by a desire to enable improvements migration handling. As you suggest, the goal is that the source QEMU be able to send a complete reliable hardware description to the destination QEMU during migration.It is getting closer, but we're not there yet. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
KVM versions, machine types and failed migrations
Hello, I'd like to ask a few questions about the way migrations work in KVM among different emulated machine types and different versions of the qemu-kvm package. I am sending to both the kvm@ and qemu-devel@ lists, please redirect me if I was wrong in doing so. In a nutshell: while trying to live-migrate a VM on ~okeanos [1], we see VM migrations fail silently if going from kvm 1.0 to kvm 1.1. The source VM is frozen, info migrate on the source monitor reports success, but the VM is dead upon arrival on the destination process. Please see [3] for the exact package versions for qemu-kvm we have tested with. Migration works if the destination kvm has been started with the same machine type as the source VM, e.g., using -M pc-1.0 specifically on the destination, when migrating a pc-1.0 machine from kvm 1.0 to kvm 1.1. How does the machine type specified with -M work in the case of migrations? Are migrations expected to fail if the machine type is different between source and destination process? If yes, shouldn't KVM be able to detect this and abort the migration instead of failing silently? For every (src, dst) pair of package version / machine type, we saw the following: dst: 1.0/pc-0.12 1.0/pc-1.0 1.1/pc-1.1 src: 1.0/pc-0.12 ok ok fails silently 1.0/pc-1.0 fails silently ok fails silently 1.1/pc-1.1 fails silently fails silently ok Machine types pc-0.12 and pc-1.0 were run with qemu-kvm package version 1.0, machine type pc-1.1 with qemu-kvm package version 1.1. Also, the migration 1.0/pc-0.12 to 1.0/pc-1.0 seems to work... How can it work or even be allowed, given that the guest finds itself running on a different hardware configuration after the migration? Regarding different package versions of qemu-kvm, it seems migrations do not work from source 0.12.5 to any other version *even* if -M pc-0.12 is specified at the incoming KVM process. For versions = 1.0 everything works provided the machine type on the destination is the same as on the source. Our goal is to patch Ganeti [2] so that it sets the destination machine type to that of the source specifically, ensuring migrations work seamlessly after a KVM upgrade. Is there a way to retrieve the machine type of a running KVM process through a monitor command? Thank you, Vangelis. [1] ~okeanos IaaS: http://okeanos.io [2] Ganeti: https://code.google.com/p/ganeti/ [3] We tested on Debian Squeeze with package versions: kvm 0.12: qemu-kvm_0.12.5+dfsg-5+squeeze8_amd64.deb kvm 1.0: qemu-kvm_1.0+dfsg-8~bpo60+1_amd64.deb kvm 1.1: qemu-kvm_1.1.2+dfsg-2~bpo60+1_amd64.deb -- Vangelis Koukis vkou...@grnet.gr OpenPGP public key ID: pub 1024D/1D038E97 2003-07-13 Vangelis Koukis vkou...@cslab.ece.ntua.gr Key fingerprint = C5CD E02E 2C78 7C10 8A00 53D8 FBFC 3799 1D03 8E97 Only those who will risk going too far can possibly find out how far one can go. -- T.S. Eliot signature.asc Description: Digital signature
Re: KVM versions, machine types and failed migrations
On Wed, Jan 09, 2013 at 02:23:50PM +0200, Vangelis Koukis wrote: Hello, I'd like to ask a few questions about the way migrations work in KVM among different emulated machine types and different versions of the qemu-kvm package. I am sending to both the kvm@ and qemu-devel@ lists, please redirect me if I was wrong in doing so. In a nutshell: while trying to live-migrate a VM on ~okeanos [1], we see VM migrations fail silently if going from kvm 1.0 to kvm 1.1. The source VM is frozen, info migrate on the source monitor reports success, but the VM is dead upon arrival on the destination process. Please see [3] for the exact package versions for qemu-kvm we have tested with. Migration works if the destination kvm has been started with the same machine type as the source VM, e.g., using -M pc-1.0 specifically on the destination, when migrating a pc-1.0 machine from kvm 1.0 to kvm 1.1. How does the machine type specified with -M work in the case of migrations? Are migrations expected to fail if the machine type is different between source and destination process? If yes, shouldn't KVM be able to detect this and abort the migration instead of failing silently? When doing migration, the fundamental requirement is that the guest OS visible machine ABI must not change. Thus there are three key things to take care of when launching QEMU on the migration target host. - The device PCI/USB addresses must be identical to the source - The machine type must be identical to the source - The CPU model must be identical to the source If you don't follow those requirements, either QEMU or the guest OS or both will crash burn during migration you get to keep both pieces :-) Regarding different package versions of qemu-kvm, it seems migrations do not work from source 0.12.5 to any other version *even* if -M pc-0.12 is specified at the incoming KVM process. For versions = 1.0 everything works provided the machine type on the destination is the same as on the source. Some older versions of QEMU were buggy causing the machine type to not correctly preserve ABI. Our goal is to patch Ganeti [2] so that it sets the destination machine type to that of the source specifically, ensuring migrations work seamlessly after a KVM upgrade. Is there a way to retrieve the machine type of a running KVM process through a monitor command? IIRC there is not a monitor command for this. The general approach to dealing with migration stability should be to launch QEMU with a canonical hardware configuration. This means explicitly setting a machine type, CPU model and PCI/USB devices addresses upfront. NB you should not use 'pc' as a machine type - if you query the list of machine types from QEMU, it will tell you what 'pc' corresponds to (pc-1.2) and then use the versioned type so you have a known machine type. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [okeanos-dev] Re: KVM versions, machine types and failed migrations
On Wed, Jan 09, 2013 at 01:10:45pm +, Daniel P. Berrange wrote: When doing migration, the fundamental requirement is that the guest OS visible machine ABI must not change. Thus there are three key things to take care of when launching QEMU on the migration target host. - The device PCI/USB addresses must be identical to the source - The machine type must be identical to the source - The CPU model must be identical to the source Thanks for the detailed list of requirements, we'll take it into account for the relevant Ganeti patch. If you don't follow those requirements, either QEMU or the guest OS or both will crash burn during migration you get to keep both pieces :-) My point is, are these requirements left up to the caller of kvm -incoming to satisfy? Since the migration will most probably break, wouldn't it be best for QEMU to detect this and complain loudly, instead of continuing with the migration, failing silently and destroying the VM? Sure there could be some yes, do it, I know it is going to break option, which will make QEMU proceed with the migration. However, in 99% of the cases this is just user error, e.g. the user has upgraded the version on the other end and has not specified -M explicitly. It would be best if QEMU was able to detect and warn the user about what is going to happen, because it does lead to the VM dying. Regarding different package versions of qemu-kvm, it seems migrations do not work from source 0.12.5 to any other version *even* if -M pc-0.12 is specified at the incoming KVM process. For versions = 1.0 everything works provided the machine type on the destination is the same as on the source. Some older versions of QEMU were buggy causing the machine type to not correctly preserve ABI. Our goal is to patch Ganeti [2] so that it sets the destination machine type to that of the source specifically, ensuring migrations work seamlessly after a KVM upgrade. Is there a way to retrieve the machine type of a running KVM process through a monitor command? IIRC there is not a monitor command for this. The general approach to dealing with migration stability should be to launch QEMU with a canonical hardware configuration. This means explicitly setting a machine type, CPU model and PCI/USB devices addresses upfront. NB you should not use 'pc' as a machine type - if you query the list of machine types from QEMU, it will tell you what 'pc' corresponds to (pc-1.2) and then use the versioned type so you have a known machine type. This is exactly what we're trying to do: specify -M explicitly in the kvm command line, instead of letting the default pc machine type change arbitrarily whenever the qemu-kvm package gets upgraded. Thanks again, Vangelis. -- Vangelis Koukis vkou...@grnet.gr OpenPGP public key ID: pub 1024D/1D038E97 2003-07-13 Vangelis Koukis vkou...@cslab.ece.ntua.gr Key fingerprint = C5CD E02E 2C78 7C10 8A00 53D8 FBFC 3799 1D03 8E97 Only those who will risk going too far can possibly find out how far one can go. -- T.S. Eliot signature.asc Description: Digital signature
