[linuxkernelnewbies] Migration - Kernel based Virtual Machine

[Peter Teoh]

http://kvm.qumranet.com/kvmwiki/Migration KVM: migrating a VM Contents KVM: migrating a VM Introduction Requirements highlights / merits User Interface Example / HOWTO Problems / Todo savevm/loadvm to an external state file (using pseudo-migration) more exec: options Algorithm (the short version) Introduction KVM currently supports savevm/loadvm and offline or live migration Migration commands are given when in qemu-monitor (Alt-Ctrl-2). Upon successful completion, the migrated VM continues to run on the destination host. Note You can migrate a guest between an AMD host to an Intel host and back. Naturally, a 64-bit guest can only be migrated to a 64-bit host, but a 32-bit guest can be migrated at will. There are some older Intel processors which don't support NX (or XD), which may cause problems in a cluster which includes NX-supporting hosts. We may add a feature to hide NX if this proves to be a problem in actual deployments. Requirements The VM image is accessible on both source and destination hosts (located on a shared storage, e.g. using nfs). It is recommended an images-directory would be found on the same path on both hosts (for migrations of a copy-on-write image -- an image created on top of a base-image using "qemu-image create -b ...") The src and dst hosts must be on the same subnet (keeping guest's network when tap is used). Do not use -snapshot qemu command line option. For tcp:// migration protocol the guest on the destination must be started the same way it was started on the source. For ssh:// migration protocol it is recommended to use vnc (not SDL) redirect nothing to/from stdin/stdout. ssh-keys have been exchanged between src and dst hosts (otherwise the user must type the password). kvm/qemu executable (e.g. /usr/bin/kvm) and data-directory (e.g. /usr/kvm/share/qemu) can be found on both hosts using the same (full) path. highlights / merits Almost unnoticeable guest down time Guest is not involved (unique to KVM Live Migration 1) Capability to tunnel VM state through an external program (unique to KVM Live Migration 1) ssh/gzip/bzip2/gpg/your own Built-in security using ssh (unique to KVM Live Migration 1) Upon success guest continues to run on destination host, upon failure guest continues to run on source host (with one exception) Short and Simple Easy to enhance Hardware independence (almost). Support for migration of stopped (paused) VMs. Open 1 These features are unique to KVM Live Migration as far as I know. If you know of other hypervisor that support any of them please update this page or let me (Uri) know. User Interface The user interface is through the qemu monitor (alt-ctrl-2 on the SDL window) Management migrate [-d] <URI> migrate_cancel The '-d' will let you query the status of the migration. With no '-d' the monitor prompt returns when the migration completes. URI can be one of 'exec:<command>', 'ssh://<ip>' or tcp://<ip:port> Status info migration Migration Parameters migrate_set_speed <speed> set bandwidth control parameters (max transfer rate per second) Example / HOWTO A is the source host, B is the destination host: TCP example: Start the VM on B with the exact same parameters as the VM on A, in migration-listen mode (-incoming tcp://0:4444 (or other PORT)) Start the migration (always on the source host): A: migrate -d tcp://B:4444 (or other PORT) Check the status (on A only): info migration SSH example: Note: It is recommended not to use SDL when using SSH. Start the migration A: migrate ssh://B Offline example: unlimit bandwidth used for migration: A: migrate_set_speed 1g stop the guest: A: stop continue with either TCP or SSH migration as described above. Problems / Todo TSC offset on the new host must be set in such a way that the guest sees a monotonically increasing TSC, otherwise the guest may hang indefinitely after migration. usbdevice tablet complains after migration. handle migration completion better (especially when network problems occur). More informative status. Migration does not work while CPU real-mode/protected mode are still changing. savevm/loadvm to an external state file (using pseudo-migration) let the state file be /saved_images/VM.STATE savevm (qemu monitor): stop migrate file:///saved_images/VM.STATE loadvm (qemu command line and qemu monitor): <qemu-command-line> -incoming file:///saved_images/VM.STATE cont more exec: options To be supported directly by Migration Protocols, but until then... Save VM state into a compressed file Save stop migrate_set_speed 4095m migrate "exec:gzip -c > STATEFILE.gz" Load gzip -c -d STATEFILE.gz | <qemu-command-line> -incoming stdio Save VM State into an encrypted file (assumes KEY has already been generated) Save stop migrate_set_speed 4095m migrate "exec:gpg -q -e -r KEY -o STATFILE.gpg" Load VM state from an encrypted file gpg -q -d -r KEY STATEFILE.gpg | <qemu-command-line> -incoming stdio Send encrypted VM state from host A to host B (Note: ssh is probably better, this is just for show) on host A migrate "exec:gpg -q -e -r KEY | nc B 4444" on host B nc -l 4444 | gpg -q -d -r KEY | <qemu-command-line> -incoming stdio Algorithm (the short version) Setup Start guest on destination, connect, enable dirty page logging and more Transfer Memory Guest continues to run Bandwidth limitation (controlled by the user) First transfer the whole memory Iteratively transfer all dirty pages (pages that were written to by the guest). Stop the guest And sync VM image(s) (guest's hard drives). Transfer State As fast as possible (no bandwidth limitation) All VM devices' state and dirty pages yet to be transferred Continue the guest On destination upon success Broadcast "I'm over here" Ethernet packet to announce new location of NIC(s). On source upon failure (with one exception). Instructions for kvm-13 and below: MigrationQemu0.8.2. Migration (last edited 2008-02-21 16:01:50 by Uri Lublin)