Hi Daniel,
Live migration is already in qemu/KVM. I was not reinventing it, just
making it work by understanding the principles and correcting the way it
works in the illumos.
On 9. 11. 2017 1:08, Daniel Plominski wrote:
I think the patches are a bit small overall, live migration is quite
complex
*
https://developers.redhat.com/blog/2015/03/24/live-migrating-qemu-kvm-virtual-machines/
If you find something not working, please let me know.
How do you solve the problem with the page (vm memory state) transfer?
Dirty pages tracking. Standard qemu and KVM way.
How do you solve the problem with the storage transfer?
* a zfs snapshot on an active zvolume, without manual sync (inside
the vm and outside the zfs txg_sync) and freeze of the kvm, i
think that is not a good idea for a consistent block storage state
This is the most tricky part. You can start VM disk migration in the
background and after you finish, you can do it again using differential
snapshot transferring just the newly written bits. But the main problem
is that when you start the destination zone (= qemu waiting for incoming
migration stream), the qemu opens the destination disk zvols and after
that point, you are not able to receive zfs incremental stream anymore
(it fails with device is busy because zvol is open). You have several
options here:
1. The best one that is not done at the moment: upgrade qemu. The new
qemu supports custom dirty block tracking maps for disk devices. That
plays nicely with zfs send/receive. For more info read here:
https://patchwork.kernel.org/patch/9709815/
https://lists.gnu.org/archive/html/qemu-devel/2017-05/msg02219.html
2. You have to pause the source VM before transferring the last
incremental zfs snapshot and before starting the dest VM. After this
last zfs transfer, you can start the qemu migration stream. This
introduces a bit of delay between pausing the source VM and resuming the
dest VM. Depends on how you implement it, you can have longer or shorter
delay.
2a: Longer delay (but easier to implement) is doing everything
sequentially: zfs full send/recv, pause src VM, zfs differential
send/recv, start dest VM, start migration (the dest VM resumes
automatically after successful state transfer). The downtime is more or
less equal to the time needed to tranfer VM RAM contents.
2b: Shorter delay (this is the best way I can think of at the moment. If
you find better way, please let me know): zfs full send/recv, start qemu
live migration that transfers the VM state into the temporary file in
/tmp (=ramdisk) at destination's host and in parallel with that, run
differential zfs snapshot transfer every 10 seconds (src VM is still
running at this time). As soon as the qemu migration finishes (src VM is
paused now), you transfer the last few disk bits using incremental
snapshot and run the dest VM + immediately push the content of the
temporary migration file from /tmp (after this is done, the VM resumes
automatically). State restore from /tmp into the qemu takes very short
time so the downtime is minimized.
How do you solve the problem with unequal kvm configs, for example
cpu_type HOST (on the source host) and QEMU(on the target host)?
You (or the migration script) are responsible for creating a destination
VM with a compatible config. If your new config differs too much (e.g.
different device count, RAM size, etc.), I expect qemu to stop/fail the
incoming migration. But you don't want to create incompatible configs
anyway.
How do you solve the problem with starting the second kvm zone on the
other host (this reserves the same MAC / IP addresses on the network)
at the same time?
Yes, it does start the zone with the same zone network config. But the
qemu inside the destination zone is not running the VM yet. AFAIK the
zone network config is mostly about restrictions on possible IP/MAC
addresses. My tests show that as soon as I unpause the destination VM,
it is reachable immediately.
I hope it helped to get an overview.
Cheers
Jan
Mit freundlichen Grüßen
*DANIEL PLOMINSKI*
Leiter – IT / Head of IT
Telefon 09265 808-151 | Mobil 0151 58026316 | [email protected]
<mailto:[email protected]>
PGP Key: http://pgp.ass.de/2B4EB20A.key
cid:C17DB6FB-5F79-4BCC-AAB4-CAB59266BC29@localdomain
ASS-Einrichtungssysteme GmbH
ASS-Adam-Stegner-Straße 19 | D-96342 Stockheim
Geschäftsführer: Matthias Stegner, Michael Stegner, Stefan Weiß
Amtsgericht Coburg HRB 3395 | Ust-ID: DE218715721
cid:E40AEC87-91EE-472A-901A-ECAD3F5801FB@localdomain
*Von:*Ján Poctavek [mailto:[email protected]]
*Gesendet:* Mittwoch, 8. November 2017 10:55
*An:* [email protected]
*Betreff:* Re: [smartos-discuss] Live migration
Hi,
Sure, the sources are here:
https://github.com/YanChii/illumos-kvm/tree/live-migration
https://github.com/YanChii/illumos-kvm-cmd/tree/live_migration
The (small) number of changes does not reflect the effort invested
into this. All dead ends and debugs are not present in the final patch :).
I just rebased and testing my changes to integrate latest commit into
illumos-kvm (regarding the coexistence with bhyve):
https://github.com/YanChii/illumos-kvm/tree/lm-merge
Jan
On 7. 11. 2017 21:03, Daniel Plominski wrote:
Hi Jan,
are the danube kvm patches public available on github?
Von meinem iPhone gesendet
Am 07.11.2017 um 11:32 schrieb Ján Poctavek
<[email protected] <mailto:[email protected]>>:
Hi Matt,
KVM live migration is possible and we are working on it right
now. The hardest part (the kernel & qemu support) is already
done and currently I'm able to successfully migrate the VMs.
Now we need two things:
1. integrate the patches into the illumos-kvm and
illumos-kvm-cmd repos (requires testing, reviews, etc.)
2. some userspace support (live migration script).
After merging the patches into SmartOS, we can focus on the
userspace part (or you can write your own migration script).
Jan
PS: in parallel with the above, the live migration will be
present in the Danube Cloud v3.0.0 expected in this December.
On 6. 11. 2017 17:05, Matthew Law wrote:
De-lurking for a moment to ask a question: is live
migration of zones and KVM VMs possible? I recall it has
been mentioned on this list before and I can find ageing
references to it, e.g:
https://github.com/joyent/smartos-live/blob/master/src/vm/README.migration
I usually get by with vmadm send and vmadm receive after
first stopping the VM and that works great but it’s a bit
of a chore stopping and migrating dozens of VMs off a busy
host when it needs to go down for maintenance.
Cheers,
Matt.
*smartos-discuss* | Archives
<https://www.listbox.com/member/archive/184463/=now>
<https://www.listbox.com/member/archive/rss/184463/28549989-eb719454>
| Modify
<https://www.listbox.com/member/?&;>
Your Subscription [Powered by Listbox] <http://www.listbox.com>
-------------------------------------------
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription:
https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com
