On Tue, Jul 5, 2016 at 3:49 AM, Hailiang Zhang <zhang.zhanghaili...@huawei.com> wrote: > On 2016/7/4 20:22, Baptiste Reynal wrote: >> >> On Thu, Jan 7, 2016 at 1:19 PM, zhanghailiang >> <zhang.zhanghaili...@huawei.com> wrote: >>> >>> For now, we still didn't support live memory snapshot, we have discussed >>> a scheme which based on userfaultfd long time ago. >>> You can find the discussion by the follow link: >>> https://lists.nongnu.org/archive/html/qemu-devel/2014-11/msg01779.html >>> >>> The scheme is based on userfaultfd's write-protect capability. >>> The userfaultfd write protection feature is available here: >>> http://www.spinics.net/lists/linux-mm/msg97422.html >>> >>> The process of this live memory scheme is like bellow: >>> 1. Pause VM >>> 2. Enable write-protect fault notification by using userfaultfd to >>> mark VM's memory to write-protect (readonly). >>> 3. Save VM's static state (here is device state) to snapshot file >>> 4. Resume VM, VM is going to run. >>> 5. Snapshot thread begins to save VM's live state (here is RAM) into >>> snapshot file. >>> 6. During this time, all the actions of writing VM's memory will be >>> blocked >>> by kernel, and kernel will wakeup the fault treating thread in qemu to >>> process this write-protect fault. The fault treating thread will >>> deliver this >>> page's address to snapshot thread. >>> 7. snapshot thread gets this address, save this page into snasphot file, >>> and then remove the write-protect by using userfaultfd API, after >>> that, >>> the actions of writing will be recovered. >>> 8. Repeat step 5~7 until all VM's memory is saved to snapshot file >>> >>> Compared with the feature of 'migrate VM's state to file', >>> the main difference for live memory snapshot is it has little time delay >>> for >>> catching VM's state. It just captures the VM's state while got users >>> snapshot >>> command, just like take a photo of VM's state. >>> >>> For now, we only support tcg accelerator, since userfaultfd is not >>> supporting >>> tracking write faults for KVM. >>> >>> Usage: >>> 1. Take a snapshot >>> #x86_64-softmmu/qemu-system-x86_64 -machine >>> pc-i440fx-2.5,accel=tcg,usb=off -drive >>> file=/mnt/windows/win7_install.qcow2.bak,if=none,id=drive-ide0-0-1,format=qcow2,cache=none >>> -device ide-hd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -vnc :7 -m >>> 8192 -smp 1 -netdev tap,id=bn0 -device virtio-net-pci,id=net-pci0,netdev=bn0 >>> --monitor stdio >>> Issue snapshot command: >>> (qemu)migrate -d file:/home/Snapshot >>> 2. Revert to the snapshot >>> #x86_64-softmmu/qemu-system-x86_64 -machine >>> pc-i440fx-2.5,accel=tcg,usb=off -drive >>> file=/mnt/windows/win7_install.qcow2.bak,if=none,id=drive-ide0-0-1,format=qcow2,cache=none >>> -device ide-hd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -vnc :7 -m >>> 8192 -smp 1 -netdev tap,id=bn0 -device virtio-net-pci,id=net-pci0,netdev=bn0 >>> --monitor stdio -incoming file:/home/Snapshot >>> >>> NOTE: >>> The userfaultfd write protection feature does not support THP for now, >>> Before taking snapshot, please disable THP by: >>> echo never > /sys/kernel/mm/transparent_hugepage/enabled >>> >>> TODO: >>> - Reduce the influence for VM while taking snapshot >>> >>> zhanghailiang (13): >>> postcopy/migration: Split fault related state into struct >>> UserfaultState >>> migration: Allow the migrate command to work on file: urls >>> migration: Allow -incoming to work on file: urls >>> migration: Create a snapshot thread to realize saving memory snapshot >>> migration: implement initialization work for snapshot >>> QEMUSizedBuffer: Introduce two help functions for qsb >>> savevm: Split qemu_savevm_state_complete_precopy() into two helper >>> functions >>> snapshot: Save VM's device state into snapshot file >>> migration/postcopy-ram: fix some helper functions to support >>> userfaultfd write-protect >>> snapshot: Enable the write-protect notification capability for VM's >>> RAM >>> snapshot/migration: Save VM's RAM into snapshot file >>> migration/ram: Fix some helper functions' parameter to use >>> PageSearchStatus >>> snapshot: Remove page's write-protect and copy the content during >>> setup stage >>> >>> include/migration/migration.h | 41 +++++-- >>> include/migration/postcopy-ram.h | 9 +- >>> include/migration/qemu-file.h | 3 +- >>> include/qemu/typedefs.h | 1 + >>> include/sysemu/sysemu.h | 3 + >>> linux-headers/linux/userfaultfd.h | 21 +++- >>> migration/fd.c | 51 ++++++++- >>> migration/migration.c | 101 ++++++++++++++++- >>> migration/postcopy-ram.c | 229 >>> ++++++++++++++++++++++++++++---------- >>> migration/qemu-file-buf.c | 61 ++++++++++ >>> migration/ram.c | 104 ++++++++++++----- >>> migration/savevm.c | 90 ++++++++++++--- >>> trace-events | 1 + >>> 13 files changed, 587 insertions(+), 128 deletions(-) >>> >>> -- >>> 1.8.3.1 >>> >>> >>> >> > > Hi, > >> Hi Hailiang, >> >> Can I get the status of this patch series ? I cannot find a v2. > > > Yes, I haven't updated it for long time, it is based on userfault-wp API > in kernel, and Andrea didn't update the related patches until recent days. > I will update this series in the next one or two weeks. But it will only > support TCG until userfault-wp API supports KVM. >
May I have a pointer to those patches ? The last I found is http://thread.gmane.org/gmane.linux.kernel.mm/141647 and it seems pretty old. >> About TCG limitation, is KVM support on a TODO list or is there a >> strong technical barrier ? >> > > I don't think there are any technical hurdles, I would like to > have a try on realizing the KVM part to support userfault-wp, > But I'm a little busy with other things now. > Andrea may has a plan to achieve it. > > To: Andrea Arcangeli <aarca...@redhat.com> > > Thanks, > Hailiang > Ok, if it is not on Andrea schedule I am willing to take the action, at least for ARM/ARM64 support. Regards, Baptiste >> Thanks, >> Baptiste >> >> . >> >