Inline
-Original Message-
From: Isaku Yamahata [mailto:yamah...@valinux.co.jp]
Sent: 14 November 2012 02:23
To: Hudzia, Benoit
Cc: quint...@redhat.com; qemu-devel qemu-devel; Orit Wasserman;
chegu_vi...@hp.com; Michael Roth
Subject: Re: Migration To-do list
On Tue, Nov 13, 2012 at 05:46:13PM +, Hudzia, Benoit wrote:
Hi,
One concept we have been playing around in the context of and hybrid
and post copy and might make sense if you are orienting your effort toward
RDMA / Post copy is to move most of the logic in the destination side.
This is one thing you might want to consider as it can solve some of the
issue you currently have and allow you to maintain almost a single API /
Protocol once integrating with post copy approach.
The idea is to drive the migration from the destination side. I.e. The page
are pulled from the destination and not pushed from the source side.
Ex: current pre-copy :
*extract dirty bitmap ( dirty bitmap extraction can be scheduled or
triggered by destination)
* send it to the destination side
* have the destination iterating over the bitmap ( can do page
prioritization here)
IIRC last year, you mentioned page prioritization, but didn't this year.
Is it still supported?
Where is it implemented? in qemu or kernel?
It is in Qemu, it is too expensive and specialised to do that within the
kernel.
I think Orit did some work regarding this aspect however I am not 100% sure it
is the stable branch yet.
* depending of protocol :
_ with standard socket ( or RDS) :
. Destination : request page(s)- can be batched
. source receive request send back the page
. destination process
_ with RDMA :
. Destination Read Page from source to local page (
the page have been mapped to RDMA at the bitmap extraction) ( RDMA
support scatter gather)
Although I'm not familiar with RDMA, RDMA requires the exchange of DMA-
address between
sender and receiver in advance and pinning down pages.
It it correct?
Yes it is correct. This is why you would be registering the memory only when
the page is dirtied. Avoiding large memory pinning for too long. ( an unpinning
upon RDMA read confirmation ).
The address is the same one as the one within the virtual memory. What you
exchange is a combination of RDMA key ( to uniquely identify the memory region
you are sharing ) and the offset start address of the MR. Then you can read
write at will within it. That is why it's a little bit tricky because the RDMA
write and read typically do not trigger any notification ( cpu / os etc..
everything is bypassed) as a result your page content can change without the
process/OS knowing it.
_ with post copy
. pretty much the same but the dirty bitmap reset is
done in kernel during the post copy operation ( provide a better dirty bit
tracking granularity)
Disadvantage:
* add a round trip that can be compensate with batch operation (
only with standard socket)
Advantage :
* most of the heavy lifting is done at the destination side leaving the
source to respond to request in an event based format
* resolve a lot of issue you have with your threading form the sender
side ( accounting etc.. )
* extremely friendly to optimised solution
* if the bitmap generation is expensive we can overlap their
generation creating a semi continuous delivery of them guaranteeing an
uninterrupted and optimised flow. = we decouple the bitmap generation
from the send/ receive operation.
Anyway , I will notify you as soon as I have the patch / library available
for
RDMA / postcopy.
Note On the fault tolerance part: this require a lot more heavy code
optimisation and poking around to guarantee efficient checkpointing. Most
of the solution we tested so far ( Remus and an old version of kemari) scale
poorly . Again, an RDMA / post copy solution is kind of necessary when you
talk about check pointing enterprise class applications.
IIRC Kemari guys evaluated IB case. I'm not sure that it was with RDMA or
IPoIB.
thanks,
Regards
Benoit
-Original Message-
From: Juan Quintela [mailto:quint...@redhat.com]
Sent: 13 November 2012 16:19
To: qemu-devel qemu-devel; Orit Wasserman; chegu_vi...@hp.com;
Hudzia, Benoit; Isaku Yamahata; Michael Roth
Subject: Migration ToDo list
Hi
If you have anything else to put, please add.
Migration Thread
* Plan is integrate it as one of first thing in December (me)
* Remove copies with buffered file (me)
Bitmap Optimization
* Finish moving to individual bitmaps for migration/vga/code
* Make sure we don't copy things around
* Shared memory bitmap with kvm?
* Move to 2MB pages bitmap and then fine grain?
QIDL
