* Hongyang Yang (yan...@cn.fujitsu.com) wrote: > > > ??? 08/28/2014 11:03 PM, Dr. David Alan Gilbert (git) ??????: > >From: "Dr. David Alan Gilbert" <dgilb...@redhat.com> > > > >Signed-off-by: Dr. David Alan Gilbert <dgilb...@redhat.com> > >---
> >+Postcopy can be combined with precopy (i.e. normal migration) so that if > >precopy > >+doesn't finish in a given time the switch is automatically made to precopy. > > I think you mean "automatically made to postcopy" here? Thanks! > >+Source behaviour > >+ > >+Until postcopy is entered the migration stream is identical to normal > >postcopy, > >+except for the addition of a 'postcopy advise' command at the beginning to > >+let the destination know that postcopy might happen. When postcopy starts > > A comma here? Yes, thanks. Dave > > >+the source sends the page discard data and then forms the 'package' > >containing: > >+ > >+ Command: 'postcopy ram listen' > >+ The device state > >+ A series of sections, identical to the precopy streams device state > >stream > >+ containing everything except postcopiable devices (i.e. RAM) > >+ Command: 'postcopy ram run' > >+ > >+The 'package' is sent as the data part of a Command: 'CMD_PACKAGED', and the > >+contents are formatted in the same way as the main migration stream. > >+ > >+Destination behaviour > >+ > >+Initially the destination looks the same as precopy, with a single thread > >+reading the migration stream; the 'postcopy advise' and 'discard' commands > >+are processed to change the way RAM is managed, but don't affect the stream > >+processing. > >+ > >+------------------------------------------------------------------------------ > >+ 1 2 3 4 5 6 7 > >+main -----DISCARD-CMD_PACKAGED ( LISTEN DEVICE DEVICE DEVICE RUN ) > >+thread | | > >+ | (page request) > >+ | \___ > >+ v \ > >+listen thread: --- page -- page -- page -- page -- page > >-- > >+ > >+ a b c > >+------------------------------------------------------------------------------ > >+ > >+On receipt of CMD_PACKAGED (1) > >+ All the data associated with the package - the ( ... ) section in the > >+diagram - is read into memory (into a QEMUSizedBuffer), and the main thread > >+recurses into qemu_loadvm_state_main to process the contents of the package > >(2) > >+which contains commands (3,6) and devices (4...) > >+ > >+On receipt of 'postcopy ram listen' - 3 -(i.e. the 1st command in the > >package) > >+a new thread (a) is started that takes over servicing the migration stream, > >+while the main thread carries on loading the package. It loads normal > >+background page data (b) but if during a device load a fault happens (5) the > >+returned page (c) is loaded by the listen thread allowing the main threads > >+device load to carry on. > >+ > >+The last thing in the CMD_PACKAGED is a 'RUN' command (6) letting the > >destination > >+CPUs start running. > >+At the end of the CMD_PACKAGED (7) the main thread returns to normal > >running behaviour > >+and is no longer used by migration, while the listen thread carries > >+on servicing page data until the end of migration. > >+ > >+=== Postcopy states === > >+ > >+Postcopy moves through a series of states (see postcopy_ram_state) > >+from ADVISE->LISTEN->RUNNING->END > >+ > >+ Advise: Set at the start of migration if postcopy is enabled, even > >+ if it hasn't had the start command; here the destination > >+ checks that its OS has the support needed for postcopy, and > >performs > >+ setup to ensure the RAM mappings are suitable for later postcopy. > >+ (Triggered by reception of POSTCOPY_RAM_ADVISE command) > >+ > >+ Listen: The first command in the package, POSTCOPY_RAM_LISTEN, switches > >+ the destination state to Listen, and starts a new thread > >+ (the 'listen thread') which takes over the job of receiving > >+ pages off the migration stream, while the main thread carries > >+ on processing the blob. With this thread able to process page > >+ reception, the destination now 'sensitises' the RAM to detect > >+ any access to missing pages (on Linux using the 'userfault' > >+ system). > >+ > >+ Running: POSTCOPY_RAM_RUN causes the destination to synchronise all > >+ state and start the CPUs and IO devices running. The main > >+ thread now finishes processing the migration package and > >+ now carries on as it would for normal precopy migration > >+ (although it can't do the cleanup it would do as it > >+ finishes a normal migration). > >+ > >+ End: The listen thread can now quit, and perform the cleanup of migration > >+ state, the migration is now complete. > >+ > >+=== Source side page maps === > >+ > >+The source side keeps two bitmaps during postcopy; 'the migration bitmap' > >+and 'sent map'. The 'migration bitmap' is basically the same as in > >+the precopy case, and holds a bit to indicate that page is 'dirty' - > >+i.e. needs sending. During the precopy phase this is updated as the CPU > >+dirties pages, however during postcopy the CPUs are stopped and nothing > >+should dirty anything any more. > >+ > >+The 'sent map' is used for the transition to postcopy. It is a bitmap that > >+has a bit set whenever a page is sent to the destination, however during > >+the transition to postcopy mode it is masked against the migration bitmap > >+(sentmap &= migrationbitmap) to generate a bitmap recording pages that > >+have been previously been sent but are now dirty again. This masked > >+sentmap is sent to the destination which discards those now dirty pages > >+before starting the CPUs. > >+ > >+Note that once in postcopy mode, the sent map is still updated; however, > >+its contents are not necessarily consistent with the pages already sent > >+due to the masking with the migration bitmap. > >+ > >+=== Destination side page maps === > >+ > >+(Needs to be changed so we can update both easily - at the moment updates > >are done > >+ with a lock) > >+The destination keeps a 'requested map' and a 'received map'. > >+Both maps are initially 0, as pages are received the bits are set in > >'received map'. > >+Incoming requests from the kernel cause the bit to be set in the 'requested > >map'. > >+When a page is received that is marked as 'requested' the kernel is > >notified. > >+If the kernel requests a page that has already been 'received' the kernel > >is notified > >+without re-requesting. > >+ > >+This leads to three valid page states: > >+page states: > >+ missing (!rc,!rq) - page not yet received or requested > >+ received (rc,!rq) - Page received > >+ requested (!rc,rq) - page requested but not yet received > >+ > >+state transitions: > >+ received -> missing (only during setup/discard) > >+ > >+ missing -> received (normal incoming page) > >+ requested -> received (incoming page previously requested) > >+ missing -> requested (userfault request) > >+ > > > > -- > Thanks, > Yang. -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
