[Qemu-devel] [PATCH v2 2/3] Migration: Add lots of trace events

2015-01-21 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Mostly on the load side, so that when we get a complaint about a migration failure we can figure out what it didn't like. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- migration/vmstate.c | 24 savevm.c

[Qemu-devel] [PATCH v2 0/3] Migration tracing and errors

2015-01-21 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Tidy up some old printfs Add some more error_reports that previously just failed with invalid. Add lots and lots of tracing. Dr. David Alan Gilbert (3): savevm: Convert fprintf to error_report Migration: Add lots of trace events Print errors

[Qemu-devel] [PATCH v2 1/3] savevm: Convert fprintf to error_report

2015-01-21 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Convert a bunch of fprintfs to error_reports Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- migration/vmstate.c | 7 --- savevm.c| 21 +++-- 2 files changed, 15 insertions(+), 13 deletions(-) diff

[Qemu-devel] [PATCH v2 1/3] Add -incoming pause

2015-02-11 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com -incoming pause causes qemu to wait for an incoming migration to be specified later. The monitor can be used to set migration capabilities that may affect the incoming connection process. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com

[Qemu-devel] [PATCH v2 0/3] -incoming pause

2015-02-11 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com This patchset provides a way of setting options on an incoming migration before the fd/process/socket has been created. start qemu with -incoming pause use the monitor to set whatever you need migrate_incoming theuri v2: Create

[Qemu-devel] [PATCH v2 3/3] Document -incoming options

2015-02-11 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Document the various URI formats for -incoming, the previous manpage and help text was wrong (out of date?) Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com Reviewed-by: Juan Quintela quint...@redhat.com --- qemu-options.hx | 29

[Qemu-devel] [PATCH v2 2/3] Add migrate_incoming

2015-02-11 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Add migrate_incoming/migrate-incoming to start an incoming migration. Once a qemu has been started with -incoming pause the migration can be started by issuing: migrate_incoming uri Signed-off-by: Dr. David Alan Gilbert

[Qemu-devel] [PATCH 1/3] Add -incoming pause

2015-02-10 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com -incoming pause causes qemu to wait for an incoming migration to be specified later. The monitor can be used to set migraiton capabilities that may affect the incoming connection process. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com

[Qemu-devel] [PATCH 3/3] Document -incoming options

2015-02-10 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Document the various URI formats for -incoming, the previous manpage and help text was wrong (out of date?) Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- qemu-options.hx | 29 ++--- 1 file changed, 26

[Qemu-devel] [PATCH 2/3] Add migrate -u option for -incoming pause

2015-02-10 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Once a qemu has been started with -incoming pause the migration can be started by issuing: migrate -u uri Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- hmp-commands.hx | 11 +++ hmp.c | 6 --

[Qemu-devel] [PATCH 0/3] -incoming pause

2015-02-10 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com This patchset provides a way of setting options on an incoming migration before the fd/process/socket has been created. start qemu with -incoming pause use the monitor to set whatever you need migrate -u theuri Dave Dr. David Alan

[Qemu-devel] [PATCH 1/1] unbreak dtrace tracing due to double _ in rdma names

2015-02-16 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com It looks like the dtrace trace code gets upset if you have trace names with __ in, which the migration/rdma.c code does. Rename the functions and the associated traces. Fixes: 733252deb8b7d37beacda5976c2769e18642b2fa Signed-off-by: Dr. David

[Qemu-devel] [PATCH v3 0/3] -incoming defer

2015-02-19 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com This patchset provides a way of setting options on an incoming migration before the fd/process/socket has been created. start qemu with -incoming defer use the monitor to set whatever you need migrate_incoming theuri v3:

[Qemu-devel] [PATCH v3 1/3] Add -incoming defer

2015-02-19 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com -incoming defer causes qemu to wait for an incoming migration to be specified later. The monitor can be used to set migration capabilities that may affect the incoming connection process. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com

[Qemu-devel] [PATCH v3 2/3] Add migrate_incoming

2015-02-19 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Add migrate_incoming/migrate-incoming to start an incoming migration. Once a qemu has been started with -incoming defer the migration can be started by issuing: migrate_incoming uri Signed-off-by: Dr. David Alan Gilbert

[Qemu-devel] [PATCH v3 3/3] Document -incoming options

2015-02-19 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Document the various URI formats for -incoming, the previous manpage and help text was wrong (out of date?) Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com Reviewed-by: Juan Quintela quint...@redhat.com --- qemu-options.hx | 29

[Qemu-devel] [RFC 0/1] Incoming migration vs early monitor commands

2015-01-29 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com The attached patch allows you to execute QMP commands from the command line prior to -incoming or loadvm. I've hit a few cases where we need to pass some state to an incoming migration, either: 1) Before it starts processing data e.g. Liang

[Qemu-devel] [RFC 1/1] Execute arbitrary QMP commands from command line

2015-01-29 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com For an incoming migration it's potentially useful to be able to set capabilities and parameters prior to opening the connection, while a separate option for that would have been possible it seems better to give access to all the existing migration

[Qemu-devel] [PATCH 1/1] Print PID and time in stderr traces

2015-01-13 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com When debugging migration it's useful to know the PID of each trace message so you can figure out if it came from the source or the destination. Printing the time makes it easy to do latency measurements or timings between trace points.

[Qemu-devel] [PATCH 0/3] Migration cancel with dead network

2015-01-08 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com If the remote host, or networking dies during a migration, the socket can be waiting for a long timeout, and migration_cancel can't complete the cancel for a long time (and you can't start a new one to somewhere else). (Where 'long' is the TCP

[Qemu-devel] [PATCH 2/3] Handle bi-directional communication for fd migration

2015-01-08 Thread Dr. David Alan Gilbert (git)
From: Cristian Klein cristian.kl...@cs.umu.se libvirt prefers opening the TCP connection itself, for two reasons. First, connection failed errors can be detected easier, without having to parse qemu's error output. Second, libvirt might be asked to secure the transfer by tunnelling the

[Qemu-devel] [PATCH 1/3] socket shutdown

2015-01-08 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Add QEMUFile interface to allow a socket to be 'shut down' - i.e. any reads/writes will fail (and any blocking read/write will be woken). Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- include/migration/qemu-file.h | 10 ++

[Qemu-devel] [PATCH 3/3] migration_cancel: shutdown migration socket

2015-01-08 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Force shutdown on migration socket on cancel to cause the cancel to complete even if the socket is blocked on a dead network. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- migration/migration.c | 12 1 file changed,

[Qemu-devel] [PATCH 0/2] RFC: Rolling statistics

2015-02-27 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Hi, This is an attempt at a generic rolling statistics utility to allow data (e.g. bandwidth usage, times etc) to be collected easily. They hold some basic values (min/max/mean/weighted mean) and the last 'n' raw values.I'd like to use this

[Qemu-devel] [PATCH 2/2] Tests for rolling statistics code

2015-02-27 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- tests/Makefile | 3 + tests/test-rolling-stats.c | 161 + 2 files changed, 164 insertions(+) create mode 100644

[Qemu-devel] [PATCH 1/2] Rolling statistics utilities

2015-02-27 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com There are various places where it's useful to hold a series of values that change over time and get summaries about them. This provides: - a count of the number of items - min/max - mean - a weighted mean (where you can set the weight

[Qemu-devel] [RFC v2 0/4] Rolling statistics utilities

2015-03-04 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Hi, This is an attempt at a generic rolling statistics utility to allow data (e.g. bandwidth usage, times etc) to be collected easily. They hold some basic values (min/max/mean/weighted mean) and the last 'n' raw values.I'd like to use this

[Qemu-devel] [RFC v2 3/4] hmp: Add a helper function for printing out a Rolling Statistics set

2015-03-04 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- hmp.c | 21 + 1 file changed, 21 insertions(+) diff --git a/hmp.c b/hmp.c index 735097c..20241d8 100644 --- a/hmp.c +++ b/hmp.c @@ -138,6 +138,27 @@ void

[Qemu-devel] [RFC v2 1/4] RollingStats qapi type

2015-03-04 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com This adds a QAPI type that holds the results of statistics collected by RStats. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- qapi-schema.json | 30 ++ 1 file changed, 30 insertions(+) diff --git

[Qemu-devel] [RFC v2 2/4] Rolling statistics utilities

2015-03-04 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com There are various places where it's useful to hold a series of values that change over time and get summaries about them. This provides: - a count of the number of items - min/max - mean - a weighted mean (where you can set the weight

[Qemu-devel] [RFC v2 4/4] Example use of rolling statistics in migration

2015-03-04 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com This is an example use of the rolling statistics to watch the 'expected downtime' in a bit more detail than the current summary figure. Example outputs from a simple run: HMP: expected downtime stats: Min/Max: 222, 1634 Mean: 983.8 (Weighted:

[Qemu-devel] [RFC 0/1] Rolling stats on colo

2015-03-05 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Hi, I'm getting COLO running on a couple of our machines here and wanted to see what was actually going on, so I merged in my recent rolling-stats code: http://lists.gnu.org/archive/html/qemu-devel/2015-03/msg00648.html with the following

[Qemu-devel] [RFC 1/1] COLO: Add primary side rolling statistics

2015-03-05 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Record: Checkpoint lifetime (ms) Pause time due to checkpoint (ms) Checkpoint size (bytes) Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- hmp.c | 12 include/migration/migration.h | 3 +++

[Qemu-devel] [PATCH v5 16/45] Add migration-capability boolean for postcopy-ram.

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com Reviewed-by: Eric Blake ebl...@redhat.com --- include/migration/migration.h | 1 + migration/migration.c | 9 + qapi-schema.json | 7 ++- 3 files changed,

[Qemu-devel] [PATCH v5 20/45] Modify savevm handlers for postcopy

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Modify save_live_pending to return separate postcopiable and non-postcopiable counts. Add 'can_postcopy' to allow a device to state if it can postcopy Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- arch_init.c | 15

[Qemu-devel] [PATCH v5 41/45] Start up a postcopy/listener thread ready for incoming page data

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com The loading of a device state (during postcopy) may access guest memory that's still on the source machine and thus might need a page fill; split off a separate thread that handles the incoming page data so that the original incoming migration code

[Qemu-devel] [PATCH v5 44/45] Disable mlock around incoming postcopy

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Userfault doesn't work with mlock; mlock is designed to nail down pages so they don't move, userfault is designed to tell you when they're not there. munlock the pages we userfault protect before postcopy. mlock everything again at the end if

[Qemu-devel] [PATCH v5 11/45] Return path: Send responses from destination to source

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Add migrate_send_rp_message to send a message from destination to source along the return path. (It uses a mutex to let it be called from multiple threads) Add migrate_send_rp_shut to send a 'shut' message to indicate the destination is

[Qemu-devel] [PATCH v5 23/45] migrate_start_postcopy: Command to trigger transition to postcopy

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Once postcopy is enabled (with migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the: migrate_start_postcopy command must be issued. Postcopy will start sometime after this (when it's

[Qemu-devel] [PATCH v5 09/45] Migration commands

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Create QEMU_VM_COMMAND section type for sending commands from source to destination. These commands are not intended to convey guest state but to control the migration process. For use in postcopy. Signed-off-by: Dr. David Alan Gilbert

[Qemu-devel] [PATCH v5 15/45] Rework loadvm path for subloops

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Postcopy needs to have two migration streams loading concurrently; one from memory (with the device state) and the other from the fd with the memory transactions. Split the core of qemu_loadvm_state out so we can use it for both. Allow the inner

[Qemu-devel] [PATCH v5 25/45] qemu_savevm_state_complete: Postcopy changes

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com When postcopy calls qemu_savevm_state_complete it's not really the end of migration, so skip: a) Finishing postcopiable iterative devices - they'll carry on b) The termination byte on the end of the stream. We then also add:

[Qemu-devel] [PATCH v5 07/45] Return path: Open a return path on QEMUFile for sockets

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Postcopy needs a method to send messages from the destination back to the source, this is the 'return path'. Wire it up for 'socket' QEMUFile's using a dup'd fd. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com ---

[Qemu-devel] [PATCH v5 30/45] Postcopy: Postcopy startup in migration thread

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Rework the migration thread to setup and start postcopy. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- include/migration/migration.h | 3 + migration/migration.c | 161 --

[Qemu-devel] [PATCH v5 05/45] Create MigrationIncomingState

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com There are currently lots of pieces of incoming migration state scattered around, and postcopy is adding more, and it seems better to try and keep it together. allocate MIS in process_incoming_migration_co Signed-off-by: Dr. David Alan Gilbert

[Qemu-devel] [PATCH v5 22/45] postcopy: OS support test

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Provide a check to see if the OS we're running on has all the bits needed for postcopy. Creates postcopy-ram.c which will get most of the other helpers we need. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com ---

[Qemu-devel] [PATCH v5 10/45] Return path: Control commands

2015-02-25 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Add two src-dest commands: * OPEN_RETURN_PATH - To request that the destination open the return path * SEND_PING - Request an acknowledge from the destination Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com ---

[Qemu-devel] [PATCH 08/10] Rework ram block hash

2015-04-20 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com RDMA uses a hash from block offset-RAM Block; this isn't needed on the destination, and now that the destination sorts the ramblock list, is harder to maintain. Split the hash so that it's only generated on the source. Signed-off-by: Dr. David

[Qemu-devel] [PATCH 06/10] Remove unneeded memset

2015-04-20 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- migration/rdma.c | 1 - 1 file changed, 1 deletion(-) diff --git a/migration/rdma.c b/migration/rdma.c index e43fae4..4f7dd0d 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@

[Qemu-devel] [PATCH 09/10] Sort destination RAMBlocks to be the same as the source

2015-04-20 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Use the order of incoming RAMBlocks from the source to record an index number; that then allows us to sort the destination local RAMBlock list to match the source. Now that the RAMBlocks are known to be in the same order, this simplifies the RDMA

[Qemu-devel] [PATCH 04/10] Translate offsets to destination address space

2015-04-20 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com The 'offset' field in RDMACompress and 'current_addr' field in RDMARegister are commented as being offsets within a particular RAMBlock, however they appear to actually be offsets within the ram_addr_t space. The code currently assumes that the

[Qemu-devel] [PATCH 00/10] Remove RDMA migration dependence on RAMBlock offset

2015-04-20 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com RDMA migration currently relies on the source and destination RAMBlocks having the same offsets within ram_addr_t space; unfortunately that's just not true when: a) You hotplug on the source but then create the device on the command line

[Qemu-devel] [PATCH 03/10] Store block name in local blocks structure

2015-04-20 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com In a later patch the block name will be used to match up two views of the block list. Keep a copy of the block name with the local block list. (At some point it could be argued that it would be best just to let migration see the innards of

[Qemu-devel] [PATCH 01/10] Rename RDMA structures to make destination clear

2015-04-20 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com RDMA has two data types that are named confusingly; RDMALocalBlock (pointed to indirectly by local_ram_blocks) RDMARemoteBlock (pointed to by block in RDMAContext) RDMALocalBlocks, as the name suggests is a data strucuture that represents

[Qemu-devel] [PATCH 07/10] Simplify rdma_delete_block and remove it's dependence on the hash

2015-04-20 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com rdma_delete_block is currently very general, but it's only used in cleanup at the end. Simplify it and remove it's dependence on the hash table and remove all of the hash-table regeneration designed to handle the (unused) case of deleting an

[Qemu-devel] [PATCH 02/10] qemu_ram_foreach_block: pass up error value, and down the ramblock name

2015-04-20 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com check the return value of the function it calls and error if it's non-0 Fixup qemu_rdma_init_one_block that is the only current caller, and rdma_add_block the only function it calls using it. Pass the name of the ramblock to the function; helps

[Qemu-devel] [PATCH 05/10] Rework ram_control_load_hook to hook during block load

2015-04-20 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com We need the names of RAMBlocks as they're loaded for RDMA, reuse an existing QEMUFile hook with some small mods. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- arch_init.c | 4 +++- include/migration/migration.h

[Qemu-devel] [PATCH 10/10] Sanity check RDMA remote data

2015-04-20 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Perform some basic (but probably not complete) sanity checking on requests from the RDMA source. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- migration/rdma.c | 30 ++ 1 file changed, 30 insertions(+)

[Qemu-devel] [PATCH 1/1] Add qemu_get_counted_string to read a string prefixed by a count byte

2015-05-15 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com and use it in loadvm_state and ram_load. Where ever it's used, check the return and error if it failed. Minor: ram_load was using a 257 byte array for its string, the maximum length is 255 bytes + 0 terminator, so fix to 256

[Qemu-devel] [PATCH 1/4] Merge section header writing

2015-05-19 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com The header writing for device sections is open coded in a few places, merge it into one. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- savevm.c | 73 +--- 1 file changed,

[Qemu-devel] [PATCH 3/4] Add a protective section footer

2015-05-19 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Badly formatted migration streams can go undetected or produce misleading errors due to a lock of checking at the end of sections. In particular a section that adds an extra 0x00 at the end causes what looks like a normal end of stream and thus

[Qemu-devel] [PATCH 2/4] Disable section footers on older machine types

2015-05-19 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com The next patch adds section footers; but we don't want to break migration compatibility so disable them on older machine types Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- hw/i386/pc_piix.c | 2 ++ hw/i386/pc_q35.c

[Qemu-devel] [PATCH 4/4] Teach analyze-migration.py about section footers

2015-05-19 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- scripts/analyze-migration.py | 5 + 1 file changed, 5 insertions(+) diff --git a/scripts/analyze-migration.py b/scripts/analyze-migration.py index 0c8b22f..f6894be 100755 ---

[Qemu-devel] [PATCH 0/4] Add section footers to detect corrupted migration streams

2015-05-19 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Badly formatted migration streams can go undetected or produce misleading errors due to a lock of checking at the end of sections. In particular a section that adds an extra 0x00 at the end causes what looks like a normal end of stream and thus

[Qemu-devel] [PATCH v6 09/47] Add wrapper for setting blocking status on a QEMUFile

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Add a wrapper to change the blocking status on a QEMUFile rather than having to use qemu_set_block(qemu_get_fd(f)); it seems best to avoid exposing the fd since not all QEMUFile's really have one. With this wrapper we could move the implementation

[Qemu-devel] [PATCH v6 07/47] Move copy out of qemu_peek_buffer

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com qemu_peek_buffer currently copies the data it reads into a buffer, however the next patch wants access to the buffer without the copy, hence rework to remove the copy to the layer above. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com

[Qemu-devel] [PATCH v6 32/47] Postcopy: Postcopy startup in migration thread

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Rework the migration thread to setup and start postcopy. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- include/migration/migration.h | 3 + migration/migration.c | 163 --

[Qemu-devel] [PATCH v6 26/47] migrate_start_postcopy: Command to trigger transition to postcopy

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Once postcopy is enabled (with migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the: migrate_start_postcopy command must be issued. Postcopy will start sometime after this (when it's

[Qemu-devel] [PATCH v6 18/47] Move loadvm_handlers into MigrationIncomingState

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com In postcopy we need the loadvm_handlers to be used in a couple of different instances of the loadvm loop/routine, and thus it can't be local any more. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com Reviewed-by: David Gibson

[Qemu-devel] [PATCH v6 37/47] postcopy_ram.c: place_page and helpers

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com postcopy_place_page (etc) provide a way for postcopy to place a page into guests memory atomically (using the copy ioctl on the ufd). Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- include/migration/migration.h| 1 +

[Qemu-devel] [PATCH v6 44/47] postcopy: Wire up loadvm_postcopy_handle_ commands

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Wire up more of the handlers for the commands on the destination side, in particular loadvm_postcopy_handle_run now has enough to start the guest running. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- savevm.c | 29

[Qemu-devel] [PATCH v6 42/47] Postcopy; Handle userfault requests

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com userfaultfd is a Linux syscall that gives an fd that receives a stream of notifications of accesses to pages registered with it and allows the program to acknowledge those stalls and tell the accessing thread to carry on. Signed-off-by: Dr. David

[Qemu-devel] [PATCH v6 45/47] End of migration for postcopy

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Tweak the end of migration cleanup; we don't want to close stuff down at the end of the main stream, since the postcopy is still sending pages on the other thread. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com ---

[Qemu-devel] [PATCH v6 01/47] Start documenting how postcopy works.

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- docs/migration.txt | 167 + 1 file changed, 167 insertions(+) diff --git a/docs/migration.txt b/docs/migration.txt index

[Qemu-devel] [PATCH v6 15/47] Return path: Send responses from destination to source

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Add migrate_send_rp_message to send a message from destination to source along the return path. (It uses a mutex to let it be called from multiple threads) Add migrate_send_rp_shut to send a 'shut' message to indicate the destination is

[Qemu-devel] [PATCH v6 14/47] Return path: Control commands

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Add two src-dest commands: * OPEN_RETURN_PATH - To request that the destination open the return path * PING - Request an acknowledge from the destination Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com Reviewed-by: David Gibson

[Qemu-devel] [PATCH v6 27/47] MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com 'MIGRATION_STATUS_POSTCOPY_ACTIVE' is entered after migrate_start_postcopy 'migration_postcopy_phase' is provided for other sections to know if they're in postcopy. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com Reviewed-by: David

[Qemu-devel] [PATCH v6 31/47] postcopy: ram_enable_notify to switch on userfault

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Mark the area of RAM as 'userfault' Start up a fault-thread to handle any userfaults we might receive from it (to be filled in later) Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com Reviewed-by: David Gibson da...@gibson.dropbear.id.au

[Qemu-devel] [PATCH v6 35/47] Page request: Process incoming page request

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com On receiving MIG_RPCOMM_REQ_PAGES look up the address and queue the page. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- arch_init.c | 64 ++- include/exec/cpu-all.h

[Qemu-devel] [PATCH v6 00/47] Postcopy implementation

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com This is the 6th cut of my version of postcopy; it is designed for use with the Linux kernel additions posted by Andrea Arcangeli here: git clone --reference linux -b userfault18 git://git.kernel.org/pub/scm/linux/kernel/git/andrea/aa.git (Note

[Qemu-devel] [PATCH v6 02/47] Split header writing out of qemu_savevm_state_begin

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Split qemu_savevm_state_begin to: qemu_savevm_state_header That writes the initial file header. qemu_savevm_state_beginThat sets up devices and does the first device pass. Used later in postcopy.

[Qemu-devel] [PATCH v6 04/47] Add qemu_get_counted_string to read a string prefixed by a count byte

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com and use it in loadvm_state and ram_load. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- arch_init.c | 5 + include/migration/qemu-file.h | 3 +++ migration/qemu-file.c | 16 savevm.c

[Qemu-devel] [PATCH v6 16/47] Return path: Source handling of return path

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Open a return path, and handle messages that are received upon it. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- include/migration/migration.h | 8 ++ migration/migration.c | 177

[Qemu-devel] [PATCH v6 10/47] Rename save_live_complete to save_live_complete_precopy

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com In postcopy we're going to need to perform the complete phase for postcopiable devices at a different point, start out by renaming all of the 'complete's to make the difference obvious. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com ---

[Qemu-devel] [PATCH v6 21/47] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com The state of the postcopy process is managed via a series of messages; * Add wrappers and handlers for sending/receiving these messages * Add state variable that track the current state of postcopy Signed-off-by: Dr. David Alan Gilbert

[Qemu-devel] [PATCH v6 41/47] Host page!=target page: Cleanup bitmaps

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Prior to the start of postcopy, ensure that everything that will be transferred later is a whole host-page in size. This is accomplished by discarding partially transferred host pages and marking any that are partially dirty as fully dirty.

[Qemu-devel] [PATCH v6 11/47] Return path: Open a return path on QEMUFile for sockets

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Postcopy needs a method to send messages from the destination back to the source, this is the 'return path'. Wire it up for 'socket' QEMUFile's using a dup'd fd. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com ---

[Qemu-devel] [PATCH v6 05/47] Create MigrationIncomingState

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com There are currently lots of pieces of incoming migration state scattered around, and postcopy is adding more, and it seems better to try and keep it together. allocate MIS in process_incoming_migration_co Signed-off-by: Dr. David Alan Gilbert

[Qemu-devel] [PATCH v6 23/47] migrate_init: Call from savevm

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Suspend to file is very much like a migrate, and it makes life easier if we have the Migration state available, so initialise it in the savevm.c code for suspending. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com Reviewed-by: David

[Qemu-devel] [PATCH v6 25/47] postcopy: OS support test

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Provide a check to see if the OS we're running on has all the bits needed for postcopy. Creates postcopy-ram.c which will get most of the other helpers we need. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com ---

[Qemu-devel] [PATCH v6 06/47] Provide runtime Target page information

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com The migration code generally is built target-independent, however there are a few places where knowing the target page size would avoid artificially moving stuff into arch_init. Provide 'qemu_target_page_bits()' that returns TARGET_PAGE_BITS to

[Qemu-devel] [PATCH v6 13/47] Migration commands

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Create QEMU_VM_COMMAND section type for sending commands from source to destination. These commands are not intended to convey guest state but to control the migration process. For use in postcopy. Signed-off-by: Dr. David Alan Gilbert

[Qemu-devel] [PATCH v6 03/47] qemu_ram_foreach_block: pass up error value, and down the ramblock name

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com check the return value of the function it calls and error if it's non-0 Fixup qemu_rdma_init_one_block that is the only current caller, and rdma_add_block the only function it calls using it. Pass the name of the ramblock to the function; helps

[Qemu-devel] [PATCH v6 12/47] Return path: socket_writev_buffer: Block even on non-blocking fd's

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com The destination sets the fd to non-blocking on incoming migrations; this also affects the return path from the destination, and thus we need to make sure we can safely write to the return path. Signed-off-by: Dr. David Alan Gilbert

[Qemu-devel] [PATCH v6 19/47] Rework loadvm path for subloops

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Postcopy needs to have two migration streams loading concurrently; one from memory (with the device state) and the other from the fd with the memory transactions. Split the core of qemu_loadvm_state out so we can use it for both. Allow the inner

[Qemu-devel] [PATCH v6 20/47] Add migration-capability boolean for postcopy-ram.

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com The 'postcopy ram' capability allows postcopy migration of RAM; note that the migration starts off in precopy mode until postcopy mode is triggered (see the migrate_start_postcopy patch later in the series). Signed-off-by: Dr. David Alan Gilbert

[Qemu-devel] [PATCH v6 24/47] Modify save_live_pending for postcopy

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Modify save_live_pending to return separate postcopiable and non-postcopiable counts. Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com --- arch_init.c | 8 ++-- include/migration/vmstate.h | 5 +++--

[Qemu-devel] [PATCH v6 30/47] postcopy: Incoming initialisation

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Signed-off-by: Dr. David Alan Gilbert dgilb...@redhat.com Reviewed-by: David Gibson da...@gibson.dropbear.id.au --- arch_init.c | 11 include/migration/migration.h| 3 + include/migration/postcopy-ram.h | 12

[Qemu-devel] [PATCH v6 28/47] Add qemu_savevm_state_complete_postcopy

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Add qemu_savevm_state_complete_postcopy to complement qemu_savevm_state_complete_precopy together with a new save_live_complete_postcopy method on devices. The save_live_complete_precopy method is called on all devices during a precopy migration,

[Qemu-devel] [PATCH v6 46/47] Disable mlock around incoming postcopy

2015-04-14 Thread Dr. David Alan Gilbert (git)
From: Dr. David Alan Gilbert dgilb...@redhat.com Userfault doesn't work with mlock; mlock is designed to nail down pages so they don't move, userfault is designed to tell you when they're not there. munlock the pages we userfault protect before postcopy. mlock everything again at the end if

<    1   2   3   4   5   6   7   8   9   10   >