Re: [Qemu-devel] [PATCH v2 0/6] vhost-user: Specify and implement device IOTLB support

[Jason Wang]

On 2017年06月01日 21:59, Michael S. Tsirkin wrote:

On Wed, May 31, 2017 at 04:33:33PM +0800, Jason Wang wrote:


On 2017年05月31日 02:20, Michael S. Tsirkin wrote:

On Fri, May 26, 2017 at 04:28:52PM +0200, Maxime Coquelin wrote:

This series aims at specifying ans implementing the protocol update
required to support device IOTLB with user backends.

In this second non-RFC version, main changes are:
   - spec fixes and clarification
   - rings information update has been restored back to ring enablement time
   - Work around GCC 4.4.7 limitation wrt assignment in unnamed union at
declaration time.

The series can be tested with vhost_iotlb_proto_v2 branch on my gitlab
account[0].

The slave requests channel part is re-used from Marc-André's series submitted
last year[1], with main changes from original version being request/feature
names renaming and addition of the REPLY_ACK feature support.

Regarding IOTLB protocol, one noticeable change is the IOTLB miss request
reply made optionnal (i.e. only if slave requests it by setting the
VHOST_USER_NEED_REPLY flag in the message header). This change provides
more flexibility in the backend implementation of the feature.

The protocol is very close to kernel backends, except that a new
communication channel is introduced to enable the slave to send
requests to the master.

[0]:https://gitlab.com/mcoquelin/dpdk-next-virtio/commits/vhost_iotlb_proto_v2
[1]:https://lists.gnu.org/archive/html/qemu-devel/2016-04/msg00095.html

Overall, this looks good to me. I do think patch 3 isn't a good idea
though, if slave wants something let it request it.

Need to find out why does vhost in kernel want the used ring iotlb at
start time - especially considering we aren't even guaranteed one entry
covers the whole ring, and invalidates should affect all addresses at
least in theory.



The reason is probably we want to verify whether or not we could correctly
access used ring in vhost_vq_init_access(). It was there since vhost_net is
introduced. We can think to remove this limitation maybe.

Thanks


Well that's only called if iotlb is disabled:

 if (!vq->iotlb &&
 !access_ok(VERIFY_READ, >used->idx, sizeof vq->used->idx)) {
 r = -EFAULT;
 goto err;
 }

Could you try removing that and see what breaks?



Looks not, the issue is vhost_update_used_flags() which needs device 
IOTLB translation. If we don't fill IOTLB in advance, it will return 
-EFAULT.


For simplicity, I don't implement control path device IOTLB miss. If you 
care about the incomplete length, we can refine vhost_iotlb_miss() to 
make sure it covers all size.


Thanks

[Qemu-devel] [PULL] qemu-sparc updates

[Mark Cave-Ayland]

Hi Peter,

This request contains the SPARC QOMify work which fell through the cracks 
during the last release.
Please pull.


ATB,

Mark.


The following changes since commit 43771d5d92312504305c19abe29ec5bfabd55f01:

  Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2017-05-31' into 
staging (2017-06-01 16:39:16 +0100)

are available in the git repository at:


  https://github.com/mcayland/qemu.git tags/qemu-sparc-signed

for you to fetch changes up to 78fb261db18d3e12f68304c0d91a85e0a3bbc757:

  hw/sparc64: QOM'ify sun4u.c (2017-06-02 05:54:43 +0100)


qemu-sparc update


xiaoqiang zhao (8):
  hw/misc: QOM'ify eccmemctl.c
  hw/dma: QOM'ify sparc32_dma.c
  hw/dma: QOM'ify sun4m_iommu.c
  hw/misc: QOM'ify slavio_misc.c
  hw/timer: QOM'ify m48txx_sysbus
  hw/timer: QOM'ify slavio_timer
  hw/sparc: QOM'ify sun4m.c
  hw/sparc64: QOM'ify sun4u.c

 hw/dma/sparc32_dma.c|   25 +-
 hw/dma/sun4m_iommu.c|   12 +--
 hw/misc/eccmemctl.c |   25 ++
 hw/misc/slavio_misc.c   |   43 +++--
 hw/sparc/sun4m.c|   54 +--
 hw/sparc64/sun4u.c  |   20 --
 hw/timer/m48t59.c   |   38 -
 hw/timer/slavio_timer.c |   12 +--
 8 files changed, 105 insertions(+), 124 deletions(-)

Re: [Qemu-devel] [PATCH v3 0/8] QOM'ify work for sparc

[Mark Cave-Ayland]

On 25/05/17 14:34, xiaoqiang zhao wrote:

> This patch set aims for QOM'ifying code relate with sparc.
> It is part of my QOM'ify work of qemu code base.
> 
> Changes:
> 
> v2 -> v3:
> * rebased on the latest master
> * squash two patches about m48txx_sysbus into one
> 
> v1 ->  v2: 
> * rebased on the latest master
> 
> xiaoqiang zhao (8):
>   hw/misc: QOM'ify eccmemctl.c
>   hw/dma: QOM'ify sparc32_dma.c
>   hw/dma: QOM'ify sun4m_iommu.c
>   hw/misc: QOM'ify slavio_misc.c
>   hw/timer: QOM'ify m48txx_sysbus
>   hw/timer: QOM'ify slavio_timer
>   hw/sparc: QOM'ify sun4m.c
>   hw/sparc64: QOM'ify sun4u.c
> 
>  hw/dma/sparc32_dma.c| 25 ++-
>  hw/dma/sun4m_iommu.c| 12 +--
>  hw/misc/eccmemctl.c | 25 ++-
>  hw/misc/slavio_misc.c   | 43 ---
>  hw/sparc/sun4m.c| 54 
> +
>  hw/sparc64/sun4u.c  | 20 +-
>  hw/timer/m48t59.c   | 38 +-
>  hw/timer/slavio_timer.c | 12 +--
>  8 files changed, 105 insertions(+), 124 deletions(-)

These look good here, so I've applied them to my qemu-sparc branch and
will send a pull request shortly.


ATB,

Mark.

Re: [Qemu-devel] [Qemu-arm] [PATCH 09/13] armv7m: Implement M profile default memory map

[Philippe Mathieu-Daudé]

Hi Peter,

On 05/30/2017 12:11 PM, Peter Maydell wrote:

On 30 May 2017 at 15:56, Philippe Mathieu-Daudé  wrote:

Hi Peter,

On 04/25/2017 09:07 AM, Peter Maydell wrote:


From: Michael Davidsaver 

Add support for the M profile default memory map which is used
if the MPU is not present or disabled.

The main differences in behaviour from implementing this
correctly are that we set the PAGE_EXEC attribute on
the right regions of memory, such that device regions
are not executable.

Signed-off-by: Michael Davidsaver 
[PMM: rephrased comment and commit message; don't mark
 the flash memory region as not-writable]



"not-writable by system caches" maybe to clarify?


(Note that this is a comment describing something I
deleted from Michael's original patch.)
No, by 'not-writable' here I mean "not writeable by the CPU".



Ok!




Signed-off-by: Peter Maydell 
---
 target/arm/helper.c | 35 ++-
 1 file changed, 26 insertions(+), 9 deletions(-)

diff --git a/target/arm/helper.c b/target/arm/helper.c
index 9e1ed1c..51662ad 100644
--- a/target/arm/helper.c
+++ b/target/arm/helper.c
@@ -8129,18 +8129,35 @@ static inline void
get_phys_addr_pmsav7_default(CPUARMState *env,
 ARMMMUIdx mmu_idx,
 int32_t address, int
*prot)
 {
-*prot = PAGE_READ | PAGE_WRITE;
-switch (address) {
-case 0xF000 ... 0x:
-if (regime_sctlr(env, mmu_idx) & SCTLR_V) { /* hivecs execing is
ok */
+if (!arm_feature(env, ARM_FEATURE_M)) {
+*prot = PAGE_READ | PAGE_WRITE;
+switch (address) {
+case 0xF000 ... 0x:
+if (regime_sctlr(env, mmu_idx) & SCTLR_V) {
+/* hivecs execing is ok */
+*prot |= PAGE_EXEC;
+}
+break;
+case 0x ... 0x7FFF:



 *prot |= PAGE_EXEC;


I checked at Table B3-1 on the ARMv7-M Architecture Reference Manual I got
at https://static.docs.arm.com/ddi0403/e/DDI0403E_B_armv7m_arm.pdf and the
on-chip peripheral address space at 0x4000 is eXecute Never.


This is the arm of the if() that deals with R profile, and R profile's


Oh I completely misunderstood that if() indeed. R and also A I suppose.


background region is completely different to M profile. (In any
case the patch shouldn't change the behaviour there.)


+break;
+}
+} else {
+/* Default system address map for M profile cores.
+ * The architecture specifies which regions are execute-never;
+ * at the MPU level no other checks are defined.
+ */
+switch (address) {
+case 0x ... 0x1fff: /* ROM */
+case 0x2000 ... 0x3fff: /* SRAM */
+case 0x6000 ... 0x7fff: /* RAM */
+case 0x8000 ... 0x9fff: /* RAM */
+*prot = PAGE_READ | PAGE_WRITE | PAGE_EXEC;
+break;
+default: /* Peripheral, 2x Device, and System */
+*prot = PAGE_READ | PAGE_WRITE;



This body is correct, however what do you think about using cases with
comments instead of 'default'? This would be clearer.


Yeah, we could do that. I think this is one of those cases where
I opted to go with how Michael had already written the code rather
than rewriting it.

/* Default system address map for M profile cores.
 * The architecture specifies which regions are execute-never;
 * at the MPU level no other checks are defined.
 */
switch (address) {
case 0x ... 0x1fff: /* ROM */
case 0x2000 ... 0x3fff: /* SRAM */
case 0x6000 ... 0x7fff: /* RAM */
case 0x8000 ... 0x9fff: /* RAM */
*prot = PAGE_READ | PAGE_WRITE | PAGE_EXEC;
break;
case 0x4000 ... 0x5fff: /* Peripheral */
case 0xa000 ... 0xbfff: /* Device */
case 0xc000 ... 0xdfff: /* Device */
case 0xe000 ... 0x: /* System */
*prot = PAGE_READ | PAGE_WRITE;
break;
default:
g_assert_not_reached();
}

would be the explicit version.



I see you added that before your pull request, thank!

For what it's worth:
Reviewed-by: Philippe Mathieu-Daudé 


 }
-break;
-case 0x ... 0x7FFF:
-*prot |= PAGE_EXEC;
-break;
 }
-
 }

 static bool get_phys_addr_pmsav7(CPUARMState *env, uint32_t address,


thanks
-- PMM

Re: [Qemu-devel] [PATCH v2 2/4] gdbstub: rename cpu_index -> cpu_gdb_index

[Philippe Mathieu-Daudé]

On 06/01/2017 11:49 AM, Alex Bennée wrote:

This is to make it clear the index is purely a gdbstub function and
should not be confused with the value of cpu->cpu_index.

Signed-off-by: Alex Bennée 


Reviewed-by: Philippe Mathieu-Daudé 


---
 gdbstub.c  | 12 ++--
 include/exec/gdbstub.h |  7 ++-
 2 files changed, 12 insertions(+), 7 deletions(-)

diff --git a/gdbstub.c b/gdbstub.c
index a249846954..026d1fe6bb 100644
--- a/gdbstub.c
+++ b/gdbstub.c
@@ -837,7 +837,7 @@ static CPUState *find_cpu(uint32_t thread_id)
 CPUState *cpu;

 CPU_FOREACH(cpu) {
-if (cpu_index(cpu) == thread_id) {
+if (cpu_gdb_index(cpu) == thread_id) {
 return cpu;
 }
 }
@@ -925,7 +925,7 @@ static int gdb_handle_vcont(GDBState *s, const char *p)
 idx = tmp;
 /* 0 means any thread, so we pick the first valid CPU */
 if (!idx) {
-idx = cpu_index(first_cpu);
+idx = cpu_gdb_index(first_cpu);
 }

 /*
@@ -975,7 +975,7 @@ static int gdb_handle_packet(GDBState *s, const char 
*line_buf)
 case '?':
 /* TODO: Make this return the correct value for user-mode.  */
 snprintf(buf, sizeof(buf), "T%02xthread:%02x;", GDB_SIGNAL_TRAP,
- cpu_index(s->c_cpu));
+ cpu_gdb_index(s->c_cpu));
 put_packet(s, buf);
 /* Remove all the breakpoints when this query is issued,
  * because gdb is doing and initial connect and the state
@@ -1243,7 +1243,7 @@ static int gdb_handle_packet(GDBState *s, const char 
*line_buf)
 } else if (strcmp(p,"sThreadInfo") == 0) {
 report_cpuinfo:
 if (s->query_cpu) {
-snprintf(buf, sizeof(buf), "m%x", cpu_index(s->query_cpu));
+snprintf(buf, sizeof(buf), "m%x", cpu_gdb_index(s->query_cpu));
 put_packet(s, buf);
 s->query_cpu = CPU_NEXT(s->query_cpu);
 } else
@@ -1400,7 +1400,7 @@ static void gdb_vm_state_change(void *opaque, int 
running, RunState state)
 }
 snprintf(buf, sizeof(buf),
  "T%02xthread:%02x;%swatch:" TARGET_FMT_lx ";",
- GDB_SIGNAL_TRAP, cpu_index(cpu), type,
+ GDB_SIGNAL_TRAP, cpu_gdb_index(cpu), type,
  (target_ulong)cpu->watchpoint_hit->vaddr);
 cpu->watchpoint_hit = NULL;
 goto send_packet;
@@ -1434,7 +1434,7 @@ static void gdb_vm_state_change(void *opaque, int 
running, RunState state)
 break;
 }
 gdb_set_stop_cpu(cpu);
-snprintf(buf, sizeof(buf), "T%02xthread:%02x;", ret, cpu_index(cpu));
+snprintf(buf, sizeof(buf), "T%02xthread:%02x;", ret, cpu_gdb_index(cpu));

 send_packet:
 put_packet(s, buf);
diff --git a/include/exec/gdbstub.h b/include/exec/gdbstub.h
index f9708bbcd6..c4fe567600 100644
--- a/include/exec/gdbstub.h
+++ b/include/exec/gdbstub.h
@@ -58,7 +58,12 @@ void gdb_register_coprocessor(CPUState *cpu,
   gdb_reg_cb get_reg, gdb_reg_cb set_reg,
   int num_regs, const char *xml, int g_pos);

-static inline int cpu_index(CPUState *cpu)
+/* Return the GDB index for a given vCPU state.
+ *
+ * For user mode this is simply the thread id. In system mode GDB
+ * numbers CPUs from 1 as 0 is reserved as an "any cpu" index.
+ */
+static inline int cpu_gdb_index(CPUState *cpu)
 {
 #if defined(CONFIG_USER_ONLY)
 return cpu->host_tid;

Re: [Qemu-devel] [PATCH 00/11] Misc migration cleanups

[Philippe Mathieu-Daudé]

for patches 1 to 10:
Reviewed-by: Philippe Mathieu-Daudé 

On 06/01/2017 06:29 PM, Juan Quintela wrote:

Hi

This are a bit of everything:
- Remove unneeded argumets for migration_channel_incomming
- Lots of changes to make migration.h local to only migration
  * Move self_annonce_delay() to misc.h
I know this conflicts for announce changes from Vlad, but
I want to remove migration.h from being exported.
  * split registrantion functions to register.h
  * Almost everything uses vmsd's for registration
  * Move constants to the places that use it
And they are not used outside of migration/*
  * create global_state.c, as they don't belong anywhere else

ToSend:
  * RAMState is a dynamic variable on my tree
  * save_{setup,cleanup} and load_{setup,cleanup}
Yes Kevin, that is for block layer
  * move all ram.c to use load_setup/cleanup

ToDo easy: (probably post 2.10)
  * split qapi functions from migration.c
They don't belong there, and it would be clearer about what is configuratio 
and what is code
  * block.c and page_cache.c still use DPRINTF, they should move to use tracing
this is easy, volunteers, please.
  * Now that it is clear what functions are exported and which not, writting 
documentation could
be a good idea for them.

ToDo, difficult:
  * rdma.c -> this needs some love, it uses a completely different set of hooks 
that everything
else, should have to integrate somehow everything together.
  * abstract compression, xbzrle, postcopy and rdma into something that
is easier to understand.

Please, review.

Thanks, Juan.


Juan Quintela (11):
  migration: Remove MigrationState from migration_channel_incomming()
  migration: Move self_announce_delay() to misc.h
  migration: Split registration functions from vmstate.h
  migration: Move dump_vmsate_json_to_file() to misc.h
  migration: Move constants to savevm.h
  migration: Commands are only used inside migration.c
  migration: ram_control_* are implemented in qemu_file
  migration: create global_state.c
  migration: Move remaining exported functions to migration/misc.h
  migration: Move migration.h to migration/
  migration: Remove unneeded includes

 hw/i386/pc_piix.c|   3 +-
 hw/net/virtio-net.c  |   1 +
 hw/net/vmxnet3.c |   1 +
 hw/ppc/spapr.c   |   4 +-
 hw/s390x/s390-skeys.c|   1 +
 hw/s390x/s390-virtio-ccw.c   |   1 +
 hw/xen/xen-common.c  |   3 +-
 include/migration/colo.h |   3 -
 include/migration/global_state.h |  25 
 include/migration/misc.h |  28 +
 include/migration/register.h |  64 ++
 include/migration/vmstate.h  |  57 -
 migration/Makefile.objs  |   2 +-
 migration/block.c|   9 +-
 migration/channel.c  |   7 +-
 migration/channel.h  |   3 +-
 migration/colo-comm.c|   2 +-
 migration/colo-failover.c|   2 +
 migration/colo.c |   4 +-
 migration/exec.c |   4 +-
 migration/fd.c   |   4 +-
 migration/global_state.c | 139 +
 migration/migration.c| 176 +--
 {include/migration => migration}/migration.h |  67 --
 migration/postcopy-ram.c |   6 +-
 migration/qemu-file.c|   4 +-
 migration/qemu-file.h|  17 +++
 migration/ram.c  |   7 +-
 migration/rdma.c |   2 +-
 migration/savevm.c   |  11 +-
 migration/savevm.h   |  15 +++
 migration/socket.c   |   5 +-
 migration/tls.c  |   4 +-
 migration/vmstate-types.c|   2 +-
 migration/vmstate.c  |   3 +-
 qdev-monitor.c   |   2 +-
 slirp/slirp.c|   1 +
 tests/test-vmstate.c |   3 +-
 ui/spice-core.c  |   2 +-
 vl.c |   2 +-
 40 files changed, 368 insertions(+), 328 deletions(-)
 create mode 100644 include/migration/global_state.h
 create mode 100644 include/migration/register.h
 create mode 100644 migration/global_state.c
 rename {include/migration => migration}/migration.h (62%)

Re: [Qemu-devel] [PATCH v2 13/14] char: make chr_fe_deinit() optionaly delete backend

[Philippe Mathieu-Daudé]

Hi Marc-André,

The new boolean argument bothered me at first but I couldn't find nicer 
way (that's why I kept this patch review so long).


On 05/29/2017 05:45 AM, Marc-André Lureau wrote:

This simplifies removing a backend for a frontend user (no need to
retrive the associated driver and seperate delete call etc).


retrieve, separate



NB: many frontends have questionable handling of ending a chardev. They
should probably delete the backend to prevent broken reusage.

Signed-off-by: Marc-André Lureau 


Reviewed-by: Philippe Mathieu-Daudé 


---
 include/chardev/char-fe.h|  6 --
 backends/rng-egd.c   |  2 +-
 chardev/char-fe.c|  5 -
 chardev/char-mux.c   |  2 +-
 gdbstub.c| 15 ++-
 hw/char/serial.c |  2 +-
 hw/char/xen_console.c|  2 +-
 hw/core/qdev-properties-system.c |  2 +-
 hw/usb/ccid-card-passthru.c  |  5 +
 hw/usb/redirect.c|  4 +---
 monitor.c|  2 +-
 net/colo-compare.c   |  8 +++-
 net/filter-mirror.c  |  6 +++---
 net/vhost-user.c |  5 +
 tests/test-char.c| 22 --
 tests/vhost-user-test.c  |  4 +---
 16 files changed, 34 insertions(+), 58 deletions(-)

diff --git a/include/chardev/char-fe.h b/include/chardev/char-fe.h
index bd82093218..2cbb262f66 100644
--- a/include/chardev/char-fe.h
+++ b/include/chardev/char-fe.h
@@ -30,12 +30,14 @@ bool qemu_chr_fe_init(CharBackend *b, Chardev *s, Error 
**errp);

 /**
  * @qemu_chr_fe_deinit:
- *
+ * @b: a CharBackend
+ * @del: if true, delete the chardev backend
+*
  * Dissociate the CharBackend from the Chardev.
  *
  * Safe to call without associated Chardev.
  */
-void qemu_chr_fe_deinit(CharBackend *b);
+void qemu_chr_fe_deinit(CharBackend *b, bool del);

 /**
  * @qemu_chr_fe_get_driver:
diff --git a/backends/rng-egd.c b/backends/rng-egd.c
index ad3e1e5edf..e7ce2cac80 100644
--- a/backends/rng-egd.c
+++ b/backends/rng-egd.c
@@ -145,7 +145,7 @@ static void rng_egd_finalize(Object *obj)
 {
 RngEgd *s = RNG_EGD(obj);

-qemu_chr_fe_deinit(>chr);
+qemu_chr_fe_deinit(>chr, false);
 g_free(s->chr_name);
 }

diff --git a/chardev/char-fe.c b/chardev/char-fe.c
index 341221d029..3f90f0567c 100644
--- a/chardev/char-fe.c
+++ b/chardev/char-fe.c
@@ -211,7 +211,7 @@ unavailable:
 return false;
 }

-void qemu_chr_fe_deinit(CharBackend *b)
+void qemu_chr_fe_deinit(CharBackend *b, bool del)
 {
 assert(b);

@@ -224,6 +224,9 @@ void qemu_chr_fe_deinit(CharBackend *b)
 MuxChardev *d = MUX_CHARDEV(b->chr);
 d->backends[b->tag] = NULL;
 }
+if (del) {
+object_unparent(OBJECT(b->chr));
+}
 b->chr = NULL;
 }
 }
diff --git a/chardev/char-mux.c b/chardev/char-mux.c
index 106c682e7f..08570b915e 100644
--- a/chardev/char-mux.c
+++ b/chardev/char-mux.c
@@ -266,7 +266,7 @@ static void char_mux_finalize(Object *obj)
 be->chr = NULL;
 }
 }
-qemu_chr_fe_deinit(>chr);
+qemu_chr_fe_deinit(>chr, false);
 }

 void mux_chr_set_handlers(Chardev *chr, GMainContext *context)
diff --git a/gdbstub.c b/gdbstub.c
index 4251d23898..ec4e4b25be 100644
--- a/gdbstub.c
+++ b/gdbstub.c
@@ -1678,9 +1678,6 @@ void gdb_exit(CPUArchState *env, int code)
 {
   GDBState *s;
   char buf[4];
-#ifndef CONFIG_USER_ONLY
-  Chardev *chr;
-#endif

   s = gdbserver_state;
   if (!s) {
@@ -1690,19 +1687,13 @@ void gdb_exit(CPUArchState *env, int code)
   if (gdbserver_fd < 0 || s->fd < 0) {
   return;
   }
-#else
-  chr = qemu_chr_fe_get_driver(>chr);
-  if (!chr) {
-  return;
-  }
 #endif

   snprintf(buf, sizeof(buf), "W%02x", (uint8_t)code);
   put_packet(s, buf);

 #ifndef CONFIG_USER_ONLY
-  qemu_chr_fe_deinit(>chr);
-  object_unparent(OBJECT(chr));
+  qemu_chr_fe_deinit(>chr, true);
 #endif
 }

@@ -2002,9 +1993,7 @@ int gdbserver_start(const char *device)
NULL, _abort);
 monitor_init(mon_chr, 0);
 } else {
-if (qemu_chr_fe_get_driver(>chr)) {
-object_unparent(OBJECT(qemu_chr_fe_get_driver(>chr)));
-}
+qemu_chr_fe_deinit(>chr, true);
 mon_chr = s->mon_chr;
 memset(s, 0, sizeof(GDBState));
 s->mon_chr = mon_chr;
diff --git a/hw/char/serial.c b/hw/char/serial.c
index 23e5fe9d18..e1f12507bf 100644
--- a/hw/char/serial.c
+++ b/hw/char/serial.c
@@ -905,7 +905,7 @@ void serial_realize_core(SerialState *s, Error **errp)

 void serial_exit_core(SerialState *s)
 {
-qemu_chr_fe_deinit(>chr);
+qemu_chr_fe_deinit(>chr, false);

 timer_del(s->modem_status_poll);
 timer_free(s->modem_status_poll);
diff --git a/hw/char/xen_console.c b/hw/char/xen_console.c
index cb849c2e3e..f9af8cadf4 100644
--- a/hw/char/xen_console.c
+++ b/hw/char/xen_console.c
@@ -261,7 +261,7 @@ 

Re: [Qemu-devel] [PATCH v3 3/5] sockets: ensure we don't accept IPv4 clients when IPv4 is disabled

[Philippe Mathieu-Daudé]

On 06/01/2017 05:29 AM, Daniel P. Berrange wrote:

Currently if you disable listening on IPv4 addresses, via the
CLI flag ipv4=off, we still mistakenly accept IPv4 clients via
the IPv6 listener socket due to IPV6_V6ONLY flag being unset.

We must ensure IPV6_V6ONLY is always set if ipv4=off

This fixes the following scenarios

  -incoming tcp::9000,ipv6=on
  -incoming tcp:[::]:9000,ipv6=on
  -chardev socket,id=cdev0,host=,port=9000,server,nowait,ipv4=off
  -chardev socket,id=cdev0,host=,port=9000,server,nowait,ipv6=on
  -chardev socket,id=cdev0,host=::,port=9000,server,nowait,ipv4=off
  -chardev socket,id=cdev0,host=::,port=9000,server,nowait,ipv6=on

which all mistakenly accepted IPv4 clients

Reviewed-by: Eric Blake 
Signed-off-by: Daniel P. Berrange 


Reviewed-by: Philippe Mathieu-Daudé 


---
 util/qemu-sockets.c | 40 +++-
 1 file changed, 31 insertions(+), 9 deletions(-)

diff --git a/util/qemu-sockets.c b/util/qemu-sockets.c
index 81bc8de..852773d 100644
--- a/util/qemu-sockets.c
+++ b/util/qemu-sockets.c
@@ -104,17 +104,16 @@ NetworkAddressFamily inet_netfamily(int family)
  *   f t   PF_INET6
  *   t -   PF_INET
  *   t f   PF_INET
- *   t t   PF_INET6
+ *   t t   PF_INET6/PF_UNSPEC
  *
  * NB, this matrix is only about getting the necessary results
  * from getaddrinfo(). Some of the cases require further work
  * after reading results from getaddrinfo in order to fully
- * apply the logic the end user wants. eg with the last case
- * ipv4=t + ipv6=t + PF_INET6, getaddrinfo alone can only
- * guarantee the ipv6=t part of the request - we need more
- * checks to provide ipv4=t part of the guarantee. This is
- * outside scope of this method and not currently handled by
- * callers at all.
+ * apply the logic the end user wants.
+ *
+ * In the first and last cases, we must set IPV6_V6ONLY=0
+ * when binding, to allow a single listener to potentially
+ * accept both IPv4+6 addresses.
  */
 int inet_ai_family_from_address(InetSocketAddress *addr,
 Error **errp)
@@ -124,6 +123,23 @@ int inet_ai_family_from_address(InetSocketAddress *addr,
 error_setg(errp, "Cannot disable IPv4 and IPv6 at same time");
 return PF_UNSPEC;
 }
+if ((addr->has_ipv6 && addr->ipv6) && (addr->has_ipv4 && addr->ipv4)) {
+/*
+ * Some backends can only do a single listener. In that case
+ * we want empty hostname to resolve to "::" and then use the
+ * flag IPV6_V6ONLY==0 to get both protocols on 1 socket. This
+ * doesn't work for addresses other than "", so they're just
+ * inevitably broken until multiple listeners can be used,
+ * and thus we honour getaddrinfo automatic protocol detection
+ * Once all backends do multi-listener, remove the PF_INET6
+ * branch entirely.
+ */
+if (!addr->host || g_str_equal(addr->host, "")) {
+return PF_INET6;
+} else {
+return PF_UNSPEC;
+}
+}
 if ((addr->has_ipv6 && addr->ipv6) || (addr->has_ipv4 && !addr->ipv4)) {
 return PF_INET6;
 }
@@ -213,8 +229,14 @@ static int inet_listen_saddr(InetSocketAddress *saddr,
 port_max = saddr->has_to ? saddr->to + port_offset : port_min;
 for (p = port_min; p <= port_max; p++) {
 #ifdef IPV6_V6ONLY
-/* listen on both ipv4 and ipv6 */
-int v6only = 0;
+/*
+ * Deals with first & last cases in matrix in comment
+ * for inet_ai_family_from_address().
+ */
+int v6only =
+((!saddr->has_ipv4 && !saddr->has_ipv6) ||
+ (saddr->has_ipv4 && saddr->ipv4 &&
+  saddr->has_ipv6 && saddr->ipv6)) ? 0 : 1;
 #endif
 inet_setport(e, p);
 #ifdef IPV6_V6ONLY

Re: [Qemu-devel] [PATCH] msi: remove return code for msi_init()

[Peter Xu]

On Thu, Jun 01, 2017 at 03:06:29PM -0700, Paul Burton wrote:
> Hi Aurelien/Paolo/Marcel,
> 
> On Thursday, 1 June 2017 12:22:06 PDT Aurelien Jarno wrote:
> > On 2017-06-01 16:23, Paolo Bonzini wrote:
> > > On 01/06/2017 10:27, Marcel Apfelbaum wrote:
> > > > On 31/05/2017 11:28, Paolo Bonzini wrote:
> > > >> No, for now I'd rather just go and remove msi_nonbroken.  When someone
> > > >> reports a bug, we can add back "msi_broken".
> > > > 
> > > > Hi,
> > > > I agree with the direction, but I am concerned msi_nonbroken is there
> > > > for a reason.
> > > > We might break some (obscure/not in use) machine.
> > > > Maybe we should CC all arch machine maintainers/contributors to give
> > > > them a chance to object...
> > > 
> > > Yeah, Alpha, MIPS and SH are those that support PCI.  Adding Richard and
> > > Aurelien, do your platforms support MSI on real hardware but not in QEMU?
> > 
> > SH clearly doesn't support MSI.
> > 
> > The oldest MIPS board also do not support MSI, but I guess the Boston
> > board might support it. I am adding Paul Burton in Cc: who probably
> > knows about that.
> > 
> > Aurelien
> 
> Indeed, real Boston hardware does support MSI (or rather, the Xilinx AXI 
> Bridge for PCI Express IP used on Boston does) & we make use of it in Linux.
> 
> Thanks,
> Paul

Does this mean that we'd better still keep the msi_nonbroken bit?

Anyway, maybe we can first merge Paolo's fix on edu device:

  [PATCH] edu: fix memory leak on msi_broken platforms

Then we can see whether we still need the rest of the changes.

Thanks,

-- 
Peter Xu

[Qemu-devel] [PATCH] gitignore: Ignore vim per-project config and clang complete

[Qu Wenruo]

Since we have already ignored ctags, ignoring .clangcomplete for
vim-clang_completion is quite reasonable for me.
(Not to mention vim-clang_completion provides better and more accurate
completion and definition search)

Also, qemu coding style is using 4 space as indent other than 8-space
tab, it also makes sense to ignore per-project vim config file.

Signed-off-by: Qu Wenruo 
---
 .gitignore | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.gitignore b/.gitignore
index 55a001e3b8..c7c6eec0b6 100644
--- a/.gitignore
+++ b/.gitignore
@@ -113,6 +113,8 @@
 cscope.*
 tags
 TAGS
+.vim*
+.clang_complete
 docker-src.*
 *~
 trace.h
-- 
2.13.0

Re: [Qemu-devel] [PATCHv5 0/4] Clean up compatibility mode handling

[no-reply]

Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 20170602031507.29881-1-da...@gibson.dropbear.id.au
Subject: [Qemu-devel] [PATCHv5 0/4] Clean up compatibility mode handling

=== TEST SCRIPT BEGIN ===
#!/bin/bash

BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0

git config --local diff.renamelimit 0
git config --local diff.renames True

commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
failed=1
echo
fi
n=$((n+1))
done

exit $failed
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
6ccf47f ppc: Rework CPU compatibility testing across migration
2b75df3 pseries: Reset CPU compatibility mode
282a89c pseries: Move CPU compatibility property to machine
8741a73 qapi: add explicit null to string input and output visitors

=== OUTPUT BEGIN ===
Checking PATCH 1/4: qapi: add explicit null to string input and output 
visitors...
Checking PATCH 2/4: pseries: Move CPU compatibility property to machine...
Checking PATCH 3/4: pseries: Reset CPU compatibility mode...
Checking PATCH 4/4: ppc: Rework CPU compatibility testing across migration...
ERROR: braces {} are necessary for all arms of this statement
#94: FILE: target/ppc/machine.c:236:
+if (cpu->compat_pvr) {
[...]
+} else
[...]

total: 1 errors, 0 warnings, 100 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

=== OUTPUT END ===

Test command exited with code: 1


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-de...@freelists.org

[Qemu-devel] [PATCHv5 3/4] pseries: Reset CPU compatibility mode

[David Gibson]

Currently, the CPU compatibility mode is set when the cpu is initialized,
then again when the guest negotiates features.  This means if a guest
negotiates a compatibility mode, then reboots, that compatibility mode
will be retained across the reset.

Usually that will get overridden when features are negotiated on the next
boot, but it's still not really correct.  This patch moves the initial set
up of the compatibility mode from cpu init to reset time.  The mode *is*
retained if the reboot was caused by the feature negotiation (it might
be important in that case, though it's unlikely).

Signed-off-by: David Gibson 
Reviewed-by: Alexey Kardashevskiy 
Reviewed-by: Michael Roth 
---
 hw/ppc/spapr.c  |  2 ++
 hw/ppc/spapr_cpu_core.c | 10 --
 2 files changed, 2 insertions(+), 10 deletions(-)

diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index 3c4e88f..2821b7e 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -1348,6 +1348,8 @@ static void ppc_spapr_reset(void)
 if (!spapr->cas_reboot) {
 spapr_ovec_cleanup(spapr->ov5_cas);
 spapr->ov5_cas = spapr_ovec_new();
+
+ppc_set_compat_all(spapr->max_compat_pvr, _abort);
 }
 
 fdt = spapr_build_fdt(spapr, rtas_addr, spapr->rtas_size);
diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c
index 846d9e7..7970871 100644
--- a/hw/ppc/spapr_cpu_core.c
+++ b/hw/ppc/spapr_cpu_core.c
@@ -121,16 +121,6 @@ static void spapr_cpu_init(sPAPRMachineState *spapr, 
PowerPCCPU *cpu,
 /* Enable PAPR mode in TCG or KVM */
 cpu_ppc_set_papr(cpu, PPC_VIRTUAL_HYPERVISOR(spapr));
 
-if (spapr->max_compat_pvr) {
-Error *local_err = NULL;
-
-ppc_set_compat(cpu, spapr->max_compat_pvr, _err);
-if (local_err) {
-error_propagate(errp, local_err);
-return;
-}
-}
-
 qemu_register_reset(spapr_cpu_reset, cpu);
 spapr_cpu_reset(cpu);
 }
-- 
2.9.4

[Qemu-devel] [PATCHv5 4/4] ppc: Rework CPU compatibility testing across migration

[David Gibson]

Migrating between different CPU versions is a bit complicated for ppc.
A long time ago, we ensured identical CPU versions at either end by
checking the PVR had the same value.  However, this breaks under KVM
HV, because we always have to use the host's PVR - it's not
virtualized.  That would mean we couldn't migrate between hosts with
different PVRs, even if the CPUs are close enough to compatible in
practice (sometimes identical cores with different surrounding logic
have different PVRs, so this happens in practice quite often).

So, we removed the PVR check, but instead checked that several flags
indicating supported instructions matched.  This turns out to be a bad
idea, because those instruction masks are not architected information, but
essentially a TCG implementation detail.  So changes to qemu internal CPU
modelling can break migration - this happened between qemu-2.6 and
qemu-2.7.  That was addressed by 146c11f1 "target-ppc: Allow eventual
removal of old migration mistakes".

Now, verification of CPU compatibility across a migration basically doesn't
happen.  We simply ignore the PVR of the incoming migration, and hope the
cpu on the destination is close enough to work.

Now that we've cleaned up handling of processor compatibility modes for
pseries machine type, we can do better.  We allow migration if:

* The source and destination PVRs are for the same type of CPU, as
  determined by CPU class's pvr_match function
OR  * When the source was in a compatibility mode, and the destination CPU
  supports the same compatibility mode

Signed-off-by: David Gibson 
Reviewed-by: Suraj Jitindar Singh 
---
 target/ppc/machine.c | 69 +---
 1 file changed, 66 insertions(+), 3 deletions(-)

diff --git a/target/ppc/machine.c b/target/ppc/machine.c
index 6cb3a48..a29aabe 100644
--- a/target/ppc/machine.c
+++ b/target/ppc/machine.c
@@ -8,6 +8,7 @@
 #include "helper_regs.h"
 #include "mmu-hash64.h"
 #include "migration/cpu.h"
+#include "qapi/error.h"
 
 static int cpu_load_old(QEMUFile *f, void *opaque, int version_id)
 {
@@ -195,6 +196,27 @@ static void cpu_pre_save(void *opaque)
 }
 }
 
+/*
+ * Determine if a given PVR is a "close enough" match to the CPU
+ * object.  For TCG and KVM PR it would probably be sufficient to
+ * require an exact PVR match.  However for KVM HV the user is
+ * restricted to a PVR exactly matching the host CPU.  The correct way
+ * to handle this is to put the guest into an architected
+ * compatibility mode.  However, to allow a more forgiving transition
+ * and migration from before this was widely done, we allow migration
+ * between sufficiently similar PVRs, as determined by the CPU class's
+ * pvr_match() hook.
+ */
+static bool pvr_match(PowerPCCPU *cpu, uint32_t pvr)
+{
+PowerPCCPUClass *pcc = POWERPC_CPU_GET_CLASS(cpu);
+
+if (pvr == pcc->pvr) {
+return true;
+}
+return pcc->pvr_match(pcc, pvr);
+}
+
 static int cpu_post_load(void *opaque, int version_id)
 {
 PowerPCCPU *cpu = opaque;
@@ -203,10 +225,31 @@ static int cpu_post_load(void *opaque, int version_id)
 target_ulong msr;
 
 /*
- * We always ignore the source PVR. The user or management
- * software has to take care of running QEMU in a compatible mode.
+ * If we're operating in compat mode, we should be ok as long as
+ * the destination supports the same compatiblity mode.
+ *
+ * Otherwise, however, we require that the destination has exactly
+ * the same CPU model as the source.
  */
-env->spr[SPR_PVR] = env->spr_cb[SPR_PVR].default_value;
+
+#if defined(TARGET_PPC64)
+if (cpu->compat_pvr) {
+Error *local_err = NULL;
+
+ppc_set_compat(cpu, cpu->compat_pvr, _err);
+if (local_err) {
+error_report_err(local_err);
+error_free(local_err);
+return -1;
+}
+} else
+#endif
+{
+if (!pvr_match(cpu, env->spr[SPR_PVR])) {
+return -1;
+}
+}
+
 env->lr = env->spr[SPR_LR];
 env->ctr = env->spr[SPR_CTR];
 cpu_write_xer(env, env->spr[SPR_XER]);
@@ -560,6 +603,25 @@ static const VMStateDescription vmstate_tlbmas = {
 }
 };
 
+static bool compat_needed(void *opaque)
+{
+PowerPCCPU *cpu = opaque;
+
+assert(!(cpu->compat_pvr && !cpu->vhyp));
+return (cpu->compat_pvr != 0);
+}
+
+static const VMStateDescription vmstate_compat = {
+.name = "cpu/compat",
+.version_id = 1,
+.minimum_version_id = 1,
+.needed = compat_needed,
+.fields = (VMStateField[]) {
+VMSTATE_UINT32(compat_pvr, PowerPCCPU),
+VMSTATE_END_OF_LIST()
+}
+};
+
 const VMStateDescription vmstate_ppc_cpu = {
 .name = "cpu",
 .version_id = 5,
@@ -613,6 +675,7 @@ const VMStateDescription vmstate_ppc_cpu = {
 _tlb6xx,
 _tlbemb,
 _tlbmas,
+_compat,
 NULL
 }
 };
-- 

[Qemu-devel] [PATCHv5 0/4] Clean up compatibility mode handling

[David Gibson]

This is a rebased and revised version of my patches revising CPU
compatiblity mode handling on ppc, last posted in November.  Since
then, many of the patches have already been merged (some for 2.9, some
since).  This is what's left.

 * There was conceptual confusion about what a compatibility mode
   means, and how it interacts with the machine type.  This cleans
   that up, clarifying that a compatibility mode (as an externally set
   option) only makes sense on machine types that don't permit the
   guest hypervisor privilege (i.e. 'pseries')

 * It was previously the user's (or management layer's) responsibility
   to determine compatibility of CPUs on either end for migration.
   This uses the compatibility modes to check that properly during an
   incoming migration.

This hasn't been extensively tested yet.  There are quite a few
migration cases to consider, for example:

Basic:

1) Boot guest with -cpu host
Should go into POWER8 compat mode after CAS
Previously would have been raw mode

2) Boot guest with -machine pseries,max-cpu-compat=power7 -cpu host
Should go into POWER7 compat mode

3) Boot guest with -cpu host,compat=power7
Should act as (2), but print a warning

4) Boot guest via libvirt with power7 compat mode specified in XML
Should act as (3), (2) once we fix libvirt

5) Hack guest to only advertise power7 compatibility, boot with -cpu host
Should go into POWER7 compat mode after CAS

6) Hack guest to only advertise real PVRs
Should remain in POWER8 raw mode after CAS

7) Hack guest to only advertise real PVRs
   Boot with -machine pseries,max-cpu-compat=power8
Should fail at CAS time

8) Hack guest to only advertise power7 compatibility, boot with -cpu host
   Reboot to normal guest
Should go to power7 compat mode after CAS of boot 1
Should revert to raw mode on reboot
SHould go to power8 compat mode after CAS of boot 2

Migration:

9) Boot guest with qemu-2.6 -machine pseries-2.6 -cpu host
   Migrate to qemu-2.8 -machine pseries-2.6 -cpu host
Should work, end up running in power8 raw mode

10) Boot guest with qemu-2.7 -machine pseries-2.7 -cpu host
Migrate to qemu-2.8 -machine pseries-2.7 -cpu host
Should work, end up running in power8 raw mode

11) Boot guest with qemu-2.7 -machine pseries-2.7 -cpu host,compat=power7
Migrate to qemu-2.8 -machine pseries-2.7 -cpu host,compat=power7
Should work, be running in POWER7 compat after, but give warning like
(3)

12) Boot guest with qemu-2.7 -machine pseries-2.7 -cpu host,compat=power7
Migrate to qemu-2.8 -machine pseries-2.7,max-cpu-compat=power7 -cpu host
Should work, be running in POWER7 compat after, no warning

13) Boot to SLOF with qemu-2.6 -machine pseries-2.6 -cpu host
Migrate to qemu-2.8 -machine pseries-2.6 -cpu host

?

14) Boot to SLOF with qemu-2.7 -machine pseries-2.7 -cpu host
Migrate to qemu-2.8 -machine pseries-2.7 -cpu host
?

15) Boot to SLOF with qemu-2.7 -machine pseries-2.7 -cpu host,compat=power7
Migrate to qemu-2.8 -machine pseries-2.7 -cpu host,compat=power7
?

16) Boot to SLOF with qemu-2.7 -machine pseries-2.7 -cpu host,compat=power7
Migrate to qemu-2.8 -machine pseries-2.7,max-cpu-compat=power7 -cpu host
?

17) Boot guest with qemu-2.6 -machine pseries-2.6 -cpu host
Migrate to qemu-2.7.z -machine pseries-2.6 -cpu host
Should work

18) Hack guest to only advertise power7 compatibility, boot with -cpu host
Boot with qemu-2.8, migrate to qemu-2.8
Should be in power7 compat mode after CAS on source, and still
in power7 compat mode on destination

Changes since v4:
  * Fixed a crash bug in the smp option compatiblity mangling
  * Removed an unnecessary fallback for missing pvr_match
  * Some spelling corrections
  * Migration core patch removed (alread merged to ppc-for-2.10)

Changes since v3:
  * Backwards compatible -cpu handling now removes compat= option from
options passed on to the cpu, so it doesn't trigger further warnings
  * Add a migration fix make cpu_synchronize_state() safe in post_load
handlers, which in turn fixes a bug in 5/5.
  * A number of bugfixes and other tweaks suggested by feedback on v2.

Changes since RFCv2:
  * Many patches dropped, since they're already merged
  * Rebased, fixed conflicts
  * Restored support for backwards migration (wasn't as complicated as
I thought)
  * Updated final patch's description to more accurately reflect the
logic

Changes since RFCv1:
  * Change CAS logic to prefer compatibility modes over raw mode
  * Simplified by giving up on half-hearted attempts to maintain
backwards migration
  * Folded migration stream changes into a single patch
  * Removed some preliminary patches which are already merged

David Gibson (3):
  pseries: Move CPU compatibility property to machine
  pseries: Reset CPU compatibility mode
  ppc: Rework CPU 

[Qemu-devel] [PATCHv5 1/4] qapi: add explicit null to string input and output visitors

[David Gibson]

From: Greg Kurz 

This may be used for deprecated object properties that are kept for
backwards compatibility.

Signed-off-by: Greg Kurz 
Reviewed-by: Markus Armbruster 
Signed-off-by: David Gibson 
---
 qapi/string-input-visitor.c  | 11 +++
 qapi/string-output-visitor.c | 14 ++
 2 files changed, 25 insertions(+)

diff --git a/qapi/string-input-visitor.c b/qapi/string-input-visitor.c
index c089491..63ae115 100644
--- a/qapi/string-input-visitor.c
+++ b/qapi/string-input-visitor.c
@@ -326,6 +326,16 @@ static void parse_type_number(Visitor *v, const char 
*name, double *obj,
 *obj = val;
 }
 
+static void parse_type_null(Visitor *v, const char *name, Error **errp)
+{
+StringInputVisitor *siv = to_siv(v);
+
+if (!siv->string || siv->string[0]) {
+error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
+   "null");
+}
+}
+
 static void string_input_free(Visitor *v)
 {
 StringInputVisitor *siv = to_siv(v);
@@ -349,6 +359,7 @@ Visitor *string_input_visitor_new(const char *str)
 v->visitor.type_bool = parse_type_bool;
 v->visitor.type_str = parse_type_str;
 v->visitor.type_number = parse_type_number;
+v->visitor.type_null = parse_type_null;
 v->visitor.start_list = start_list;
 v->visitor.next_list = next_list;
 v->visitor.check_list = check_list;
diff --git a/qapi/string-output-visitor.c b/qapi/string-output-visitor.c
index 53c2175..af649e1 100644
--- a/qapi/string-output-visitor.c
+++ b/qapi/string-output-visitor.c
@@ -256,6 +256,19 @@ static void print_type_number(Visitor *v, const char 
*name, double *obj,
 string_output_set(sov, g_strdup_printf("%f", *obj));
 }
 
+static void print_type_null(Visitor *v, const char *name, Error **errp)
+{
+StringOutputVisitor *sov = to_sov(v);
+char *out;
+
+if (sov->human) {
+out = g_strdup("");
+} else {
+out = g_strdup("");
+}
+string_output_set(sov, out);
+}
+
 static void
 start_list(Visitor *v, const char *name, GenericList **list, size_t size,
Error **errp)
@@ -341,6 +354,7 @@ Visitor *string_output_visitor_new(bool human, char 
**result)
 v->visitor.type_bool = print_type_bool;
 v->visitor.type_str = print_type_str;
 v->visitor.type_number = print_type_number;
+v->visitor.type_null = print_type_null;
 v->visitor.start_list = start_list;
 v->visitor.next_list = next_list;
 v->visitor.end_list = end_list;
-- 
2.9.4

[Qemu-devel] [PATCHv5 2/4] pseries: Move CPU compatibility property to machine

[David Gibson]

Server class POWER CPUs have a "compat" property, which is used to set the
backwards compatibility mode for the processor.  However, this only makes
sense for machine types which don't give the guest access to hypervisor
privilege - otherwise the compatibility level is under the guest's control.

To reflect this, this removes the CPU 'compat' property and instead
creates a 'max-cpu-compat' property on the pseries machine.  Strictly
speaking this breaks compatibility, but AFAIK the 'compat' option was
never (directly) used with -device or device_add.

The option was used with -cpu.  So, to maintain compatibility, this
patch adds a hack to the cpu option parsing to strip out any compat
options supplied with -cpu and set them on the machine property
instead of the now deprecated cpu property.

Signed-off-by: David Gibson 
---
 hw/ppc/spapr.c  |   6 ++-
 hw/ppc/spapr_cpu_core.c |  55 +++-
 hw/ppc/spapr_hcall.c|   8 ++--
 include/hw/ppc/spapr.h  |  12 --
 target/ppc/compat.c | 102 
 target/ppc/cpu.h|   5 ++-
 target/ppc/translate_init.c |  86 +++--
 7 files changed, 201 insertions(+), 73 deletions(-)

diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index ab3aab1..3c4e88f 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -2134,7 +2134,7 @@ static void ppc_spapr_init(MachineState *machine)
 machine->cpu_model = kvm_enabled() ? "host" : smc->tcg_default_cpu;
 }
 
-ppc_cpu_parse_features(machine->cpu_model);
+spapr_cpu_parse_features(spapr);
 
 spapr_init_cpus(spapr);
 
@@ -2497,6 +2497,10 @@ static void spapr_machine_initfn(Object *obj)
 " place of standard EPOW events when 
possible"
 " (required for memory hot-unplug 
support)",
 NULL);
+
+ppc_compat_add_property(obj, "max-cpu-compat", >max_compat_pvr,
+"Maximum permitted CPU compatibility mode",
+_fatal);
 }
 
 static void spapr_machine_finalizefn(Object *obj)
diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c
index ff7058e..846d9e7 100644
--- a/hw/ppc/spapr_cpu_core.c
+++ b/hw/ppc/spapr_cpu_core.c
@@ -20,6 +20,57 @@
 #include "sysemu/numa.h"
 #include "qemu/error-report.h"
 
+void spapr_cpu_parse_features(sPAPRMachineState *spapr)
+{
+/*
+ * Backwards compatibility hack:
+ *
+ *   CPUs had a "compat=" property which didn't make sense for
+ *   anything except pseries.  It was replaced by "max-cpu-compat"
+ *   machine option.  This supports old command lines like
+ *   -cpu POWER8,compat=power7
+ *   By stripping the compat option and applying it to the machine
+ *   before passing it on to the cpu level parser.
+ */
+gchar **inpieces;
+int i, j;
+gchar *compat_str = NULL;
+
+inpieces = g_strsplit(MACHINE(spapr)->cpu_model, ",", 0);
+
+/* inpieces[0] is the actual model string */
+i = 1;
+j = 1;
+while (inpieces[i]) {
+if (g_str_has_prefix(inpieces[i], "compat=")) {
+/* in case of multiple compat= options */
+g_free(compat_str);
+compat_str = inpieces[i];
+} else {
+j++;
+}
+
+i++;
+/* Excise compat options from list */
+inpieces[j] = inpieces[i];
+}
+
+if (compat_str) {
+char *val = compat_str + strlen("compat=");
+gchar *newprops = g_strjoinv(",", inpieces);
+
+object_property_set_str(OBJECT(spapr), val, "max-cpu-compat",
+_fatal);
+
+ppc_cpu_parse_features(newprops);
+g_free(newprops);
+} else {
+ppc_cpu_parse_features(MACHINE(spapr)->cpu_model);
+}
+
+g_strfreev(inpieces);
+}
+
 static void spapr_cpu_reset(void *opaque)
 {
 sPAPRMachineState *spapr = SPAPR_MACHINE(qdev_get_machine());
@@ -70,10 +121,10 @@ static void spapr_cpu_init(sPAPRMachineState *spapr, 
PowerPCCPU *cpu,
 /* Enable PAPR mode in TCG or KVM */
 cpu_ppc_set_papr(cpu, PPC_VIRTUAL_HYPERVISOR(spapr));
 
-if (cpu->max_compat) {
+if (spapr->max_compat_pvr) {
 Error *local_err = NULL;
 
-ppc_set_compat(cpu, cpu->max_compat, _err);
+ppc_set_compat(cpu, spapr->max_compat_pvr, _err);
 if (local_err) {
 error_propagate(errp, local_err);
 return;
diff --git a/hw/ppc/spapr_hcall.c b/hw/ppc/spapr_hcall.c
index aae5a62..a9bb3ed 100644
--- a/hw/ppc/spapr_hcall.c
+++ b/hw/ppc/spapr_hcall.c
@@ -1044,11 +1044,11 @@ static target_ulong h_signal_sys_reset(PowerPCCPU *cpu,
 }
 }
 
-static uint32_t cas_check_pvr(PowerPCCPU *cpu, target_ulong *addr,
-  Error **errp)
+static uint32_t cas_check_pvr(sPAPRMachineState *spapr, PowerPCCPU *cpu,
+  

Re: [Qemu-devel] [PATCHv4 0/5] Clean up compatibility mode handling

[David Gibson]

On Thu, Jun 01, 2017 at 01:59:14PM +0200, Cédric Le Goater wrote:
> On 06/01/2017 08:52 AM, David Gibson wrote:
> > On Wed, May 31, 2017 at 10:58:57AM +0200, Greg Kurz wrote:
> >> On Wed, 31 May 2017 12:57:48 +1000
> >> David Gibson  wrote:
> >>> [...]
>  All old non-pseries machine types already complain when started with
>  a POWER7 or newer CPU. Providing the extra error message looks weird:
> 
>  qemu-system-ppc64 -machine ppce500 \
>    -cpu POWER7,compat=power6
>  qemu-system-ppc64: CPU 'compat' property is deprecated and has no effect;
>   use max-cpu-compat machine property instead
>  MMU model 983043 not supported by this machine.
> 
>  but I guess it's better than crashing. :)  
> >>>
> >>> Well, sure POWER7 doesn't make sense for an e500 machine for other
> >>> reasons.  But POWER7 or POWER8 _would_ make sense for powernv, where
> >>> compat= doesn't.
> >>>
> >>
> >> The powernv machine type doesn't even support CPU features at all:
> >>
> >> chip_typename = g_strdup_printf(TYPE_PNV_CHIP "-%s", 
> >> machine->cpu_model);
> >> if (!object_class_by_name(chip_typename)) {
> >> error_report("invalid CPU model '%s' for %s machine",
> >>  machine->cpu_model, MACHINE_GET_CLASS(machine)->name);
> >> exit(1);
> >> }
> > 
> > Ah, well, that's another bug, but not one that's in scope for this
> > series.
> 
> PowerNV is still work in progress. I would not worry about it too much.

I wasn't intending to :).

-- 
David Gibson| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson


signature.asc
Description: PGP signature

Re: [Qemu-devel] [PATCHv4 5/5] ppc: Rework CPU compatibility testing across migration

[David Gibson]

On Thu, Jun 01, 2017 at 10:23:18AM +0200, Greg Kurz wrote:
> On Fri, 26 May 2017 15:23:19 +1000
> David Gibson  wrote:
> 
> > Migrating between different CPU versions is a bit complicated for ppc.
> > A long time ago, we ensured identical CPU versions at either end by
> > checking the PVR had the same value.  However, this breaks under KVM
> > HV, because we always have to use the host's PVR - it's not
> > virtualized.  That would mean we couldn't migrate between hosts with
> > different PVRs, even if the CPUs are close enough to compatible in
> > practice (sometimes identical cores with different surrounding logic
> > have different PVRs, so this happens in practice quite often).
> > 
> > So, we removed the PVR check, but instead checked that several flags
> > indicating supported instructions matched.  This turns out to be a bad
> > idea, because those instruction masks are not architected information, but
> > essentially a TCG implementation detail.  So changes to qemu internal CPU
> > modelling can break migration - this happened between qemu-2.6 and
> > qemu-2.7.  That was addressed by 146c11f1 "target-ppc: Allow eventual
> > removal of old migration mistakes".
> > 
> > Now, verification of CPU compatibility across a migration basically doesn't
> > happen.  We simply ignore the PVR of the incoming migration, and hope the
> > cpu on the destination is close enough to work.
> > 
> > Now that we've cleaned up handling of processor compatibility modes for
> > pseries machine type, we can do better.  We allow migration if:
> > 
> > * The source and destination PVRs are for the same type of CPU, as
> >   determined by CPU class's pvr_match function
> > OR  * When the source was in a compatibility mode, and the destination CPU
> >   supports the same compatibility mode
> > 
> > Signed-off-by: David Gibson 
> > ---
> >  target/ppc/machine.c | 72 
> > +---
> >  1 file changed, 69 insertions(+), 3 deletions(-)
> > 
> > diff --git a/target/ppc/machine.c b/target/ppc/machine.c
> > index 6cb3a48..2c6d9dc 100644
> > --- a/target/ppc/machine.c
> > +++ b/target/ppc/machine.c
> > @@ -8,6 +8,7 @@
> >  #include "helper_regs.h"
> >  #include "mmu-hash64.h"
> >  #include "migration/cpu.h"
> > +#include "qapi/error.h"
> >  
> >  static int cpu_load_old(QEMUFile *f, void *opaque, int version_id)
> >  {
> > @@ -195,6 +196,30 @@ static void cpu_pre_save(void *opaque)
> >  }
> >  }
> >  
> > +/*
> > + * Determine if a given PVR is a "close enough" match to the CPU
> > + * object.  For TCG and KVM PR it would probably be sufficient to
> > + * require an exact PVR match.  However for KVM HV the user is
> > + * restricted to a PVR exactly matching the host CPU.  The correct way
> > + * to handle this is to put the guest into an architected
> > + * compatibility mode.  However, to allow a more forgiving transition
> > + * and migration from before this was widely done, we allow migration
> > + * between sufficiently similar PVRs, as determined by the CPU class's
> > + * pvr_match() hook.
> > + */
> > +static bool pvr_match(PowerPCCPU *cpu, uint32_t pvr)
> > +{
> > +PowerPCCPUClass *pcc = POWERPC_CPU_GET_CLASS(cpu);
> > +
> > +if (pvr == pcc->pvr) {
> > +return true;
> > +}
> > +if (pcc->pvr_match) {
> > +return pcc->pvr_match(pcc, pvr);
> > +}
> > +return false;
> > +}
> 
> The base class provides a fallback for pcc->pvr_match that does:
> 
> static bool ppc_pvr_match_default(PowerPCCPUClass *pcc, uint32_t pvr)
> {
> return pcc->pvr == pvr;
> }
> 
> so I'm not sure this function is needed, but maybe I'm missing something.

Ah, yes, I think you're right.  I've simplified accordingly.

> > +
> >  static int cpu_post_load(void *opaque, int version_id)
> >  {
> >  PowerPCCPU *cpu = opaque;
> > @@ -203,10 +228,31 @@ static int cpu_post_load(void *opaque, int version_id)
> >  target_ulong msr;
> >  
> >  /*
> > - * We always ignore the source PVR. The user or management
> > - * software has to take care of running QEMU in a compatible mode.
> > + * If we're operating in compat mode, we should be ok as long as
> > + * the destination supports the same compatiblity mode.
> > + *
> > + * Otherwise, however, we require that the destination has exactly
> > + * the same CPU model as the source.
> >   */
> > -env->spr[SPR_PVR] = env->spr_cb[SPR_PVR].default_value;
> > +
> > +#if defined(TARGET_PPC64)
> > +if (cpu->compat_pvr) {
> > +Error *local_err = NULL;
> > +
> > +ppc_set_compat(cpu, cpu->compat_pvr, _err);
> > +if (local_err) {
> > +error_report_err(local_err);
> > +error_free(local_err);
> > +return -1;
> > +}
> > +} else
> > +#endif
> > +{
> > +if (!pvr_match(cpu, env->spr[SPR_PVR])) {
> > +return -1;
> > +}
> > +}

Re: [Qemu-devel] [PATCHv4 0/5] Clean up compatibility mode handling

[David Gibson]

On Thu, Jun 01, 2017 at 03:09:15PM +0200, Greg Kurz wrote:
> On Thu, 1 Jun 2017 13:59:14 +0200
> Cédric Le Goater  wrote:
> 
> > On 06/01/2017 08:52 AM, David Gibson wrote:
> > > On Wed, May 31, 2017 at 10:58:57AM +0200, Greg Kurz wrote:  
> > >> On Wed, 31 May 2017 12:57:48 +1000
> > >> David Gibson  wrote:  
> > >>> [...]  
> >  All old non-pseries machine types already complain when started with
> >  a POWER7 or newer CPU. Providing the extra error message looks weird:
> > 
> >  qemu-system-ppc64 -machine ppce500 \
> >    -cpu POWER7,compat=power6
> >  qemu-system-ppc64: CPU 'compat' property is deprecated and has no 
> >  effect;
> >   use max-cpu-compat machine property instead
> >  MMU model 983043 not supported by this machine.
> > 
> >  but I guess it's better than crashing. :)
> > >>>
> > >>> Well, sure POWER7 doesn't make sense for an e500 machine for other
> > >>> reasons.  But POWER7 or POWER8 _would_ make sense for powernv, where
> > >>> compat= doesn't.
> > >>>  
> > >>
> > >> The powernv machine type doesn't even support CPU features at all:
> > >>
> > >> chip_typename = g_strdup_printf(TYPE_PNV_CHIP "-%s", 
> > >> machine->cpu_model);
> > >> if (!object_class_by_name(chip_typename)) {
> > >> error_report("invalid CPU model '%s' for %s machine",
> > >>  machine->cpu_model, 
> > >> MACHINE_GET_CLASS(machine)->name);
> > >> exit(1);
> > >> }  
> > > 
> > > Ah, well, that's another bug, but not one that's in scope for this
> > > series.  
> > 
> > PowerNV is still work in progress. I would not worry about it too much.
> > 
> 
> Of course and this isn't the purpose of the discussion actually. We were
> talking about CPU features being relevant or not depending on the machine
> type.
> 
> But I'm not even sure that CPU features are useful at all for ppc, not to
> say very confusing (otherwise this series wouldn't be needed for example).
> 
> Speaking of PowerNV, just as an example, I guess the fix would be to
> forbid machine->cpu_model if it contains features. And probably the same
> for all other machine types, except pseries for backward compatibility
> reasons.

I don't think that's correct in principle.  I can imagine CPU
properties it might make sense to really set on the cpu, regardless of
machine type.  A quick look says we don't have any such at the moment,
but I don't think it's something we should prevent as a matter of policy.

-- 
David Gibson| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson


signature.asc
Description: PGP signature

[Qemu-devel] NVME: is there any plan to support SGL data transfer?

[Qu Wenruo]

Hi,

When going through NVMe specification and hw/block/nvme.c,
I found that it seems that NVMe qemu implementation only support PRP for 
sq entry.

And NvmeRwCmd doesn't even use union to define DPTR, but just prp1 and prp2.

Although I am just a newbie, but I'm quite interested in NVMe and want 
to try to implement SGL support for qemu NVMe.


Is there anyone already doing such work? Or is there any plan on 
implement such feature?


Thanks,
Qu

[Qemu-devel] Target AVR

[Anichang via Qemu-devel]

Hi all,

I just resurrected the target-avr patchset from Michael Rolnik. Following the 
details:

commit f2bca179dbfc3f378b131ed619d07db946bae598
Merge: 43771d5 ed250c0
Author: Ani Chang 
Date: Fri Jun 2 01:17:34 2017 +0200

target/avr: resurrected (see mailing list qemu-devel, Richard Henderson on Sep 
20, 2016 at 8:35pm)
and fixed (it builds).

Details:
- merge remote git://github.com/rth7680/qemu.git tags/pull-avr-20160920 into 
master
- fixed include/sysemu/arch_init.h (i.e.: bump QEMU_ARCH_AVR from 1<<17 to 
1<<18)
- fixed target/avr/cpu.c (i.e.: remove one function arg)
- fixed target/avr/machine.c (i.e.: fix a bunch of getters/setters signatures)

Running the sample board outputs:

$ ./qemu-system-avr
Unexpected error in object_property_add() at qom/object.c:940:
qemu-system-avr: attempt to add duplicate property 'memory' to object (type 
'avr5-avr')
Aborted (core dumped)
$

Signed-off-by: Ani Chang 

commit 43771d5d92312504305c19abe29ec5bfabd55f01
Merge: c077a99 c064477
Author: Peter Maydell 
Date: Thu Jun 1 16:39:16 2017 +0100

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2017-05-31' into 
staging
...
---

Following the output of 'make check'.

...
GTESTER check-qtest-avr
Unexpected error in object_property_add() at qom/object.c:940:
attempt to add duplicate property 'memory' to object (type 'xmega7-avr')
Broken pipe
GTester: last random seed: R02Sb7127f88337efa767b5e96a88046ebc1
Unexpected error in object_property_add() at qom/object.c:940:
qemu-system-avr: attempt to add duplicate property 'memory' to object (type 
'avr5-avr')
Broken pipe
GTester: last random seed: R02S94aa640298a8d5a71d11208b95363edd
Unexpected error in object_property_add() at qom/object.c:940:
qemu-system-avr: attempt to add duplicate property 'memory' to object (type 
'avr5-avr')
Broken pipe
GTester: last random seed: R02S76c62d67e22fbb237a3431358e65d6c2
/qemu-test/tests/Makefile.include:824: recipe for target 'check-qtest-avr' 
failed
make: *** [check-qtest-avr] Error 1
$
---

I have no idea what to do from here. How to solve the "attempt to add duplicate 
property 'memory' to object" error?

Regards

Re: [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap

[John Snow]

On 06/01/2017 03:30 AM, Sementsov-Ogievskiy Vladimir wrote:
> Hi John!
> 
> Look at our discussion about this in v18 thread.
> > Shortly: readonly is not the same as disabled. disabled= bitmap just
> ignores all writes. readonly= writes are not allowed at all.
> 
> And I think, I'll try to go through way 2: "dirty" field instead of
> "readonly" (look at v18 discussion), as it a bit more flexible.
> 

Not sure which I prefer...

Method 1 is attractive in that it is fairly simple, and enforces fairly
loudly the inability to write to devices with RO bitmaps. It's a natural
extension of your current approach.

Method 2 is attractive in that it seems a little more efficient, and is
a little more clever. A dirty flag lets us avoid flushing bitmaps we
never even changed (though we still need to clean up the in_use flags.)

What I wonder about #2 is what happens when a write sneaks in (due to a
bug or a use case we didn't see) on a bitmap attached to a read-only
node. We fail later on invalidate? It shouldn't happen in normal
circumstances, but I worry that the failure mode is messier.


Well, either way I will be happy for now I think -- pick whichever
option feels easiest or best for you to implement.

Thanks!

> 
> On 01.06.2017 02:48, John Snow wrote:
>>
>> On 05/30/2017 04:17 AM, Vladimir Sementsov-Ogievskiy wrote:
>>> It will be needed in following commits for persistent bitmaps.
>>> If bitmap is loaded from read-only storage (and we can't mark it
>>> "in use" in this storage) corresponding BdrvDirtyBitmap should be
>>> read-only.
>>>
>>> Signed-off-by: Vladimir Sementsov-Ogievskiy 
>>> ---
>>>   block/dirty-bitmap.c | 28 
>>>   block/io.c   |  8 
>>>   blockdev.c   |  6 ++
>>>   include/block/dirty-bitmap.h |  4 
>>>   4 files changed, 46 insertions(+)
>>>
>>> diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
>>> index 90af37287f..733f19ca5e 100644
>>> --- a/block/dirty-bitmap.c
>>> +++ b/block/dirty-bitmap.c
>>> @@ -44,6 +44,8 @@ struct BdrvDirtyBitmap {
>>>   int64_t size;   /* Size of the bitmap (Number of
>>> sectors) */
>>>   bool disabled;  /* Bitmap is read-only */
>>>   int active_iterators;   /* How many iterators are active */
>>> +bool readonly;  /* Bitmap is read-only and may be
>>> changed only
>>> +   by deserialize* functions */
>>>   QLIST_ENTRY(BdrvDirtyBitmap) list;
>>>   };
>>>   @@ -436,6 +438,7 @@ void bdrv_set_dirty_bitmap(BdrvDirtyBitmap
>>> *bitmap,
>>>  int64_t cur_sector, int64_t nr_sectors)
>>>   {
>>>   assert(bdrv_dirty_bitmap_enabled(bitmap));
>>> +assert(!bdrv_dirty_bitmap_readonly(bitmap));
>> Not reasonable to add the condition for !readonly into
>> bdrv_dirty_bitmap_enabled?
>>
>> As is:
>>
>> If readonly is set to true on a bitmap, bdrv_dirty_bitmap_status is
>> going to return ACTIVE for such bitmaps, but DISABLED might be more
>> appropriate to indicate the read-only nature.
>>
>> If you add this condition into _enabled(), you can skip the extra
>> assertions you've added here.
>>
>>>   hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors);
>>>   }
>>>   @@ -443,12 +446,14 @@ void bdrv_reset_dirty_bitmap(BdrvDirtyBitmap
>>> *bitmap,
>>>int64_t cur_sector, int64_t nr_sectors)
>>>   {
>>>   assert(bdrv_dirty_bitmap_enabled(bitmap));
>>> +assert(!bdrv_dirty_bitmap_readonly(bitmap));
>>>   hbitmap_reset(bitmap->bitmap, cur_sector, nr_sectors);
>>>   }
>>> void bdrv_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap, HBitmap **out)
>>>   {
>>>   assert(bdrv_dirty_bitmap_enabled(bitmap));
>>> +assert(!bdrv_dirty_bitmap_readonly(bitmap));
>>>   if (!out) {
>>>   hbitmap_reset_all(bitmap->bitmap);
>>>   } else {
>>> @@ -519,6 +524,7 @@ void bdrv_set_dirty(BlockDriverState *bs, int64_t
>>> cur_sector,
>>>   if (!bdrv_dirty_bitmap_enabled(bitmap)) {
>>>   continue;
>>>   }
>>> +assert(!bdrv_dirty_bitmap_readonly(bitmap));
>>>   hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors);
>>>   }
>>>   }
>>> @@ -540,3 +546,25 @@ int64_t
>>> bdrv_get_meta_dirty_count(BdrvDirtyBitmap *bitmap)
>>>   {
>>>   return hbitmap_count(bitmap->meta);
>>>   }
>>> +
>>> +bool bdrv_dirty_bitmap_readonly(const BdrvDirtyBitmap *bitmap)
>>> +{
>>> +return bitmap->readonly;
>>> +}
>>> +
>>> +void bdrv_dirty_bitmap_set_readonly(BdrvDirtyBitmap *bitmap)
>>> +{
>>> +bitmap->readonly = true;
>>> +}
>>> +
>>> +bool bdrv_has_readonly_bitmaps(BlockDriverState *bs)
>>> +{
>>> +BdrvDirtyBitmap *bm;
>>> +QLIST_FOREACH(bm, >dirty_bitmaps, list) {
>>> +if (bm->readonly) {
>>> +return true;
>>> +}
>>> +}
>>> +
>>> +return false;
>>> +}
>>> diff --git a/block/io.c b/block/io.c
>>> index fdd7485c22..0e28a1f595 

Re: [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap

[John Snow]

On 05/31/2017 11:53 AM, Max Reitz wrote:
> On 2017-05-31 17:05, Vladimir Sementsov-Ogievskiy wrote:
>> 31.05.2017 17:44, Max Reitz wrote:
>>> On 2017-05-31 16:29, Vladimir Sementsov-Ogievskiy wrote:
 31.05.2017 16:43, Max Reitz wrote:
> On 2017-05-30 08:50, Vladimir Sementsov-Ogievskiy wrote:
>> Thank you for this scenario. Hmm.
>>
>> So, as I need guarantee that image and bitmap are unchanged,
>> bdrv_set_dirty should return error and fail the whole write. Ok?
> I don't know. That would mean that you couldn't commit to an image that
> has a persistent auto-loading bitmap, which doesn't seem very nice
> to me.
>
> I'm not quite sure what to do myself. So first I'd definitely want the
> commit operation to succeed. That means we'd have to automatically make
> the bitmap non-readonly once we write to it. The "readonly" flag would
> then be an "unchanged" flag, rather, to signify that the bitmap has not
> been changed since it was loaded, which means that it does not need to
> be written back to the image file.
>
> Now the issue remains that if you modify a persistent bitmap that is
> stored in an image file that is opened RO when it's closed, you
> won't be
> able to write the modifications back.
>
> So in addition, I guess we'd need to "flush" all persistent bitmaps
> (that is, write all modifications back to the file and set the
> "unchanged" flag (you could also call it "dirty" and then mean the
> opposite) for each bitmap) not only when the image is closed or
> invalidated, but also when it is reopened read-only.
>
> (block-commit reopens the backing BDS R/W, then writes to them, thus
> modifying the dirty bitmaps, and finally reopens the BDS as read-only;
> before that happens, we will have to flush the modified bitmap data.)
 Ok, understand.

 We need to consider also setting in_use flag in the image. We _must not_
 write to image with dirty bitmap,
 if in_use flag of this dirty bitmap is not set, as in case of something
 fail we will have image with wrong bitmap with
 unset in_use flag (which looks ok).
>>> Right.
>>>
 I see two ways to handle it:

 variant 1:
 1. readonly field stays as is (see v19, with normal errors, not only
 asserts)
 2. immediately after reopening r/w we do "reopening bitmaps r/w", i.e.
 set in_use in the image and set BdrvDirtyBitmap.readonly = false
 3. in reopen_prepare, if reopening r-o do "reopening bitmaps r-o", i.e.
 save them into the image and set BdrvDirtyBitmap.readonly = true
>>> Sounds good, yes.
>>>
 variant 2:
 1. instead of 'readonly' add 'dirty' field, set dirty to 0 for all
 bitmaps on create
 2. before write/discard check this field in all related bitmaps, and if
 dirty=0 (and persistent=1), write IN_USE flag into the image first, set
 dirty=1, and only then do write. (if writing IN_USE=1 failed, fail the
 whole write)
 3. in reopen_prepare, if reopening r-o do "reopening bitmaps r-o", i.e.
 save them into the image and set BdrvDirtyBitmap.dirty = 0
>>> Works, too.
>>>
>>> I think the second variant would the more "efficient" way (because you
>>> only have to flush out dirty dirty bitmaps), but the first one would be
>>> simpler and has the great advantage of not requiring a write to the
>>> image file when you just want to set a bit in the in-memory dirty
>>> bitmap. So I'd personally go for the first variant.
>>
>> Hmm, why not requiring? Both 1 and 2 do write in_use=1, but (1) do this
>> on open/reopen, and (2) before the first write to the image.
> 
> Oh, I didn't read the "before write/discard". Yes, if you check it
> before writing, then you won't have to set the flag through
> bdrv_set_dirty().
> 
>> "set a bit in the in-memory..." - are you saying about not-persistent
>> dirty bitmaps? In this case, of course, nothing should be written into
>> the image, just set dirty=1.
> 
> No, I did mean persistent bitmaps, but bdrv_set_dirty() just sets the
> bit in main memory, of course. It only gets written to the image later
> (on reopen/close/invalidate).
> 

There may be some benefit to setting in_use immediately as soon as we
admit that we are willing to tolerate writes to the bitmap. It's a
performance hit, but it may help on-disk consistency.

Or maybe that's a fool's errand? This is a design question we've largely
ignored so far, but it's something that will need investigating sooner
or later.

> Well, your choice. I think both will work. :-)
> 
> Max
> 

Re: [Qemu-devel] [PATCH 14/19] nbd/server: nbd_negotiate: return 1 on NBD_OPT_ABORT

[Eric Blake]

On 05/30/2017 09:30 AM, Vladimir Sementsov-Ogievskiy wrote:
> Separate case when client sent NBD_OPT_ABORT from other errors.
> It will be needed for the following patch, where errors will be
> reported.
> Considered case is not actually the error - it honestly follows NBD
> protocol. Therefore it should not be reported like an error.
> -EPIPE case means client not read server reply on NBD_OPT_ABORT,
> which is also OK.
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy 
> ---
>  nbd/server.c | 20 +++-
>  1 file changed, 15 insertions(+), 5 deletions(-)
> 
> diff --git a/nbd/server.c b/nbd/server.c
> index 30dfb81a5c..0e53d3dd91 100644
> --- a/nbd/server.c
> +++ b/nbd/server.c
> @@ -369,9 +369,13 @@ static QIOChannel 
> *nbd_negotiate_handle_starttls(NBDClient *client,
>  return QIO_CHANNEL(tioc);
>  }
>  
> -
> -/* Process all NBD_OPT_* client option commands.
> - * Return -errno on error, 0 on success. */
> +/* nbd_negotiate_options
> + * Process all NBD_OPT_* client option commands.
> + * Return:
> + * < 0 on error

Do you want to be specific that this is a negative errno value, or is it
just any negative value with no correlation to errno?

> + * 0   on successful negotiation
> + * 1   if client sent NBD_OPT_ABORT, i.e. on legal disconnect
> + */
>  static int nbd_negotiate_options(NBDClient *client)
>  {
>  int ret;
> @@ -483,7 +487,7 @@ static int nbd_negotiate_options(NBDClient *client)
>  }
>  /* Let the client keep trying, unless they asked to quit */
>  if (clientflags == NBD_OPT_ABORT) {
> -return -EINVAL;
> +return 1;
>  }
>  break;
>  }
> @@ -502,7 +506,7 @@ static int nbd_negotiate_options(NBDClient *client)
>   * guests that don't wait for our reply. */
>  ret = nbd_negotiate_send_rep(client->ioc, NBD_REP_ACK,
>   clientflags);
> -return ret < 0 ? ret : -EINVAL;
> +return ret < 0 && ret != -EPIPE ? ret : 1;

This should just be 'return 1;', which means you don't need to capture
and check 'ret'.

>  
>  case NBD_OPT_EXPORT_NAME:
>  return nbd_negotiate_handle_export_name(client, length);
> @@ -560,6 +564,12 @@ static int nbd_negotiate_options(NBDClient *client)
>  }
>  }
>  
> +/* nbd_negotiate
> + * Return:
> + * < 0 on error

Again, if this is reliably a negative errno, specifically document that.

> + * 0   on successful negotiation
> + * 1   if client sent NBD_OPT_ABORT, i.e. on legal disconnect
> + */
>  static coroutine_fn int nbd_negotiate(NBDClient *client)
>  {
>  char buf[8 + 8 + 8 + 128];
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap

[John Snow]

On 05/31/2017 09:43 AM, Max Reitz wrote:
> On 2017-05-30 08:50, Vladimir Sementsov-Ogievskiy wrote:
>> Thank you for this scenario. Hmm.
>>
>> So, as I need guarantee that image and bitmap are unchanged,
>> bdrv_set_dirty should return error and fail the whole write. Ok?
> 
> I don't know. That would mean that you couldn't commit to an image that
> has a persistent auto-loading bitmap, which doesn't seem very nice to me.
> 
> I'm not quite sure what to do myself. So first I'd definitely want the
> commit operation to succeed. That means we'd have to automatically make
> the bitmap non-readonly once we write to it. The "readonly" flag would
> then be an "unchanged" flag, rather, to signify that the bitmap has not
> been changed since it was loaded, which means that it does not need to
> be written back to the image file.
> 
> Now the issue remains that if you modify a persistent bitmap that is
> stored in an image file that is opened RO when it's closed, you won't be
> able to write the modifications back.
> > So in addition, I guess we'd need to "flush" all persistent bitmaps
> (that is, write all modifications back to the file and set the
> "unchanged" flag (you could also call it "dirty" and then mean the
> opposite) for each bitmap) not only when the image is closed or
> invalidated, but also when it is reopened read-only.
>

Makes sense.

> (block-commit reopens the backing BDS R/W, then writes to them, thus
> modifying the dirty bitmaps, and finally reopens the BDS as read-only;
> before that happens, we will have to flush the modified bitmap data.)
> 

OK, so it would perhaps be enough to toggle the RO flag on/off when
nodes get reopened. When they get reopened RO, we'd need to flush at
that point.

(Right?)

Of course, a changed flag makes this a little moot as it is probably
more flexible; but there is something slightly attractive about the more
rigid form.

(Hmm, for the purposes of periodic flushing, we may want a changed flag
anyway...)

> Max
> 
Thanks for the scenario and the explainer.

Re: [Qemu-devel] [PATCH 13/19] nbd/server: return original error codes

[Eric Blake]

On 05/30/2017 09:30 AM, Vladimir Sementsov-Ogievskiy wrote:
> The code in many cases return -EINVAL or -EIO instead of original error
> code from, for example, write_sync(). Following patch will need EPIPE
> handling, so, let's refactor this where possible (the only exclusion
> is nbd_co_receive_request, with own return-code convention)

Do we still want/need EPIPE handling, given the discussion on the
previous two patches?

> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy 
> ---
>  nbd/server.c | 124 
> +--
>  1 file changed, 77 insertions(+), 47 deletions(-)
> 

Feels weird to have a net gain in code, but maybe worthwhile.

> diff --git a/nbd/server.c b/nbd/server.c
> index a47f13e4fb..30dfb81a5c 100644
> --- a/nbd/server.c
> +++ b/nbd/server.c
> @@ -136,30 +136,38 @@ static void nbd_client_receive_next_request(NBDClient 
> *client);
>  static int nbd_negotiate_send_rep_len(QIOChannel *ioc, uint32_t type,
>uint32_t opt, uint32_t len)
>  {
> +int ret;
>  uint64_t magic;
>  
>  TRACE("Reply opt=%" PRIx32 " type=%" PRIx32 " len=%" PRIu32,
>type, opt, len);
>  
>  magic = cpu_to_be64(NBD_REP_MAGIC);
> -if (write_sync(ioc, , sizeof(magic), NULL) < 0) {
> +ret = write_sync(ioc, , sizeof(magic), NULL);
> +if (ret < 0) {
>  LOG("write failed (rep magic)");
> -return -EINVAL;
> +return ret;
>  }

Constructs like this should get shorter once we plumb errp all the way
through.  Okay, I can live with the temporary verbosity.

You may still have to make changes due to rebasing (in which case I'll
definitely want to review again); but if this patch doesn't need further
rework, you can add:
Reviewed-by: Eric Blake 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org



signature.asc
Description: OpenPGP digital signature

[Qemu-devel] [PATCH 5/5] migration: Make compression_threads use save/load_setup/cleanup()

[Juan Quintela]

Once there, be consistent and use
compress_thread_{save,load}_{setup,cleanup}.

Signed-off-by: Juan Quintela 
---
 migration/migration.c |  5 -
 migration/ram.c   | 12 
 migration/ram.h   |  5 -
 3 files changed, 8 insertions(+), 14 deletions(-)

diff --git a/migration/migration.c b/migration/migration.c
index b75aebc..4040b33 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -284,7 +284,6 @@ static void process_incoming_migration_bh(void *opaque)
 } else {
 runstate_set(global_state_get_runstate());
 }
-migrate_decompress_threads_join();
 /*
  * This must happen after any state changes since as soon as an external
  * observer sees this event they might start to prod at the VM assuming
@@ -347,7 +346,6 @@ static void process_incoming_migration_co(void *opaque)
 migrate_set_state(>state, MIGRATION_STATUS_ACTIVE,
   MIGRATION_STATUS_FAILED);
 error_report("load of migration failed: %s", strerror(-ret));
-migrate_decompress_threads_join();
 exit(EXIT_FAILURE);
 }
 mis->bh = qemu_bh_new(process_incoming_migration_bh, mis);
@@ -358,7 +356,6 @@ void migration_fd_process_incoming(QEMUFile *f)
 {
 Coroutine *co = qemu_coroutine_create(process_incoming_migration_co, f);
 
-migrate_decompress_threads_create();
 qemu_file_set_blocking(f, false);
 qemu_coroutine_enter(co);
 }
@@ -823,7 +820,6 @@ static void migrate_fd_cleanup(void *opaque)
 }
 qemu_mutex_lock_iothread();
 
-migrate_compress_threads_join();
 qemu_fclose(s->to_dst_file);
 s->to_dst_file = NULL;
 }
@@ -1974,7 +1970,6 @@ void migrate_fd_connect(MigrationState *s)
 }
 }
 
-migrate_compress_threads_create();
 qemu_thread_create(>thread, "live_migration", migration_thread, s,
QEMU_THREAD_JOINABLE);
 s->migration_thread_running = true;
diff --git a/migration/ram.c b/migration/ram.c
index 3349116..e9d97d9 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -299,7 +299,7 @@ static inline void terminate_compression_threads(void)
 }
 }
 
-void migrate_compress_threads_join(void)
+static void compress_threads_save_cleanup(void)
 {
 int i, thread_count;
 
@@ -322,7 +322,7 @@ void migrate_compress_threads_join(void)
 comp_param = NULL;
 }
 
-void migrate_compress_threads_create(void)
+static void compress_threads_save_setup(void)
 {
 int i, thread_count;
 
@@ -1383,6 +1383,7 @@ static void ram_save_cleanup(void *opaque)
 }
 XBZRLE_cache_unlock();
 migration_page_queue_free(*rsp);
+compress_threads_save_cleanup();
 g_free(*rsp);
 *rsp = NULL;
 }
@@ -1915,6 +1916,7 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
 }
 
 rcu_read_unlock();
+compress_threads_save_setup();
 
 ram_control_before_iterate(f, RAM_CONTROL_SETUP);
 ram_control_after_iterate(f, RAM_CONTROL_SETUP);
@@ -2220,7 +,7 @@ static void wait_for_decompress_done(void)
 qemu_mutex_unlock(_done_lock);
 }
 
-void migrate_decompress_threads_create(void)
+static void compress_threads_load_setup(void)
 {
 int i, thread_count;
 
@@ -2241,7 +2243,7 @@ void migrate_decompress_threads_create(void)
 }
 }
 
-void migrate_decompress_threads_join(void)
+static void compress_threads_load_cleanup(void)
 {
 int i, thread_count;
 
@@ -2304,12 +2306,14 @@ static void decompress_data_with_multi_threads(QEMUFile 
*f,
 static int ram_load_setup(QEMUFile *f, void *opaque)
 {
 xbzrle_load_setup();
+compress_threads_load_setup();
 return 0;
 }
 
 static int ram_load_cleanup(void *opaque)
 {
 xbzrle_load_cleanup();
+compress_threads_load_cleanup();
 return 0;
 }
 
diff --git a/migration/ram.h b/migration/ram.h
index e49dd3f..bfb7b04 100644
--- a/migration/ram.h
+++ b/migration/ram.h
@@ -38,11 +38,6 @@ extern XBZRLECacheStats xbzrle_counters;
 int64_t xbzrle_cache_resize(int64_t new_size);
 uint64_t ram_bytes_total(void);
 
-void migrate_compress_threads_create(void);
-void migrate_compress_threads_join(void);
-void migrate_decompress_threads_create(void);
-void migrate_decompress_threads_join(void);
-
 uint64_t ram_pagesize_summary(void);
 int ram_save_queue_pages(const char *rbname, ram_addr_t start, ram_addr_t len);
 void acct_update_position(QEMUFile *f, size_t size, bool zero);
-- 
2.9.4

[Qemu-devel] [PATCH 3/5] migration: Create load_setup()/cleanup() methods

[Juan Quintela]

We need to do things at load time.

Signed-off-by: Juan Quintela 
---
 include/migration/register.h |  2 ++
 migration/savevm.c   | 31 +++
 2 files changed, 33 insertions(+)

diff --git a/include/migration/register.h b/include/migration/register.h
index 8e37185..0862bb8 100644
--- a/include/migration/register.h
+++ b/include/migration/register.h
@@ -42,6 +42,8 @@ typedef struct SaveVMHandlers {
   uint64_t *non_postcopiable_pending,
   uint64_t *postcopiable_pending);
 LoadStateHandler *load_state;
+int (*load_setup)(QEMUFile *f, void *opaque);
+int (*load_cleanup)(void *opaque);
 } SaveVMHandlers;
 
 int register_savevm(DeviceState *dev,
diff --git a/migration/savevm.c b/migration/savevm.c
index 0ec2cb0..1e5dfad 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -1928,6 +1928,31 @@ qemu_loadvm_section_part_end(QEMUFile *f, 
MigrationIncomingState *mis)
 return 0;
 }
 
+static int qemu_savevm_load_state_begin(QEMUFile *f)
+{
+SaveStateEntry *se;
+int ret;
+
+trace_savevm_state_begin();
+QTAILQ_FOREACH(se, _state.handlers, entry) {
+if (!se->ops || !se->ops->load_setup) {
+continue;
+}
+if (se->ops && se->ops->is_active) {
+if (!se->ops->is_active(se->opaque)) {
+continue;
+}
+}
+
+ret = se->ops->load_setup(f, se->opaque);
+if (ret < 0) {
+qemu_file_set_error(f, ret);
+return ret;
+}
+}
+return 0;
+}
+
 static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis)
 {
 uint8_t section_type;
@@ -2000,6 +2025,12 @@ int qemu_loadvm_state(QEMUFile *f)
 return -ENOTSUP;
 }
 
+
+if (qemu_savevm_load_state_begin(f) != 0) {
+error_report("Load state of one device failed");
+return -EINVAL;
+}
+
 if (!savevm_state.skip_configuration || enforce_config_section()) {
 if (qemu_get_byte(f) != QEMU_VM_CONFIGURATION) {
 error_report("Configuration section missing");
-- 
2.9.4

[Qemu-devel] [PATCH 4/5] migration: Convert ram to use new load_setup()/load_cleanup()

[Juan Quintela]

Once there, I rename ram_migration_cleanup() to ram_save_cleanup().
Notice that this is the first pass, and I only passed XBZRLE to the
new scheme.  Moved decoded_buf to inside XBZRLE struct.
As a bonus, I don't have to export xbzrle functions from ram.c.

Signed-off-by: Juan Quintela 
---
 migration/migration.c |  3 ---
 migration/ram.c   | 52 +++
 migration/ram.h   |  1 -
 3 files changed, 36 insertions(+), 20 deletions(-)

diff --git a/migration/migration.c b/migration/migration.c
index 331cab7..b75aebc 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -350,9 +350,6 @@ static void process_incoming_migration_co(void *opaque)
 migrate_decompress_threads_join();
 exit(EXIT_FAILURE);
 }
-
-free_xbzrle_decoded_buf();
-
 mis->bh = qemu_bh_new(process_incoming_migration_bh, mis);
 qemu_bh_schedule(mis->bh);
 }
diff --git a/migration/ram.c b/migration/ram.c
index d3d2ef1..3349116 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -85,11 +85,10 @@ static struct {
 QemuMutex lock;
 /* it will store a page full of zeros */
 uint8_t *zero_target_page;
+/* buffer used for XBZRLE decoding */
+uint8_t *decoded_buf;
 } XBZRLE;
 
-/* buffer used for XBZRLE decoding */
-static uint8_t *xbzrle_decoded_buf;
-
 static void XBZRLE_cache_lock(void)
 {
 if (migrate_use_xbzrle())
@@ -1343,13 +1342,18 @@ uint64_t ram_bytes_total(void)
 return total;
 }
 
-void free_xbzrle_decoded_buf(void)
+static void xbzrle_load_setup(void)
 {
-g_free(xbzrle_decoded_buf);
-xbzrle_decoded_buf = NULL;
+XBZRLE.decoded_buf = g_malloc(TARGET_PAGE_SIZE);
 }
 
-static void ram_migration_cleanup(void *opaque)
+static void xbzrle_load_cleanup(void)
+{
+g_free(XBZRLE.decoded_buf);
+XBZRLE.decoded_buf = NULL;
+}
+
+static void ram_save_cleanup(void *opaque)
 {
 RAMState **rsp = opaque;
 RAMBlock *block;
@@ -2068,12 +2072,6 @@ static int load_xbzrle(QEMUFile *f, ram_addr_t addr, 
void *host)
 {
 unsigned int xh_len;
 int xh_flags;
-uint8_t *loaded_data;
-
-if (!xbzrle_decoded_buf) {
-xbzrle_decoded_buf = g_malloc(TARGET_PAGE_SIZE);
-}
-loaded_data = xbzrle_decoded_buf;
 
 /* extract RLE header */
 xh_flags = qemu_get_byte(f);
@@ -2089,10 +2087,10 @@ static int load_xbzrle(QEMUFile *f, ram_addr_t addr, 
void *host)
 return -1;
 }
 /* load data and decode */
-qemu_get_buffer_in_place(f, _data, xh_len);
+qemu_get_buffer_in_place(f, _buf, xh_len);
 
 /* decode RLE */
-if (xbzrle_decode_buffer(loaded_data, xh_len, host,
+if (xbzrle_decode_buffer(XBZRLE.decoded_buf, xh_len, host,
  TARGET_PAGE_SIZE) == -1) {
 error_report("Failed to load XBZRLE page - decode error!");
 return -1;
@@ -2296,6 +2294,26 @@ static void decompress_data_with_multi_threads(QEMUFile 
*f,
 }
 
 /**
+ * ram_load_setup: Setup RAM for migration incoming side
+ *
+ * Returns zero to indicate success and negative for error
+ *
+ * @f: QEMUFile where to receive the data
+ * @opaque: RAMState pointer
+ */
+static int ram_load_setup(QEMUFile *f, void *opaque)
+{
+xbzrle_load_setup();
+return 0;
+}
+
+static int ram_load_cleanup(void *opaque)
+{
+xbzrle_load_cleanup();
+return 0;
+}
+
+/**
  * ram_postcopy_incoming_init: allocate postcopy data structures
  *
  * Returns 0 for success and negative if there was one error
@@ -2603,7 +2621,9 @@ static SaveVMHandlers savevm_ram_handlers = {
 .save_live_complete_precopy = ram_save_complete,
 .save_live_pending = ram_save_pending,
 .load_state = ram_load,
-.save_cleanup = ram_migration_cleanup,
+.save_cleanup = ram_save_cleanup,
+.load_setup = ram_load_setup,
+.load_cleanup = ram_load_cleanup,
 };
 
 void ram_mig_init(void)
diff --git a/migration/ram.h b/migration/ram.h
index 9eadc8c..e49dd3f 100644
--- a/migration/ram.h
+++ b/migration/ram.h
@@ -46,7 +46,6 @@ void migrate_decompress_threads_join(void);
 uint64_t ram_pagesize_summary(void);
 int ram_save_queue_pages(const char *rbname, ram_addr_t start, ram_addr_t len);
 void acct_update_position(QEMUFile *f, size_t size, bool zero);
-void free_xbzrle_decoded_buf(void);
 void ram_debug_dump_bitmap(unsigned long *todump, bool expected,
unsigned long pages);
 void ram_postcopy_migrated_memory_release(MigrationState *ms);
-- 
2.9.4

[Qemu-devel] [PATCH 2/5] migration: Rename cleanup() to save_cleanup()

[Juan Quintela]

We need a cleanup for loads.

Signed-off-by: Juan Quintela 
---
 hw/ppc/spapr.c   | 2 +-
 include/migration/register.h | 2 +-
 migration/block.c| 2 +-
 migration/ram.c  | 2 +-
 migration/savevm.c   | 4 ++--
 5 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index d51a361..0c11613 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -1876,7 +1876,7 @@ static SaveVMHandlers savevm_htab_handlers = {
 .save_setup = htab_save_setup,
 .save_live_iterate = htab_save_iterate,
 .save_live_complete_precopy = htab_save_complete,
-.cleanup = htab_cleanup,
+.save_cleanup = htab_cleanup,
 .load_state = htab_load,
 };
 
diff --git a/include/migration/register.h b/include/migration/register.h
index 6cb27f4..8e37185 100644
--- a/include/migration/register.h
+++ b/include/migration/register.h
@@ -21,7 +21,7 @@ typedef struct SaveVMHandlers {
 /* This runs inside the iothread lock.  */
 SaveStateHandler *save_state;
 
-void (*cleanup)(void *opaque);
+void (*save_cleanup)(void *opaque);
 int (*save_live_complete_postcopy)(QEMUFile *f, void *opaque);
 int (*save_live_complete_precopy)(QEMUFile *f, void *opaque);
 
diff --git a/migration/block.c b/migration/block.c
index 976c42e..efa0805 100644
--- a/migration/block.c
+++ b/migration/block.c
@@ -1000,7 +1000,7 @@ static SaveVMHandlers savevm_block_handlers = {
 .save_live_complete_precopy = block_save_complete,
 .save_live_pending = block_save_pending,
 .load_state = block_load,
-.cleanup = block_migration_cleanup,
+.save_cleanup = block_migration_cleanup,
 .is_active = block_is_active,
 };
 
diff --git a/migration/ram.c b/migration/ram.c
index f2c13e2..d3d2ef1 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -2603,7 +2603,7 @@ static SaveVMHandlers savevm_ram_handlers = {
 .save_live_complete_precopy = ram_save_complete,
 .save_live_pending = ram_save_pending,
 .load_state = ram_load,
-.cleanup = ram_migration_cleanup,
+.save_cleanup = ram_migration_cleanup,
 };
 
 void ram_mig_init(void)
diff --git a/migration/savevm.c b/migration/savevm.c
index 67c9c39..0ec2cb0 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -1242,8 +1242,8 @@ void qemu_savevm_state_cleanup(void)
 
 trace_savevm_state_cleanup();
 QTAILQ_FOREACH(se, _state.handlers, entry) {
-if (se->ops && se->ops->cleanup) {
-se->ops->cleanup(se->opaque);
+if (se->ops && se->ops->save_cleanup) {
+se->ops->save_cleanup(se->opaque);
 }
 }
 }
-- 
2.9.4

[Qemu-devel] [PATCH 1/5] migration: Rename save_live_setup() to save_setup()

[Juan Quintela]

We are going to use it now for more than save live regions.

Signed-off-by: Juan Quintela 
---
 hw/ppc/spapr.c   | 2 +-
 include/migration/register.h | 2 +-
 migration/block.c| 2 +-
 migration/ram.c  | 2 +-
 migration/savevm.c   | 6 +++---
 5 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index 0aadde6..d51a361 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -1873,7 +1873,7 @@ static void htab_cleanup(void *opaque)
 }
 
 static SaveVMHandlers savevm_htab_handlers = {
-.save_live_setup = htab_save_setup,
+.save_setup = htab_save_setup,
 .save_live_iterate = htab_save_iterate,
 .save_live_complete_precopy = htab_save_complete,
 .cleanup = htab_cleanup,
diff --git a/include/migration/register.h b/include/migration/register.h
index 844afaf..6cb27f4 100644
--- a/include/migration/register.h
+++ b/include/migration/register.h
@@ -36,7 +36,7 @@ typedef struct SaveVMHandlers {
 int (*save_live_iterate)(QEMUFile *f, void *opaque);
 
 /* This runs outside the iothread lock!  */
-int (*save_live_setup)(QEMUFile *f, void *opaque);
+int (*save_setup)(QEMUFile *f, void *opaque);
 void (*save_live_pending)(QEMUFile *f, void *opaque,
   uint64_t threshold_size,
   uint64_t *non_postcopiable_pending,
diff --git a/migration/block.c b/migration/block.c
index 14a8a84..976c42e 100644
--- a/migration/block.c
+++ b/migration/block.c
@@ -995,7 +995,7 @@ static bool block_is_active(void *opaque)
 }
 
 static SaveVMHandlers savevm_block_handlers = {
-.save_live_setup = block_save_setup,
+.save_setup = block_save_setup,
 .save_live_iterate = block_save_iterate,
 .save_live_complete_precopy = block_save_complete,
 .save_live_pending = block_save_pending,
diff --git a/migration/ram.c b/migration/ram.c
index 1164f14..f2c13e2 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -2597,7 +2597,7 @@ static int ram_load(QEMUFile *f, void *opaque, int 
version_id)
 }
 
 static SaveVMHandlers savevm_ram_handlers = {
-.save_live_setup = ram_save_setup,
+.save_setup = ram_save_setup,
 .save_live_iterate = ram_save_iterate,
 .save_live_complete_postcopy = ram_save_complete,
 .save_live_complete_precopy = ram_save_complete,
diff --git a/migration/savevm.c b/migration/savevm.c
index 8489abf..67c9c39 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -606,7 +606,7 @@ int register_savevm_live(DeviceState *dev,
 se->opaque = opaque;
 se->vmsd = NULL;
 /* if this is a live_savem then set is_ram */
-if (ops->save_live_setup != NULL) {
+if (ops->save_setup != NULL) {
 se->is_ram = 1;
 }
 
@@ -1000,7 +1000,7 @@ void qemu_savevm_state_begin(QEMUFile *f)
 
 trace_savevm_state_begin();
 QTAILQ_FOREACH(se, _state.handlers, entry) {
-if (!se->ops || !se->ops->save_live_setup) {
+if (!se->ops || !se->ops->save_setup) {
 continue;
 }
 if (se->ops && se->ops->is_active) {
@@ -1010,7 +1010,7 @@ void qemu_savevm_state_begin(QEMUFile *f)
 }
 save_section_header(f, se, QEMU_VM_SECTION_START);
 
-ret = se->ops->save_live_setup(f, se->opaque);
+ret = se->ops->save_setup(f, se->opaque);
 save_section_footer(f, se);
 if (ret < 0) {
 qemu_file_set_error(f, ret);
-- 
2.9.4

[Qemu-devel] [PATCH 0/5] Create setup/cleanup methods for migration incoming side

[Juan Quintela]

Hi

This series make:
- use of cleanup/save methods generic, not only for save_live methods
- create the equivalent methods for the load side (load_setup/cleanup)
- Make ram use this methods to see how/when they are used.

Stefan, Kevin, this were the methods that you asked for the block.c
migration, right?  Please, comment if they are enough for you.

This are on top of the ramstate_dynamic series that I just posted.

Please, review.

Later, Juan.

Juan Quintela (5):
  migration: Rename save_live_setup() to save_setup()
  migration: Rename cleanup() to save_cleanup()
  migration: Create load_setup()/cleanup() methods
  migration: Convert ram to use new load_setup()/load_cleanup()
  migration: Make compression_threads use save/load_setup/cleanup()

 hw/ppc/spapr.c   |  4 +--
 include/migration/register.h |  6 ++--
 migration/block.c|  4 +--
 migration/migration.c|  8 --
 migration/ram.c  | 66 ++--
 migration/ram.h  |  6 
 migration/savevm.c   | 41 +++
 7 files changed, 89 insertions(+), 46 deletions(-)

-- 
2.9.4

Re: [Qemu-devel] [PATCH 12/19] nbd/common: nbd_wr_syncv handle QIO_CHANNEL_ERR_EPIPE

[Eric Blake]

On 05/30/2017 09:30 AM, Vladimir Sementsov-Ogievskiy wrote:
> Return EPIPE in case of QIO_CHANNEL_ERR_EPIPE, we will need it to
> improve error path in nbd server.
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy 
> ---
>  nbd/common.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

I agree with Daniel's assessment that we probably aren't doing things
right in the previous patch, and therefore probably don't need this one.

> 
> diff --git a/nbd/common.c b/nbd/common.c
> index e520aae741..88e0297fb2 100644
> --- a/nbd/common.c
> +++ b/nbd/common.c
> @@ -52,7 +52,7 @@ ssize_t nbd_wr_syncv(QIOChannel *ioc,
>  continue;
>  }
>  if (len < 0) {
> -done = -EIO;
> +done = len == QIO_CHANNEL_ERR_EPIPE ? -EPIPE : -EIO;
>  goto cleanup;
>  }
>  
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org



signature.asc
Description: OpenPGP digital signature

[Qemu-devel] [PATCH 2/5] ram: Move ZERO_TARGET_PAGE inside XBZRLE

[Juan Quintela]

It was only used by XBZRLE anyways.

Signed-off-by: Juan Quintela 
---
 migration/ram.c | 11 ++-
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/migration/ram.c b/migration/ram.c
index e503277..04b55a7 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -66,8 +66,6 @@
 /* 0x80 is reserved in migration.h start with 0x100 next */
 #define RAM_SAVE_FLAG_COMPRESS_PAGE0x100
 
-static uint8_t *ZERO_TARGET_PAGE;
-
 static inline bool is_zero_range(uint8_t *p, uint64_t size)
 {
 return buffer_is_zero(p, size);
@@ -83,6 +81,8 @@ static struct {
 /* Cache for XBZRLE, Protected by lock. */
 PageCache *cache;
 QemuMutex lock;
+/* it will store a page full of zeros */
+uint8_t *zero_target_page;
 } XBZRLE;
 
 /* buffer used for XBZRLE decoding */
@@ -509,7 +509,7 @@ static void xbzrle_cache_zero_page(RAMState *rs, ram_addr_t 
current_addr)
 
 /* We don't care if this fails to allocate a new cache page
  * as long as it updated an old one */
-cache_insert(XBZRLE.cache, current_addr, ZERO_TARGET_PAGE,
+cache_insert(XBZRLE.cache, current_addr, XBZRLE.zero_target_page,
  rs->bitmap_sync_count);
 }
 
@@ -1453,10 +1453,11 @@ static void ram_migration_cleanup(void *opaque)
 cache_fini(XBZRLE.cache);
 g_free(XBZRLE.encoded_buf);
 g_free(XBZRLE.current_buf);
-g_free(ZERO_TARGET_PAGE);
+g_free(XBZRLE.zero_target_page);
 XBZRLE.cache = NULL;
 XBZRLE.encoded_buf = NULL;
 XBZRLE.current_buf = NULL;
+XBZRLE.zero_target_page = NULL;
 }
 XBZRLE_cache_unlock();
 migration_page_queue_free(rs);
@@ -1877,7 +1878,7 @@ static int ram_state_init(RAMState *rs)
 
 if (migrate_use_xbzrle()) {
 XBZRLE_cache_lock();
-ZERO_TARGET_PAGE = g_malloc0(TARGET_PAGE_SIZE);
+XBZRLE.zero_target_page = g_malloc0(TARGET_PAGE_SIZE);
 XBZRLE.cache = cache_init(migrate_xbzrle_cache_size() /
   TARGET_PAGE_SIZE,
   TARGET_PAGE_SIZE);
-- 
2.9.4

[Qemu-devel] [PATCH 5/5] ram: Make RAMState dynamic

[Juan Quintela]

We create the variable while we are at migration and we remove it
after migration.

Signed-off-by: Juan Quintela 
---
 migration/ram.c | 52 
 1 file changed, 32 insertions(+), 20 deletions(-)

diff --git a/migration/ram.c b/migration/ram.c
index 6c48219..1164f14 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -199,7 +199,7 @@ struct RAMState {
 };
 typedef struct RAMState RAMState;
 
-static RAMState ram_state;
+static RAMState *ram_state;
 
 MigrationStats ram_counters;
 
@@ -783,7 +783,7 @@ static int ram_save_page(RAMState *rs, PageSearchStatus 
*pss, bool last_stage)
 static int do_compress_ram_page(QEMUFile *f, RAMBlock *block,
 ram_addr_t offset)
 {
-RAMState *rs = _state;
+RAMState *rs = ram_state;
 int bytes_sent, blen;
 uint8_t *p = block->host + (offset & TARGET_PAGE_MASK);
 
@@ -1130,7 +1130,7 @@ static void migration_page_queue_free(RAMState *rs)
 int ram_save_queue_pages(const char *rbname, ram_addr_t start, ram_addr_t len)
 {
 RAMBlock *ramblock;
-RAMState *rs = _state;
+RAMState *rs = ram_state;
 
 ram_counters.postcopy_requests++;
 rcu_read_lock();
@@ -1351,7 +1351,7 @@ void free_xbzrle_decoded_buf(void)
 
 static void ram_migration_cleanup(void *opaque)
 {
-RAMState *rs = opaque;
+RAMState **rsp = opaque;
 RAMBlock *block;
 
 /* caller have hold iothread lock or is in a bh, so there is
@@ -1378,7 +1378,9 @@ static void ram_migration_cleanup(void *opaque)
 XBZRLE.zero_target_page = NULL;
 }
 XBZRLE_cache_unlock();
-migration_page_queue_free(rs);
+migration_page_queue_free(*rsp);
+g_free(*rsp);
+*rsp = NULL;
 }
 
 static void ram_state_reset(RAMState *rs)
@@ -1703,7 +1705,7 @@ static int postcopy_chunk_hostpages(MigrationState *ms, 
RAMBlock *block)
  */
 int ram_postcopy_send_discard_bitmap(MigrationState *ms)
 {
-RAMState *rs = _state;
+RAMState *rs = ram_state;
 RAMBlock *block;
 int ret;
 
@@ -1786,12 +1788,13 @@ err:
 return ret;
 }
 
-static int ram_state_init(RAMState *rs)
+static int ram_state_init(RAMState **rsp)
 {
-memset(rs, 0, sizeof(*rs));
-qemu_mutex_init(>bitmap_mutex);
-qemu_mutex_init(>src_page_req_mutex);
-QSIMPLEQ_INIT(>src_page_requests);
+*rsp = g_new0(RAMState, 1);
+
+qemu_mutex_init(&(*rsp)->bitmap_mutex);
+qemu_mutex_init(&(*rsp)->src_page_req_mutex);
+QSIMPLEQ_INIT(&(*rsp)->src_page_requests);
 
 if (migrate_use_xbzrle()) {
 XBZRLE_cache_lock();
@@ -1802,6 +1805,8 @@ static int ram_state_init(RAMState *rs)
 if (!XBZRLE.cache) {
 XBZRLE_cache_unlock();
 error_report("Error creating cache");
+g_free(*rsp);
+*rsp = NULL;
 return -1;
 }
 XBZRLE_cache_unlock();
@@ -1810,6 +1815,8 @@ static int ram_state_init(RAMState *rs)
 XBZRLE.encoded_buf = g_try_malloc0(TARGET_PAGE_SIZE);
 if (!XBZRLE.encoded_buf) {
 error_report("Error allocating encoded_buf");
+g_free(*rsp);
+*rsp = NULL;
 return -1;
 }
 
@@ -1818,6 +1825,8 @@ static int ram_state_init(RAMState *rs)
 error_report("Error allocating current_buf");
 g_free(XBZRLE.encoded_buf);
 XBZRLE.encoded_buf = NULL;
+g_free(*rsp);
+*rsp = NULL;
 return -1;
 }
 }
@@ -1827,7 +1836,7 @@ static int ram_state_init(RAMState *rs)
 
 qemu_mutex_lock_ramlist();
 rcu_read_lock();
-ram_state_reset(rs);
+ram_state_reset(*rsp);
 
 /* Skip setting bitmap if there is no RAM */
 if (ram_bytes_total()) {
@@ -1852,7 +1861,7 @@ static int ram_state_init(RAMState *rs)
 ram_counters.remaining_pages = ram_bytes_total() >> TARGET_PAGE_BITS;
 
 memory_global_dirty_log_start();
-migration_bitmap_sync(rs);
+migration_bitmap_sync(*rsp);
 qemu_mutex_unlock_ramlist();
 qemu_mutex_unlock_iothread();
 rcu_read_unlock();
@@ -1877,16 +1886,16 @@ static int ram_state_init(RAMState *rs)
  */
 static int ram_save_setup(QEMUFile *f, void *opaque)
 {
-RAMState *rs = opaque;
+RAMState **rsp = opaque;
 RAMBlock *block;
 
 /* migration has already setup the bitmap, reuse it. */
 if (!migration_in_colo_state()) {
-if (ram_state_init(rs) < 0) {
+if (ram_state_init(rsp) != 0) {
 return -1;
- }
+}
 }
-rs->f = f;
+(*rsp)->f = f;
 
 rcu_read_lock();
 
@@ -1921,7 +1930,8 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
  */
 static int ram_save_iterate(QEMUFile *f, void *opaque)
 {
-RAMState *rs = opaque;
+RAMState **temp = opaque;
+RAMState *rs = *temp;
 int ret;
 int i;
 int64_t t0;
@@ -1996,7 +2006,8 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
  */
 static int ram_save_complete(QEMUFile *f, void *opaque)
 {
-  

[Qemu-devel] [PATCH 1/5] ram: Call migration_page_queue_free() at ram_migration_cleanup()

[Juan Quintela]

We shouldn't be using memory later than that.

Signed-off-by: Juan Quintela 
---
 migration/migration.c | 2 --
 migration/ram.c   | 5 +++--
 migration/ram.h   | 1 -
 3 files changed, 3 insertions(+), 5 deletions(-)

diff --git a/migration/migration.c b/migration/migration.c
index af4c2cc..ea3d41c 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -814,8 +814,6 @@ static void migrate_fd_cleanup(void *opaque)
 qemu_bh_delete(s->cleanup_bh);
 s->cleanup_bh = NULL;
 
-migration_page_queue_free();
-
 if (s->to_dst_file) {
 trace_migrate_fd_cleanup();
 qemu_mutex_unlock_iothread();
diff --git a/migration/ram.c b/migration/ram.c
index db7f4b0..e503277 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -1181,10 +1181,9 @@ static bool get_queued_page(RAMState *rs, 
PageSearchStatus *pss)
  * be some left.  in case that there is any page left, we drop it.
  *
  */
-void migration_page_queue_free(void)
+static void migration_page_queue_free(RAMState *rs)
 {
 struct RAMSrcPageRequest *mspr, *next_mspr;
-RAMState *rs = _state;
 /* This queue generally should be empty - but in the case of a failed
  * migration might have some droppings in.
  */
@@ -1434,6 +1433,7 @@ void free_xbzrle_decoded_buf(void)
 
 static void ram_migration_cleanup(void *opaque)
 {
+RAMState *rs = opaque;
 RAMBlock *block;
 
 /* caller have hold iothread lock or is in a bh, so there is
@@ -1459,6 +1459,7 @@ static void ram_migration_cleanup(void *opaque)
 XBZRLE.current_buf = NULL;
 }
 XBZRLE_cache_unlock();
+migration_page_queue_free(rs);
 }
 
 static void ram_state_reset(RAMState *rs)
diff --git a/migration/ram.h b/migration/ram.h
index c9563d1..d4da419 100644
--- a/migration/ram.h
+++ b/migration/ram.h
@@ -53,7 +53,6 @@ void migrate_decompress_threads_create(void);
 void migrate_decompress_threads_join(void);
 
 uint64_t ram_pagesize_summary(void);
-void migration_page_queue_free(void);
 int ram_save_queue_pages(const char *rbname, ram_addr_t start, ram_addr_t len);
 void acct_update_position(QEMUFile *f, size_t size, bool zero);
 void free_xbzrle_decoded_buf(void);
-- 
2.9.4

[Qemu-devel] [PATCH 4/5] ram: Use MigrationStats for statistics

[Juan Quintela]

RAM Statistics need to survive migration to make info migrate work, so we
need to store them outside of RAMState.  As we already have an struct
with those fields, just used them. (MigrationStats and XBZRLECacheStats).

Signed-off-by: Juan Quintela 
---
 migration/migration.c |  33 +-
 migration/ram.c   | 179 ++
 migration/ram.h   |  15 +
 3 files changed, 68 insertions(+), 159 deletions(-)

diff --git a/migration/migration.c b/migration/migration.c
index 2c13217..331cab7 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -480,28 +480,28 @@ static void populate_ram_info(MigrationInfo *info, 
MigrationState *s)
 {
 info->has_ram = true;
 info->ram = g_malloc0(sizeof(*info->ram));
-info->ram->transferred = ram_bytes_transferred();
+info->ram->transferred = ram_counters.transferred;
 info->ram->total = ram_bytes_total();
-info->ram->duplicate = dup_mig_pages_transferred();
+info->ram->duplicate = ram_counters.duplicate;
 /* legacy value.  It is not used anymore */
 info->ram->skipped = 0;
-info->ram->normal = norm_mig_pages_transferred();
-info->ram->normal_bytes = norm_mig_pages_transferred() *
+info->ram->normal = ram_counters.normal;
+info->ram->normal_bytes = ram_counters.normal *
 qemu_target_page_size();
 info->ram->mbps = s->mbps;
-info->ram->dirty_sync_count = ram_dirty_sync_count();
-info->ram->postcopy_requests = ram_postcopy_requests();
+info->ram->dirty_sync_count = ram_counters.dirty_sync_count;
+info->ram->postcopy_requests = ram_counters.postcopy_requests;
 info->ram->page_size = qemu_target_page_size();
 
 if (migrate_use_xbzrle()) {
 info->has_xbzrle_cache = true;
 info->xbzrle_cache = g_malloc0(sizeof(*info->xbzrle_cache));
 info->xbzrle_cache->cache_size = migrate_xbzrle_cache_size();
-info->xbzrle_cache->bytes = xbzrle_mig_bytes_transferred();
-info->xbzrle_cache->pages = xbzrle_mig_pages_transferred();
-info->xbzrle_cache->cache_miss = xbzrle_mig_pages_cache_miss();
-info->xbzrle_cache->cache_miss_rate = xbzrle_mig_cache_miss_rate();
-info->xbzrle_cache->overflow = xbzrle_mig_pages_overflow();
+info->xbzrle_cache->bytes = xbzrle_counters.bytes;
+info->xbzrle_cache->pages = xbzrle_counters.pages;
+info->xbzrle_cache->cache_miss = xbzrle_counters.cache_miss;
+info->xbzrle_cache->cache_miss_rate = xbzrle_counters.cache_miss_rate;
+info->xbzrle_cache->overflow = xbzrle_counters.overflow;
 }
 
 if (cpu_throttle_active()) {
@@ -518,10 +518,11 @@ static void populate_ram_info(MigrationInfo *info, 
MigrationState *s)
 }
 
 if (s->state != MIGRATION_STATUS_COMPLETED) {
-info->ram->remaining_pages = ram_pages_remaining();
-info->ram->remaining = ram_pages_remaining() *
+
+info->ram->remaining_pages = ram_counters.remaining_pages;
+info->ram->remaining = ram_counters.remaining_pages *
 qemu_target_page_size();
-info->ram->dirty_pages_rate = ram_dirty_pages_rate();
+info->ram->dirty_pages_rate = ram_counters.dirty_pages_rate;
 }
 }
 
@@ -1886,8 +1887,8 @@ static void *migration_thread(void *opaque)
   bandwidth, threshold_size);
 /* if we haven't sent anything, we don't want to recalculate
1 is a small enough number for our purposes */
-if (ram_dirty_pages_rate() && transferred_bytes > 1) {
-s->expected_downtime = ram_dirty_pages_rate() *
+if (ram_counters.dirty_pages_rate && transferred_bytes > 1) {
+s->expected_downtime = ram_counters.dirty_pages_rate *
 qemu_target_page_size() / bandwidth;
 }
 
diff --git a/migration/ram.c b/migration/ram.c
index 30519e1..6c48219 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -71,6 +71,8 @@ static inline bool is_zero_range(uint8_t *p, uint64_t size)
 return buffer_is_zero(p, size);
 }
 
+XBZRLECacheStats xbzrle_counters;
+
 /* struct contains XBZRLE cache and a static page
used by the compression */
 static struct {
@@ -174,8 +176,6 @@ struct RAMState {
 bool ram_bulk_stage;
 /* How many times we have dirty too many pages */
 int dirty_rate_high_cnt;
-/* How many times we have synchronized the bitmap */
-uint64_t bitmap_sync_count;
 /* these variables are used for bitmap sync */
 /* last time we did a full bitmap_sync */
 int64_t time_last_bitmap_sync;
@@ -187,32 +187,8 @@ struct RAMState {
 uint64_t xbzrle_cache_miss_prev;
 /* number of iterations at the beginning of period */
 uint64_t iterations_prev;
-/* Accounting fields */
-/* number of zero pages.  It used to be pages filled by the same char. */
-uint64_t zero_pages;
-/* number of normal transferred 

[Qemu-devel] [PATCH 0/5] Make RAMState dynamic

[Juan Quintela]

Hi

This series make RAMState a dynamic variable.  We create it at the
beggining of migration and remove it when migration ends.

- Move ZERO_TARGET_PAGE to XBZRLE
- print the number of remaining pages not only the number of bytes
  this makes easier to use statistics.
- How to export statistics to use for "info migrate"?  It is
  complicated, right now we use accessor functions for RAMState.  But we
  already have a struct with the fields that we need.  MigrationStats.
  Use them instead of having to create a new accessor function for
  each new field that we print.  There is another reason for this: We
  want to make RAMstate dynamic. And we access states after migration
  has finished.
- We end making RAMState dynamic.

This series is on top of the "previous consistent ouput"

Please, review.

Thanks, Juan.

Juan Quintela (5):
  ram: Call migration_page_queue_free() at ram_migration_cleanup()
  ram: Move ZERO_TARGET_PAGE inside XBZRLE
  migration: Print statistics about the number of remaining target pages
  ram: Use MigrationStats for statistics
  ram: Make RAMState dynamic

 migration/migration.c |  35 
 migration/ram.c   | 243 ++
 migration/ram.h   |  16 +---
 qapi-schema.json  |   6 +-
 4 files changed, 113 insertions(+), 187 deletions(-)

-- 
2.9.4

[Qemu-devel] [PATCH 3/5] migration: Print statistics about the number of remaining target pages

[Juan Quintela]

Signed-off-by: Juan Quintela 
---
 migration/migration.c | 4 +++-
 migration/ram.c   | 4 ++--
 migration/ram.h   | 2 +-
 qapi-schema.json  | 6 +-
 4 files changed, 11 insertions(+), 5 deletions(-)

diff --git a/migration/migration.c b/migration/migration.c
index ea3d41c..2c13217 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -518,7 +518,9 @@ static void populate_ram_info(MigrationInfo *info, 
MigrationState *s)
 }
 
 if (s->state != MIGRATION_STATUS_COMPLETED) {
-info->ram->remaining = ram_bytes_remaining();
+info->ram->remaining_pages = ram_pages_remaining();
+info->ram->remaining = ram_pages_remaining() *
+qemu_target_page_size();
 info->ram->dirty_pages_rate = ram_dirty_pages_rate();
 }
 }
diff --git a/migration/ram.c b/migration/ram.c
index 04b55a7..30519e1 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -265,9 +265,9 @@ uint64_t ram_bytes_transferred(void)
 return ram_state.bytes_transferred;
 }
 
-uint64_t ram_bytes_remaining(void)
+uint64_t ram_pages_remaining(void)
 {
-return ram_state.migration_dirty_pages * TARGET_PAGE_SIZE;
+return ram_state.migration_dirty_pages;
 }
 
 uint64_t ram_dirty_sync_count(void)
diff --git a/migration/ram.h b/migration/ram.h
index d4da419..5864470 100644
--- a/migration/ram.h
+++ b/migration/ram.h
@@ -41,7 +41,7 @@ uint64_t xbzrle_mig_pages_cache_miss(void);
 double xbzrle_mig_cache_miss_rate(void);
 uint64_t xbzrle_mig_pages_overflow(void);
 uint64_t ram_bytes_transferred(void);
-uint64_t ram_bytes_remaining(void);
+uint64_t ram_pages_remaining(void);
 uint64_t ram_dirty_sync_count(void);
 uint64_t ram_dirty_pages_rate(void);
 uint64_t ram_postcopy_requests(void);
diff --git a/qapi-schema.json b/qapi-schema.json
index 4b50b65..ff1c048 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -601,6 +601,9 @@
 # @page-size: The number of bytes per page for the various page-based
 #statistics (since 2.10)
 #
+# @remaining-pages: amount of pages remaining to be transferred to the target 
VM
+#(since 2.10)
+#
 # Since: 0.14.0
 ##
 { 'struct': 'MigrationStats',
@@ -608,7 +611,8 @@
'duplicate': 'int', 'skipped': 'int', 'normal': 'int',
'normal-bytes': 'int', 'dirty-pages-rate' : 'int',
'mbps' : 'number', 'dirty-sync-count' : 'int',
-   'postcopy-requests' : 'int', 'page-size' : 'int' } }
+   'postcopy-requests' : 'int', 'page-size' : 'int',
+   'remaining-pages' : 'int' } }
 
 ##
 # @XBZRLECacheStats:
-- 
2.9.4

Re: [Qemu-devel] [PATCH] msi: remove return code for msi_init()

[Paul Burton]

Hi Aurelien/Paolo/Marcel,

On Thursday, 1 June 2017 12:22:06 PDT Aurelien Jarno wrote:
> On 2017-06-01 16:23, Paolo Bonzini wrote:
> > On 01/06/2017 10:27, Marcel Apfelbaum wrote:
> > > On 31/05/2017 11:28, Paolo Bonzini wrote:
> > >> No, for now I'd rather just go and remove msi_nonbroken.  When someone
> > >> reports a bug, we can add back "msi_broken".
> > > 
> > > Hi,
> > > I agree with the direction, but I am concerned msi_nonbroken is there
> > > for a reason.
> > > We might break some (obscure/not in use) machine.
> > > Maybe we should CC all arch machine maintainers/contributors to give
> > > them a chance to object...
> > 
> > Yeah, Alpha, MIPS and SH are those that support PCI.  Adding Richard and
> > Aurelien, do your platforms support MSI on real hardware but not in QEMU?
> 
> SH clearly doesn't support MSI.
> 
> The oldest MIPS board also do not support MSI, but I guess the Boston
> board might support it. I am adding Paul Burton in Cc: who probably
> knows about that.
> 
> Aurelien

Indeed, real Boston hardware does support MSI (or rather, the Xilinx AXI 
Bridge for PCI Express IP used on Boston does) & we make use of it in Linux.

Thanks,
Paul

signature.asc
Description: This is a digitally signed message part.

[Qemu-devel] [PATCH 4/4] ram: Now POSTCOPY_ACTIVE is the same that STATUS_ACTIVE

[Juan Quintela]

Merge them.

Signed-off-by: Juan Quintela 
---
 migration/migration.c | 12 
 1 file changed, 12 deletions(-)

diff --git a/migration/migration.c b/migration/migration.c
index 8289544..af4c2cc 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -538,19 +538,7 @@ MigrationInfo *qmp_query_migrate(Error **errp)
 break;
 case MIGRATION_STATUS_ACTIVE:
 case MIGRATION_STATUS_CANCELLING:
-info->has_status = true;
-info->has_total_time = true;
-info->total_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME)
-- s->total_time;
-info->has_expected_downtime = true;
-info->expected_downtime = s->expected_downtime;
-info->has_setup_time = true;
-info->setup_time = s->setup_time;
-
-populate_ram_info(info, s);
-break;
 case MIGRATION_STATUS_POSTCOPY_ACTIVE:
-/* Mostly the same as active; TODO add some postcopy stats */
 info->has_status = true;
 info->has_total_time = true;
 info->total_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME)
-- 
2.9.4

[Qemu-devel] [PATCH 2/4] ram: We only print throttling information sometimes

[Juan Quintela]

Change it to be consistent with everything else.

Signed-off-by: Juan Quintela 
---
 migration/migration.c | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/migration/migration.c b/migration/migration.c
index fd8b406..7f79da0 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -504,6 +504,11 @@ static void populate_ram_info(MigrationInfo *info, 
MigrationState *s)
 info->xbzrle_cache->overflow = xbzrle_mig_pages_overflow();
 }
 
+if (cpu_throttle_active()) {
+info->has_cpu_throttle_percentage = true;
+info->cpu_throttle_percentage = cpu_throttle_get_percentage();
+}
+
 if (s->state != MIGRATION_STATUS_COMPLETED) {
 info->ram->remaining = ram_bytes_remaining();
 info->ram->dirty_pages_rate = ram_dirty_pages_rate();
@@ -544,11 +549,6 @@ MigrationInfo *qmp_query_migrate(Error **errp)
 info->disk->total = blk_mig_bytes_total();
 }
 
-if (cpu_throttle_active()) {
-info->has_cpu_throttle_percentage = true;
-info->cpu_throttle_percentage = cpu_throttle_get_percentage();
-}
-
 break;
 case MIGRATION_STATUS_POSTCOPY_ACTIVE:
 /* Mostly the same as active; TODO add some postcopy stats */
-- 
2.9.4

[Qemu-devel] [PATCH 3/4] ram: Print block stats also in the complete case

[Juan Quintela]

To make things easier, I just moved it to populate_ram_info().

Signed-off-by: Juan Quintela 
---
 migration/migration.c | 26 --
 1 file changed, 8 insertions(+), 18 deletions(-)

diff --git a/migration/migration.c b/migration/migration.c
index 7f79da0..8289544 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -509,6 +509,14 @@ static void populate_ram_info(MigrationInfo *info, 
MigrationState *s)
 info->cpu_throttle_percentage = cpu_throttle_get_percentage();
 }
 
+if (blk_mig_active()) {
+info->has_disk = true;
+info->disk = g_malloc0(sizeof(*info->disk));
+info->disk->transferred = blk_mig_bytes_transferred();
+info->disk->remaining = blk_mig_bytes_remaining();
+info->disk->total = blk_mig_bytes_total();
+}
+
 if (s->state != MIGRATION_STATUS_COMPLETED) {
 info->ram->remaining = ram_bytes_remaining();
 info->ram->dirty_pages_rate = ram_dirty_pages_rate();
@@ -540,15 +548,6 @@ MigrationInfo *qmp_query_migrate(Error **errp)
 info->setup_time = s->setup_time;
 
 populate_ram_info(info, s);
-
-if (blk_mig_active()) {
-info->has_disk = true;
-info->disk = g_malloc0(sizeof(*info->disk));
-info->disk->transferred = blk_mig_bytes_transferred();
-info->disk->remaining = blk_mig_bytes_remaining();
-info->disk->total = blk_mig_bytes_total();
-}
-
 break;
 case MIGRATION_STATUS_POSTCOPY_ACTIVE:
 /* Mostly the same as active; TODO add some postcopy stats */
@@ -562,15 +561,6 @@ MigrationInfo *qmp_query_migrate(Error **errp)
 info->setup_time = s->setup_time;
 
 populate_ram_info(info, s);
-
-if (blk_mig_active()) {
-info->has_disk = true;
-info->disk = g_malloc0(sizeof(*info->disk));
-info->disk->transferred = blk_mig_bytes_transferred();
-info->disk->remaining = blk_mig_bytes_remaining();
-info->disk->total = blk_mig_bytes_total();
-}
-
 break;
 case MIGRATION_STATUS_COLO:
 info->has_status = true;
-- 
2.9.4

[Qemu-devel] [PATCH 1/4] ram: Unfold get_xbzrle_cache_stats() into populate_ram_info()

[Juan Quintela]

They were called consequtiveley always.

Signed-off-by: Juan Quintela 
---
 migration/migration.c | 29 +++--
 1 file changed, 11 insertions(+), 18 deletions(-)

diff --git a/migration/migration.c b/migration/migration.c
index 60da9d6..fd8b406 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -476,20 +476,6 @@ static bool migration_is_setup_or_active(int state)
 }
 }
 
-static void get_xbzrle_cache_stats(MigrationInfo *info)
-{
-if (migrate_use_xbzrle()) {
-info->has_xbzrle_cache = true;
-info->xbzrle_cache = g_malloc0(sizeof(*info->xbzrle_cache));
-info->xbzrle_cache->cache_size = migrate_xbzrle_cache_size();
-info->xbzrle_cache->bytes = xbzrle_mig_bytes_transferred();
-info->xbzrle_cache->pages = xbzrle_mig_pages_transferred();
-info->xbzrle_cache->cache_miss = xbzrle_mig_pages_cache_miss();
-info->xbzrle_cache->cache_miss_rate = xbzrle_mig_cache_miss_rate();
-info->xbzrle_cache->overflow = xbzrle_mig_pages_overflow();
-}
-}
-
 static void populate_ram_info(MigrationInfo *info, MigrationState *s)
 {
 info->has_ram = true;
@@ -507,6 +493,17 @@ static void populate_ram_info(MigrationInfo *info, 
MigrationState *s)
 info->ram->postcopy_requests = ram_postcopy_requests();
 info->ram->page_size = qemu_target_page_size();
 
+if (migrate_use_xbzrle()) {
+info->has_xbzrle_cache = true;
+info->xbzrle_cache = g_malloc0(sizeof(*info->xbzrle_cache));
+info->xbzrle_cache->cache_size = migrate_xbzrle_cache_size();
+info->xbzrle_cache->bytes = xbzrle_mig_bytes_transferred();
+info->xbzrle_cache->pages = xbzrle_mig_pages_transferred();
+info->xbzrle_cache->cache_miss = xbzrle_mig_pages_cache_miss();
+info->xbzrle_cache->cache_miss_rate = xbzrle_mig_cache_miss_rate();
+info->xbzrle_cache->overflow = xbzrle_mig_pages_overflow();
+}
+
 if (s->state != MIGRATION_STATUS_COMPLETED) {
 info->ram->remaining = ram_bytes_remaining();
 info->ram->dirty_pages_rate = ram_dirty_pages_rate();
@@ -552,7 +549,6 @@ MigrationInfo *qmp_query_migrate(Error **errp)
 info->cpu_throttle_percentage = cpu_throttle_get_percentage();
 }
 
-get_xbzrle_cache_stats(info);
 break;
 case MIGRATION_STATUS_POSTCOPY_ACTIVE:
 /* Mostly the same as active; TODO add some postcopy stats */
@@ -575,15 +571,12 @@ MigrationInfo *qmp_query_migrate(Error **errp)
 info->disk->total = blk_mig_bytes_total();
 }
 
-get_xbzrle_cache_stats(info);
 break;
 case MIGRATION_STATUS_COLO:
 info->has_status = true;
 /* TODO: display COLO specific information (checkpoint info etc.) */
 break;
 case MIGRATION_STATUS_COMPLETED:
-get_xbzrle_cache_stats(info);
-
 info->has_status = true;
 info->has_total_time = true;
 info->total_time = s->total_time;
-- 
2.9.4

[Qemu-devel] [PATCH 0/4] Make info migrate output consistent

[Juan Quintela]

Hi

This series make the output of info migrate be printed in a single
place for the states ACTIVE, CANCELLING and POSTCOPY_ACTIVE.

The code was duplicated, just use the same code.  Once done that, fix
the inconsistences that were not needed.

Please, review.

Thanks, Juan.

Juan Quintela (4):
  ram: Unfold get_xbzrle_cache_stats() into populate_ram_info()
  ram: We only print throttling information sometimes
  ram: Print block stats also in the complete case
  ram: Now POSTCOPY_ACTIVE is the same that STATUS_ACTIVE

 migration/migration.c | 77 ---
 1 file changed, 24 insertions(+), 53 deletions(-)

-- 
2.9.4

[Qemu-devel] [PATCH 09/11] migration: Move remaining exported functions to migration/misc.h

[Juan Quintela]

Signed-off-by: Juan Quintela 
---
 hw/i386/pc_piix.c |  2 +-
 hw/ppc/spapr.c|  2 +-
 hw/xen/xen-common.c   |  2 +-
 include/migration/migration.h | 13 -
 include/migration/misc.h  | 12 
 migration/migration.c |  1 +
 qdev-monitor.c|  2 +-
 ui/spice-core.c   |  2 +-
 vl.c  |  1 -
 9 files changed, 18 insertions(+), 19 deletions(-)

diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
index dc19d96..46a2bc4 100644
--- a/hw/i386/pc_piix.c
+++ b/hw/i386/pc_piix.c
@@ -53,7 +53,7 @@
 #include "hw/xen/xen_pt.h"
 #endif
 #include "migration/global_state.h"
-#include "migration/migration.h"
+#include "migration/misc.h"
 #include "kvm_i386.h"
 #include "sysemu/numa.h"
 
diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index ac8a317..0aadde6 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -38,7 +38,7 @@
 #include "sysemu/cpus.h"
 #include "sysemu/hw_accel.h"
 #include "kvm_ppc.h"
-#include "migration/migration.h"
+#include "migration/misc.h"
 #include "migration/global_state.h"
 #include "migration/register.h"
 #include "mmu-hash64.h"
diff --git a/hw/xen/xen-common.c b/hw/xen/xen-common.c
index e265445..19ad79c 100644
--- a/hw/xen/xen-common.c
+++ b/hw/xen/xen-common.c
@@ -13,7 +13,7 @@
 #include "qmp-commands.h"
 #include "sysemu/char.h"
 #include "sysemu/accel.h"
-#include "migration/migration.h"
+#include "migration/misc.h"
 #include "migration/global_state.h"
 
 //#define DEBUG_XEN
diff --git a/include/migration/migration.h b/include/migration/migration.h
index 5050454..601e4ab 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -122,26 +122,16 @@ void migrate_set_state(int *state, int old_state, int 
new_state);
 
 void migration_fd_process_incoming(QEMUFile *f);
 
-void qemu_start_incoming_migration(const char *uri, Error **errp);
-
 uint64_t migrate_max_downtime(void);
 
 void migrate_fd_error(MigrationState *s, const Error *error);
 
 void migrate_fd_connect(MigrationState *s);
 
-void add_migration_state_change_notifier(Notifier *notify);
-void remove_migration_state_change_notifier(Notifier *notify);
 MigrationState *migrate_init(void);
 bool migration_is_blocked(Error **errp);
-bool migration_in_setup(MigrationState *);
-bool migration_is_idle(void);
-bool migration_has_finished(MigrationState *);
-bool migration_has_failed(MigrationState *);
 /* True if outgoing migration has entered postcopy phase */
 bool migration_in_postcopy(void);
-/* ...and after the device transmission */
-bool migration_in_postcopy_after_devices(MigrationState *);
 MigrationState *migrate_get_current(void);
 
 bool migrate_release_ram(void);
@@ -171,7 +161,4 @@ void migrate_send_rp_pong(MigrationIncomingState *mis,
 void migrate_send_rp_req_pages(MigrationIncomingState *mis, const char* rbname,
   ram_addr_t start, size_t len);
 
-void savevm_skip_section_footers(void);
-void savevm_skip_configuration(void);
-
 #endif
diff --git a/include/migration/misc.h b/include/migration/misc.h
index 026b561..d5a433a 100644
--- a/include/migration/misc.h
+++ b/include/migration/misc.h
@@ -39,5 +39,17 @@ int64_t self_announce_delay(int round)
 /* migration/savevm.c */
 
 void dump_vmstate_json_to_file(FILE *out_fp);
+void savevm_skip_section_footers(void);
+void savevm_skip_configuration(void);
 
+/* migration/migration.c */
+void qemu_start_incoming_migration(const char *uri, Error **errp);
+bool migration_is_idle(void);
+void add_migration_state_change_notifier(Notifier *notify);
+void remove_migration_state_change_notifier(Notifier *notify);
+bool migration_in_setup(MigrationState *);
+bool migration_has_finished(MigrationState *);
+bool migration_has_failed(MigrationState *);
+/* ...and after the device transmission */
+bool migration_in_postcopy_after_devices(MigrationState *);
 #endif
diff --git a/migration/migration.c b/migration/migration.c
index 6a7620b..b6c0cfd 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -24,6 +24,7 @@
 #include "rdma.h"
 #include "ram.h"
 #include "migration/global_state.h"
+#include "migration/misc.h"
 #include "migration/migration.h"
 #include "savevm.h"
 #include "qemu-file-channel.h"
diff --git a/qdev-monitor.c b/qdev-monitor.c
index 3ecbf0b..8fd6df9 100644
--- a/qdev-monitor.c
+++ b/qdev-monitor.c
@@ -29,7 +29,7 @@
 #include "qemu/error-report.h"
 #include "qemu/help_option.h"
 #include "sysemu/block-backend.h"
-#include "migration/migration.h"
+#include "migration/misc.h"
 
 /*
  * Aliases were a bad idea from the start.  Let's keep them
diff --git a/ui/spice-core.c b/ui/spice-core.c
index 804abc5..a087ad5 100644
--- a/ui/spice-core.c
+++ b/ui/spice-core.c
@@ -35,7 +35,7 @@
 #include "qapi/qmp/qstring.h"
 #include "qapi/qmp/qjson.h"
 #include "qemu/notify.h"
-#include "migration/migration.h"
+#include "migration/misc.h"
 #include "hw/hw.h"
 #include "ui/spice-display.h"
 #include 

[Qemu-devel] [PATCH 10/11] migration: Move migration.h to migration/

[Juan Quintela]

Nothing uses it outside of migration.h

Signed-off-by: Juan Quintela 
---
 migration/block.c| 2 +-
 migration/channel.c  | 2 +-
 migration/colo-comm.c| 2 +-
 migration/colo.c | 2 +-
 migration/exec.c | 2 +-
 migration/fd.c   | 2 +-
 migration/migration.c| 2 +-
 {include/migration => migration}/migration.h | 0
 migration/postcopy-ram.c | 2 +-
 migration/qemu-file.c| 2 +-
 migration/ram.c  | 2 +-
 migration/rdma.c | 2 +-
 migration/savevm.c   | 2 +-
 migration/socket.c   | 2 +-
 migration/tls.c  | 2 +-
 migration/vmstate-types.c| 2 +-
 migration/vmstate.c  | 2 +-
 tests/test-vmstate.c | 2 +-
 18 files changed, 17 insertions(+), 17 deletions(-)
 rename {include/migration => migration}/migration.h (100%)

diff --git a/migration/block.c b/migration/block.c
index 30ab531..21c7fac 100644
--- a/migration/block.c
+++ b/migration/block.c
@@ -25,7 +25,7 @@
 #include "qemu/timer.h"
 #include "block.h"
 #include "migration/misc.h"
-#include "migration/migration.h"
+#include "migration.h"
 #include "migration/register.h"
 #include "sysemu/blockdev.h"
 #include "qemu-file.h"
diff --git a/migration/channel.c b/migration/channel.c
index 92f6f99..3b7252f 100644
--- a/migration/channel.c
+++ b/migration/channel.c
@@ -13,7 +13,7 @@
 #include "qemu/osdep.h"
 #include "channel.h"
 #include "tls.h"
-#include "migration/migration.h"
+#include "migration.h"
 #include "qemu-file-channel.h"
 #include "trace.h"
 #include "qapi/error.h"
diff --git a/migration/colo-comm.c b/migration/colo-comm.c
index 8bfdf68..b61aa19 100644
--- a/migration/colo-comm.c
+++ b/migration/colo-comm.c
@@ -12,7 +12,7 @@
  */
 
 #include "qemu/osdep.h"
-#include "migration/migration.h"
+#include "migration.h"
 #include "migration/colo.h"
 #include "migration/vmstate.h"
 #include "trace.h"
diff --git a/migration/colo.c b/migration/colo.c
index 111b715..45e9b46 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -14,7 +14,7 @@
 #include "qemu/timer.h"
 #include "sysemu/sysemu.h"
 #include "qemu-file-channel.h"
-#include "migration/migration.h"
+#include "migration.h"
 #include "qemu-file.h"
 #include "savevm.h"
 #include "migration/colo.h"
diff --git a/migration/exec.c b/migration/exec.c
index fc78eeb..b1de445 100644
--- a/migration/exec.c
+++ b/migration/exec.c
@@ -22,7 +22,7 @@
 #include "qemu-common.h"
 #include "channel.h"
 #include "exec.h"
-#include "migration/migration.h"
+#include "migration.h"
 #include "io/channel-command.h"
 #include "trace.h"
 
diff --git a/migration/fd.c b/migration/fd.c
index 8a04dcd..b2384bf 100644
--- a/migration/fd.c
+++ b/migration/fd.c
@@ -19,7 +19,7 @@
 #include "qemu-common.h"
 #include "channel.h"
 #include "fd.h"
-#include "migration/migration.h"
+#include "migration.h"
 #include "monitor/monitor.h"
 #include "io/channel-util.h"
 #include "trace.h"
diff --git a/migration/migration.c b/migration/migration.c
index b6c0cfd..956e36c 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -25,7 +25,7 @@
 #include "ram.h"
 #include "migration/global_state.h"
 #include "migration/misc.h"
-#include "migration/migration.h"
+#include "migration.h"
 #include "savevm.h"
 #include "qemu-file-channel.h"
 #include "qemu-file.h"
diff --git a/include/migration/migration.h b/migration/migration.h
similarity index 100%
rename from include/migration/migration.h
rename to migration/migration.h
diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c
index 9c41887..c8c4500 100644
--- a/migration/postcopy-ram.c
+++ b/migration/postcopy-ram.c
@@ -20,7 +20,7 @@
 
 #include "qemu-common.h"
 #include "exec/target_page.h"
-#include "migration/migration.h"
+#include "migration.h"
 #include "qemu-file.h"
 #include "savevm.h"
 #include "postcopy-ram.h"
diff --git a/migration/qemu-file.c b/migration/qemu-file.c
index ab26f4e..e65c373 100644
--- a/migration/qemu-file.c
+++ b/migration/qemu-file.c
@@ -28,7 +28,7 @@
 #include "qemu/iov.h"
 #include "qemu/sockets.h"
 #include "qemu/coroutine.h"
-#include "migration/migration.h"
+#include "migration.h"
 #include "qemu-file.h"
 #include "trace.h"
 
diff --git a/migration/ram.c b/migration/ram.c
index 31b4d6a..fe307ce 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -37,7 +37,7 @@
 #include "qemu/main-loop.h"
 #include "xbzrle.h"
 #include "ram.h"
-#include "migration/migration.h"
+#include "migration.h"
 #include "migration/register.h"
 #include "migration/misc.h"
 #include "qemu-file.h"
diff --git a/migration/rdma.c b/migration/rdma.c
index e446c6f..c6bc607 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -18,7 +18,7 @@
 #include 

[Qemu-devel] [PATCH 07/11] migration: ram_control_* are implemented in qemu_file

[Juan Quintela]

Signed-off-by: Juan Quintela 
---
 include/migration/migration.h | 18 --
 migration/qemu-file.h | 17 +
 2 files changed, 17 insertions(+), 18 deletions(-)

diff --git a/include/migration/migration.h b/include/migration/migration.h
index 108212c..cb894b8 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -171,24 +171,6 @@ void migrate_send_rp_pong(MigrationIncomingState *mis,
 void migrate_send_rp_req_pages(MigrationIncomingState *mis, const char* rbname,
   ram_addr_t start, size_t len);
 
-void ram_control_before_iterate(QEMUFile *f, uint64_t flags);
-void ram_control_after_iterate(QEMUFile *f, uint64_t flags);
-void ram_control_load_hook(QEMUFile *f, uint64_t flags, void *data);
-
-/* Whenever this is found in the data stream, the flags
- * will be passed to ram_control_load_hook in the incoming-migration
- * side. This lets before_ram_iterate/after_ram_iterate add
- * transport-specific sections to the RAM migration data.
- */
-#define RAM_SAVE_FLAG_HOOK 0x80
-
-#define RAM_SAVE_CONTROL_NOT_SUPP -1000
-#define RAM_SAVE_CONTROL_DELAYED  -2000
-
-size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset,
- ram_addr_t offset, size_t size,
- uint64_t *bytes_sent);
-
 void savevm_skip_section_footers(void);
 void register_global_state(void);
 void global_state_set_optional(void);
diff --git a/migration/qemu-file.h b/migration/qemu-file.h
index 49fd697..aae4e5e 100644
--- a/migration/qemu-file.h
+++ b/migration/qemu-file.h
@@ -156,5 +156,22 @@ void qemu_file_set_blocking(QEMUFile *f, bool block);
 
 size_t qemu_get_counted_string(QEMUFile *f, char buf[256]);
 
+void ram_control_before_iterate(QEMUFile *f, uint64_t flags);
+void ram_control_after_iterate(QEMUFile *f, uint64_t flags);
+void ram_control_load_hook(QEMUFile *f, uint64_t flags, void *data);
+
+/* Whenever this is found in the data stream, the flags
+ * will be passed to ram_control_load_hook in the incoming-migration
+ * side. This lets before_ram_iterate/after_ram_iterate add
+ * transport-specific sections to the RAM migration data.
+ */
+#define RAM_SAVE_FLAG_HOOK 0x80
+
+#define RAM_SAVE_CONTROL_NOT_SUPP -1000
+#define RAM_SAVE_CONTROL_DELAYED  -2000
+
+size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset,
+ ram_addr_t offset, size_t size,
+ uint64_t *bytes_sent);
 
 #endif
-- 
2.9.4

[Qemu-devel] [PATCH 03/11] migration: Split registration functions from vmstate.h

[Juan Quintela]

They are indpendent, and nowadays almost every device register things
with qdev->vmsd.

Signed-off-by: Juan Quintela 
---
 hw/net/vmxnet3.c |  1 +
 hw/ppc/spapr.c   |  1 +
 hw/s390x/s390-skeys.c|  1 +
 hw/s390x/s390-virtio-ccw.c   |  1 +
 include/migration/register.h | 64 
 include/migration/vmstate.h  | 45 ---
 migration/block.c|  1 +
 migration/ram.c  |  1 +
 migration/savevm.c   |  1 +
 slirp/slirp.c|  1 +
 10 files changed, 72 insertions(+), 45 deletions(-)
 create mode 100644 include/migration/register.h

diff --git a/hw/net/vmxnet3.c b/hw/net/vmxnet3.c
index 8b1fab2..21ac646 100644
--- a/hw/net/vmxnet3.c
+++ b/hw/net/vmxnet3.c
@@ -26,6 +26,7 @@
 #include "qemu/bswap.h"
 #include "hw/pci/msix.h"
 #include "hw/pci/msi.h"
+#include "migration/register.h"
 
 #include "vmxnet3.h"
 #include "vmxnet_debug.h"
diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index ab3aab1..a44efbf 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -39,6 +39,7 @@
 #include "sysemu/hw_accel.h"
 #include "kvm_ppc.h"
 #include "migration/migration.h"
+#include "migration/register.h"
 #include "mmu-hash64.h"
 #include "mmu-book3s-v3.h"
 #include "qom/cpu.h"
diff --git a/hw/s390x/s390-skeys.c b/hw/s390x/s390-skeys.c
index 619152c..58f084a 100644
--- a/hw/s390x/s390-skeys.c
+++ b/hw/s390x/s390-skeys.c
@@ -15,6 +15,7 @@
 #include "hw/s390x/storage-keys.h"
 #include "qemu/error-report.h"
 #include "sysemu/kvm.h"
+#include "migration/register.h"
 
 #define S390_SKEYS_BUFFER_SIZE 131072  /* Room for 128k storage keys */
 #define S390_SKEYS_SAVE_FLAG_EOS 0x01
diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
index c9021f2..51d14d3 100644
--- a/hw/s390x/s390-virtio-ccw.c
+++ b/hw/s390x/s390-virtio-ccw.c
@@ -28,6 +28,7 @@
 #include "ipl.h"
 #include "hw/s390x/s390-virtio-ccw.h"
 #include "hw/s390x/css-bridge.h"
+#include "migration/register.h"
 
 static const char *const reset_dev_types[] = {
 TYPE_VIRTUAL_CSS_BRIDGE,
diff --git a/include/migration/register.h b/include/migration/register.h
new file mode 100644
index 000..844afaf
--- /dev/null
+++ b/include/migration/register.h
@@ -0,0 +1,64 @@
+/*
+ * QEMU migration vmstate registration
+ *
+ * Copyright IBM, Corp. 2008
+ *
+ * Authors:
+ *  Anthony Liguori   
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef MIGRATION_REGISTER_H
+#define MIGRATION_REGISTER_H
+
+typedef void SaveStateHandler(QEMUFile *f, void *opaque);
+typedef int LoadStateHandler(QEMUFile *f, void *opaque, int version_id);
+
+typedef struct SaveVMHandlers {
+/* This runs inside the iothread lock.  */
+SaveStateHandler *save_state;
+
+void (*cleanup)(void *opaque);
+int (*save_live_complete_postcopy)(QEMUFile *f, void *opaque);
+int (*save_live_complete_precopy)(QEMUFile *f, void *opaque);
+
+/* This runs both outside and inside the iothread lock.  */
+bool (*is_active)(void *opaque);
+
+/* This runs outside the iothread lock in the migration case, and
+ * within the lock in the savevm case.  The callback had better only
+ * use data that is local to the migration thread or protected
+ * by other locks.
+ */
+int (*save_live_iterate)(QEMUFile *f, void *opaque);
+
+/* This runs outside the iothread lock!  */
+int (*save_live_setup)(QEMUFile *f, void *opaque);
+void (*save_live_pending)(QEMUFile *f, void *opaque,
+  uint64_t threshold_size,
+  uint64_t *non_postcopiable_pending,
+  uint64_t *postcopiable_pending);
+LoadStateHandler *load_state;
+} SaveVMHandlers;
+
+int register_savevm(DeviceState *dev,
+const char *idstr,
+int instance_id,
+int version_id,
+SaveStateHandler *save_state,
+LoadStateHandler *load_state,
+void *opaque);
+
+int register_savevm_live(DeviceState *dev,
+ const char *idstr,
+ int instance_id,
+ int version_id,
+ SaveVMHandlers *ops,
+ void *opaque);
+
+void unregister_savevm(DeviceState *dev, const char *idstr, void *opaque);
+
+#endif
diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
index b95c9bb..6de554d 100644
--- a/include/migration/vmstate.h
+++ b/include/migration/vmstate.h
@@ -29,53 +29,8 @@
 
 #include "migration/qjson.h"
 
-typedef void SaveStateHandler(QEMUFile *f, void *opaque);
 typedef int LoadStateHandler(QEMUFile *f, void *opaque, int version_id);
 
-typedef struct SaveVMHandlers {
-/* This runs inside the iothread lock.  */
-SaveStateHandler *save_state;
-
-  

[Qemu-devel] [PATCH 11/11] migration: Remove unneeded includes

[Juan Quintela]

Signed-off-by: Juan Quintela 
---
 include/migration/colo.h  | 3 ---
 include/migration/misc.h  | 2 ++
 migration/block.c | 6 --
 migration/colo-failover.c | 2 ++
 migration/colo.c  | 2 --
 migration/exec.c  | 2 --
 migration/fd.c| 2 --
 migration/global_state.c  | 1 -
 migration/migration.c | 6 --
 migration/migration.h | 2 --
 migration/postcopy-ram.c  | 4 +---
 migration/qemu-file.c | 2 --
 migration/ram.c   | 4 
 migration/savevm.c| 6 --
 14 files changed, 5 insertions(+), 39 deletions(-)

diff --git a/include/migration/colo.h b/include/migration/colo.h
index ba0bb6e..be6beba 100644
--- a/include/migration/colo.h
+++ b/include/migration/colo.h
@@ -14,9 +14,6 @@
 #define QEMU_COLO_H
 
 #include "qemu-common.h"
-#include "qemu/coroutine_int.h"
-#include "qemu/thread.h"
-#include "qemu/main-loop.h"
 
 bool colo_supported(void);
 void colo_info_init(void);
diff --git a/include/migration/misc.h b/include/migration/misc.h
index d5a433a..65c7070 100644
--- a/include/migration/misc.h
+++ b/include/migration/misc.h
@@ -14,6 +14,8 @@
 #ifndef MIGRATION_MISC_H
 #define MIGRATION_MISC_H
 
+#include "qemu/notify.h"
+
 /* migration/ram.c */
 
 void ram_mig_init(void);
diff --git a/migration/block.c b/migration/block.c
index 21c7fac..14a8a84 100644
--- a/migration/block.c
+++ b/migration/block.c
@@ -15,19 +15,13 @@
 
 #include "qemu/osdep.h"
 #include "qapi/error.h"
-#include "qemu-common.h"
-#include "block/block.h"
 #include "qemu/error-report.h"
-#include "qemu/main-loop.h"
-#include "hw/hw.h"
 #include "qemu/cutils.h"
 #include "qemu/queue.h"
-#include "qemu/timer.h"
 #include "block.h"
 #include "migration/misc.h"
 #include "migration.h"
 #include "migration/register.h"
-#include "sysemu/blockdev.h"
 #include "qemu-file.h"
 #include "migration/vmstate.h"
 #include "sysemu/block-backend.h"
diff --git a/migration/colo-failover.c b/migration/colo-failover.c
index cc229f5..f991486 100644
--- a/migration/colo-failover.c
+++ b/migration/colo-failover.c
@@ -13,6 +13,8 @@
 #include "qemu/osdep.h"
 #include "migration/colo.h"
 #include "migration/failover.h"
+#include "qemu/main-loop.h"
+#include "migration.h"
 #include "qmp-commands.h"
 #include "qapi/qmp/qerror.h"
 #include "qemu/error-report.h"
diff --git a/migration/colo.c b/migration/colo.c
index 45e9b46..c436d63 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -11,7 +11,6 @@
  */
 
 #include "qemu/osdep.h"
-#include "qemu/timer.h"
 #include "sysemu/sysemu.h"
 #include "qemu-file-channel.h"
 #include "migration.h"
@@ -22,7 +21,6 @@
 #include "io/channel-buffer.h"
 #include "trace.h"
 #include "qemu/error-report.h"
-#include "qapi/error.h"
 #include "migration/failover.h"
 #include "replication.h"
 #include "qmp-commands.h"
diff --git a/migration/exec.c b/migration/exec.c
index b1de445..08b599e 100644
--- a/migration/exec.c
+++ b/migration/exec.c
@@ -19,10 +19,8 @@
 
 #include "qemu/osdep.h"
 #include "qapi/error.h"
-#include "qemu-common.h"
 #include "channel.h"
 #include "exec.h"
-#include "migration.h"
 #include "io/channel-command.h"
 #include "trace.h"
 
diff --git a/migration/fd.c b/migration/fd.c
index b2384bf..30f5258 100644
--- a/migration/fd.c
+++ b/migration/fd.c
@@ -16,10 +16,8 @@
 
 #include "qemu/osdep.h"
 #include "qapi/error.h"
-#include "qemu-common.h"
 #include "channel.h"
 #include "fd.h"
-#include "migration.h"
 #include "monitor/monitor.h"
 #include "io/channel-util.h"
 #include "trace.h"
diff --git a/migration/global_state.c b/migration/global_state.c
index 16ac63f..f792cf5 100644
--- a/migration/global_state.c
+++ b/migration/global_state.c
@@ -17,7 +17,6 @@
 #include "qapi/util.h"
 #include "migration/global_state.h"
 #include "migration/vmstate.h"
-#include "sysemu/sysemu.h"
 #include "trace.h"
 
 typedef struct {
diff --git a/migration/migration.c b/migration/migration.c
index 956e36c..60da9d6 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -16,7 +16,6 @@
 #include "qemu/osdep.h"
 #include "qemu/cutils.h"
 #include "qemu/error-report.h"
-#include "qemu/main-loop.h"
 #include "migration/blocker.h"
 #include "exec.h"
 #include "fd.h"
@@ -30,11 +29,9 @@
 #include "qemu-file-channel.h"
 #include "qemu-file.h"
 #include "migration/vmstate.h"
-#include "sysemu/sysemu.h"
 #include "block/block.h"
 #include "qapi/qmp/qerror.h"
 #include "qapi/util.h"
-#include "qemu/sockets.h"
 #include "qemu/rcu.h"
 #include "block.h"
 #include "postcopy-ram.h"
@@ -42,9 +39,6 @@
 #include "qmp-commands.h"
 #include "trace.h"
 #include "qapi-event.h"
-#include "qom/cpu.h"
-#include "exec/memory.h"
-#include "exec/address-spaces.h"
 #include "exec/target_page.h"
 #include "io/channel-buffer.h"
 #include "migration/colo.h"
diff --git a/migration/migration.h b/migration/migration.h
index 601e4ab..d9a268a 100644
--- a/migration/migration.h
+++ b/migration/migration.h
@@ -14,10 +14,8 @@
 #ifndef QEMU_MIGRATION_H
 #define QEMU_MIGRATION_H
 

[Qemu-devel] [PATCH 05/11] migration: Move constants to savevm.h

[Juan Quintela]

Signed-off-by: Juan Quintela 
---
 include/migration/migration.h | 15 ---
 migration/savevm.h| 15 +++
 migration/vmstate.c   |  1 +
 tests/test-vmstate.c  |  1 +
 4 files changed, 17 insertions(+), 15 deletions(-)

diff --git a/include/migration/migration.h b/include/migration/migration.h
index 79b5484..dd52d3c 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -22,21 +22,6 @@
 #include "exec/cpu-common.h"
 #include "qemu/coroutine_int.h"
 
-#define QEMU_VM_FILE_MAGIC   0x5145564d
-#define QEMU_VM_FILE_VERSION_COMPAT  0x0002
-#define QEMU_VM_FILE_VERSION 0x0003
-
-#define QEMU_VM_EOF  0x00
-#define QEMU_VM_SECTION_START0x01
-#define QEMU_VM_SECTION_PART 0x02
-#define QEMU_VM_SECTION_END  0x03
-#define QEMU_VM_SECTION_FULL 0x04
-#define QEMU_VM_SUBSECTION   0x05
-#define QEMU_VM_VMDESCRIPTION0x06
-#define QEMU_VM_CONFIGURATION0x07
-#define QEMU_VM_COMMAND  0x08
-#define QEMU_VM_SECTION_FOOTER   0x7e
-
 /* Messages sent on the return path from destination to source */
 enum mig_rp_message_type {
 MIG_RP_MSG_INVALID = 0,  /* Must be 0 */
diff --git a/migration/savevm.h b/migration/savevm.h
index eb44877..45b59c1 100644
--- a/migration/savevm.h
+++ b/migration/savevm.h
@@ -14,6 +14,21 @@
 #ifndef MIGRATION_SAVEVM_H
 #define MIGRATION_SAVEVM_H
 
+#define QEMU_VM_FILE_MAGIC   0x5145564d
+#define QEMU_VM_FILE_VERSION_COMPAT  0x0002
+#define QEMU_VM_FILE_VERSION 0x0003
+
+#define QEMU_VM_EOF  0x00
+#define QEMU_VM_SECTION_START0x01
+#define QEMU_VM_SECTION_PART 0x02
+#define QEMU_VM_SECTION_END  0x03
+#define QEMU_VM_SECTION_FULL 0x04
+#define QEMU_VM_SUBSECTION   0x05
+#define QEMU_VM_VMDESCRIPTION0x06
+#define QEMU_VM_CONFIGURATION0x07
+#define QEMU_VM_COMMAND  0x08
+#define QEMU_VM_SECTION_FOOTER   0x7e
+
 bool qemu_savevm_state_blocked(Error **errp);
 void qemu_savevm_state_begin(QEMUFile *f);
 void qemu_savevm_state_header(QEMUFile *f);
diff --git a/migration/vmstate.c b/migration/vmstate.c
index 51a19b6..377d951 100644
--- a/migration/vmstate.c
+++ b/migration/vmstate.c
@@ -14,6 +14,7 @@
 #include "qemu-common.h"
 #include "migration/migration.h"
 #include "migration/vmstate.h"
+#include "migration/savevm.h"
 #include "qemu-file.h"
 #include "qemu/bitops.h"
 #include "qemu/error-report.h"
diff --git a/tests/test-vmstate.c b/tests/test-vmstate.c
index c52aff9..f30433a 100644
--- a/tests/test-vmstate.c
+++ b/tests/test-vmstate.c
@@ -30,6 +30,7 @@
 #include "migration/qemu-file-types.h"
 #include "../migration/qemu-file.h"
 #include "../migration/qemu-file-channel.h"
+#include "../migration/savevm.h"
 #include "qemu/coroutine.h"
 #include "io/channel-file.h"
 
-- 
2.9.4

[Qemu-devel] [PATCH 06/11] migration: Commands are only used inside migration.c

[Juan Quintela]

So, move them there.  Notice that we export functions that send
commands, not the command themselves.

Signed-off-by: Juan Quintela 
---
 include/migration/migration.h | 15 --
 migration/migration.c | 46 +++
 2 files changed, 29 insertions(+), 32 deletions(-)

diff --git a/include/migration/migration.h b/include/migration/migration.h
index dd52d3c..108212c 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -22,18 +22,6 @@
 #include "exec/cpu-common.h"
 #include "qemu/coroutine_int.h"
 
-/* Messages sent on the return path from destination to source */
-enum mig_rp_message_type {
-MIG_RP_MSG_INVALID = 0,  /* Must be 0 */
-MIG_RP_MSG_SHUT, /* sibling will not send any more RP messages */
-MIG_RP_MSG_PONG, /* Response to a PING; data (seq: be32 ) */
-
-MIG_RP_MSG_REQ_PAGES_ID, /* data (start: be64, len: be32, id: string) */
-MIG_RP_MSG_REQ_PAGES,/* data (start: be64, len: be32) */
-
-MIG_RP_MSG_MAX
-};
-
 /* State for the incoming migration */
 struct MigrationIncomingState {
 QEMUFile *from_src_file;
@@ -176,9 +164,6 @@ int migrate_decompress_threads(void);
 bool migrate_use_events(void);
 
 /* Sending on the return path - generic and then for each message type */
-void migrate_send_rp_message(MigrationIncomingState *mis,
- enum mig_rp_message_type message_type,
- uint16_t len, void *data);
 void migrate_send_rp_shut(MigrationIncomingState *mis,
   uint32_t value);
 void migrate_send_rp_pong(MigrationIncomingState *mis,
diff --git a/migration/migration.c b/migration/migration.c
index 48c94c9..603d39d 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -86,6 +86,18 @@ static NotifierList migration_state_notifiers =
 
 static bool deferred_incoming;
 
+/* Messages sent on the return path from destination to source */
+enum mig_rp_message_type {
+MIG_RP_MSG_INVALID = 0,  /* Must be 0 */
+MIG_RP_MSG_SHUT, /* sibling will not send any more RP messages */
+MIG_RP_MSG_PONG, /* Response to a PING; data (seq: be32 ) */
+
+MIG_RP_MSG_REQ_PAGES_ID, /* data (start: be64, len: be32, id: string) */
+MIG_RP_MSG_REQ_PAGES,/* data (start: be64, len: be32) */
+
+MIG_RP_MSG_MAX
+};
+
 /* When we add fault tolerance, we could have several
migrations at once.  For now we don't need to add
dynamic creation of migration */
@@ -292,6 +304,23 @@ static void deferred_incoming_migration(Error **errp)
 deferred_incoming = true;
 }
 
+/*
+ * Send a message on the return channel back to the source
+ * of the migration.
+ */
+static void migrate_send_rp_message(MigrationIncomingState *mis,
+enum mig_rp_message_type message_type,
+uint16_t len, void *data)
+{
+trace_migrate_send_rp_message((int)message_type, len);
+qemu_mutex_lock(>rp_mutex);
+qemu_put_be16(mis->to_src_file, (unsigned int)message_type);
+qemu_put_be16(mis->to_src_file, len);
+qemu_put_buffer(mis->to_src_file, data, len);
+qemu_fflush(mis->to_src_file);
+qemu_mutex_unlock(>rp_mutex);
+}
+
 /* Request a range of pages from the source VM at the given
  * start address.
  *   rbname: Name of the RAMBlock to request the page in, if NULL it's the same
@@ -462,23 +491,6 @@ void migration_fd_process_incoming(QEMUFile *f)
 }
 
 /*
- * Send a message on the return channel back to the source
- * of the migration.
- */
-void migrate_send_rp_message(MigrationIncomingState *mis,
- enum mig_rp_message_type message_type,
- uint16_t len, void *data)
-{
-trace_migrate_send_rp_message((int)message_type, len);
-qemu_mutex_lock(>rp_mutex);
-qemu_put_be16(mis->to_src_file, (unsigned int)message_type);
-qemu_put_be16(mis->to_src_file, len);
-qemu_put_buffer(mis->to_src_file, data, len);
-qemu_fflush(mis->to_src_file);
-qemu_mutex_unlock(>rp_mutex);
-}
-
-/*
  * Send a 'SHUT' message on the return channel with the given value
  * to indicate that we've finished with the RP.  Non-0 value indicates
  * error.
-- 
2.9.4

[Qemu-devel] [PATCH 02/11] migration: Move self_announce_delay() to misc.h

[Juan Quintela]

Signed-off-by: Juan Quintela 
---
 hw/net/virtio-net.c |  1 +
 include/migration/misc.h| 10 ++
 include/migration/vmstate.h | 10 --
 migration/savevm.c  |  1 +
 4 files changed, 12 insertions(+), 10 deletions(-)

diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index 9a3d769..91eddaf 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -25,6 +25,7 @@
 #include "qapi/qmp/qjson.h"
 #include "qapi-event.h"
 #include "hw/virtio/virtio-access.h"
+#include "migration/misc.h"
 
 #define VIRTIO_NET_VM_VERSION11
 
diff --git a/include/migration/misc.h b/include/migration/misc.h
index d7892b7..60486d4 100644
--- a/include/migration/misc.h
+++ b/include/migration/misc.h
@@ -26,4 +26,14 @@ void blk_mig_init(void);
 static inline void blk_mig_init(void) {}
 #endif
 
+#define SELF_ANNOUNCE_ROUNDS 5
+
+static inline
+int64_t self_announce_delay(int round)
+{
+assert(round < SELF_ANNOUNCE_ROUNDS && round > 0);
+/* delay 50ms, 150ms, 250ms, ... */
+return 50 + (SELF_ANNOUNCE_ROUNDS - round - 1) * 100;
+}
+
 #endif
diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
index 6689562..b95c9bb 100644
--- a/include/migration/vmstate.h
+++ b/include/migration/vmstate.h
@@ -1018,8 +1018,6 @@ extern const VMStateInfo vmstate_info_qtailq;
 #define VMSTATE_END_OF_LIST() \
 {}
 
-#define SELF_ANNOUNCE_ROUNDS 5
-
 int vmstate_load_state(QEMUFile *f, const VMStateDescription *vmsd,
void *opaque, int version_id);
 void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
@@ -1051,14 +1049,6 @@ void vmstate_register_ram(struct MemoryRegion *memory, 
DeviceState *dev);
 void vmstate_unregister_ram(struct MemoryRegion *memory, DeviceState *dev);
 void vmstate_register_ram_global(struct MemoryRegion *memory);
 
-static inline
-int64_t self_announce_delay(int round)
-{
-assert(round < SELF_ANNOUNCE_ROUNDS && round > 0);
-/* delay 50ms, 150ms, 250ms, ... */
-return 50 + (SELF_ANNOUNCE_ROUNDS - round - 1) * 100;
-}
-
 void dump_vmstate_json_to_file(FILE *out_fp);
 
 bool vmstate_check_only_migratable(const VMStateDescription *vmsd);
diff --git a/migration/savevm.c b/migration/savevm.c
index 9c320f5..d683877 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -36,6 +36,7 @@
 #include "qemu/timer.h"
 #include "migration/migration.h"
 #include "migration/snapshot.h"
+#include "migration/misc.h"
 #include "ram.h"
 #include "qemu-file-channel.h"
 #include "qemu-file.h"
-- 
2.9.4

[Qemu-devel] [PATCH 08/11] migration: create global_state.c

[Juan Quintela]

It don't belong anywhere else, just the global state where everybody
can stick other things.

Signed-off-by: Juan Quintela 
---
 hw/i386/pc_piix.c|   1 +
 hw/ppc/spapr.c   |   1 +
 hw/xen/xen-common.c  |   1 +
 include/migration/global_state.h |  25 +++
 include/migration/migration.h|   4 --
 migration/Makefile.objs  |   2 +-
 migration/global_state.c | 140 +++
 migration/migration.c| 121 +
 migration/savevm.c   |   1 +
 vl.c |   1 +
 10 files changed, 172 insertions(+), 125 deletions(-)
 create mode 100644 include/migration/global_state.h
 create mode 100644 migration/global_state.c

diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
index 2234bd0..dc19d96 100644
--- a/hw/i386/pc_piix.c
+++ b/hw/i386/pc_piix.c
@@ -52,6 +52,7 @@
 #include 
 #include "hw/xen/xen_pt.h"
 #endif
+#include "migration/global_state.h"
 #include "migration/migration.h"
 #include "kvm_i386.h"
 #include "sysemu/numa.h"
diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index a44efbf..ac8a317 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -39,6 +39,7 @@
 #include "sysemu/hw_accel.h"
 #include "kvm_ppc.h"
 #include "migration/migration.h"
+#include "migration/global_state.h"
 #include "migration/register.h"
 #include "mmu-hash64.h"
 #include "mmu-book3s-v3.h"
diff --git a/hw/xen/xen-common.c b/hw/xen/xen-common.c
index a9055e9..e265445 100644
--- a/hw/xen/xen-common.c
+++ b/hw/xen/xen-common.c
@@ -14,6 +14,7 @@
 #include "sysemu/char.h"
 #include "sysemu/accel.h"
 #include "migration/migration.h"
+#include "migration/global_state.h"
 
 //#define DEBUG_XEN
 
diff --git a/include/migration/global_state.h b/include/migration/global_state.h
new file mode 100644
index 000..90faea7
--- /dev/null
+++ b/include/migration/global_state.h
@@ -0,0 +1,25 @@
+/*
+ * Global State configuration
+ *
+ * Copyright (c) 2014-2017 Red Hat Inc
+ *
+ * Authors:
+ *  Juan Quintela 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#ifndef QEMU_MIGRATION_GLOBAL_STATE_H
+#define QEMU_MIGRATION_GLOBAL_STATE_H
+
+#include "sysemu/sysemu.h"
+
+void register_global_state(void);
+void global_state_set_optional(void);
+int global_state_store(void);
+void global_state_store_running(void);
+bool global_state_received(void);
+RunState global_state_get_runstate(void);
+
+#endif
diff --git a/include/migration/migration.h b/include/migration/migration.h
index cb894b8..5050454 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -172,10 +172,6 @@ void migrate_send_rp_req_pages(MigrationIncomingState 
*mis, const char* rbname,
   ram_addr_t start, size_t len);
 
 void savevm_skip_section_footers(void);
-void register_global_state(void);
-void global_state_set_optional(void);
 void savevm_skip_configuration(void);
-int global_state_store(void);
-void global_state_store_running(void);
 
 #endif
diff --git a/migration/Makefile.objs b/migration/Makefile.objs
index 90f8c1f..1c7770d 100644
--- a/migration/Makefile.objs
+++ b/migration/Makefile.objs
@@ -2,7 +2,7 @@ common-obj-y += migration.o socket.o fd.o exec.o
 common-obj-y += tls.o channel.o savevm.o
 common-obj-y += colo-comm.o colo.o colo-failover.o
 common-obj-y += vmstate.o vmstate-types.o page_cache.o
-common-obj-y += qemu-file.o
+common-obj-y += qemu-file.o global_state.o
 common-obj-y += qemu-file-channel.o
 common-obj-y += xbzrle.o postcopy-ram.o
 common-obj-y += qjson.o
diff --git a/migration/global_state.c b/migration/global_state.c
new file mode 100644
index 000..16ac63f
--- /dev/null
+++ b/migration/global_state.c
@@ -0,0 +1,140 @@
+/*
+ * Global State configuration
+ *
+ * Copyright (c) 2014-2017 Red Hat Inc
+ *
+ * Authors:
+ *  Juan Quintela 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/cutils.h"
+#include "qemu/error-report.h"
+#include "qapi/error.h"
+#include "qapi/util.h"
+#include "migration/global_state.h"
+#include "migration/vmstate.h"
+#include "sysemu/sysemu.h"
+#include "trace.h"
+
+typedef struct {
+bool optional;
+uint32_t size;
+uint8_t runstate[100];
+RunState state;
+bool received;
+} GlobalState;
+
+static GlobalState global_state;
+
+int global_state_store(void)
+{
+if (!runstate_store((char *)global_state.runstate,
+sizeof(global_state.runstate))) {
+error_report("runstate name too big: %s", global_state.runstate);
+trace_migrate_state_too_big();
+return -EINVAL;
+}
+return 0;
+}
+
+void global_state_store_running(void)
+{
+const char *state = 

[Qemu-devel] [PATCH 04/11] migration: Move dump_vmsate_json_to_file() to misc.h

[Juan Quintela]

It was not from vmstate.c to start with.

Signed-off-by: Juan Quintela 
---
 include/migration/misc.h| 4 
 include/migration/vmstate.h | 2 --
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/include/migration/misc.h b/include/migration/misc.h
index 60486d4..026b561 100644
--- a/include/migration/misc.h
+++ b/include/migration/misc.h
@@ -36,4 +36,8 @@ int64_t self_announce_delay(int round)
 return 50 + (SELF_ANNOUNCE_ROUNDS - round - 1) * 100;
 }
 
+/* migration/savevm.c */
+
+void dump_vmstate_json_to_file(FILE *out_fp);
+
 #endif
diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
index 6de554d..ad243d4 100644
--- a/include/migration/vmstate.h
+++ b/include/migration/vmstate.h
@@ -1004,8 +1004,6 @@ void vmstate_register_ram(struct MemoryRegion *memory, 
DeviceState *dev);
 void vmstate_unregister_ram(struct MemoryRegion *memory, DeviceState *dev);
 void vmstate_register_ram_global(struct MemoryRegion *memory);
 
-void dump_vmstate_json_to_file(FILE *out_fp);
-
 bool vmstate_check_only_migratable(const VMStateDescription *vmsd);
 
 #endif
-- 
2.9.4

[Qemu-devel] [PATCH 00/11] Misc migration cleanups

[Juan Quintela]

Hi

This are a bit of everything:
- Remove unneeded argumets for migration_channel_incomming
- Lots of changes to make migration.h local to only migration
  * Move self_annonce_delay() to misc.h
I know this conflicts for announce changes from Vlad, but
I want to remove migration.h from being exported.
  * split registrantion functions to register.h
  * Almost everything uses vmsd's for registration
  * Move constants to the places that use it
And they are not used outside of migration/*
  * create global_state.c, as they don't belong anywhere else

ToSend:
  * RAMState is a dynamic variable on my tree
  * save_{setup,cleanup} and load_{setup,cleanup}
Yes Kevin, that is for block layer
  * move all ram.c to use load_setup/cleanup

ToDo easy: (probably post 2.10)
  * split qapi functions from migration.c
They don't belong there, and it would be clearer about what is configuratio 
and what is code
  * block.c and page_cache.c still use DPRINTF, they should move to use tracing
this is easy, volunteers, please.
  * Now that it is clear what functions are exported and which not, writting 
documentation could
be a good idea for them.

ToDo, difficult:
  * rdma.c -> this needs some love, it uses a completely different set of hooks 
that everything
else, should have to integrate somehow everything together.
  * abstract compression, xbzrle, postcopy and rdma into something that
is easier to understand.

Please, review.

Thanks, Juan.


Juan Quintela (11):
  migration: Remove MigrationState from migration_channel_incomming()
  migration: Move self_announce_delay() to misc.h
  migration: Split registration functions from vmstate.h
  migration: Move dump_vmsate_json_to_file() to misc.h
  migration: Move constants to savevm.h
  migration: Commands are only used inside migration.c
  migration: ram_control_* are implemented in qemu_file
  migration: create global_state.c
  migration: Move remaining exported functions to migration/misc.h
  migration: Move migration.h to migration/
  migration: Remove unneeded includes

 hw/i386/pc_piix.c|   3 +-
 hw/net/virtio-net.c  |   1 +
 hw/net/vmxnet3.c |   1 +
 hw/ppc/spapr.c   |   4 +-
 hw/s390x/s390-skeys.c|   1 +
 hw/s390x/s390-virtio-ccw.c   |   1 +
 hw/xen/xen-common.c  |   3 +-
 include/migration/colo.h |   3 -
 include/migration/global_state.h |  25 
 include/migration/misc.h |  28 +
 include/migration/register.h |  64 ++
 include/migration/vmstate.h  |  57 -
 migration/Makefile.objs  |   2 +-
 migration/block.c|   9 +-
 migration/channel.c  |   7 +-
 migration/channel.h  |   3 +-
 migration/colo-comm.c|   2 +-
 migration/colo-failover.c|   2 +
 migration/colo.c |   4 +-
 migration/exec.c |   4 +-
 migration/fd.c   |   4 +-
 migration/global_state.c | 139 +
 migration/migration.c| 176 +--
 {include/migration => migration}/migration.h |  67 --
 migration/postcopy-ram.c |   6 +-
 migration/qemu-file.c|   4 +-
 migration/qemu-file.h|  17 +++
 migration/ram.c  |   7 +-
 migration/rdma.c |   2 +-
 migration/savevm.c   |  11 +-
 migration/savevm.h   |  15 +++
 migration/socket.c   |   5 +-
 migration/tls.c  |   4 +-
 migration/vmstate-types.c|   2 +-
 migration/vmstate.c  |   3 +-
 qdev-monitor.c   |   2 +-
 slirp/slirp.c|   1 +
 tests/test-vmstate.c |   3 +-
 ui/spice-core.c  |   2 +-
 vl.c |   2 +-
 40 files changed, 368 insertions(+), 328 deletions(-)
 create mode 100644 include/migration/global_state.h
 create mode 100644 include/migration/register.h
 create mode 100644 migration/global_state.c
 rename {include/migration => migration}/migration.h (62%)

-- 
2.9.4

[Qemu-devel] [PATCH 01/11] migration: Remove MigrationState from migration_channel_incomming()

[Juan Quintela]

All callers were calling migrate_get_current(), so do it inside the function.

Signed-off-by: Juan Quintela 
---
 migration/channel.c | 5 +++--
 migration/channel.h | 3 +--
 migration/exec.c| 2 +-
 migration/fd.c  | 2 +-
 migration/socket.c  | 3 +--
 migration/tls.c | 2 +-
 6 files changed, 8 insertions(+), 9 deletions(-)

diff --git a/migration/channel.c b/migration/channel.c
index eae1d9e..92f6f99 100644
--- a/migration/channel.c
+++ b/migration/channel.c
@@ -19,9 +19,10 @@
 #include "qapi/error.h"
 #include "io/channel-tls.h"
 
-void migration_channel_process_incoming(MigrationState *s,
-QIOChannel *ioc)
+void migration_channel_process_incoming(QIOChannel *ioc)
 {
+MigrationState *s = migrate_get_current();
+
 trace_migration_set_incoming_channel(
 ioc, object_get_typename(OBJECT(ioc)));
 
diff --git a/migration/channel.h b/migration/channel.h
index 2e0a7e3..e4b4057 100644
--- a/migration/channel.h
+++ b/migration/channel.h
@@ -18,8 +18,7 @@
 
 #include "io/channel.h"
 
-void migration_channel_process_incoming(MigrationState *s,
-QIOChannel *ioc);
+void migration_channel_process_incoming(QIOChannel *ioc);
 
 void migration_channel_connect(MigrationState *s,
QIOChannel *ioc,
diff --git a/migration/exec.c b/migration/exec.c
index 9077024..fc78eeb 100644
--- a/migration/exec.c
+++ b/migration/exec.c
@@ -49,7 +49,7 @@ static gboolean exec_accept_incoming_migration(QIOChannel 
*ioc,
GIOCondition condition,
gpointer opaque)
 {
-migration_channel_process_incoming(migrate_get_current(), ioc);
+migration_channel_process_incoming(ioc);
 object_unref(OBJECT(ioc));
 return FALSE; /* unregister */
 }
diff --git a/migration/fd.c b/migration/fd.c
index 0077a50..8a04dcd 100644
--- a/migration/fd.c
+++ b/migration/fd.c
@@ -49,7 +49,7 @@ static gboolean fd_accept_incoming_migration(QIOChannel *ioc,
  GIOCondition condition,
  gpointer opaque)
 {
-migration_channel_process_incoming(migrate_get_current(), ioc);
+migration_channel_process_incoming(ioc);
 object_unref(OBJECT(ioc));
 return FALSE; /* unregister */
 }
diff --git a/migration/socket.c b/migration/socket.c
index 85bfdcc..50dc8d8 100644
--- a/migration/socket.c
+++ b/migration/socket.c
@@ -148,8 +148,7 @@ static gboolean socket_accept_incoming_migration(QIOChannel 
*ioc,
 trace_migration_socket_incoming_accepted();
 
 qio_channel_set_name(QIO_CHANNEL(sioc), "migration-socket-incoming");
-migration_channel_process_incoming(migrate_get_current(),
-   QIO_CHANNEL(sioc));
+migration_channel_process_incoming(QIO_CHANNEL(sioc));
 object_unref(OBJECT(sioc));
 
 out:
diff --git a/migration/tls.c b/migration/tls.c
index bae9aca..d3abd6e 100644
--- a/migration/tls.c
+++ b/migration/tls.c
@@ -74,7 +74,7 @@ static void migration_tls_incoming_handshake(QIOTask *task,
 error_report_err(err);
 } else {
 trace_migration_tls_incoming_handshake_complete();
-migration_channel_process_incoming(migrate_get_current(), ioc);
+migration_channel_process_incoming(ioc);
 }
 object_unref(OBJECT(ioc));
 }
-- 
2.9.4

Re: [Qemu-devel] [RFC 3/6] vfio: Setup IGD stolen memory

[Alex Williamson]

On Tue, 30 May 2017 01:30:33 +0800
Zhi Wang  wrote:

> We still keep using VM dedicated memory for isolation to support IGD
> stolen in the guest. Becuase of the PA of the stolen memory can not be
> moved after the system is powered-up, we wish the PA of the guest stolen
> memory can sit in the same PA of host. A new memory region is allocated,
> and the memory region will be marked as reserved in guest E820 table.
> 
> We don't need to take care of GGMS, as the accesses to GGMS from HW bypass
> IOMMU.

:-O  So the device has access to a reserved region of host memory
regardless of the IOMMU.  Should the kernel be doing something to
save/restore that area or scrub that region before we get access to the
device?

 
> Suggested-by: Xiong Zhang 
> Signed-off-by: Zhi Wang 
> ---
>  hw/vfio/pci-quirks.c | 83 
> ++--
>  1 file changed, 29 insertions(+), 54 deletions(-)
> 
> diff --git a/hw/vfio/pci-quirks.c b/hw/vfio/pci-quirks.c
> index e0a0c13..5a083c1 100644
> --- a/hw/vfio/pci-quirks.c
> +++ b/hw/vfio/pci-quirks.c
> @@ -18,6 +18,7 @@
>  #include "pci.h"
>  #include "trace.h"
>  #include "intel-platform.h"
> +#include "hw/i386/pc.h"
>  
>  /* Use uin32_t for vendor & device so PCI_ANY_ID expands and cannot match hw 
> */
>  static bool vfio_pci_is(VFIOPCIDevice *vdev, uint32_t vendor, uint32_t 
> device)
> @@ -1362,9 +1363,10 @@ static void vfio_probe_igd_bar4_quirk(VFIOPCIDevice 
> *vdev, int nr)
>  VFIOQuirk *quirk;
>  VFIOIGDQuirk *igd;
>  const struct intel_device_info *info;
> +void *stolen;
>  PCIDevice *lpc_bridge;
> -int i, ret, ggms_mb, gms_mb = 0, gen;
> -uint64_t *bdsm_size;
> +int i, ret;
> +uint64_t bdsm_size;
>  uint32_t gmch;
>  uint16_t cmd_orig, cmd;
>  Error *err = NULL;
> @@ -1393,16 +1395,38 @@ static void vfio_probe_igd_bar4_quirk(VFIOPCIDevice 
> *vdev, int nr)
>  return;
>  }
>  
> -gen = info->gen;
> -
>  /* Setup our quirk to munge GTT addresses to the VM allocated buffer */
>  quirk = g_malloc0(sizeof(*quirk));
> +quirk->mem = g_new0(MemoryRegion, 1);
> +quirk->nr_mem = 1;
> +
>  igd = quirk->data = g_malloc0(sizeof(*igd));
>  igd->vdev = vdev;
>  igd->index = ~0;
>  igd->bdsm = vfio_pci_read_config(>pdev, IGD_BDSM, 4);
>  igd->bdsm &= ~((1 << 20) - 1); /* 1MB aligned */
>  
> +/* Setup stolen memory for IGD device. */
> +gmch = vfio_pci_read_config(>pdev, IGD_GMCH, 4);
> +bdsm_size = info->get_stolen_size(gmch);
> +
> +stolen = qemu_memalign(bdsm_size, bdsm_size);

This only needs to be 1MB aligned, not naturally aligned, right?

> +
> +memory_region_init_ram_ptr(>mem[0], OBJECT(vdev),
> +   "vfio-igd-stolen", bdsm_size, stolen);
> +memory_region_add_subregion_overlap(get_system_memory(),
> +igd->bdsm, >mem[0], 1);

We discussed off-list that maybe it's an acceptable solution to waste
VM memory for stolen memory, ie. let QEMU allocate the buffer and map
it into the VM at the same address as the host.  But I'm not really
sure what problem doing that here is solving other than we don't yet
expose IGD stolen memory as a device specific region on the vfio
device.  Is that plan to add that device specific region and do an mmap
of it rather than the above memalign when available?  That way we're
only wasting the memory we're overlapping, which may not even be
allocated yet.

There are also some hotplug issues here.  A) We cannot do this for a
hot-added device, there's a test later in the code for disabling legacy
mode for hot-added devices.  B) Is it possible to do cleanup on
hot-remove or do we need to disable the ability to hot-remove IGD
devices?

> +
> +e820_add_entry(igd->bdsm, bdsm_size, E820_RESERVED);
> +
> +/* GMCH is read-only, emulated */
> +pci_set_long(vdev->pdev.wmask + IGD_GMCH, 0);
> +pci_set_long(vdev->emulated_config_bits + IGD_GMCH, ~0);
> +
> +/* BDSM is read-only, emulated */
> +pci_set_long(vdev->pdev.wmask + IGD_BDSM, 0);
> +pci_set_long(vdev->emulated_config_bits + IGD_BDSM, ~0);
> +
>  /*
>   * We need to create an LPC/ISA bridge at PCI bus address 00:1f.0 that we
>   * can stuff host values into, so if there's already one there and it's 
> not
> @@ -1472,8 +1496,6 @@ static void vfio_probe_igd_bar4_quirk(VFIOPCIDevice 
> *vdev, int nr)
>  goto out;
>  }
>  
> -gmch = vfio_pci_read_config(>pdev, IGD_GMCH, 4);
> -
>  /*
>   * If IGD VGA Disable is clear (expected) and VGA is not already enabled,
>   * try to enable it.  Probably shouldn't be using legacy mode without 
> VGA,
> @@ -1528,53 +1550,6 @@ static void vfio_probe_igd_bar4_quirk(VFIOPCIDevice 
> *vdev, int nr)
>  
>  QLIST_INSERT_HEAD(>bars[nr].quirks, quirk, next);
>  
> -/* Determine the size of stolen memory needed for GTT */

Re: [Qemu-devel] [RFC PATCH] OvmfPkg/AcpiPlatformDxe: lift 4 GB alloc limit for modern ACPI systems

[Laszlo Ersek]

On 06/01/17 14:25, Laszlo Ersek wrote:

> In QEMU, we could tie both of these extensions to new machine types.
> 
> The result would be:
> 
>   firmware  QEMU  QEMU machine type  result
>       -  ---
>   old   new   oldallocate blobs under 4GB
>   old   new   newbreakage, but that's OK, we can
>require refreshed firmware for
>new machine types
>   new   old   oldallocate blobs under 4GB
>   new   new   oldallocate blobs under 4GB
>   new   new   newallocate blobs from 64-bit space

I think the situation is easier than this. We don't have to tie the
extensions to machine types.

The reason is that old firmware is allowed to fail on new QEMU
(regardless of machine type). Example: the WRITE_POINTER command,
originally introduced for VMGENID. If you run a SeaBIOS binary without
WRITE_POINTER support, in a QEMU VM with "-device vmgenid", the device
will not work. And QEMU doesn't try to prevent that by binding vmgenid
to machine types. Instead, QEMU bundled a SeaBIOS binary with
WRITE_POINTER support, for the release that introduced VMGENID.

(There's no reason for not bundling OVMF and ArmVirtQemu binaries with
QEMU releases now. Gerd already has a build service up and running, at
.)

The scenario that we *should* avoid is new firmware failing on old QEMU.
And this patch is actually that case, because the new fw would allocate
blobs with such 8-byte addresses that might not fit into 32-bit blob
fields. So, the extensions are necessary, but tying them to machine
types isn't.

  firmware  QEMU  result
      --
  old   new   breakage, but that's OK; we can require refreshed
firmware for new QEMU releases
  new   old   allocate blobs under 4GB (alloc zone extension is
necessary)
  new   new   allocate blobs from any address range

Thanks
Laszlo

Re: [Qemu-devel] [RFC 2/6] vfio: Setup IGD quirks earlier

[Alex Williamson]

On Tue, 30 May 2017 01:30:32 +0800
Zhi Wang  wrote:

> Initialize IGD quirks a bit earlier since we're going to support IGD
> stolen memory under both primary mode (legacy mode) and secondary mode
> and we need one extra memory region from IGD quirks.
> 
> Suggested-by: Xiong Zhang 
> Signed-off-by: Zhi Wang 
> ---
>  hw/vfio/pci-quirks.c | 30 +-
>  1 file changed, 17 insertions(+), 13 deletions(-)
> 
> diff --git a/hw/vfio/pci-quirks.c b/hw/vfio/pci-quirks.c
> index 71360ef..e0a0c13 100644
> --- a/hw/vfio/pci-quirks.c
> +++ b/hw/vfio/pci-quirks.c
> @@ -1395,6 +1395,14 @@ static void vfio_probe_igd_bar4_quirk(VFIOPCIDevice 
> *vdev, int nr)
>  
>  gen = info->gen;
>  
> +/* Setup our quirk to munge GTT addresses to the VM allocated buffer */
> +quirk = g_malloc0(sizeof(*quirk));
> +igd = quirk->data = g_malloc0(sizeof(*igd));
> +igd->vdev = vdev;
> +igd->index = ~0;
> +igd->bdsm = vfio_pci_read_config(>pdev, IGD_BDSM, 4);
> +igd->bdsm &= ~((1 << 20) - 1); /* 1MB aligned */
> +

This patch does not stand on its own, we alloc memory and setup a few
things earlier, but there are numerous returns before we get to the
next chunk below.  So the most obvious result of this patch alone is
that it introduces a memory leak.  Thanks,

Alex

>  /*
>   * We need to create an LPC/ISA bridge at PCI bus address 00:1f.0 that we
>   * can stuff host values into, so if there's already one there and it's 
> not
> @@ -1502,23 +1510,19 @@ static void vfio_probe_igd_bar4_quirk(VFIOPCIDevice 
> *vdev, int nr)
>  goto out;
>  }
>  
> -/* Setup our quirk to munge GTT addresses to the VM allocated buffer */
> -quirk = g_malloc0(sizeof(*quirk));
> -quirk->mem = g_new0(MemoryRegion, 2);
> -quirk->nr_mem = 2;
> -igd = quirk->data = g_malloc0(sizeof(*igd));
> -igd->vdev = vdev;
> -igd->index = ~0;
> -igd->bdsm = vfio_pci_read_config(>pdev, IGD_BDSM, 4);
> -igd->bdsm &= ~((1 << 20) - 1); /* 1MB aligned */
> +quirk->mem = g_renew(MemoryRegion, quirk->mem, 2);
> +
> +memory_region_init_io(>mem[quirk->nr_mem++], OBJECT(vdev),
> +  _igd_index_quirk, igd, "vfio-igd-index-quirk",
> +  4);
>  
> -memory_region_init_io(>mem[0], OBJECT(vdev), 
> _igd_index_quirk,
> -  igd, "vfio-igd-index-quirk", 4);
>  memory_region_add_subregion_overlap(vdev->bars[nr].region.mem,
>  0, >mem[0], 1);
>  
> -memory_region_init_io(>mem[1], OBJECT(vdev), _igd_data_quirk,
> -  igd, "vfio-igd-data-quirk", 4);
> +memory_region_init_io(>mem[quirk->nr_mem++], OBJECT(vdev),
> +  _igd_data_quirk, igd, "vfio-igd-data-quirk",
> +  4);
> +
>  memory_region_add_subregion_overlap(vdev->bars[nr].region.mem,
>  4, >mem[1], 1);
>  

Re: [Qemu-devel] [RFC 1/6] vfio: Add Intel platform definitions

[Alex Williamson]

On Tue, 30 May 2017 01:30:38 +0800
Zhi Wang  wrote:

> This patch introduces device descriptions for Intel platforms. Most of
> the Intel device definitions come from i915.
> 
> Suggested-by: Xiong Zhang 
> Signed-off-by: Zhi Wang 
> ---
>  hw/vfio/Makefile.objs|   2 +-
>  hw/vfio/intel-platform.c | 366 
> +++
>  hw/vfio/intel-platform.h |  38 +
>  hw/vfio/pci-quirks.c |  28 ++--
>  4 files changed, 421 insertions(+), 13 deletions(-)
>  create mode 100644 hw/vfio/intel-platform.c
>  create mode 100644 hw/vfio/intel-platform.h

Thanks to IGD hardware designers seeing fit to change the
implementation on a whim, this has a non-trivial future maintenance
burden.  Is Intel signing up to support this?  I would suggest being
overly explicit with where each define and function lives in the Linux
kernel so that it doesn't take a great deal of research to lift new
defines here.  Extra points if we could simply include Linux kernel
headers in a way to pick them up automatically.

Along those same lines, there's a subtle behavior change here where
igd_gen() hopes that Intel is eventually converging on a stable layout
and therefore assumes newer devices are compatible with the latest
version we know about.  That's removed here, so it seems we'll always
be trailing the latest hardware.  The comment:

> +error_report("IGD device %s is unsupported in legacy mode, "
> + "try SandyBridge or newer", vdev->vbasedev.name);

is also really no longer accurate.  igd_gen() had code to match older
hardware as specifically unsupported vs only unknown (and assumed to be
supported using the newest generation we know about).  Now the unknown
device could be new or old, so suggesting SandyBridge or newer isn't
helpful advice for a user.

Why "intel-platform"?  Maybe this is subtly trying to indicate that IGD
is really a broken mix of PCI and "platform" devices since it's PCI yet
relies on various non-PCI resources, like stolen memory.  In any case,
it's a bit confusing from a vfio perspective.  Perhaps pci-igd?  Thanks,

Alex

Re: [Qemu-devel] [PATCH v3 30/30] target/s390x: update maximum TCG model to z800

[David Hildenbrand]

On 01.06.2017 21:17, Aurelien Jarno wrote:
> On 2017-06-01 10:38, David Hildenbrand wrote:
>> On 01.06.2017 00:01, Aurelien Jarno wrote:
>>> At the same time fix the TCG version of get_max_cpu_model to return the
>>> maximum model like on KVM. Remove the ETF2 and long-displacement
>>
>> I don't understand the part
>> "fix the TCG version of get_max_cpu_model to return the maximum model
>> like on KVM".
>>
>> Can you elaborate?
> 
> Currently get_max_cpu_model returns the features of the base model, so
> for example the one of a z900 even on a z800. This makes impossible to
> enable the features that are provided by a z800 like etf2 or ldisp.
> 

Right, you can always change the max_cpu_model, e.g. bumping up the
version or adding new features, that is just fine.

> For what I understand from the KVM code (but I haven't tested), the
> function return all the features that are supported by the current CPU,
> not all the features that are supported by the base model of the current
> CPU.

Correct, for KVM it is the detected model, that means: Base features +
optional features.


> 
> 
>>> facilities from the additional features as it is included in the z800.
>>>
>>> Signed-off-by: Aurelien Jarno 
>>> ---
>>>  target/s390x/cpu_models.c | 13 ++---
>>>  1 file changed, 6 insertions(+), 7 deletions(-)
>>>
>>> diff --git a/target/s390x/cpu_models.c b/target/s390x/cpu_models.c
>>> index fc3cb25cc3..c13bbd852c 100644
>>> --- a/target/s390x/cpu_models.c
>>> +++ b/target/s390x/cpu_models.c
>>> @@ -668,8 +668,6 @@ static void add_qemu_cpu_model_features(S390FeatBitmap 
>>> fbm)
>>>  static const int feats[] = {
>>>  S390_FEAT_STFLE,
>>>  S390_FEAT_EXTENDED_IMMEDIATE,
>>> -S390_FEAT_EXTENDED_TRANSLATION_2,
>>> -S390_FEAT_LONG_DISPLACEMENT,
>>>  S390_FEAT_LONG_DISPLACEMENT_FAST,
>>>  S390_FEAT_ETF2_ENH,
>>>  S390_FEAT_STORE_CLOCK_FAST,
>>> @@ -696,9 +694,9 @@ static S390CPUModel *get_max_cpu_model(Error **errp)
>>>  if (kvm_enabled()) {
>>>  kvm_s390_get_host_cpu_model(_model, errp);
>>>  } else {
>>> -/* TCG emulates a z900 (with some optional additional features) */
>>> -max_model.def = _cpu_defs[0];
>>> -bitmap_copy(max_model.features, max_model.def->default_feat,
>>> +/* TCG emulates a z800 (with some optional additional features) */
>>> +max_model.def = s390_find_cpu_def(0x2066, 7, 3, NULL);
>>> +bitmap_copy(max_model.features, max_model.def->full_feat,
>>>  S390_FEAT_MAX);
>>>  add_qemu_cpu_model_features(max_model.features);
>>>  }
>>> @@ -956,8 +954,9 @@ static void s390_qemu_cpu_model_initfn(Object *obj)
>>>  S390CPU *cpu = S390_CPU(obj);
>>>  
>>>  cpu->model = g_malloc0(sizeof(*cpu->model));
>>> -/* TCG emulates a z900 (with some optional additional features) */
>>> -memcpy(_qemu_cpu_defs, _cpu_defs[0], 
>>> sizeof(s390_qemu_cpu_defs));
>>> +/* TCG emulates a z800 (with some optional additional features) */
>>> +memcpy(_qemu_cpu_defs, s390_find_cpu_def(0x2066, 7, 3, NULL),
>>> +   sizeof(s390_qemu_cpu_defs));
>>
>> No changing the qemu model without compatibility handling.
> 
> This patch series is based on the patch from Thomas Huth. It means the
> QEMU model is still based on a z900, but that it is possible to enable
> some more features like etf2.

Thomas' code did neither change features nor the "model definition". It
just allows for some more feature to be set. It is a hack.

I am pretty sure that expanding the "qemu" CPU model now (QMP
query-cpu-model-expansion) will indicate a z800, not a z900.

See cpu_info_from_model(). And that is a problem, because the QEMU CPU
model is a "migration-safe" CPU model, meaning it must remain equal for
every compatibility machine.

Thanks.

> 
> Aurelien
> 


-- 

Thanks,

David

Re: [Qemu-devel] [PATCH v7 0/7] trace: [tcg] Optimize per-vCPU tracing states with separate TB caches

[Emilio G. Cota]

On Fri, Jan 13, 2017 at 21:48:09 +0100, Lluís Vilanova wrote:
(snip)
> To handle both issues, this series integrates the dynamic tracing event state
> into the TB hashing function, so that vCPUs tracing different events will use
> separate TBs. Note that only events with the 'vcpu' property are used for
> hashing (as stored in the bitmap of CPUState->trace_dstate).

Is this going to be picked up by anyone? AFAICT the patchset is close
to being merge-ready.

Lluís: I'm very interested in your instrumentation work [1]:

- How much up to date are the branches in [1]? I couldn't find this
  v7 iteration in there, although maybe I didn't look carefully enough.

- Are you planning on upstreaming it? I have some time to help with
  that if you're interested.

- Do you have instrumentation examples beyond what's in
  docs/instrumentation.txt? In particular I'd like to see how the basic
  block (BBL) instrumentation works, i.e. how a 'skeleton' simulator
  would work to decode the guest instructions and also track their
  dependences.

Thanks,

Emilio

[1] https://projects.gso.ac.upc.edu/projects/qemu-dbi

Re: [Qemu-devel] [PATCH v2 02/15] file-posix: support BDRV_REQ_ALLOCATE

[Eric Blake]

On 06/01/2017 02:49 PM, Eric Blake wrote:
> On 06/01/2017 10:14 AM, Anton Nefedov wrote:
>> Current write_zeroes implementation is good enough to satisfy this flag too
>>
>> Signed-off-by: Anton Nefedov 
>> ---
>>  block/file-posix.c | 9 -
>>  1 file changed, 8 insertions(+), 1 deletion(-)
> 
> Are we sure that fallocate() is always fast, or are there some file
> systems where it is no faster than manually writing zeroes?  I'm worried
> that blindly claiming BDRV_REQ_ALLOCATE may fail if we encounter a libc

not so much fail as in "break the guest", but fail as in "take far more
time than we were expecting, pessimising our behavior to worse than if
we had not tried the allocation at all"

> or kernel-based fallback that takes a slow patch on our behalf.
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] [PATCH v2 03/15] blkdebug: support BDRV_REQ_ALLOCATE

[Eric Blake]

On 06/01/2017 10:14 AM, Anton Nefedov wrote:
> Support the flag if the underlying BDS supports it
> 
> Signed-off-by: Anton Nefedov 
> ---
>  block/blkdebug.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)

Shouldn't other passthrough drivers (like raw-format.c) make this change
as well?

> 
> diff --git a/block/blkdebug.c b/block/blkdebug.c
> index a5196e8..8b1401b 100644
> --- a/block/blkdebug.c
> +++ b/block/blkdebug.c
> @@ -415,7 +415,8 @@ static int blkdebug_open(BlockDriverState *bs, QDict 
> *options, int flags,
>  
>  bs->supported_write_flags = BDRV_REQ_FUA &
>  bs->file->bs->supported_write_flags;
> -bs->supported_zero_flags = (BDRV_REQ_FUA | BDRV_REQ_MAY_UNMAP) &
> +bs->supported_zero_flags =
> +(BDRV_REQ_FUA | BDRV_REQ_MAY_UNMAP | BDRV_REQ_ALLOCATE) &
>  bs->file->bs->supported_zero_flags;
>  ret = -EINVAL;
>  
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] [PATCH v2 02/15] file-posix: support BDRV_REQ_ALLOCATE

[Eric Blake]

On 06/01/2017 10:14 AM, Anton Nefedov wrote:
> Current write_zeroes implementation is good enough to satisfy this flag too
> 
> Signed-off-by: Anton Nefedov 
> ---
>  block/file-posix.c | 9 -
>  1 file changed, 8 insertions(+), 1 deletion(-)

Are we sure that fallocate() is always fast, or are there some file
systems where it is no faster than manually writing zeroes?  I'm worried
that blindly claiming BDRV_REQ_ALLOCATE may fail if we encounter a libc
or kernel-based fallback that takes a slow patch on our behalf.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org



signature.asc
Description: OpenPGP digital signature

[Qemu-devel] [PULL 14/15] migration: Export ram.c functions in its own file

[Juan Quintela]

All functions are internal except for ram_mig_init().  Create
migration/misc.h for this kind of functions.

Signed-off-by: Juan Quintela 
Reviewed-by: Dr. David Alan Gilbert 
---
 include/migration/migration.h | 38 ---
 include/migration/misc.h  | 21 +
 migration/migration.c |  1 +
 migration/postcopy-ram.c  |  1 +
 migration/ram.c   |  2 ++
 migration/ram.h   | 70 +++
 migration/rdma.c  |  2 +-
 migration/savevm.c|  1 +
 vl.c  |  1 +
 9 files changed, 98 insertions(+), 39 deletions(-)
 create mode 100644 include/migration/misc.h
 create mode 100644 migration/ram.h

diff --git a/include/migration/migration.h b/include/migration/migration.h
index 8d29bc9..79b5484 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -171,38 +171,6 @@ bool migration_in_postcopy(void);
 bool migration_in_postcopy_after_devices(MigrationState *);
 MigrationState *migrate_get_current(void);
 
-void migrate_compress_threads_create(void);
-void migrate_compress_threads_join(void);
-void migrate_decompress_threads_create(void);
-void migrate_decompress_threads_join(void);
-uint64_t ram_bytes_remaining(void);
-uint64_t ram_bytes_transferred(void);
-uint64_t ram_bytes_total(void);
-uint64_t ram_dirty_sync_count(void);
-uint64_t ram_dirty_pages_rate(void);
-uint64_t ram_postcopy_requests(void);
-void free_xbzrle_decoded_buf(void);
-
-void acct_update_position(QEMUFile *f, size_t size, bool zero);
-
-uint64_t dup_mig_pages_transferred(void);
-uint64_t norm_mig_pages_transferred(void);
-uint64_t xbzrle_mig_bytes_transferred(void);
-uint64_t xbzrle_mig_pages_transferred(void);
-uint64_t xbzrle_mig_pages_overflow(void);
-uint64_t xbzrle_mig_pages_cache_miss(void);
-double xbzrle_mig_cache_miss_rate(void);
-
-void ram_handle_compressed(void *host, uint8_t ch, uint64_t size);
-void ram_debug_dump_bitmap(unsigned long *todump, bool expected,
-   unsigned long pages);
-/* For outgoing discard bitmap */
-int ram_postcopy_send_discard_bitmap(MigrationState *ms);
-/* For incoming postcopy discard */
-int ram_discard_range(const char *block_name, uint64_t start, size_t length);
-int ram_postcopy_incoming_init(MigrationIncomingState *mis);
-void ram_postcopy_migrated_memory_release(MigrationState *ms);
-
 bool migrate_release_ram(void);
 bool migrate_postcopy_ram(void);
 bool migrate_zero_blocks(void);
@@ -213,8 +181,6 @@ int migrate_use_xbzrle(void);
 int64_t migrate_xbzrle_cache_size(void);
 bool migrate_colo_enabled(void);
 
-int64_t xbzrle_cache_resize(int64_t new_size);
-
 bool migrate_use_block(void);
 bool migrate_use_block_incremental(void);
 
@@ -253,7 +219,6 @@ size_t ram_control_save_page(QEMUFile *f, ram_addr_t 
block_offset,
  ram_addr_t offset, size_t size,
  uint64_t *bytes_sent);
 
-void ram_mig_init(void);
 void savevm_skip_section_footers(void);
 void register_global_state(void);
 void global_state_set_optional(void);
@@ -261,7 +226,4 @@ void savevm_skip_configuration(void);
 int global_state_store(void);
 void global_state_store_running(void);
 
-void migration_page_queue_free(void);
-int ram_save_queue_pages(const char *rbname, ram_addr_t start, ram_addr_t len);
-uint64_t ram_pagesize_summary(void);
 #endif
diff --git a/include/migration/misc.h b/include/migration/misc.h
new file mode 100644
index 000..0b37714
--- /dev/null
+++ b/include/migration/misc.h
@@ -0,0 +1,21 @@
+/*
+ * QEMU migration miscellaneus exported functions
+ *
+ * Copyright IBM, Corp. 2008
+ *
+ * Authors:
+ *  Anthony Liguori   
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef MIGRATION_MISC_H
+#define MIGRATION_MISC_H
+
+/* migration/ram.c */
+
+void ram_mig_init(void);
+
+#endif
diff --git a/migration/migration.c b/migration/migration.c
index ec79aff..6e5afa4 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -22,6 +22,7 @@
 #include "fd.h"
 #include "socket.h"
 #include "rdma.h"
+#include "ram.h"
 #include "migration/migration.h"
 #include "savevm.h"
 #include "qemu-file-channel.h"
diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c
index 71f4389..9c41887 100644
--- a/migration/postcopy-ram.c
+++ b/migration/postcopy-ram.c
@@ -24,6 +24,7 @@
 #include "qemu-file.h"
 #include "savevm.h"
 #include "postcopy-ram.h"
+#include "ram.h"
 #include "sysemu/sysemu.h"
 #include "sysemu/balloon.h"
 #include "qemu/error-report.h"
diff --git a/migration/ram.c b/migration/ram.c
index 390f714..f387e9c 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -36,7 +36,9 @@
 #include "qemu/timer.h"
 #include "qemu/main-loop.h"
 #include "xbzrle.h"
+#include "ram.h"
 #include "migration/migration.h"
+#include 

[Qemu-devel] [PULL 12/15] migration: Export rdma.c functions in its own file

[Juan Quintela]

Signed-off-by: Juan Quintela 
Reviewed-by: Dr. David Alan Gilbert 
---
 include/migration/migration.h |  4 
 migration/migration.c |  1 +
 migration/rdma.c  |  1 +
 migration/rdma.h  | 25 +
 4 files changed, 27 insertions(+), 4 deletions(-)
 create mode 100644 migration/rdma.h

diff --git a/include/migration/migration.h b/include/migration/migration.h
index 29fda5b..8d29bc9 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -153,10 +153,6 @@ void qemu_start_incoming_migration(const char *uri, Error 
**errp);
 
 uint64_t migrate_max_downtime(void);
 
-void rdma_start_outgoing_migration(void *opaque, const char *host_port, Error 
**errp);
-
-void rdma_start_incoming_migration(const char *host_port, Error **errp);
-
 void migrate_fd_error(MigrationState *s, const Error *error);
 
 void migrate_fd_connect(MigrationState *s);
diff --git a/migration/migration.c b/migration/migration.c
index fe6dc18..ec79aff 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -21,6 +21,7 @@
 #include "exec.h"
 #include "fd.h"
 #include "socket.h"
+#include "rdma.h"
 #include "migration/migration.h"
 #include "savevm.h"
 #include "qemu-file-channel.h"
diff --git a/migration/rdma.c b/migration/rdma.c
index 4cb5bf8..fab30ea 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -17,6 +17,7 @@
 #include "qapi/error.h"
 #include "qemu-common.h"
 #include "qemu/cutils.h"
+#include "rdma.h"
 #include "migration/migration.h"
 #include "qemu-file.h"
 #include "exec/cpu-common.h"
diff --git a/migration/rdma.h b/migration/rdma.h
new file mode 100644
index 000..de2ba09
--- /dev/null
+++ b/migration/rdma.h
@@ -0,0 +1,25 @@
+/*
+ * RDMA protocol and interfaces
+ *
+ * Copyright IBM, Corp. 2010-2013
+ * Copyright Red Hat, Inc. 2015-2016
+ *
+ * Authors:
+ *  Michael R. Hines 
+ *  Jiuxing Liu 
+ *  Daniel P. Berrange 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef QEMU_MIGRATION_RDMA_H
+#define QEMU_MIGRATION_RDMA_H
+
+void rdma_start_outgoing_migration(void *opaque, const char *host_port,
+   Error **errp);
+
+void rdma_start_incoming_migration(const char *host_port, Error **errp);
+
+#endif
-- 
2.9.4

Re: [Qemu-devel] [PATCH v7 4/7] exec: [tcg] Use different TBs according to the vCPU's dynamic tracing state

[Emilio G. Cota]

On Fri, Jan 13, 2017 at 21:48:35 +0100, Lluís Vilanova wrote:
>  9 files changed, 54 insertions(+), 11 deletions(-)
> 
> diff --git a/cpu-exec.c b/cpu-exec.c
> index 4188fed3c6..36709cba1f 100644
> --- a/cpu-exec.c
> +++ b/cpu-exec.c
> @@ -261,6 +261,7 @@ struct tb_desc {
>  CPUArchState *env;
>  tb_page_addr_t phys_page1;
>  uint32_t flags;
> +TRACE_QHT_VCPU_DSTATE_TYPE trace_vcpu_dstate;
(snip)
> diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
> index 57cd978578..ae74f61ea2 100644
> --- a/include/exec/exec-all.h
> +++ b/include/exec/exec-all.h
> @@ -200,6 +200,10 @@ static inline void tlb_flush_by_mmuidx(CPUState *cpu, 
> ...)
>  #define USE_DIRECT_JUMP
>  #endif
>  
> +/**
> + * TranslationBlock:
> + * @trace_vcpu_dstate: Per-vCPU dynamic tracing state used to generate this 
> TB.
> + */
>  struct TranslationBlock {
>  target_ulong pc;   /* simulated PC corresponding to this block (EIP + CS 
> base) */
>  target_ulong cs_base; /* CS base for this block */
> @@ -215,6 +219,7 @@ struct TranslationBlock {
>  #define CF_IGNORE_ICOUNT 0x4 /* Do not generate icount code */
>  
>  uint16_t invalid;
> +TRACE_QHT_VCPU_DSTATE_TYPE trace_vcpu_dstate;
(snip)
> --- a/include/exec/tb-hash-xx.h
> +++ b/include/exec/tb-hash-xx.h
> @@ -35,6 +35,7 @@
>  #define EXEC_TB_HASH_XX_H
>  
>  #include "qemu/bitops.h"
> +#include "qemu-common.h"
>  
>  #define PRIME32_1   2654435761U
>  #define PRIME32_2   2246822519U
> @@ -49,7 +50,8 @@
>   * contiguous in memory.
>   */
>  static inline
> -uint32_t tb_hash_func5(uint64_t a0, uint64_t b0, uint32_t e)
> +uint32_t tb_hash_func6(uint64_t a0, uint64_t b0, uint32_t e,
> +   TRACE_QHT_VCPU_DSTATE_TYPE f)

I find this typedef unnecessary. Why not use u32 everywhere?
If ever we need more bits, then we'll add additional u32's here
as well.

Also, including above qemu-common.h goes against the spirit of
keeping this file (as well as tb-hash.h) free of external
dependences. (originally tb-hash.h's contents were in exec-all.h)

If we're worried about forgetting to update the hash function,
we could have a compile-time check + a comment elsewhere
(e.g. translate-all.c).

>  {
>  uint32_t v1 = TB_HASH_XX_SEED + PRIME32_1 + PRIME32_2;
>  uint32_t v2 = TB_HASH_XX_SEED + PRIME32_2;
> @@ -83,6 +85,10 @@ uint32_t tb_hash_func5(uint64_t a0, uint64_t b0, uint32_t 
> e)
Right here you should also do:

@@ -78,7 +78,7 @@ uint32_t tb_hash_func5(uint64_t a0, uint64_t b0, uint32_t e)
 v4 *= PRIME32_1;

 h32 = rol32(v1, 1) + rol32(v2, 7) + rol32(v3, 12) + rol32(v4, 18);
-h32 += 20;
+h32 += 24;

to take into account the newly added parameter.

>  h32 += e * PRIME32_3;
>  h32  = rol32(h32, 17) * PRIME32_4;
>  
> +QEMU_BUILD_BUG_ON(sizeof(TRACE_QHT_VCPU_DSTATE_TYPE) != 
> sizeof(uint32_t));
> +h32 += f * PRIME32_3;
> +h32  = rol32(h32, 17) * PRIME32_4;

If we get rid of the typedef, this compile-time check will go away too.

(snip)
> diff --git a/include/qemu-common.h b/include/qemu-common.h
> index 1430390eb6..73a6fe 100644
> --- a/include/qemu-common.h
> +++ b/include/qemu-common.h
> @@ -151,4 +151,7 @@ void page_size_init(void);
>   * returned. */
>  bool dump_in_progress(void);
>  
> +/* Use a macro to allow safe changes to its size in the future */
> +#define TRACE_QHT_VCPU_DSTATE_TYPE uint32_t
> +
(snip)
> diff --git a/translate-all.c b/translate-all.c
> index 29ccb9e546..6e1b1d474c 100644
> --- a/translate-all.c
> +++ b/translate-all.c
> @@ -54,6 +54,7 @@
>  #include "exec/tb-hash.h"
>  #include "translate-all.h"
>  #include "qemu/bitmap.h"
> +#include "qemu/error-report.h"
>  #include "qemu/timer.h"
>  #include "exec/log.h"
>  
> @@ -813,6 +814,12 @@ static void tb_htable_init(void)
>  {
>  unsigned int mode = QHT_MODE_AUTO_RESIZE;
>  
> +/* Ensure TB hash function covers the bitmap size */
> +if (DIV_ROUND_UP(trace_get_vcpu_event_count(), BITS_PER_BYTE) >
> +sizeof(TRACE_QHT_VCPU_DSTATE_TYPE)) {
> +error_report("too many 'vcpu' events for the TB hash function");
> +}
> +

This is a better place to do the above check, I think.

Thanks,

Emilio

[Qemu-devel] [PULL 08/15] migration: Export exec.c functions in its own file

[Juan Quintela]

Signed-off-by: Juan Quintela 
Reviewed-by: Dr. David Alan Gilbert 
---
 include/migration/migration.h |  4 
 migration/exec.c  |  1 +
 migration/exec.h  | 26 ++
 migration/migration.c |  1 +
 4 files changed, 28 insertions(+), 4 deletions(-)
 create mode 100644 migration/exec.h

diff --git a/include/migration/migration.h b/include/migration/migration.h
index d1a353a..d04f045 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -163,10 +163,6 @@ void migration_tls_channel_connect(MigrationState *s,
 
 uint64_t migrate_max_downtime(void);
 
-void exec_start_incoming_migration(const char *host_port, Error **errp);
-
-void exec_start_outgoing_migration(MigrationState *s, const char *host_port, 
Error **errp);
-
 void tcp_start_incoming_migration(const char *host_port, Error **errp);
 
 void tcp_start_outgoing_migration(MigrationState *s, const char *host_port, 
Error **errp);
diff --git a/migration/exec.c b/migration/exec.c
index 57a9335..9077024 100644
--- a/migration/exec.c
+++ b/migration/exec.c
@@ -21,6 +21,7 @@
 #include "qapi/error.h"
 #include "qemu-common.h"
 #include "channel.h"
+#include "exec.h"
 #include "migration/migration.h"
 #include "io/channel-command.h"
 #include "trace.h"
diff --git a/migration/exec.h b/migration/exec.h
new file mode 100644
index 000..b210ffd
--- /dev/null
+++ b/migration/exec.h
@@ -0,0 +1,26 @@
+/*
+ * QEMU live migration
+ *
+ * Copyright IBM, Corp. 2008
+ * Copyright Dell MessageOne 2008
+ * Copyright Red Hat, Inc. 2015-2016
+ *
+ * Authors:
+ *  Anthony Liguori   
+ *  Charles Duffy 
+ *  Daniel P. Berrange 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ *
+ * Contributions after 2012-01-13 are licensed under the terms of the
+ * GNU GPL, version 2 or (at your option) any later version.
+ */
+
+#ifndef QEMU_MIGRATION_EXEC_H
+#define QEMU_MIGRATION_EXEC_H
+void exec_start_incoming_migration(const char *host_port, Error **errp);
+
+void exec_start_outgoing_migration(MigrationState *s, const char *host_port,
+   Error **errp);
+#endif
diff --git a/migration/migration.c b/migration/migration.c
index 97bbb9f..c400388 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -18,6 +18,7 @@
 #include "qemu/error-report.h"
 #include "qemu/main-loop.h"
 #include "migration/blocker.h"
+#include "exec.h"
 #include "migration/migration.h"
 #include "savevm.h"
 #include "qemu-file-channel.h"
-- 
2.9.4

[Qemu-devel] [PULL 15/15] migration: Move include/migration/block.h into migration/

[Juan Quintela]

All functions were internal, except blk_mig_init() that is exported in
misc.h now.

Signed-off-by: Juan Quintela 
Reviewed-by: Dr. David Alan Gilbert 
---
 include/migration/misc.h | 8 
 migration/block.c| 3 ++-
 {include/migration => migration}/block.h | 2 --
 migration/colo.c | 2 +-
 migration/migration.c| 2 +-
 vl.c | 1 -
 6 files changed, 12 insertions(+), 6 deletions(-)
 rename {include/migration => migration}/block.h (93%)

diff --git a/include/migration/misc.h b/include/migration/misc.h
index 0b37714..d7892b7 100644
--- a/include/migration/misc.h
+++ b/include/migration/misc.h
@@ -18,4 +18,12 @@
 
 void ram_mig_init(void);
 
+/* migration/block.c */
+
+#ifdef CONFIG_LIVE_BLOCK_MIGRATION
+void blk_mig_init(void);
+#else
+static inline void blk_mig_init(void) {}
+#endif
+
 #endif
diff --git a/migration/block.c b/migration/block.c
index 3e27499..4d8c2e9 100644
--- a/migration/block.c
+++ b/migration/block.c
@@ -23,7 +23,8 @@
 #include "qemu/cutils.h"
 #include "qemu/queue.h"
 #include "qemu/timer.h"
-#include "migration/block.h"
+#include "block.h"
+#include "migration/misc.h"
 #include "migration/migration.h"
 #include "sysemu/blockdev.h"
 #include "qemu-file.h"
diff --git a/include/migration/block.h b/migration/block.h
similarity index 93%
rename from include/migration/block.h
rename to migration/block.h
index 28cff53..22ebe94 100644
--- a/include/migration/block.h
+++ b/migration/block.h
@@ -15,14 +15,12 @@
 #define MIGRATION_BLOCK_H
 
 #ifdef CONFIG_LIVE_BLOCK_MIGRATION
-void blk_mig_init(void);
 int blk_mig_active(void);
 uint64_t blk_mig_bytes_transferred(void);
 uint64_t blk_mig_bytes_remaining(void);
 uint64_t blk_mig_bytes_total(void);
 
 #else
-static inline void blk_mig_init(void) { }
 static inline int blk_mig_active(void)
 {
 return false;
diff --git a/migration/colo.c b/migration/colo.c
index 4f1f3b8..111b715 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -18,7 +18,7 @@
 #include "qemu-file.h"
 #include "savevm.h"
 #include "migration/colo.h"
-#include "migration/block.h"
+#include "block.h"
 #include "io/channel-buffer.h"
 #include "trace.h"
 #include "qemu/error-report.h"
diff --git a/migration/migration.c b/migration/migration.c
index 6e5afa4..48c94c9 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -34,7 +34,7 @@
 #include "qapi/util.h"
 #include "qemu/sockets.h"
 #include "qemu/rcu.h"
-#include "migration/block.h"
+#include "block.h"
 #include "postcopy-ram.h"
 #include "qemu/thread.h"
 #include "qmp-commands.h"
diff --git a/vl.c b/vl.c
index 13deeba..80b86c0 100644
--- a/vl.c
+++ b/vl.c
@@ -86,7 +86,6 @@ int main(int argc, char **argv)
 #include "qemu/log.h"
 #include "sysemu/blockdev.h"
 #include "hw/block/block.h"
-#include "migration/block.h"
 #include "migration/misc.h"
 #include "migration/snapshot.h"
 #include "sysemu/tpm.h"
-- 
2.9.4

[Qemu-devel] [PULL 06/15] migration: Remove unneeded includes of migration/vmstate.h

[Juan Quintela]

Signed-off-by: Juan Quintela 
Reviewed-by: Philippe Mathieu-Daudé 
Reviewed-by: Dr. David Alan Gilbert 
---
 hw/core/qdev.c   | 1 -
 include/hw/acpi/memory_hotplug.h | 1 -
 include/hw/acpi/pcihp.h  | 1 -
 include/hw/pci/shpc.h| 1 -
 target/alpha/cpu.c   | 1 -
 target/hppa/cpu.c| 1 -
 target/s390x/cpu.c   | 1 -
 target/tilegx/cpu.c  | 1 -
 8 files changed, 8 deletions(-)

diff --git a/hw/core/qdev.c b/hw/core/qdev.c
index 71ff95f..0ce45a2 100644
--- a/hw/core/qdev.c
+++ b/hw/core/qdev.c
@@ -37,7 +37,6 @@
 #include "hw/boards.h"
 #include "hw/sysbus.h"
 #include "qapi-event.h"
-#include "migration/vmstate.h"
 
 bool qdev_hotplug = false;
 static bool qdev_hot_added = false;
diff --git a/include/hw/acpi/memory_hotplug.h b/include/hw/acpi/memory_hotplug.h
index db8ebc9..77c6576 100644
--- a/include/hw/acpi/memory_hotplug.h
+++ b/include/hw/acpi/memory_hotplug.h
@@ -3,7 +3,6 @@
 
 #include "hw/qdev-core.h"
 #include "hw/acpi/acpi.h"
-#include "migration/vmstate.h"
 #include "hw/acpi/aml-build.h"
 
 /**
diff --git a/include/hw/acpi/pcihp.h b/include/hw/acpi/pcihp.h
index 04528b7..8a65f99 100644
--- a/include/hw/acpi/pcihp.h
+++ b/include/hw/acpi/pcihp.h
@@ -28,7 +28,6 @@
 #define HW_ACPI_PCIHP_H
 
 #include "hw/acpi/acpi.h"
-#include "migration/vmstate.h"
 #include "hw/hotplug.h"
 
 #define ACPI_PCIHP_IO_BASE_PROP "acpi-pcihp-io-base"
diff --git a/include/hw/pci/shpc.h b/include/hw/pci/shpc.h
index b208554..71e836b 100644
--- a/include/hw/pci/shpc.h
+++ b/include/hw/pci/shpc.h
@@ -3,7 +3,6 @@
 
 #include "qemu-common.h"
 #include "exec/memory.h"
-#include "migration/vmstate.h"
 #include "hw/hotplug.h"
 #include "hw/pci/pci.h"
 
diff --git a/target/alpha/cpu.c b/target/alpha/cpu.c
index b4f9798..8186c9d 100644
--- a/target/alpha/cpu.c
+++ b/target/alpha/cpu.c
@@ -23,7 +23,6 @@
 #include "qapi/error.h"
 #include "cpu.h"
 #include "qemu-common.h"
-#include "migration/vmstate.h"
 #include "exec/exec-all.h"
 
 
diff --git a/target/hppa/cpu.c b/target/hppa/cpu.c
index 1d791d0..30299e9 100644
--- a/target/hppa/cpu.c
+++ b/target/hppa/cpu.c
@@ -22,7 +22,6 @@
 #include "qapi/error.h"
 #include "cpu.h"
 #include "qemu-common.h"
-#include "migration/vmstate.h"
 #include "exec/exec-all.h"
 
 
diff --git a/target/s390x/cpu.c b/target/s390x/cpu.c
index a69005d..accef03 100644
--- a/target/s390x/cpu.c
+++ b/target/s390x/cpu.c
@@ -32,7 +32,6 @@
 #include "qemu/error-report.h"
 #include "trace.h"
 #include "qapi/visitor.h"
-#include "migration/vmstate.h"
 #include "exec/exec-all.h"
 #ifndef CONFIG_USER_ONLY
 #include "hw/hw.h"
diff --git a/target/tilegx/cpu.c b/target/tilegx/cpu.c
index d90e38e..4532639 100644
--- a/target/tilegx/cpu.c
+++ b/target/tilegx/cpu.c
@@ -23,7 +23,6 @@
 #include "cpu.h"
 #include "qemu-common.h"
 #include "hw/qdev-properties.h"
-#include "migration/vmstate.h"
 #include "linux-user/syscall_defs.h"
 #include "exec/exec-all.h"
 
-- 
2.9.4

[Qemu-devel] [PULL 13/15] migration: Create include for migration snapshots

[Juan Quintela]

Start removing migration code from sysemu/sysemu.h.

Signed-off-by: Juan Quintela 
Reviewed-by: Dr. David Alan Gilbert 
---
 hmp.c|  5 +++--
 include/block/block_int.h|  4 ++--
 include/migration/snapshot.h | 21 +
 include/sysemu/sysemu.h  |  3 ---
 migration/savevm.c   |  5 +++--
 replay/replay-snapshot.c |  5 +++--
 vl.c |  3 ++-
 7 files changed, 34 insertions(+), 12 deletions(-)
 create mode 100644 include/migration/snapshot.h

diff --git a/hmp.c b/hmp.c
index 20f5dab..ad72390 100644
--- a/hmp.c
+++ b/hmp.c
@@ -42,6 +42,7 @@
 #include "qemu/error-report.h"
 #include "exec/ramlist.h"
 #include "hw/intc/intc.h"
+#include "migration/snapshot.h"
 
 #ifdef CONFIG_SPICE
 #include 
@@ -1284,7 +1285,7 @@ void hmp_loadvm(Monitor *mon, const QDict *qdict)
 
 vm_stop(RUN_STATE_RESTORE_VM);
 
-if (load_vmstate(name, ) == 0 && saved_vm_running) {
+if (load_snapshot(name, ) == 0 && saved_vm_running) {
 vm_start();
 }
 hmp_handle_error(mon, );
@@ -1294,7 +1295,7 @@ void hmp_savevm(Monitor *mon, const QDict *qdict)
 {
 Error *err = NULL;
 
-save_vmstate(qdict_get_try_str(qdict, "name"), );
+save_snapshot(qdict_get_try_str(qdict, "name"), );
 hmp_handle_error(mon, );
 }
 
diff --git a/include/block/block_int.h b/include/block/block_int.h
index e5eb473..cb78c4f 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -601,8 +601,8 @@ struct BlockDriverState {
 int copy_on_read;
 
 /* If we are reading a disk image, give its size in sectors.
- * Generally read-only; it is written to by load_vmstate and save_vmstate,
- * but the block layer is quiescent during those.
+ * Generally read-only; it is written to by load_snapshot and
+ * save_snaphost, but the block layer is quiescent during those.
  */
 int64_t total_sectors;
 
diff --git a/include/migration/snapshot.h b/include/migration/snapshot.h
new file mode 100644
index 000..c85b6ec
--- /dev/null
+++ b/include/migration/snapshot.h
@@ -0,0 +1,21 @@
+/*
+ * QEMU snapshots
+ *
+ * Copyright (c) 2004-2008 Fabrice Bellard
+ * Copyright (c) 2009-2015 Red Hat Inc
+ *
+ * Authors:
+ *  Juan Quintela 
+ *
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#ifndef QEMU_MIGRATION_SNAPSHOT_H
+#define QEMU_MIGRATION_SNAPSHOT_H
+
+int save_snapshot(const char *name, Error **errp);
+int load_snapshot(const char *name, Error **errp);
+
+#endif
diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
index 723c8dc..9841a52 100644
--- a/include/sysemu/sysemu.h
+++ b/include/sysemu/sysemu.h
@@ -92,9 +92,6 @@ void qemu_remove_exit_notifier(Notifier *notify);
 void qemu_add_machine_init_done_notifier(Notifier *notify);
 void qemu_remove_machine_init_done_notifier(Notifier *notify);
 
-int save_vmstate(const char *name, Error **errp);
-int load_vmstate(const char *name, Error **errp);
-
 void qemu_announce_self(void);
 
 extern int autostart;
diff --git a/migration/savevm.c b/migration/savevm.c
index bb3f9ec..f2664f3 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -35,6 +35,7 @@
 #include "sysemu/sysemu.h"
 #include "qemu/timer.h"
 #include "migration/migration.h"
+#include "migration/snapshot.h"
 #include "qemu-file-channel.h"
 #include "qemu-file.h"
 #include "savevm.h"
@@ -2067,7 +2068,7 @@ int qemu_loadvm_state(QEMUFile *f)
 return ret;
 }
 
-int save_vmstate(const char *name, Error **errp)
+int save_snapshot(const char *name, Error **errp)
 {
 BlockDriverState *bs, *bs1;
 QEMUSnapshotInfo sn1, *sn = , old_sn1, *old_sn = _sn1;
@@ -2224,7 +2225,7 @@ void qmp_xen_load_devices_state(const char *filename, 
Error **errp)
 migration_incoming_state_destroy();
 }
 
-int load_vmstate(const char *name, Error **errp)
+int load_snapshot(const char *name, Error **errp)
 {
 BlockDriverState *bs, *bs_vm_state;
 QEMUSnapshotInfo sn;
diff --git a/replay/replay-snapshot.c b/replay/replay-snapshot.c
index c75cd38..a4ded29 100644
--- a/replay/replay-snapshot.c
+++ b/replay/replay-snapshot.c
@@ -19,6 +19,7 @@
 #include "qapi/qmp/qstring.h"
 #include "qemu/error-report.h"
 #include "migration/vmstate.h"
+#include "migration/snapshot.h"
 
 static void replay_pre_save(void *opaque)
 {
@@ -66,13 +67,13 @@ void replay_vmstate_init(void)
 
 if (replay_snapshot) {
 if (replay_mode == REPLAY_MODE_RECORD) {
-if (save_vmstate(replay_snapshot, ) != 0) {
+if (save_snapshot(replay_snapshot, ) != 0) {
 error_report_err(err);
 error_report("Could not create snapshot for icount record");
 exit(1);
 }
 } else if (replay_mode == REPLAY_MODE_PLAY) {
-if (load_vmstate(replay_snapshot, ) != 0) {
+if 

[Qemu-devel] [PULL 04/15] migration: fix leak of src file on dst

[Juan Quintela]

From: Peter Xu 

The return path channel is possibly leaked. Fix it.

Signed-off-by: Peter Xu 
Reviewed-by: Juan Quintela 
Signed-off-by: Juan Quintela 
---
 migration/migration.c | 5 +
 1 file changed, 5 insertions(+)

diff --git a/migration/migration.c b/migration/migration.c
index c3218cd..b90e399 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -133,6 +133,11 @@ void migration_incoming_state_destroy(void)
 {
 struct MigrationIncomingState *mis = migration_incoming_get_current();
 
+if (mis->to_src_file) {
+qemu_fclose(mis->to_src_file);
+mis->to_src_file = NULL;
+}
+
 qemu_event_destroy(>main_thread_load_event);
 }
 
-- 
2.9.4

[Qemu-devel] [PULL 11/15] migration: Export tls.c functions in its own file

[Juan Quintela]

Just for the functions exported from tls.c.  Notice that we can't
remove the migration/migration.h include from tls.c because it access
directly MigrationState for the tls params.

Signed-off-by: Juan Quintela 
Reviewed-by: Dr. David Alan Gilbert 
---
 include/migration/migration.h | 10 --
 migration/channel.c   |  1 +
 migration/migration.c |  1 -
 migration/tls.c   |  1 +
 migration/tls.h   | 34 ++
 5 files changed, 36 insertions(+), 11 deletions(-)
 create mode 100644 migration/tls.h

diff --git a/include/migration/migration.h b/include/migration/migration.h
index 9d311ed..29fda5b 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -18,7 +18,6 @@
 #include "qemu-common.h"
 #include "qemu/thread.h"
 #include "qemu/notify.h"
-#include "io/channel.h"
 #include "qapi-types.h"
 #include "exec/cpu-common.h"
 #include "qemu/coroutine_int.h"
@@ -152,15 +151,6 @@ void migration_fd_process_incoming(QEMUFile *f);
 
 void qemu_start_incoming_migration(const char *uri, Error **errp);
 
-void migration_tls_channel_process_incoming(MigrationState *s,
-QIOChannel *ioc,
-Error **errp);
-
-void migration_tls_channel_connect(MigrationState *s,
-   QIOChannel *ioc,
-   const char *hostname,
-   Error **errp);
-
 uint64_t migrate_max_downtime(void);
 
 void rdma_start_outgoing_migration(void *opaque, const char *host_port, Error 
**errp);
diff --git a/migration/channel.c b/migration/channel.c
index 2e78905cc..eae1d9e 100644
--- a/migration/channel.c
+++ b/migration/channel.c
@@ -12,6 +12,7 @@
 
 #include "qemu/osdep.h"
 #include "channel.h"
+#include "tls.h"
 #include "migration/migration.h"
 #include "qemu-file-channel.h"
 #include "trace.h"
diff --git a/migration/migration.c b/migration/migration.c
index e6e36fa..fe6dc18 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -43,7 +43,6 @@
 #include "exec/address-spaces.h"
 #include "exec/target_page.h"
 #include "io/channel-buffer.h"
-#include "io/channel-tls.h"
 #include "migration/colo.h"
 
 #define MAX_THROTTLE  (32 << 20)  /* Migration transfer speed throttling */
diff --git a/migration/tls.c b/migration/tls.c
index 34ad121..bae9aca 100644
--- a/migration/tls.c
+++ b/migration/tls.c
@@ -21,6 +21,7 @@
 #include "qemu/osdep.h"
 #include "channel.h"
 #include "migration/migration.h"
+#include "tls.h"
 #include "io/channel-tls.h"
 #include "crypto/tlscreds.h"
 #include "qemu/error-report.h"
diff --git a/migration/tls.h b/migration/tls.h
new file mode 100644
index 000..cdd7000
--- /dev/null
+++ b/migration/tls.h
@@ -0,0 +1,34 @@
+/*
+ * QEMU migration TLS support
+ *
+ * Copyright (c) 2015 Red Hat, Inc.
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, see .
+ *
+ */
+
+#ifndef QEMU_MIGRATION_TLS_H
+#define QEMU_MIGRATION_TLS_H
+
+#include "io/channel.h"
+
+void migration_tls_channel_process_incoming(MigrationState *s,
+QIOChannel *ioc,
+Error **errp);
+
+void migration_tls_channel_connect(MigrationState *s,
+   QIOChannel *ioc,
+   const char *hostname,
+   Error **errp);
+#endif
-- 
2.9.4

[Qemu-devel] [PULL 10/15] migration: Export socket.c functions in its own file

[Juan Quintela]

Signed-off-by: Juan Quintela 
Reviewed-by: Dr. David Alan Gilbert 
---
 include/migration/migration.h |  8 
 migration/migration.c |  1 +
 migration/socket.c|  1 +
 migration/socket.h| 28 
 4 files changed, 30 insertions(+), 8 deletions(-)
 create mode 100644 migration/socket.h

diff --git a/include/migration/migration.h b/include/migration/migration.h
index d249c13..9d311ed 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -163,14 +163,6 @@ void migration_tls_channel_connect(MigrationState *s,
 
 uint64_t migrate_max_downtime(void);
 
-void tcp_start_incoming_migration(const char *host_port, Error **errp);
-
-void tcp_start_outgoing_migration(MigrationState *s, const char *host_port, 
Error **errp);
-
-void unix_start_incoming_migration(const char *path, Error **errp);
-
-void unix_start_outgoing_migration(MigrationState *s, const char *path, Error 
**errp);
-
 void rdma_start_outgoing_migration(void *opaque, const char *host_port, Error 
**errp);
 
 void rdma_start_incoming_migration(const char *host_port, Error **errp);
diff --git a/migration/migration.c b/migration/migration.c
index ec93f7a..e6e36fa 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -20,6 +20,7 @@
 #include "migration/blocker.h"
 #include "exec.h"
 #include "fd.h"
+#include "socket.h"
 #include "migration/migration.h"
 #include "savevm.h"
 #include "qemu-file-channel.h"
diff --git a/migration/socket.c b/migration/socket.c
index 3f8ffc9..85bfdcc 100644
--- a/migration/socket.c
+++ b/migration/socket.c
@@ -20,6 +20,7 @@
 #include "qemu/error-report.h"
 #include "qapi/error.h"
 #include "channel.h"
+#include "socket.h"
 #include "migration/migration.h"
 #include "qemu-file.h"
 #include "io/channel-socket.h"
diff --git a/migration/socket.h b/migration/socket.h
new file mode 100644
index 000..6b91e9d
--- /dev/null
+++ b/migration/socket.h
@@ -0,0 +1,28 @@
+/*
+ * QEMU live migration via socket
+ *
+ * Copyright Red Hat, Inc. 2009-2016
+ *
+ * Authors:
+ *  Chris Lalancette 
+ *  Daniel P. Berrange 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ *
+ * Contributions after 2012-01-13 are licensed under the terms of the
+ * GNU GPL, version 2 or (at your option) any later version.
+ */
+
+#ifndef QEMU_MIGRATION_SOCKET_H
+#define QEMU_MIGRATION_SOCKET_H
+void tcp_start_incoming_migration(const char *host_port, Error **errp);
+
+void tcp_start_outgoing_migration(MigrationState *s, const char *host_port,
+  Error **errp);
+
+void unix_start_incoming_migration(const char *path, Error **errp);
+
+void unix_start_outgoing_migration(MigrationState *s, const char *path,
+   Error **errp);
+#endif
-- 
2.9.4

[Qemu-devel] [PULL 03/15] migration: Remove section_id parameter from vmstate_load

[Juan Quintela]

Everything else assumes that we always load a device from its own
savevm handler.

Signed-off-by: Juan Quintela 
Reviewed-by: Laurent Vivier 
Reviewed-by: Peter Xu 
Reviewed-by: Dr. David Alan Gilbert 
---
 migration/savevm.c | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/migration/savevm.c b/migration/savevm.c
index d96209b..2d1d4bc 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -746,13 +746,13 @@ void vmstate_unregister(DeviceState *dev, const 
VMStateDescription *vmsd,
 }
 }
 
-static int vmstate_load(QEMUFile *f, SaveStateEntry *se, int version_id)
+static int vmstate_load(QEMUFile *f, SaveStateEntry *se)
 {
 trace_vmstate_load(se->idstr, se->vmsd ? se->vmsd->name : "(old)");
 if (!se->vmsd) { /* Old style */
-return se->ops->load_state(f, se->opaque, version_id);
+return se->ops->load_state(f, se->opaque, se->load_version_id);
 }
-return vmstate_load_state(f, se->vmsd, se->opaque, version_id);
+return vmstate_load_state(f, se->vmsd, se->opaque, se->load_version_id);
 }
 
 static void vmstate_save_old_style(QEMUFile *f, SaveStateEntry *se, QJSON 
*vmdesc)
@@ -1882,7 +1882,7 @@ qemu_loadvm_section_start_full(QEMUFile *f, 
MigrationIncomingState *mis)
 return -EINVAL;
 }
 
-ret = vmstate_load(f, se, se->load_version_id);
+ret = vmstate_load(f, se);
 if (ret < 0) {
 error_report("error while loading state for instance 0x%x of"
  " device '%s'", instance_id, idstr);
@@ -1915,7 +1915,7 @@ qemu_loadvm_section_part_end(QEMUFile *f, 
MigrationIncomingState *mis)
 return -EINVAL;
 }
 
-ret = vmstate_load(f, se, se->load_version_id);
+ret = vmstate_load(f, se);
 if (ret < 0) {
 error_report("error while loading state section id %d(%s)",
  section_id, se->idstr);
-- 
2.9.4

[Qemu-devel] [PULL 09/15] migration: Export fd.c functions in its own file

[Juan Quintela]

Signed-off-by: Juan Quintela 
Reviewed-by: Dr. David Alan Gilbert 
---
 include/migration/migration.h |  4 
 migration/fd.c|  1 +
 migration/fd.h| 23 +++
 migration/migration.c |  1 +
 4 files changed, 25 insertions(+), 4 deletions(-)
 create mode 100644 migration/fd.h

diff --git a/include/migration/migration.h b/include/migration/migration.h
index d04f045..d249c13 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -171,10 +171,6 @@ void unix_start_incoming_migration(const char *path, Error 
**errp);
 
 void unix_start_outgoing_migration(MigrationState *s, const char *path, Error 
**errp);
 
-void fd_start_incoming_migration(const char *path, Error **errp);
-
-void fd_start_outgoing_migration(MigrationState *s, const char *fdname, Error 
**errp);
-
 void rdma_start_outgoing_migration(void *opaque, const char *host_port, Error 
**errp);
 
 void rdma_start_incoming_migration(const char *host_port, Error **errp);
diff --git a/migration/fd.c b/migration/fd.c
index 05e0a5c..0077a50 100644
--- a/migration/fd.c
+++ b/migration/fd.c
@@ -18,6 +18,7 @@
 #include "qapi/error.h"
 #include "qemu-common.h"
 #include "channel.h"
+#include "fd.h"
 #include "migration/migration.h"
 #include "monitor/monitor.h"
 #include "io/channel-util.h"
diff --git a/migration/fd.h b/migration/fd.h
new file mode 100644
index 000..a14a63c
--- /dev/null
+++ b/migration/fd.h
@@ -0,0 +1,23 @@
+/*
+ * QEMU live migration via generic fd
+ *
+ * Copyright Red Hat, Inc. 2009-2016
+ *
+ * Authors:
+ *  Chris Lalancette 
+ *  Daniel P. Berrange 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ *
+ * Contributions after 2012-01-13 are licensed under the terms of the
+ * GNU GPL, version 2 or (at your option) any later version.
+ */
+
+#ifndef QEMU_MIGRATION_FD_H
+#define QEMU_MIGRATION_FD_H
+void fd_start_incoming_migration(const char *path, Error **errp);
+
+void fd_start_outgoing_migration(MigrationState *s, const char *fdname,
+ Error **errp);
+#endif
diff --git a/migration/migration.c b/migration/migration.c
index c400388..ec93f7a 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -19,6 +19,7 @@
 #include "qemu/main-loop.h"
 #include "migration/blocker.h"
 #include "exec.h"
+#include "fd.h"
 #include "migration/migration.h"
 #include "savevm.h"
 #include "qemu-file-channel.h"
-- 
2.9.4

[Qemu-devel] [PULL 05/15] migration: shut src return path unconditionally

[Juan Quintela]

From: Peter Xu 

We were do the shutting off only for postcopy. Now we do this as long as
the source return path is there.

Moving the cleanup of from_src_file there too.

Signed-off-by: Peter Xu 
Reviewed-by: Juan Quintela 
Signed-off-by: Juan Quintela 
---
 migration/migration.c| 8 +++-
 migration/postcopy-ram.c | 1 -
 2 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/migration/migration.c b/migration/migration.c
index b90e399..5d9ccf1 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -134,10 +134,17 @@ void migration_incoming_state_destroy(void)
 struct MigrationIncomingState *mis = migration_incoming_get_current();
 
 if (mis->to_src_file) {
+/* Tell source that we are done */
+migrate_send_rp_shut(mis, qemu_file_get_error(mis->from_src_file) != 
0);
 qemu_fclose(mis->to_src_file);
 mis->to_src_file = NULL;
 }
 
+if (mis->from_src_file) {
+qemu_fclose(mis->from_src_file);
+mis->from_src_file = NULL;
+}
+
 qemu_event_destroy(>main_thread_load_event);
 }
 
@@ -435,7 +442,6 @@ static void process_incoming_migration_co(void *opaque)
 exit(EXIT_FAILURE);
 }
 
-qemu_fclose(f);
 free_xbzrle_decoded_buf();
 
 mis->bh = qemu_bh_new(process_incoming_migration_bh, mis);
diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c
index 3f9ae1b..5ceb623 100644
--- a/migration/postcopy-ram.c
+++ b/migration/postcopy-ram.c
@@ -333,7 +333,6 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingState 
*mis)
 }
 
 postcopy_state_set(POSTCOPY_INCOMING_END);
-migrate_send_rp_shut(mis, qemu_file_get_error(mis->from_src_file) != 0);
 
 if (mis->postcopy_tmp_page) {
 munmap(mis->postcopy_tmp_page, mis->largest_page_size);
-- 
2.9.4

[Qemu-devel] [PULL 07/15] migration: Split qemu-file.h

[Juan Quintela]

Split the file into public and internal interfaces.  I have to rename
the external one because we can't have two include files with the same
name in the same directory.  Build system gets confused.  The only
exported functions are the ones that handle basic types.

Signed-off-by: Juan Quintela 
Reviewed-by: Dr. David Alan Gilbert 
---
 hw/i2c/i2c-ddc.c |   1 +
 hw/intc/s390_flic.c  |   1 -
 hw/intc/s390_flic_kvm.c  |   1 -
 hw/s390x/s390-skeys.c|   1 -
 include/hw/hw.h  |   2 +-
 include/migration/qemu-file-types.h  | 164 +++
 migration/block.c|   2 +-
 migration/colo.c |   2 +-
 migration/migration.c|   2 +-
 migration/postcopy-ram.c |   2 +-
 migration/qemu-file-channel.c|   3 +-
 migration/qemu-file.c|   2 +-
 {include/migration => migration}/qemu-file.h | 157 +
 migration/ram.c  |   2 +-
 migration/rdma.c |   2 +-
 migration/savevm.c   |   1 +
 migration/socket.c   |   2 +-
 migration/vmstate-types.c|   3 +-
 migration/vmstate.c  |   2 +-
 tests/test-vmstate.c |   3 +-
 20 files changed, 186 insertions(+), 169 deletions(-)
 create mode 100644 include/migration/qemu-file-types.h
 rename {include/migration => migration}/qemu-file.h (62%)

diff --git a/hw/i2c/i2c-ddc.c b/hw/i2c/i2c-ddc.c
index 66899d7..6b92e95 100644
--- a/hw/i2c/i2c-ddc.c
+++ b/hw/i2c/i2c-ddc.c
@@ -17,6 +17,7 @@
  */
 
 #include "qemu/osdep.h"
+#include "qemu-common.h"
 #include "qemu/log.h"
 #include "hw/i2c/i2c.h"
 #include "hw/i2c/i2c-ddc.h"
diff --git a/hw/intc/s390_flic.c b/hw/intc/s390_flic.c
index 711c114..a26e906 100644
--- a/hw/intc/s390_flic.c
+++ b/hw/intc/s390_flic.c
@@ -13,7 +13,6 @@
 #include "qemu/osdep.h"
 #include "qemu/error-report.h"
 #include "hw/sysbus.h"
-#include "migration/qemu-file.h"
 #include "hw/s390x/s390_flic.h"
 #include "trace.h"
 #include "hw/qdev.h"
diff --git a/hw/intc/s390_flic_kvm.c b/hw/intc/s390_flic_kvm.c
index cc44bc4..b4c61d8 100644
--- a/hw/intc/s390_flic_kvm.c
+++ b/hw/intc/s390_flic_kvm.c
@@ -17,7 +17,6 @@
 #include "qemu/error-report.h"
 #include "hw/sysbus.h"
 #include "sysemu/kvm.h"
-#include "migration/qemu-file.h"
 #include "hw/s390x/s390_flic.h"
 #include "hw/s390x/adapter.h"
 #include "trace.h"
diff --git a/hw/s390x/s390-skeys.c b/hw/s390x/s390-skeys.c
index e2d4e1a..619152c 100644
--- a/hw/s390x/s390-skeys.c
+++ b/hw/s390x/s390-skeys.c
@@ -12,7 +12,6 @@
 #include "qemu/osdep.h"
 #include "hw/boards.h"
 #include "qmp-commands.h"
-#include "migration/qemu-file.h"
 #include "hw/s390x/storage-keys.h"
 #include "qemu/error-report.h"
 #include "sysemu/kvm.h"
diff --git a/include/hw/hw.h b/include/hw/hw.h
index af9eae1..ab4950c 100644
--- a/include/hw/hw.h
+++ b/include/hw/hw.h
@@ -11,7 +11,7 @@
 #include "exec/memory.h"
 #include "hw/irq.h"
 #include "migration/vmstate.h"
-#include "migration/qemu-file.h"
+#include "migration/qemu-file-types.h"
 #include "qemu/module.h"
 #include "sysemu/reset.h"
 
diff --git a/include/migration/qemu-file-types.h 
b/include/migration/qemu-file-types.h
new file mode 100644
index 000..bd6d7dd
--- /dev/null
+++ b/include/migration/qemu-file-types.h
@@ -0,0 +1,164 @@
+/*
+ * QEMU System Emulator
+ *
+ * Copyright (c) 2003-2008 Fabrice Bellard
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to 
deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING 
FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+
+#ifndef QEMU_FILE_H
+#define QEMU_FILE_H
+
+void qemu_put_buffer(QEMUFile *f, const uint8_t *buf, size_t size);
+void qemu_put_byte(QEMUFile *f, int v);
+
+#define qemu_put_sbyte qemu_put_byte
+
+void 

[Qemu-devel] [PULL 02/15] migration: loadvm handlers are not used

[Juan Quintela]

So we remove all traces of them.

Signed-off-by: Juan Quintela 
Reviewed-by: Laurent Vivier 
Reviewed-by: Peter Xu 
Reviewed-by: Dr. David Alan Gilbert 
---
 include/migration/migration.h |  5 -
 include/migration/vmstate.h   |  2 --
 include/qemu/typedefs.h   |  1 -
 migration/migration.c |  2 --
 migration/savevm.c| 26 --
 5 files changed, 36 deletions(-)

diff --git a/include/migration/migration.h b/include/migration/migration.h
index 0e807b6..d1a353a 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -50,8 +50,6 @@ enum mig_rp_message_type {
 MIG_RP_MSG_MAX
 };
 
-typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
-
 /* State for the incoming migration */
 struct MigrationIncomingState {
 QEMUFile *from_src_file;
@@ -89,9 +87,6 @@ struct MigrationIncomingState {
 /* The coroutine we should enter (back) after failover */
 Coroutine *migration_incoming_co;
 QemuSemaphore colo_incoming_sem;
-
-/* See savevm.c */
-LoadStateEntry_Head loadvm_handlers;
 };
 
 MigrationIncomingState *migration_incoming_get_current(void);
diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
index f97411d..6689562 100644
--- a/include/migration/vmstate.h
+++ b/include/migration/vmstate.h
@@ -1020,8 +1020,6 @@ extern const VMStateInfo vmstate_info_qtailq;
 
 #define SELF_ANNOUNCE_ROUNDS 5
 
-void loadvm_free_handlers(MigrationIncomingState *mis);
-
 int vmstate_load_state(QEMUFile *f, const VMStateDescription *vmsd,
void *opaque, int version_id);
 void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
diff --git a/include/qemu/typedefs.h b/include/qemu/typedefs.h
index 33a6aa1..51958bf 100644
--- a/include/qemu/typedefs.h
+++ b/include/qemu/typedefs.h
@@ -39,7 +39,6 @@ typedef struct I2SCodec I2SCodec;
 typedef struct ISABus ISABus;
 typedef struct ISADevice ISADevice;
 typedef struct IsaDma IsaDma;
-typedef struct LoadStateEntry LoadStateEntry;
 typedef struct MACAddr MACAddr;
 typedef struct MachineClass MachineClass;
 typedef struct MachineState MachineState;
diff --git a/migration/migration.c b/migration/migration.c
index 7087d1a..c3218cd 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -122,7 +122,6 @@ MigrationIncomingState *migration_incoming_get_current(void)
 if (!once) {
 mis_current.state = MIGRATION_STATUS_NONE;
 memset(_current, 0, sizeof(MigrationIncomingState));
-QLIST_INIT(_current.loadvm_handlers);
 qemu_mutex_init(_current.rp_mutex);
 qemu_event_init(_current.main_thread_load_event, false);
 once = true;
@@ -135,7 +134,6 @@ void migration_incoming_state_destroy(void)
 struct MigrationIncomingState *mis = migration_incoming_get_current();
 
 qemu_event_destroy(>main_thread_load_event);
-loadvm_free_handlers(mis);
 }
 
 
diff --git a/migration/savevm.c b/migration/savevm.c
index 07646f4..d96209b 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -1804,13 +1804,6 @@ static int loadvm_process_command(QEMUFile *f)
 return 0;
 }
 
-struct LoadStateEntry {
-QLIST_ENTRY(LoadStateEntry) entry;
-SaveStateEntry *se;
-int section_id;
-int version_id;
-};
-
 /*
  * Read a footer off the wire and check that it matches the expected section
  *
@@ -1846,22 +1839,11 @@ static bool check_section_footer(QEMUFile *f, 
SaveStateEntry *se)
 return true;
 }
 
-void loadvm_free_handlers(MigrationIncomingState *mis)
-{
-LoadStateEntry *le, *new_le;
-
-QLIST_FOREACH_SAFE(le, >loadvm_handlers, entry, new_le) {
-QLIST_REMOVE(le, entry);
-g_free(le);
-}
-}
-
 static int
 qemu_loadvm_section_start_full(QEMUFile *f, MigrationIncomingState *mis)
 {
 uint32_t instance_id, version_id, section_id;
 SaveStateEntry *se;
-LoadStateEntry *le;
 char idstr[256];
 int ret;
 
@@ -1900,14 +1882,6 @@ qemu_loadvm_section_start_full(QEMUFile *f, 
MigrationIncomingState *mis)
 return -EINVAL;
 }
 
-/* Add entry */
-le = g_malloc0(sizeof(*le));
-
-le->se = se;
-le->section_id = section_id;
-le->version_id = version_id;
-QLIST_INSERT_HEAD(>loadvm_handlers, le, entry);
-
 ret = vmstate_load(f, se, se->load_version_id);
 if (ret < 0) {
 error_report("error while loading state for instance 0x%x of"
-- 
2.9.4

[Qemu-devel] [PULL 01/15] migration: Use savevm_handlers instead of loadvm copy

[Juan Quintela]

There is no reason for having the loadvm_handlers at all.  There is
only one use, and we can use the savevm handlers.

We will remove the loadvm handlers on a following patch.

Signed-off-by: Juan Quintela 
Reviewed-by: Laurent Vivier 
Reviewed-by: Peter Xu 
Reviewed-by: Dr. David Alan Gilbert 

--

- Added load_version_id: version_id read from the stream (laurent)
- Added load_section_id: section_id read from the stream (dave)
---
 migration/savevm.c | 32 +++-
 1 file changed, 19 insertions(+), 13 deletions(-)

diff --git a/migration/savevm.c b/migration/savevm.c
index a2d4f9c..07646f4 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -272,7 +272,11 @@ typedef struct SaveStateEntry {
 int instance_id;
 int alias_id;
 int version_id;
+/* version id read from the stream */
+int load_version_id;
 int section_id;
+/* section id read from the stream */
+int load_section_id;
 SaveVMHandlers *ops;
 const VMStateDescription *vmsd;
 void *opaque;
@@ -1813,7 +1817,7 @@ struct LoadStateEntry {
  * Returns: true if the footer was good
  *  false if there is a problem (and calls error_report to say why)
  */
-static bool check_section_footer(QEMUFile *f, LoadStateEntry *le)
+static bool check_section_footer(QEMUFile *f, SaveStateEntry *se)
 {
 uint8_t read_mark;
 uint32_t read_section_id;
@@ -1826,15 +1830,15 @@ static bool check_section_footer(QEMUFile *f, 
LoadStateEntry *le)
 read_mark = qemu_get_byte(f);
 
 if (read_mark != QEMU_VM_SECTION_FOOTER) {
-error_report("Missing section footer for %s", le->se->idstr);
+error_report("Missing section footer for %s", se->idstr);
 return false;
 }
 
 read_section_id = qemu_get_be32(f);
-if (read_section_id != le->section_id) {
+if (read_section_id != se->load_section_id) {
 error_report("Mismatched section id in footer for %s -"
  " read 0x%x expected 0x%x",
- le->se->idstr, read_section_id, le->section_id);
+ se->idstr, read_section_id, se->load_section_id);
 return false;
 }
 
@@ -1887,6 +1891,8 @@ qemu_loadvm_section_start_full(QEMUFile *f, 
MigrationIncomingState *mis)
  version_id, idstr, se->version_id);
 return -EINVAL;
 }
+se->load_version_id = version_id;
+se->load_section_id = section_id;
 
 /* Validate if it is a device's state */
 if (xen_enabled() && se->is_ram) {
@@ -1902,13 +1908,13 @@ qemu_loadvm_section_start_full(QEMUFile *f, 
MigrationIncomingState *mis)
 le->version_id = version_id;
 QLIST_INSERT_HEAD(>loadvm_handlers, le, entry);
 
-ret = vmstate_load(f, le->se, le->version_id);
+ret = vmstate_load(f, se, se->load_version_id);
 if (ret < 0) {
 error_report("error while loading state for instance 0x%x of"
  " device '%s'", instance_id, idstr);
 return ret;
 }
-if (!check_section_footer(f, le)) {
+if (!check_section_footer(f, se)) {
 return -EINVAL;
 }
 
@@ -1919,29 +1925,29 @@ static int
 qemu_loadvm_section_part_end(QEMUFile *f, MigrationIncomingState *mis)
 {
 uint32_t section_id;
-LoadStateEntry *le;
+SaveStateEntry *se;
 int ret;
 
 section_id = qemu_get_be32(f);
 
 trace_qemu_loadvm_state_section_partend(section_id);
-QLIST_FOREACH(le, >loadvm_handlers, entry) {
-if (le->section_id == section_id) {
+QTAILQ_FOREACH(se, _state.handlers, entry) {
+if (se->load_section_id == section_id) {
 break;
 }
 }
-if (le == NULL) {
+if (se == NULL) {
 error_report("Unknown savevm section %d", section_id);
 return -EINVAL;
 }
 
-ret = vmstate_load(f, le->se, le->version_id);
+ret = vmstate_load(f, se, se->load_version_id);
 if (ret < 0) {
 error_report("error while loading state section id %d(%s)",
- section_id, le->se->idstr);
+ section_id, se->idstr);
 return ret;
 }
-if (!check_section_footer(f, le)) {
+if (!check_section_footer(f, se)) {
 return -EINVAL;
 }
 
-- 
2.9.4

[Qemu-devel] [PULL 00/15] Migration PULL request

[Juan Quintela]

Hi
This pull requset includes:
- fd leaks for return patch (peter)
- remove loadvm_handlers
- cleanup of include files for migration

Please, apply.

Thanks, Juan.


The following changes since commit c077a998eb3fcae2d048e3baeb5bc592d30fddde:

  Merge remote-tracking branch 'remotes/riku/tags/pull-linux-user-20170531' 
into staging (2017-06-01 15:50:40 +0100)

are available in the git repository at:

  git://github.com/juanquintela/qemu.git tags/migration/20170601

for you to fetch changes up to 2c9e6fec89ff032a3f75a5a1caccc31901fb4056:

  migration: Move include/migration/block.h into migration/ (2017-06-01 
18:49:24 +0200)


migration/next for 20170601


Juan Quintela (13):
  migration: Use savevm_handlers instead of loadvm copy
  migration: loadvm handlers are not used
  migration: Remove section_id parameter from vmstate_load
  migration: Remove unneeded includes of migration/vmstate.h
  migration: Split qemu-file.h
  migration: Export exec.c functions in its own file
  migration: Export fd.c functions in its own file
  migration: Export socket.c functions in its own file
  migration: Export tls.c functions in its own file
  migration: Export rdma.c functions in its own file
  migration: Create include for migration snapshots
  migration: Export ram.c functions in its own file
  migration: Move include/migration/block.h into migration/

Peter Xu (2):
  migration: fix leak of src file on dst
  migration: shut src return path unconditionally

 hmp.c|   5 +-
 hw/core/qdev.c   |   1 -
 hw/i2c/i2c-ddc.c |   1 +
 hw/intc/s390_flic.c  |   1 -
 hw/intc/s390_flic_kvm.c  |   1 -
 hw/s390x/s390-skeys.c|   1 -
 include/block/block_int.h|   4 +-
 include/hw/acpi/memory_hotplug.h |   1 -
 include/hw/acpi/pcihp.h  |   1 -
 include/hw/hw.h  |   2 +-
 include/hw/pci/shpc.h|   1 -
 include/migration/migration.h|  73 
 include/migration/misc.h |  29 +
 include/migration/qemu-file-types.h  | 164 +++
 include/migration/snapshot.h |  21 
 include/migration/vmstate.h  |   2 -
 include/qemu/typedefs.h  |   1 -
 include/sysemu/sysemu.h  |   3 -
 migration/block.c|   5 +-
 {include/migration => migration}/block.h |   2 -
 migration/channel.c  |   1 +
 migration/colo.c |   4 +-
 migration/exec.c |   1 +
 migration/exec.h |  26 +
 migration/fd.c   |   1 +
 migration/fd.h   |  23 
 migration/migration.c|  25 +++-
 migration/postcopy-ram.c |   4 +-
 migration/qemu-file-channel.c|   3 +-
 migration/qemu-file.c|   2 +-
 {include/migration => migration}/qemu-file.h | 157 +
 migration/ram.c  |   4 +-
 migration/ram.h  |  70 
 migration/rdma.c |   5 +-
 migration/rdma.h |  25 
 migration/savevm.c   |  71 +---
 migration/socket.c   |   3 +-
 migration/socket.h   |  28 +
 migration/tls.c  |   1 +
 migration/tls.h  |  34 ++
 migration/vmstate-types.c|   3 +-
 migration/vmstate.c  |   2 +-
 replay/replay-snapshot.c |   5 +-
 target/alpha/cpu.c   |   1 -
 target/hppa/cpu.c|   1 -
 target/s390x/cpu.c   |   1 -
 target/tilegx/cpu.c  |   1 -
 tests/test-vmstate.c |   3 +-
 vl.c |   5 +-
 49 files changed, 510 insertions(+), 319 deletions(-)
 create mode 100644 include/migration/misc.h
 create mode 100644 include/migration/qemu-file-types.h
 create mode 100644 include/migration/snapshot.h
 rename {include/migration => migration}/block.h (93%)
 create mode 100644 migration/exec.h
 create mode 100644 migration/fd.h
 rename {include/migration => migration}/qemu-file.h (62%)
 create mode 100644 migration/ram.h
 create mode 100644 migration/rdma.h
 create mode 100644 migration/socket.h
 create mode 1006

Re: [Qemu-devel] [PATCH] msi: remove return code for msi_init()

[Aurelien Jarno]

On 2017-06-01 16:23, Paolo Bonzini wrote:
> 
> 
> On 01/06/2017 10:27, Marcel Apfelbaum wrote:
> > On 31/05/2017 11:28, Paolo Bonzini wrote:
> >> No, for now I'd rather just go and remove msi_nonbroken.  When someone
> >> reports a bug, we can add back "msi_broken".
> > 
> > Hi,
> > I agree with the direction, but I am concerned msi_nonbroken is there
> > for a reason.
> > We might break some (obscure/not in use) machine.
> > Maybe we should CC all arch machine maintainers/contributors to give
> > them a chance to object...
> 
> Yeah, Alpha, MIPS and SH are those that support PCI.  Adding Richard and
> Aurelien, do your platforms support MSI on real hardware but not in QEMU?

SH clearly doesn't support MSI.

The oldest MIPS board also do not support MSI, but I guess the Boston
board might support it. I am adding Paul Burton in Cc: who probably
knows about that.

Aurelien

-- 
Aurelien Jarno  GPG: 4096R/1DDD8C9B
aurel...@aurel32.net http://www.aurel32.net

Re: [Qemu-devel] [PATCH v1] s390x/cpumodel: wire up cpu type + id for TCG

[Aurelien Jarno]

On 2017-06-01 12:14, David Hildenbrand wrote:
> Let's properly expose the CPU type (machine-type number) via "STORE CPU
> ID" and "STORE SUBSYSTEM INFORMATION".
> 
> As TCG emulates basic mode, the CPU identification number has the format
> "An", whereby A is the CPU address, and n are parts of the CPU serial
> number (0 for us for now).
> 
> Signed-off-by: David Hildenbrand 
> ---
> 
> Tested stidp with a kvm-unit-test that is still being worked on (waiting
> for Thomas' interception test to integrate). I think we are missing quite
> some "operand alignment checks" in other handlers, too.
> 
> ---
>  target/s390x/cpu.h |  1 -
>  target/s390x/cpu_models.c  |  2 --
>  target/s390x/helper.h  |  1 +
>  target/s390x/insn-data.def |  2 +-
>  target/s390x/misc_helper.c | 26 +++---
>  target/s390x/translate.c   | 11 ---
>  6 files changed, 29 insertions(+), 14 deletions(-)
> 
> diff --git a/target/s390x/cpu.h b/target/s390x/cpu.h
> index c74b419..02bd8bf 100644
> --- a/target/s390x/cpu.h
> +++ b/target/s390x/cpu.h
> @@ -147,7 +147,6 @@ typedef struct CPUS390XState {
>  CPU_COMMON
>  
>  uint32_t cpu_num;
> -uint32_t machine_type;
>  
>  uint64_t tod_offset;
>  uint64_t tod_basetime;
> diff --git a/target/s390x/cpu_models.c b/target/s390x/cpu_models.c
> index b6220c8..99ec0c8 100644
> --- a/target/s390x/cpu_models.c
> +++ b/target/s390x/cpu_models.c
> @@ -710,8 +710,6 @@ static inline void apply_cpu_model(const S390CPUModel 
> *model, Error **errp)
>  
>  if (kvm_enabled()) {
>  kvm_s390_apply_cpu_model(model, errp);
> -} else if (model) {
> -/* FIXME TCG - use data for stdip/stfl */
>  }
>  
>  if (!*errp) {
> diff --git a/target/s390x/helper.h b/target/s390x/helper.h
> index 0b70770..0c8f745 100644
> --- a/target/s390x/helper.h
> +++ b/target/s390x/helper.h
> @@ -121,6 +121,7 @@ DEF_HELPER_FLAGS_3(sturg, TCG_CALL_NO_WG, void, env, i64, 
> i64)
>  DEF_HELPER_1(per_check_exception, void, env)
>  DEF_HELPER_FLAGS_3(per_branch, TCG_CALL_NO_RWG, void, env, i64, i64)
>  DEF_HELPER_FLAGS_2(per_ifetch, TCG_CALL_NO_RWG, void, env, i64)
> +DEF_HELPER_2(stidp, void, env, i64)
>  
>  DEF_HELPER_2(xsch, void, env, i64)
>  DEF_HELPER_2(csch, void, env, i64)
> diff --git a/target/s390x/insn-data.def b/target/s390x/insn-data.def
> index 55a7c52..83e7d01 100644
> --- a/target/s390x/insn-data.def
> +++ b/target/s390x/insn-data.def
> @@ -902,7 +902,7 @@
>  /* STORE CPU ADDRESS */
>  C(0xb212, STAP,S, Z,   la2, 0, new, m1_16, stap, 0)
>  /* STORE CPU ID */
> -C(0xb202, STIDP,   S, Z,   la2, 0, new, m1_64, stidp, 0)
> +C(0xb202, STIDP,   S, Z,   0, a2, 0, 0, stidp, 0)
>  /* STORE CPU TIMER */
>  C(0xb209, STPT,S, Z,   la2, 0, new, m1_64, stpt, 0)
>  /* STORE FACILITY LIST */
> diff --git a/target/s390x/misc_helper.c b/target/s390x/misc_helper.c
> index 1b9f448..f682511 100644
> --- a/target/s390x/misc_helper.c
> +++ b/target/s390x/misc_helper.c
> @@ -383,6 +383,7 @@ uint64_t HELPER(stpt)(CPUS390XState *env)
>  uint32_t HELPER(stsi)(CPUS390XState *env, uint64_t a0,
>uint64_t r0, uint64_t r1)
>  {
> +S390CPU *cpu = s390_env_get_cpu(env);
>  int cc = 0;
>  int sel1, sel2;
>  
> @@ -402,12 +403,14 @@ uint32_t HELPER(stsi)(CPUS390XState *env, uint64_t a0,
>  if ((sel1 == 1) && (sel2 == 1)) {
>  /* Basic Machine Configuration */
>  struct sysib_111 sysib;
> +char type[5] = {};
>  
>  memset(, 0, sizeof(sysib));
>  ebcdic_put(sysib.manuf, "QEMU", 16);
> -/* same as machine type number in STORE CPU ID */
> -ebcdic_put(sysib.type, "QEMU", 4);
> -/* same as model number in STORE CPU ID */
> +/* same as machine type number in STORE CPU ID, but in EBCDIC */
> +snprintf(type, ARRAY_SIZE(type), "%X", cpu->model->def->type);
> +ebcdic_put(sysib.type, type, 4);
> +/* model number (not stored in STORE CPU ID for z/Architecure) */
>  ebcdic_put(sysib.model, "QEMU", 16);
>  ebcdic_put(sysib.sequence, "QEMU", 16);
>  ebcdic_put(sysib.plant, "QEMU", 4);
> @@ -736,3 +739,20 @@ uint32_t HELPER(stfle)(CPUS390XState *env, uint64_t addr)
>  env->regs[0] = deposit64(env->regs[0], 0, 8, max_m1);
>  return (count_m1 >= max_m1 ? 0 : 3);
>  }
> +
> +#ifndef CONFIG_USER_ONLY
> +void HELPER(stidp)(CPUS390XState *env, uint64_t addr)
> +{
> +S390CPU *cpu = s390_env_get_cpu(env);
> +uint64_t cpuid = s390_cpuid_from_cpu_model(cpu->model);
> +
> +if (addr & 0x7) {
> +program_interrupt(env, PGM_SPECIFICATION, ILEN_LATER_INC);
> +return;
> +}
> +
> +/* basic mode, write the cpu address into the first 4 bit of the ID */
> +cpuid |= ((uint64_t)env->cpu_num & 0xf) << 54;
> +cpu_stq_data(env, addr, cpuid);
> +}
> 

Re: [Qemu-devel] [PATCH v3 30/30] target/s390x: update maximum TCG model to z800

[Aurelien Jarno]

On 2017-06-01 11:04, David Hildenbrand wrote:
> On 01.06.2017 10:38, David Hildenbrand wrote:
> > On 01.06.2017 00:01, Aurelien Jarno wrote:
> >> At the same time fix the TCG version of get_max_cpu_model to return the
> >> maximum model like on KVM. Remove the ETF2 and long-displacement
> > 
> > I don't understand the part
> > "fix the TCG version of get_max_cpu_model to return the maximum model
> > like on KVM".
> > 
> > Can you elaborate?
> > 
> >> facilities from the additional features as it is included in the z800.
> >>
> >> Signed-off-by: Aurelien Jarno 
> >> ---
> >>  target/s390x/cpu_models.c | 13 ++---
> >>  1 file changed, 6 insertions(+), 7 deletions(-)
> >>
> >> diff --git a/target/s390x/cpu_models.c b/target/s390x/cpu_models.c
> >> index fc3cb25cc3..c13bbd852c 100644
> >> --- a/target/s390x/cpu_models.c
> >> +++ b/target/s390x/cpu_models.c
> >> @@ -668,8 +668,6 @@ static void add_qemu_cpu_model_features(S390FeatBitmap 
> >> fbm)
> >>  static const int feats[] = {
> >>  S390_FEAT_STFLE,
> >>  S390_FEAT_EXTENDED_IMMEDIATE,
> >> -S390_FEAT_EXTENDED_TRANSLATION_2,
> >> -S390_FEAT_LONG_DISPLACEMENT,
> >>  S390_FEAT_LONG_DISPLACEMENT_FAST,
> >>  S390_FEAT_ETF2_ENH,
> >>  S390_FEAT_STORE_CLOCK_FAST,
> >> @@ -696,9 +694,9 @@ static S390CPUModel *get_max_cpu_model(Error **errp)
> >>  if (kvm_enabled()) {
> >>  kvm_s390_get_host_cpu_model(_model, errp);
> >>  } else {
> >> -/* TCG emulates a z900 (with some optional additional features) */
> >> -max_model.def = _cpu_defs[0];
> >> -bitmap_copy(max_model.features, max_model.def->default_feat,
> >> +/* TCG emulates a z800 (with some optional additional features) */
> >> +max_model.def = s390_find_cpu_def(0x2066, 7, 3, NULL);
> >> +bitmap_copy(max_model.features, max_model.def->full_feat,
> >>  S390_FEAT_MAX);
> 
> This is most likely wrong: you're indicating features here that are not
> available on tcg. esp. S390_FEAT_SIE_F2 and friends.
> 
> I think should only copy the base features and add whatever else is
> available via add_qemu_cpu_model_features() as already done.

The patch series added all the z800 features exposed via STFL/STFLE.
Indeed the SIE features are missing, but anyway QEMU doesn't emulate SIE
at all so the lack of these features are not exposed to the guest. In that
regard QEMU already wrongly claim to emulate a z900.


> >>  add_qemu_cpu_model_features(max_model.features);
> >>  }
> >> @@ -956,8 +954,9 @@ static void s390_qemu_cpu_model_initfn(Object *obj)
> >>  S390CPU *cpu = S390_CPU(obj);
> >>  
> >>  cpu->model = g_malloc0(sizeof(*cpu->model));
> >> -/* TCG emulates a z900 (with some optional additional features) */
> >> -memcpy(_qemu_cpu_defs, _cpu_defs[0], 
> >> sizeof(s390_qemu_cpu_defs));
> >> +/* TCG emulates a z800 (with some optional additional features) */
> >> +memcpy(_qemu_cpu_defs, s390_find_cpu_def(0x2066, 7, 3, NULL),
> >> +   sizeof(s390_qemu_cpu_defs));
> > 
> > No changing the qemu model without compatibility handling.
> > 
> Please have a look at the following mail for a possible solution:
> 
> https://lists.gnu.org/archive/html/qemu-devel/2017-05/msg06030.html
> 
> This could be moved to a separate patch. So this patch really should
> just care about the maximum model, not the qemu model.

From what I understand from this thread, the patch from Thomas Huth was
finally considered acceptable. I am adding him in Cc: so that he can
comment.

Aurelien

-- 
Aurelien Jarno  GPG: 4096R/1DDD8C9B
aurel...@aurel32.net http://www.aurel32.net

Re: [Qemu-devel] [PATCH v3 30/30] target/s390x: update maximum TCG model to z800

[Aurelien Jarno]

On 2017-06-01 10:38, David Hildenbrand wrote:
> On 01.06.2017 00:01, Aurelien Jarno wrote:
> > At the same time fix the TCG version of get_max_cpu_model to return the
> > maximum model like on KVM. Remove the ETF2 and long-displacement
> 
> I don't understand the part
> "fix the TCG version of get_max_cpu_model to return the maximum model
> like on KVM".
> 
> Can you elaborate?

Currently get_max_cpu_model returns the features of the base model, so
for example the one of a z900 even on a z800. This makes impossible to
enable the features that are provided by a z800 like etf2 or ldisp.

For what I understand from the KVM code (but I haven't tested), the
function return all the features that are supported by the current CPU,
not all the features that are supported by the base model of the current
CPU.


> > facilities from the additional features as it is included in the z800.
> > 
> > Signed-off-by: Aurelien Jarno 
> > ---
> >  target/s390x/cpu_models.c | 13 ++---
> >  1 file changed, 6 insertions(+), 7 deletions(-)
> > 
> > diff --git a/target/s390x/cpu_models.c b/target/s390x/cpu_models.c
> > index fc3cb25cc3..c13bbd852c 100644
> > --- a/target/s390x/cpu_models.c
> > +++ b/target/s390x/cpu_models.c
> > @@ -668,8 +668,6 @@ static void add_qemu_cpu_model_features(S390FeatBitmap 
> > fbm)
> >  static const int feats[] = {
> >  S390_FEAT_STFLE,
> >  S390_FEAT_EXTENDED_IMMEDIATE,
> > -S390_FEAT_EXTENDED_TRANSLATION_2,
> > -S390_FEAT_LONG_DISPLACEMENT,
> >  S390_FEAT_LONG_DISPLACEMENT_FAST,
> >  S390_FEAT_ETF2_ENH,
> >  S390_FEAT_STORE_CLOCK_FAST,
> > @@ -696,9 +694,9 @@ static S390CPUModel *get_max_cpu_model(Error **errp)
> >  if (kvm_enabled()) {
> >  kvm_s390_get_host_cpu_model(_model, errp);
> >  } else {
> > -/* TCG emulates a z900 (with some optional additional features) */
> > -max_model.def = _cpu_defs[0];
> > -bitmap_copy(max_model.features, max_model.def->default_feat,
> > +/* TCG emulates a z800 (with some optional additional features) */
> > +max_model.def = s390_find_cpu_def(0x2066, 7, 3, NULL);
> > +bitmap_copy(max_model.features, max_model.def->full_feat,
> >  S390_FEAT_MAX);
> >  add_qemu_cpu_model_features(max_model.features);
> >  }
> > @@ -956,8 +954,9 @@ static void s390_qemu_cpu_model_initfn(Object *obj)
> >  S390CPU *cpu = S390_CPU(obj);
> >  
> >  cpu->model = g_malloc0(sizeof(*cpu->model));
> > -/* TCG emulates a z900 (with some optional additional features) */
> > -memcpy(_qemu_cpu_defs, _cpu_defs[0], 
> > sizeof(s390_qemu_cpu_defs));
> > +/* TCG emulates a z800 (with some optional additional features) */
> > +memcpy(_qemu_cpu_defs, s390_find_cpu_def(0x2066, 7, 3, NULL),
> > +   sizeof(s390_qemu_cpu_defs));
> 
> No changing the qemu model without compatibility handling.

This patch series is based on the patch from Thomas Huth. It means the
QEMU model is still based on a z900, but that it is possible to enable
some more features like etf2.

Aurelien

-- 
Aurelien Jarno  GPG: 4096R/1DDD8C9B
aurel...@aurel32.net http://www.aurel32.net

Re: [Qemu-devel] [PATCH v2 01/15] block: introduce BDRV_REQ_ALLOCATE flag

[Eric Blake]

On 06/01/2017 10:14 AM, Anton Nefedov wrote:
> The flag is supposed to indicate that the region of the disk image has
> to be sufficiently allocated so it reads as zeroes. The call with the flag
> set has to return -ENOTSUP if allocation cannot be done efficiently
> (i.e. without falling back to writing actual buffers)
> 
> Signed-off-by: Anton Nefedov 
> ---
>  block/io.c| 19 ---
>  block/trace-events|  1 +
>  include/block/block.h |  6 +-
>  3 files changed, 22 insertions(+), 4 deletions(-)

You may want to 'git config diff.orderFile /path/to/file' (with a
suitably populated file) so that .h files come first in your diffs, as
that can aid reviewers.  At one point, there was a thread about adding
such a file to qemu.git proper for everyone to share, although it seems
to have stalled.

> 
> diff --git a/block/io.c b/block/io.c
> index ed31810..d47efa9 100644
> --- a/block/io.c
> +++ b/block/io.c
> @@ -1272,7 +1272,7 @@ static int coroutine_fn 
> bdrv_co_do_pwrite_zeroes(BlockDriverState *bs,
>  assert(!bs->supported_zero_flags);
>  }
>  
> -if (ret == -ENOTSUP) {
> +if (ret == -ENOTSUP && !(flags & BDRV_REQ_ALLOCATE)) {

I'd feel MUCH better if you first fixed the conditional just above this
point to ensure that if the caller requests BDRV_REQ_ALLOCATE that we do
not call bdrv->bdrv_co_pwrite_zeroes() unless bs->supported_zero_flags
also mentions this bit.

Remember, the existing semantics of .bdrv_co_pwrite_zeroes() merely
state that we must return -ENOTSUP unless we can guarantee that we read
back as zeroes, but puts no timing constraints on it.  A driver that has
not been retrofitted to understand the BDRV_REQ_ALLOCATE flag will
therefore risk taking too long.  Using bs->supported_zero_flags as your
gate is what will let you avoid calling into a driver that has not been
audited for fitting the new contract.

>  /* Fall back to bounce buffer if write zeroes is unsupported */
>  BdrvRequestFlags write_flags = flags & ~BDRV_REQ_ZERO_WRITE;
>  
> @@ -1355,8 +1355,8 @@ static int coroutine_fn bdrv_aligned_pwritev(BdrvChild 
> *child,
>  ret = notifier_with_return_list_notify(>before_write_notifiers, req);
>  
>  if (!ret && bs->detect_zeroes != BLOCKDEV_DETECT_ZEROES_OPTIONS_OFF &&
> -!(flags & BDRV_REQ_ZERO_WRITE) && drv->bdrv_co_pwrite_zeroes &&
> -qemu_iovec_is_zero(qiov)) {
> +!(flags & BDRV_REQ_ZERO_WRITE) && !(flags & BDRV_REQ_ALLOCATE) &&
> +drv->bdrv_co_pwrite_zeroes && qemu_iovec_is_zero(qiov)) {
>  flags |= BDRV_REQ_ZERO_WRITE;
>  if (bs->detect_zeroes == BLOCKDEV_DETECT_ZEROES_OPTIONS_UNMAP) {
>  flags |= BDRV_REQ_MAY_UNMAP;
> @@ -1436,6 +1436,9 @@ static int coroutine_fn 
> bdrv_co_do_zero_pwritev(BdrvChild *child,
>  
>  assert(flags & BDRV_REQ_ZERO_WRITE);
>  if (head_padding_bytes || tail_padding_bytes) {
> +if (flags & BDRV_REQ_ALLOCATE) {
> +return -ENOTSUP;
> +}

Can we assert that BDRV_REQ_ALLOCATE will only be supplied by a caller
that is already using aligned values?  Or is that too strict?

>  buf = qemu_blockalign(bs, align);
>  iov = (struct iovec) {
>  .iov_base   = buf,
> @@ -1534,6 +1537,11 @@ int coroutine_fn bdrv_co_pwritev(BdrvChild *child,
>  return ret;
>  }
>  
> +if (qiov && flags & BDRV_REQ_ALLOCATE) {
> +/* allocation request with qiov provided doesn't make much sense */
> +return -ENOTSUP;

Should this be an assertion (bug in the program for mixing things that
don't make sense) rather than just a runtime error return?

> +}
> +
>  bdrv_inc_in_flight(bs);
>  /*
>   * Align write if necessary by performing a read-modify-write cycle.
> @@ -1665,6 +1673,11 @@ int coroutine_fn bdrv_co_pwrite_zeroes(BdrvChild 
> *child, int64_t offset,
>  {
>  trace_bdrv_co_pwrite_zeroes(child->bs, offset, count, flags);
>  
> +if (flags & BDRV_REQ_MAY_UNMAP && flags & BDRV_REQ_ALLOCATE) {
> +/* nonsense */
> +return -ENOTSUP;
> +}

Ditto.

> +
>  if (!(child->bs->open_flags & BDRV_O_UNMAP)) {
>  flags &= ~BDRV_REQ_MAY_UNMAP;
>  }
> diff --git a/block/trace-events b/block/trace-events
> index 9a71c7f..a15c2cc 100644
> --- a/block/trace-events
> +++ b/block/trace-events
> @@ -15,6 +15,7 @@ bdrv_aio_writev(void *bs, int64_t sector_num, int 
> nb_sectors, void *opaque) "bs
>  bdrv_co_readv(void *bs, int64_t sector_num, int nb_sector) "bs %p sector_num 
> %"PRId64" nb_sectors %d"
>  bdrv_co_writev(void *bs, int64_t sector_num, int nb_sector) "bs %p 
> sector_num %"PRId64" nb_sectors %d"
>  bdrv_co_pwrite_zeroes(void *bs, int64_t offset, int count, int flags) "bs %p 
> offset %"PRId64" count %d flags %#x"
> +bdrv_co_allocate(void *bs, int64_t offset, int count) "bs %p offset 
> %"PRId64" count %d"
>  bdrv_co_do_copy_on_readv(void *bs, int64_t offset, 

Re: [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap

[John Snow]

On 06/01/2017 03:30 AM, Sementsov-Ogievskiy Vladimir wrote:
> Hi John!
> 
> Look at our discussion about this in v18 thread.
> 

I'm batting zero today; I literally missed that entire subthread.

Sigh, thanks, I'll go back and read.

Re: [Qemu-devel] [PATCH v2 00/15] qcow2: space preallocation and COW improvements

[John Snow]

Missing qemu-block@, CCing.

On 06/01/2017 11:14 AM, Anton Nefedov wrote:
> Changes in v2:
>   - introduce new BDRV flag for write_zeroes()
>   instead of using driver callback directly.
> Skipped introducing new functions like bdrv_co_pallocate() for now:
>   1. it seems ok to keep calling this write_zeroes() as zeroes
>   are expected;
>   2. most of the code can be reused now anyway, so changes to
>   write_zeroes() path are not significant
>   3. write_zeroes() alignment and max-request limits can also be reused
> 
> As a possible alternative we can have bdrv_co_pallocate() which can
> switch to pwrite_zeroes(,flags|=BDRV_REQ_ALLOCATE) early.
> 
> 
> 
> This pull request is to address a few performance problems of qcow2 format:
> 
>   1. non cluster-aligned write requests (to unallocated clusters) explicitly
> pad data with zeroes if there is no backing data. This can be avoided
> and the whole clusters are preallocated and zeroed in a single
> efficient write_zeroes() operation, also providing better host file
> continuity
> 
>   2. moreover, efficient write_zeroes() operation can be used to preallocate
> space megabytes ahead which gives noticeable improvement on some storage
> types (e.g. distributed storages where space allocation operation is
> expensive)
> 
>   3. preallocating/zeroing the clusters in advance makes possible to enable
> simultaneous writes to the same unallocated cluster, which is beneficial
> for parallel sequential write operations which are not cluster-aligned
> 
> Performance test results are added to commit messages (see patch 3, 12)
> 
> Anton Nefedov (11):
>   block: introduce BDRV_REQ_ALLOCATE flag
>   file-posix: support BDRV_REQ_ALLOCATE
>   blkdebug: support BDRV_REQ_ALLOCATE
>   qcow2: do not COW the empty areas
>   qcow2: set inactive flag
>   qcow2: handle_prealloc(): find out if area zeroed by earlier
> preallocation
>   qcow2: fix misleading comment about L2 linking
>   qcow2-cluster: slightly refactor handle_dependencies()
>   qcow2-cluster: make handle_dependencies() logic easier to follow
>   qcow2: allow concurrent unaligned writes to the same clusters
>   iotest 046: test simultaneous cluster write error case
> 
> Denis V. Lunev (3):
>   qcow2: alloc space for COW in one chunk
>   qcow2: preallocation at image expand
>   qcow2: truncate preallocated space
> 
> Pavel Butsykin (1):
>   qcow2: check space leak at the end of the image
> 
>  block/blkdebug.c   |   3 +-
>  block/file-posix.c |   9 +-
>  block/io.c |  19 ++-
>  block/qcow2-cache.c|   3 +
>  block/qcow2-cluster.c  | 218 +++--
>  block/qcow2-refcount.c |  21 +++
>  block/qcow2.c  | 273 
> -
>  block/qcow2.h  |  26 
>  block/trace-events |   1 +
>  include/block/block.h  |   6 +-
>  tests/qemu-iotests/026.out | 104 ++
>  tests/qemu-iotests/026.out.nocache | 104 ++
>  tests/qemu-iotests/029.out |   5 +-
>  tests/qemu-iotests/046 |  38 +-
>  tests/qemu-iotests/046.out |  23 
>  tests/qemu-iotests/060 |   2 +-
>  tests/qemu-iotests/060.out |  13 +-
>  tests/qemu-iotests/061.out |   5 +-
>  tests/qemu-iotests/066 |   2 +-
>  tests/qemu-iotests/066.out |   9 +-
>  tests/qemu-iotests/098.out |   7 +-
>  tests/qemu-iotests/108.out |   5 +-
>  tests/qemu-iotests/112.out |   5 +-
>  23 files changed, 789 insertions(+), 112 deletions(-)
> 

Re: [Qemu-devel] [PATCH v2 2/4] dump: add vmcoreinfo ELF note

[Laszlo Ersek]

On 06/01/17 15:03, Marc-André Lureau wrote:
> Read vmcoreinfo note from guest memory when dump_info provides the
> address, and write it as an ELF note in the dump.
> 
> NUMBER(phys_base) in vmcoreinfo has only been recently introduced in
> Linux 4.10 ("kexec: export the value of phys_base instead of symbol
> address"). To accomadate for older kernels, modify the vmcoreinfo to add
> the new fields and help newer crash that will use it.

I think here you mean

  modify the DumpState structure

rather than

  modify the vmcoreinfo

> 
> Signed-off-by: Marc-André Lureau 
> ---
>  include/sysemu/dump.h |   2 +
>  dump.c| 133 
> ++
>  2 files changed, 135 insertions(+)
> 
> diff --git a/include/sysemu/dump.h b/include/sysemu/dump.h
> index 2672a15f8b..b8a7a1e41d 100644
> --- a/include/sysemu/dump.h
> +++ b/include/sysemu/dump.h
> @@ -192,6 +192,8 @@ typedef struct DumpState {
>* this could be used to calculate
>* how much work we have
>* finished. */
> +uint8_t *vmcoreinfo;

Can you document that this is an ELF note?

> +size_t vmcoreinfo_size;
>  } DumpState;
>  
>  uint16_t cpu_to_dump16(DumpState *s, uint16_t val);
> diff --git a/dump.c b/dump.c
> index bdf3270f02..6911ffad8b 100644
> --- a/dump.c
> +++ b/dump.c
> @@ -27,6 +27,7 @@
>  #include "qapi/qmp/qerror.h"
>  #include "qmp-commands.h"
>  #include "qapi-event.h"
> +#include "qemu/error-report.h"
>  
>  #include 
>  #ifdef CONFIG_LZO
> @@ -88,6 +89,8 @@ static int dump_cleanup(DumpState *s)
>  qemu_mutex_unlock_iothread();
>  }
>  }
> +g_free(s->vmcoreinfo);
> +s->vmcoreinfo = NULL;
>  
>  return 0;
>  }

I vaguely feel that this should be moved in front of resuming VM
execution. I don't have a strong reason, just consistency with the rest
of the cleanup.

> @@ -238,6 +241,19 @@ static inline int cpu_index(CPUState *cpu)
>  return cpu->cpu_index + 1;
>  }
>  
> +static void write_vmcoreinfo_note(WriteCoreDumpFunction f, DumpState *s,
> +  Error **errp)
> +{
> +int ret;
> +
> +if (s->vmcoreinfo) {
> +ret = f(s->vmcoreinfo, s->vmcoreinfo_size, s);
> +if (ret < 0) {
> +error_setg(errp, "dump: failed to write vmcoreinfo");
> +}
> +}
> +}
> +
>  static void write_elf64_notes(WriteCoreDumpFunction f, DumpState *s,
>Error **errp)
>  {
> @@ -261,6 +277,8 @@ static void write_elf64_notes(WriteCoreDumpFunction f, 
> DumpState *s,
>  return;
>  }
>  }
> +
> +write_vmcoreinfo_note(f, s, errp);
>  }
>  
>  static void write_elf32_note(DumpState *s, Error **errp)
> @@ -306,6 +324,8 @@ static void write_elf32_notes(WriteCoreDumpFunction f, 
> DumpState *s,
>  return;
>  }
>  }
> +
> +write_vmcoreinfo_note(f, s, errp);
>  }
>  
>  static void write_elf_section(DumpState *s, int type, Error **errp)
> @@ -717,6 +737,50 @@ static int buf_write_note(const void *buf, size_t size, 
> void *opaque)
>  return 0;
>  }
>  
> +static void get_note_sizes(DumpState *s, const void *note,
> +   uint64_t *note_head_size,
> +   uint64_t *name_size,
> +   uint64_t *desc_size)
> +{

I'm not happy that I have to reverse engineer what this function does.
Please document it in the commit message and/or in a function-level
comment, especially regarding the actual permitted types of *note.

Very similar functionality exists in "target/i386/arch_dump.c" already.
Is there a (remote) possibility to extract / refactor / share code?

> +uint64_t note_head_sz;
> +uint64_t name_sz;
> +uint64_t desc_sz;
> +
> +if (s->dump_info.d_class == ELFCLASS64) {

Ugh, this is extremely confusing. This refers to DumpState.dump_info,
which has type ArchDumpInfo. But in the previous patch we also introduce
a global "dump_info" variable, of type DumpInfo.

Worse, ArchDumpInfo already has a field called "phys_base" (comment:
"The target's physmem base"), and it's even filled in in
"target/arm/arch_dump.c", function cpu_get_dump_info():

/* Take a best guess at the phys_base. If we get it wrong then crash
 * will need '--machdep phys_offset=' added to its command
 * line, which isn't any worse than assuming we can use zero, but being
 * wrong. This is the same algorithm the crash utility uses when
 * attempting to guess as it loads non-dumpfile formatted files.
 */

Looks like we already have some overlapping code / functionality for
this, for the ARM target?

Sorry, I'm totally lost. It must have been years since I last looked at
this code. I guess my comments might not make much sense, even.

Please post a version 3, with as detailed as possible commit messages,
explaining your entire thought process, the data 

Re: [Qemu-devel] [PATCH v2 1/4] dump: add DumpInfo structure

[Eric Blake]

On 06/01/2017 01:06 PM, Laszlo Ersek wrote:
> On 06/01/17 15:03, Marc-André Lureau wrote:
>> One way or another, the guest could communicate various dump info (via
>> guest agent or vmcoreinfo device) and populate that structure. It can
>> then be used to augment the dump with various details, as done in the
>> following patch.
>>
>> Signed-off-by: Marc-André Lureau 
>> ---
>>  include/sysemu/dump-info.h | 18 ++
>>  dump.c |  3 +++
>>  2 files changed, 21 insertions(+)
>>  create mode 100644 include/sysemu/dump-info.h
>>
>> diff --git a/include/sysemu/dump-info.h b/include/sysemu/dump-info.h
>> new file mode 100644
>> index 00..d2378e15e2
>> --- /dev/null
>> +++ b/include/sysemu/dump-info.h
>> @@ -0,0 +1,18 @@
>> +#ifndef DUMP_INFO_H
>> +#define DUMP_INFO_H

>>
> 
> Can you please spell out, in the commit message, the reason for
> introducing a new header file? (I suspect your reason, but it should be
> documented explicitly.)

Also, should you have a copyright header in the new file?  And does
MAINTAINERS cover it?

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] [PATCH v8 19/20] qcow2: report encryption specific image information

[Eric Blake]

On 06/01/2017 12:27 PM, Daniel P. Berrange wrote:
> Currently 'qemu-img info' reports a simple "encrypted: yes"
> field. This is not very useful now that qcow2 can support
> multiple encryption formats. Users want to know which format
> is in use and some data related to it.
> 

> 
> Signed-off-by: Daniel P. Berrange 
> ---
>  block/qcow2.c| 32 +++-
>  qapi/block-core.json | 27 ++-
>  2 files changed, 57 insertions(+), 2 deletions(-)
> 

Reviewed-by: Eric Blake 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] [PATCH v2 1/4] dump: add DumpInfo structure

[Laszlo Ersek]

On 06/01/17 15:03, Marc-André Lureau wrote:
> One way or another, the guest could communicate various dump info (via
> guest agent or vmcoreinfo device) and populate that structure. It can
> then be used to augment the dump with various details, as done in the
> following patch.
> 
> Signed-off-by: Marc-André Lureau 
> ---
>  include/sysemu/dump-info.h | 18 ++
>  dump.c |  3 +++
>  2 files changed, 21 insertions(+)
>  create mode 100644 include/sysemu/dump-info.h
> 
> diff --git a/include/sysemu/dump-info.h b/include/sysemu/dump-info.h
> new file mode 100644
> index 00..d2378e15e2
> --- /dev/null
> +++ b/include/sysemu/dump-info.h
> @@ -0,0 +1,18 @@
> +#ifndef DUMP_INFO_H
> +#define DUMP_INFO_H
> +
> +typedef struct DumpInfo {
> +bool received;
> +/* kernel base address */
> +bool has_phys_base;
> +uint64_t phys_base;
> +/* "_text" symbol location */
> +bool has_text;
> +uint64_t text;
> +/* the content of /sys/kernel/vmcoreinfo on Linux */
> +char *vmcoreinfo;
> +} DumpInfo;
> +
> +extern DumpInfo dump_info;
> +
> +#endif /* DUMP_INFO_H */
> diff --git a/dump.c b/dump.c
> index d9090a24cc..bdf3270f02 100644
> --- a/dump.c
> +++ b/dump.c
> @@ -20,6 +20,7 @@
>  #include "monitor/monitor.h"
>  #include "sysemu/kvm.h"
>  #include "sysemu/dump.h"
> +#include "sysemu/dump-info.h"
>  #include "sysemu/sysemu.h"
>  #include "sysemu/memory_mapping.h"
>  #include "sysemu/cpus.h"
> @@ -38,6 +39,8 @@
>  #define ELF_MACHINE_UNAME "Unknown"
>  #endif
>  
> +DumpInfo dump_info = { 0, };
> +
>  uint16_t cpu_to_dump16(DumpState *s, uint16_t val)
>  {
>  if (s->dump_info.d_endian == ELFDATA2LSB) {
> 

Can you please spell out, in the commit message, the reason for
introducing a new header file? (I suspect your reason, but it should be
documented explicitly.)

Thanks
Laszlo

Re: [Qemu-devel] [PATCH v8 12/20] qcow2: extend specification to cover LUKS encryption

[Eric Blake]

On 06/01/2017 12:27 PM, Daniel P. Berrange wrote:
> Update the qcow2 specification to describe how the LUKS header is
> placed inside a qcow2 file, when using LUKS encryption for the
> qcow2 payload instead of the legacy AES-CBC encryption
> 
> Reviewed-by: Eric Blake 
> Reviewed-by: Alberto Garcia 
> Reviewed-by: Max Reitz 
> Signed-off-by: Daniel P. Berrange 
> ---
>  docs/specs/qcow2.txt | 103 
> +++
>  1 file changed, 103 insertions(+)
> 

> +== Data encryption ==
> +
> +When an encryption method is requested in the header, the image payload
> +data must be encrypted/decrypted on every write/read. The image headers
> +and metadata are never encrypted.
> +
> +The algorithms used for encryption vary depending on the method
> +
> + - AES:
> +
> +   The AES cipher, in CBC mode, with 256 bit keys.
> +
> +   Initialization vectors generated using plain64 method, with
> +   the virtual disk sector as the input tweak.
> +
> +   This format is no longer supported in QEMU system emulators, due
> +   to a number of design flaws affecting it security. It is only

s/affecting it/affecting its/

Can keep my R-b.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] [PULL v2 00/34] Misc patches for 2016-06-01

[Peter Maydell]

On 1 June 2017 at 18:53, Peter Maydell  wrote:
> Test failure on OSX:
>
> TEST: tests/device-introspect-test... (pid=66373)
>   /aarch64/device/introspect/list: OK
>   /aarch64/device/introspect/none: OK
>   /aarch64/device/introspect/abstract: OK
>   /aarch64/device/introspect/concrete: **
> ERROR:/root/qemu/qom/object.c:364:object_initialize_with_type:
> assertion failed: (type != NULL)
> Broken pipe

Got those the wrong way round -- this is the FreeBSD failure
and the other lot are OSX. Pretty sure it's the same error,
though -- it's just that for some reason my OSX setup doesn't
actually cause make to exit with an error when a test fails,
so it goes on to hit what's probably the same bug in all the
other check-qtest-$ARCH targets rather than bailing out.

thanks
-- PMM

Re: [Qemu-devel] [PULL v2 00/34] Misc patches for 2016-06-01

[Peter Maydell]

On 1 June 2017 at 18:07, Paolo Bonzini  wrote:
> The following changes since commit 56821559f0ba682fe6b367815572e6f974d329ab:
>
>   Merge remote-tracking branch 'dgilbert/tags/pull-hmp-20170517' into staging 
> (2017-05-18 13:36:15 +0100)
>
> are available in the git repository at:
>
>   git://github.com/bonzini/qemu.git tags/for-upstream
>
> for you to fetch changes up to f3c3ad2dc98c6610152355f6a4de1445efc87421:
>
>   cpus: reset throttle_thread_scheduled after sleep (2017-06-01 18:01:05 
> +0200)
>
> 
> * virtio-scsi use-after-free fix (Fam)
> * vhost-user-scsi support (Felipe)
> * SMM fixes and improvements for TCG (myself, Mihail)
> * irqchip and AddressSpaceDispatch cleanups and fixes (Peter)
> * Coverity fix (Stefano)
> * NBD cleanups and fixes (Vladimir, Eric, myself)
> * RTC accuracy improvements and code cleanups (Guangrong+Yunfang)
> * socket error reporting improvement (Daniel)
> * GDB XML description for SSE registers (Abdallah)
> * kvmclock update fix (Denis)
> * SMM memory savings (Gonglei)
> * -cpu 486 fix (Paolo)
> * various bugfixes (Roman, Peter, myself, Thomas)
> * rtc-test improvement (Guangrong)
> * migration throttling fix (Felipe)

Test failure on OSX:

TEST: tests/device-introspect-test... (pid=66373)
  /aarch64/device/introspect/list: OK
  /aarch64/device/introspect/none: OK
  /aarch64/device/introspect/abstract: OK
  /aarch64/device/introspect/concrete: **
ERROR:/root/qemu/qom/object.c:364:object_initialize_with_type:
assertion failed: (type != NULL)
Broken pipe

Test failures on FreeBSD:
  GTESTER check-qtest-aarch64
**
ERROR:/Users/pm215/src/qemu-for-merges/qom/object.c:364:void
object_initialize_with_type(void *, size_t, TypeImpl *): assertion
failed: (type != NULL)
Broken pipe
GTester: last random seed: R02S8983f4bd019deabaf27dbf8281490ab9
  GTESTER check-qtest-alpha
**
ERROR:/Users/pm215/src/qemu-for-merges/qom/object.c:364:void
object_initialize_with_type(void *, size_t, TypeImpl *): assertion
failed: (type != NULL)
Broken pipe
GTester: last random seed: R02S5ba80c00f420a9cbe79b7926bc90901c
  GTESTER check-qtest-arm
**
ERROR:/Users/pm215/src/qemu-for-merges/qom/object.c:364:void
object_initialize_with_type(void *, size_t, TypeImpl *): assertion
failed: (type != NULL)
Broken pipe
GTester: last random seed: R02S9a3f10d16d54467f421a79a355a25714
  GTESTER check-qtest-cris
  GTESTER check-qtest-i386
"kvm" accelerator not found.
"kvm" accelerator not found.
"kvm" accelerator not found.
"kvm" accelerator not found.
"kvm" accelerator not found.
"kvm" accelerator not found.
"kvm" accelerator not found.
"kvm" accelerator not found.
"kvm" accelerator not found.
"kvm" accelerator not found.
**
ERROR:/Users/pm215/src/qemu-for-merges/qom/object.c:364:void
object_initialize_with_type(void *, size_t, TypeImpl *): assertion
failed: (type != NULL)
Broken pipe
GTester: last random seed: R02S29797c8935576bfd6237cce3e02075c9
  GTESTER check-qtest-lm32
  GTESTER check-qtest-m68k
  GTESTER check-qtest-microblaze
  GTESTER check-qtest-microblazeel
  GTESTER check-qtest-mips
**
ERROR:/Users/pm215/src/qemu-for-merges/qom/object.c:364:void
object_initialize_with_type(void *, size_t, TypeImpl *): assertion
failed: (type != NULL)
Broken pipe
GTester: last random seed: R02S1d5b7d14f648cc07be3bfc003b7c66a2
  GTESTER check-qtest-mips64
**
ERROR:/Users/pm215/src/qemu-for-merges/qom/object.c:364:void
object_initialize_with_type(void *, size_t, TypeImpl *): assertion
failed: (type != NULL)
Broken pipe
GTester: last random seed: R02S6c9d59bd8f80319effd07c498108c129
  GTESTER check-qtest-mips64el
**
ERROR:/Users/pm215/src/qemu-for-merges/qom/object.c:364:void
object_initialize_with_type(void *, size_t, TypeImpl *): assertion
failed: (type != NULL)
Broken pipe
GTester: last random seed: R02S530b42302d04c589e6a5f1fc49268dcb
  GTESTER check-qtest-mipsel
**
ERROR:/Users/pm215/src/qemu-for-merges/qom/object.c:364:void
object_initialize_with_type(void *, size_t, TypeImpl *): assertion
failed: (type != NULL)
Broken pipe
GTester: last random seed: R02S404f19f557f26102621573b7b8641871
  GTESTER check-qtest-moxie
  GTESTER check-qtest-nios2
  GTESTER check-qtest-or1k
  GTESTER check-qtest-ppc
**
ERROR:/Users/pm215/src/qemu-for-merges/qom/object.c:364:void
object_initialize_with_type(void *, size_t, TypeImpl *): assertion
failed: (type != NULL)
Broken pipe
GTester: last random seed: R02S8a9874d645fde55f36b7df0204c82959
  GTESTER check-qtest-ppc64
**
ERROR:/Users/pm215/src/qemu-for-merges/qom/object.c:364:void
object_initialize_with_type(void *, size_t, TypeImpl *): assertion
failed: (type != NULL)
Broken pipe
GTester: last random seed: R02S1bf51dd7caadd650ac3be82bd21cbae5
  GTESTER check-qtest-ppcemb
**
ERROR:/Users/pm215/src/qemu-for-merges/qom/object.c:364:void
object_initialize_with_type(void *, 

[Qemu-devel] [PATCH v8 17/20] block: remove all encryption handling APIs

[Daniel P. Berrange]

Now that all encryption keys must be provided upfront via
the QCryptoSecret API and associated block driver properties
there is no need for any explicit encryption handling APIs
in the block layer. Encryption can be handled transparently
within the block driver. We only retain an API for querying
whether an image is encrypted or not, since that is a
potentially useful piece of metadata to report to the user.

Reviewed-by: Alberto Garcia 
Reviewed-by: Max Reitz 
Signed-off-by: Daniel P. Berrange 
---
 block.c   | 77 +--
 block/crypto.c|  1 -
 block/qapi.c  |  2 +-
 block/qcow.c  |  8 -
 block/qcow2.c |  1 -
 blockdev.c| 37 ++-
 hmp-commands.hx   |  2 ++
 include/block/block.h |  3 --
 include/block/block_int.h |  1 -
 include/qapi/error.h  |  1 -
 qapi/block-core.json  | 37 ++-
 qapi/common.json  |  5 +--
 12 files changed, 16 insertions(+), 159 deletions(-)

diff --git a/block.c b/block.c
index fa1d06d..440649c 100644
--- a/block.c
+++ b/block.c
@@ -2569,15 +2569,7 @@ static BlockDriverState *bdrv_open_inherit(const char 
*filename,
 goto close_and_fail;
 }
 
-if (!bdrv_key_required(bs)) {
-bdrv_parent_cb_change_media(bs, true);
-} else if (!runstate_check(RUN_STATE_PRELAUNCH)
-   && !runstate_check(RUN_STATE_INMIGRATE)
-   && !runstate_check(RUN_STATE_PAUSED)) { /* HACK */
-error_setg(errp,
-   "Guest must be stopped for opening of encrypted image");
-goto close_and_fail;
-}
+bdrv_parent_cb_change_media(bs, true);
 
 QDECREF(options);
 
@@ -3068,7 +3060,6 @@ static void bdrv_close(BlockDriverState *bs)
 bs->backing_format[0] = '\0';
 bs->total_sectors = 0;
 bs->encrypted = false;
-bs->valid_key = false;
 bs->sg = false;
 QDECREF(bs->options);
 QDECREF(bs->explicit_options);
@@ -3498,72 +3489,6 @@ bool bdrv_is_encrypted(BlockDriverState *bs)
 return bs->encrypted;
 }
 
-bool bdrv_key_required(BlockDriverState *bs)
-{
-BdrvChild *backing = bs->backing;
-
-if (backing && backing->bs->encrypted && !backing->bs->valid_key) {
-return true;
-}
-return (bs->encrypted && !bs->valid_key);
-}
-
-int bdrv_set_key(BlockDriverState *bs, const char *key)
-{
-int ret;
-if (bs->backing && bs->backing->bs->encrypted) {
-ret = bdrv_set_key(bs->backing->bs, key);
-if (ret < 0)
-return ret;
-if (!bs->encrypted)
-return 0;
-}
-if (!bs->encrypted) {
-return -EINVAL;
-} else if (!bs->drv || !bs->drv->bdrv_set_key) {
-return -ENOMEDIUM;
-}
-ret = bs->drv->bdrv_set_key(bs, key);
-if (ret < 0) {
-bs->valid_key = false;
-} else if (!bs->valid_key) {
-/* call the change callback now, we skipped it on open */
-bs->valid_key = true;
-bdrv_parent_cb_change_media(bs, true);
-}
-return ret;
-}
-
-/*
- * Provide an encryption key for @bs.
- * If @key is non-null:
- * If @bs is not encrypted, fail.
- * Else if the key is invalid, fail.
- * Else set @bs's key to @key, replacing the existing key, if any.
- * If @key is null:
- * If @bs is encrypted and still lacks a key, fail.
- * Else do nothing.
- * On failure, store an error object through @errp if non-null.
- */
-void bdrv_add_key(BlockDriverState *bs, const char *key, Error **errp)
-{
-if (key) {
-if (!bdrv_is_encrypted(bs)) {
-error_setg(errp, "Node '%s' is not encrypted",
-  bdrv_get_device_or_node_name(bs));
-} else if (bdrv_set_key(bs, key) < 0) {
-error_setg(errp, QERR_INVALID_PASSWORD);
-}
-} else {
-if (bdrv_key_required(bs)) {
-error_set(errp, ERROR_CLASS_DEVICE_ENCRYPTED,
-  "'%s' (%s) is encrypted",
-  bdrv_get_device_or_node_name(bs),
-  bdrv_get_encrypted_filename(bs));
-}
-}
-}
-
 const char *bdrv_get_format_name(BlockDriverState *bs)
 {
 return bs->drv ? bs->drv->format_name : NULL;
diff --git a/block/crypto.c b/block/crypto.c
index da4be74..3ad4b20 100644
--- a/block/crypto.c
+++ b/block/crypto.c
@@ -308,7 +308,6 @@ static int block_crypto_open_generic(QCryptoBlockFormat 
format,
 }
 
 bs->encrypted = true;
-bs->valid_key = true;
 
 ret = 0;
  cleanup:
diff --git a/block/qapi.c b/block/qapi.c
index a40922e..9d724c2 100644
--- a/block/qapi.c
+++ b/block/qapi.c
@@ -45,7 +45,7 @@ BlockDeviceInfo *bdrv_block_device_info(BlockBackend *blk,
 info->ro = bs->read_only;
 info->drv= g_strdup(bs->drv->format_name);
 info->encrypted  = bs->encrypted;
-   

[Qemu-devel] [PATCH v8 14/20] qcow2: add iotests to cover LUKS encryption support

[Daniel P. Berrange]

This extends the 087 iotest to cover LUKS encryption when doing
blockdev-add.

Two further tests are added to validate read/write of LUKS
encrypted images with a single file and with a backing file.

Reviewed-by: Alberto Garcia 
Reviewed-by: Max Reitz 
Signed-off-by: Daniel P. Berrange 
---
 tests/qemu-iotests/087 | 35 ++-
 tests/qemu-iotests/087.out | 14 +++-
 tests/qemu-iotests/183 | 76 
 tests/qemu-iotests/183.out | 18 ++
 tests/qemu-iotests/184 | 86 ++
 tests/qemu-iotests/184.out | 26 ++
 tests/qemu-iotests/group   |  2 ++
 7 files changed, 255 insertions(+), 2 deletions(-)
 create mode 100755 tests/qemu-iotests/183
 create mode 100644 tests/qemu-iotests/183.out
 create mode 100755 tests/qemu-iotests/184
 create mode 100644 tests/qemu-iotests/184.out

diff --git a/tests/qemu-iotests/087 b/tests/qemu-iotests/087
index 1d595b2..f8e4903 100755
--- a/tests/qemu-iotests/087
+++ b/tests/qemu-iotests/087
@@ -119,7 +119,7 @@ run_qemu .
+#
+
+# creator
+owner=berra...@redhat.com
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+status=1   # failure is the default!
+
+_cleanup()
+{
+   _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+_supported_fmt qcow2
+_supported_proto generic
+_supported_os Linux
+
+
+size=16M
+
+SECRET="secret,id=sec0,data=astrochicken"
+SECRETALT="secret,id=sec0,data=platypus"
+
+_make_test_img --object $SECRET -o 
"encrypt.format=luks,encrypt.key-secret=sec0,encrypt.iter-time=10" $size
+
+IMGSPEC="driver=$IMGFMT,file.filename=$TEST_IMG,encrypt.key-secret=sec0"
+
+QEMU_IO_OPTIONS=$QEMU_IO_OPTIONS_NO_FMT
+
+echo
+echo "== reading whole image =="
+$QEMU_IO --object $SECRET -c "read -P 0 0 $size" --image-opts $IMGSPEC | 
_filter_qemu_io | _filter_testdir
+
+echo
+echo "== rewriting whole image =="
+$QEMU_IO --object $SECRET -c "write -P 0xa 0 $size" --image-opts $IMGSPEC | 
_filter_qemu_io | _filter_testdir
+
+echo
+echo "== verify pattern =="
+$QEMU_IO --object $SECRET -c "read -P 0xa 0 $size"  --image-opts $IMGSPEC | 
_filter_qemu_io | _filter_testdir
+
+echo
+echo "== verify open failure with wrong password =="
+$QEMU_IO --object $SECRETALT -c "read -P 0xa 0 $size" --image-opts $IMGSPEC | 
_filter_qemu_io | _filter_testdir
+
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/183.out b/tests/qemu-iotests/183.out
new file mode 100644
index 000..a095077
--- /dev/null
+++ b/tests/qemu-iotests/183.out
@@ -0,0 +1,18 @@
+QA output created by 183
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=16777216 encrypt.format=luks 
encrypt.key-secret=sec0 encrypt.iter-time=10
+
+== reading whole image ==
+read 16777216/16777216 bytes at offset 0
+16 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+== rewriting whole image ==
+wrote 16777216/16777216 bytes at offset 0
+16 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+== verify pattern ==
+read 16777216/16777216 bytes at offset 0
+16 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+== verify open failure with wrong password ==
+can't open: Invalid password, cannot unlock any keyslot
+*** done
diff --git a/tests/qemu-iotests/184 b/tests/qemu-iotests/184
new file mode 100755
index 000..54ad980
--- /dev/null
+++ b/tests/qemu-iotests/184
@@ -0,0 +1,86 @@
+#!/bin/bash
+#
+# Test encrypted read/write using backing files
+#
+# Copyright (C) 2017 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see .
+#
+
+# creator
+owner=berra...@redhat.com
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+status=1   # failure is the default!
+
+_cleanup()
+{
+   _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+_supported_fmt qcow2
+_supported_proto generic
+_supported_os Linux
+
+
+size=16M
+TEST_IMG_BASE=$TEST_IMG.base
+SECRET0="secret,id=sec0,data=astrochicken"
+SECRET1="secret,id=sec1,data=furby"
+
+TEST_IMG_SAVE=$TEST_IMG
+TEST_IMG=$TEST_IMG_BASE
+echo "== create base =="
+_make_test_img --object $SECRET0 -o 

[Qemu-devel] [PATCH v8 11/20] qcow2: convert QCow2 to use QCryptoBlock for encryption

[Daniel P. Berrange]

This converts the qcow2 driver to make use of the QCryptoBlock
APIs for encrypting image content, using the legacy QCow2 AES
scheme.

With this change it is now required to use the QCryptoSecret
object for providing passwords, instead of the current block
password APIs / interactive prompting.

  $QEMU \
-object secret,id=sec0,filename=/home/berrange/encrypted.pw \
-drive file=/home/berrange/encrypted.qcow2,encrypt.key-secret=sec0

The test 087 could be simplified since there is no longer a
difference in behaviour when using blockdev_add with encrypted
images for the running vs stopped CPU state.

Reviewed-by: Alberto Garcia 
Reviewed-by: Eric Blake 
Signed-off-by: Daniel P. Berrange 
---
 block/qcow2-cluster.c  |  47 +-
 block/qcow2.c  | 226 ++---
 block/qcow2.h  |   5 +-
 qapi/block-core.json   |  27 +-
 tests/qemu-iotests/049 |   2 +-
 tests/qemu-iotests/049.out |   4 +-
 tests/qemu-iotests/082.out |  27 ++
 tests/qemu-iotests/087 |  28 +++---
 tests/qemu-iotests/087.out |  12 +--
 tests/qemu-iotests/134 |  18 +++-
 tests/qemu-iotests/134.out |  10 +-
 tests/qemu-iotests/158 |  19 ++--
 tests/qemu-iotests/158.out |  14 +--
 tests/qemu-iotests/common  |  10 +-
 14 files changed, 263 insertions(+), 186 deletions(-)

diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index 6400147..c4a256d 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -357,47 +357,6 @@ static int count_contiguous_clusters_unallocated(int 
nb_clusters,
 return i;
 }
 
-/* The crypt function is compatible with the linux cryptoloop
-   algorithm for < 4 GB images. */
-int qcow2_encrypt_sectors(BDRVQcow2State *s, int64_t sector_num,
-  uint8_t *buf, int nb_sectors, bool enc,
-  Error **errp)
-{
-union {
-uint64_t ll[2];
-uint8_t b[16];
-} ivec;
-int i;
-int ret;
-
-for(i = 0; i < nb_sectors; i++) {
-ivec.ll[0] = cpu_to_le64(sector_num);
-ivec.ll[1] = 0;
-if (qcrypto_cipher_setiv(s->cipher,
- ivec.b, G_N_ELEMENTS(ivec.b),
- errp) < 0) {
-return -1;
-}
-if (enc) {
-ret = qcrypto_cipher_encrypt(s->cipher,
- buf, buf,
- 512,
- errp);
-} else {
-ret = qcrypto_cipher_decrypt(s->cipher,
- buf, buf,
- 512,
- errp);
-}
-if (ret < 0) {
-return -1;
-}
-sector_num++;
-buf += 512;
-}
-return 0;
-}
-
 static int coroutine_fn do_perform_cow(BlockDriverState *bs,
uint64_t src_cluster_offset,
uint64_t cluster_offset,
@@ -438,11 +397,11 @@ static int coroutine_fn do_perform_cow(BlockDriverState 
*bs,
 Error *err = NULL;
 int64_t sector = (src_cluster_offset + offset_in_cluster)
  >> BDRV_SECTOR_BITS;
-assert(s->cipher);
 assert((offset_in_cluster & ~BDRV_SECTOR_MASK) == 0);
 assert((bytes & ~BDRV_SECTOR_MASK) == 0);
-if (qcow2_encrypt_sectors(s, sector, iov.iov_base,
-  bytes >> BDRV_SECTOR_BITS, true, ) < 0) {
+assert(s->crypto);
+if (qcrypto_block_encrypt(s->crypto, sector, iov.iov_base,
+  bytes, ) < 0) {
 ret = -EIO;
 error_free(err);
 goto out;
diff --git a/block/qcow2.c b/block/qcow2.c
index a6ed3dc..38c0420 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -37,6 +37,9 @@
 #include "qemu/option_int.h"
 #include "qemu/cutils.h"
 #include "qemu/bswap.h"
+#include "qapi/opts-visitor.h"
+#include "qapi-visit.h"
+#include "block/crypto.h"
 
 /*
   Differences with QCOW:
@@ -461,6 +464,7 @@ static QemuOptsList qcow2_runtime_opts = {
 .type = QEMU_OPT_NUMBER,
 .help = "Clean unused cache entries after this time (in seconds)",
 },
+BLOCK_CRYPTO_OPT_DEF_QCOW_KEY_SECRET("encrypt."),
 { /* end of list */ }
 },
 };
@@ -585,6 +589,7 @@ typedef struct Qcow2ReopenState {
 int overlap_check;
 bool discard_passthrough[QCOW2_DISCARD_MAX];
 uint64_t cache_clean_interval;
+QCryptoBlockOpenOptions *crypto_opts; /* Disk encryption runtime options */
 } Qcow2ReopenState;
 
 static int qcow2_update_options_prepare(BlockDriverState *bs,
@@ -598,9 +603,14 @@ static int qcow2_update_options_prepare(BlockDriverState 
*bs,
 int overlap_check_template = 0;
 uint64_t l2_cache_size, refcount_cache_size;
 int i;
+const