Re: [Xen-devel] [PATCH 1/3] Don't create default ioreq server

[Wen Congyang]

At 2016/11/30 20:25, Andrew Cooper wrote:

On 30/11/16 09:47, Zhang Chen wrote:

The ioreq server make colo run failed.

Signed-off-by: Zhang Chen <zhangchen.f...@cn.fujitsu.com>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>


Nack.

You can simply "fix" a COLO issue by breaking a much more common usecase.


Yes, this patch is wrong. It is only in my git tree to let COLO work.




What actually breaks in the COLO case here?


I have reported this BUG last year:
https://lists.xenproject.org/archives/html/xen-devel/2015-12/msg02850.html


Thanks
Wen Congyang



~Andrew


---
  xen/arch/x86/hvm/hvm.c | 11 ---
  1 file changed, 11 deletions(-)

diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index 25dc759..8522852 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -5339,17 +5339,6 @@ static int hvmop_get_param(
  case HVM_PARAM_IOREQ_PFN:
  case HVM_PARAM_BUFIOREQ_PFN:
  case HVM_PARAM_BUFIOREQ_EVTCHN:
-{
-domid_t domid;
-
-/* May need to create server. */
-domid = d->arch.hvm_domain.params[HVM_PARAM_DM_DOMAIN];
-rc = hvm_create_ioreq_server(d, domid, 1,
- HVM_IOREQSRV_BUFIOREQ_LEGACY, NULL);
-if ( rc != 0 && rc != -EEXIST )
-goto out;
-}
-/*FALLTHRU*/
  default:
  a.value = d->arch.hvm_domain.params[a.index];
  break;



___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel



___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Error migrating VM to secondary host using COLO replication

[Wen Congyang]

On 11/10/2016 01:31 AM, Sadi wrote:
> Hello again,
> 
> Looking at the primary host's syslog, the arptables command from 
> xen/etc/scripts/colo-proxy-setup has failed.
> 
> Here's the log:
> 
> Nov  9 14:43:39 colop colop: /etc/xen/scripts/colo-proxy-setup: setup 
> XENBUS_PATH=
> Nov  9 14:43:39 colop kernel: [  302.825788] u32 classifier
> Nov  9 14:43:39 colop kernel: [  302.825791] Actions configured
> Nov  9 14:43:39 colop kernel: [  302.835407] Mirror/redirect action on
> Nov  9 14:43:39 colop kernel: [  302.919605] ip6_tables: (C) 2000-2006 
> Netfilter Core Team
> Nov  9 14:43:39 colop kernel: [  302.941511] arp_tables: (C) 2002 David S. 
> Miller
> Nov  9 14:43:39 colop colop: /etc/xen/scripts/colo-proxy-setup: arptables -I 
> INPUT -i eth1 -j MARK --set-mark 1 failed
> Nov  9 14:43:39 colop colop: /etc/xen/scripts/colo-proxy-setup: Writing 
> /hotplug-status connected to xenstore.
> Nov  9 14:43:39 colop colop: /etc/xen/scripts/colo-proxy-setup: Successful 
> colo-proxy-setup setup for vif2.0-emu. mode = primary  vifname: vif2.0-emu, 
> index: 1, forwarddev: eth1.
> 
> It's ok for the --set-mark argument to have value equal '1' , not a hex?

This log is very useful, we will investigate it.

Thanks
Wen Congyang

> 
> The i got running the command is:
> root@colop:~# arptables -I INPUT -i eth1 -j MARK --set-mark 1
> Bad argument `1'
> 
> Thanks, Sadi.
> 
> 
> 
> On Tue, Nov 8, 2016 at 6:53 PM, Sadi <sadi...@gmail.com 
> <mailto:sadi...@gmail.com>> wrote:
> 
> Hi,
> 
> Apparently vif2.0-emu was already binded with br0 when "brctl addif br0 
> vif2.0-emu" failed.
> 
> root@colob-HP-Compaq-6005-Pro-MT-PC:~# brctl addif br0 vif2.0-emu
> device vif2.0-emu is already a member of a bridge; can't enslave it to 
> bridge br0.
> root@colob-HP-Compaq-6005-Pro-MT-PC:~# brctl show
> bridge name bridge id   STP enabled interfaces
> br0 8000.001a3fc46255   no  eth0
> vif2.0
> vif2.0-emu
> br1 8000.   no
> 
> About the iptables, it seems like SECCOLO target can't be recognised.
> 
> root@colob-HP-Compaq-6005-Pro-MT-PC:~# iptables -t mangle -D PREROUTING 
> -m physdev --physdev-in vif2.0-emu -j SECCOLO --index 1
> iptables: No chain/target/match by that name.
> 
> Here is my active modules matching colo:
> 
> root@colob-HP-Compaq-6005-Pro-MT-PC:~# lsmod | grep -i colo
> xt_SECCOLO 16384  1
> nf_conntrack_colo  16384  2 xt_SECCOLO
> x_tables   36864  8 
> xt_physdev,ip6table_mangle,ip_tables,xt_SECCOLO,xt_tcpudp,iptable_filter,iptable_mangle,ip6_tables
> nf_conntrack  106496  4 
> xt_SECCOLO,nf_nat,nf_conntrack_colo,nf_conntrack_ipv4
> 
> So i was looking in the iptables and this really looks like the source of 
> the problem. 
> 
> Sadi.
> 
> On Tue, Nov 8, 2016 at 5:57 PM, Konrad Rzeszutek Wilk 
> <konrad.w...@oracle.com <mailto:konrad.w...@oracle.com>> wrote:
> 
> > entered forwarding state
> > Nov  7 18:10:30 colob NetworkManager[907]:   (vif2.0-emu): 
> enslaved
> > to br0
> > Nov  7 18:10:30 colob root: /etc/xen/scripts/colo-proxy-setup: 
> brctl addif
> > br0 vif2.0-emu failed
> 
> 
> How come this failed?
> 
> > Nov  7 18:10:30 colob root: /etc/xen/scripts/colo-proxy-setup: 
> iptables -t
> > mangle -D PREROUTING -m physdev --physdev-in vif2.0-emu -j SECCOLO 
> --index
> > 1 failed
> 
> Ah b/c of this. Are there any errors of what exactly failed?
> 
> 
> 
> 
> -- 
> Sadi.
> 
> 
> 
> 
> -- 
> Sadi.




___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Error migrating VM to secondary host using COLO replication

[Wen Congyang]

On 10/28/2016 06:56 AM, Sadi wrote:
> Hello,
> 
> I've been trying to set COLO replication to work but i'm stuck on a problem 
> when migrating de primary VM to secondary host.
> 
> I have been following the instructions from this wiki
> 
> - http://wiki.xenproject.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
> 
> and this mail thread
> 
>  - 
> http://xen.markmail.org/search/?q=COLO#query:COLO+page:1+mid:fb7wrn62vbks4unn+state:results
> 
> I'm anexing the steps i took setting the environment before facing this 
> problem when executing 'xl remus' command:

Sorry for the late reply.

> 
>>migration target: Ready to receive domain.
>>Saving to migration stream new xl format (info 0x3/0x0/2840)
>>Loading new save file  (new xl fmt info 
>>0x3/0x0/2840)
>>Savefile contains xl domain config in JSON format
>>Parsing config from 
>>xc: info: Saving domain 2, type x86 HVM
>>xc: info: Found x86 HVM domain from Xen 4.7
>>xc: info: Restoring domain
>>xc: Frames iteration 0 of 5: 1045504/1045504  100%
>>xc: Domain now suspended: 0/00%
>>libxl: error: libxl_qmp.c:702:libxl__qmp_initialize: Connection error: No 
>>such file or directory
>>libxl: error: libxl_colo_restore.c:817:colo_restore_setup_cds_done: COLO: 
>>failed to setup device >for guest with domid 1

According to the log, we cannot connect to the secondary qemu. Can you give the 
secondry qemu log?

Thanks
Wen Congyang

>>xc: error: Restore failed (38 = Function not implemented): Internal error
>>libxl: info: libxl_colo_restore.c:320:libxl__colo_restore_teardown: colo fails
>>libxl: error: libxl_stream_read.c:852:libxl__xc_domain_restore_done: 
>>restoring domain: Function >not implemented
>>libxl: info: libxl_colo_restore.c:320:libxl__colo_restore_teardown: colo fails
> 
> I'm hoping that someone could provide with directions.
> 
> Thanks for your time and sory for bad english (not native language).
> 
> 
> Sadi.
> 
> 
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH for-4.7] tools/libxl: Fix legacy migration following COLO backchannel breakage

[Wen Congyang]

On 04/15/2016 03:54 AM, Andrew Cooper wrote:
> c/s f5d947bf1b "tools/libxl: add back channel support to read stream"
> made a bogus adjustment to libxl__stream_read_start(), including
> removing the comment hinting at what was going on, which breaks
> conversion of a legacy migration stream.
> 
> Symptoms look like:
> 
>   root@anonymi:~ # xl migrate domU host
>   migration target: Ready to receive domain.
>   Saving to migration stream new xl format (info 0x1/0x0/2677)
>   xc: error: error polling suspend notification channel: -1: Internal error
>   Loading new save file  (new xl fmt info 
> 0x1/0x0/2677)
>Savefile contains xl domain config in JSON format
>   Parsing config from 
>   libxl: error: libxl_stream_read.c:327:stream_header_done: Invalid ident: 
> expected 0x4c6962786c466d74, got 0x01f00f00
>   libxl: error: libxl_utils.c:430:libxl_read_exactly: file/stream truncated 
> reading ipc msg header from domain 1 save/restore helper stdout pipe
> 
> The adjustment is not required for backchannel support (as there is no
> interaction between back channels and legacy conversion), and caused
> stream->fd to be latched in the datacopier before legacy conversion
> substitutes it for the fd which is the output of the conversion script.
> 
> This causes libxl to consume data from the legacy stream rather than the
> v2 stream, and for the conversion script to encounter an error as the
> legacy stream appears to skip ahead.
> 
> Undo the adjustments to libxl__stream_read_start(), and introduce a
> better description of what is going on.  Introduce some extra assertions
> to try and catch similar breakage in the future.
> 
> Reported-by: Olaf Hering <o...@aepfle.de>
> Signed-off-by: Andrew Cooper <andrew.coop...@citrix.com>

Reviewed-by: Wen Congyang <we...@cn.fujitsu.com>

> ---
> CC: Ian Jackson <ian.jack...@eu.citrix.com>
> CC: Wei Liu <wei.l...@citrix.com>
> CC: Olaf Hering <o...@aepfle.de>
> CC: Yang Hongyang <hongyang.y...@easystack.cn>
> CC: Wen Congyang <we...@cn.fujitsu.com>
> CC: Changlong Xie <xiecl.f...@cn.fujitsu.com>
> ---
>  tools/libxl/libxl_stream_read.c | 33 -
>  1 file changed, 24 insertions(+), 9 deletions(-)
> 
> diff --git a/tools/libxl/libxl_stream_read.c b/tools/libxl/libxl_stream_read.c
> index 9659051..89c2f21 100644
> --- a/tools/libxl/libxl_stream_read.c
> +++ b/tools/libxl/libxl_stream_read.c
> @@ -234,16 +234,16 @@ void libxl__stream_read_start(libxl__egc *egc,
>  stream->running = true;
>  stream->phase   = SRS_PHASE_NORMAL;
>  
> -dc->ao   = stream->ao;
> -dc->copywhat = "restore v2 stream";
> -dc->readfd = stream->fd;
> -dc->writefd  = -1;
> -
> -if (stream->back_channel)
> -return;
> -
>  if (stream->legacy) {
> -/* Convert the legacy stream. */
> +/*
> + * Convert the legacy stream.
> + *
> + * This results in a fork()/exec() of conversion helper script.  It 
> is
> + * passed the exiting stream->fd as an input, and returns the
> + * transformed stream via a new pipe.  The fd of this new pipe then
> + * replaces stream->fd, to make the rest of the stream read code
> + * agnostic to whether legacy conversion is happening or not.
> + */
>  libxl__conversion_helper_state *chs = >chs;
>  
>  chs->legacy_fd = stream->fd;
> @@ -258,10 +258,25 @@ void libxl__stream_read_start(libxl__egc *egc,
>  goto err;
>  }
>  
> +/* There should be no interaction of COLO backchannels and legacy
> + * stream conversion. */
> +assert(!stream->back_channel);
> +
> +/* Confirm *dc is still zeroed out, while we shuffle stream->fd. */
> +assert(dc->ao == NULL);
>  assert(stream->chs.v2_carefd);
>  stream->fd = libxl__carefd_fd(stream->chs.v2_carefd);
>  stream->dcs->libxc_fd = stream->fd;
>  }
> +/* stream->fd is now a v2 stream. */
> +
> +dc->ao   = stream->ao;
> +dc->copywhat = "restore v2 stream";
> +dc->readfd   = stream->fd;
> +dc->writefd  = -1;
> +
> +if (stream->back_channel)
> +return;
>  
>  /* Start reading the stream header. */
>  rc = setup_read(stream, "stream header",
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH 0/3] COLO: only build on Linux

[Wen Congyang]

On 04/06/2016 06:17 PM, Wei Liu wrote:
> On Wed, Apr 06, 2016 at 09:24:26AM +0800, Wen Congyang wrote:
>> On 04/06/2016 04:05 AM, Wei Liu wrote:
>>> COLO depends on netlink which is only available on Linux. This series 
>>> cleans up
>>> COLO code and make it only build on Linux. This should fix FreeBSD build.
>>>
>>> Congyang and Changlong, please review this series as soon as possible. I 
>>> also
>>> have a question why COLO doesn't use libnl? It is using Linux header 
>>> directly.
>>
>> It is implemented by Yang one year ago. I don't know the reason. We will 
>> investigate
>> how to use libnl.
>>
> 
> It's easy -- change inclusion of linux/netlink.h to netlink/netlink.h. I
> already have a series to do that -- this series is actually my second
> attempt to fix COLO.
> 
> If you can confirm COLO should depend on libnl, I will post my patches.

I think COLO should depend on libnl.

Thanks
Wen Congyang

> 
> Wei.
> 
>> Thanks
>> Wen Congyang
>>
>>>
>>> Roger, can you run functional tests on FreeBSD? I don't have FreeBSD Dom0.
>>>
>>> Wei.
>>>
>>> Wei Liu (3):
>>>   libxl: colo: rearrange things in header files
>>>   libxl: colo: move netlink related stuff to libxl_colo_proxy.c
>>>   libxl: colo: only build COLO on Linux
>>>
>>>  tools/libxl/Makefile   |  5 
>>>  tools/libxl/libxl_colo.h   | 52 +--
>>>  tools/libxl/libxl_colo_proxy.c | 13 +
>>>  tools/libxl/libxl_internal.h   | 36 
>>>  tools/libxl/libxl_no_colo.c| 62 
>>> ++
>>>  5 files changed, 117 insertions(+), 51 deletions(-)
>>>  create mode 100644 tools/libxl/libxl_no_colo.c
>>>
>>
>>
>>
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH 0/3] COLO: only build on Linux

[Wen Congyang]

On 04/06/2016 04:05 AM, Wei Liu wrote:
> COLO depends on netlink which is only available on Linux. This series cleans 
> up
> COLO code and make it only build on Linux. This should fix FreeBSD build.
> 
> Congyang and Changlong, please review this series as soon as possible. I also
> have a question why COLO doesn't use libnl? It is using Linux header directly.

It is implemented by Yang one year ago. I don't know the reason. We will 
investigate
how to use libnl.

Thanks
Wen Congyang

> 
> Roger, can you run functional tests on FreeBSD? I don't have FreeBSD Dom0.
> 
> Wei.
> 
> Wei Liu (3):
>   libxl: colo: rearrange things in header files
>   libxl: colo: move netlink related stuff to libxl_colo_proxy.c
>   libxl: colo: only build COLO on Linux
> 
>  tools/libxl/Makefile   |  5 
>  tools/libxl/libxl_colo.h   | 52 +--
>  tools/libxl/libxl_colo_proxy.c | 13 +
>  tools/libxl/libxl_internal.h   | 36 
>  tools/libxl/libxl_no_colo.c| 62 
> ++
>  5 files changed, 117 insertions(+), 51 deletions(-)
>  create mode 100644 tools/libxl/libxl_no_colo.c
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v13 00/26] COarse-grain LOck-stepping Virtual Machines for Non-stop Service

[Wen Congyang]

On 03/30/2016 10:50 PM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v13 00/26] COarse-grain LOck-stepping Virtual 
> Machines for Non-stop Service"):
>> This patchset implemented the COLO feature for Xen.
>> For detail/install/use of COLO feature, refer to:
>> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
>>
>> You can get the codes from here:
>> https://github.com/Pating/xen/tree/changlox/colo_v13
> 
> I fetched the branches `colo_v13' and `colo_v13_fixup' and it seems
> that I can get the proper versions of v13.1 from the latter.  I have
> acked them accordingly.
> 
> Can you confirm that that branch is what you intend for upstream ?
> 
> If so, I have a question about it:
> 
> There are two patches in it which have not been posted as part of your
> series and are marked as "[DO NOT MERGE]".  (Thanks for your admirably
> clear marking, btw.)
> 
> They are
>   [DONT MERGE] don't create default ioreq server
>   [DONT MERGE] tools/libxc: support to resume uncooperative HVM guests
> 
> 
> The latter patch "support to resume uncooperative HVM guests" seems to
> have been posted a number of times and AFAICT most recently as
>   [PATCH v8 05/13] tools/libxc: support to resume uncooperative HVM guests
> on the 18th of February.
> 
> Is that not required for COLO ?  We need to sort this out, I think.

Yes, it is required for COLO.

> 
> 
> What is the status of the default ioreq server patch ?  Why is it in
> your git branch ?

I have reported this bug last year:
http://lists.xenproject.org/archives/html/xen-devel/2015-12/msg02850.html

This patch is just a temporary patch.

Thanks
Wen Congyang

> 
> 
> Thanks,
> Ian.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v11 20/27] Support colo mode for qemu disk

[Wen Congyang]

On 03/18/2016 01:18 AM, Ian Jackson wrote:
> Wen Congyang writes ("Re: [PATCH v11 20/27] Support colo mode for qemu disk"):
>> How does block replication work:
> 
> Thanks for this explanation, which is reallt helpful.
> 
> I would like to repeat back to you what I think I have understood:
> 
> Between resynchs, you allow each VM to run in parallel and to generate
> possibly-divergent disk writes.
> 
> So on host B you retain both the A and B disk writes.  They are
> stored as differences (qcow2) for performance reasons.
> 
> If A fails and it becomes necessary to resume the VM only on B, you
> use B's version of the VM (both disk and memory).
> 
> If B fails then A can use the A version (disk and memory).
> 
> If the two are still up, but they diverge in network traffic, you
> resynch the memory from A to B, and drop B's disk and replace it with
> a copy of A's.
> 
> Have I understood correctly ?

Yes.

> 
> 
> If so, what software, where, arranges for the management of the
> different qcow2 `layers' ?  Ie, what creates the layers; what resynchs
> them, etc. ?

active disk and hidden disk are seperate disk. The management application
can create an empty qcow disk before running COLO. These two disks are
empty disk, and have the same size with the secondary disk.

Thanks
Wen Congyang

> 
> The reason I started asking all these questions is because of these
> parameters in your disk config:
> 
>  +("colo_enable", libxl_defbool),
>  +("colo_restore_enable", libxl_defbool),
>  +("colo_host", string),
>  +("colo_port", string),
>  +("colo_export", string),
>  +("active_disk", string),
>  +("hidden_disk", string)
> 
> For COLO to work properly it is necessary that the `active_disk' and
> `hidden_disk' to relate in a specific way to the main disk: they must
> be related snapshots (qcow2, currently).
> 
> Would it be possible for these disk names to have formulaic,
> predicatable, names, so that they wouldn't need to be specified
> separately ?
> 
> Is there any value in being able to specify them separately ?
> 
> Ian.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 3/3] libxl: rename checkpointed_stream to stream_type

[Wen Congyang]

On 03/15/2016 08:54 PM, Wei Liu wrote:
> On Tue, Mar 15, 2016 at 09:58:37AM +0800, Wen Congyang wrote:
>> Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
>> ---
>> v3->v4: Remove the new macro, and updte the macro 
>> LIBXL_HAVE_CHECKPOINTED_STREAM
>>  tools/libxl/libxl.c  |  4 ++--
>>  tools/libxl/libxl.h  |  4 +++-
>>  tools/libxl/libxl_create.c   |  4 ++--
>>  tools/libxl/libxl_dom_save.c |  6 +++---
>>  tools/libxl/libxl_internal.h |  2 +-
>>  tools/libxl/libxl_save_callout.c |  4 ++--
>>  tools/libxl/libxl_stream_read.c  |  4 ++--
>>  tools/libxl/libxl_stream_write.c |  2 +-
>>  tools/libxl/libxl_types.idl  |  2 +-
>>  tools/libxl/xl_cmdimpl.c | 16 
>>  10 files changed, 25 insertions(+), 23 deletions(-)
>>
>> diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
>> index 93e228d..7579dd2 100644
>> --- a/tools/libxl/libxl.c
>> +++ b/tools/libxl/libxl.c
>> @@ -876,7 +876,7 @@ int libxl_domain_remus_start(libxl_ctx *ctx, 
>> libxl_domain_remus_info *info,
>>  dss->live = 1;
>>  dss->debug = 0;
>>  dss->remus = info;
>> -dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
>> +dss->stream_type = LIBXL_CHECKPOINTED_STREAM_REMUS;
>>  
>>  assert(info);
>>  
>> @@ -937,7 +937,7 @@ int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, 
>> int fd, int flags,
>>  dss->type = type;
>>  dss->live = flags & LIBXL_SUSPEND_LIVE;
>>  dss->debug = flags & LIBXL_SUSPEND_DEBUG;
>> -dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_NONE;
>> +dss->stream_type = LIBXL_CHECKPOINTED_STREAM_NONE;
>>  
>>  rc = libxl__fd_flags_modify_save(gc, dss->fd,
>>   ~(O_NONBLOCK|O_NDELAY), 0,
>> diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
>> index f9e3ef5..c55094b 100644
>> --- a/tools/libxl/libxl.h
>> +++ b/tools/libxl/libxl.h
>> @@ -879,7 +879,9 @@ void libxl_mac_copy(libxl_ctx *ctx, libxl_mac *dst, 
>> libxl_mac *src);
>>  /*
>>   * LIBXL_HAVE_CHECKPOINTED_STREAM
>>   *
>> - * If this is defined, then libxl_checkpointed_stream exists.
>> + * If this is defined, then libxl_checkpointed_stream exists, and the
>> + * libxl_domain_create_restore() interface's parameter checkpointed_stream
>> + * is renamed to stream_type
>>   */
>>  #define LIBXL_HAVE_CHECKPOINTED_STREAM 1
> 
> I just realised this patch have not provided compatibility shim for
> older version of LIBXL_API.
> 
> It now breaks backward compatibility because it changes existing API.
> It's my fault for not observing this earlier, sorry.
> 
> On the other hand, I don't think this patch is strictly necessary.
> Andrew's original complain was about libxc not exporting XC_MIG_* types.
> I think libxl can function just fine as is with the old name.
> 
> So if you really wish to change the name of that field, I'm afraid
> compatibility shim for all functions that accept
> libxl_domain_restore_params is required.
> 
> I suggest we drop this patch for now and focus on core COLO
> functionality, what do you think?

It is OK to drop this patch.

Thanks
Wen Congyang

> 
> Wei.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v4 2/3] tools: change checkpointed_stream's type from int to xc_migration_stream_t

[Wen Congyang]

checkpointed_stream is also renamed to stream_type

Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
Acked-by: Wei Liu <wei.l...@citrix.com>
---
v2->v3: Rename checkpointed_stream to stream_type
 tools/libxc/include/xenguest.h  |  8 
 tools/libxc/xc_nomigrate.c  |  4 ++--
 tools/libxc/xc_sr_restore.c | 10 +-
 tools/libxc/xc_sr_save.c|  8 
 tools/libxl/libxl_save_helper.c | 42 -
 5 files changed, 36 insertions(+), 36 deletions(-)

diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index cf521c3..4f0b06e 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -86,14 +86,14 @@ typedef enum {
  * @parm xch a handle to an open hypervisor interface
  * @parm fd the file descriptor to save a domain to
  * @parm dom the id of the domain
- * @param checkpointed_stream XC_MIG_STREAM_NONE if the far end of the stream
+ * @param stream_type XC_MIG_STREAM_NONE if the far end of the stream
  *doesn't use checkpointing
  * @return 0 on success, -1 on failure
  */
 int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t 
max_iters,
uint32_t max_factor, uint32_t flags /* XCFLAGS_xxx */,
struct save_callbacks* callbacks, int hvm,
-   int checkpointed_stream);
+   xc_migration_stream_t stream_type);
 
 /* callbacks provided by xc_domain_restore */
 struct restore_callbacks {
@@ -121,7 +121,7 @@ struct restore_callbacks {
  * @parm hvm non-zero if this is a HVM restore
  * @parm pae non-zero if this HVM domain has PAE support enabled
  * @parm superpages non-zero to allocate guest memory with superpages
- * @parm checkpointed_stream non-zero if the far end of the stream is using 
checkpointing
+ * @parm stream_type non-zero if the far end of the stream is using 
checkpointing
  * @parm callbacks non-NULL to receive a callback to restore toolstack
  *   specific data
  * @return 0 on success, -1 on failure
@@ -131,7 +131,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, 
uint32_t dom,
   domid_t store_domid, unsigned int console_evtchn,
   unsigned long *console_mfn, domid_t console_domid,
   unsigned int hvm, unsigned int pae, int superpages,
-  int checkpointed_stream,
+  xc_migration_stream_t stream_type,
   struct restore_callbacks *callbacks);
 
 /**
diff --git a/tools/libxc/xc_nomigrate.c b/tools/libxc/xc_nomigrate.c
index c9124df..08e1f8c 100644
--- a/tools/libxc/xc_nomigrate.c
+++ b/tools/libxc/xc_nomigrate.c
@@ -23,7 +23,7 @@
 int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t 
max_iters,
uint32_t max_factor, uint32_t flags,
struct save_callbacks* callbacks, int hvm,
-   int checkpointed_stream)
+   xc_migration_stream_t stream_type)
 {
 errno = ENOSYS;
 return -1;
@@ -34,7 +34,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t 
dom,
   domid_t store_domid, unsigned int console_evtchn,
   unsigned long *console_mfn, domid_t console_domid,
   unsigned int hvm, unsigned int pae, int superpages,
-  int checkpointed_stream,
+  xc_migration_stream_t stream_type,
   struct restore_callbacks *callbacks)
 {
 errno = ENOSYS;
diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
index d4d33fd..819401d 100644
--- a/tools/libxc/xc_sr_restore.c
+++ b/tools/libxc/xc_sr_restore.c
@@ -725,7 +725,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, 
uint32_t dom,
   domid_t store_domid, unsigned int console_evtchn,
   unsigned long *console_gfn, domid_t console_domid,
   unsigned int hvm, unsigned int pae, int superpages,
-  int checkpointed_stream,
+  xc_migration_stream_t stream_type,
   struct restore_callbacks *callbacks)
 {
 struct xc_sr_context ctx =
@@ -739,16 +739,16 @@ int xc_domain_restore(xc_interface *xch, int io_fd, 
uint32_t dom,
 ctx.restore.console_domid = console_domid;
 ctx.restore.xenstore_evtchn = store_evtchn;
 ctx.restore.xenstore_domid = store_domid;
-ctx.restore.checkpointed = checkpointed_stream;
+ctx.restore.checkpointed = stream_type;
 ctx.restore.callbacks = callbacks;
 
 /* Sanity checks for callbacks. */
-if ( checkpointed_stream )
+if ( stream_type )
 assert(callbacks->checkpoint);
 
 DPRINTF("fd %d, dom %u, hvm %u, pae %u, superpages %d"
-", checkpointed_stream %d", io_fd, dom, hvm, pae,
-superpages, checkpointed_stream);
+&q

[Xen-devel] [PATCH v4 3/3] libxl: rename checkpointed_stream to stream_type

[Wen Congyang]

Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
---
v3->v4: Remove the new macro, and updte the macro LIBXL_HAVE_CHECKPOINTED_STREAM
 tools/libxl/libxl.c  |  4 ++--
 tools/libxl/libxl.h  |  4 +++-
 tools/libxl/libxl_create.c   |  4 ++--
 tools/libxl/libxl_dom_save.c |  6 +++---
 tools/libxl/libxl_internal.h |  2 +-
 tools/libxl/libxl_save_callout.c |  4 ++--
 tools/libxl/libxl_stream_read.c  |  4 ++--
 tools/libxl/libxl_stream_write.c |  2 +-
 tools/libxl/libxl_types.idl  |  2 +-
 tools/libxl/xl_cmdimpl.c | 16 
 10 files changed, 25 insertions(+), 23 deletions(-)

diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 93e228d..7579dd2 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -876,7 +876,7 @@ int libxl_domain_remus_start(libxl_ctx *ctx, 
libxl_domain_remus_info *info,
 dss->live = 1;
 dss->debug = 0;
 dss->remus = info;
-dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
+dss->stream_type = LIBXL_CHECKPOINTED_STREAM_REMUS;
 
 assert(info);
 
@@ -937,7 +937,7 @@ int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, 
int fd, int flags,
 dss->type = type;
 dss->live = flags & LIBXL_SUSPEND_LIVE;
 dss->debug = flags & LIBXL_SUSPEND_DEBUG;
-dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_NONE;
+dss->stream_type = LIBXL_CHECKPOINTED_STREAM_NONE;
 
 rc = libxl__fd_flags_modify_save(gc, dss->fd,
  ~(O_NONBLOCK|O_NDELAY), 0,
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index f9e3ef5..c55094b 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -879,7 +879,9 @@ void libxl_mac_copy(libxl_ctx *ctx, libxl_mac *dst, 
libxl_mac *src);
 /*
  * LIBXL_HAVE_CHECKPOINTED_STREAM
  *
- * If this is defined, then libxl_checkpointed_stream exists.
+ * If this is defined, then libxl_checkpointed_stream exists, and the
+ * libxl_domain_create_restore() interface's parameter checkpointed_stream
+ * is renamed to stream_type
  */
 #define LIBXL_HAVE_CHECKPOINTED_STREAM 1
 
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index f1028bc..b28eb89 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -993,7 +993,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
 libxl_domain_config *const d_config = dcs->guest_config;
 const int restore_fd = dcs->restore_fd;
 libxl__domain_build_state *const state = >build_state;
-const int checkpointed_stream = dcs->restore_params.checkpointed_stream;
+const int stream_type = dcs->restore_params.stream_type;
 
 if (rc) {
 domcreate_rebuild_done(egc, dcs, rc);
@@ -1033,7 +1033,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
 dcs->srs.completion_callback = domcreate_stream_done;
 
 if (restore_fd >= 0) {
-switch (checkpointed_stream) {
+switch (stream_type) {
 case LIBXL_CHECKPOINTED_STREAM_REMUS:
 libxl__remus_restore_setup(egc, dcs);
 /* fall through */
diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
index f3288b9..ff92ef0 100644
--- a/tools/libxl/libxl_dom_save.c
+++ b/tools/libxl/libxl_dom_save.c
@@ -338,7 +338,7 @@ void libxl__domain_save(libxl__egc *egc, 
libxl__domain_save_state *dss)
 unsigned int nr_vnodes = 0, nr_vmemranges = 0, nr_vcpus = 0;
 libxl__domain_suspend_state *dsps = >dsps;
 
-if (dss->checkpointed_stream != LIBXL_CHECKPOINTED_STREAM_NONE && !r_info) 
{
+if (dss->stream_type != LIBXL_CHECKPOINTED_STREAM_NONE && !r_info) {
 LOG(ERROR, "Migration stream is checkpointed, but there's no "
"checkpoint info!");
 rc = ERROR_INVAL;
@@ -383,12 +383,12 @@ void libxl__domain_save(libxl__egc *egc, 
libxl__domain_save_state *dss)
 goto out;
 }
 
-if (dss->checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_REMUS) {
+if (dss->stream_type == LIBXL_CHECKPOINTED_STREAM_REMUS) {
 if (libxl_defbool_val(r_info->compression))
 dss->xcflags |= XCFLAGS_CHECKPOINT_COMPRESS;
 }
 
-if (dss->checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_NONE)
+if (dss->stream_type == LIBXL_CHECKPOINTED_STREAM_NONE)
 callbacks->suspend = libxl__domain_suspend_callback;
 
 callbacks->switch_qemu_logdirty = 
libxl__domain_suspend_common_switch_qemu_logdirty;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 005fe53..0aada0d 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3128,7 +3128,7 @@ struct libxl__domain_save_state {
 libxl_domain_type type;
 int live;
 int debug;
-int checkpointed_stream;
+libxl_checkpointed_stream stream_type;
 const libxl_domain_remus_info *remus;
 /* private */
 

[Xen-devel] [PATCH v4 1/3] libxc: move migration_stream's definition to xenguest.h

[Wen Congyang]

xc_domain_save() and xc_domain_restore's parameter will use this type,
so it should be public.

Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
Acked-by: Wei Liu <wei.l...@citrix.com>
---
v2->v3: Rename MIG_STREAM_* to XC_MIG_STREAM_*
 tools/libxc/include/xenguest.h |  7 ++-
 tools/libxc/xc_sr_common.h | 10 --
 tools/libxc/xc_sr_save.c   | 12 ++--
 3 files changed, 12 insertions(+), 17 deletions(-)

diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index affc42b..cf521c3 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -75,13 +75,18 @@ struct save_callbacks {
 void* data;
 };
 
+typedef enum {
+XC_MIG_STREAM_NONE, /* plain stream */
+XC_MIG_STREAM_REMUS,
+} xc_migration_stream_t;
+
 /**
  * This function will save a running domain.
  *
  * @parm xch a handle to an open hypervisor interface
  * @parm fd the file descriptor to save a domain to
  * @parm dom the id of the domain
- * @param checkpointed_stream MIG_STREAM_NONE if the far end of the stream
+ * @param checkpointed_stream XC_MIG_STREAM_NONE if the far end of the stream
  *doesn't use checkpointing
  * @return 0 on success, -1 on failure
  */
diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 66f595f..e7568b5 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -180,16 +180,6 @@ struct xc_sr_context
 
 xc_dominfo_t dominfo;
 
-/*
- * migration stream
- * 0: Plain VM
- * 1: Remus
- */
-enum {
-MIG_STREAM_NONE, /* plain stream */
-MIG_STREAM_REMUS,
-} migration_stream;
-
 union /* Common save or restore data. */
 {
 struct /* Save data. */
diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
index e258b7c..ab59673 100644
--- a/tools/libxc/xc_sr_save.c
+++ b/tools/libxc/xc_sr_save.c
@@ -629,7 +629,7 @@ static int send_domain_memory_live(struct xc_sr_context 
*ctx)
 if ( rc )
 goto out;
 
-if ( ctx->save.debug && ctx->save.checkpointed != MIG_STREAM_NONE )
+if ( ctx->save.debug && ctx->save.checkpointed != XC_MIG_STREAM_NONE )
 {
 rc = verify_frames(ctx);
 if ( rc )
@@ -758,7 +758,7 @@ static int save(struct xc_sr_context *ctx, uint16_t 
guest_type)
 
 if ( ctx->save.live )
 rc = send_domain_memory_live(ctx);
-else if ( ctx->save.checkpointed != MIG_STREAM_NONE )
+else if ( ctx->save.checkpointed != XC_MIG_STREAM_NONE )
 rc = send_domain_memory_checkpointed(ctx);
 else
 rc = send_domain_memory_nonlive(ctx);
@@ -778,7 +778,7 @@ static int save(struct xc_sr_context *ctx, uint16_t 
guest_type)
 if ( rc )
 goto err;
 
-if ( ctx->save.checkpointed != MIG_STREAM_NONE )
+if ( ctx->save.checkpointed != XC_MIG_STREAM_NONE )
 {
 /*
  * We have now completed the initial live portion of the checkpoint
@@ -799,7 +799,7 @@ static int save(struct xc_sr_context *ctx, uint16_t 
guest_type)
 if ( rc <= 0 )
 goto err;
 }
-} while ( ctx->save.checkpointed != MIG_STREAM_NONE );
+} while ( ctx->save.checkpointed != XC_MIG_STREAM_NONE );
 
 xc_report_progress_single(xch, "End of stream");
 
@@ -845,8 +845,8 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t 
dom,
 ctx.save.checkpointed = checkpointed_stream;
 
 /* If altering migration_stream update this assert too. */
-assert(checkpointed_stream == MIG_STREAM_NONE ||
-   checkpointed_stream == MIG_STREAM_REMUS);
+assert(checkpointed_stream == XC_MIG_STREAM_NONE ||
+   checkpointed_stream == XC_MIG_STREAM_REMUS);
 
 /*
  * TODO: Find some time to better tweak the live migration algorithm.
-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v10 26/31] COLO proxy: implement setup/teardown of COLO proxy module

[Wen Congyang]

c = timeout_us % 100;
>> +setsockopt(cps->sock_fd, SOL_SOCKET, SO_RCVTIMEO, , sizeof(tv));
>> +}
>> +
>> +iov.iov_base = tmp;
>> +iov.iov_len = size;
>> +next:
>> +ret = recvmsg(cps->sock_fd, , 0);
>> +if (ret <= 0) {
>> +if (errno != EAGAIN && errno != EWOULDBLOCK)
> 
> -EINTR ?

IIRC, WAGAIN and EWOULDBLOCK may have different value in some system.
EINTR is not handled here.

> 
>> +LOGE(ERROR, "can't recv msg from kernel by netlink");
>> +goto err;
>> +}
>> +
>> +len += ret;
>> +if (mh.msg_flags & MSG_TRUNC) {
>> +size += 16384;
>> +tmp = libxl__realloc(NOGC, tmp, size);
> 
> You really should check 'tmp'.
> 
> If this loop continues on for some time the 'size' may be
> in milions and this realloc will fail.

OK, will fix it in the next version.

> 
>> +iov.iov_base = tmp + len;
>> +iov.iov_len = size - len;
>> +goto next;
> 
>> +}
>> +
>> +*buff = tmp;
>> +ret = len;
>> +goto out;
>> +
>> +err:
>> +free(tmp);
>> +*buff = NULL;
>> +
>> +out:
>> +if (timeout_us) {
>> +tv.tv_sec = 0;
>> +tv.tv_usec = 0;
>> +setsockopt(cps->sock_fd, SOL_SOCKET, SO_RCVTIMEO, , sizeof(tv));
>> +}
>> +return ret;
>> +}
>> +
>> +/* = colo-proxy: setup and teardown == */
>> +
>> +int colo_proxy_setup(libxl__colo_proxy_state *cps)
>> +{
>> +int skfd = 0;
>> +struct sockaddr_nl sa;
>> +struct nlmsghdr *h;
>> +int i = 1;
>> +int ret = ERROR_FAIL;
>> +uint8_t *buff = NULL;
>> +int64_t size;
>> +
>> +STATE_AO_GC(cps->ao);
>> +
>> +skfd = socket(PF_NETLINK, SOCK_RAW, NETLINK_COLO);
>> +if (skfd < 0) {
>> +LOG(ERROR, "can not create a netlink socket: %s", strerror(errno));
>> +goto out;
>> +}
>> +cps->sock_fd = skfd;
>> +memset(, 0, sizeof(sa));
>> +sa.nl_family = AF_NETLINK;
>> +sa.nl_groups = 0;
>> +retry:
>> +sa.nl_pid = i++;
>> +
>> +if (i > 10) {
>> +LOG(ERROR, "netlink bind error");
>> +goto out;
>> +}
>> +
>> +ret = bind(skfd, (struct sockaddr *), sizeof(sa));
>> +if (ret < 0 && errno == EADDRINUSE) {
>> +LOG(ERROR, "colo index %d has already in used", sa.nl_pid);
>> +goto retry;
>> +} else if (ret < 0) {
>> +LOG(ERROR, "netlink bind error");
>> +goto out;
>> +}
>> +
>> +cps->index = sa.nl_pid;
>> +ret = colo_proxy_send(cps, NULL, 0, COLO_PROXY_INIT);
>> +if (ret < 0) {
>> +goto out;
>> +}
> 
> Ditto. You can remove it.

OK, will check all codes.

Thanks
Wen Congyang

> 
>> +/* receive ack */
>> +size = colo_proxy_recv(cps, , 50);
>> +if (size < 0) {
>> +LOG(ERROR, "Can't recv msg from kernel by netlink: %s",
>> +strerror(errno));
>> +goto out;
>> +}
>> +
>> +if (size) {
>> +h = (struct nlmsghdr *)buff;
>> +if (h->nlmsg_type == NLMSG_ERROR) {
>> +/* ack's type is NLMSG_ERROR */
>> +struct nlmsgerr *err = (struct nlmsgerr *)NLMSG_DATA(h);
>> +
>> +if (size - sizeof(*h) < sizeof(*err)) {
>> +LOG(ERROR, "NLMSG_LENGTH is too short");
>> +goto out;
>> +}
>> +
>> +if (err->error) {
>> +LOG(ERROR, "NLMSG_ERROR contains error %d", err->error);
>> +goto out;
>> +}
>> +}
>> +}
>> +
>> +ret = 0;
>> +
>> +out:
>> +free(buff);
>> +if (ret) {
>> +close(cps->sock_fd);
>> +cps->sock_fd = -1;
>> +}
>> +return ret;
>> +}
>> +
>> +void colo_proxy_teardown(libxl__colo_proxy_state *cps)
>> +{
>> +if (cps->sock_fd >= 0) {
>> +close(cps->sock_fd);
>> +cps->sock_fd = -1;
>> +}
>> +}
>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>> index 3af5fdd..3b44b09 100644
>> --- a/tools/libxl/libxl_internal.h
>> +++ b/tools/libxl/libxl_internal.h
>> @@ -3112,6 +3112,15 @@ libxl__stream_read_inuse(const 
>> libxl__stream_read_state *stream)
>>  }
>>  
>>  /*- colo related state structure -*/
>> +typedef struct libxl__colo_proxy_state libxl__colo_proxy_state;
>> +struct libxl__colo_proxy_state {
>> +/* set by caller of colo_proxy_setup */
>> +libxl__ao *ao;
>> +
>> +int sock_fd;
>> +int index;
>> +};
>> +
>>  typedef struct libxl__colo_save_state libxl__colo_save_state;
>>  struct libxl__colo_save_state {
>>  int send_fd;
>> @@ -3126,6 +3135,9 @@ struct libxl__colo_save_state {
>>  /* private, used by qdisk block replication */
>>  bool qdisk_used;
>>  bool qdisk_setuped;
>> +
>> +/* private, used by colo-proxy */
>> +libxl__colo_proxy_state cps;
>>  };
>>  
>>  /*- Domain suspend (save) state structure -*/
>> @@ -3535,6 +3547,9 @@ struct libxl__colo_restore_state {
>>  bool qdisk_setuped;
>>  const char *host;
>>  const char *port;
>> +
>> +/* private, used by colo-proxy */
>> +libxl__colo_proxy_state cps;
>>  };
>>  
>>  struct libxl__domain_create_state {
>> -- 
>> 2.5.0
>>
>>
>>
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v11 10/27] tools/libxl: add back channel support to write stream

[Wen Congyang]

On 03/05/2016 01:00 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 10/27] tools/libxl: add back channel 
> support to write stream"):
>> From: Wen Congyang <we...@cn.fujitsu.com>
>>
>> Add back channel support to write stream. If the write stream is
>> a back channel stream, this means the write stream is used by
>> Secondary to send some records back.
> 
> The general idea seems fine to me but I want an opinion from Andrew.
> 
> If I'm not mistaken there is no call site for this yet.  In which case
> this should be mentioned in the commit message.
> 
>> +/*- checkpoint state -*/
>> +void libxl__stream_write_checkpoint_state(libxl__egc *egc,
>> +  libxl__stream_write_state *stream,
>> +  libxl_sr_checkpoint_state *srcs)
> 
> Firstly, missing blank line.
> 
> Secondly, reading all this leads me to think that maybe the
> `checkpoint_state' record should be called something different.  Is it
> only ever going to be used for COLO ?  Maybe it should be
> `COLOHA_STATE' or something (and all the functions etc. renamed
> consequently) ?

Another question: what should be renamed? The function name or the type name
or both?

Thanks
Wen Congyang

> 
> What do you think ?
> 
> Thanks,
> Ian.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v11 07/27] docs/libxl: Introduce CHECKPOINT_CONTEXT to support migration v2 colo streams

[Wen Congyang]

On 03/05/2016 12:51 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 07/27] docs/libxl: Introduce 
> CHECKPOINT_CONTEXT to support migration v2 colo streams"):
>> From: Wen Congyang <we...@cn.fujitsu.com>
> 
> I think we will want to see an ack from Andy Cooper on this, in due
> course.
> 
>> It is the negotiation record for COLO.
>> Primary->Secondary:
>> control_id  0x: Secondary VM is out of sync, start a new 
>> checkpoint
>> Secondary->Primary:
>> 0x0001: Secondary VM is suspended
>> 0x0002: Secondary VM is ready
>> 0x0003: Secondary VM is resumed
> 
> I don't think it is necessary to repeat the enum assignment here in
> the commit message.

OK, will fix it in the next version.

> 
> 
>> +CHECKPOINT\_STATE
>> +--
> 
> This documentation patch ought to explicitly mention COLO, and have
> cross-references to the various documents (eg, the README added in the
> previous patch).
> 
>> +A checkpoint state record contains the control information for checkpoint.
>> +
>> + 0 1 2 3 4 5 6 7 octet
>> ++++
>> +| control_id | padding|
>> ++++
>> +
>> +
>> +FieldDescription
>> + ---
>> +control_id   0x: Secondary VM is out of sync, start a new 
>> checkpoint
>> + (Primary -> Secondary)
>> +
>> + 0x0001: Secondary VM is suspended (Secondary -> 
>> Primary)
>> +
>> + 0x0002: Secondary VM is ready (Secondary -> Primary)
>> +
>> + 0x0003: Secondary VM is resumed (Secondary -> Primary)
> 
> I think this should be accompanied by an explanation of what order
> these messages are sent in, and what both ends may or may not do
> during that time.

OK, will fix it in the next version.

> 
> 
>> @@ -212,6 +214,11 @@ class VerifyLibxl(VerifyBase):
>>  if len(content) != 0:
>>  raise RecordError("Checkpoint end record with non-zero length")
>>  
>> +def verify_record_checkpoint_state(self, content):
>> +""" Checkpoint state """
>> +if len(content) == 0:
>> +raise RecordError("Checkpoint state record with zero length")
>> +
> 
> I'm not verify familiar with this area of the code, but I think that
> this should probably check that the control_id is as expected.  Can it
> know what the right sequencing is ?

To Andrew Cooper:
What is the purpost of this script? If it is not used for live system, I think
the stream should not contain checkpoint state record.

Thanks
Wen Congyang

> 
> Ian.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v3 1/3] libxc: move migration_stream's definition to xenguest.h

[Wen Congyang]

xc_domain_save() and xc_domain_restore's parameter will use this type,
so it should be public.

Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
---
v2->v3: Rename MIG_STREAM_* to XC_MIG_STREAM_*
 tools/libxc/include/xenguest.h |  7 ++-
 tools/libxc/xc_sr_common.h | 10 --
 tools/libxc/xc_sr_save.c   | 12 ++--
 3 files changed, 12 insertions(+), 17 deletions(-)

diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index affc42b..cf521c3 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -75,13 +75,18 @@ struct save_callbacks {
 void* data;
 };
 
+typedef enum {
+XC_MIG_STREAM_NONE, /* plain stream */
+XC_MIG_STREAM_REMUS,
+} xc_migration_stream_t;
+
 /**
  * This function will save a running domain.
  *
  * @parm xch a handle to an open hypervisor interface
  * @parm fd the file descriptor to save a domain to
  * @parm dom the id of the domain
- * @param checkpointed_stream MIG_STREAM_NONE if the far end of the stream
+ * @param checkpointed_stream XC_MIG_STREAM_NONE if the far end of the stream
  *doesn't use checkpointing
  * @return 0 on success, -1 on failure
  */
diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 66f595f..e7568b5 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -180,16 +180,6 @@ struct xc_sr_context
 
 xc_dominfo_t dominfo;
 
-/*
- * migration stream
- * 0: Plain VM
- * 1: Remus
- */
-enum {
-MIG_STREAM_NONE, /* plain stream */
-MIG_STREAM_REMUS,
-} migration_stream;
-
 union /* Common save or restore data. */
 {
 struct /* Save data. */
diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
index e258b7c..ab59673 100644
--- a/tools/libxc/xc_sr_save.c
+++ b/tools/libxc/xc_sr_save.c
@@ -629,7 +629,7 @@ static int send_domain_memory_live(struct xc_sr_context 
*ctx)
 if ( rc )
 goto out;
 
-if ( ctx->save.debug && ctx->save.checkpointed != MIG_STREAM_NONE )
+if ( ctx->save.debug && ctx->save.checkpointed != XC_MIG_STREAM_NONE )
 {
 rc = verify_frames(ctx);
 if ( rc )
@@ -758,7 +758,7 @@ static int save(struct xc_sr_context *ctx, uint16_t 
guest_type)
 
 if ( ctx->save.live )
 rc = send_domain_memory_live(ctx);
-else if ( ctx->save.checkpointed != MIG_STREAM_NONE )
+else if ( ctx->save.checkpointed != XC_MIG_STREAM_NONE )
 rc = send_domain_memory_checkpointed(ctx);
 else
 rc = send_domain_memory_nonlive(ctx);
@@ -778,7 +778,7 @@ static int save(struct xc_sr_context *ctx, uint16_t 
guest_type)
 if ( rc )
 goto err;
 
-if ( ctx->save.checkpointed != MIG_STREAM_NONE )
+if ( ctx->save.checkpointed != XC_MIG_STREAM_NONE )
 {
 /*
  * We have now completed the initial live portion of the checkpoint
@@ -799,7 +799,7 @@ static int save(struct xc_sr_context *ctx, uint16_t 
guest_type)
 if ( rc <= 0 )
 goto err;
 }
-} while ( ctx->save.checkpointed != MIG_STREAM_NONE );
+} while ( ctx->save.checkpointed != XC_MIG_STREAM_NONE );
 
 xc_report_progress_single(xch, "End of stream");
 
@@ -845,8 +845,8 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t 
dom,
 ctx.save.checkpointed = checkpointed_stream;
 
 /* If altering migration_stream update this assert too. */
-assert(checkpointed_stream == MIG_STREAM_NONE ||
-   checkpointed_stream == MIG_STREAM_REMUS);
+assert(checkpointed_stream == XC_MIG_STREAM_NONE ||
+   checkpointed_stream == XC_MIG_STREAM_REMUS);
 
 /*
  * TODO: Find some time to better tweak the live migration algorithm.
-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v3 3/3] libxl: rename checkpointed_stream to stream_type

[Wen Congyang]

Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
---
 tools/libxl/libxl.c  |  4 ++--
 tools/libxl/libxl.h  |  8 
 tools/libxl/libxl_create.c   |  4 ++--
 tools/libxl/libxl_dom_save.c |  6 +++---
 tools/libxl/libxl_internal.h |  2 +-
 tools/libxl/libxl_save_callout.c |  4 ++--
 tools/libxl/libxl_stream_read.c  |  4 ++--
 tools/libxl/libxl_stream_write.c |  2 +-
 tools/libxl/libxl_types.idl  |  2 +-
 tools/libxl/xl_cmdimpl.c | 16 
 10 files changed, 30 insertions(+), 22 deletions(-)

diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 4cdc169..7c2c9fc 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -876,7 +876,7 @@ int libxl_domain_remus_start(libxl_ctx *ctx, 
libxl_domain_remus_info *info,
 dss->live = 1;
 dss->debug = 0;
 dss->remus = info;
-dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
+dss->stream_type = LIBXL_CHECKPOINTED_STREAM_REMUS;
 
 assert(info);
 
@@ -937,7 +937,7 @@ int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, 
int fd, int flags,
 dss->type = type;
 dss->live = flags & LIBXL_SUSPEND_LIVE;
 dss->debug = flags & LIBXL_SUSPEND_DEBUG;
-dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_NONE;
+dss->stream_type = LIBXL_CHECKPOINTED_STREAM_NONE;
 
 rc = libxl__fd_flags_modify_save(gc, dss->fd,
  ~(O_NONBLOCK|O_NDELAY), 0,
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index f9e3ef5..86b1d06 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -863,6 +863,14 @@ void libxl_mac_copy(libxl_ctx *ctx, libxl_mac *dst, 
libxl_mac *src);
 #define LIBXL_HAVE_SRM_V1 1
 
 /*
+ * LIBXL_HAVE_STREAM_TYPE
+ *
+ * If this is define, then the libxl_domain_create_restore() interfaces;s
+ * parameter checkpointed_stream is renamed to stream_type
+ */
+#define LIBXL_HAVE_STREAM_TYPE 1
+
+/*
  * libxl_domain_build_info has the u.hvm.gfx_passthru_kind field and
  * the libxl_gfx_passthru_kind enumeration is defined.
 */
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index f1028bc..b28eb89 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -993,7 +993,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
 libxl_domain_config *const d_config = dcs->guest_config;
 const int restore_fd = dcs->restore_fd;
 libxl__domain_build_state *const state = >build_state;
-const int checkpointed_stream = dcs->restore_params.checkpointed_stream;
+const int stream_type = dcs->restore_params.stream_type;
 
 if (rc) {
 domcreate_rebuild_done(egc, dcs, rc);
@@ -1033,7 +1033,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
 dcs->srs.completion_callback = domcreate_stream_done;
 
 if (restore_fd >= 0) {
-switch (checkpointed_stream) {
+switch (stream_type) {
 case LIBXL_CHECKPOINTED_STREAM_REMUS:
 libxl__remus_restore_setup(egc, dcs);
 /* fall through */
diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
index f3288b9..ff92ef0 100644
--- a/tools/libxl/libxl_dom_save.c
+++ b/tools/libxl/libxl_dom_save.c
@@ -338,7 +338,7 @@ void libxl__domain_save(libxl__egc *egc, 
libxl__domain_save_state *dss)
 unsigned int nr_vnodes = 0, nr_vmemranges = 0, nr_vcpus = 0;
 libxl__domain_suspend_state *dsps = >dsps;
 
-if (dss->checkpointed_stream != LIBXL_CHECKPOINTED_STREAM_NONE && !r_info) 
{
+if (dss->stream_type != LIBXL_CHECKPOINTED_STREAM_NONE && !r_info) {
 LOG(ERROR, "Migration stream is checkpointed, but there's no "
"checkpoint info!");
 rc = ERROR_INVAL;
@@ -383,12 +383,12 @@ void libxl__domain_save(libxl__egc *egc, 
libxl__domain_save_state *dss)
 goto out;
 }
 
-if (dss->checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_REMUS) {
+if (dss->stream_type == LIBXL_CHECKPOINTED_STREAM_REMUS) {
 if (libxl_defbool_val(r_info->compression))
 dss->xcflags |= XCFLAGS_CHECKPOINT_COMPRESS;
 }
 
-if (dss->checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_NONE)
+if (dss->stream_type == LIBXL_CHECKPOINTED_STREAM_NONE)
 callbacks->suspend = libxl__domain_suspend_callback;
 
 callbacks->switch_qemu_logdirty = 
libxl__domain_suspend_common_switch_qemu_logdirty;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 005fe53..0aada0d 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3128,7 +3128,7 @@ struct libxl__domain_save_state {
 libxl_domain_type type;
 int live;
 int debug;
-int checkpointed_stream;
+libxl_checkpointed_stream stream_type;
 const libxl_domain_remus_info *remus;
 /* private */
 int rc;
diff --git a/tools/libxl/libxl

[Xen-devel] [PATCH v3 2/3] tools: change checkpointed_stream's type from int to xc_migration_stream_t

[Wen Congyang]

checkpointed_stream is also renamed to stream_type

Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
---
v2->v3: Rename checkpointed_stream to stream_type
 tools/libxc/include/xenguest.h  |  8 
 tools/libxc/xc_nomigrate.c  |  4 ++--
 tools/libxc/xc_sr_restore.c | 10 +-
 tools/libxc/xc_sr_save.c|  8 
 tools/libxl/libxl_save_helper.c | 42 -
 5 files changed, 36 insertions(+), 36 deletions(-)

diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index cf521c3..4f0b06e 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -86,14 +86,14 @@ typedef enum {
  * @parm xch a handle to an open hypervisor interface
  * @parm fd the file descriptor to save a domain to
  * @parm dom the id of the domain
- * @param checkpointed_stream XC_MIG_STREAM_NONE if the far end of the stream
+ * @param stream_type XC_MIG_STREAM_NONE if the far end of the stream
  *doesn't use checkpointing
  * @return 0 on success, -1 on failure
  */
 int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t 
max_iters,
uint32_t max_factor, uint32_t flags /* XCFLAGS_xxx */,
struct save_callbacks* callbacks, int hvm,
-   int checkpointed_stream);
+   xc_migration_stream_t stream_type);
 
 /* callbacks provided by xc_domain_restore */
 struct restore_callbacks {
@@ -121,7 +121,7 @@ struct restore_callbacks {
  * @parm hvm non-zero if this is a HVM restore
  * @parm pae non-zero if this HVM domain has PAE support enabled
  * @parm superpages non-zero to allocate guest memory with superpages
- * @parm checkpointed_stream non-zero if the far end of the stream is using 
checkpointing
+ * @parm stream_type non-zero if the far end of the stream is using 
checkpointing
  * @parm callbacks non-NULL to receive a callback to restore toolstack
  *   specific data
  * @return 0 on success, -1 on failure
@@ -131,7 +131,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, 
uint32_t dom,
   domid_t store_domid, unsigned int console_evtchn,
   unsigned long *console_mfn, domid_t console_domid,
   unsigned int hvm, unsigned int pae, int superpages,
-  int checkpointed_stream,
+  xc_migration_stream_t stream_type,
   struct restore_callbacks *callbacks);
 
 /**
diff --git a/tools/libxc/xc_nomigrate.c b/tools/libxc/xc_nomigrate.c
index c9124df..08e1f8c 100644
--- a/tools/libxc/xc_nomigrate.c
+++ b/tools/libxc/xc_nomigrate.c
@@ -23,7 +23,7 @@
 int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t 
max_iters,
uint32_t max_factor, uint32_t flags,
struct save_callbacks* callbacks, int hvm,
-   int checkpointed_stream)
+   xc_migration_stream_t stream_type)
 {
 errno = ENOSYS;
 return -1;
@@ -34,7 +34,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t 
dom,
   domid_t store_domid, unsigned int console_evtchn,
   unsigned long *console_mfn, domid_t console_domid,
   unsigned int hvm, unsigned int pae, int superpages,
-  int checkpointed_stream,
+  xc_migration_stream_t stream_type,
   struct restore_callbacks *callbacks)
 {
 errno = ENOSYS;
diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
index d4d33fd..819401d 100644
--- a/tools/libxc/xc_sr_restore.c
+++ b/tools/libxc/xc_sr_restore.c
@@ -725,7 +725,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, 
uint32_t dom,
   domid_t store_domid, unsigned int console_evtchn,
   unsigned long *console_gfn, domid_t console_domid,
   unsigned int hvm, unsigned int pae, int superpages,
-  int checkpointed_stream,
+  xc_migration_stream_t stream_type,
   struct restore_callbacks *callbacks)
 {
 struct xc_sr_context ctx =
@@ -739,16 +739,16 @@ int xc_domain_restore(xc_interface *xch, int io_fd, 
uint32_t dom,
 ctx.restore.console_domid = console_domid;
 ctx.restore.xenstore_evtchn = store_evtchn;
 ctx.restore.xenstore_domid = store_domid;
-ctx.restore.checkpointed = checkpointed_stream;
+ctx.restore.checkpointed = stream_type;
 ctx.restore.callbacks = callbacks;
 
 /* Sanity checks for callbacks. */
-if ( checkpointed_stream )
+if ( stream_type )
 assert(callbacks->checkpoint);
 
 DPRINTF("fd %d, dom %u, hvm %u, pae %u, superpages %d"
-", checkpointed_stream %d", io_fd, dom, hvm, pae,
-superpages, checkpointed_stream);
+", stream_type %d", io_fd, dom, hvm,

[Xen-devel] [PATCH v2 2/2] tools: change checkpointed_stream's type from int to xc_migration_stream_t

[Wen Congyang]

Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
---
 tools/libxc/include/xenguest.h  |  4 ++--
 tools/libxc/xc_nomigrate.c  |  4 ++--
 tools/libxc/xc_sr_restore.c |  2 +-
 tools/libxc/xc_sr_save.c|  2 +-
 tools/libxl/libxl_save_helper.c | 38 +++---
 5 files changed, 25 insertions(+), 25 deletions(-)

diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index 973e5a6..029082d 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -93,7 +93,7 @@ typedef enum {
 int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t 
max_iters,
uint32_t max_factor, uint32_t flags /* XCFLAGS_xxx */,
struct save_callbacks* callbacks, int hvm,
-   int checkpointed_stream);
+   xc_migration_stream_t checkpointed_stream);
 
 /* callbacks provided by xc_domain_restore */
 struct restore_callbacks {
@@ -131,7 +131,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, 
uint32_t dom,
   domid_t store_domid, unsigned int console_evtchn,
   unsigned long *console_mfn, domid_t console_domid,
   unsigned int hvm, unsigned int pae, int superpages,
-  int checkpointed_stream,
+  xc_migration_stream_t checkpointed_stream,
   struct restore_callbacks *callbacks);
 
 /**
diff --git a/tools/libxc/xc_nomigrate.c b/tools/libxc/xc_nomigrate.c
index c9124df..77db59c 100644
--- a/tools/libxc/xc_nomigrate.c
+++ b/tools/libxc/xc_nomigrate.c
@@ -23,7 +23,7 @@
 int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t 
max_iters,
uint32_t max_factor, uint32_t flags,
struct save_callbacks* callbacks, int hvm,
-   int checkpointed_stream)
+   xc_migration_stream_t checkpointed_stream)
 {
 errno = ENOSYS;
 return -1;
@@ -34,7 +34,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t 
dom,
   domid_t store_domid, unsigned int console_evtchn,
   unsigned long *console_mfn, domid_t console_domid,
   unsigned int hvm, unsigned int pae, int superpages,
-  int checkpointed_stream,
+  xc_migration_stream_t checkpointed_stream,
   struct restore_callbacks *callbacks)
 {
 errno = ENOSYS;
diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
index d4d33fd..b2a3558 100644
--- a/tools/libxc/xc_sr_restore.c
+++ b/tools/libxc/xc_sr_restore.c
@@ -725,7 +725,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, 
uint32_t dom,
   domid_t store_domid, unsigned int console_evtchn,
   unsigned long *console_gfn, domid_t console_domid,
   unsigned int hvm, unsigned int pae, int superpages,
-  int checkpointed_stream,
+  xc_migration_stream_t checkpointed_stream,
   struct restore_callbacks *callbacks)
 {
 struct xc_sr_context ctx =
diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
index e258b7c..fc51f60 100644
--- a/tools/libxc/xc_sr_save.c
+++ b/tools/libxc/xc_sr_save.c
@@ -830,7 +830,7 @@ static int save(struct xc_sr_context *ctx, uint16_t 
guest_type)
 int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom,
uint32_t max_iters, uint32_t max_factor, uint32_t flags,
struct save_callbacks* callbacks, int hvm,
-   int checkpointed_stream)
+   xc_migration_stream_t checkpointed_stream)
 {
 struct xc_sr_context ctx =
 {
diff --git a/tools/libxl/libxl_save_helper.c b/tools/libxl/libxl_save_helper.c
index 6bdcf13..5503311 100644
--- a/tools/libxl/libxl_save_helper.c
+++ b/tools/libxl/libxl_save_helper.c
@@ -246,14 +246,14 @@ int main(int argc, char **argv)
 
 if (!strcmp(mode,"--save-domain")) {
 
-io_fd =atoi(NEXTARG);
-uint32_t dom = strtoul(NEXTARG,0,10);
-uint32_t max_iters =   strtoul(NEXTARG,0,10);
-uint32_t max_factor =  strtoul(NEXTARG,0,10);
-uint32_t flags =   strtoul(NEXTARG,0,10);
-int hvm =  atoi(NEXTARG);
-unsigned cbflags = strtoul(NEXTARG,0,10);
-int checkpointed_stream =  strtoul(NEXTARG,0,10);
+io_fd = atoi(NEXTARG);
+uint32_t dom =  strtoul(NEXTARG,0,10);
+uint32_t max_iters =strtoul(NEXTARG,0,10);
+uint32_t max_factor =   strtoul(NEXTARG,0,10);
+uint32_t flags =strtoul(NEXTARG,0,10);
+int hvm =   atoi(NEXTARG);
+

[Xen-devel] [PATCH v2 1/2] libxc: move migration_stream's definition to xenguest.h

[Wen Congyang]

xc_domain_save() and xc_domain_restore's parameter will use this type,
so it should be public.

Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
---
 tools/libxc/include/xenguest.h |  5 +
 tools/libxc/xc_sr_common.h | 10 --
 2 files changed, 5 insertions(+), 10 deletions(-)

diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index affc42b..973e5a6 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -75,6 +75,11 @@ struct save_callbacks {
 void* data;
 };
 
+typedef enum {
+MIG_STREAM_NONE, /* plain stream */
+MIG_STREAM_REMUS,
+} xc_migration_stream_t;
+
 /**
  * This function will save a running domain.
  *
diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 66f595f..e7568b5 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -180,16 +180,6 @@ struct xc_sr_context
 
 xc_dominfo_t dominfo;
 
-/*
- * migration stream
- * 0: Plain VM
- * 1: Remus
- */
-enum {
-MIG_STREAM_NONE, /* plain stream */
-MIG_STREAM_REMUS,
-} migration_stream;
-
 union /* Common save or restore data. */
 {
 struct /* Save data. */
-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH] libxc: move migration_stream's definition to xenguest.h

[Wen Congyang]

On 03/08/2016 06:38 PM, Andrew Cooper wrote:
> On 08/03/16 05:32, Wen Congyang wrote:
>> xc_save_domain()'s parameter use this type, so it should
>> be public.
> 
> xc_domain_save() currently uses an int, which also needs fixing.

OK. Will fix it in the next version.

> 
>>
>> Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
> 
> Does this even compile?  You have removed a variable without any
> replacement.

Yes, I compile it. The variable is not used.

> 
>> ---
>>  tools/libxc/include/xenguest.h | 10 ++
>>  tools/libxc/xc_sr_common.h | 10 --
>>  2 files changed, 10 insertions(+), 10 deletions(-)
>>
>> diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
>> index affc42b..888536e 100644
>> --- a/tools/libxc/include/xenguest.h
>> +++ b/tools/libxc/include/xenguest.h
>> @@ -238,4 +238,9 @@ xen_pfn_t *xc_map_m2p(xc_interface *xch,
>>unsigned long max_mfn,
>>int prot,
>>unsigned long *mfn0);
>> +
>> +typedef enum {
>> +MIG_STREAM_NONE, /* plain stream */
>> +MIG_STREAM_REMUS,
>> +} migration_stream;
> 
> This typedef should be beside xc_domain_save() as that is where it is
> intended to be used.  It also needs xc_ prefixes as it is part of the
> public interface, and a typedef wants a _t suffix.

OK. Will fix it in the next version.

Thanks
Wen Congyang

> 
> ~Andrew
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH] libxc: move migration_stream's definition to xenguest.h

[Wen Congyang]

xc_save_domain()'s parameter use this type, so it should
be public.

Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
---
 tools/libxc/include/xenguest.h | 10 ++
 tools/libxc/xc_sr_common.h | 10 --
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index affc42b..888536e 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -238,4 +238,9 @@ xen_pfn_t *xc_map_m2p(xc_interface *xch,
   unsigned long max_mfn,
   int prot,
   unsigned long *mfn0);
+
+typedef enum {
+MIG_STREAM_NONE, /* plain stream */
+MIG_STREAM_REMUS,
+} migration_stream;
 #endif /* XENGUEST_H */
diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 66f595f..e7568b5 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -180,16 +180,6 @@ struct xc_sr_context
 
 xc_dominfo_t dominfo;
 
-/*
- * migration stream
- * 0: Plain VM
- * 1: Remus
- */
-enum {
-MIG_STREAM_NONE, /* plain stream */
-MIG_STREAM_REMUS,
-} migration_stream;
-
 union /* Common save or restore data. */
 {
 struct /* Save data. */
-- 
2.5.0



___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v11 18/27] implement the cmdline for COLO

[Wen Congyang]

On 03/05/2016 01:22 AM, Ian Jackson wrote:
>> --- a/docs/man/xl.pod.1
>> +++ b/docs/man/xl.pod.1
> ...
>> + COLO support in xl is still in experimental (proof-of-concept) phase.
>> + There is no support for network or disk at the moment.
> 
> I think you need to spell out the lack of storage and network handling
> means that the guest will corrupt its disk and confuse its network
> peers.

OK, will fix it in the next version.

> 
>> @@ -875,7 +890,10 @@ int libxl_domain_remus_start(libxl_ctx *ctx, 
>> libxl_domain_remus_info *info,
>>  dss->live = 1;
>>  dss->debug = 0;
>>  dss->remus = info;
>> -dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
>> +if (libxl_defbool_val(info->colo))
>> +dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_COLO;
>> +else
>> +dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
> 
> If you prefer, a ? : expression would do here as well.
> 
>  +dss->checkpointed_stream =
>libxl_defbool_val() ? LIBXL_CHECKPOINTED_STREAM_COLO :...

If so, this line will contains more than 80 characters. So I will not
change it.

Thanks
Wen Congyang

> 
> (only completed with sensible formatting).  Up to you - it's fine as
> it is, too.
> 
> Most of this patch looks good to me.
> 
> Ian.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v11 17/27] libxc/save: support COLO save

[Wen Congyang]

On 03/05/2016 01:18 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 17/27] libxc/save: support COLO save"):
>> From: Wen Congyang <we...@cn.fujitsu.com>
>>
>> After suspend primary vm, get dirty bitmap on secondary vm,
>> and send pages both dirty on primary/secondary to secondary.
> 
> This patch again seems like a plausible kind of thing.  Again, I'd
> like to hear from Andrew.
> 
>> +static int merge_secondary_dirty_bitmap(struct xc_sr_context *ctx)
>> +{
> 
> This function might want the word `colo' in its name somewhere.

OK, will fix it in the next version.

Thanks
Wen Congyang

> 
> Thanks,
> Ian.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v11 15/27] primary vm suspend/resume/checkpoint code

[Wen Congyang]

On 03/05/2016 01:14 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 15/27] primary vm suspend/resume/checkpoint 
> code"):
>> From: Wen Congyang <we...@cn.fujitsu.com>
> 
> I would look at this on the same basis as the previous patch.
> 
>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>> index 48b4858..5160939 100644
>> --- a/tools/libxl/libxl_internal.h
>> +++ b/tools/libxl/libxl_internal.h
> 
>> +struct libxl__stream_read_state {
>> +/* filled by the user */
>> +libxl__ao *ao;
>> +libxl__domain_create_state *dcs;
>> +int fd;
>> +bool legacy;
> 
> Can you please split out this code motion into a separate patch ?
> As it is it is very difficult to review.

OK, will fix it in the next version.

Thanks
Wen Congyang

> 
> Thanks,
> Ian.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v11 14/27] secondary vm suspend/resume/checkpoint code

[Wen Congyang]

On 03/05/2016 01:11 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 14/27] secondary vm 
> suspend/resume/checkpoint code"):
>> From: Wen Congyang <we...@cn.fujitsu.com>
>>
>> Secondary vm is running in colo mode. So we will do
>> the following things again and again:
> 
> I don't propose to review this in detail.  Skimreading it, it looks
> plausible.  I don't think a detailed review is needed.
> 
> I will review the changes to the core code.
> 
>> diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
>> index 5d9f497..2bfed64 100644
>> --- a/tools/libxc/xc_sr_common.h
>> +++ b/tools/libxc/xc_sr_common.h
>> @@ -184,10 +184,12 @@ struct xc_sr_context
>>   * migration stream
>>   * 0: Plain VM
>>   * 1: Remus
>> + * 2: COLO
>>   */
>>  enum {
>>  MIG_STREAM_NONE, /* plain stream */
>>  MIG_STREAM_REMUS,
>> +MIG_STREAM_COLO,
> 
> I think this shows that the duplicated list (in the comment, above the
> enum) is a mistake.  I would prefer it to be removed.

Do you mean remove the comments?

> 
>> +/* = colo: common functions = */

Add a blank line here? Will fix it in the next version.

>> +static void colo_enable_logdirty(libxl__colo_restore_state *crs, libxl__egc 
>> *egc)
> 
> Here's another missing blank line.  This seems to be a general theme:
> can you change this everywhere ?  Thanks.
> 
>> @@ -994,6 +1011,8 @@ static void domcreate_bootloader_done(libxl__egc *egc,
>>  const int restore_fd = dcs->restore_fd;
>>  libxl__domain_build_state *const state = >build_state;
>>  const int checkpointed_stream = dcs->restore_params.checkpointed_stream;
>> +libxl__colo_restore_state *const crs = >crs;
>> +libxl_domain_build_info *const info = _config->b_info;
>>  
>>  if (rc) {
>>  domcreate_rebuild_done(egc, dcs, rc);
>> @@ -1022,6 +1041,13 @@ static void domcreate_bootloader_done(libxl__egc *egc,
>>  
>>  /* Restore */
>>  
>> +/* COLO only supports HVM now */
>> +if (info->type != LIBXL_DOMAIN_TYPE_HVM &&
>> +checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_COLO) {
>> +rc = ERROR_FAIL;
>> +goto out;
> 
> Please log something here, or it may be very mysterious.

OK. Will add some comments to explain why only support pure HVM now.

> 
>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>> index 6307b71..48b4858 100644
>> --- a/tools/libxl/libxl_internal.h
>> +++ b/tools/libxl/libxl_internal.h
>> @@ -87,6 +87,8 @@
>> @@ -3468,7 +3464,6 @@ libxl__stream_read_inuse(const 
>> libxl__stream_read_state *stream)
>>  return stream->running;
>>  }
>>  
>> -
>>  struct libxl__domain_create_state {
>>  /* filled in by user */
>>  libxl__ao *ao;
>> @@ -3484,6 +3479,8 @@ struct libxl__domain_create_state {
> 
> Unintentional whitespace change.

Sorry for the mistake. Will fix it in the next version.

Thanks
Wen Congyang

> 
> 
> Ian.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v11 12/27] tools/libx{l, c}: introduce wait_checkpoint callback

[Wen Congyang]

On 03/05/2016 04:23 AM, Konrad Rzeszutek Wilk wrote:
> On Fri, Mar 04, 2016 at 05:03:16PM +, Ian Jackson wrote:
>> Changlong Xie writes ("[PATCH v11 12/27] tools/libx{l,c}: introduce 
>> wait_checkpoint callback"):
>>> From: Wen Congyang <we...@cn.fujitsu.com>
>>>
>>> Under COLO, we are doing checkpoint on demand, if this
>>> callback returns 1, we will take another checkpoint.
>>> 0 indicates unexpected error.
>>
>> This doesn't seem to have a corresponding implementation.  I think the
>> implementation ought to be in the same patch.
>>
>> If 0 is always an `unexpected error', perhaps the return value should
>> be an error code or something ?  I'm not sure.
> 
> I struggled with this API.
> 
> I like the idea of that negative value would imply 'unexpected error'.
> And 1 for 'OK, take another checkpoint'. Not sure if zero would be a valid
> return value..

IIRC, save/restore callback always use 0 for unexpected error, 1 for OK.
negative value for pipe is broken.

Thanks
Wen Congyang

> 
> 
>>
>> Ian.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v11 10/27] tools/libxl: add back channel support to write stream

[Wen Congyang]

On 03/05/2016 01:00 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 10/27] tools/libxl: add back channel 
> support to write stream"):
>> From: Wen Congyang <we...@cn.fujitsu.com>
>>
>> Add back channel support to write stream. If the write stream is
>> a back channel stream, this means the write stream is used by
>> Secondary to send some records back.
> 
> The general idea seems fine to me but I want an opinion from Andrew.
> 
> If I'm not mistaken there is no call site for this yet.  In which case
> this should be mentioned in the commit message.
> 
>> +/*- checkpoint state -*/
>> +void libxl__stream_write_checkpoint_state(libxl__egc *egc,
>> +  libxl__stream_write_state *stream,
>> +  libxl_sr_checkpoint_state *srcs)
> 
> Firstly, missing blank line.
> 
> Secondly, reading all this leads me to think that maybe the
> `checkpoint_state' record should be called something different.  Is it
> only ever going to be used for COLO ?  Maybe it should be

Yes, it is only used for COLO now.

> `COLOHA_STATE' or something (and all the functions etc. renamed
> consequently) ?
> 
> What do you think ?

COLO is FT, not HA. What aboyt COLOFT_STATE?

Thanks
Wen Congyang

> 
> Thanks,
> Ian.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v11 20/27] Support colo mode for qemu disk

[Wen Congyang]

On 03/05/2016 04:30 AM, Konrad Rzeszutek Wilk wrote:
> On Fri, Mar 04, 2016 at 05:52:09PM +, Ian Jackson wrote:
>> Changlong Xie writes ("[PATCH v11 20/27] Support colo mode for qemu disk"):
>>> +Enable COLO HA for disk. For better understanding block replication on
>>> +QEMU, please refer to:
>>> +http://wiki.qemu.org/Features/BlockReplication
>>
>> Sorry, I missed this link on my first pass.  I still think that at the
>> very least this needs something more user-facing (ie, how should one
>> set this up).
>>
>> But, I'm kind of worried that qemu is the wrong place to be doing
>> this.
>>
>> How can this be made to work with PV guests ?
> 
> QEMU can also serve PV guests (qdisk).
> 
> I think your question is more of - what about making this work with
> PV block backend?

I don't know how to work with PV block backend. It is one reason that
why we only support pure HVM now.
For PV block backend, there is also other problem. For exampe resuming
it in the secondary side is very slow, because we need to disconnect and
reconnect.

Thanks
Wen Congyang

>>
>> What if an HVM guest has PV-on-HVM drivers ?  In this case there might
>> be two relevant qemus, one for the qdisk Xen PV block backend, and one
>> for the emulated IDE.
> 
> In both cases QEMU would use the same underlaying API to actually write/read
> out the blocks. That API would then use NBD, etc to replicate writes.
> 
> Maybe a little ASCII art?
> 
>   qdisk  ide
> \/
>\  /
>block API
> |
>QCOW2
> |
>NBD
> 
> Or such?
> 
>>
>> I don't understand how discrepant writes are detected.  Surely they
>> might occur and should trigger a resynch ?
>>
>> Ian.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v11 20/27] Support colo mode for qemu disk

[Wen Congyang]

On 03/05/2016 01:44 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 20/27] Support colo mode for qemu disk"):
>> From: Wen Congyang <we...@cn.fujitsu.com>
>>
>> Usage: disk = 
>> ['...,colo,colo-host=xxx,colo-port=xxx,colo-export=xxx,active-disk=xxx,hidden-disk=xxx...']
>> For QEMU block replication details:
>> http://wiki.qemu.org/Features/BlockReplication
> 
> So now I am slightly confused by the design, I think.
> 
> When you replicate a VM with COLO using xl, its memory state is
> transferred over ssh.  But its disk replication is done unencrypted
> and unauthenticated ?

Yes, it is a problem. I will think how to improve it.

> 
> And the disk replication is, out of band, and needs to be configured
> separately ?  This is rather awkward, although maybe not a
> showstopper.  (Maybe we can have a plan to fix it in the future...)

colo-host,colo-port should be the global configuration. And colo-export,
active-disk,hidden-disk must be configured separately, because each
disk should have a different configuration.

> 
> And, how does the disk replication, which doesn't depend on there
> being xl running, relate to the vm state replication, which does ?  I
> think at the very least I'd like to see some information about the
> principles of operation - either explained, or referred to, in the
> user manual.

OK. The disk replication doesn't depend on xl. We only can operate it
via qemu monitor command:
1. stop the vm
2. do the checkpoint
3. start the vm
1/3 is suspend/resume the guest. We only need to do 2 when both vm are
in the consistent state.

> 
> Is it possible to use COLO with an existing full-service disk
> replication service such as DRBD ?

DRBD doesn's support the case like COLO. Because both primary guest
and secondary guest need to write to the disk.

> 
>> +(a) An example for COLO replication's configuration: disk 
>> =['...,colo,colo-host
>> +=xxx,colo-port=xxx,colo-export=xxx,active-disk=xxx,hidden-disk=xxx...']
>> +
>> +=item B  :Secondary host's ip address.
>> +
>> +=item B  :Secondary host's port, we will run a nbd server on
>> +secondary host, and the nbd server will listen this port.
>> +
>> +=item B:Nbd server's disk export name of secondary host.
>> +
>> +=item B:Secondary's guest write will be buffered in this 
>> disk,
>> +and it's used by secondary.
>> +
>> +=item B:Primary's modified contents will be buffered in 
>> this
>> +disk, and it's used by secondary.
> 
> What would a typical configuration look like ?  I don't understand the
> relationship between active-disk and hidden-disk, etc.

QEMU has a feature: backing file
For example: A's backing file is B
1. If we read from A, but the sector is not allocated in A. We wil return a zero
   sector to the guest. If A has a backing file, we will read the sector from B
   instead of returning a zero sector.
2. The backing file doesn't affect the write operation.

QEMU has another feature: backup block job
Backup job has two file: one is source and another is the target. It has some 
running
mode. For block replication, we use the mode "sync=none". In this mode, we will 
read
the data from the source disk before we modify it, and write it to the target 
disk.
We keep a bitmap to remeber which sector is backuped from the source disk to the
target disk. If the target disk is an empty disk, and empty disk's backing file 
is
the source disk, we can read from the target disk to get the source disk's 
originnal data.


How does block replication work:
A. primary qemu:
1. use the block driver quorum: it will read from all children and write to all 
children.
   child 0: real disk
   child 1: nbd client
   reading from child 1 will fail, but we use the fifo mode. In this mode, we 
read from
   child 0 will success and we don't read from child 0
   write to child 1: because child 1 is nbd client, it will forward the write 
request to
   nbd server

B. secondary qemu:
We have 3 disks: active disk(called it A), hidden disk(called it H), and 
secondary disk
(real disk, called it S).
A's backing file is H, and H's backing file is S.
We also start a backup job: the source disk is S, and the target disk is H.
we run nbd server in secondary qemu. And the nbd server will write to S.

Before resuming both primary vm and secondary vm: the state is:
1. primary disk and secondary disk are in the consistent state(contain the same 
data)
2. active disk and hidden disk are the empty disk
When the guest is running:
1. NBD server receives the primary write operation and writes the data to S
2. Before we write data to S, the backup job will read the original data and 
backup it
   to H
3. The secondary vm will write data to A.
4. If secondary vm will read data from A:
   I. If the sector 

Re: [Xen-devel] Patching error while setting up COLO

[Wen Congyang]

On 03/05/2016 09:51 AM, Yu-An(Victor) Chen wrote:
> Hi Congyang,
> 
> Thanks for your reply,
> 
> even with your script, and I modify the "path_to_xen_source" to point where 
> my xen directory is. I still got this error.
> 
> ERROR: User requested feature xen
>configure was not able to find it.
>Install xen devel
> 
> What do you think what I am missing? Thank you!

Do you build xen before?

Thanks
Wen Congyang

> 
> Victor
> 
> 
> 
> On Thu, Mar 3, 2016 at 6:15 PM, Wen Congyang <we...@cn.fujitsu.com 
> <mailto:we...@cn.fujitsu.com>> wrote:
> 
> On 03/04/2016 10:01 AM, Yu-An(Victor) Chen wrote:
> > Hi,
> >
> > So I git clone 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_wencongyang_qemu-2Dxen.git=CwICaQ=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI=IitX1U91-NhsQt0q4MJOLQ=4j1T2HKL4uKodf62b4Tz1XtOvX81uAqCqfOcD90CRAY=s0fo5ej8_vZ1PmOkDCuyIroS5Zi_KpDSHI8jqodSmrg=
> >
> > but i only see branch "con-xen-v2" instead of " colo-xen-v2" so I 
> assume I use just use con-xen-v2.
> >
> > But then the following step:
> >
> > in both ~/qemu-colo and ~/qemu-xen
> >
> > ./configure --enable-xen --target-list=x86_64-softmmu 
> --extra-cflags="-I$path_to_xen_source/tools/include 
> -I$path_to_xen_source/tools/libxc -I$path_to_xen_source/tools/xenstore" 
> --extra-ldflags="-L$path_to_xen_source/tools/libxc 
> -L$path_to_xen_source/tools/xenstore"
> 
> 
> This command line is out of dated. The following is my building scripts:
> #! /bin/bash
> 
> path_to_xen_source=/work/src/xen
> #./configure --enable-xen --target-list=i386-softmmu \
> #--extra-cflags="-I$path_to_xen_source/tools/include 
> -I$path_to_xen_source/tools/libxc/include 
> -I$path_to_xen_source/tools/xenstore/include" \
> #--extra-ldflags="-L$path_to_xen_source/tools/libxc 
> -L$path_to_xen_source/tools/xenstore"
> 
> extra_cflags=""
> extra_cflags+=" -DXC_WANT_COMPAT_EVTCHN_API=1"
> extra_cflags+=" -DXC_WANT_COMPAT_GNTTAB_API=1"
> extra_cflags+=" -DXC_WANT_COMPAT_MAP_FOREIGN_API=1"
> extra_cflags+=" -I$path_to_xen_source/tools/include"
> extra_cflags+=" -I$path_to_xen_source/tools/libs/toollog/include"
> extra_cflags+=" -I$path_to_xen_source/tools/libs/evtchn/include"
> extra_cflags+=" -I$path_to_xen_source/tools/libs/gnttab/include"
> extra_cflags+=" -I$path_to_xen_source/tools/libs/foreignmemory/include"
> extra_cflags+=" -I$path_to_xen_source/tools/libxc/include"
> extra_cflags+=" -I$path_to_xen_source/tools/xenstore/include"
> extra_cflags+=" -I$path_to_xen_source/tools/xenstore/compat/include"
> extra_cflags+=" "
> 
> extra_ldflags=""
> extra_ldflags+=" -L$path_to_xen_source/tools/libxc"
> extra_ldflags+=" -L$path_to_xen_source/tools/xenstore"
> extra_ldflags+=" -L$path_to_xen_source/tools/libs/evtchn"
> extra_ldflags+=" -L$path_to_xen_source/tools/libs/gnttab"
> extra_ldflags+=" -L$path_to_xen_source/tools/libs/foreignmemory"
> extra_ldflags+=" -Wl,-rpath-link=$path_to_xen_source/tools/libs/toollog"
> extra_ldflags+=" -Wl,-rpath-link=$path_to_xen_source/tools/libs/evtchn"
> extra_ldflags+=" -Wl,-rpath-link=$path_to_xen_source/tools/libs/gnttab"
> extra_ldflags+=" -Wl,-rpath-link=$path_to_xen_source/tools/libs/call"
> extra_ldflags+=" 
> -Wl,-rpath-link=$path_to_xen_source/tools/libs/foreignmemory"
> extra_ldflags+=" "
> 
> ./configure --enable-xen --target-list=i386-softmmu \
> --extra-cflags="$extra_cflags" \
> --extra-ldflags="$extra_ldflags"
> 
> if [[ $? -ne 0 ]]; then
> exit 1
> fi
> 
> #make -j8 && make clean
> make -j8
> 
> You can find the newest building way in tools/Makefile(xen's codes):
> subdir-all-qemu-xen-dir: qemu-xen-dir-find
> if test -d $(QEMU_UPSTREAM_LOC) ; then \
> source=$(QEMU_UPSTREAM_LOC); \
> else \
>         source=.; \
> fi; \
> cd qemu-xen-dir; \
> if $$source/scripts/tracetool.py --check-backend --backend stderr 
> ; then \
> enable_trace_backend='--enable-trace-backend=stderr'; \
> else \
> enable_t

Re: [Xen-devel] Patching error while setting up COLO

[Wen Congyang]

On 03/04/2016 10:01 AM, Yu-An(Victor) Chen wrote:
> Hi,
> 
> So I git clone https://github.com/wencongyang/qemu-xen.git
> 
> but i only see branch "con-xen-v2" instead of " colo-xen-v2" so I assume I 
> use just use con-xen-v2.
> 
> But then the following step:
> 
> in both ~/qemu-colo and ~/qemu-xen
> 
> ./configure --enable-xen --target-list=x86_64-softmmu 
> --extra-cflags="-I$path_to_xen_source/tools/include 
> -I$path_to_xen_source/tools/libxc -I$path_to_xen_source/tools/xenstore" 
> --extra-ldflags="-L$path_to_xen_source/tools/libxc 
> -L$path_to_xen_source/tools/xenstore"


This command line is out of dated. The following is my building scripts:
#! /bin/bash

path_to_xen_source=/work/src/xen
#./configure --enable-xen --target-list=i386-softmmu \
#--extra-cflags="-I$path_to_xen_source/tools/include 
-I$path_to_xen_source/tools/libxc/include 
-I$path_to_xen_source/tools/xenstore/include" \
#--extra-ldflags="-L$path_to_xen_source/tools/libxc 
-L$path_to_xen_source/tools/xenstore"

extra_cflags=""
extra_cflags+=" -DXC_WANT_COMPAT_EVTCHN_API=1"
extra_cflags+=" -DXC_WANT_COMPAT_GNTTAB_API=1"
extra_cflags+=" -DXC_WANT_COMPAT_MAP_FOREIGN_API=1"
extra_cflags+=" -I$path_to_xen_source/tools/include"
extra_cflags+=" -I$path_to_xen_source/tools/libs/toollog/include"
extra_cflags+=" -I$path_to_xen_source/tools/libs/evtchn/include"
extra_cflags+=" -I$path_to_xen_source/tools/libs/gnttab/include"
extra_cflags+=" -I$path_to_xen_source/tools/libs/foreignmemory/include"
extra_cflags+=" -I$path_to_xen_source/tools/libxc/include"
extra_cflags+=" -I$path_to_xen_source/tools/xenstore/include"
extra_cflags+=" -I$path_to_xen_source/tools/xenstore/compat/include"
extra_cflags+=" "

extra_ldflags=""
extra_ldflags+=" -L$path_to_xen_source/tools/libxc"
extra_ldflags+=" -L$path_to_xen_source/tools/xenstore"
extra_ldflags+=" -L$path_to_xen_source/tools/libs/evtchn"
extra_ldflags+=" -L$path_to_xen_source/tools/libs/gnttab"
extra_ldflags+=" -L$path_to_xen_source/tools/libs/foreignmemory"
extra_ldflags+=" -Wl,-rpath-link=$path_to_xen_source/tools/libs/toollog"
extra_ldflags+=" -Wl,-rpath-link=$path_to_xen_source/tools/libs/evtchn"
extra_ldflags+=" -Wl,-rpath-link=$path_to_xen_source/tools/libs/gnttab"
extra_ldflags+=" -Wl,-rpath-link=$path_to_xen_source/tools/libs/call"
extra_ldflags+=" -Wl,-rpath-link=$path_to_xen_source/tools/libs/foreignmemory"
extra_ldflags+=" "

./configure --enable-xen --target-list=i386-softmmu \
--extra-cflags="$extra_cflags" \
--extra-ldflags="$extra_ldflags"

if [[ $? -ne 0 ]]; then
exit 1
fi

#make -j8 && make clean
make -j8

You can find the newest building way in tools/Makefile(xen's codes):
subdir-all-qemu-xen-dir: qemu-xen-dir-find  
if test -d $(QEMU_UPSTREAM_LOC) ; then \
source=$(QEMU_UPSTREAM_LOC); \  
else \  
source=.; \ 
fi; \   
cd qemu-xen-dir; \  
if $$source/scripts/tracetool.py --check-backend --backend stderr ; 
then \
enable_trace_backend='--enable-trace-backend=stderr'; \ 
else \  
enable_trace_backend='' ; \ 
fi ; \  
$$source/configure --enable-xen --target-list=i386-softmmu \
$(QEMU_XEN_ENABLE_DEBUG) \  
$$enable_trace_backend \
--prefix=$(LIBEXEC) \   
--libdir=$(LIBEXEC_LIB) \   
--includedir=$(LIBEXEC_INC) \


Thanks
Wen Congyang

> 
> 
> I got the following error message:
> 
> "ERROR: User requested feature xen
>configure was not able to find it.
>    Install xen devel"
> 
> I found out the the error came from just simply doing this:
> 
> ./configure --enable-xen  
> 
> I am thinking the reason is because I did this step wrong:
> 
> "path_to_xen_source=~/xen"
> 
> Do I just simply c

Re: [Xen-devel] Patching error while setting up COLO

[Wen Congyang]

On 03/03/2016 05:39 PM, Yu-An(Victor) Chen wrote:
> Hi Changlong,
> 
> Thanks for the reply,
> 
> Again when I am trying to do the following:
> 
> 5. build qemu-colo
> 1) cd ~/qemu-colo/; *git checkout colo-xen-v2*
> *
> *
> I got this error message *"error: pathspec 'colo-xen-v2' did not match any 
> file(s) known to git."* Even if I do git fetch, I still get the same error.
> 
> the qemu-colo I cloned from is provided by you 
> https://github.com/wencongyang/qemu-colo.git

https://github.com/wencongyang/qemu-xen, not qemu-colo

> 
> Thank you!
> 
> Victor
> 
> 
> 
> On Thu, Feb 25, 2016 at 9:07 PM, Changlong Xie  > wrote:
> 
> On 02/26/2016 12:55 PM, Yu-An(Victor) Chen wrote:
> 
> Hi Changlong,
> 
> Are you suggesting I should hold off on setting up COLO for now?
> 
> 
> No, just following my steps.
> 
> Thanks
> -Xie
> 
> 
> Thanks!
> 
> Victor
> 
> On Thu, Feb 25, 2016 at 8:19 PM, Changlong Xie 
> >
> wrote:
> 
> On 02/26/2016 11:38 AM, Yu-An(Victor) Chen wrote:
> 
> Hi Changlong,
> 
> Thanks for the reply!
> 
> So I am trying to follow your new instructions, but when I am 
> trying to do
> this:
> 
>cd ~/colo-proxy/; git checkout 405527cbfa9f
> 
> I got the following error:
> 
> "error: pathspec '405527cbfa9f' did not match any file(s) 
> known to git."
> 
> I assume it is just a typo? Thank you!
> 
> 
> Hi victor
> 
> Please git clone
> 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Pating_colo-2Dproxy_tree_changlox=CwICaQ=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI=IitX1U91-NhsQt0q4MJOLQ=pCAkg_8tEQmGEZZoUlyePZjK7z-6aEmp-n6UrQRLWo4=Ww-EAIszC-zQuVcDc4XpigwVbMG_4t2SpTg2PV6HTjM=
> *Notice* that, currently we implement colo proxy as a kernel 
> module what
> is a temporary measure. But further more we'll intergrate it in 
> qemu and
> drop this one, so both qemu-colo and xen-colo will share the same 
> proxy.
> Please don't test this colo proxy now, there maybe some bugs, but 
> it's
> acceptable.
> 
> Thanks
>  -Xie
> 
> 
> Victor
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v10 31/31] cmdline switches and config vars to control colo-proxy

[Wen Congyang]

On 03/02/2016 11:05 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:35AM +0800, Wen Congyang wrote:
>> Add cmdline switches to 'xl migrate-receive' command to specify
>> a domain-specific hotplug script to setup COLO proxy.
>>
>> Add a new config var 'colo.default.agentscript' to xl.conf, that
>> allows the user to override the default global script used to
>> setup COLO proxy.
>>
>> Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
>> Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
>> ---
>>  docs/man/xl.conf.pod.5  |  6 ++
>>  docs/man/xl.pod.1   |  1 -
>>  tools/libxl/libxl.c |  6 ++
>>  tools/libxl/libxl_create.c  | 14 --
>>  tools/libxl/libxl_types.idl |  1 +
>>  tools/libxl/xl.c|  3 +++
>>  tools/libxl/xl.h|  1 +
>>  tools/libxl/xl_cmdimpl.c| 47 
>> ++---
>>  8 files changed, 65 insertions(+), 14 deletions(-)
>>
>> diff --git a/docs/man/xl.conf.pod.5 b/docs/man/xl.conf.pod.5
>> index 8ae19bb..8f7fd28 100644
>> --- a/docs/man/xl.conf.pod.5
>> +++ b/docs/man/xl.conf.pod.5
>> @@ -111,6 +111,12 @@ Configures the default script used by Remus to setup 
>> network buffering.
>>  
>>  Default: C
>>  
>> +=item B

Re: [Xen-devel] [PATCH v10 24/31] Support colo mode for qemu disk

[Wen Congyang]

On 03/02/2016 11:04 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:28AM +0800, Wen Congyang wrote:
>> Usage: disk = 
>> ['...,colo,colo-host=xxx,colo-port=xxx,colo-export=xxx,active-disk=xxx,hidden-disk=xxx...']
>> For QEMU block replication details:
>> http://wiki.qemu.org/Features/BlockReplication
>>
>> Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
>> Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
>> ---
>>  docs/man/xl.pod.1   |   2 +-
>>  docs/misc/xl-disk-configuration.txt |  50 ++
>>  tools/libxl/libxl.c |  62 +++-
>>  tools/libxl/libxl_create.c  |  25 -
>>  tools/libxl/libxl_device.c  |  54 +++
>>  tools/libxl/libxl_dm.c  | 184 
>> ++--
>>  tools/libxl/libxl_types.idl |   7 ++
>>  tools/libxl/libxlu_disk_l.l |   7 ++
>>  8 files changed, 382 insertions(+), 9 deletions(-)
>>
>> diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
>> index 1c6dd87..4f1901d 100644
>> --- a/docs/man/xl.pod.1
>> +++ b/docs/man/xl.pod.1
>> @@ -454,7 +454,7 @@ N.B: Remus support in xl is still in experimental 
>> (proof-of-concept) phase.
>>   Disk replication support is limited to DRBD disks.
>>  
>>   COLO support in xl is still in experimental (proof-of-concept) phase.
>> - There is no support for network or disk at the moment.
>> + There is no support for network at the moment.
> 
> You need some document here for the syntax, otherwise users have no clue
> how to configure disk replicate support. I also won't be able to
> meaningfully review this patch without a reference.

OK. will fix it in the next version.

> 
>>  
>>  B
>>  
>> diff --git a/docs/misc/xl-disk-configuration.txt 
>> b/docs/misc/xl-disk-configuration.txt
>> index 29f6ddb..6f23c2d 100644
>> --- a/docs/misc/xl-disk-configuration.txt
>> +++ b/docs/misc/xl-disk-configuration.txt
>> @@ -234,6 +234,56 @@ were intentionally created non-sparse to avoid 
>> fragmentation of the
>>  file.
>>  
>>  
> 
> Some nitpicking about the format below.
> 
>> +===
>> +COLO PARAMETERS
>> +===
>> +
>> +
>> +colo
>> +
>> +
>> +Enable COLO HA for disk. For better understanding block replication on
>> +QEMU, please refer to:
>> +http://wiki.qemu.org/Features/BlockReplication
>> +
>> +
>> +colo-host
>> +-
> 
> Blank line here please.
> 
>> +Description:   Secondary host's address
>> +Mandatory: Yes when COLO enabled
>> +
>> +
>> +colo-port
>> +-
> 
> Ditto.
> 
>> +Description:   Secondary port
>> +   We will run a nbd server on secondary host,
>> +   and the nbd server will listen this port.
>> +Mandatory: Yes when COLO enabled
>> +
>> +
>> +colo-export
>> +-
> 
> Here as well. And some more "-"s to match "colo-export".
> 
>> +Description:   We will run a nbd server on secondary host,
>> +   exportname is the nbd server's disk export name.
>> +Mandatory: Yes when COLO enabled
>> +
>> +
>> +active-disk
>> +---
>> +
>> +Description:   This is used by secondary. Secondary guest's write
>> +   will be buffered in this disk.
>> +Mandatory: Yes when COLO enabled
>> +
>> +
>> +hidden-disk
>> +---
>> +
>> +Description:   This is used by secondary. It buffers the original
>> +   content that is modified by the primary VM.
>> +Mandatory: Yes when COLO enabled
>> +
>> +
> 
> The rest of the patch is mainly for manipulating QEMU parameters. I've
> skipped it for now.

If you want to know about how qemu block repication works, you can see:
http://wiki.qemu.org/Features/BlockReplication

> 
>>  
>>  DEPRECATED PARAMETERS, PREFIXES AND SYNTAXES
>>  
>> diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
>> index 12df81a..f691628 100644
>> --- a/tools/libxl/libxl.c
>> +++ b/tools/libxl/libxl.c
>> @@ -2309,6 +2309,8 @@ int libxl__device_disk_setdefault(libxl__gc *gc, 
>> libxl_device_disk *disk)
>>  int rc;
>>  
>>  libxl_

Re: [Xen-devel] [PATCH v10 22/31] implement the cmdline for COLO

[Wen Congyang]

On 03/02/2016 11:03 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:26AM +0800, Wen Congyang wrote:
> [...]
>> +if (libxl_defbool_val(info->colo)) {
>> +if (libxl_defbool_val(info->compression)) {
> 
> This can be simplified as
> 
>if (libxl_defbool_val(xxx) && libxl_defbool_val(yyy))

OK. will fix it in the next version.

> 
>> +LOG(ERROR, "cannot use memory checkpoint compression in COLO 
>> mode");
>> +rc = ERROR_FAIL;
>> +goto out;
>> +}
>> +}
>> +
>>  if (!libxl_defbool_val(info->allow_unsafe) &&
>>  (libxl_defbool_val(info->blackhole) ||
>>   !libxl_defbool_val(info->netbuf) ||
>> @@ -876,7 +892,10 @@ int libxl_domain_remus_start(libxl_ctx *ctx, 
>> libxl_domain_remus_info *info,
>>  dss->live = 1;
>>  dss->debug = 0;
>>  dss->remus = info;
>> -dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
>> +if (libxl_defbool_val(info->colo))
>> +dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_COLO;
>> +else
>> +dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
>>  
>>  assert(info);
>>  
>> diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
>> index df7268b..0dc7220 100644
>> --- a/tools/libxl/xl_cmdimpl.c
>> +++ b/tools/libxl/xl_cmdimpl.c
>> @@ -4440,6 +4440,8 @@ static void migrate_receive(int debug, int daemonize, 
>> int monitor,
>>  char rc_buf;
>>  char *migration_domname;
>>  struct domain_create dom_info;
>> +const char *ha = checkpointed == LIBXL_CHECKPOINTED_STREAM_COLO ?
>> + "COLO" : "Remus";
>>  
>>  signal(SIGPIPE, SIG_IGN);
>>  /* if we get SIGPIPE we'd rather just have it as an error */
>> @@ -4460,6 +4462,9 @@ static void migrate_receive(int debug, int daemonize, 
>> int monitor,
>>  dom_info.send_back_fd = send_fd;
>>  dom_info.migration_domname_r = _domname;
>>  dom_info.checkpointed_stream = checkpointed;
>> +if (checkpointed == LIBXL_CHECKPOINTED_STREAM_COLO)
>> +/* COLO uses stdout to send control message to master */
>> +dom_info.quiet = 1;
>>  
> 
> It seems that dom_info->quiet affects stderr, not stdout. See the only
> place that checks this in xl_cmdimpl.c.
> 
>>  rc = create_domain(_info);
>>  if (rc < 0) {
>> @@ -4472,11 +4477,12 @@ static void migrate_receive(int debug, int 
>> daemonize, int monitor,
>>  
>>  switch (checkpointed) {
>>  case LIBXL_CHECKPOINTED_STREAM_REMUS:
>> +case LIBXL_CHECKPOINTED_STREAM_COLO:
>>  /* If we are here, it means that the sender (primary) has crashed.
>>   * TODO: Split-Brain Check.
>>   */
>> -fprintf(stderr, "migration target: Remus Failover for domain %u\n",
>> -domid);
>> +fprintf(stderr, "migration target: %s Failover for domain %u\n",
>> +ha, domid);
>>  
>>  /*
>>   * If domain renaming fails, lets just continue (as we need the 
>> domain
>> @@ -4492,16 +4498,20 @@ static void migrate_receive(int debug, int 
>> daemonize, int monitor,
>>  rc = libxl_domain_rename(ctx, domid, migration_domname,
>>   common_domname);
>>  if (rc)
>> -fprintf(stderr, "migration target (Remus): "
>> +fprintf(stderr, "migration target (%s): "
>>  "Failed to rename domain from %s to %s:%d\n",
>> -migration_domname, common_domname, rc);
>> +ha, migration_domname, common_domname, rc);
>>  }
>>  
>> +if (checkpointed == LIBXL_CHECKPOINTED_STREAM_COLO)
>> +/* The guest is running after failover in COLO mode */
>> +exit(rc ? -ERROR_FAIL: 0);
>> +
>>  rc = libxl_domain_unpause(ctx, domid);
>>  if (rc)
>> -fprintf(stderr, "migration target (Remus): "
>> +fprintf(stderr, "migration target (%s): "
>>  "Failed to unpause domain %s (id: %u):%d\n",
>> -common_domname, domid, rc);
>> +ha, common_domname, domid, rc);
>>  
>>  exit(rc ? -ERROR_FAIL: 0);
>>  defaul

Re: [Xen-devel] [PATCH v10 10/31] tools/libxl: add back channel support to write stream

[Wen Congyang]

On 03/02/2016 11:02 PM, Wei Liu wrote:
> On Fri, Feb 26, 2016 at 10:11:27AM +0800, Wen Congyang wrote:
>> On 02/25/2016 11:54 PM, Wei Liu wrote:
>>> On Mon, Feb 22, 2016 at 10:52:14AM +0800, Wen Congyang wrote:
>>>> Add back channel support to write stream. If the write stream is
>>>> a back channel stream, this means the write stream is used by
>>>> Secondary to send some records back.
>>>>
>>>> Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
>>>> Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
>>>> ---
>>>>  tools/libxl/libxl_dom_save.c |  1 +
>>>>  tools/libxl/libxl_internal.h |  1 +
>>>>  tools/libxl/libxl_stream_write.c | 26 --
>>>>  3 files changed, 22 insertions(+), 6 deletions(-)
>>>>
>>>> diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
>>>> index 72b61c7..18946c5 100644
>>>> --- a/tools/libxl/libxl_dom_save.c
>>>> +++ b/tools/libxl/libxl_dom_save.c
>>>> @@ -404,6 +404,7 @@ void libxl__domain_save(libxl__egc *egc, 
>>>> libxl__domain_save_state *dss)
>>>>  dss->sws.ao  = dss->ao;
>>>>  dss->sws.dss = dss;
>>>>  dss->sws.fd  = dss->fd;
>>>> +dss->sws.back_channel = false;
>>>>  dss->sws.completion_callback = stream_done;
>>>>  
>>>>  libxl__stream_write_start(egc, >sws);
>>>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>>>> index 3d3e8e8..e02e554 100644
>>>> --- a/tools/libxl/libxl_internal.h
>>>> +++ b/tools/libxl/libxl_internal.h
>>>> @@ -3044,6 +3044,7 @@ struct libxl__stream_write_state {
>>>>  libxl__ao *ao;
>>>>  libxl__domain_save_state *dss;
>>>>  int fd;
>>>> +bool back_channel;
>>>>  void (*completion_callback)(libxl__egc *egc,
>>>>  libxl__stream_write_state *sws,
>>>>  int rc);
>>>> diff --git a/tools/libxl/libxl_stream_write.c 
>>>> b/tools/libxl/libxl_stream_write.c
>>>> index f6ea55d..5379126 100644
>>>> --- a/tools/libxl/libxl_stream_write.c
>>>> +++ b/tools/libxl/libxl_stream_write.c
>>>> @@ -49,6 +49,13 @@
>>>>   *  - if (hvm)
>>>>   *  - Emulator context record
>>>>   *  - Checkpoint end record
>>>> + *
>>>> + * For back channel stream:
>>>> + * - libxl__stream_write_start()
>>>> + *- Set up the stream to running state
>>>> + *
>>>> + * - Add a new API to write the record. When the record is written
>>>> + *   out, call stream->checkpoint_callback() to return.
>>>
>>> What does this mean? Which new API?
>>
>> The next patch introduces this API. The commits is very old.
>>
>> I think I can merge these two patches into one patch.
>>
> 
> Please reference the actual function / API.
> 
>>>
>>>>   */
>>>>  
>>>>  /* Success/error/cleanup handling. */
>>>> @@ -225,6 +232,15 @@ void libxl__stream_write_start(libxl__egc *egc,
>>>>  
>>>>  stream->running = true;
>>>>  
>>>> +dc->ao= ao;
>>>> +dc->readfd= -1;
>>>> +dc->copywhat  = "save v2 stream";
>>>> +dc->writefd   = stream->fd;
>>>> +dc->maxsz = -1;
>>>> +
>>>> +if (stream->back_channel)
>>>> +return;
>>>> +
>>>
>>> There seems to be very subtle change of behaviour.
>>>
>>> Before this patch, the dc->* are not set until the emulator check is
>>> done. With this path, it is possible in the normal case some of the
>>> fields are initialised in the error path.
>>>
>>> I think this is OK given the callbacks in the upper layer and in
>>> the writer don't rely on those fields to clean up. Just one thing to
>>> note.
>>>
>>> That said, I suggest you move all initialisation of dc->* in one place.
>>
>> OK, I will do it.
>>
>>>
>>>>  if (dss->type == LIBXL_DOMAIN_TYPE_HVM) {
>>>>  stream->device_model_version =
>>>>  libxl__device_model_version_running(gc, dss->domid);
>>>> @@ -249,12 +265,7

Re: [Xen-devel] [PATCH v10 16/31] secondary vm suspend/resume/checkpoint code

[Wen Congyang]

On 02/25/2016 11:56 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:20AM +0800, Wen Congyang wrote:
>> Secondary vm is running in colo mode. So we will do
>> the following things again and again:
>> 1. Resume secondary vm
>>a. Send CHECKPOINT_SVM_READY to master.
>>b. If it is not the first resume, call 
>> libxl__checkpoint_devices_preresume().
>>c. If it is the first resume(resume right after live migration),
>>   - call libxl__xc_domain_restore_done() to build the secondary vm.
>>   - enable secondary vm's logdirty.
>>   - call libxl__domain_resume() to resume secondary vm.
>>   - call libxl__checkpoint_devices_setup() to setup checkpoint devices.
>>d. Send CHECKPOINT_SVM_RESUMED to master.
>> 2. Wait a new checkpoint
>>a. Call libxl__checkpoint_devices_commit().
>>b. Read CHECKPOINT_NEW from master.
>> 3. Suspend secondary vm
>>a. Suspend secondary vm.
>>b. Call libxl__checkpoint_devices_postsuspend().
>>c. Send CHECKPOINT_SVM_SUSPENDED to master.
>>
>> Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
>> Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
>> ---
>>  tools/libxc/xc_sr_common.h   |2 +
>>  tools/libxc/xc_sr_save.c |3 +-
>>  tools/libxl/Makefile |1 +
>>  tools/libxl/libxl_colo.h |   24 +
>>  tools/libxl/libxl_colo_restore.c | 1038 
>> ++
>>  tools/libxl/libxl_create.c   |   37 ++
>>  tools/libxl/libxl_internal.h |   19 +
>>  tools/libxl/libxl_save_callout.c |7 +-
>>  tools/libxl/libxl_stream_read.c  |   12 +
>>  tools/libxl/libxl_types.idl  |1 +
> 
> There is a bunch of TODOs in libxl_colo.c but I don't think you're in a
> better position to judge whether they should be blocker or not.
> 
>>  10 files changed, 1142 insertions(+), 2 deletions(-)
>>  create mode 100644 tools/libxl/libxl_colo.h
>>  create mode 100644 tools/libxl/libxl_colo_restore.c
>>
>> diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
>> index 5d9f497..2bfed64 100644
>> --- a/tools/libxc/xc_sr_common.h
>> +++ b/tools/libxc/xc_sr_common.h
>> @@ -184,10 +184,12 @@ struct xc_sr_context
>>   * migration stream
>>   * 0: Plain VM
>>   * 1: Remus
>> + * 2: COLO
>>   */
>>  enum {
>>  MIG_STREAM_NONE, /* plain stream */
>>  MIG_STREAM_REMUS,
>> +MIG_STREAM_COLO,
>>  } migration_stream;
>>  
>>  union /* Common save or restore data. */
>> diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
>> index fe210cc..7393355 100644
>> --- a/tools/libxc/xc_sr_save.c
>> +++ b/tools/libxc/xc_sr_save.c
>> @@ -846,7 +846,8 @@ int xc_domain_save(xc_interface *xch, int io_fd, 
>> uint32_t dom,
>>  
>>  /* If altering migration_stream update this assert too. */
>>  assert(checkpointed_stream == MIG_STREAM_NONE ||
>> -   checkpointed_stream == MIG_STREAM_REMUS);
>> +   checkpointed_stream == MIG_STREAM_REMUS ||
>> +   checkpointed_stream == MIG_STREAM_COLO);
>>  
>>  /*
>>   * TODO: Find some time to better tweak the live migration algorithm.
> 
> [...]
> 
>> +
>> +#include "libxl_osdeps.h" /* must come before any other headers */
>> +
>> +#include "libxl_internal.h"
>> +#include "libxl_colo.h"
>> +#include "libxl_sr_stream_format.h"
>> +
>> +enum {
>> +LIBXL_COLO_SETUPED,
>> +LIBXL_COLO_SUSPENDED,
>> +LIBXL_COLO_RESUMED,
>> +};
>> +
>> +typedef struct libxl__colo_restore_checkpoint_state 
>> libxl__colo_restore_checkpoint_state;
>> +struct libxl__colo_restore_checkpoint_state {
>> +libxl__domain_suspend_state dsps;
>> +libxl__logdirty_switch lds;
>> +libxl__colo_restore_state *crs;
>> +libxl__stream_write_state sws;
>> +int status;
>> +bool preresume;
>> +/* used for teardown */
>> +int teardown_devices;
>> +int saved_rc;
>> +char *state_file;
>> +
>> +void (*callback)(libxl__egc *,
>> + libxl__colo_restore_checkpoint_state *,
>> + int);
>> +};
>> +
> 
> Shouldn't the enum and struct belong to libxl_colo.h ?

If we inlucde libxl_colo.h in libxl_internal.h, we cannot move this into 
colo.h, because
this structure needs libxl__domain_suspend_state, libxl__logdirty_switch, ...
We cannot ju

Re: [Xen-devel] [PATCH v10 21/31] libxc/save: support COLO save

[Wen Congyang]

On 02/25/2016 11:58 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:25AM +0800, Wen Congyang wrote:
> [...]
>>  /*
>>   * Suspend the domain and send dirty memory.
>>   * This is the last iteration of the live migration and the
>> @@ -558,6 +610,16 @@ static int suspend_and_send_dirty(struct xc_sr_context 
>> *ctx)
>>  
>>  bitmap_or(dirty_bitmap, ctx->save.deferred_pages, ctx->save.p2m_size);
>>  
>> +if ( !ctx->save.live && ctx->save.checkpointed == MIG_STREAM_COLO )
>> +{
>> +rc = merge_secondary_dirty_bitmap(ctx);
>> +if ( rc )
>> +{
>> +PERROR("Failed to get secondary vm's dirty pages");
>> +goto out;
>> +}
>> +}
>> +
>>  rc = send_dirty_pages(ctx, stats.dirty_count + 
>> ctx->save.nr_deferred_pages);
>>  if ( rc )
>>  goto out;
>> @@ -791,13 +853,42 @@ static int save(struct xc_sr_context *ctx, uint16_t 
>> guest_type)
>>  if ( rc )
>>  goto err;
>>  
>> -rc = ctx->save.callbacks->postcopy(ctx->save.callbacks->data);
>> -if ( rc <= 0 )
>> -goto err;
>> +if ( ctx->save.checkpointed == MIG_STREAM_COLO )
>> +{
>> +rc = 
>> ctx->save.callbacks->checkpoint(ctx->save.callbacks->data);
>> +if ( !rc )
>> +{
>> +rc = -1;
>> +goto err;
>> +}
>> +}
>>  
>> -        rc = ctx->save.callbacks->checkpoint(ctx->save.callbacks->data);
>> -if ( rc <= 0 )
>> +rc = ctx->save.callbacks->postcopy(ctx->save.callbacks->data);
>> +if ( !rc )
> 
> This original code for checking postcopy return value is if ( rc <= 0 ).

OK, I will check it.

Thanks
Wen Congyang

> 
> Wei.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v10 18/31] libxc/restore: support COLO restore

[Wen Congyang]

On 02/25/2016 11:57 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:22AM +0800, Wen Congyang wrote:
> [...]
>> - * With Remus, we buffer the records sent by the primary at checkpoint,
>> + * With Remus/COLO, we buffer the records sent by the primary at checkpoint,
>>   * in case the primary will fail, we can recover from the last
>>   * checkpoint state.
>>   * This should be enough for most of the cases because primary only send
>> diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
>> index aef9bca..2ae8154 100644
>> --- a/tools/libxc/xc_sr_restore.c
>> +++ b/tools/libxc/xc_sr_restore.c
>> @@ -460,6 +460,49 @@ static int handle_checkpoint(struct xc_sr_context *ctx)
>>  else
>>  ctx->restore.buffer_all_records = true;
>>  
>> +if ( ctx->restore.checkpointed == MIG_STREAM_COLO )
>> +{
>> +#define HANDLE_CALLBACK_RETURN_VALUE(ret)   \
>> +do {\
>> +if ( ret == 1 ) \
>> +rc = 0; /* Success */   \
>> +else\
>> +{   \
>> +if ( ret == 2 ) \
>> +rc = BROKEN_CHANNEL;\
>> +else\
>> +rc = -1; /* Some unspecified error */   \
>> +goto err;   \
>> +}   \
>> +} while (0)
>> +
>> +/* COLO */
>> +
>> +/* We need to resume guest */
>> +rc = ctx->restore.ops.stream_complete(ctx);
>> +if ( rc )
>> +goto err;
>> +
>> +/* TODO: call restore_results */
>> +
>> +/* Resume secondary vm */
>> +ret = 
>> ctx->restore.callbacks->postcopy(ctx->restore.callbacks->data);
>> +HANDLE_CALLBACK_RETURN_VALUE(ret);
>> +
>> +/* Wait for a new checkpoint */
>> +ret = ctx->restore.callbacks->wait_checkpoint(
>> +
>> ctx->restore.callbacks->data);
>> +HANDLE_CALLBACK_RETURN_VALUE(ret);
>> +
>> +/* suspend secondary vm */
>> +ret = ctx->restore.callbacks->suspend(ctx->restore.callbacks->data);
>> +HANDLE_CALLBACK_RETURN_VALUE(ret);
>> +
>> +#undef HANDLE_CALLBACK_RETURN_VALUE
>> +
>> +/* TODO: send dirty pfn list to primary */
> 
> You replace the TODOs with actual code in the next two patches.
> 
> You can rearrange them a bit so that you don't need to add TODOs at all.

Yes, will fix it in the next version.

Thanks
Wen Congyang

> 
> 
> Wei.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v10 17/31] primary vm suspend/resume/checkpoint code

[Wen Congyang]

On 02/25/2016 11:57 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:21AM +0800, Wen Congyang wrote:
> [...]
>> +#include "libxl_osdeps.h" /* must come before any other headers */
>> +
>> +#include "libxl_internal.h"
>> +#include "libxl_colo.h"
>> +
>> +static const libxl__checkpoint_device_instance_ops *colo_ops[] = {
>> +NULL,
>> +};
>> +
>> +/* = helper functions = */
>> +static int init_device_subkind(libxl__checkpoint_devices_state *cds)
>> +{
>> +/* init device subkind-specific state in the libxl ctx */
>> +int rc;
>> +STATE_AO_GC(cds->ao);
>> +
>> +rc = 0;
>> +return rc;
>> +}
>> +
>> +static void cleanup_device_subkind(libxl__checkpoint_devices_state *cds)
>> +{
>> +/* cleanup device subkind-specific state in the libxl ctx */
>> +STATE_AO_GC(cds->ao);
>> +}
>> +
>> +/* = colo: setup save environment = */
>> +static void colo_save_setup_done(libxl__egc *egc,
>> + libxl__checkpoint_devices_state *cds,
>> + int rc);
>> +static void colo_save_setup_failed(libxl__egc *egc,
>> +   libxl__checkpoint_devices_state *cds,
>> +   int rc);
>> +static void libxl__colo_save_domain_suspend_callback(void *data);
>> +static void libxl__colo_save_domain_checkpoint_callback(void *data);
>> +static void libxl__colo_save_domain_resume_callback(void *data);
>> +static void libxl__colo_save_domain_wait_checkpoint_callback(void *data);
>> +
> 
> 
> An ordered list of callbacks is appreciated. This doesn't seem to cover
> all callbacks and I'm not sure if this is the order they are supposed to
> fire either.

OK, I will check it and fix it in the next version.

Thanks
Wen Congyang

> 
> Wei.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v10 16/31] secondary vm suspend/resume/checkpoint code

[Wen Congyang]

On 02/25/2016 11:56 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:20AM +0800, Wen Congyang wrote:
>> Secondary vm is running in colo mode. So we will do
>> the following things again and again:
>> 1. Resume secondary vm
>>a. Send CHECKPOINT_SVM_READY to master.
>>b. If it is not the first resume, call 
>> libxl__checkpoint_devices_preresume().
>>c. If it is the first resume(resume right after live migration),
>>   - call libxl__xc_domain_restore_done() to build the secondary vm.
>>   - enable secondary vm's logdirty.
>>   - call libxl__domain_resume() to resume secondary vm.
>>   - call libxl__checkpoint_devices_setup() to setup checkpoint devices.
>>d. Send CHECKPOINT_SVM_RESUMED to master.
>> 2. Wait a new checkpoint
>>a. Call libxl__checkpoint_devices_commit().
>>b. Read CHECKPOINT_NEW from master.
>> 3. Suspend secondary vm
>>a. Suspend secondary vm.
>>b. Call libxl__checkpoint_devices_postsuspend().
>>c. Send CHECKPOINT_SVM_SUSPENDED to master.
>>
>> Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
>> Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
>> ---
>>  tools/libxc/xc_sr_common.h   |2 +
>>  tools/libxc/xc_sr_save.c |3 +-
>>  tools/libxl/Makefile |1 +
>>  tools/libxl/libxl_colo.h |   24 +
>>  tools/libxl/libxl_colo_restore.c | 1038 
>> ++
>>  tools/libxl/libxl_create.c   |   37 ++
>>  tools/libxl/libxl_internal.h |   19 +
>>  tools/libxl/libxl_save_callout.c |7 +-
>>  tools/libxl/libxl_stream_read.c  |   12 +
>>  tools/libxl/libxl_types.idl  |1 +
> 
> There is a bunch of TODOs in libxl_colo.c but I don't think you're in a
> better position to judge whether they should be blocker or not.
> 
>>  10 files changed, 1142 insertions(+), 2 deletions(-)
>>  create mode 100644 tools/libxl/libxl_colo.h
>>  create mode 100644 tools/libxl/libxl_colo_restore.c
>>
>> diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
>> index 5d9f497..2bfed64 100644
>> --- a/tools/libxc/xc_sr_common.h
>> +++ b/tools/libxc/xc_sr_common.h
>> @@ -184,10 +184,12 @@ struct xc_sr_context
>>   * migration stream
>>   * 0: Plain VM
>>   * 1: Remus
>> + * 2: COLO
>>   */
>>  enum {
>>  MIG_STREAM_NONE, /* plain stream */
>>  MIG_STREAM_REMUS,
>> +MIG_STREAM_COLO,
>>  } migration_stream;
>>  
>>  union /* Common save or restore data. */
>> diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
>> index fe210cc..7393355 100644
>> --- a/tools/libxc/xc_sr_save.c
>> +++ b/tools/libxc/xc_sr_save.c
>> @@ -846,7 +846,8 @@ int xc_domain_save(xc_interface *xch, int io_fd, 
>> uint32_t dom,
>>  
>>  /* If altering migration_stream update this assert too. */
>>  assert(checkpointed_stream == MIG_STREAM_NONE ||
>> -   checkpointed_stream == MIG_STREAM_REMUS);
>> +   checkpointed_stream == MIG_STREAM_REMUS ||
>> +   checkpointed_stream == MIG_STREAM_COLO);
>>  
>>  /*
>>   * TODO: Find some time to better tweak the live migration algorithm.
> 
> [...]
> 
>> +
>> +#include "libxl_osdeps.h" /* must come before any other headers */
>> +
>> +#include "libxl_internal.h"
>> +#include "libxl_colo.h"
>> +#include "libxl_sr_stream_format.h"
>> +
>> +enum {
>> +LIBXL_COLO_SETUPED,
>> +LIBXL_COLO_SUSPENDED,
>> +LIBXL_COLO_RESUMED,
>> +};
>> +
>> +typedef struct libxl__colo_restore_checkpoint_state 
>> libxl__colo_restore_checkpoint_state;
>> +struct libxl__colo_restore_checkpoint_state {
>> +libxl__domain_suspend_state dsps;
>> +libxl__logdirty_switch lds;
>> +libxl__colo_restore_state *crs;
>> +libxl__stream_write_state sws;
>> +int status;
>> +bool preresume;
>> +/* used for teardown */
>> +int teardown_devices;
>> +int saved_rc;
>> +char *state_file;
>> +
>> +void (*callback)(libxl__egc *,
>> + libxl__colo_restore_checkpoint_state *,
>> + int);
>> +};
>> +
> 
> Shouldn't the enum and struct belong to libxl_colo.h ?

It only be used by restore side. I think it is OK to move them to libxl_colo.h.

> 
>> +
>> +static void libxl__colo_restore_domain_resume_callback(void *data);
>

Re: [Xen-devel] [PATCH v10 12/31] tools/libxl: add back channel support to read stream

[Wen Congyang]

On 02/25/2016 11:54 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:16AM +0800, Wen Congyang wrote:
>> This is used by primay to read records sent by secondary.
>>
>> Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
>> Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
>> ---
>>  tools/libxl/libxl_create.c  |  1 +
>>  tools/libxl/libxl_internal.h|  1 +
>>  tools/libxl/libxl_stream_read.c | 27 +++
>>  3 files changed, 25 insertions(+), 4 deletions(-)
>>
>> diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
>> index 525bf85..fc746fb 100644
>> --- a/tools/libxl/libxl_create.c
>> +++ b/tools/libxl/libxl_create.c
>> @@ -1030,6 +1030,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
>>  dcs->srs.dcs = dcs;
>>  dcs->srs.fd = restore_fd;
>>  dcs->srs.legacy = (dcs->restore_params.stream_version == 1);
>> +dcs->srs.back_channel = false;
>>  dcs->srs.completion_callback = domcreate_stream_done;
>>  
>>  if (restore_fd >= 0) {
>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>> index cdf6751..32957ca 100644
>> --- a/tools/libxl/libxl_internal.h
>> +++ b/tools/libxl/libxl_internal.h
>> @@ -3420,6 +3420,7 @@ struct libxl__stream_read_state {
>>  libxl__domain_create_state *dcs;
>>  int fd;
>>  bool legacy;
>> +bool back_channel;
>>  void (*completion_callback)(libxl__egc *egc,
>>  libxl__stream_read_state *srs,
>>  int rc);
>> diff --git a/tools/libxl/libxl_stream_read.c 
>> b/tools/libxl/libxl_stream_read.c
>> index f4781eb..02a2c46 100644
>> --- a/tools/libxl/libxl_stream_read.c
>> +++ b/tools/libxl/libxl_stream_read.c
>> @@ -118,6 +118,15 @@
>>   *record, and therefore the buffered state is inconsistent. In
>>   *libxl__xc_domain_restore_done(), we just complete the stream and
>>   *stream->completion_callback() will be called to resume the guest
>> + *
>> + * For back channel stream:
>> + * - libxl__stream_read_start()
>> + *- Set up the stream to running state
>> + *
>> + * - libxl__stream_read_continue()
>> + * - Set up reading the next record from a started stream.
>> + *   Add some codes to process_record() to handle the record.
>> + *   Then call stream->checkpoint_callback() to return.
>>   */
>>  
>>  /* Success/error/cleanup handling. */
>> @@ -221,6 +230,17 @@ void libxl__stream_read_start(libxl__egc *egc,
>>  stream->running = true;
>>  stream->phase   = SRS_PHASE_NORMAL;
>>  
>> +dc->ao   = stream->ao;
>> +dc->copywhat = "restore v2 stream";
>> +dc->writefd  = -1;
>> +
>> +if (stream->back_channel) {
>> +assert(!stream->legacy);
>> +
>> +dc->readfd = stream->fd;
> 
> Why is this needed? dc->readfd is set to stream->fd no matter it is back
> channel or not. This can be moved outside this if {}.

Yes, will fix it in the next version.

> 
>> +return;
>> +}
>> +
>>  if (stream->legacy) {
>>  /* Convert the legacy stream. */
>>  libxl__conversion_helper_state *chs = >chs;
>> @@ -243,10 +263,7 @@ void libxl__stream_read_start(libxl__egc *egc,
>>  }
>>  /* stream->fd is now a v2 stream. */
>>  
>> -dc->ao   = stream->ao;
>> -dc->copywhat = "restore v2 stream";
>>  dc->readfd   = stream->fd;
>> -dc->writefd  = -1;
>>  
>>  /* Start reading the stream header. */
>>  rc = setup_read(stream, "stream header",
>> @@ -762,7 +779,9 @@ static void stream_done(libxl__egc *egc,
>>  LIBXL_STAILQ_FOREACH_SAFE(rec, >record_queue, entry, trec)
>>      free_record(rec);
>>  
>> -check_all_finished(egc, stream, rc);
>> +if (!stream->back_channel)
>> +/* back channel stream doesn't have restore helper */
>> +check_all_finished(egc, stream, rc);
> 
> Even if it doesn't have restore helper, check_all_finished also checks
> if the stream and the conversion helper are till in use.  The
> explanation in the comment doesn't seem to justify this change.

In stream_done(), stream->running is set to false, so 
libxl__stream_read_in_use()
returns false.

Back channel stream doesn't support legacy stream, so there is no conversion 
helper.

I will update the comments in the next version.

Thanks
Wen Congyang

> 
> Wei.
> 
>>  }
>>  
>>  void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
>> -- 
>> 2.5.0
>>
>>
>>
>>
>> ___
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v10 10/31] tools/libxl: add back channel support to write stream

[Wen Congyang]

On 02/25/2016 11:54 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:14AM +0800, Wen Congyang wrote:
>> Add back channel support to write stream. If the write stream is
>> a back channel stream, this means the write stream is used by
>> Secondary to send some records back.
>>
>> Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
>> Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
>> ---
>>  tools/libxl/libxl_dom_save.c |  1 +
>>  tools/libxl/libxl_internal.h |  1 +
>>  tools/libxl/libxl_stream_write.c | 26 --
>>  3 files changed, 22 insertions(+), 6 deletions(-)
>>
>> diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
>> index 72b61c7..18946c5 100644
>> --- a/tools/libxl/libxl_dom_save.c
>> +++ b/tools/libxl/libxl_dom_save.c
>> @@ -404,6 +404,7 @@ void libxl__domain_save(libxl__egc *egc, 
>> libxl__domain_save_state *dss)
>>  dss->sws.ao  = dss->ao;
>>  dss->sws.dss = dss;
>>  dss->sws.fd  = dss->fd;
>> +dss->sws.back_channel = false;
>>  dss->sws.completion_callback = stream_done;
>>  
>>  libxl__stream_write_start(egc, >sws);
>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>> index 3d3e8e8..e02e554 100644
>> --- a/tools/libxl/libxl_internal.h
>> +++ b/tools/libxl/libxl_internal.h
>> @@ -3044,6 +3044,7 @@ struct libxl__stream_write_state {
>>  libxl__ao *ao;
>>  libxl__domain_save_state *dss;
>>  int fd;
>> +bool back_channel;
>>  void (*completion_callback)(libxl__egc *egc,
>>  libxl__stream_write_state *sws,
>>  int rc);
>> diff --git a/tools/libxl/libxl_stream_write.c 
>> b/tools/libxl/libxl_stream_write.c
>> index f6ea55d..5379126 100644
>> --- a/tools/libxl/libxl_stream_write.c
>> +++ b/tools/libxl/libxl_stream_write.c
>> @@ -49,6 +49,13 @@
>>   *  - if (hvm)
>>   *  - Emulator context record
>>   *  - Checkpoint end record
>> + *
>> + * For back channel stream:
>> + * - libxl__stream_write_start()
>> + *- Set up the stream to running state
>> + *
>> + * - Add a new API to write the record. When the record is written
>> + *   out, call stream->checkpoint_callback() to return.
> 
> What does this mean? Which new API?

The next patch introduces this API. The commits is very old.

I think I can merge these two patches into one patch.

> 
>>   */
>>  
>>  /* Success/error/cleanup handling. */
>> @@ -225,6 +232,15 @@ void libxl__stream_write_start(libxl__egc *egc,
>>  
>>  stream->running = true;
>>  
>> +dc->ao= ao;
>> +dc->readfd= -1;
>> +dc->copywhat  = "save v2 stream";
>> +dc->writefd   = stream->fd;
>> +dc->maxsz = -1;
>> +
>> +if (stream->back_channel)
>> +return;
>> +
> 
> There seems to be very subtle change of behaviour.
> 
> Before this patch, the dc->* are not set until the emulator check is
> done. With this path, it is possible in the normal case some of the
> fields are initialised in the error path.
> 
> I think this is OK given the callbacks in the upper layer and in
> the writer don't rely on those fields to clean up. Just one thing to
> note.
> 
> That said, I suggest you move all initialisation of dc->* in one place.

OK, I will do it.

> 
>>  if (dss->type == LIBXL_DOMAIN_TYPE_HVM) {
>>  stream->device_model_version =
>>  libxl__device_model_version_running(gc, dss->domid);
>> @@ -249,12 +265,7 @@ void libxl__stream_write_start(libxl__egc *egc,
>>  stream->emu_sub_hdr.index = 0;
>>  }
>>  
>> -dc->ao= ao;
>> -dc->readfd= -1;
>>  dc->writewhat = "stream header";
>> -dc->copywhat  = "save v2 stream";
>> -dc->writefd   = stream->fd;
>> -dc->maxsz = -1;
>>  dc->callback  = stream_header_done;
>>  
>>  rc = libxl__datacopier_start(dc);
>> @@ -279,6 +290,7 @@ void libxl__stream_write_start_checkpoint(libxl__egc 
>> *egc,
>>  {
>>  assert(stream->running);
>>  assert(!stream->in_checkpoint);
>> +assert(!stream->back_channel);
>>  stream->in_checkpoint = true;
>>  
>>  write_emulator_xenstore_record(egc, stream);
>> @@ -590,7 +602,9 @@ static void stream_done(libxl__egc *egc,

Re: [Xen-devel] [PATCH v10 07/31] docs/libxl: Introduce CHECKPOINT_CONTEXT to support migration v2 colo streams

[Wen Congyang]

On 02/25/2016 11:54 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:11AM +0800, Wen Congyang wrote:
>> It is the negotiation record for COLO.
>> Primary->Secondary:
>> control_id  0x: Secondary VM is out of sync, start a new 
>> checkpoint
>> Secondary->Primary:
>> 0x0001: Secondary VM is suspended
>> 0x0002: Secondary VM is ready
>> 0x0003: Secondary VM is resumed
>>
>> Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
>> Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
>> ---
>>  docs/specs/libxl-migration-stream.pandoc | 25 +++--
>>  tools/libxl/libxl_sr_stream_format.h | 11 +++
>>  tools/python/xen/migration/libxl.py  |  9 +
>>  3 files changed, 43 insertions(+), 2 deletions(-)
>>
>> diff --git a/docs/specs/libxl-migration-stream.pandoc 
>> b/docs/specs/libxl-migration-stream.pandoc
>> index 2c97d86..5166d66 100644
>> --- a/docs/specs/libxl-migration-stream.pandoc
>> +++ b/docs/specs/libxl-migration-stream.pandoc
>> @@ -1,6 +1,6 @@
>>  % LibXenLight Domain Image Format
>>  % Andrew Cooper <<andrew.coop...@citrix.com>>
> 
> You can add yours and Hongyang's name and email here as well.

OK

> 
>> -% Revision 1
>> +% Revision 2
>>  
>>  Introduction
>>  
>> @@ -119,7 +119,9 @@ type 0x: END
>>  
>>   0x0004: CHECKPOINT_END
>>  
>> - 0x0005 - 0x7FFF: Reserved for future _mandatory_
>> + 0x0005: CHECKPOINT_STATE
>> +
>> + 0x0006 - 0x7FFF: Reserved for future _mandatory_
>>   records.
>>  
>>   0x8000 - 0x: Reserved for future _optional_
>> @@ -249,6 +251,25 @@ A checkpoint end record marks the end of a checkpoint 
>> in the image.
>>  The end record contains no fields; its body_length is 0.
>>  
>>  
>> +CHECKPOINT\_STATE
>> +--
>> +
>> +A checkpoint state record contains the control information for checkpoint.
>> +
>> + 0 1 2 3 4 5 6 7 octet
>> ++++
>> +| control_id | padding|
>> ++++
>> +
>> +
>> +FieldDescription
>> + ---
>> +control_id   0x: Secondary VM is out of sync, start a new 
>> checkpoint
>> + 0x0001: Secondary VM is suspended
>> + 0x0002: Secondary VM is ready
>> + 0x0003: Secondary VM is resumed
> 
> You commit message seems to suggest 0 can only go from primary to
> secondary, while the other three only go from secondary to primary. You
> probably need to spell out the direction here (and later enforce this
> behaviour in code).

OK, will fix it in the next version.

Thanks
Wen Congyang

> 
> Wei.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v10 01/31] tools/libxl: introduce libxl__domain_restore_device_model to load qemu state

[Wen Congyang]

On 02/25/2016 11:53 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:05AM +0800, Wen Congyang wrote:
>> In normal migration, the qemu state is passed to qemu as a parameter.
>> With COLO, secondary vm is running. So we will do the following steps
>> at every checkpoint:
>> 1. suspend both primary vm and secondary vm
>> 2. sync the state
>> 3. resume both primary vm and secondary vm
>> Primary will send qemu's state in step2, and secondary's qemu should
>> read it and restore the state before it is resumed. We can not pass
>> the state to qemu as a parameter because secondary QEMU already started
> 
> is already started.
> 
>> at this point, so we introduce libxl__domain_restore_device_model() to
>> do it. This API MUST be called before resuming secondary vm.
>>
> 
> The last sentence is of no relevancy of this function itself, so I would
> just remove it if this patch will be resent in its current form.
> 
> And some comments below.
> 
>> Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
>> Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
>> Cc: Anthony Perard <anthony.per...@citrix.com>
>> Reviewed-by: Konrad Rzeszutek Wilk <konrad.w...@oracle.com>
>> ---
>>  tools/libxl/libxl_dom_save.c | 20 
>>  tools/libxl/libxl_internal.h |  4 
>>  tools/libxl/libxl_qmp.c  | 10 ++
>>  3 files changed, 34 insertions(+)
>>
>> diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
>> index 4eb7960..7d345f9 100644
>> --- a/tools/libxl/libxl_dom_save.c
>> +++ b/tools/libxl/libxl_dom_save.c
>> @@ -512,6 +512,26 @@ int 
>> libxl__restore_emulator_xenstore_data(libxl__domain_create_state *dcs,
>>  return rc;
>>  }
>>  
>> +int libxl__domain_restore_device_model(libxl__gc *gc, uint32_t domid,
>> +   const char *restore_file)
>> +{
>> +int rc;
>> +
>> +switch (libxl__device_model_version_running(gc, domid)) {
>> +case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN_TRADITIONAL:
>> +/* Will never be supported. */
>> +rc = ERROR_INVAL;
>> +break;
>> +case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN:
>> +rc = libxl__qmp_restore(gc, domid, restore_file);
>> +break;
>> +default:
>> +rc = ERROR_INVAL;
>> +}
>> +
>> +return rc;
>> +}
>> +
> 
> The function name suggests that it should be universally applied to all
> device model cases and all places that involves restoring device model.
> This is not the case in this series.  I search for this function in the
> rest of this series, it seems to be only used by COLO (in patch #10).
> 
> It's also unclear where the other half libxl__domain_save_device_model
> is.  I don't think this is your problem, though.
> 
> Based on the two reasons above, I would say let's not have this function
> at all.  You can call libxl__qmp_restore in COLO code directly.
> 
> Does this sound plausible?
> 
> If you agree, this patch can be turned into introduction of
> libxl__qmp_restore.

Yes, it is only used by COLO. I will update it in the next version.

Thanks
Wen Congyang

> 
>>  /*
>>   * Local variables:
>>   * mode: C
>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>> index a1aae97..5320e5e 100644
>> --- a/tools/libxl/libxl_internal.h
>> +++ b/tools/libxl/libxl_internal.h
>> @@ -1119,6 +1119,8 @@ _hidden int libxl__domain_rename(libxl__gc *gc, 
>> uint32_t domid,
>>   const char *old_name, const char *new_name,
>>   xs_transaction_t trans);
>>  
>> +_hidden int libxl__domain_restore_device_model(libxl__gc *gc, uint32_t 
>> domid,
>> +   const char *restore_file);
>>  _hidden int libxl__domain_resume_device_model(libxl__gc *gc, uint32_t 
>> domid);
>>  
>>  _hidden const char *libxl__userdata_path(libxl__gc *gc, uint32_t domid,
>> @@ -1762,6 +1764,8 @@ _hidden int libxl__qmp_stop(libxl__gc *gc, int domid);
>>  _hidden int libxl__qmp_resume(libxl__gc *gc, int domid);
>>  /* Save current QEMU state into fd. */
>>  _hidden int libxl__qmp_save(libxl__gc *gc, int domid, const char *filename);
>> +/* Load current QEMU state from file. */
>> +_hidden int libxl__qmp_restore(libxl__gc *gc, int domid, const char 
>> *filename);
>>  /* Set dirty bitmap logging status */
>>  _hidden int libxl__qmp_set_global_dirty_log(libxl

[Xen-devel] [PATCH v10 03/31] tools/libxl: Add back channel to allow migration target send data back

[Wen Congyang]

In COLO mode, secondary needs to send the following data to primary:
1. In libxl
   Secondary sends the following CHECKPOINT_CONTEXT to primary:
   CHECKPOINT_SVM_SUSPENDED, CHECKPOINT_SVM_READY and CHECKPOINT_SVM_RESUMED
2. In libxc
   Secondary sends the dirty pfn list to primary

But the io_fd only can be written in primary, and only can be read in
secondary. Save recv_fd in domain_suspend_state, and send_fd in
domain_create_state. Extend libxl_domain_create_restore API, add a
send_fd param to it. Add LIBXL_HAVE_CREATE_RESTORE_SEND_FD to indicate
the API change.

Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Acked-by: Wei Liu <wei.l...@citrix.com>
---
 tools/libxl/libxl.c  |  3 +--
 tools/libxl/libxl.h  | 29 -
 tools/libxl/libxl_create.c   | 10 ++
 tools/libxl/libxl_internal.h |  2 ++
 tools/libxl/xl_cmdimpl.c |  8 +++-
 tools/ocaml/libs/xl/xenlight_stubs.c |  2 +-
 6 files changed, 45 insertions(+), 9 deletions(-)

diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 4cdc169..e9ab78c 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -834,7 +834,6 @@ out:
 static void remus_failover_cb(libxl__egc *egc,
   libxl__domain_save_state *dss, int rc);
 
-/* TODO: Explicit Checkpoint acknowledgements via recv_fd. */
 int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
  uint32_t domid, int send_fd, int recv_fd,
  const libxl_asyncop_how *ao_how)
@@ -871,7 +870,7 @@ int libxl_domain_remus_start(libxl_ctx *ctx, 
libxl_domain_remus_info *info,
 dss->callback = remus_failover_cb;
 dss->domid = domid;
 dss->fd = send_fd;
-/* TODO do something with recv_fd */
+dss->recv_fd = recv_fd;
 dss->type = type;
 dss->live = 1;
 dss->debug = 0;
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index f9e3ef5..5a286ec 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -639,6 +639,15 @@ typedef struct libxl__ctx libxl_ctx;
 #define LIBXL_HAVE_DOMAIN_CREATE_RESTORE_PARAMS 1
 
 /*
+ * LIBXL_HAVE_DOMAIN_CREATE_RESTORE_SEND_BACK_FD 1
+ *
+ * If this is defined, libxl_domain_create_restore()'s API has changed to
+ * include a send_back_fd param which used for libxl migration back channel
+ * during COLO.
+ */
+#define LIBXL_HAVE_DOMAIN_CREATE_RESTORE_SEND_BACK_FD 1
+
+/*
  * LIBXL_HAVE_CREATEINFO_PVH
  * If this is defined, then libxl supports creation of a PVH guest.
  */
@@ -1165,6 +1174,7 @@ int libxl_domain_create_new(libxl_ctx *ctx, 
libxl_domain_config *d_config,
 LIBXL_EXTERNAL_CALLERS_ONLY;
 int libxl_domain_create_restore(libxl_ctx *ctx, libxl_domain_config *d_config,
 uint32_t *domid, int restore_fd,
+int send_back_fd,
 const libxl_domain_restore_params *params,
 const libxl_asyncop_how *ao_how,
 const libxl_asyncprogress_how *aop_console_how)
@@ -1185,7 +1195,7 @@ int static inline libxl_domain_create_restore_0x040200(
 libxl_domain_restore_params_init();
 
 ret = libxl_domain_create_restore(
-ctx, d_config, domid, restore_fd, , ao_how, aop_console_how);
+ctx, d_config, domid, restore_fd, -1, , ao_how, 
aop_console_how);
 
 libxl_domain_restore_params_dispose();
 return ret;
@@ -1193,6 +1203,23 @@ int static inline libxl_domain_create_restore_0x040200(
 
 #define libxl_domain_create_restore libxl_domain_create_restore_0x040200
 
+#elif defined(LIBXL_API_VERSION) && LIBXL_API_VERSION >= 0x040400 \
+ && LIBXL_API_VERSION < 0x040700
+
+int static inline libxl_domain_create_restore_0x040400(
+libxl_ctx *ctx, libxl_domain_config *d_config,
+uint32_t *domid, int restore_fd,
+const libxl_domain_restore_params *params,
+const libxl_asyncop_how *ao_how,
+const libxl_asyncprogress_how *aop_console_how)
+LIBXL_EXTERNAL_CALLERS_ONLY
+{
+return libxl_domain_create_restore(ctx, d_config, domid, restore_fd,
+   -1, params, ao_how, aop_console_how);
+}
+
+#define libxl_domain_create_restore libxl_domain_create_restore_0x040400
+
 #endif
 
 int libxl_domain_soft_reset(libxl_ctx *ctx,
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index f1028bc..525bf85 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1572,7 +1572,7 @@ static void domain_create_cb(libxl__egc *egc,
  int rc, uint32_t domid);
 
 static int do_domain_create(libxl_ctx *ctx, libxl_domain_config *d_config,
-uint32_t *domid, int restore_fd,
+

[Xen-devel] [PATCH v10 02/31] tools/libxl: introduce libxl__domain_common_switch_qemu_logdirty()

[Wen Congyang]

Secondary vm is running in COLO mode, we need to send secondary
vm's dirty page information to primary host at checkpoint, so we
have to enable qemu logdirty on secondary.

libxl__domain_suspend_common_switch_qemu_logdirty() is to enable
qemu logdirty. But it uses libxl__domain_save_state, and calls
libxl__xc_domain_saverestore_async_callback_done() before exits.
This can not be used for secondary vm.

Update libxl__domain_suspend_common_switch_qemu_logdirty() to
introduce a new API libxl__domain_common_switch_qemu_logdirty().
This API only uses libxl__logdirty_switch, and calls
lds->callback before exits. This new API will be used by the patch:
  secondary vm suspend/resume/checkpoint code

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
CC: Andrew Cooper <andrew.coop...@citrix.com>
Acked-by: Ian Campbell <ian.campb...@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.w...@oracle.com>
Acked-by: Wei Liu <wei.l...@citrix.com>
---
 tools/libxl/libxl_dom_save.c | 95 
 tools/libxl/libxl_internal.h |  8 
 2 files changed, 60 insertions(+), 43 deletions(-)

diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
index 7d345f9..72b61c7 100644
--- a/tools/libxl/libxl_dom_save.c
+++ b/tools/libxl/libxl_dom_save.c
@@ -42,7 +42,7 @@ static void switch_logdirty_timeout(libxl__egc *egc, 
libxl__ev_time *ev,
 static void switch_logdirty_xswatch(libxl__egc *egc, libxl__ev_xswatch*,
 const char *watch_path, const char *event_path);
 static void switch_logdirty_done(libxl__egc *egc,
- libxl__domain_save_state *dss, int rc);
+ libxl__logdirty_switch *lds, int rc);
 
 void libxl__logdirty_init(libxl__logdirty_switch *lds)
 {
@@ -52,13 +52,10 @@ void libxl__logdirty_init(libxl__logdirty_switch *lds)
 }
 
 static void domain_suspend_switch_qemu_xen_traditional_logdirty
-   (int domid, unsigned enable,
-libxl__save_helper_state *shs)
+   (libxl__egc *egc, int domid, unsigned enable,
+libxl__logdirty_switch *lds)
 {
-libxl__egc *egc = shs->egc;
-libxl__domain_save_state *dss = shs->caller_state;
-libxl__logdirty_switch *lds = >logdirty;
-STATE_AO_GC(dss->ao);
+STATE_AO_GC(lds->ao);
 int rc;
 xs_transaction_t t = 0;
 const char *got;
@@ -120,26 +117,34 @@ static void 
domain_suspend_switch_qemu_xen_traditional_logdirty
  out:
 LOG(ERROR,"logdirty switch failed (rc=%d), abandoning suspend",rc);
 libxl__xs_transaction_abort(gc, );
-switch_logdirty_done(egc,dss,rc);
+switch_logdirty_done(egc,lds,rc);
 }
 
 static void domain_suspend_switch_qemu_xen_logdirty
-   (int domid, unsigned enable,
-libxl__save_helper_state *shs)
+   (libxl__egc *egc, int domid, unsigned enable,
+libxl__logdirty_switch *lds)
 {
-libxl__egc *egc = shs->egc;
-libxl__domain_save_state *dss = shs->caller_state;
-STATE_AO_GC(dss->ao);
+STATE_AO_GC(lds->ao);
 int rc;
 
 rc = libxl__qmp_set_global_dirty_log(gc, domid, enable);
-if (!rc) {
-libxl__xc_domain_saverestore_async_callback_done(egc, shs, 0);
-} else {
+if (rc)
 LOG(ERROR,"logdirty switch failed (rc=%d), abandoning suspend",rc);
+
+lds->callback(egc, lds, rc);
+}
+
+static void domain_suspend_switch_qemu_logdirty_done
+(libxl__egc *egc, libxl__logdirty_switch *lds, int rc)
+{
+libxl__domain_save_state *dss = CONTAINER_OF(lds, *dss, logdirty);
+
+if (rc) {
 dss->rc = rc;
-libxl__xc_domain_saverestore_async_callback_done(egc, shs, -1);
-}
+libxl__xc_domain_saverestore_async_callback_done(egc,
+ >sws.shs, -1);
+} else
+libxl__xc_domain_saverestore_async_callback_done(egc, >sws.shs, 
0);
 }
 
 void libxl__domain_suspend_common_switch_qemu_logdirty
@@ -148,42 +153,52 @@ void libxl__domain_suspend_common_switch_qemu_logdirty
 libxl__save_helper_state *shs = user;
 libxl__egc *egc = shs->egc;
 libxl__domain_save_state *dss = shs->caller_state;
-STATE_AO_GC(dss->ao);
+
+/* Convenience aliases. */
+libxl__logdirty_switch *const lds = >logdirty;
+
+lds->callback = domain_suspend_switch_qemu_logdirty_done;
+libxl__domain_common_switch_qemu_logdirty(egc, domid, enable, lds);
+}
+
+void libxl__domain_common_switch_qemu_logdirty(libxl__egc *egc,
+   int domid, unsigned enable,
+   libxl__logdirty_swit

[Xen-devel] [PATCH v10 15/31] tools/libx{l, c}: add postcopy/suspend callback to restore side

[Wen Congyang]

Secondary(restore side) is running under COLO, we also need
postcopy/suspend callbacks.

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
---
 tools/libxc/include/xenguest.h | 10 ++
 tools/libxl/libxl_save_msgs_gen.pl |  4 ++--
 2 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index f598bec..cb0bbcf 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -101,6 +101,16 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t 
dom, uint32_t max_iter
 
 /* callbacks provided by xc_domain_restore */
 struct restore_callbacks {
+/* Called after a new checkpoint to suspend the guest.
+ */
+int (*suspend)(void* data);
+
+/* Called after the secondary vm is ready to resume.
+ * Callback function resumes the guest & the device model,
+ * returns to xc_domain_restore.
+ */
+int (*postcopy)(void* data);
+
 /* A checkpoint record has been found in the stream.
  * returns: */
 #define XGR_CHECKPOINT_ERROR0 /* Terminate processing */
diff --git a/tools/libxl/libxl_save_msgs_gen.pl 
b/tools/libxl/libxl_save_msgs_gen.pl
index 51d004d..6016706 100755
--- a/tools/libxl/libxl_save_msgs_gen.pl
+++ b/tools/libxl/libxl_save_msgs_gen.pl
@@ -23,8 +23,8 @@ our @msgs = (
  STRING doing_what),
 'unsigned long', 'done',
 'unsigned long', 'total'] ],
-[  3, 'scxA',   "suspend", [] ],
-[  4, 'scxA',   "postcopy", [] ],
+[  3, 'srcxA',  "suspend", [] ],
+[  4, 'srcxA',  "postcopy", [] ],
 [  5, 'srcxA',  "checkpoint", [] ],
 [  6, 'srcxA',  "wait_checkpoint", [] ],
 [  7, 'scxA',   "switch_qemu_logdirty",  [qw(int domid
-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v10 07/31] docs/libxl: Introduce CHECKPOINT_CONTEXT to support migration v2 colo streams

[Wen Congyang]

It is the negotiation record for COLO.
Primary->Secondary:
control_id  0x: Secondary VM is out of sync, start a new checkpoint
Secondary->Primary:
0x0001: Secondary VM is suspended
0x0002: Secondary VM is ready
0x0003: Secondary VM is resumed

Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
---
 docs/specs/libxl-migration-stream.pandoc | 25 +++--
 tools/libxl/libxl_sr_stream_format.h | 11 +++
 tools/python/xen/migration/libxl.py  |  9 +
 3 files changed, 43 insertions(+), 2 deletions(-)

diff --git a/docs/specs/libxl-migration-stream.pandoc 
b/docs/specs/libxl-migration-stream.pandoc
index 2c97d86..5166d66 100644
--- a/docs/specs/libxl-migration-stream.pandoc
+++ b/docs/specs/libxl-migration-stream.pandoc
@@ -1,6 +1,6 @@
 % LibXenLight Domain Image Format
 % Andrew Cooper <<andrew.coop...@citrix.com>>
-% Revision 1
+% Revision 2
 
 Introduction
 
@@ -119,7 +119,9 @@ type 0x: END
 
  0x0004: CHECKPOINT_END
 
- 0x0005 - 0x7FFF: Reserved for future _mandatory_
+ 0x0005: CHECKPOINT_STATE
+
+ 0x0006 - 0x7FFF: Reserved for future _mandatory_
  records.
 
  0x8000 - 0x: Reserved for future _optional_
@@ -249,6 +251,25 @@ A checkpoint end record marks the end of a checkpoint in 
the image.
 The end record contains no fields; its body_length is 0.
 
 
+CHECKPOINT\_STATE
+--
+
+A checkpoint state record contains the control information for checkpoint.
+
+ 0 1 2 3 4 5 6 7 octet
++++
+| control_id | padding|
++++
+
+
+FieldDescription
+ ---
+control_id   0x: Secondary VM is out of sync, start a new 
checkpoint
+ 0x0001: Secondary VM is suspended
+ 0x0002: Secondary VM is ready
+ 0x0003: Secondary VM is resumed
+
+
 Future Extensions
 =
 
diff --git a/tools/libxl/libxl_sr_stream_format.h 
b/tools/libxl/libxl_sr_stream_format.h
index 54da360..75f5190 100644
--- a/tools/libxl/libxl_sr_stream_format.h
+++ b/tools/libxl/libxl_sr_stream_format.h
@@ -36,6 +36,7 @@ typedef struct libxl__sr_rec_hdr
 #define REC_TYPE_EMULATOR_XENSTORE_DATA 0x0002U
 #define REC_TYPE_EMULATOR_CONTEXT   0x0003U
 #define REC_TYPE_CHECKPOINT_END 0x0004U
+#define REC_TYPE_CHECKPOINT_STATE   0x0005U
 
 typedef struct libxl__sr_emulator_hdr
 {
@@ -47,6 +48,16 @@ typedef struct libxl__sr_emulator_hdr
 #define EMULATOR_QEMU_TRADITIONAL0x0001U
 #define EMULATOR_QEMU_UPSTREAM   0x0002U
 
+typedef struct libxl_sr_checkpoint_state
+{
+uint32_t id;
+} libxl_sr_checkpoint_state;
+
+#define CHECKPOINT_NEW   0xU
+#define CHECKPOINT_SVM_SUSPENDED 0x0001U
+#define CHECKPOINT_SVM_READY 0x0002U
+#define CHECKPOINT_SVM_RESUMED   0x0003U
+
 #endif /* LIBXL__SR_STREAM_FORMAT_H */
 
 /*
diff --git a/tools/python/xen/migration/libxl.py 
b/tools/python/xen/migration/libxl.py
index fc0acf6..d5f54dc 100644
--- a/tools/python/xen/migration/libxl.py
+++ b/tools/python/xen/migration/libxl.py
@@ -37,6 +37,7 @@ REC_TYPE_libxc_context  = 0x0001
 REC_TYPE_emulator_xenstore_data = 0x0002
 REC_TYPE_emulator_context   = 0x0003
 REC_TYPE_checkpoint_end = 0x0004
+REC_TYPE_checkpoint_state   = 0x0005
 
 rec_type_to_str = {
 REC_TYPE_end: "End",
@@ -44,6 +45,7 @@ rec_type_to_str = {
 REC_TYPE_emulator_xenstore_data : "Emulator xenstore data",
 REC_TYPE_emulator_context   : "Emulator context",
 REC_TYPE_checkpoint_end : "Checkpoint end",
+REC_TYPE_checkpoint_state   : "Checkpoint state"
 }
 
 # emulator_* header
@@ -212,6 +214,11 @@ class VerifyLibxl(VerifyBase):
 if len(content) != 0:
 raise RecordError("Checkpoint end record with non-zero length")
 
+def verify_record_checkpoint_state(self, content):
+""" Checkpoint state """
+if len(content) == 0:
+raise RecordError("Checkpoint state record with zero length")
+
 
 record_verifiers = {
 REC_TYPE_end:
@@ -224,4 +231,6 @@ record_verifiers = {
 VerifyLibxl.verify_record_emulator_context,
 REC_TYPE_checkpoint_end:
 VerifyLib

[Xen-devel] [PATCH v10 06/31] docs: add colo readme

[Wen Congyang]

add colo readme, refer to
http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Acked-by: Ian Campbell <ian.campb...@citrix.com>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
---
 docs/README.colo | 9 +
 1 file changed, 9 insertions(+)
 create mode 100644 docs/README.colo

diff --git a/docs/README.colo b/docs/README.colo
new file mode 100644
index 000..466eb72
--- /dev/null
+++ b/docs/README.colo
@@ -0,0 +1,9 @@
+COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service)
+project is a high availability solution. Both primary VM (PVM) and secondary VM
+(SVM) run in parallel. They receive the same request from client, and generate
+response in parallel too. If the response packets from PVM and SVM are
+identical, they are released immediately. Otherwise, a VM checkpoint (on 
demand)
+is conducted.
+
+See the website at http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
+for details.
-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v10 11/31] tools/libxl: write checkpoint_state records into the stream

[Wen Congyang]

write checkpoint_state records into the stream, used by both
primary and secondary to send colo context.

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
---
 tools/libxl/libxl_internal.h |  5 
 tools/libxl/libxl_stream_write.c | 60 +++-
 2 files changed, 64 insertions(+), 1 deletion(-)

diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index e02e554..cdf6751 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3056,6 +3056,7 @@ struct libxl__stream_write_state {
 bool running;
 bool in_checkpoint;
 bool sync_teardown;  /* Only used to coordinate shutdown on error path. */
+bool in_checkpoint_state;
 libxl__save_helper_state shs;
 
 /* Main stream-writing data. */
@@ -3079,6 +3080,10 @@ _hidden void libxl__stream_write_start(libxl__egc *egc,
 _hidden void
 libxl__stream_write_start_checkpoint(libxl__egc *egc,
  libxl__stream_write_state *stream);
+_hidden void
+libxl__stream_write_checkpoint_state(libxl__egc *egc,
+ libxl__stream_write_state *stream,
+ libxl_sr_checkpoint_state *srcs);
 _hidden void libxl__stream_write_abort(libxl__egc *egc,
libxl__stream_write_state *stream,
int rc);
diff --git a/tools/libxl/libxl_stream_write.c b/tools/libxl/libxl_stream_write.c
index 5379126..8e0e57e 100644
--- a/tools/libxl/libxl_stream_write.c
+++ b/tools/libxl/libxl_stream_write.c
@@ -98,6 +98,12 @@ static void write_checkpoint_end_record(libxl__egc *egc,
 static void checkpoint_end_record_done(libxl__egc *egc,
libxl__stream_write_state *stream);
 
+/* checkpoint state */
+static void write_checkpoint_state_done(libxl__egc *egc,
+libxl__stream_write_state *stream);
+static void checkpoint_state_done(libxl__egc *egc,
+  libxl__stream_write_state *stream, int rc);
+
 /*- Helpers -*/
 
 static void write_done(libxl__egc *egc,
@@ -589,6 +595,21 @@ static void stream_complete(libxl__egc *egc,
 return;
 }
 
+if (stream->in_checkpoint_state) {
+assert(rc);
+
+/*
+ * If an error is encountered while in a checkpoint, pass it
+ * back to libxc.  The failure will come back around to us via
+ * 1. normal stream
+ *libxl__xc_domain_save_done()
+ * 2. back_channel stream
+ *libxl__stream_write_abort()
+ */
+checkpoint_state_done(egc, stream, rc);
+return;
+}
+
 stream_done(egc, stream, rc);
 }
 
@@ -596,6 +617,7 @@ static void stream_done(libxl__egc *egc,
 libxl__stream_write_state *stream, int rc)
 {
 assert(stream->running);
+assert(!stream->in_checkpoint_state);
 stream->running = false;
 
 if (stream->emu_carefd)
@@ -656,7 +678,43 @@ static void check_all_finished(libxl__egc *egc,
 libxl__save_helper_inuse(>shs))
 return;
 
-stream->completion_callback(egc, stream, stream->rc);
+if (stream->completion_callback)
+/* back channel stream doesn't have completion_callback() */
+stream->completion_callback(egc, stream, stream->rc);
+}
+
+/*- checkpoint state -*/
+void libxl__stream_write_checkpoint_state(libxl__egc *egc,
+  libxl__stream_write_state *stream,
+  libxl_sr_checkpoint_state *srcs)
+{
+struct libxl__sr_rec_hdr rec;
+
+assert(stream->running);
+assert(!stream->in_checkpoint);
+assert(!stream->in_checkpoint_state);
+stream->in_checkpoint_state = true;
+
+FILLZERO(rec);
+rec.type = REC_TYPE_CHECKPOINT_STATE;
+rec.length = sizeof(*srcs);
+
+setup_write(egc, stream, "checkpoint state", ,
+srcs, write_checkpoint_state_done);
+}
+
+static void write_checkpoint_state_done(libxl__egc *egc,
+libxl__stream_write_state *stream)
+{
+checkpoint_state_done(egc, stream, 0);
+}
+
+static void checkpoint_state_done(libxl__egc *egc,
+  libxl__stream_write_state *stream, int rc)
+{
+assert(stream->in_checkpoint_state);
+stream->in_checkpoint_state = false;
+stream->checkpoint_callback(egc, stream, rc);
 }
 
 /*
-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v10 01/31] tools/libxl: introduce libxl__domain_restore_device_model to load qemu state

[Wen Congyang]

In normal migration, the qemu state is passed to qemu as a parameter.
With COLO, secondary vm is running. So we will do the following steps
at every checkpoint:
1. suspend both primary vm and secondary vm
2. sync the state
3. resume both primary vm and secondary vm
Primary will send qemu's state in step2, and secondary's qemu should
read it and restore the state before it is resumed. We can not pass
the state to qemu as a parameter because secondary QEMU already started
at this point, so we introduce libxl__domain_restore_device_model() to
do it. This API MUST be called before resuming secondary vm.

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
Cc: Anthony Perard <anthony.per...@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.w...@oracle.com>
---
 tools/libxl/libxl_dom_save.c | 20 
 tools/libxl/libxl_internal.h |  4 
 tools/libxl/libxl_qmp.c  | 10 ++
 3 files changed, 34 insertions(+)

diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
index 4eb7960..7d345f9 100644
--- a/tools/libxl/libxl_dom_save.c
+++ b/tools/libxl/libxl_dom_save.c
@@ -512,6 +512,26 @@ int 
libxl__restore_emulator_xenstore_data(libxl__domain_create_state *dcs,
 return rc;
 }
 
+int libxl__domain_restore_device_model(libxl__gc *gc, uint32_t domid,
+   const char *restore_file)
+{
+int rc;
+
+switch (libxl__device_model_version_running(gc, domid)) {
+case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN_TRADITIONAL:
+/* Will never be supported. */
+rc = ERROR_INVAL;
+break;
+case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN:
+rc = libxl__qmp_restore(gc, domid, restore_file);
+break;
+default:
+rc = ERROR_INVAL;
+}
+
+return rc;
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index a1aae97..5320e5e 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -1119,6 +1119,8 @@ _hidden int libxl__domain_rename(libxl__gc *gc, uint32_t 
domid,
  const char *old_name, const char *new_name,
  xs_transaction_t trans);
 
+_hidden int libxl__domain_restore_device_model(libxl__gc *gc, uint32_t domid,
+   const char *restore_file);
 _hidden int libxl__domain_resume_device_model(libxl__gc *gc, uint32_t domid);
 
 _hidden const char *libxl__userdata_path(libxl__gc *gc, uint32_t domid,
@@ -1762,6 +1764,8 @@ _hidden int libxl__qmp_stop(libxl__gc *gc, int domid);
 _hidden int libxl__qmp_resume(libxl__gc *gc, int domid);
 /* Save current QEMU state into fd. */
 _hidden int libxl__qmp_save(libxl__gc *gc, int domid, const char *filename);
+/* Load current QEMU state from file. */
+_hidden int libxl__qmp_restore(libxl__gc *gc, int domid, const char *filename);
 /* Set dirty bitmap logging status */
 _hidden int libxl__qmp_set_global_dirty_log(libxl__gc *gc, int domid, bool 
enable);
 _hidden int libxl__qmp_insert_cdrom(libxl__gc *gc, int domid, const 
libxl_device_disk *disk);
diff --git a/tools/libxl/libxl_qmp.c b/tools/libxl/libxl_qmp.c
index 714038b..eec8a44 100644
--- a/tools/libxl/libxl_qmp.c
+++ b/tools/libxl/libxl_qmp.c
@@ -905,6 +905,16 @@ int libxl__qmp_save(libxl__gc *gc, int domid, const char 
*filename)
NULL, NULL);
 }
 
+int libxl__qmp_restore(libxl__gc *gc, int domid, const char *state_file)
+{
+libxl__json_object *args = NULL;
+
+qmp_parameters_add_string(gc, , "filename", state_file);
+
+return qmp_run_command(gc, domid, "xen-load-devices-state", args,
+   NULL, NULL);
+}
+
 static int qmp_change(libxl__gc *gc, libxl__qmp_handler *qmp,
   char *device, char *target, char *arg)
 {
-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v10 20/31] send store gfn and console gfn to xl before resuming secondary vm

[Wen Congyang]

We will call libxl__xc_domain_restore_done() to rebuild secondary vm. But
we need store gfn and console gfn when rebuilding secondary vm. So make
restore_results a function pointer in callback struct and struct
{save,restore}_callbacks, and use this callback to send store gfn and
console gfn to xl.

Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
CC: Andrew Cooper <andrew.coop...@citrix.com>
---
 tools/libxc/include/xenguest.h | 8 
 tools/libxc/xc_sr_restore.c| 7 +--
 tools/libxl/libxl_colo_restore.c   | 5 -
 tools/libxl/libxl_create.c | 3 +++
 tools/libxl/libxl_save_msgs_gen.pl | 4 ++--
 5 files changed, 18 insertions(+), 9 deletions(-)

diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index cb0bbcf..67cb540 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -127,6 +127,14 @@ struct restore_callbacks {
  */
 int (*wait_checkpoint)(void* data);
 
+/*
+ * callback to send store gfn and console gfn to xl
+ * if we want to resume vm before xc_domain_save()
+ * exits.
+ */
+void (*restore_results)(xen_pfn_t store_gfn, xen_pfn_t console_gfn,
+void *data);
+
 /* to be provided as the last argument to each callback function */
 void* data;
 };
diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
index 37c0416..24c4719 100644
--- a/tools/libxc/xc_sr_restore.c
+++ b/tools/libxc/xc_sr_restore.c
@@ -569,7 +569,9 @@ static int handle_checkpoint(struct xc_sr_context *ctx)
 if ( rc )
 goto err;
 
-/* TODO: call restore_results */
+ctx->restore.callbacks->restore_results(ctx->restore.xenstore_gfn,
+ctx->restore.console_gfn,
+ctx->restore.callbacks->data);
 
 /* Resume secondary vm */
 ret = ctx->restore.callbacks->postcopy(ctx->restore.callbacks->data);
@@ -852,7 +854,8 @@ int xc_domain_restore(xc_interface *xch, int io_fd, 
uint32_t dom,
 /* this is COLO restore */
 assert(callbacks->suspend &&
callbacks->postcopy &&
-   callbacks->wait_checkpoint);
+   callbacks->wait_checkpoint &&
+   callbacks->restore_results);
 }
 
 DPRINTF("fd %d, dom %u, hvm %u, pae %u, superpages %d"
diff --git a/tools/libxl/libxl_colo_restore.c b/tools/libxl/libxl_colo_restore.c
index a417e6e..0ef5b5f 100644
--- a/tools/libxl/libxl_colo_restore.c
+++ b/tools/libxl/libxl_colo_restore.c
@@ -139,11 +139,6 @@ static void colo_resume_vm(libxl__egc *egc,
 return;
 }
 
-/*
- * TODO: get store gfn and console gfn
- *  We should call the callback restore_results in
- *  xc_domain_restore() before resuming the guest.
- */
 libxl__xc_domain_restore_done(egc, dcs, 0, 0, 0);
 
 return;
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 39458b7..f32ca98 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1014,6 +1014,8 @@ static void domcreate_bootloader_done(libxl__egc *egc,
 const int checkpointed_stream = dcs->restore_params.checkpointed_stream;
 libxl__colo_restore_state *const crs = >crs;
 libxl_domain_build_info *const info = _config->b_info;
+libxl__srm_restore_autogen_callbacks *const callbacks =
+>srs.shs.callbacks.restore.a;
 
 if (rc) {
 domcreate_rebuild_done(egc, dcs, rc);
@@ -1041,6 +1043,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
 }
 
 /* Restore */
+callbacks->restore_results = libxl__srm_callout_callback_restore_results;
 
 /* COLO only supports HVM now */
 if (info->type != LIBXL_DOMAIN_TYPE_HVM &&
diff --git a/tools/libxl/libxl_save_msgs_gen.pl 
b/tools/libxl/libxl_save_msgs_gen.pl
index 6016706..c2243f2 100755
--- a/tools/libxl/libxl_save_msgs_gen.pl
+++ b/tools/libxl/libxl_save_msgs_gen.pl
@@ -29,8 +29,8 @@ our @msgs = (
 [  6, 'srcxA',  "wait_checkpoint", [] ],
 [  7, 'scxA',   "switch_qemu_logdirty",  [qw(int domid
   unsigned enable)] ],
-[  8, 'r',  "restore_results",   ['unsigned long', 'store_mfn',
-  'unsigned long', 'console_mfn'] 
],
+[  8, 'rcx',"restore_results",   ['unsigned long', 'store_gfn',
+  'unsigned long', 'console_gfn'] 
],
 [  9, 'srW',"complete",  [qw(int retval
  int errnoval)] ],
 );
-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v10 08/31] libxc/migration: Specification update for DIRTY_PFN_LIST records

[Wen Congyang]

Used by secondary to send it's dirty bitmap to primary under COLO.

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
---
 docs/specs/libxc-migration-stream.pandoc | 24 ++-
 tools/libxc/xc_sr_common.c   | 31 ---
 tools/libxc/xc_sr_stream_format.h| 31 ---
 tools/python/xen/migration/libxc.py  | 68 ++--
 4 files changed, 93 insertions(+), 61 deletions(-)

diff --git a/docs/specs/libxc-migration-stream.pandoc 
b/docs/specs/libxc-migration-stream.pandoc
index 8cd678f..36b880d 100644
--- a/docs/specs/libxc-migration-stream.pandoc
+++ b/docs/specs/libxc-migration-stream.pandoc
@@ -227,7 +227,9 @@ type 0x: END
 
  0x000E: CHECKPOINT
 
- 0x000F - 0x7FFF: Reserved for future _mandatory_
+ 0x000F: CHECKPOINT_DIRTY_PFN_LIST
+
+ 0x0010 - 0x7FFF: Reserved for future _mandatory_
  records.
 
  0x8000 - 0x: Reserved for future _optional_
@@ -599,6 +601,26 @@ CHECKPOINT record or an END record.
 
 \clearpage
 
+CHECKPOINT_DIRTY_PFN_LIST
+
+
+A checkpoint dirty pfn list record is used to convey information about
+dirty memory in the VM. It is an unordered list of PFNs. Currently only
+applicable in the backchannel of a checkpointed stream.
+
+ 0 1 2 3 4 5 6 7 octet
++-+
+| pfn[0]  |
++-+
+...
++-+
+| pfn[C-1]|
++-+
+
+The count of pfns is: record->length/sizeof(uint64_t).
+
+\clearpage
+
 Layout
 ==
 
diff --git a/tools/libxc/xc_sr_common.c b/tools/libxc/xc_sr_common.c
index 945cfa6..3313a90 100644
--- a/tools/libxc/xc_sr_common.c
+++ b/tools/libxc/xc_sr_common.c
@@ -20,21 +20,22 @@ const char *dhdr_type_to_str(uint32_t type)
 
 static const char *mandatory_rec_types[] =
 {
-[REC_TYPE_END]  = "End",
-[REC_TYPE_PAGE_DATA]= "Page data",
-[REC_TYPE_X86_PV_INFO]  = "x86 PV info",
-[REC_TYPE_X86_PV_P2M_FRAMES]= "x86 PV P2M frames",
-[REC_TYPE_X86_PV_VCPU_BASIC]= "x86 PV vcpu basic",
-[REC_TYPE_X86_PV_VCPU_EXTENDED] = "x86 PV vcpu extended",
-[REC_TYPE_X86_PV_VCPU_XSAVE]= "x86 PV vcpu xsave",
-[REC_TYPE_SHARED_INFO]  = "Shared info",
-[REC_TYPE_TSC_INFO] = "TSC info",
-[REC_TYPE_HVM_CONTEXT]  = "HVM context",
-[REC_TYPE_HVM_PARAMS]   = "HVM params",
-[REC_TYPE_TOOLSTACK]= "Toolstack",
-[REC_TYPE_X86_PV_VCPU_MSRS] = "x86 PV vcpu msrs",
-[REC_TYPE_VERIFY]   = "Verify",
-[REC_TYPE_CHECKPOINT]   = "Checkpoint",
+[REC_TYPE_END]  = "End",
+[REC_TYPE_PAGE_DATA]= "Page data",
+[REC_TYPE_X86_PV_INFO]  = "x86 PV info",
+[REC_TYPE_X86_PV_P2M_FRAMES]= "x86 PV P2M frames",
+[REC_TYPE_X86_PV_VCPU_BASIC]= "x86 PV vcpu basic",
+[REC_TYPE_X86_PV_VCPU_EXTENDED] = "x86 PV vcpu extended",
+[REC_TYPE_X86_PV_VCPU_XSAVE]= "x86 PV vcpu xsave",
+[REC_TYPE_SHARED_INFO]  = "Shared info",
+[REC_TYPE_TSC_INFO] = "TSC info",
+[REC_TYPE_HVM_CONTEXT]  = "HVM context",
+[REC_TYPE_HVM_PARAMS]   = "HVM params",
+[REC_TYPE_TOOLSTACK]= "Toolstack",
+[REC_TYPE_X86_PV_VCPU_MSRS] = "x86 PV vcpu msrs",
+[REC_TYPE_VERIFY]   = "Verify",
+[REC_TYPE_CHECKPOINT]   = "Checkpoint",
+[REC_TYPE_CHECKPOINT_DIRTY_PFN_LIST]= "Checkpoint dirty pfn list",
 };
 
 const char *rec_type_to_str(uint32_t type)
diff --git a/tools/libxc/xc_sr_stream_format.h 
b/tools/libxc/xc_sr_stream_format.h
index 6d0f8fd..3291b25 100644
--- a/tools/libxc/xc_sr_stream_format.h
+++ b/tools/libxc/xc_sr_stream_format.h
@@ -60,21 +60,22 @@ struct xc_sr_rhdr
 /* Somewhat arbitrary - 8MB */
 #define REC_LENGTH_MAX(8U << 20)
 
-#define REC_TYPE_END  0xU
-#define REC_TYPE_PAGE_DATA0x0001U
-#define REC_TYPE_X86_PV_INFO  0x0002U
-#define REC_TYPE_X86_PV_P2M_FRAMES0x0003U
-#define REC_TYPE_X86_PV_VCPU_BASIC0x0004U
-#

[Xen-devel] [PATCH v10 19/31] libxc/restore: send dirty pfn list to primary when checkpoint under colo

[Wen Congyang]

Send dirty pfn list to primary when checkpoint under colo.

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
---
 tools/libxc/xc_sr_common.h  |   4 ++
 tools/libxc/xc_sr_restore.c | 120 +++-
 2 files changed, 123 insertions(+), 1 deletion(-)

diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index a24a9ad..29ab4eb 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -226,6 +226,10 @@ struct xc_sr_context
 struct xc_sr_restore_ops ops;
 struct restore_callbacks *callbacks;
 
+int send_back_fd;
+unsigned long p2m_size;
+xc_hypercall_buffer_t dirty_bitmap_hbuf;
+
 /* From Image Header. */
 uint32_t format_version;
 
diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
index 2ae8154..37c0416 100644
--- a/tools/libxc/xc_sr_restore.c
+++ b/tools/libxc/xc_sr_restore.c
@@ -411,6 +411,92 @@ static int handle_page_data(struct xc_sr_context *ctx, 
struct xc_sr_record *rec)
 return rc;
 }
 
+/*
+ * Send checkpoint dirty pfn list to primary.
+ */
+static int send_checkpoint_dirty_pfn_list(struct xc_sr_context *ctx)
+{
+xc_interface *xch = ctx->xch;
+int rc = -1;
+unsigned count, written;
+uint64_t i, *pfns = NULL;
+struct iovec *iov = NULL;
+xc_shadow_op_stats_t stats = { 0, ctx->restore.p2m_size };
+struct xc_sr_record rec =
+{
+.type = REC_TYPE_CHECKPOINT_DIRTY_PFN_LIST,
+};
+DECLARE_HYPERCALL_BUFFER_SHADOW(unsigned long, dirty_bitmap,
+>restore.dirty_bitmap_hbuf);
+
+if ( xc_shadow_control(
+ xch, ctx->domid, XEN_DOMCTL_SHADOW_OP_CLEAN,
+ HYPERCALL_BUFFER(dirty_bitmap), ctx->restore.p2m_size,
+ NULL, 0, ) != ctx->restore.p2m_size )
+{
+PERROR("Failed to retrieve logdirty bitmap");
+goto err;
+}
+
+for ( i = 0, count = 0; i < ctx->restore.p2m_size; i++ )
+{
+if ( test_bit(i, dirty_bitmap) )
+count++;
+}
+
+
+pfns = malloc(count * sizeof(*pfns));
+if ( !pfns )
+{
+ERROR("Unable to allocate %zu bytes of memory for dirty pfn list",
+  count * sizeof(*pfns));
+goto err;
+}
+
+for ( i = 0, written = 0; i < ctx->restore.p2m_size; ++i )
+{
+if ( !test_bit(i, dirty_bitmap) )
+continue;
+
+if ( written > count )
+{
+ERROR("Dirty pfn list exceed");
+goto err;
+}
+
+pfns[written++] = i;
+}
+
+/* iovec[] for writev(). */
+iov = malloc(3 * sizeof(*iov));
+if ( !iov )
+{
+ERROR("Unable to allocate memory for sending dirty bitmap");
+goto err;
+}
+
+rec.length = count * sizeof(*pfns);
+
+iov[0].iov_base = 
+iov[0].iov_len = sizeof(rec.type);
+
+iov[1].iov_base = 
+iov[1].iov_len = sizeof(rec.length);
+
+iov[2].iov_base = pfns;
+iov[2].iov_len = count * sizeof(*pfns);
+
+if ( writev_exact(ctx->restore.send_back_fd, iov, 3) )
+{
+PERROR("Failed to write dirty bitmap to stream");
+goto err;
+}
+
+rc = 0;
+ err:
+return rc;
+}
+
 static int process_record(struct xc_sr_context *ctx, struct xc_sr_record *rec);
 static int handle_checkpoint(struct xc_sr_context *ctx)
 {
@@ -500,7 +586,9 @@ static int handle_checkpoint(struct xc_sr_context *ctx)
 
 #undef HANDLE_CALLBACK_RETURN_VALUE
 
-/* TODO: send dirty pfn list to primary */
+rc = send_checkpoint_dirty_pfn_list(ctx);
+if ( rc )
+goto err;
 }
 
  err:
@@ -572,6 +660,21 @@ static int setup(struct xc_sr_context *ctx)
 {
 xc_interface *xch = ctx->xch;
 int rc;
+DECLARE_HYPERCALL_BUFFER_SHADOW(unsigned long, dirty_bitmap,
+>restore.dirty_bitmap_hbuf);
+
+if ( ctx->restore.checkpointed == MIG_STREAM_COLO )
+{
+dirty_bitmap = xc_hypercall_buffer_alloc_pages(xch, dirty_bitmap,
+NRPAGES(bitmap_size(ctx->restore.p2m_size)));
+
+if ( !dirty_bitmap )
+{
+ERROR("Unable to allocate memory for dirty bitmap");
+rc = -1;
+goto err;
+}
+}
 
 rc = ctx->restore.ops.setup(ctx);
 if ( rc )
@@ -605,10 +708,15 @@ static void cleanup(struct xc_sr_context *ctx)
 {
 xc_interface *xch = ctx->xch;
 unsigned i;
+DECLARE_HYPERCALL_BUFFER_SHADOW(unsigned long, dirty_bitmap,
+>restore.dirty_bitmap_hbuf);
 
 for ( i = 0; i < ctx->restore.buffered_rec_num; i++ )
 free(ctx->restore.buffered_records[i].data);
 
+if ( ctx->restore.checkpointed == MIG_STREAM_COL

[Xen-devel] [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service

[Wen Congyang]

This patchset implemented the COLO feature for Xen.
For detail/install/use of COLO feature, refer to:
  http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping

This patchset is based on:
1. http://lists.xenproject.org/archives/html/xen-devel/2016-02/msg02515.html

You can get the codes from here:
https://github.com/wencongyang/xen/tree/colo_v10

Changlog from v9 to v10
1. Rebased to the upstream xen
2. Fix one bug found in the test
3. Merge some patches from prepare series
4. Split patch 5 to two patches(patch 4 and 5) according to the comments from
   Wei Liu

Changlog from v8 to v9:
1. Rebased to the upstream xen
2. Fix some bugs found in the test

Changelog from v7 to v8:
1. Rebased to the latest libxl migration v2.

Changelog from v6 to v7:
1. Ported to Libxl migration v2
2. Send dirty bitmap from secondary to primary on libxc side
3. Address review comments

Changelog from v5 to v6:
1. based on migration v2(libxc)
2. split the patchset into prerequisite patchset and this main patchset.

Changelog from v4 to v5:
1. rebase to the latest xen upstream
2. disk replication: blktap2->qdisk
3. nic replication: colo-agent->colo-proxy

Changelog from v3 to v4:
1. rebase to newest xen
2. bug fix

Changlog from v2 to v3:
1. rebase to newest remus
2. add nic replication support

Changlog from v1 to v2:
1. rebase to newest remus
2. add disk replication support

Wen Congyang (31):
  tools/libxl: introduce libxl__domain_restore_device_model to load qemu
state
  tools/libxl: introduce libxl__domain_common_switch_qemu_logdirty()
  tools/libxl: Add back channel to allow migration target send data back
  tools/libxl: Introduce new helper function dup_fd_helper()
  tools/libx{l,c}: add back channel to libxc
  docs: add colo readme
  docs/libxl: Introduce CHECKPOINT_CONTEXT to support migration v2 colo
streams
  libxc/migration: Specification update for DIRTY_PFN_LIST records
  libxc/migration: export read_record for common use
  tools/libxl: add back channel support to write stream
  tools/libxl: write checkpoint_state records into the stream
  tools/libxl: add back channel support to read stream
  tools/libxl: handle checkpoint_state records in a libxl migration v2
read stream
  tools/libx{l,c}: introduce wait_checkpoint callback
  tools/libx{l,c}: add postcopy/suspend callback to restore side
  secondary vm suspend/resume/checkpoint code
  primary vm suspend/resume/checkpoint code
  libxc/restore: support COLO restore
  libxc/restore: send dirty pfn list to primary when checkpoint under
colo
  send store gfn and console gfn to xl before resuming secondary vm
  libxc/save: support COLO save
  implement the cmdline for COLO
  COLO: introduce new API to prepare/start/do/get_error/stop replication
  Support colo mode for qemu disk
  COLO: use qemu block replication
  COLO proxy: implement setup/teardown of COLO proxy module
  COLO proxy: preresume, postresume and checkpoint
  COLO nic: implement COLO nic subkind
  setup and control colo proxy on primary side
  setup and control colo proxy on secondary side
  cmdline switches and config vars to control colo-proxy

 docs/README.colo |9 +
 docs/man/xl.conf.pod.5   |6 +
 docs/man/xl.pod.1|   11 +-
 docs/misc/xl-disk-configuration.txt  |   50 ++
 docs/specs/libxc-migration-stream.pandoc |   24 +-
 docs/specs/libxl-migration-stream.pandoc |   25 +-
 tools/hotplug/Linux/Makefile |1 +
 tools/hotplug/Linux/colo-proxy-setup |  135 
 tools/libxc/include/xenguest.h   |   40 +-
 tools/libxc/xc_nomigrate.c   |4 +-
 tools/libxc/xc_sr_common.c   |   80 ++-
 tools/libxc/xc_sr_common.h   |   26 +-
 tools/libxc/xc_sr_restore.c  |  246 +--
 tools/libxc/xc_sr_save.c |  109 ++-
 tools/libxc/xc_sr_stream_format.h|   31 +-
 tools/libxl/Makefile |4 +
 tools/libxl/libxl.c  |  100 ++-
 tools/libxl/libxl.h  |   29 +-
 tools/libxl/libxl_colo.h |   36 +
 tools/libxl/libxl_colo_nic.c |  321 +
 tools/libxl/libxl_colo_proxy.c   |  292 
 tools/libxl/libxl_colo_qdisk.c   |  226 +++
 tools/libxl/libxl_colo_restore.c | 1091 ++
 tools/libxl/libxl_colo_save.c|  721 
 tools/libxl/libxl_create.c   |   86 ++-
 tools/libxl/libxl_device.c   |   54 ++
 tools/libxl/libxl_dm.c   |  184 -
 tools/libxl/libxl_dom_save.c |  124 ++--
 tools/libxl/libxl_internal.h |  247 +--
 tools/libxl/libxl_qmp.c  |  106 +++
 tools/libxl/libxl_save_callout.c |   52 +-
 tools/libxl/libxl_save_helper.c  |8 +-
 tools/libxl/libxl_save_msgs_gen.pl   |   13 +-
 tools/libxl/libxl_sr_stream_format.h |   11 +
 tools

[Xen-devel] [PATCH v10 17/31] primary vm suspend/resume/checkpoint code

[Wen Congyang]

We will do the following things again and again:
1. Suspend primary vm
   a. Suspend primary vm
   b. do postsuspend
   c. Read CHECKPOINT_SVM_SUSPENDED sent by secondary
2. Resume primary vm
   a. Read CHECKPOINT_SVM_READY from slave
   b. Do presume
   c. Resume primary vm
   d. Read CHECKPOINT_SVM_RESUMED from slave
3. Wait a new checkpoint
   a. Wait a new checkpoint(not implemented)
   b. Send CHECKPOINT_NEW to slave

Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
---
 tools/libxl/Makefile  |   2 +-
 tools/libxl/libxl.c   |   6 +-
 tools/libxl/libxl_colo.h  |   6 +
 tools/libxl/libxl_colo_save.c | 563 ++
 tools/libxl/libxl_dom_save.c  |   8 +-
 tools/libxl/libxl_internal.h  | 168 +++--
 tools/libxl/libxl_types.idl   |   1 +
 7 files changed, 674 insertions(+), 80 deletions(-)
 create mode 100644 tools/libxl/libxl_colo_save.c

diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index d8612eb..2016393 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -65,7 +65,7 @@ LIBXL_OBJS-y += libxl_no_convert_callout.o
 endif
 
 LIBXL_OBJS-y += libxl_remus.o libxl_checkpoint_device.o libxl_remus_disk_drbd.o
-LIBXL_OBJS-y += libxl_colo_restore.o
+LIBXL_OBJS-y += libxl_colo_restore.o libxl_colo_save.o
 
 LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o libxl_psr.o
 LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_arm.o libxl_libfdt_compat.o
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index e9ab78c..7236175 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -17,6 +17,7 @@
 #include "libxl_osdeps.h"
 
 #include "libxl_internal.h"
+#include "libxl_colo.h"
 
 #define PAGE_TO_MEMKB(pages) ((pages) * 4)
 #define BACKEND_STRING_SIZE 5
@@ -880,7 +881,10 @@ int libxl_domain_remus_start(libxl_ctx *ctx, 
libxl_domain_remus_info *info,
 assert(info);
 
 /* Point of no return */
-libxl__remus_setup(egc, >rs);
+if (libxl_defbool_val(info->colo))
+libxl__colo_save_setup(egc, >css);
+else
+libxl__remus_setup(egc, >rs);
 return AO_INPROGRESS;
 
  out:
diff --git a/tools/libxl/libxl_colo.h b/tools/libxl/libxl_colo.h
index 8bea1a2..b08f672 100644
--- a/tools/libxl/libxl_colo.h
+++ b/tools/libxl/libxl_colo.h
@@ -21,4 +21,10 @@ extern void libxl__colo_restore_setup(libxl__egc *egc,
 extern void libxl__colo_restore_teardown(libxl__egc *egc, void *dcs_void,
  int ret, int retval, int errnoval);
 
+extern void libxl__colo_save_setup(libxl__egc *egc,
+   libxl__colo_save_state *css);
+extern void libxl__colo_save_teardown(libxl__egc *egc,
+  libxl__colo_save_state *css,
+  int rc);
+
 #endif
diff --git a/tools/libxl/libxl_colo_save.c b/tools/libxl/libxl_colo_save.c
new file mode 100644
index 000..8b18a91
--- /dev/null
+++ b/tools/libxl/libxl_colo_save.c
@@ -0,0 +1,563 @@
+/*
+ * Copyright (C) 2014 FUJITSU LIMITED
+ * Author: Wen Congyang <we...@cn.fujitsu.com>
+ * Yang Hongyang <hongyang.y...@easystack.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+#include "libxl_colo.h"
+
+static const libxl__checkpoint_device_instance_ops *colo_ops[] = {
+NULL,
+};
+
+/* = helper functions = */
+static int init_device_subkind(libxl__checkpoint_devices_state *cds)
+{
+/* init device subkind-specific state in the libxl ctx */
+int rc;
+STATE_AO_GC(cds->ao);
+
+rc = 0;
+return rc;
+}
+
+static void cleanup_device_subkind(libxl__checkpoint_devices_state *cds)
+{
+/* cleanup device subkind-specific state in the libxl ctx */
+STATE_AO_GC(cds->ao);
+}
+
+/* = colo: setup save environment = */
+static void colo_save_setup_done(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc);
+static void colo_save_setup_failed(libxl__egc *egc,
+   libxl__checkpoint_devices_state *cds,
+   int rc);
+static void libxl__colo_save_domain_suspend_callback(void *data);
+static void libxl__

[Xen-devel] [PATCH v10 22/31] implement the cmdline for COLO

[Wen Congyang]

Add a new option -c to the command 'xl remus'. If you want
to use COLO HA instead of Remus HA, please use -c option.

Update man pages to reflect the addition of a new option to
'xl remus' command.

Also add a new option -c to the internal command 'xl migrate-receive'.

Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
---
 docs/man/xl.pod.1 | 12 +++--
 tools/libxl/libxl.c   | 23 --
 tools/libxl/xl_cmdimpl.c  | 62 ---
 tools/libxl/xl_cmdtable.c |  4 ++-
 4 files changed, 82 insertions(+), 19 deletions(-)

diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
index 4279c7c..1c6dd87 100644
--- a/docs/man/xl.pod.1
+++ b/docs/man/xl.pod.1
@@ -447,12 +447,15 @@ Print huge (!) amount of debug during the migration 
process.
 
 =item B [I] I I
 
-Enable Remus HA for domain. By default B relies on ssh as a transport
-mechanism between the two hosts.
+Enable Remus HA or COLO HA for domain. By default B relies on ssh as a
+transport mechanism between the two hosts.
 
 N.B: Remus support in xl is still in experimental (proof-of-concept) phase.
  Disk replication support is limited to DRBD disks.
 
+ COLO support in xl is still in experimental (proof-of-concept) phase.
+ There is no support for network or disk at the moment.
+
 B
 
 =over 4
@@ -498,6 +501,11 @@ Disable network output buffering. Requires enabling unsafe 
mode.
 
 Disable disk replication. Requires enabling unsafe mode.
 
+=item B<-c>
+
+Enable COLO HA. This conflicts with B<-i> and B<-b>, and memory
+checkpoint compression must be disabled.
+
 =back
 
 =item B I
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 7236175..12df81a 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -849,12 +849,28 @@ int libxl_domain_remus_start(libxl_ctx *ctx, 
libxl_domain_remus_info *info,
 goto out;
 }
 
+/* The caller must set this defbool */
+if (libxl_defbool_is_default(info->colo)) {
+LOG(ERROR, "colo mode must be enabled/disabled");
+rc = ERROR_FAIL;
+goto out;
+}
+
 libxl_defbool_setdefault(>allow_unsafe, false);
 libxl_defbool_setdefault(>blackhole, false);
-libxl_defbool_setdefault(>compression, true);
+libxl_defbool_setdefault(>compression,
+ !libxl_defbool_val(info->colo));
 libxl_defbool_setdefault(>netbuf, true);
 libxl_defbool_setdefault(>diskbuf, true);
 
+if (libxl_defbool_val(info->colo)) {
+if (libxl_defbool_val(info->compression)) {
+LOG(ERROR, "cannot use memory checkpoint compression in COLO 
mode");
+rc = ERROR_FAIL;
+goto out;
+}
+}
+
 if (!libxl_defbool_val(info->allow_unsafe) &&
 (libxl_defbool_val(info->blackhole) ||
  !libxl_defbool_val(info->netbuf) ||
@@ -876,7 +892,10 @@ int libxl_domain_remus_start(libxl_ctx *ctx, 
libxl_domain_remus_info *info,
 dss->live = 1;
 dss->debug = 0;
 dss->remus = info;
-dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
+if (libxl_defbool_val(info->colo))
+dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_COLO;
+else
+dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
 
 assert(info);
 
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index df7268b..0dc7220 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -4440,6 +4440,8 @@ static void migrate_receive(int debug, int daemonize, int 
monitor,
 char rc_buf;
 char *migration_domname;
 struct domain_create dom_info;
+const char *ha = checkpointed == LIBXL_CHECKPOINTED_STREAM_COLO ?
+ "COLO" : "Remus";
 
 signal(SIGPIPE, SIG_IGN);
 /* if we get SIGPIPE we'd rather just have it as an error */
@@ -4460,6 +4462,9 @@ static void migrate_receive(int debug, int daemonize, int 
monitor,
 dom_info.send_back_fd = send_fd;
 dom_info.migration_domname_r = _domname;
 dom_info.checkpointed_stream = checkpointed;
+if (checkpointed == LIBXL_CHECKPOINTED_STREAM_COLO)
+/* COLO uses stdout to send control message to master */
+dom_info.quiet = 1;
 
 rc = create_domain(_info);
 if (rc < 0) {
@@ -4472,11 +4477,12 @@ static void migrate_receive(int debug, int daemonize, 
int monitor,
 
 switch (checkpointed) {
 case LIBXL_CHECKPOINTED_STREAM_REMUS:
+case LIBXL_CHECKPOINTED_STREAM_COLO:
 /* If we are here, it means that the sender (primary) has crashed.
  * TODO: Split-Brain Check.
  */
-fprintf(stderr, "migration target: Remus Failover for domain %u\n",
-domid);
+fprintf(stderr, "migration target: %s Failover for domain %u\n"

[Xen-devel] [PATCH v10 31/31] cmdline switches and config vars to control colo-proxy

[Wen Congyang]

Add cmdline switches to 'xl migrate-receive' command to specify
a domain-specific hotplug script to setup COLO proxy.

Add a new config var 'colo.default.agentscript' to xl.conf, that
allows the user to override the default global script used to
setup COLO proxy.

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
---
 docs/man/xl.conf.pod.5  |  6 ++
 docs/man/xl.pod.1   |  1 -
 tools/libxl/libxl.c |  6 ++
 tools/libxl/libxl_create.c  | 14 --
 tools/libxl/libxl_types.idl |  1 +
 tools/libxl/xl.c|  3 +++
 tools/libxl/xl.h|  1 +
 tools/libxl/xl_cmdimpl.c| 47 ++---
 8 files changed, 65 insertions(+), 14 deletions(-)

diff --git a/docs/man/xl.conf.pod.5 b/docs/man/xl.conf.pod.5
index 8ae19bb..8f7fd28 100644
--- a/docs/man/xl.conf.pod.5
+++ b/docs/man/xl.conf.pod.5
@@ -111,6 +111,12 @@ Configures the default script used by Remus to setup 
network buffering.
 
 Default: C
 
+=item B

[Xen-devel] [PATCH v10 16/31] secondary vm suspend/resume/checkpoint code

[Wen Congyang]

Secondary vm is running in colo mode. So we will do
the following things again and again:
1. Resume secondary vm
   a. Send CHECKPOINT_SVM_READY to master.
   b. If it is not the first resume, call libxl__checkpoint_devices_preresume().
   c. If it is the first resume(resume right after live migration),
  - call libxl__xc_domain_restore_done() to build the secondary vm.
  - enable secondary vm's logdirty.
  - call libxl__domain_resume() to resume secondary vm.
  - call libxl__checkpoint_devices_setup() to setup checkpoint devices.
   d. Send CHECKPOINT_SVM_RESUMED to master.
2. Wait a new checkpoint
   a. Call libxl__checkpoint_devices_commit().
   b. Read CHECKPOINT_NEW from master.
3. Suspend secondary vm
   a. Suspend secondary vm.
   b. Call libxl__checkpoint_devices_postsuspend().
   c. Send CHECKPOINT_SVM_SUSPENDED to master.

Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
---
 tools/libxc/xc_sr_common.h   |2 +
 tools/libxc/xc_sr_save.c |3 +-
 tools/libxl/Makefile |1 +
 tools/libxl/libxl_colo.h |   24 +
 tools/libxl/libxl_colo_restore.c | 1038 ++
 tools/libxl/libxl_create.c   |   37 ++
 tools/libxl/libxl_internal.h |   19 +
 tools/libxl/libxl_save_callout.c |7 +-
 tools/libxl/libxl_stream_read.c  |   12 +
 tools/libxl/libxl_types.idl  |1 +
 10 files changed, 1142 insertions(+), 2 deletions(-)
 create mode 100644 tools/libxl/libxl_colo.h
 create mode 100644 tools/libxl/libxl_colo_restore.c

diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 5d9f497..2bfed64 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -184,10 +184,12 @@ struct xc_sr_context
  * migration stream
  * 0: Plain VM
  * 1: Remus
+ * 2: COLO
  */
 enum {
 MIG_STREAM_NONE, /* plain stream */
 MIG_STREAM_REMUS,
+MIG_STREAM_COLO,
 } migration_stream;
 
 union /* Common save or restore data. */
diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
index fe210cc..7393355 100644
--- a/tools/libxc/xc_sr_save.c
+++ b/tools/libxc/xc_sr_save.c
@@ -846,7 +846,8 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t 
dom,
 
 /* If altering migration_stream update this assert too. */
 assert(checkpointed_stream == MIG_STREAM_NONE ||
-   checkpointed_stream == MIG_STREAM_REMUS);
+   checkpointed_stream == MIG_STREAM_REMUS ||
+   checkpointed_stream == MIG_STREAM_COLO);
 
 /*
  * TODO: Find some time to better tweak the live migration algorithm.
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 789a12e..d8612eb 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -65,6 +65,7 @@ LIBXL_OBJS-y += libxl_no_convert_callout.o
 endif
 
 LIBXL_OBJS-y += libxl_remus.o libxl_checkpoint_device.o libxl_remus_disk_drbd.o
+LIBXL_OBJS-y += libxl_colo_restore.o
 
 LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o libxl_psr.o
 LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_arm.o libxl_libfdt_compat.o
diff --git a/tools/libxl/libxl_colo.h b/tools/libxl/libxl_colo.h
new file mode 100644
index 000..8bea1a2
--- /dev/null
+++ b/tools/libxl/libxl_colo.h
@@ -0,0 +1,24 @@
+/*
+ * Copyright (C) 2014 FUJITSU LIMITED
+ * Author: Wen Congyang <we...@cn.fujitsu.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#ifndef LIBXL_COLO_H
+#define LIBXL_COLO_H
+
+extern void libxl__colo_restore_setup(libxl__egc *egc,
+  libxl__colo_restore_state *crs);
+extern void libxl__colo_restore_teardown(libxl__egc *egc, void *dcs_void,
+ int ret, int retval, int errnoval);
+
+#endif
diff --git a/tools/libxl/libxl_colo_restore.c b/tools/libxl/libxl_colo_restore.c
new file mode 100644
index 000..a417e6e
--- /dev/null
+++ b/tools/libxl/libxl_colo_restore.c
@@ -0,0 +1,1038 @@
+/*
+ * Copyright (C) 2014 FUJITSU LIMITED
+ * Author: Wen Congyang <we...@cn.fujitsu.com>
+ * Yang Hongyang <hongyang.y...@easystack.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in t

[Xen-devel] [PATCH v10 27/31] COLO proxy: preresume, postresume and checkpoint

[Wen Congyang]

preresume, postresume and checkpoint

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
---
 tools/libxl/libxl_colo.h   |  4 +++
 tools/libxl/libxl_colo_proxy.c | 62 ++
 2 files changed, 66 insertions(+)

diff --git a/tools/libxl/libxl_colo.h b/tools/libxl/libxl_colo.h
index 6d8cd4f..17dd07f 100644
--- a/tools/libxl/libxl_colo.h
+++ b/tools/libxl/libxl_colo.h
@@ -29,4 +29,8 @@ extern void libxl__colo_save_teardown(libxl__egc *egc,
 
 extern int colo_proxy_setup(libxl__colo_proxy_state *cps);
 extern void colo_proxy_teardown(libxl__colo_proxy_state *cps);
+extern void colo_proxy_preresume(libxl__colo_proxy_state *cps);
+extern void colo_proxy_postresume(libxl__colo_proxy_state *cps);
+extern int colo_proxy_checkpoint(libxl__colo_proxy_state *cps,
+ unsigned int timeout_us);
 #endif
diff --git a/tools/libxl/libxl_colo_proxy.c b/tools/libxl/libxl_colo_proxy.c
index e07e640..c714b52 100644
--- a/tools/libxl/libxl_colo_proxy.c
+++ b/tools/libxl/libxl_colo_proxy.c
@@ -228,3 +228,65 @@ void colo_proxy_teardown(libxl__colo_proxy_state *cps)
 cps->sock_fd = -1;
 }
 }
+
+/* = colo-proxy: preresume, postresume and checkpoint == */
+
+void colo_proxy_preresume(libxl__colo_proxy_state *cps)
+{
+colo_proxy_send(cps, NULL, 0, COLO_CHECKPOINT);
+/* TODO: need to handle if the call fails... */
+}
+
+void colo_proxy_postresume(libxl__colo_proxy_state *cps)
+{
+/* nothing to do... */
+}
+
+typedef struct colo_msg {
+bool is_checkpoint;
+} colo_msg;
+
+/*
+ * Return value:
+ * -1: error
+ *  0: no checkpoint event is received before timeout
+ *  1: do checkpoint
+ */
+int colo_proxy_checkpoint(libxl__colo_proxy_state *cps,
+  unsigned int timeout_us)
+{
+uint8_t *buff;
+int64_t size;
+struct nlmsghdr *h;
+struct colo_msg *m;
+int ret = -1;
+
+STATE_AO_GC(cps->ao);
+
+size = colo_proxy_recv(cps, , timeout_us);
+
+/* timeout, return no checkpoint message. */
+if (size <= 0) {
+return 0;
+}
+
+h = (struct nlmsghdr *) buff;
+
+if (h->nlmsg_type == NLMSG_ERROR) {
+LOG(ERROR, "receive NLMSG_ERROR");
+goto out;
+}
+
+if (h->nlmsg_len < NLMSG_LENGTH(sizeof(*m))) {
+LOG(ERROR, "NLMSG_LENGTH is too short");
+goto out;
+}
+
+m = NLMSG_DATA(h);
+
+ret = m->is_checkpoint ? 1 : 0;
+
+out:
+free(buff);
+return ret;
+}
-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v10 26/31] COLO proxy: implement setup/teardown of COLO proxy module

[Wen Congyang]

setup/teardown of COLO proxy module.
we use netlink to communicate with proxy module.
About colo-proxy module:
https://lkml.org/lkml/2015/6/18/32
How to use:
http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
---
 tools/libxl/Makefile   |   1 +
 tools/libxl/libxl_colo.h   |   2 +
 tools/libxl/libxl_colo_proxy.c | 230 +
 tools/libxl/libxl_internal.h   |  15 +++
 4 files changed, 248 insertions(+)
 create mode 100644 tools/libxl/libxl_colo_proxy.c

diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 28d54d0..6fea9e0 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -67,6 +67,7 @@ endif
 LIBXL_OBJS-y += libxl_remus.o libxl_checkpoint_device.o libxl_remus_disk_drbd.o
 LIBXL_OBJS-y += libxl_colo_restore.o libxl_colo_save.o
 LIBXL_OBJS-y += libxl_colo_qdisk.o
+LIBXL_OBJS-y += libxl_colo_proxy.o
 
 LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o libxl_psr.o
 LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_arm.o libxl_libfdt_compat.o
diff --git a/tools/libxl/libxl_colo.h b/tools/libxl/libxl_colo.h
index b08f672..6d8cd4f 100644
--- a/tools/libxl/libxl_colo.h
+++ b/tools/libxl/libxl_colo.h
@@ -27,4 +27,6 @@ extern void libxl__colo_save_teardown(libxl__egc *egc,
   libxl__colo_save_state *css,
   int rc);
 
+extern int colo_proxy_setup(libxl__colo_proxy_state *cps);
+extern void colo_proxy_teardown(libxl__colo_proxy_state *cps);
 #endif
diff --git a/tools/libxl/libxl_colo_proxy.c b/tools/libxl/libxl_colo_proxy.c
new file mode 100644
index 000..e07e640
--- /dev/null
+++ b/tools/libxl/libxl_colo_proxy.c
@@ -0,0 +1,230 @@
+/*
+ * Copyright (C) 2015 FUJITSU LIMITED
+ * Author: Yang Hongyang <hongyang.y...@easystack.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+#include "libxl_colo.h"
+#include 
+
+#define NETLINK_COLO 28
+
+enum colo_netlink_op {
+COLO_QUERY_CHECKPOINT = (NLMSG_MIN_TYPE + 1),
+COLO_CHECKPOINT,
+COLO_FAILOVER,
+COLO_PROXY_INIT,
+COLO_PROXY_RESET, /* UNUSED, will be used for continuous FT */
+};
+
+/* = colo-proxy: helper functions == */
+
+static int colo_proxy_send(libxl__colo_proxy_state *cps, uint8_t *buff,
+   uint64_t size, int type)
+{
+struct sockaddr_nl sa;
+struct nlmsghdr msg;
+struct iovec iov;
+struct msghdr mh;
+int ret;
+
+STATE_AO_GC(cps->ao);
+
+memset(, 0, sizeof(sa));
+sa.nl_family = AF_NETLINK;
+sa.nl_pid = 0;
+sa.nl_groups = 0;
+
+msg.nlmsg_len = NLMSG_SPACE(0);
+msg.nlmsg_flags = NLM_F_REQUEST;
+if (type == COLO_PROXY_INIT) {
+msg.nlmsg_flags |= NLM_F_ACK;
+}
+msg.nlmsg_seq = 0;
+/* This is untrusty */
+msg.nlmsg_pid = cps->index;
+msg.nlmsg_type = type;
+
+iov.iov_base = 
+iov.iov_len = msg.nlmsg_len;
+
+mh.msg_name = 
+mh.msg_namelen = sizeof(sa);
+mh.msg_iov = 
+mh.msg_iovlen = 1;
+mh.msg_control = NULL;
+mh.msg_controllen = 0;
+mh.msg_flags = 0;
+
+ret = sendmsg(cps->sock_fd, , 0);
+if (ret <= 0) {
+LOG(ERROR, "can't send msg to kernel by netlink: %s",
+strerror(errno));
+}
+
+return ret;
+}
+
+/* error: return -1, otherwise return 0 */
+static int64_t colo_proxy_recv(libxl__colo_proxy_state *cps, uint8_t **buff,
+   unsigned int timeout_us)
+{
+struct sockaddr_nl sa;
+struct iovec iov;
+struct msghdr mh = {
+.msg_name = ,
+.msg_namelen = sizeof(sa),
+.msg_iov = ,
+.msg_iovlen = 1,
+};
+struct timeval tv;
+uint32_t size = 16384;
+int64_t len = 0;
+int ret;
+
+STATE_AO_GC(cps->ao);
+uint8_t *tmp = libxl__malloc(NOGC, size);
+
+if (timeout_us) {
+tv.tv_sec = timeout_us / 100;
+tv.tv_usec = timeout_us % 100;
+setsockopt(cps->sock_fd, SOL_SOCKET, SO_RCVTIMEO, , sizeof(tv));
+}
+
+iov.iov_base = tmp;
+iov.iov_len = size;
+next:
+ret = recvmsg(cps->sock_fd, , 0);
+if (ret <= 0) {
+if (errno != EAGAIN && errno != EWOULDBLOCK)
+LOGE(ER

[Xen-devel] [PATCH v10 10/31] tools/libxl: add back channel support to write stream

[Wen Congyang]

Add back channel support to write stream. If the write stream is
a back channel stream, this means the write stream is used by
Secondary to send some records back.

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
---
 tools/libxl/libxl_dom_save.c |  1 +
 tools/libxl/libxl_internal.h |  1 +
 tools/libxl/libxl_stream_write.c | 26 --
 3 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
index 72b61c7..18946c5 100644
--- a/tools/libxl/libxl_dom_save.c
+++ b/tools/libxl/libxl_dom_save.c
@@ -404,6 +404,7 @@ void libxl__domain_save(libxl__egc *egc, 
libxl__domain_save_state *dss)
 dss->sws.ao  = dss->ao;
 dss->sws.dss = dss;
 dss->sws.fd  = dss->fd;
+dss->sws.back_channel = false;
 dss->sws.completion_callback = stream_done;
 
 libxl__stream_write_start(egc, >sws);
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 3d3e8e8..e02e554 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3044,6 +3044,7 @@ struct libxl__stream_write_state {
 libxl__ao *ao;
 libxl__domain_save_state *dss;
 int fd;
+bool back_channel;
 void (*completion_callback)(libxl__egc *egc,
 libxl__stream_write_state *sws,
 int rc);
diff --git a/tools/libxl/libxl_stream_write.c b/tools/libxl/libxl_stream_write.c
index f6ea55d..5379126 100644
--- a/tools/libxl/libxl_stream_write.c
+++ b/tools/libxl/libxl_stream_write.c
@@ -49,6 +49,13 @@
  *  - if (hvm)
  *  - Emulator context record
  *  - Checkpoint end record
+ *
+ * For back channel stream:
+ * - libxl__stream_write_start()
+ *- Set up the stream to running state
+ *
+ * - Add a new API to write the record. When the record is written
+ *   out, call stream->checkpoint_callback() to return.
  */
 
 /* Success/error/cleanup handling. */
@@ -225,6 +232,15 @@ void libxl__stream_write_start(libxl__egc *egc,
 
 stream->running = true;
 
+dc->ao= ao;
+dc->readfd= -1;
+dc->copywhat  = "save v2 stream";
+dc->writefd   = stream->fd;
+dc->maxsz = -1;
+
+if (stream->back_channel)
+return;
+
 if (dss->type == LIBXL_DOMAIN_TYPE_HVM) {
 stream->device_model_version =
 libxl__device_model_version_running(gc, dss->domid);
@@ -249,12 +265,7 @@ void libxl__stream_write_start(libxl__egc *egc,
 stream->emu_sub_hdr.index = 0;
 }
 
-dc->ao= ao;
-dc->readfd= -1;
 dc->writewhat = "stream header";
-dc->copywhat  = "save v2 stream";
-dc->writefd   = stream->fd;
-dc->maxsz = -1;
 dc->callback  = stream_header_done;
 
 rc = libxl__datacopier_start(dc);
@@ -279,6 +290,7 @@ void libxl__stream_write_start_checkpoint(libxl__egc *egc,
 {
 assert(stream->running);
 assert(!stream->in_checkpoint);
+assert(!stream->back_channel);
 stream->in_checkpoint = true;
 
 write_emulator_xenstore_record(egc, stream);
@@ -590,7 +602,9 @@ static void stream_done(libxl__egc *egc,
 libxl__carefd_close(stream->emu_carefd);
 free(stream->emu_body);
 
-check_all_finished(egc, stream, rc);
+if (!stream->back_channel)
+/* back channel stream doesn't have save helper */
+check_all_finished(egc, stream, rc);
 }
 
 static void checkpoint_done(libxl__egc *egc,
-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v10 12/31] tools/libxl: add back channel support to read stream

[Wen Congyang]

This is used by primay to read records sent by secondary.

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
---
 tools/libxl/libxl_create.c  |  1 +
 tools/libxl/libxl_internal.h|  1 +
 tools/libxl/libxl_stream_read.c | 27 +++
 3 files changed, 25 insertions(+), 4 deletions(-)

diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 525bf85..fc746fb 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1030,6 +1030,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
 dcs->srs.dcs = dcs;
 dcs->srs.fd = restore_fd;
 dcs->srs.legacy = (dcs->restore_params.stream_version == 1);
+dcs->srs.back_channel = false;
 dcs->srs.completion_callback = domcreate_stream_done;
 
 if (restore_fd >= 0) {
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index cdf6751..32957ca 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3420,6 +3420,7 @@ struct libxl__stream_read_state {
 libxl__domain_create_state *dcs;
 int fd;
 bool legacy;
+bool back_channel;
 void (*completion_callback)(libxl__egc *egc,
 libxl__stream_read_state *srs,
 int rc);
diff --git a/tools/libxl/libxl_stream_read.c b/tools/libxl/libxl_stream_read.c
index f4781eb..02a2c46 100644
--- a/tools/libxl/libxl_stream_read.c
+++ b/tools/libxl/libxl_stream_read.c
@@ -118,6 +118,15 @@
  *record, and therefore the buffered state is inconsistent. In
  *libxl__xc_domain_restore_done(), we just complete the stream and
  *stream->completion_callback() will be called to resume the guest
+ *
+ * For back channel stream:
+ * - libxl__stream_read_start()
+ *- Set up the stream to running state
+ *
+ * - libxl__stream_read_continue()
+ * - Set up reading the next record from a started stream.
+ *   Add some codes to process_record() to handle the record.
+ *   Then call stream->checkpoint_callback() to return.
  */
 
 /* Success/error/cleanup handling. */
@@ -221,6 +230,17 @@ void libxl__stream_read_start(libxl__egc *egc,
 stream->running = true;
 stream->phase   = SRS_PHASE_NORMAL;
 
+dc->ao   = stream->ao;
+dc->copywhat = "restore v2 stream";
+dc->writefd  = -1;
+
+if (stream->back_channel) {
+assert(!stream->legacy);
+
+dc->readfd = stream->fd;
+return;
+}
+
 if (stream->legacy) {
 /* Convert the legacy stream. */
 libxl__conversion_helper_state *chs = >chs;
@@ -243,10 +263,7 @@ void libxl__stream_read_start(libxl__egc *egc,
 }
 /* stream->fd is now a v2 stream. */
 
-dc->ao   = stream->ao;
-dc->copywhat = "restore v2 stream";
 dc->readfd   = stream->fd;
-dc->writefd  = -1;
 
 /* Start reading the stream header. */
 rc = setup_read(stream, "stream header",
@@ -762,7 +779,9 @@ static void stream_done(libxl__egc *egc,
 LIBXL_STAILQ_FOREACH_SAFE(rec, >record_queue, entry, trec)
 free_record(rec);
 
-check_all_finished(egc, stream, rc);
+if (!stream->back_channel)
+/* back channel stream doesn't have restore helper */
+check_all_finished(egc, stream, rc);
 }
 
 void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v10 25/31] COLO: use qemu block replication

[Wen Congyang]

Use qemu block replication as our block replication solution.
Note that guest must be paused before starting COLO, otherwise,
the disk won't be consistent between primary and secondary.

Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
---
 tools/libxl/Makefile |   1 +
 tools/libxl/libxl_colo_qdisk.c   | 226 +++
 tools/libxl/libxl_colo_restore.c |  42 +++-
 tools/libxl/libxl_colo_save.c|  54 +-
 tools/libxl/libxl_internal.h |  13 +++
 5 files changed, 331 insertions(+), 5 deletions(-)
 create mode 100644 tools/libxl/libxl_colo_qdisk.c

diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 2016393..28d54d0 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -66,6 +66,7 @@ endif
 
 LIBXL_OBJS-y += libxl_remus.o libxl_checkpoint_device.o libxl_remus_disk_drbd.o
 LIBXL_OBJS-y += libxl_colo_restore.o libxl_colo_save.o
+LIBXL_OBJS-y += libxl_colo_qdisk.o
 
 LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o libxl_psr.o
 LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_arm.o libxl_libfdt_compat.o
diff --git a/tools/libxl/libxl_colo_qdisk.c b/tools/libxl/libxl_colo_qdisk.c
new file mode 100644
index 000..6179947
--- /dev/null
+++ b/tools/libxl/libxl_colo_qdisk.c
@@ -0,0 +1,226 @@
+/*
+ * Copyright (C) 2015 FUJITSU LIMITED
+ * Author: Wen Congyang <we...@cn.fujitsu.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+
+typedef struct libxl__colo_qdisk {
+bool setuped;
+} libxl__colo_qdisk;
+
+/* == init() and cleanup() == */
+int init_subkind_qdisk(libxl__checkpoint_devices_state *cds)
+{
+/*
+ * We don't know if we use qemu block replication, so
+ * we cannot start block replication here.
+ */
+return 0;
+}
+
+void cleanup_subkind_qdisk(libxl__checkpoint_devices_state *cds)
+{
+}
+
+/* == setup() and teardown() == */
+static void colo_qdisk_setup(libxl__egc *egc, libxl__checkpoint_device *dev,
+ bool primary)
+{
+const libxl_device_disk *disk = dev->backend_dev;
+int ret, rc = 0;
+libxl__colo_qdisk *colo_qdisk = NULL;
+
+/* Convenience aliases */
+libxl__checkpoint_devices_state *const cds = dev->cds;
+const char *host = disk->colo_host;
+const char *port = disk->colo_port;
+const char *export_name = disk->colo_export;
+const int domid = cds->domid;
+
+STATE_AO_GC(dev->cds->ao);
+
+if (disk->backend != LIBXL_DISK_BACKEND_QDISK ||
+!libxl_defbool_val(disk->colo_enable)) {
+rc = ERROR_CHECKPOINT_DEVOPS_DOES_NOT_MATCH;
+goto out;
+}
+
+dev->matched = true;
+
+GCNEW(colo_qdisk);
+dev->concrete_data = colo_qdisk;
+
+if (primary) {
+libxl__colo_save_state *css = cds->concrete_data;
+
+css->qdisk_used = true;
+/* NBD server is not ready, so we cannot start block replication now */
+goto out;
+} else {
+libxl__colo_restore_state *crs = cds->concrete_data;
+
+if (!crs->qdisk_used) {
+/* start nbd server */
+ret = libxl__qmp_nbd_server_start(gc, domid, host, port);
+if (ret) {
+rc = ERROR_FAIL;
+goto out;
+}
+crs->host = host;
+crs->port = port;
+} else {
+if (strcmp(crs->host, host) || strcmp(crs->port, port)) {
+LOG(ERROR, "The host and port of all disks must be the same");
+rc = ERROR_FAIL;
+goto out;
+}
+}
+
+crs->qdisk_used = true;
+
+ret = libxl__qmp_nbd_server_add(gc, domid, export_name);
+if (ret)
+rc = ERROR_FAIL;
+
+colo_qdisk->setuped = true;
+}
+
+out:
+dev->aodev.rc = rc;
+dev->aodev.callback(egc, >aodev);
+}
+
+static void colo_qdisk_teardown(libxl__egc *egc, libxl__checkpoint_device *dev,
+bool primary)
+{
+int ret, rc = 0;
+const libxl__colo_qdisk *colo_qdisk = dev->concrete_data;
+const libxl_device_disk *disk = dev->backend_dev;
+
+/* Convenience aliases */
+libxl__checkpoint_devices_state *const cds =

[Xen-devel] [PATCH v10 28/31] COLO nic: implement COLO nic subkind

[Wen Congyang]

implement COLO nic subkind.

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
---
 tools/hotplug/Linux/Makefile |   1 +
 tools/hotplug/Linux/colo-proxy-setup | 135 +++
 tools/libxl/Makefile |   1 +
 tools/libxl/libxl_colo_nic.c | 321 +++
 tools/libxl/libxl_internal.h |   5 +
 tools/libxl/libxl_types.idl  |   1 +
 6 files changed, 464 insertions(+)
 create mode 100755 tools/hotplug/Linux/colo-proxy-setup
 create mode 100644 tools/libxl/libxl_colo_nic.c

diff --git a/tools/hotplug/Linux/Makefile b/tools/hotplug/Linux/Makefile
index 6e10118..9bb852b 100644
--- a/tools/hotplug/Linux/Makefile
+++ b/tools/hotplug/Linux/Makefile
@@ -28,6 +28,7 @@ XEN_SCRIPTS += block-iscsi
 XEN_SCRIPTS += block-tap
 XEN_SCRIPTS += block-drbd-probe
 XEN_SCRIPTS += $(XEN_SCRIPTS-y)
+XEN_SCRIPTS += colo-proxy-setup
 
 SUBDIRS-$(CONFIG_SYSTEMD) += systemd
 
diff --git a/tools/hotplug/Linux/colo-proxy-setup 
b/tools/hotplug/Linux/colo-proxy-setup
new file mode 100755
index 000..94e2034
--- /dev/null
+++ b/tools/hotplug/Linux/colo-proxy-setup
@@ -0,0 +1,135 @@
+#! /bin/bash
+
+dir=$(dirname "$0")
+. "$dir/xen-hotplug-common.sh"
+. "$dir/hotplugpath.sh"
+
+findCommand "$@"
+
+if [ "$command" != "setup" -a  "$command" != "teardown" ]
+then
+echo "Invalid command: $command"
+log err "Invalid command: $command"
+exit 1
+fi
+
+evalVariables "$@"
+
+: ${vifname:?}
+: ${forwarddev:?}
+: ${mode:?}
+: ${index:?}
+: ${bridge:?}
+
+forwardbr="colobr0"
+
+if [ "$mode" != "primary" -a "$mode" != "secondary" ]
+then
+echo "Invalid mode: $mode"
+log err "Invalid mode: $mode"
+exit 1
+fi
+
+if [ $index -lt 0 ] || [ $index -gt 100 ]; then
+echo "index overflow"
+exit 1
+fi
+
+function setup_primary()
+{
+do_without_error tc qdisc add dev $vifname root handle 1: prio
+do_without_error tc filter add dev $vifname parent 1: protocol ip prio 10 \
+u32 match u32 0 0 flowid 1:2 action mirred egress mirror dev 
$forwarddev
+do_without_error tc filter add dev $vifname parent 1: protocol arp prio 11 
\
+u32 match u32 0 0 flowid 1:2 action mirred egress mirror dev 
$forwarddev
+do_without_error tc filter add dev $vifname parent 1: protocol ipv6 prio \
+12 u32 match u32 0 0 flowid 1:2 action mirred egress mirror \
+dev $forwarddev
+
+do_without_error modprobe nf_conntrack_ipv4
+do_without_error modprobe xt_PMYCOLO sec_dev=$forwarddev
+
+iptables -t mangle -I PREROUTING -m physdev --physdev-in \
+$vifname -j PMYCOLO --index $index
+ip6tables -t mangle -I PREROUTING -m physdev --physdev-in \
+$vifname -j PMYCOLO --index $index
+do_without_error arptables -I INPUT -i $forwarddev -j MARK --set-mark 
$index
+}
+
+function teardown_primary()
+{
+do_without_error tc filter del dev $vifname parent 1: protocol ip prio 10 
u32 match u32 \
+0 0 flowid 1:2 action mirred egress mirror dev $forwarddev
+do_without_error tc filter del dev $vifname parent 1: protocol arp prio 11 
u32 match u32 \
+0 0 flowid 1:2 action mirred egress mirror dev $forwarddev
+do_without_error tc filter del dev $vifname parent 1: protocol ipv6 prio 
12 u32 match u32 \
+0 0 flowid 1:2 action mirred egress mirror dev $forwarddev
+do_without_error tc qdisc del dev $vifname root handle 1: prio
+
+do_without_error iptables -t mangle -D PREROUTING -m physdev --physdev-in \
+$vifname -j PMYCOLO --index $index
+do_without_error ip6tables -t mangle -D PREROUTING -m physdev --physdev-in 
\
+$vifname -j PMYCOLO --index $index
+do_without_error arptables -F
+do_without_error rmmod xt_PMYCOLO
+}
+
+function setup_secondary()
+{
+do_without_error brctl delif $bridge $vifname
+do_without_error brctl addbr $forwardbr
+do_without_error brctl addif $forwardbr $vifname
+do_without_error brctl addif $forwardbr $forwarddev
+do_without_error ip link set dev $forwardbr up
+do_without_error modprobe xt_SECCOLO
+
+iptables -t mangle -I PREROUTING -m physdev --physdev-in \
+$vifname -j SECCOLO --index $index
+ip6tables -t mangle -I PREROUTING -m physdev --physdev-in \
+$vifname -j SECCOLO --index $index
+}
+
+function teardown_secondary()
+{
+do_without_error brctl delif $forwardbr $forwarddev
+do_without_error brctl delif $forwardbr $vifname
+do_without_error brctl delbr $forwardbr
+do_without_error brctl addif $bridge $vifname
+
+do_without_error iptables -t mangle -D PREROUTING -m physdev --physdev-in \
+$vifname -j SECCOLO --index $index
+do_without_error ip6tables -

[Xen-devel] [PATCH v10 09/31] libxc/migration: export read_record for common use

[Wen Congyang]

read_record() could be used by primary to read dirty bitmap
record sent by secondary under COLO.
When used by save side, we need to pass the backchannel fd
instead of ctx->fd to read_record(), so we added a fd param to
it.
No functional changes.

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
CC: Andrew Cooper <andrew.coop...@citrix.com>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
---
 tools/libxc/xc_sr_common.c  | 49 +++
 tools/libxc/xc_sr_common.h  | 14 ++
 tools/libxc/xc_sr_restore.c | 63 +
 3 files changed, 64 insertions(+), 62 deletions(-)

diff --git a/tools/libxc/xc_sr_common.c b/tools/libxc/xc_sr_common.c
index 3313a90..b228a15 100644
--- a/tools/libxc/xc_sr_common.c
+++ b/tools/libxc/xc_sr_common.c
@@ -89,6 +89,55 @@ int write_split_record(struct xc_sr_context *ctx, struct 
xc_sr_record *rec,
 return -1;
 }
 
+int read_record(struct xc_sr_context *ctx, int fd, struct xc_sr_record *rec)
+{
+xc_interface *xch = ctx->xch;
+struct xc_sr_rhdr rhdr;
+size_t datasz;
+
+if ( read_exact(fd, , sizeof(rhdr)) )
+{
+PERROR("Failed to read Record Header from stream");
+return -1;
+}
+else if ( rhdr.length > REC_LENGTH_MAX )
+{
+ERROR("Record (0x%08x, %s) length %#x exceeds max (%#x)", rhdr.type,
+  rec_type_to_str(rhdr.type), rhdr.length, REC_LENGTH_MAX);
+return -1;
+}
+
+datasz = ROUNDUP(rhdr.length, REC_ALIGN_ORDER);
+
+if ( datasz )
+{
+rec->data = malloc(datasz);
+
+if ( !rec->data )
+{
+ERROR("Unable to allocate %zu bytes for record data (0x%08x, %s)",
+  datasz, rhdr.type, rec_type_to_str(rhdr.type));
+return -1;
+}
+
+if ( read_exact(fd, rec->data, datasz) )
+{
+free(rec->data);
+rec->data = NULL;
+PERROR("Failed to read %zu bytes of data for record (0x%08x, %s)",
+   datasz, rhdr.type, rec_type_to_str(rhdr.type));
+return -1;
+}
+}
+else
+rec->data = NULL;
+
+rec->type   = rhdr.type;
+rec->length = rhdr.length;
+
+return 0;
+};
+
 static void __attribute__((unused)) build_assertions(void)
 {
 XC_BUILD_BUG_ON(sizeof(struct xc_sr_ihdr) != 24);
diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 66f595f..5d9f497 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -382,6 +382,20 @@ static inline int write_record(struct xc_sr_context *ctx,
 }
 
 /*
+ * Reads a record from the stream, and fills in the record structure.
+ *
+ * Returns 0 on success and non-0 on failure.
+ *
+ * On success, the records type and size shall be valid.
+ * - If size is 0, data shall be NULL.
+ * - If size is non-0, data shall be a buffer allocated by malloc() which must
+ *   be passed to free() by the caller.
+ *
+ * On failure, the contents of the record structure are undefined.
+ */
+int read_record(struct xc_sr_context *ctx, int fd, struct xc_sr_record *rec);
+
+/*
  * This would ideally be private in restore.c, but is needed by
  * x86_pv_localise_page() if we receive pagetables frames ahead of the
  * contents of the frames they point at.
diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
index e24560e..aef9bca 100644
--- a/tools/libxc/xc_sr_restore.c
+++ b/tools/libxc/xc_sr_restore.c
@@ -69,67 +69,6 @@ static int read_headers(struct xc_sr_context *ctx)
 }
 
 /*
- * Reads a record from the stream, and fills in the record structure.
- *
- * Returns 0 on success and non-0 on failure.
- *
- * On success, the records type and size shall be valid.
- * - If size is 0, data shall be NULL.
- * - If size is non-0, data shall be a buffer allocated by malloc() which must
- *   be passed to free() by the caller.
- *
- * On failure, the contents of the record structure are undefined.
- */
-static int read_record(struct xc_sr_context *ctx, struct xc_sr_record *rec)
-{
-xc_interface *xch = ctx->xch;
-struct xc_sr_rhdr rhdr;
-size_t datasz;
-
-if ( read_exact(ctx->fd, , sizeof(rhdr)) )
-{
-PERROR("Failed to read Record Header from stream");
-return -1;
-}
-else if ( rhdr.length > REC_LENGTH_MAX )
-{
-ERROR("Record (0x%08x, %s) length %#x exceeds max (%#x)", rhdr.type,
-  rec_type_to_str(rhdr.type), rhdr.length, REC_LENGTH_MAX);
-return -1;
-}
-
-datasz = ROUNDUP(rhdr.length, REC_ALIGN_ORDER);
-
-if ( datasz )
-{
-rec->data = malloc(datasz);
-
-if ( !rec->data )
-{
-ERROR("Unable to allocate %zu bytes for record data (0x%08x, %s)",
-  datasz, rhdr.type, rec_type_to_str(rhdr.type));
-return -1;
- 

[Xen-devel] [PATCH v10 24/31] Support colo mode for qemu disk

[Wen Congyang]

Usage: disk = 
['...,colo,colo-host=xxx,colo-port=xxx,colo-export=xxx,active-disk=xxx,hidden-disk=xxx...']
For QEMU block replication details:
http://wiki.qemu.org/Features/BlockReplication

Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
---
 docs/man/xl.pod.1   |   2 +-
 docs/misc/xl-disk-configuration.txt |  50 ++
 tools/libxl/libxl.c |  62 +++-
 tools/libxl/libxl_create.c  |  25 -
 tools/libxl/libxl_device.c  |  54 +++
 tools/libxl/libxl_dm.c  | 184 ++--
 tools/libxl/libxl_types.idl |   7 ++
 tools/libxl/libxlu_disk_l.l |   7 ++
 8 files changed, 382 insertions(+), 9 deletions(-)

diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
index 1c6dd87..4f1901d 100644
--- a/docs/man/xl.pod.1
+++ b/docs/man/xl.pod.1
@@ -454,7 +454,7 @@ N.B: Remus support in xl is still in experimental 
(proof-of-concept) phase.
  Disk replication support is limited to DRBD disks.
 
  COLO support in xl is still in experimental (proof-of-concept) phase.
- There is no support for network or disk at the moment.
+ There is no support for network at the moment.
 
 B
 
diff --git a/docs/misc/xl-disk-configuration.txt 
b/docs/misc/xl-disk-configuration.txt
index 29f6ddb..6f23c2d 100644
--- a/docs/misc/xl-disk-configuration.txt
+++ b/docs/misc/xl-disk-configuration.txt
@@ -234,6 +234,56 @@ were intentionally created non-sparse to avoid 
fragmentation of the
 file.
 
 
+===
+COLO PARAMETERS
+===
+
+
+colo
+
+
+Enable COLO HA for disk. For better understanding block replication on
+QEMU, please refer to:
+http://wiki.qemu.org/Features/BlockReplication
+
+
+colo-host
+-
+Description:   Secondary host's address
+Mandatory: Yes when COLO enabled
+
+
+colo-port
+-
+Description:   Secondary port
+   We will run a nbd server on secondary host,
+   and the nbd server will listen this port.
+Mandatory: Yes when COLO enabled
+
+
+colo-export
+-
+Description:   We will run a nbd server on secondary host,
+   exportname is the nbd server's disk export name.
+Mandatory: Yes when COLO enabled
+
+
+active-disk
+---
+
+Description:   This is used by secondary. Secondary guest's write
+   will be buffered in this disk.
+Mandatory: Yes when COLO enabled
+
+
+hidden-disk
+---
+
+Description:   This is used by secondary. It buffers the original
+   content that is modified by the primary VM.
+Mandatory: Yes when COLO enabled
+
+
 
 DEPRECATED PARAMETERS, PREFIXES AND SYNTAXES
 
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 12df81a..f691628 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -2309,6 +2309,8 @@ int libxl__device_disk_setdefault(libxl__gc *gc, 
libxl_device_disk *disk)
 int rc;
 
 libxl_defbool_setdefault(>discard_enable, !!disk->readwrite);
+libxl_defbool_setdefault(>colo_enable, false);
+libxl_defbool_setdefault(>colo_restore_enable, false);
 
 rc = libxl__resolve_domid(gc, disk->backend_domname, >backend_domid);
 if (rc < 0) return rc;
@@ -2507,6 +2509,18 @@ static void device_disk_add(libxl__egc *egc, uint32_t 
domid,
 flexarray_append(back, "params");
 flexarray_append(back, GCSPRINTF("%s:%s",
   
libxl__device_disk_string_of_format(disk->format), disk->pdev_path));
+if (libxl_defbool_val(disk->colo_enable)) {
+flexarray_append(back, "colo-host");
+flexarray_append(back, libxl__sprintf(gc, "%s", 
disk->colo_host));
+flexarray_append(back, "colo-port");
+flexarray_append(back, libxl__sprintf(gc, "%s", 
disk->colo_port));
+flexarray_append(back, "colo-export");
+flexarray_append(back, libxl__sprintf(gc, "%s", 
disk->colo_export));
+flexarray_append(back, "active-disk");
+flexarray_append(back, libxl__sprintf(gc, "%s", 
disk->active_disk));
+flexarray_append(back, "hidden-disk");
+flexarray_append(back, libxl__sprintf(gc, "%s", 
disk->hidden_disk));
+}
 assert(device->backend_kind == LIBXL__DEVICE_KIND_QDISK);
 break;
 default:
@@ -2622,7 +2636,10 @@ static int libxl__device_disk_from_xs_be(libxl__gc *gc,
 goto 

[Xen-devel] [PATCH v10 18/31] libxc/restore: support COLO restore

[Wen Congyang]

call the callbacks resume/checkpoint/suspend while secondary vm
status is consistent with primary.

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
CC: Andrew Cooper <andrew.coop...@citrix.com>
---
 tools/libxc/xc_sr_common.h  |  4 +--
 tools/libxc/xc_sr_restore.c | 60 +
 2 files changed, 62 insertions(+), 2 deletions(-)

diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 2bfed64..a24a9ad 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -234,13 +234,13 @@ struct xc_sr_context
 uint32_t guest_page_size;
 
 /* Plain VM, or checkpoints over time. */
-bool checkpointed;
+int checkpointed;
 
 /* Currently buffering records between a checkpoint */
 bool buffer_all_records;
 
 /*
- * With Remus, we buffer the records sent by the primary at checkpoint,
+ * With Remus/COLO, we buffer the records sent by the primary at checkpoint,
  * in case the primary will fail, we can recover from the last
  * checkpoint state.
  * This should be enough for most of the cases because primary only send
diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
index aef9bca..2ae8154 100644
--- a/tools/libxc/xc_sr_restore.c
+++ b/tools/libxc/xc_sr_restore.c
@@ -460,6 +460,49 @@ static int handle_checkpoint(struct xc_sr_context *ctx)
 else
 ctx->restore.buffer_all_records = true;
 
+if ( ctx->restore.checkpointed == MIG_STREAM_COLO )
+{
+#define HANDLE_CALLBACK_RETURN_VALUE(ret)   \
+do {\
+if ( ret == 1 ) \
+rc = 0; /* Success */   \
+else\
+{   \
+if ( ret == 2 ) \
+rc = BROKEN_CHANNEL;\
+else\
+rc = -1; /* Some unspecified error */   \
+goto err;   \
+}   \
+} while (0)
+
+/* COLO */
+
+/* We need to resume guest */
+rc = ctx->restore.ops.stream_complete(ctx);
+if ( rc )
+goto err;
+
+/* TODO: call restore_results */
+
+/* Resume secondary vm */
+ret = ctx->restore.callbacks->postcopy(ctx->restore.callbacks->data);
+HANDLE_CALLBACK_RETURN_VALUE(ret);
+
+/* Wait for a new checkpoint */
+ret = ctx->restore.callbacks->wait_checkpoint(
+ctx->restore.callbacks->data);
+HANDLE_CALLBACK_RETURN_VALUE(ret);
+
+/* suspend secondary vm */
+ret = ctx->restore.callbacks->suspend(ctx->restore.callbacks->data);
+HANDLE_CALLBACK_RETURN_VALUE(ret);
+
+#undef HANDLE_CALLBACK_RETURN_VALUE
+
+/* TODO: send dirty pfn list to primary */
+}
+
  err:
 return rc;
 }
@@ -631,6 +674,15 @@ static int restore(struct xc_sr_context *ctx)
 } while ( rec.type != REC_TYPE_END );
 
  remus_failover:
+
+if ( ctx->restore.checkpointed == MIG_STREAM_COLO )
+{
+/* With COLO, we have already called stream_complete */
+rc = 0;
+IPRINTF("COLO Failover");
+goto done;
+}
+
 /*
  * With Remus, if we reach here, there must be some error on primary,
  * failover from the last checkpoint state.
@@ -685,6 +737,14 @@ int xc_domain_restore(xc_interface *xch, int io_fd, 
uint32_t dom,
 if ( checkpointed_stream )
 assert(callbacks->checkpoint);
 
+if ( ctx.restore.checkpointed == MIG_STREAM_COLO )
+{
+/* this is COLO restore */
+assert(callbacks->suspend &&
+   callbacks->postcopy &&
+   callbacks->wait_checkpoint);
+}
+
 DPRINTF("fd %d, dom %u, hvm %u, pae %u, superpages %d"
 ", checkpointed_stream %d", io_fd, dom, hvm, pae,
 superpages, checkpointed_stream);
-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v10 23/31] COLO: introduce new API to prepare/start/do/get_error/stop replication

[Wen Congyang]

We will use qemu block replication, and qemu provides some qmp commands
to prepare replication, start replication, get replication error, and
stop replication. Introduce new API to execute these qmp commands.

Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
---
 tools/libxl/libxl_internal.h | 20 +
 tools/libxl/libxl_qmp.c  | 96 
 2 files changed, 116 insertions(+)

diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index d23956c..227b1d3 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -1771,6 +1771,26 @@ _hidden int libxl__qmp_set_global_dirty_log(libxl__gc 
*gc, int domid, bool enabl
 _hidden int libxl__qmp_insert_cdrom(libxl__gc *gc, int domid, const 
libxl_device_disk *disk);
 /* Add a virtual CPU */
 _hidden int libxl__qmp_cpu_add(libxl__gc *gc, int domid, int index);
+/* Start NBD server */
+_hidden int libxl__qmp_nbd_server_start(libxl__gc *gc, int domid,
+const char *host, const char *port);
+/* Add a disk to NBD server */
+_hidden int libxl__qmp_nbd_server_add(libxl__gc *gc, int domid, const char 
*disk);
+/* Start replication */
+_hidden int libxl__qmp_start_replication(libxl__gc *gc, int domid, bool 
primary);
+/* Get replication error that occurs when the vm is running */
+_hidden int libxl__qmp_get_replication_error(libxl__gc *gc, int domid);
+/* Do checkpoint */
+_hidden int libxl__qmp_do_checkpoint(libxl__gc *gc, int domid);
+/* Stop replication */
+_hidden int libxl__qmp_stop_replication(libxl__gc *gc, int domid, bool 
primary);
+/* Stop NBD server */
+_hidden int libxl__qmp_nbd_server_stop(libxl__gc *gc, int domid);
+/* Add or remove a child to/from quorum */
+_hidden int libxl__qmp_x_blockdev_change(libxl__gc *gc, int domid, const char 
*parant,
+ const char *child, const char *node);
+/* run a hmp command in qmp mode */
+_hidden int libxl__qmp_hmp(libxl__gc *gc, int domid, const char *command_line);
 /* close and free the QMP handler */
 _hidden void libxl__qmp_close(libxl__qmp_handler *qmp);
 /* remove the socket file, if the file has already been removed,
diff --git a/tools/libxl/libxl_qmp.c b/tools/libxl/libxl_qmp.c
index eec8a44..53860fc 100644
--- a/tools/libxl/libxl_qmp.c
+++ b/tools/libxl/libxl_qmp.c
@@ -978,6 +978,102 @@ int libxl__qmp_cpu_add(libxl__gc *gc, int domid, int idx)
 return qmp_run_command(gc, domid, "cpu-add", args, NULL, NULL);
 }
 
+int libxl__qmp_nbd_server_start(libxl__gc *gc, int domid,
+const char *host, const char *port)
+{
+libxl__json_object *args = NULL;
+libxl__json_object *addr = NULL;
+libxl__json_object *data = NULL;
+
+/* 'addr': {
+ *   'type': 'inet',
+ *   'data': {
+ * 'host': '$nbd_host',
+ * 'port': '$nbd_port'
+ *   }
+ * }
+ */
+qmp_parameters_add_string(gc, , "host", host);
+qmp_parameters_add_string(gc, , "port", port);
+
+qmp_parameters_add_string(gc, , "type", "inet");
+qmp_parameters_common_add(gc, , "data", data);
+
+qmp_parameters_common_add(gc, , "addr", addr);
+
+return qmp_run_command(gc, domid, "nbd-server-start", args, NULL, NULL);
+}
+
+int libxl__qmp_nbd_server_add(libxl__gc *gc, int domid, const char *disk)
+{
+libxl__json_object *args = NULL;
+
+qmp_parameters_add_string(gc, , "device", disk);
+qmp_parameters_add_bool(gc, , "writable", true);
+
+return qmp_run_command(gc, domid, "nbd-server-add", args, NULL, NULL);
+}
+
+int libxl__qmp_start_replication(libxl__gc *gc, int domid, bool primary)
+{
+libxl__json_object *args = NULL;
+
+qmp_parameters_add_bool(gc, , "enable", true);
+qmp_parameters_add_bool(gc, , "primary", primary);
+
+return qmp_run_command(gc, domid, "xen-set-replication", args, NULL, NULL);
+}
+
+int libxl__qmp_get_replication_error(libxl__gc *gc, int domid)
+{
+return qmp_run_command(gc, domid, "xen-get-replication-error", NULL,
+   NULL, NULL);
+}
+
+int libxl__qmp_do_checkpoint(libxl__gc *gc, int domid)
+{
+return qmp_run_command(gc, domid, "xen-do-checkpoint", NULL, NULL, NULL);
+}
+
+int libxl__qmp_stop_replication(libxl__gc *gc, int domid, bool primary)
+{
+libxl__json_object *args = NULL;
+
+qmp_parameters_add_bool(gc, , "enable", false);
+qmp_parameters_add_bool(gc, , "primary", primary);
+
+return qmp_run_command(gc, domid, "xen-set-replication", args, NULL, NULL);
+}
+
+int libxl__qmp_nbd_server_stop(libxl__gc *gc, int domid)
+{
+return qmp_run_command(gc, domid, "nbd-server-stop", NULL, NULL, NULL);
+}
+
+int libxl__qmp_x_blockdev_change(libxl__gc *gc, int domid, const char *parent,
+ c

[Xen-devel] [PATCH v10 13/31] tools/libxl: handle checkpoint_state records in a libxl migration v2 read stream

[Wen Congyang]

Read a checkpoint_state and call stream->checkpoint_callback to handle it.

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
---
 tools/libxl/libxl_internal.h|  3 +++
 tools/libxl/libxl_stream_read.c | 57 -
 2 files changed, 59 insertions(+), 1 deletion(-)

diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 32957ca..33b658d 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3432,6 +3432,7 @@ struct libxl__stream_read_state {
 bool running;
 bool in_checkpoint;
 bool sync_teardown; /* Only used to coordinate shutdown on error path. */
+bool in_checkpoint_state;
 libxl__save_helper_state shs;
 libxl__conversion_helper_state chs;
 
@@ -3459,6 +3460,8 @@ _hidden void libxl__stream_read_start(libxl__egc *egc,
   libxl__stream_read_state *stream);
 _hidden void libxl__stream_read_start_checkpoint(libxl__egc *egc,
  libxl__stream_read_state 
*stream);
+_hidden void libxl__stream_read_checkpoint_state(libxl__egc *egc,
+ libxl__stream_read_state 
*stream);
 _hidden void libxl__stream_read_abort(libxl__egc *egc,
   libxl__stream_read_state *stream, int 
rc);
 static inline bool
diff --git a/tools/libxl/libxl_stream_read.c b/tools/libxl/libxl_stream_read.c
index 02a2c46..5d980d9 100644
--- a/tools/libxl/libxl_stream_read.c
+++ b/tools/libxl/libxl_stream_read.c
@@ -166,6 +166,10 @@ static void write_emulator_done(libxl__egc *egc,
 libxl__datacopier_state *dc,
 int rc, int onwrite, int errnoval);
 
+/* Handlers for checkpoint state mini-loop */
+static void checkpoint_state_done(libxl__egc *egc,
+  libxl__stream_read_state *stream, int rc);
+
 /*- Helpers -*/
 
 /* Helper to set up reading some data from the stream. */
@@ -549,6 +553,7 @@ static bool process_record(libxl__egc *egc,
 STATE_AO_GC(stream->ao);
 libxl__domain_create_state *dcs = stream->dcs;
 libxl__sr_record_buf *rec;
+libxl_sr_checkpoint_state *srcs;
 bool further_action_needed = false;
 int rc = 0;
 
@@ -619,6 +624,17 @@ static bool process_record(libxl__egc *egc,
 checkpoint_done(egc, stream, 0);
 break;
 
+case REC_TYPE_CHECKPOINT_STATE:
+if (!stream->in_checkpoint_state) {
+LOG(ERROR, "Unexpected CHECKPOINT_STATE record in stream");
+rc = ERROR_FAIL;
+goto err;
+}
+
+srcs = rec->body;
+checkpoint_state_done(egc, stream, srcs->id);
+break;
+
 default:
 LOG(ERROR, "Unrecognised record 0x%08x", rec->hdr.type);
 rc = ERROR_FAIL;
@@ -730,6 +746,21 @@ static void stream_complete(libxl__egc *egc,
 return;
 }
 
+if (stream->in_checkpoint_state) {
+assert(rc);
+
+/*
+ * If an error is encountered while in a checkpoint, pass it
+ * back to libxc.  The failure will come back around to us via
+ * 1. normal stream
+ *libxl__xc_domain_restore_done()
+ * 2. back_channel stream
+ *libxl__stream_read_abort()
+ */
+checkpoint_state_done(egc, stream, rc);
+return;
+}
+
 stream_done(egc, stream, rc);
 }
 
@@ -760,6 +791,7 @@ static void stream_done(libxl__egc *egc,
 
 assert(stream->running);
 assert(!stream->in_checkpoint);
+assert(!stream->in_checkpoint_state);
 stream->running = false;
 
 if (stream->incoming_record)
@@ -881,7 +913,30 @@ static void check_all_finished(libxl__egc *egc,
 libxl__conversion_helper_inuse(>chs))
 return;
 
-stream->completion_callback(egc, stream, stream->rc);
+if (stream->completion_callback)
+/* back channel stream doesn't have completion_callback() */
+stream->completion_callback(egc, stream, stream->rc);
+}
+
+/*- Checkpoint state handlers -*/
+
+void libxl__stream_read_checkpoint_state(libxl__egc *egc,
+ libxl__stream_read_state *stream)
+{
+assert(stream->running);
+assert(!stream->in_checkpoint);
+assert(!stream->in_checkpoint_state);
+stream->in_checkpoint_state = true;
+
+setup_read_record(egc, stream);
+}
+
+static void checkpoint_state_done(libxl__egc *egc,
+  libxl__stream_read_state *stream, int rc)
+{
+assert(stream->in_checkpoint_state);
+stream->in_checkpoint_state = false;
+stream->checkpoint_callback(egc, stream, rc);
 }
 
 /*
-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v10 30/31] setup and control colo proxy on secondary side

[Wen Congyang]

setup and control colo proxy on secondary side

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
---
 tools/libxl/libxl_colo_restore.c | 28 +---
 1 file changed, 25 insertions(+), 3 deletions(-)

diff --git a/tools/libxl/libxl_colo_restore.c b/tools/libxl/libxl_colo_restore.c
index eff80ca..c4b1fb4 100644
--- a/tools/libxl/libxl_colo_restore.c
+++ b/tools/libxl/libxl_colo_restore.c
@@ -50,9 +50,11 @@ static void 
libxl__colo_restore_domain_checkpoint_callback(void *data);
 static void libxl__colo_restore_domain_wait_checkpoint_callback(void *data);
 static void libxl__colo_restore_domain_suspend_callback(void *data);
 
+extern const libxl__checkpoint_device_instance_ops colo_restore_device_nic;
 extern const libxl__checkpoint_device_instance_ops colo_restore_device_qdisk;
 
 static const libxl__checkpoint_device_instance_ops *colo_restore_ops[] = {
+_restore_device_nic,
 _restore_device_qdisk,
 NULL,
 };
@@ -153,8 +155,14 @@ static int 
init_device_subkind(libxl__checkpoint_devices_state *cds)
 int rc;
 STATE_AO_GC(cds->ao);
 
+rc = init_subkind_colo_nic(cds);
+if (rc) goto out;
+
 rc = init_subkind_qdisk(cds);
-if (rc)  goto out;
+if (rc) {
+cleanup_subkind_colo_nic(cds);
+goto out;
+}
 
 rc = 0;
 out:
@@ -166,6 +174,7 @@ static void 
cleanup_device_subkind(libxl__checkpoint_devices_state *cds)
 /* cleanup device subkind-specific state in the libxl ctx */
 STATE_AO_GC(cds->ao);
 
+cleanup_subkind_colo_nic(cds);
 cleanup_subkind_qdisk(cds);
 }
 
@@ -347,6 +356,8 @@ static void colo_restore_teardown_devices_done(libxl__egc 
*egc,
 if (crcs->teardown_devices)
 cleanup_device_subkind(cds);
 
+colo_proxy_teardown(>cps);
+
 rc = crcs->saved_rc;
 if (!rc) {
 crcs->callback = do_failover_done;
@@ -605,6 +616,8 @@ static void colo_restore_preresume_cb(libxl__egc *egc,
 }
 }
 
+colo_proxy_preresume(>cps);
+
 colo_restore_resume_vm(egc, crcs);
 
 return;
@@ -641,6 +654,8 @@ static void colo_resume_vm_done(libxl__egc *egc,
 
 crcs->status = LIBXL_COLO_RESUMED;
 
+colo_proxy_postresume(>cps);
+
 /* avoid calling stream->completion_callback() more than once */
 if (crs->saved_cb) {
 dcs->callback = crs->saved_cb;
@@ -762,13 +777,20 @@ static void colo_setup_checkpoint_devices(libxl__egc *egc,
 
 STATE_AO_GC(crs->ao);
 
-/* TODO: nic support */
-cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VBD);
+cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VIF) |
+ (1 << LIBXL__DEVICE_KIND_VBD);
 cds->callback = colo_restore_setup_cds_done;
 cds->ao = ao;
 cds->domid = crs->domid;
 cds->ops = colo_restore_ops;
 
+crs->cps.ao = ao;
+if (colo_proxy_setup(>cps)) {
+LOG(ERROR, "COLO: failed to setup colo proxy for guest with domid %u",
+cds->domid);
+goto out;
+}
+
 if (init_device_subkind(cds))
 goto out;
 
-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v10 05/31] tools/libx{l, c}: add back channel to libxc

[Wen Congyang]

In COLO mode, both VMs are running, and are considered in sync if the
visible network traffic is identical.  After some time, they fall out of
sync.

At this point, the two VMs have definitely diverged.  Lets call the
primary dirty bitmap set A, while the secondary dirty bitmap set B.

Sets A and B are different.

Under normal migration, the page data for set A will be sent from the
primary to the secondary.

However, the set difference B - A (the one in B but not in A, lets
call this C) is out-of-date on the secondary (with respect to the
primary) and will not be sent by the primary (to secondary), as it
was not memory dirtied by the primary. The secondary needs C page data
to reconstruct an exact copy of the primary at the checkpoint.

The secondary cannot calculate C as it doesn't know A.  Instead, the
secondary must send B to the primary, at which point the primary
calculates the union of A and B (lets call this D) which is all the
pages dirtied by both the primary and the secondary, and sends all page
data covered by D.

In the general case, D is a superset of both A and B.  Without the
backchannel dirty bitmap, a COLO checkpoint can't reconstruct a valid
copy of the primary.

We transfer the dirty bitmap on libxc side, so we need to introduce back
channel to libxc.

Note: it is different from the paper. We change the original design to
the current one, according to our following concerns:
1. The original design needs extra memory on Secondary host. When there's
   multiple backups on one host, the memory cost is high.
2. The memory cache code will be another 1k+, it will make the review
   more time consuming.

Note: the back channel will be used in the patch
 libxc/restore: send dirty pfn list to primary when checkpoint under COLO
to send dirty pfn list from secondary to primary. The patch is posted in
another series.

Signed-off-by: Yang Hongyang 
Signed-off-by: Andrew Cooper 
CC: Ian Campbell 
CC: Ian Jackson 
CC: Wei Liu 
Reviewed-by: Konrad Rzeszutek Wilk 
---
 tools/libxc/include/xenguest.h   |  4 ++--
 tools/libxc/xc_nomigrate.c   |  4 ++--
 tools/libxc/xc_sr_restore.c  |  2 +-
 tools/libxc/xc_sr_save.c |  2 +-
 tools/libxl/libxl_save_callout.c | 21 +++--
 tools/libxl/libxl_save_helper.c  |  8 ++--
 6 files changed, 27 insertions(+), 14 deletions(-)

diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index affc42b..8040ac7 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -88,7 +88,7 @@ struct save_callbacks {
 int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t 
max_iters,
uint32_t max_factor, uint32_t flags /* XCFLAGS_xxx */,
struct save_callbacks* callbacks, int hvm,
-   int checkpointed_stream);
+   int checkpointed_stream, int recv_fd);
 
 /* callbacks provided by xc_domain_restore */
 struct restore_callbacks {
@@ -127,7 +127,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, 
uint32_t dom,
   unsigned long *console_mfn, domid_t console_domid,
   unsigned int hvm, unsigned int pae, int superpages,
   int checkpointed_stream,
-  struct restore_callbacks *callbacks);
+  struct restore_callbacks *callbacks, int send_back_fd);
 
 /**
  * This function will create a domain for a paravirtualized Linux
diff --git a/tools/libxc/xc_nomigrate.c b/tools/libxc/xc_nomigrate.c
index c9124df..3634743 100644
--- a/tools/libxc/xc_nomigrate.c
+++ b/tools/libxc/xc_nomigrate.c
@@ -23,7 +23,7 @@
 int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t 
max_iters,
uint32_t max_factor, uint32_t flags,
struct save_callbacks* callbacks, int hvm,
-   int checkpointed_stream)
+   int checkpointed_stream, int recv_fd)
 {
 errno = ENOSYS;
 return -1;
@@ -35,7 +35,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t 
dom,
   unsigned long *console_mfn, domid_t console_domid,
   unsigned int hvm, unsigned int pae, int superpages,
   int checkpointed_stream,
-  struct restore_callbacks *callbacks)
+  struct restore_callbacks *callbacks, int send_back_fd)
 {
 errno = ENOSYS;
 return -1;
diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
index d4d33fd..e24560e 100644
--- a/tools/libxc/xc_sr_restore.c
+++ b/tools/libxc/xc_sr_restore.c
@@ -726,7 +726,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, 
uint32_t dom,
   unsigned long *console_gfn, domid_t console_domid,
   unsigned int hvm, unsigned int pae, 

[Xen-devel] [PATCH v10 04/31] tools/libxl: Introduce new helper function dup_fd_helper()

[Wen Congyang]

It is pure refactoring and no functional changes.

Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
---
 tools/libxl/libxl_save_callout.c | 24 
 1 file changed, 16 insertions(+), 8 deletions(-)

diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index 7f1f5d4..a885504 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -118,6 +118,21 @@ void libxl__save_helper_init(libxl__save_helper_state *shs)
 }
 
 /*- helper execution -*/
+static int dup_fd_helper(libxl__gc *gc, int fd, const char *what)
+{
+int dup_fd = fd;
+
+if (fd <= 2) {
+dup_fd = dup(fd);
+if (dup_fd < 0) {
+LOGE(ERROR,"dup %s", what);
+exit(-1);
+}
+}
+libxl_fd_set_cloexec(CTX, dup_fd, 0);
+
+return dup_fd;
+}
 
 /*
  * Both save and restore share four parameters:
@@ -186,14 +201,7 @@ static void run_helper(libxl__egc *egc, 
libxl__save_helper_state *shs,
 
 pid_t pid = libxl__ev_child_fork(gc, >child, helper_exited);
 if (!pid) {
-if (stream_fd <= 2) {
-stream_fd = dup(stream_fd);
-if (stream_fd < 0) {
-LOGE(ERROR,"dup migration stream fd");
-exit(-1);
-}
-}
-libxl_fd_set_cloexec(CTX, stream_fd, 0);
+stream_fd = dup_fd_helper(gc, stream_fd, "migration stream fd");
 *stream_fd_arg = GCSPRINTF("%d", stream_fd);
 
 for (i=0; i<num_preserve_fds; i++)
-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v10 14/31] tools/libx{l, c}: introduce wait_checkpoint callback

[Wen Congyang]

Under COLO, we are doing checkpoint on demand, if this
callback returns 1, we will take another checkpoint.
0 indicates unexpected error.

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
---
 tools/libxc/include/xenguest.h | 18 ++
 tools/libxl/libxl_save_msgs_gen.pl |  7 ---
 2 files changed, 22 insertions(+), 3 deletions(-)

diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index 8040ac7..f598bec 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -68,6 +68,15 @@ struct save_callbacks {
  * 1: take another checkpoint */
 int (*checkpoint)(void* data);
 
+/*
+ * Called after the checkpoint callback.
+ *
+ * returns:
+ * 0: terminate checkpointing gracefully
+ * 1: take another checkpoint
+ */
+int (*wait_checkpoint)(void* data);
+
 /* Enable qemu-dm logging dirty pages to xen */
 int (*switch_qemu_logdirty)(int domid, unsigned enable, void *data); /* 
HVM only */
 
@@ -99,6 +108,15 @@ struct restore_callbacks {
 #define XGR_CHECKPOINT_FAILOVER 2 /* Failover and resume VM */
 int (*checkpoint)(void* data);
 
+/*
+ * Called after the checkpoint callback.
+ *
+ * returns:
+ * 0: terminate checkpointing gracefully
+ * 1: take another checkpoint
+ */
+int (*wait_checkpoint)(void* data);
+
 /* to be provided as the last argument to each callback function */
 void* data;
 };
diff --git a/tools/libxl/libxl_save_msgs_gen.pl 
b/tools/libxl/libxl_save_msgs_gen.pl
index d6d2967..51d004d 100755
--- a/tools/libxl/libxl_save_msgs_gen.pl
+++ b/tools/libxl/libxl_save_msgs_gen.pl
@@ -26,11 +26,12 @@ our @msgs = (
 [  3, 'scxA',   "suspend", [] ],
 [  4, 'scxA',   "postcopy", [] ],
 [  5, 'srcxA',  "checkpoint", [] ],
-[  6, 'scxA',   "switch_qemu_logdirty",  [qw(int domid
+[  6, 'srcxA',  "wait_checkpoint", [] ],
+[  7, 'scxA',   "switch_qemu_logdirty",  [qw(int domid
   unsigned enable)] ],
-[  7, 'r',  "restore_results",   ['unsigned long', 'store_mfn',
+[  8, 'r',  "restore_results",   ['unsigned long', 'store_mfn',
   'unsigned long', 'console_mfn'] 
],
-[  8, 'srW',"complete",  [qw(int retval
+[  9, 'srW',"complete",  [qw(int retval
  int errnoval)] ],
 );
 
-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v10 21/31] libxc/save: support COLO save

[Wen Congyang]

After suspend primary vm, get dirty bitmap on secondary vm,
and send pages both dirty on primary/secondary to secondary.

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
CC: Andrew Cooper <andrew.coop...@citrix.com>
---
 tools/libxc/xc_sr_common.h |   2 +
 tools/libxc/xc_sr_save.c   | 104 ++---
 2 files changed, 101 insertions(+), 5 deletions(-)

diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 29ab4eb..5301746 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -196,6 +196,8 @@ struct xc_sr_context
 {
 struct /* Save data. */
 {
+int recv_fd;
+
 struct xc_sr_save_ops ops;
 struct save_callbacks *callbacks;
 
diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
index 7393355..89a5eaf 100644
--- a/tools/libxc/xc_sr_save.c
+++ b/tools/libxc/xc_sr_save.c
@@ -517,6 +517,58 @@ static int send_memory_live(struct xc_sr_context *ctx)
 return rc;
 }
 
+static int merge_secondary_dirty_bitmap(struct xc_sr_context *ctx)
+{
+xc_interface *xch = ctx->xch;
+struct xc_sr_record rec;
+uint64_t *pfns = NULL;
+uint64_t pfn;
+unsigned count, i;
+int rc;
+DECLARE_HYPERCALL_BUFFER_SHADOW(unsigned long, dirty_bitmap,
+>save.dirty_bitmap_hbuf);
+
+rc = read_record(ctx, ctx->save.recv_fd, );
+if ( rc )
+goto err;
+
+if ( rec.type != REC_TYPE_CHECKPOINT_DIRTY_PFN_LIST )
+{
+PERROR("Expect dirty bitmap record, but received %u", rec.type );
+rc = -1;
+goto err;
+}
+
+if ( rec.length % sizeof(*pfns) )
+{
+PERROR("Invalid dirty pfn list record length %u", rec.length );
+rc = -1;
+goto err;
+}
+
+count = rec.length / sizeof(*pfns);
+pfns = rec.data;
+
+for ( i = 0; i < count; i++ )
+{
+pfn = pfns[i];
+if (pfn > ctx->save.p2m_size)
+{
+PERROR("Invalid pfn %#lx", pfn );
+rc = -1;
+goto err;
+}
+
+set_bit(pfn, dirty_bitmap);
+}
+
+rc = 0;
+
+ err:
+free(rec.data);
+return rc;
+}
+
 /*
  * Suspend the domain and send dirty memory.
  * This is the last iteration of the live migration and the
@@ -558,6 +610,16 @@ static int suspend_and_send_dirty(struct xc_sr_context 
*ctx)
 
 bitmap_or(dirty_bitmap, ctx->save.deferred_pages, ctx->save.p2m_size);
 
+if ( !ctx->save.live && ctx->save.checkpointed == MIG_STREAM_COLO )
+{
+rc = merge_secondary_dirty_bitmap(ctx);
+if ( rc )
+{
+PERROR("Failed to get secondary vm's dirty pages");
+goto out;
+}
+}
+
 rc = send_dirty_pages(ctx, stats.dirty_count + 
ctx->save.nr_deferred_pages);
 if ( rc )
 goto out;
@@ -791,13 +853,42 @@ static int save(struct xc_sr_context *ctx, uint16_t 
guest_type)
 if ( rc )
 goto err;
 
-rc = ctx->save.callbacks->postcopy(ctx->save.callbacks->data);
-if ( rc <= 0 )
-goto err;
+if ( ctx->save.checkpointed == MIG_STREAM_COLO )
+{
+rc = 
ctx->save.callbacks->checkpoint(ctx->save.callbacks->data);
+if ( !rc )
+{
+rc = -1;
+goto err;
+}
+}
 
-rc = ctx->save.callbacks->checkpoint(ctx->save.callbacks->data);
-if ( rc <= 0 )
+rc = ctx->save.callbacks->postcopy(ctx->save.callbacks->data);
+if ( !rc )
+{
+rc = -1;
 goto err;
+}
+
+if ( ctx->save.checkpointed == MIG_STREAM_COLO )
+{
+rc = ctx->save.callbacks->wait_checkpoint(
+ctx->save.callbacks->data);
+if ( rc <= 0 )
+goto err;
+}
+else if ( ctx->save.checkpointed == MIG_STREAM_REMUS )
+{
+rc = 
ctx->save.callbacks->checkpoint(ctx->save.callbacks->data);
+if ( rc <= 0 )
+goto err;
+}
+else
+{
+ERROR("Unknown checkpointed stream");
+rc = -1;
+goto err;
+}
 }
 } while ( ctx->save.checkpointed != MIG_STREAM_NONE );
 
@@ -843,6 +934,7 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t 
dom,
 ctx.save.live  = !!(flags & XCFLAGS_LIVE);
 ctx.save.debug = !!(flags & XCFLAGS_DEBUG);
 ctx.save.checkpointed = checkpointed_stre

[Xen-devel] [PATCH v10 29/31] setup and control colo proxy on primary side

[Wen Congyang]

setup and control colo proxy on primary side

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
---
 tools/libxl/libxl_colo_save.c | 124 +++---
 tools/libxl/libxl_internal.h  |   1 +
 2 files changed, 117 insertions(+), 8 deletions(-)

diff --git a/tools/libxl/libxl_colo_save.c b/tools/libxl/libxl_colo_save.c
index 5f8456c..ccfbb17 100644
--- a/tools/libxl/libxl_colo_save.c
+++ b/tools/libxl/libxl_colo_save.c
@@ -19,9 +19,11 @@
 #include "libxl_internal.h"
 #include "libxl_colo.h"
 
+extern const libxl__checkpoint_device_instance_ops colo_save_device_nic;
 extern const libxl__checkpoint_device_instance_ops colo_save_device_qdisk;
 
 static const libxl__checkpoint_device_instance_ops *colo_ops[] = {
+_save_device_nic,
 _save_device_qdisk,
 NULL,
 };
@@ -33,9 +35,15 @@ static int 
init_device_subkind(libxl__checkpoint_devices_state *cds)
 int rc;
 STATE_AO_GC(cds->ao);
 
-rc = init_subkind_qdisk(cds);
+rc = init_subkind_colo_nic(cds);
 if (rc) goto out;
 
+rc = init_subkind_qdisk(cds);
+if (rc) {
+cleanup_subkind_colo_nic(cds);
+goto out;
+}
+
 rc = 0;
 out:
 return rc;
@@ -46,6 +54,7 @@ static void 
cleanup_device_subkind(libxl__checkpoint_devices_state *cds)
 /* cleanup device subkind-specific state in the libxl ctx */
 STATE_AO_GC(cds->ao);
 
+cleanup_subkind_colo_nic(cds);
 cleanup_subkind_qdisk(cds);
 }
 
@@ -83,9 +92,16 @@ void libxl__colo_save_setup(libxl__egc *egc, 
libxl__colo_save_state *css)
 css->paused = true;
 css->qdisk_setuped = false;
 css->qdisk_used = false;
+libxl__ev_child_init(>child);
 
-/* TODO: nic support */
-cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VBD);
+if (dss->remus->netbufscript)
+css->colo_proxy_script = libxl__strdup(gc, dss->remus->netbufscript);
+else
+css->colo_proxy_script = GCSPRINTF("%s/colo-proxy-setup",
+   libxl__xen_script_dir_path());
+
+cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VIF) |
+ (1 << LIBXL__DEVICE_KIND_VBD);
 cds->ops = colo_ops;
 cds->callback = colo_save_setup_done;
 cds->ao = ao;
@@ -96,6 +112,12 @@ void libxl__colo_save_setup(libxl__egc *egc, 
libxl__colo_save_state *css)
 css->srs.fd = css->recv_fd;
 css->srs.back_channel = true;
 libxl__stream_read_start(egc, >srs);
+css->cps.ao = ao;
+if (colo_proxy_setup(>cps)) {
+LOG(ERROR, "COLO: failed to setup colo proxy for guest with domid %u",
+cds->domid);
+goto out;
+}
 
 if (init_device_subkind(cds))
 goto out;
@@ -183,6 +205,7 @@ static void colo_teardown_done(libxl__egc *egc,
 libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
 
 cleanup_device_subkind(cds);
+colo_proxy_teardown(>cps);
 dss->callback(egc, dss, rc);
 }
 
@@ -382,6 +405,8 @@ static void colo_read_svm_ready_done(libxl__egc *egc,
 goto out;
 }
 
+colo_proxy_preresume(>cps);
+
 css->svm_running = true;
 dss->cds.callback = colo_preresume_cb;
 libxl__checkpoint_devices_preresume(egc, >cds);
@@ -466,6 +491,8 @@ static void colo_read_svm_resumed_done(libxl__egc *egc,
 goto out;
 }
 
+colo_proxy_postresume(>cps);
+
 ok = 1;
 
 out:
@@ -474,6 +501,91 @@ out:
 
 
 /* = colo: wait new checkpoint = */
+
+static void colo_start_new_checkpoint(libxl__egc *egc,
+  libxl__checkpoint_devices_state *cds,
+  int rc);
+static void colo_proxy_async_wait_for_checkpoint(libxl__colo_save_state *css);
+static void colo_proxy_async_call_done(libxl__egc *egc,
+   libxl__ev_child *child,
+   int pid,
+   int status);
+
+static void colo_proxy_async_call(libxl__egc *egc,
+  libxl__colo_save_state *css,
+  void func(libxl__colo_save_state *),
+  libxl__ev_child_callback callback)
+{
+libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+int pid = -1, rc;
+
+STATE_AO_GC(dss->cds.ao);
+
+/* Fork and call */
+pid = libxl__ev_child_fork(gc, >child, callback);
+if (pid == -1) {
+LOG(ERROR, "unable to fork");
+rc = ERROR_FAIL;
+goto out;
+}
+
+if (!pid) {
+/* child */
+func(css);
+/* notreached */
+abort();
+}
+
+return;
+
+out:
+callback(egc, >child, -1,

[Xen-devel] [PATCH v8 06/13] tools/libxl: introduce enum type libxl_checkpointed_stream

[Wen Congyang]

Introduce enum type libxl_checkpointed_stream in IDL.
rename the last argument of migrate_receive from "remus" to
"checkpointed" since the semantics of this parameter has
changed.

NOTE:
 libxl_domain_restore_params and domain_create aren't changed here,
 checkpointed_stream is still an int. Because we will pass the
 value from libxl to libxc.

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.w...@oracle.com>
Acked-by: Wei Liu <wei.l...@citrix.com>
---
 tools/libxl/libxl.h |  7 +++
 tools/libxl/libxl_create.c  |  8 ++--
 tools/libxl/libxl_stream_read.c |  7 +--
 tools/libxl/libxl_types.idl |  5 +
 tools/libxl/xl_cmdimpl.c| 18 --
 5 files changed, 35 insertions(+), 10 deletions(-)

diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index fa87f53..6225db1 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -876,6 +876,13 @@ void libxl_mac_copy(libxl_ctx *ctx, libxl_mac *dst, 
libxl_mac *src);
  */
 #define LIBXL_HAVE_DEVICE_MODEL_VERSION_NONE 1
 
+/*
+ * LIBXL_HAVE_CHECKPOINTED_STREAM
+ *
+ * If this is defined, then libxl_checkpointed_stream exists.
+ */
+#define LIBXL_HAVE_CHECKPOINTED_STREAM 1
+
 typedef char **libxl_string_list;
 void libxl_string_list_dispose(libxl_string_list *sl);
 int libxl_string_list_length(const libxl_string_list *sl);
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index ad1d50c..f1028bc 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1033,9 +1033,13 @@ static void domcreate_bootloader_done(libxl__egc *egc,
 dcs->srs.completion_callback = domcreate_stream_done;
 
 if (restore_fd >= 0) {
-if (checkpointed_stream)
+switch (checkpointed_stream) {
+case LIBXL_CHECKPOINTED_STREAM_REMUS:
 libxl__remus_restore_setup(egc, dcs);
-libxl__stream_read_start(egc, >srs);
+/* fall through */
+case LIBXL_CHECKPOINTED_STREAM_NONE:
+libxl__stream_read_start(egc, >srs);
+}
 return;
 }
 
diff --git a/tools/libxl/libxl_stream_read.c b/tools/libxl/libxl_stream_read.c
index dac134e..f4781eb 100644
--- a/tools/libxl/libxl_stream_read.c
+++ b/tools/libxl/libxl_stream_read.c
@@ -794,19 +794,22 @@ void libxl__xc_domain_restore_done(libxl__egc *egc, void 
*dcs_void,
  * If the stream is not still alive, we must not continue any work.
  */
 if (libxl__stream_read_inuse(stream)) {
-if (checkpointed_stream) {
+switch (checkpointed_stream) {
+case LIBXL_CHECKPOINTED_STREAM_REMUS:
 /*
  * Failover from primary. Domain state is currently at a
  * consistent checkpoint, complete the stream, and call
  * stream->completion_callback() to resume the guest.
  */
 stream_complete(egc, stream, 0);
-} else {
+break;
+case LIBXL_CHECKPOINTED_STREAM_NONE:
 /*
  * Libxc has indicated that it is done with the stream.
  * Resume reading libxl records from it.
  */
 stream_continue(egc, stream);
+break;
 }
 }
 }
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 9ad7eba..b8fb22f 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -228,6 +228,11 @@ libxl_hdtype = Enumeration("hdtype", [
 (2, "AHCI"),
 ], init_val = "LIBXL_HDTYPE_IDE")
 
+libxl_checkpointed_stream = Enumeration("checkpointed_stream", [
+(0, "NONE"),
+(1, "REMUS"),
+])
+
 #
 # Complex libxl types
 #
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index d07ccb2..6597ebd 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -4426,7 +4426,8 @@ static void migrate_domain(uint32_t domid, const char 
*rune, int debug,
 }
 
 static void migrate_receive(int debug, int daemonize, int monitor,
-int send_fd, int recv_fd, int remus)
+int send_fd, int recv_fd,
+libxl_checkpointed_stream checkpointed)
 {
 uint32_t domid;
 int rc, rc2;
@@ -4451,7 +4452,7 @@ static void migrate_receive(int debug, int daemonize, int 
monitor,
 dom_info.paused = 1;
 dom_info.migrate_fd = recv_fd;
 dom_info.migration_domname_r = _domname;
-dom_info.checkpointed_stream = remus;
+dom_info.checkpointed_stream = checkpointed;
 
 rc = create_domain(_info);
 if (rc < 0) {
@@ -4462,7 +4463,8 @@ static void migrate_receive(int debug, int daemonize, int 
monitor,
 
 domid = rc;
 
-if (remus) {
+switch (checkpointed) {
+case LIBXL_CHECKPOINTED_STREAM_REMUS:
 /* If we are here, it means 

[Xen-devel] [PATCH v8 04/13] libxl/save: Refactor libxl__domain_suspend_state

[Wen Congyang]

Currently struct libxl__domain_suspend_state contains 2 type of states,
one is save state, another is suspend state. This patch separates those
two out.
The motivation of this is that COLO will need to do suspend/resume
continuously, we need a more common suspend state.

After this change, dss stands for libxl__domain_save_state,
dsps stands for libxl__domain_suspend_state.

Also introduce libxl__domain_suspend_init to initialise the
libxl__domain_suspend_state.

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
CC: Ian Campbell <ian.campb...@citrix.com>
CC: Ian Jackson <ian.jack...@eu.citrix.com>
CC: Wei Liu <wei.l...@citrix.com>
CC: Andrew Cooper <andrew.coop...@citrix.com>
Acked-by:Ian Campbell <ian.campb...@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.w...@oracle.com>
Acked-by: Wei Liu <wei.l...@citrix.com>
---
 tools/libxl/libxl.c  |  10 +-
 tools/libxl/libxl_create.c   |  10 +-
 tools/libxl/libxl_dom_save.c |  61 
 tools/libxl/libxl_dom_suspend.c  | 207 ---
 tools/libxl/libxl_internal.h |  61 +++-
 tools/libxl/libxl_netbuffer.c|   2 +-
 tools/libxl/libxl_remus.c|  37 +++
 tools/libxl/libxl_save_callout.c |   2 +-
 tools/libxl/libxl_stream_write.c |  16 +--
 9 files changed, 227 insertions(+), 179 deletions(-)

diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index d6ce7da..db5732c 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -832,7 +832,7 @@ out:
 }
 
 static void remus_failover_cb(libxl__egc *egc,
-  libxl__domain_suspend_state *dss, int rc);
+  libxl__domain_save_state *dss, int rc);
 
 /* TODO: Explicit Checkpoint acknowledgements via recv_fd. */
 int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
@@ -840,7 +840,7 @@ int libxl_domain_remus_start(libxl_ctx *ctx, 
libxl_domain_remus_info *info,
  const libxl_asyncop_how *ao_how)
 {
 AO_CREATE(ctx, domid, ao_how);
-libxl__domain_suspend_state *dss;
+libxl__domain_save_state *dss;
 int rc;
 
 libxl_domain_type type = libxl__domain_type(gc, domid);
@@ -888,7 +888,7 @@ int libxl_domain_remus_start(libxl_ctx *ctx, 
libxl_domain_remus_info *info,
 }
 
 static void remus_failover_cb(libxl__egc *egc,
-  libxl__domain_suspend_state *dss, int rc)
+  libxl__domain_save_state *dss, int rc)
 {
 STATE_AO_GC(dss->ao);
 /*
@@ -900,7 +900,7 @@ static void remus_failover_cb(libxl__egc *egc,
 }
 
 static void domain_suspend_cb(libxl__egc *egc,
-  libxl__domain_suspend_state *dss, int rc)
+  libxl__domain_save_state *dss, int rc)
 {
 STATE_AO_GC(dss->ao);
 int flrc;
@@ -925,7 +925,7 @@ int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, 
int fd, int flags,
 goto out_err;
 }
 
-libxl__domain_suspend_state *dss;
+libxl__domain_save_state *dss;
 GCNEW(dss);
 
 dss->ao = ao;
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index e421d36..ad1d50c 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1558,7 +1558,7 @@ typedef struct {
 typedef struct {
 libxl__app_domain_create_state cdcs;
 libxl__domain_destroy_state dds;
-libxl__domain_suspend_state dss;
+libxl__domain_save_state dss;
 char *toolstack_buf;
 uint32_t toolstack_len;
 } libxl__domain_soft_reset_state;
@@ -1653,7 +1653,7 @@ static int do_domain_soft_reset(libxl_ctx *ctx,
 libxl__app_domain_create_state *cdcs;
 libxl__domain_create_state *dcs;
 libxl__domain_build_state *state;
-libxl__domain_suspend_state *dss;
+libxl__domain_save_state *dss;
 char *dom_path, *xs_store_mfn, *xs_console_mfn;
 uint32_t domid_out;
 int rc;
@@ -1697,8 +1697,8 @@ static int do_domain_soft_reset(libxl_ctx *ctx,
 
 dss->ao = ao;
 dss->domid = domid_soft_reset;
-dss->dm_savefile = GCSPRINTF(LIBXL_DEVICE_MODEL_SAVE_FILE".%d",
- domid_soft_reset);
+dss->dsps.dm_savefile = GCSPRINTF(LIBXL_DEVICE_MODEL_SAVE_FILE".%d",
+  domid_soft_reset);
 
 rc = libxl__save_emulator_xenstore_data(dss, >toolstack_buf,
 >toolstack_len);
@@ -1707,7 +1707,7 @@ static int do_domain_soft_reset(libxl_ctx *ctx,
 goto out;
 }
 
-rc = libxl__domain_suspend_device_model(gc, dss);
+rc = libxl__domain_suspend_device_model(gc, >dsps);
 if (rc) {
 LOG(ERROR, "failed to suspend device model.");
 goto out;
diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
index cca3404..aead042 100644
--- a/tools/li

[Xen-devel] [PATCH v8 12/13] tools/libxl: move remus state into a seperate structure

[Wen Congyang]

Add a new structure remus state, and move concrete layer's private
member to remus state.
it is pure refactoring and no functional changes.
Init interval in libxl__remus_setup(). It is safe to move this initialisation,
because this value is only used for remus, and remus will use this value after
libxl__remus_setup().

Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.w...@oracle.com>
Acked-by: Wei Liu <wei.l...@citrix.com>
---
 tools/libxl/libxl.c |  2 +-
 tools/libxl/libxl_dom_save.c|  3 +--
 tools/libxl/libxl_internal.h| 35 +++---
 tools/libxl/libxl_netbuffer.c   | 49 +
 tools/libxl/libxl_remus.c   | 24 --
 tools/libxl/libxl_remus_disk_drbd.c |  8 +++---
 6 files changed, 72 insertions(+), 49 deletions(-)

diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 58b4574..4cdc169 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -881,7 +881,7 @@ int libxl_domain_remus_start(libxl_ctx *ctx, 
libxl_domain_remus_info *info,
 assert(info);
 
 /* Point of no return */
-libxl__remus_setup(egc, dss);
+libxl__remus_setup(egc, >rs);
 return AO_INPROGRESS;
 
  out:
diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
index 28e2a41..4eb7960 100644
--- a/tools/libxl/libxl_dom_save.c
+++ b/tools/libxl/libxl_dom_save.c
@@ -383,7 +383,6 @@ void libxl__domain_save(libxl__egc *egc, 
libxl__domain_save_state *dss)
 }
 
 if (dss->checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_REMUS) {
-dss->interval = r_info->interval;
 if (libxl_defbool_val(r_info->compression))
 dss->xcflags |= XCFLAGS_CHECKPOINT_COMPRESS;
 }
@@ -433,7 +432,7 @@ static void domain_save_done(libxl__egc *egc,
  * from sending checkpoints. Teardown the network buffers and
  * release netlink resources.  This is an async op.
  */
-libxl__remus_teardown(egc, dss, rc);
+libxl__remus_teardown(egc, >rs, rc);
 return;
 }
 
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 2847d13..a1aae97 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2894,6 +2894,7 @@ struct libxl__checkpoint_devices_state {
 libxl__ao *ao;
 uint32_t domid;
 libxl__checkpoint_callback *callback;
+void *concrete_data;
 int device_kind_flags;
 /* The ops must be pointer array, and the last ops must be NULL. */
 const libxl__checkpoint_device_instance_ops **ops;
@@ -2917,16 +2918,6 @@ struct libxl__checkpoint_devices_state {
 int num_disks;
 
 libxl__multidev multidev;
-
-/*- private for concrete (device-specific) layer only -*/
-
-/* private for nic device subkind ops */
-char *netbufscript;
-struct nl_sock *nlsock;
-struct nl_cache *qdisc_cache;
-
-/* private for drbd disk subkind ops */
-char *drbd_probe_script;
 };
 
 /*
@@ -2974,6 +2965,23 @@ _hidden void 
libxl__checkpoint_devices_preresume(libxl__egc *egc,
 libxl__checkpoint_devices_state *cds);
 _hidden void libxl__checkpoint_devices_commit(libxl__egc *egc,
 libxl__checkpoint_devices_state *cds);
+
+/*- Remus related state structure -*/
+typedef struct libxl__remus_state libxl__remus_state;
+struct libxl__remus_state {
+/* private */
+libxl__ev_time checkpoint_timeout; /* used for Remus checkpoint */
+int interval; /* checkpoint interval */
+
+/*- private for concrete (device-specific) layer only -*/
+/* private for nic device subkind ops */
+char *netbufscript;
+struct nl_sock *nlsock;
+struct nl_cache *qdisc_cache;
+
+/* private for drbd disk subkind ops */
+char *drbd_probe_script;
+};
 _hidden int libxl__netbuffer_enabled(libxl__gc *gc);
 
 /*- Legacy conversion helper -*/
@@ -3127,9 +3135,8 @@ struct libxl__domain_save_state {
 int hvm;
 int xcflags;
 libxl__domain_suspend_state dsps;
+libxl__remus_state rs;
 libxl__checkpoint_devices_state cds;
-libxl__ev_time checkpoint_timeout; /* used for Remus checkpoint */
-int interval; /* checkpoint interval (for Remus) */
 libxl__stream_write_state sws;
 libxl__logdirty_switch logdirty;
 };
@@ -3535,9 +3542,9 @@ _hidden void libxl__domain_suspend_callback(void *data);
 
 /* Remus setup and teardown */
 _hidden void libxl__remus_setup(libxl__egc *egc,
-libxl__domain_save_state *dss);
+libxl__remus_state *rs);
 _hidden void libxl__remus_teardown(libxl__egc *egc,
-   libxl__domain_save_state *dss,
+   libxl__remus_state *rs,
 

[Xen-devel] [PATCH v8 08/13] tools/libxl: export logdirty_init

[Wen Congyang]

We need to enable logdirty on secondary, so we export logdirty_init
for internal use. Rename it to libxl__logdirty_init.

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
Acked-by: Ian Campbell <ian.campb...@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.w...@oracle.com>
Acked-by: Wei Liu <wei.l...@citrix.com>
---
 tools/libxl/libxl_dom_save.c | 4 ++--
 tools/libxl/libxl_internal.h | 2 ++
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
index a385500..28e2a41 100644
--- a/tools/libxl/libxl_dom_save.c
+++ b/tools/libxl/libxl_dom_save.c
@@ -44,7 +44,7 @@ static void switch_logdirty_xswatch(libxl__egc *egc, 
libxl__ev_xswatch*,
 static void switch_logdirty_done(libxl__egc *egc,
  libxl__domain_save_state *dss, int rc);
 
-static void logdirty_init(libxl__logdirty_switch *lds)
+void libxl__logdirty_init(libxl__logdirty_switch *lds)
 {
 lds->cmd_path = 0;
 libxl__ev_xswatch_init(>watch);
@@ -345,7 +345,7 @@ void libxl__domain_save(libxl__egc *egc, 
libxl__domain_save_state *dss)
 }
 
 dss->rc = 0;
-logdirty_init(>logdirty);
+libxl__logdirty_init(>logdirty);
 dsps->ao = ao;
 dsps->domid = domid;
 rc = libxl__domain_suspend_init(egc, dsps, type);
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index ac6457f..656bccd 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3078,6 +3078,8 @@ typedef struct libxl__logdirty_switch {
 libxl__ev_time timeout;
 } libxl__logdirty_switch;
 
+_hidden void libxl__logdirty_init(libxl__logdirty_switch *lds);
+
 struct libxl__domain_suspend_state {
 /* set by caller of libxl__domain_suspend_init */
 libxl__ao *ao;
-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v8 03/13] tools/libxl: move save/restore code into libxl_dom_save.c

[Wen Congyang]

This is purely code motion.

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
CC: Ian Jackson <ian.jack...@eu.citrix.com>
Acked-by: Ian Campbell <ian.campb...@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.w...@oracle.com>
Acked-by: Wei Liu <wei.l...@citrix.com>
---
 tools/libxl/Makefile |   2 +-
 tools/libxl/libxl_dom.c  | 509 
 tools/libxl/libxl_dom_save.c | 538 +++
 3 files changed, 539 insertions(+), 510 deletions(-)
 create mode 100644 tools/libxl/libxl_dom_save.c

diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 7d64ecc..263ea0e 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -105,7 +105,7 @@ LIBXL_OBJS = flexarray.o libxl.o libxl_create.o libxl_dm.o 
libxl_pci.o \
libxl_stream_read.o libxl_stream_write.o \
libxl_save_callout.o _libxl_save_msgs_callout.o \
libxl_qmp.o libxl_event.o libxl_fork.o \
-   libxl_dom_suspend.o $(LIBXL_OBJS-y)
+   libxl_dom_suspend.o libxl_dom_save.o $(LIBXL_OBJS-y)
 LIBXL_OBJS += libxl_genid.o
 LIBXL_OBJS += _libxl_types.o libxl_flask.o _libxl_types_internal.o
 
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index d74f1a4..664adad 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -24,7 +24,6 @@
 #include 
 #include 
 #include 
-#include 
 
 libxl_domain_type libxl__domain_type(libxl__gc *gc, uint32_t domid)
 {
@@ -1107,514 +1106,6 @@ int libxl__qemu_traditional_cmd(libxl__gc *gc, uint32_t 
domid,
 return libxl__xs_printf(gc, XBT_NULL, path, "%s", cmd);
 }
 
-/*
- * Inspect the buffer between start and end, and return a pointer to the
- * character following the NUL terminator of start, or NULL if start is not
- * terminated before end.
- */
-static const char *next_string(const char *start, const char *end)
-{
-if (start >= end) return NULL;
-
-size_t total_len = end - start;
-size_t len = strnlen(start, total_len);
-
-if (len == total_len)
-return NULL;
-else
-return start + len + 1;
-}
-
-int libxl__restore_emulator_xenstore_data(libxl__domain_create_state *dcs,
-  const char *ptr, uint32_t size)
-{
-STATE_AO_GC(dcs->ao);
-const char *next = ptr, *end = ptr + size, *key, *val;
-int rc;
-
-const uint32_t domid = dcs->guest_domid;
-const uint32_t dm_domid = libxl_get_stubdom_id(CTX, domid);
-const char *xs_root = libxl__device_model_xs_path(gc, dm_domid, domid, "");
-
-while (next < end) {
-key = next;
-next = next_string(next, end);
-
-/* Sanitise 'key'. */
-if (!next) {
-rc = ERROR_FAIL;
-LOG(ERROR, "Key in xenstore data not NUL terminated");
-goto out;
-}
-if (key[0] == '\0') {
-rc = ERROR_FAIL;
-LOG(ERROR, "empty key found in xenstore data");
-goto out;
-}
-if (key[0] == '/') {
-rc = ERROR_FAIL;
-LOG(ERROR, "Key in xenstore data not relative");
-goto out;
-}
-
-val = next;
-next = next_string(next, end);
-
-/* Sanitise 'val'. */
-if (!next) {
-rc = ERROR_FAIL;
-LOG(ERROR, "Val in xenstore data not NUL terminated");
-goto out;
-}
-
-libxl__xs_printf(gc, XBT_NULL,
- GCSPRINTF("%s/%s", xs_root, key),
- "%s", val);
-}
-
-rc = 0;
-
- out:
-return rc;
-}
-
-/* Domain suspend (save) */
-
-static void stream_done(libxl__egc *egc,
-libxl__stream_write_state *sws, int rc);
-static void domain_save_done(libxl__egc *egc,
- libxl__domain_suspend_state *dss, int rc);
-
-/*- complicated callback, called by xc_domain_save -*/
-
-/*
- * We implement the other end of protocol for controlling qemu-dm's
- * logdirty.  There is no documentation for this protocol, but our
- * counterparty's implementation is in
- * qemu-xen-traditional.git:xenstore.c in the function
- * xenstore_process_logdirty_event
- */
-
-static void switch_logdirty_timeout(libxl__egc *egc, libxl__ev_time *ev,
-const struct timeval *requested_abs,
-int rc);
-static void switch_logdirty_xswatch(libxl__egc *egc, libxl__ev_xswatch*,
-const char *watch_path, const char *event_path);
-static void switch_logdirty_done(libxl__egc *egc,
- libxl__domain_suspend_state *dss, int rc);
-
-static void logdirty_ini

[Xen-devel] [PATCH v8 02/13] tools/libxl: move remus code into libxl_remus.c

[Wen Congyang]

After previous refactoring, we are now able to move all remus code
into a separate file libxl_remus.c.

Export following functions for internal use:
- setup/teardown Remus:
  * libxl__remus_setup
  * libxl__remus_teardown
  * libxl__remus_restore_setup

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
CC: Ian Campbell <ian.campb...@citrix.com>
CC: Ian Jackson <ian.jack...@eu.citrix.com>
Acked-by:Ian Campbell <ian.campb...@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.w...@oracle.com>
Acked-by: Wei Liu <wei.l...@citrix.com>
---
 tools/libxl/Makefile |   2 +-
 tools/libxl/libxl.c  |  75 -
 tools/libxl/libxl_create.c   |  32 
 tools/libxl/libxl_dom.c  | 223 --
 tools/libxl/libxl_internal.h |  14 +-
 tools/libxl/libxl_remus.c| 362 +++
 6 files changed, 371 insertions(+), 337 deletions(-)
 create mode 100644 tools/libxl/libxl_remus.c

diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 620720e..7d64ecc 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -64,7 +64,7 @@ else
 LIBXL_OBJS-y += libxl_no_convert_callout.o
 endif
 
-LIBXL_OBJS-y += libxl_remus_device.o libxl_remus_disk_drbd.o
+LIBXL_OBJS-y += libxl_remus.o libxl_remus_device.o libxl_remus_disk_drbd.o
 
 LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o libxl_psr.o
 LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_arm.o libxl_libfdt_compat.o
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 38029cd..d6ce7da 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -831,12 +831,6 @@ out:
 return ptr;
 }
 
-static void libxl__remus_setup(libxl__egc *egc,
-   libxl__domain_suspend_state *dss);
-static void remus_setup_done(libxl__egc *egc,
- libxl__remus_devices_state *rds, int rc);
-static void remus_setup_failed(libxl__egc *egc,
-   libxl__remus_devices_state *rds, int rc);
 static void remus_failover_cb(libxl__egc *egc,
   libxl__domain_suspend_state *dss, int rc);
 
@@ -893,75 +887,6 @@ int libxl_domain_remus_start(libxl_ctx *ctx, 
libxl_domain_remus_info *info,
 return AO_CREATE_FAIL(rc);
 }
 
-static void libxl__remus_setup(libxl__egc *egc,
-   libxl__domain_suspend_state *dss)
-{
-/* Convenience aliases */
-libxl__remus_devices_state *const rds = >rds;
-const libxl_domain_remus_info *const info = dss->remus;
-libxl__srm_save_autogen_callbacks *const callbacks =
->sws.shs.callbacks.save.a;
-
-STATE_AO_GC(dss->ao);
-
-if (libxl_defbool_val(info->netbuf)) {
-if (!libxl__netbuffer_enabled(gc)) {
-LOG(ERROR, "Remus: No support for network buffering");
-goto out;
-}
-rds->device_kind_flags |= (1 << LIBXL__DEVICE_KIND_VIF);
-}
-
-if (libxl_defbool_val(info->diskbuf))
-rds->device_kind_flags |= (1 << LIBXL__DEVICE_KIND_VBD);
-
-rds->ao = ao;
-rds->domid = dss->domid;
-rds->callback = remus_setup_done;
-
-dss->sws.checkpoint_callback = remus_checkpoint_stream_written;
-
-callbacks->suspend = libxl__remus_domain_suspend_callback;
-callbacks->postcopy = libxl__remus_domain_resume_callback;
-callbacks->checkpoint = libxl__remus_domain_save_checkpoint_callback;
-
-libxl__remus_devices_setup(egc, rds);
-return;
-
-out:
-dss->callback(egc, dss, ERROR_FAIL);
-}
-
-static void remus_setup_done(libxl__egc *egc,
- libxl__remus_devices_state *rds, int rc)
-{
-libxl__domain_suspend_state *dss = CONTAINER_OF(rds, *dss, rds);
-STATE_AO_GC(dss->ao);
-
-if (!rc) {
-libxl__domain_save(egc, dss);
-return;
-}
-
-LOG(ERROR, "Remus: failed to setup device for guest with domid %u, rc %d",
-dss->domid, rc);
-rds->callback = remus_setup_failed;
-libxl__remus_devices_teardown(egc, rds);
-}
-
-static void remus_setup_failed(libxl__egc *egc,
-   libxl__remus_devices_state *rds, int rc)
-{
-libxl__domain_suspend_state *dss = CONTAINER_OF(rds, *dss, rds);
-STATE_AO_GC(dss->ao);
-
-if (rc)
-LOG(ERROR, "Remus: failed to teardown device after setup failed"
-" for guest with domid %u, rc %d", dss->domid, rc);
-
-dss->callback(egc, dss, rc);
-}
-
 static void remus_failover_cb(libxl__egc *egc,
   libxl__domain_suspend_state *dss, int rc)
 {
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 7293d0b..e421d36 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -709,38 +709,6 @@ static int store_libxl_entry(libxl__gc *gc, uint

[Xen-devel] [PATCH v8 10/13] tools/libxl: adjust the indentation

[Wen Congyang]

This is just tidying up after the "tools/libxl: rename remus device
to checkpoint device" patch automatic renaming.

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
Acked-by: Ian Campbell <ian.campb...@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.w...@oracle.com>
Acked-by: Wei Liu <wei.l...@citrix.com>
---
 tools/libxl/libxl_checkpoint_device.c | 21 +++--
 tools/libxl/libxl_internal.h  | 19 +++
 2 files changed, 22 insertions(+), 18 deletions(-)

diff --git a/tools/libxl/libxl_checkpoint_device.c 
b/tools/libxl/libxl_checkpoint_device.c
index 109cd23..226f159 100644
--- a/tools/libxl/libxl_checkpoint_device.c
+++ b/tools/libxl/libxl_checkpoint_device.c
@@ -73,9 +73,9 @@ static void devices_teardown_cb(libxl__egc *egc,
 /* checkpoint device setup and teardown */
 
 static libxl__checkpoint_device* checkpoint_device_init(libxl__egc *egc,
-  libxl__checkpoint_devices_state 
*cds,
-  libxl__device_kind kind,
-  void *libxl_dev)
+libxl__checkpoint_devices_state *cds,
+libxl__device_kind kind,
+void *libxl_dev)
 {
 libxl__checkpoint_device *dev = NULL;
 
@@ -89,9 +89,10 @@ static libxl__checkpoint_device* 
checkpoint_device_init(libxl__egc *egc,
 }
 
 static void checkpoint_devices_setup(libxl__egc *egc,
-libxl__checkpoint_devices_state *cds);
+ libxl__checkpoint_devices_state *cds);
 
-void libxl__checkpoint_devices_setup(libxl__egc *egc, 
libxl__checkpoint_devices_state *cds)
+void libxl__checkpoint_devices_setup(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds)
 {
 int i, rc;
 
@@ -137,7 +138,7 @@ out:
 }
 
 static void checkpoint_devices_setup(libxl__egc *egc,
-libxl__checkpoint_devices_state *cds)
+ libxl__checkpoint_devices_state *cds)
 {
 int i, rc;
 
@@ -285,12 +286,12 @@ static void devices_checkpoint_cb(libxl__egc *egc,
 
 /* API implementations */
 
-#define define_checkpoint_api(api)\
-void libxl__checkpoint_devices_##api(libxl__egc *egc,\
-libxl__checkpoint_devices_state *cds)\
+#define define_checkpoint_api(api)  \
+void libxl__checkpoint_devices_##api(libxl__egc *egc,   \
+libxl__checkpoint_devices_state *cds)   \
 {   \
 int i;  \
-libxl__checkpoint_device *dev;   \
+libxl__checkpoint_device *dev;  \
 \
 STATE_AO_GC(cds->ao);   \
 \
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 630f048..bde7a15 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2818,7 +2818,8 @@ typedef struct libxl__save_helper_state {
  * Each device type needs to implement the interfaces specified in
  * the libxl__checkpoint_device_instance_ops if it wishes to support Remus.
  *
- * The high-level control flow through the checkpoint device layer is shown 
below:
+ * The high-level control flow through the checkpoint device layer is shown
+ * below:
  *
  * xl remus
  *  |->  libxl_domain_remus_start
@@ -2879,7 +2880,8 @@ int 
init_subkind_drbd_disk(libxl__checkpoint_devices_state *cds);
 void cleanup_subkind_drbd_disk(libxl__checkpoint_devices_state *cds);
 
 typedef void libxl__checkpoint_callback(libxl__egc *,
-   libxl__checkpoint_devices_state *, int rc);
+libxl__checkpoint_devices_state *,
+int rc);
 
 /*
  * State associated with a checkpoint invocation, including parameters
@@ -2887,7 +2889,7 @@ typedef void libxl__checkpoint_callback(libxl__egc *,
  * save/restore machinery.
  */
 struct libxl__checkpoint_devices_state {
-/* must be set by caller of libxl__checkpoint_device_(setup|teardown) 
*/
+/*-- must be set by caller of libxl__checkpoint_device_(setup|teardown) 
--*/
 
 libxl__ao *ao;
 uint32_t domid;
@@ -2900,7 +2902,8 @@ struct libxl__checkpoint_devices_state {
 /*
  * this array is allocated before setup the checkpoint devices by the
 

[Xen-devel] [PATCH v8 13/13] tools/libxl: seperate device init/cleanup from checkpoint device layer

[Wen Congyang]

we call (init|cleanup)_subkind_nic and (init|cleanup)_subkind_drbd_disk
directly in checkpoint device. Move them to libxl_remus.c, Call them before
calling libxl__checkpoint_devices_setup() or after calling
libxl__checkpoint_devices_teardown().
it is pure refactoring and no functional changes.

Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Acked-by: Ian Campbell <ian.campb...@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.w...@oracle.com>
Acked-by: Wei Liu <wei.l...@citrix.com>
---
 tools/libxl/libxl_checkpoint_device.c | 42 ++-
 tools/libxl/libxl_remus.c | 42 +++
 2 files changed, 44 insertions(+), 40 deletions(-)

diff --git a/tools/libxl/libxl_checkpoint_device.c 
b/tools/libxl/libxl_checkpoint_device.c
index bbc6dc4..0a16dbb 100644
--- a/tools/libxl/libxl_checkpoint_device.c
+++ b/tools/libxl/libxl_checkpoint_device.c
@@ -17,38 +17,6 @@
 
 #include "libxl_internal.h"
 
-/*- helper functions -*/
-
-static int init_device_subkind(libxl__checkpoint_devices_state *cds)
-{
-/* init device subkind-specific state in the libxl ctx */
-int rc;
-STATE_AO_GC(cds->ao);
-
-if (libxl__netbuffer_enabled(gc)) {
-rc = init_subkind_nic(cds);
-if (rc) goto out;
-}
-
-rc = init_subkind_drbd_disk(cds);
-if (rc) goto out;
-
-rc = 0;
-out:
-return rc;
-}
-
-static void cleanup_device_subkind(libxl__checkpoint_devices_state *cds)
-{
-/* cleanup device subkind-specific state in the libxl ctx */
-STATE_AO_GC(cds->ao);
-
-if (libxl__netbuffer_enabled(gc))
-cleanup_subkind_nic(cds);
-
-cleanup_subkind_drbd_disk(cds);
-}
-
 /*- setup() and teardown() -*/
 
 /* callbacks */
@@ -86,14 +54,10 @@ static void checkpoint_devices_setup(libxl__egc *egc,
 void libxl__checkpoint_devices_setup(libxl__egc *egc,
  libxl__checkpoint_devices_state *cds)
 {
-int i, rc;
+int i;
 
 STATE_AO_GC(cds->ao);
 
-rc = init_device_subkind(cds);
-if (rc)
-goto out;
-
 cds->num_devices = 0;
 cds->num_nics = 0;
 cds->num_disks = 0;
@@ -126,7 +90,7 @@ void libxl__checkpoint_devices_setup(libxl__egc *egc,
 return;
 
 out:
-cds->callback(egc, cds, rc);
+cds->callback(egc, cds, 0);
 }
 
 static void checkpoint_devices_setup(libxl__egc *egc,
@@ -263,8 +227,6 @@ static void devices_teardown_cb(libxl__egc *egc,
 cds->disks = NULL;
 cds->num_disks = 0;
 
-cleanup_device_subkind(cds);
-
 cds->callback(egc, cds, rc);
 }
 
diff --git a/tools/libxl/libxl_remus.c b/tools/libxl/libxl_remus.c
index e83cdc9..54ec7de 100644
--- a/tools/libxl/libxl_remus.c
+++ b/tools/libxl/libxl_remus.c
@@ -26,6 +26,38 @@ static const libxl__checkpoint_device_instance_ops 
*remus_ops[] = {
 NULL,
 };
 
+/*- helper functions -*/
+
+static int init_device_subkind(libxl__checkpoint_devices_state *cds)
+{
+/* init device subkind-specific state in the libxl ctx */
+int rc;
+STATE_AO_GC(cds->ao);
+
+if (libxl__netbuffer_enabled(gc)) {
+rc = init_subkind_nic(cds);
+if (rc) goto out;
+}
+
+rc = init_subkind_drbd_disk(cds);
+if (rc) goto out;
+
+rc = 0;
+out:
+return rc;
+}
+
+static void cleanup_device_subkind(libxl__checkpoint_devices_state *cds)
+{
+/* cleanup device subkind-specific state in the libxl ctx */
+STATE_AO_GC(cds->ao);
+
+if (libxl__netbuffer_enabled(gc))
+cleanup_subkind_nic(cds);
+
+cleanup_subkind_drbd_disk(cds);
+}
+
 /* Remus setup and teardown -*/
 
 static void remus_setup_done(libxl__egc *egc,
@@ -68,6 +100,12 @@ void libxl__remus_setup(libxl__egc *egc, libxl__remus_state 
*rs)
 cds->concrete_data = rs;
 rs->interval = info->interval;
 
+if (init_device_subkind(cds)) {
+LOG(ERROR, "Remus: failed to init device subkind for guest %u",
+dss->domid);
+goto out;
+}
+
 dss->sws.checkpoint_callback = remus_checkpoint_stream_written;
 
 callbacks->suspend = libxl__remus_domain_suspend_callback;
@@ -108,6 +146,8 @@ static void remus_setup_failed(libxl__egc *egc,
 LOG(ERROR, "Remus: failed to teardown device after setup failed"
 " for guest with domid %u, rc %d", dss->domid, rc);
 
+cleanup_device_subkind(cds);
+
 dss->callback(egc, dss, rc);
 }
 
@@ -142,6 +182,8 @@ static void remus_teardown_done(libxl__egc *egc,
 LOG(ERROR, "Remus: failed to teardown device for guest with domid %u,"
 " rc %d", dss->domid, rc);
 
+cleanup_device_subkind(cds);
+
 dss->callback(egc, dss, rc);
 }
 
-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v8 00/13] Prerequisite patches for COLO

[Wen Congyang]

This patchset is Prerequisite for COLO feature. Refer to:
http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping

Patch status:
1. Acked patches: patch 2-4, 6-13
2. Reviewd patches: all
3. New patches: none
Note:
1. Patch 1 and 7 is updated according to Wei Liu's comments
2. Patch 2-3 is updated because patch 1 is updated
3. Patch 8, 9, 11, 12 in v7 is moved to another series
4. Patch 13, 14 in v7 is fold into one patch(patch 9)
5. The commit message for patch 5 is not updated(wait the reply
   from Ian C, and Ian J)

You can get the codes from here:
https://github.com/wencongyang/xen/tree/colo_pre_v8
You can get the whole colo related patches from here:
https://github.com/wencongyang/xen/tree/colo_v10

v6->v7:
 - Addressed comments from Konrad Rzeszutek Wilk

v5->v6:
 - Fix some bugs found in the test

v4->v5:
 - Rebased to the latest xen
 - Addressed comments from last round

v3->v4:
 - Rebased to the latest migration v2 branch
 - Addressed comments from last round

v2->v3:
 - Merge '[PATCH v2 0/6] Misc cleanups for libxl' into this patchset
   for easy review
 - Addressed review comments
 - Add back channel to libxc
 - Introduce should_checkpoint callback
 - Introduce DIRTY_BITMAP record on libxc side
 - Introduce COLO_CONTEXT record on libxl side
 - Ported to Libxl migration v2

v1->v2:
 - Rebased to [PATCH v2 0/6] Misc cleanups for libxl
 - Add a bugfix for the error handling of process_record

Wen Congyang (13):
  libxl/remus: init checkpoint callback in Remus setup callback
  tools/libxl: move remus code into libxl_remus.c
  tools/libxl: move save/restore code into libxl_dom_save.c
  libxl/save: Refactor libxl__domain_suspend_state
  tools/libxc: support to resume uncooperative HVM guests
  tools/libxl: introduce enum type libxl_checkpointed_stream
  migration/save: pass checkpointed_stream from libxl to libxc
  tools/libxl: export logdirty_init
  tools/libxl: rename remus device to checkpoint device
  tools/libxl: adjust the indentation
  tools/libxl: store remus_ops in checkpoint device state
  tools/libxl: move remus state into a seperate structure
  tools/libxl: seperate device init/cleanup from checkpoint device layer

 tools/libxc/include/xenguest.h|   6 +-
 tools/libxc/xc_nomigrate.c|   3 +-
 tools/libxc/xc_resume.c   |  25 +-
 tools/libxc/xc_sr_common.h|  12 +-
 tools/libxc/xc_sr_save.c  |  17 +-
 tools/libxl/Makefile  |   4 +-
 tools/libxl/libxl.c   |  81 +---
 tools/libxl/libxl.h   |  19 +
 tools/libxl/libxl_checkpoint_device.c | 282 +
 tools/libxl/libxl_create.c|  44 +-
 tools/libxl/libxl_dom.c   | 740 --
 tools/libxl/libxl_dom_save.c  | 521 
 tools/libxl/libxl_dom_suspend.c   | 207 ++
 tools/libxl/libxl_internal.h  | 217 ++
 tools/libxl/libxl_netbuffer.c | 117 +++---
 tools/libxl/libxl_nonetbuffer.c   |  10 +-
 tools/libxl/libxl_remus.c | 424 +++
 tools/libxl/libxl_remus_device.c  | 327 ---
 tools/libxl/libxl_remus_disk_drbd.c   |  56 +--
 tools/libxl/libxl_save_callout.c  |   4 +-
 tools/libxl/libxl_save_helper.c   |   3 +-
 tools/libxl/libxl_stream_read.c   |   7 +-
 tools/libxl/libxl_stream_write.c  |  18 +-
 tools/libxl/libxl_types.idl   |  10 +-
 tools/libxl/xl_cmdimpl.c  |  18 +-
 25 files changed, 1709 insertions(+), 1463 deletions(-)
 create mode 100644 tools/libxl/libxl_checkpoint_device.c
 create mode 100644 tools/libxl/libxl_dom_save.c
 create mode 100644 tools/libxl/libxl_remus.c
 delete mode 100644 tools/libxl/libxl_remus_device.c

-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v8 01/13] libxl/remus: init checkpoint callback in Remus setup callback

[Wen Congyang]

Init stream {read/write} state checkpoint_callback, suspend/resume/checkpoint
callback in Remus setup callback.
There's no functional change, it's just refactoring so that we can move
all remus code into one file.

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
CC: Andrew Cooper <andrew.coop...@citrix.com>
CC: Ian Campbell <ian.campb...@citrix.com>
CC: Ian Jackson <ian.jack...@eu.citrix.com>
CC: Wei Liu <wei.l...@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.w...@oracle.com>
---
 tools/libxl/libxl.c  |  8 
 tools/libxl/libxl_create.c   | 18 ++
 tools/libxl/libxl_dom.c  | 18 +-
 tools/libxl/libxl_internal.h |  7 +++
 4 files changed, 34 insertions(+), 17 deletions(-)

diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 2d18b8d..38029cd 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -899,6 +899,8 @@ static void libxl__remus_setup(libxl__egc *egc,
 /* Convenience aliases */
 libxl__remus_devices_state *const rds = >rds;
 const libxl_domain_remus_info *const info = dss->remus;
+libxl__srm_save_autogen_callbacks *const callbacks =
+>sws.shs.callbacks.save.a;
 
 STATE_AO_GC(dss->ao);
 
@@ -917,6 +919,12 @@ static void libxl__remus_setup(libxl__egc *egc,
 rds->domid = dss->domid;
 rds->callback = remus_setup_done;
 
+dss->sws.checkpoint_callback = remus_checkpoint_stream_written;
+
+callbacks->suspend = libxl__remus_domain_suspend_callback;
+callbacks->postcopy = libxl__remus_domain_resume_callback;
+callbacks->checkpoint = libxl__remus_domain_save_checkpoint_callback;
+
 libxl__remus_devices_setup(egc, rds);
 return;
 
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index de5d27f..7293d0b 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -730,6 +730,17 @@ static void remus_checkpoint_stream_done(
 libxl__xc_domain_saverestore_async_callback_done(egc, >shs, rc);
 }
 
+static void libxl__remus_restore_setup(libxl__egc *egc,
+   libxl__domain_create_state *dcs)
+{
+/* Convenience aliases */
+libxl__srm_restore_autogen_callbacks *const callbacks =
+>srs.shs.callbacks.restore.a;
+
+callbacks->checkpoint = libxl__remus_domain_restore_checkpoint_callback;
+dcs->srs.checkpoint_callback = remus_checkpoint_stream_done;
+}
+
 /*- main domain creation -*/
 
 /* We have a linear control flow; only one event callback is
@@ -1014,8 +1025,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
 libxl_domain_config *const d_config = dcs->guest_config;
 const int restore_fd = dcs->restore_fd;
 libxl__domain_build_state *const state = >build_state;
-libxl__srm_restore_autogen_callbacks *const callbacks =
->srs.shs.callbacks.restore.a;
+const int checkpointed_stream = dcs->restore_params.checkpointed_stream;
 
 if (rc) {
 domcreate_rebuild_done(egc, dcs, rc);
@@ -1043,7 +1053,6 @@ static void domcreate_bootloader_done(libxl__egc *egc,
 }
 
 /* Restore */
-callbacks->checkpoint = libxl__remus_domain_restore_checkpoint_callback;
 
 rc = libxl__build_pre(gc, domid, d_config, state);
 if (rc)
@@ -1054,9 +1063,10 @@ static void domcreate_bootloader_done(libxl__egc *egc,
 dcs->srs.fd = restore_fd;
 dcs->srs.legacy = (dcs->restore_params.stream_version == 1);
 dcs->srs.completion_callback = domcreate_stream_done;
-dcs->srs.checkpoint_callback = remus_checkpoint_stream_done;
 
 if (restore_fd >= 0) {
+if (checkpointed_stream)
+libxl__remus_restore_setup(egc, dcs);
 libxl__stream_read_start(egc, >srs);
 return;
 }
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index 2269998..7835d4d 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -1489,7 +1489,7 @@ static void remus_devices_preresume_cb(libxl__egc *egc,
libxl__remus_devices_state *rds,
int rc);
 
-static void libxl__remus_domain_suspend_callback(void *data)
+void libxl__remus_domain_suspend_callback(void *data)
 {
 libxl__save_helper_state *shs = data;
 libxl__egc *egc = shs->egc;
@@ -1532,7 +1532,7 @@ out:
 libxl__xc_domain_saverestore_async_callback_done(egc, >sws.shs, !rc);
 }
 
-static void libxl__remus_domain_resume_callback(void *data)
+void libxl__remus_domain_resume_callback(void *data)
 {
 libxl__save_helper_state *shs = data;
 libxl__egc *egc = shs->egc;
@@ -1569,8 +1569,6 @@ out:
 
 /*- remus asynchronous checkpoint callback -*/
 
-static void remus_checkpoint_stream_written(
-libxl__egc *egc, libxl__stream_write_state *s

[Xen-devel] [PATCH v8 07/13] migration/save: pass checkpointed_stream from libxl to libxc

[Wen Congyang]

Pass checkpointed_stream from libxl to libxc.
It won't affact legacy migration because legacy migration
won't use this param.

Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
CC: Ian Campbell <ian.campb...@citrix.com>
CC: Ian Jackson <ian.jack...@eu.citrix.com>
CC: Wei Liu <wei.l...@citrix.com>
CC: Andrew Cooper <andrew.coop...@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.w...@oracle.com>
Acked-by: Wei Liu <wei.l...@citrix.com>
---
 tools/libxc/include/xenguest.h   |  6 --
 tools/libxc/xc_nomigrate.c   |  3 ++-
 tools/libxc/xc_sr_common.h   | 12 +++-
 tools/libxc/xc_sr_save.c | 17 +++--
 tools/libxl/libxl.c  |  2 ++
 tools/libxl/libxl_dom_save.c | 11 ---
 tools/libxl/libxl_internal.h |  1 +
 tools/libxl/libxl_save_callout.c |  2 +-
 tools/libxl/libxl_save_helper.c  |  3 ++-
 tools/libxl/libxl_stream_write.c |  2 +-
 tools/libxl/libxl_types.idl  |  1 +
 11 files changed, 44 insertions(+), 16 deletions(-)

diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index d48b3ff..affc42b 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -29,7 +29,6 @@
 #define XCFLAGS_HVM   (1 << 2)
 #define XCFLAGS_STDVGA(1 << 3)
 #define XCFLAGS_CHECKPOINT_COMPRESS(1 << 4)
-#define XCFLAGS_CHECKPOINTED(1 << 5)
 
 #define X86_64_B_SIZE   64 
 #define X86_32_B_SIZE   32
@@ -82,11 +81,14 @@ struct save_callbacks {
  * @parm xch a handle to an open hypervisor interface
  * @parm fd the file descriptor to save a domain to
  * @parm dom the id of the domain
+ * @param checkpointed_stream MIG_STREAM_NONE if the far end of the stream
+ *doesn't use checkpointing
  * @return 0 on success, -1 on failure
  */
 int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t 
max_iters,
uint32_t max_factor, uint32_t flags /* XCFLAGS_xxx */,
-   struct save_callbacks* callbacks, int hvm);
+   struct save_callbacks* callbacks, int hvm,
+   int checkpointed_stream);
 
 /* callbacks provided by xc_domain_restore */
 struct restore_callbacks {
diff --git a/tools/libxc/xc_nomigrate.c b/tools/libxc/xc_nomigrate.c
index 902429e..c9124df 100644
--- a/tools/libxc/xc_nomigrate.c
+++ b/tools/libxc/xc_nomigrate.c
@@ -22,7 +22,8 @@
 
 int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t 
max_iters,
uint32_t max_factor, uint32_t flags,
-   struct save_callbacks* callbacks, int hvm)
+   struct save_callbacks* callbacks, int hvm,
+   int checkpointed_stream)
 {
 errno = ENOSYS;
 return -1;
diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 60b43e8..66f595f 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -180,6 +180,16 @@ struct xc_sr_context
 
 xc_dominfo_t dominfo;
 
+/*
+ * migration stream
+ * 0: Plain VM
+ * 1: Remus
+ */
+enum {
+MIG_STREAM_NONE, /* plain stream */
+MIG_STREAM_REMUS,
+} migration_stream;
+
 union /* Common save or restore data. */
 {
 struct /* Save data. */
@@ -191,7 +201,7 @@ struct xc_sr_context
 bool live;
 
 /* Plain VM, or checkpoints over time. */
-bool checkpointed;
+int checkpointed;
 
 /* Further debugging information in the stream. */
 bool debug;
diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
index ccb000e..e258b7c 100644
--- a/tools/libxc/xc_sr_save.c
+++ b/tools/libxc/xc_sr_save.c
@@ -629,7 +629,7 @@ static int send_domain_memory_live(struct xc_sr_context 
*ctx)
 if ( rc )
 goto out;
 
-if ( ctx->save.debug && !ctx->save.checkpointed )
+if ( ctx->save.debug && ctx->save.checkpointed != MIG_STREAM_NONE )
 {
 rc = verify_frames(ctx);
 if ( rc )
@@ -758,7 +758,7 @@ static int save(struct xc_sr_context *ctx, uint16_t 
guest_type)
 
 if ( ctx->save.live )
 rc = send_domain_memory_live(ctx);
-else if ( ctx->save.checkpointed )
+else if ( ctx->save.checkpointed != MIG_STREAM_NONE )
 rc = send_domain_memory_checkpointed(ctx);
 else
 rc = send_domain_memory_nonlive(ctx);
@@ -778,7 +778,7 @@ static int save(struct xc_sr_context *ctx, uint16_t 
guest_type)
 if ( rc )
 goto err;
 
-if ( ctx->save.checkpointed )
+if ( ctx->save.checkpointed != MIG_STREAM_NONE )
 {
 /*
  * We have now completed the initial live portion of the checkpoint
@@ -799,7 +799,7 @@ static int save(struct xc_sr_context *ctx, uint16_t 
guest_type)
 if ( rc <= 0 

[Xen-devel] [PATCH v8 09/13] tools/libxl: rename remus device to checkpoint device

[Wen Congyang]

This patch is auto generated by the following commands:
 1. git mv tools/libxl/libxl_remus_device.c 
tools/libxl/libxl_checkpoint_device.c
 2. perl -pi -e 's/libxl_remus_device/libxl_checkpoint_device/g' 
tools/libxl/Makefile
 3. perl -pi -e 's/\blibxl__remus_devices/libxl__checkpoint_devices/g' 
tools/libxl/*.[ch]
 4. perl -pi -e 's/\blibxl__remus_device\b/libxl__checkpoint_device/g' 
tools/libxl/*.[ch]
 5. perl -pi -e 
's/\blibxl__remus_device_instance_ops\b/libxl__checkpoint_device_instance_ops/g'
 tools/libxl/*.[ch]
 6. perl -pi -e 's/\blibxl__remus_callback\b/libxl__checkpoint_callback/g' 
tools/libxl/*.[ch]
 7. perl -pi -e 's/\bremus_device_init\b/checkpoint_device_init/g' 
tools/libxl/*.[ch]
 8. perl -pi -e 's/\bremus_devices_setup\b/checkpoint_devices_setup/g' 
tools/libxl/*.[ch]
 9. perl -pi -e 's/\bdefine_remus_checkpoint_api\b/define_checkpoint_api/g' 
tools/libxl/*.[ch]
10. perl -pi -e 's/\brds\b/cds/g' tools/libxl/*.[ch]
11. perl -pi -e 's/REMUS_DEVICE/CHECKPOINT_DEVICE/g' tools/libxl/*.[ch] 
tools/libxl/*.idl
12. perl -pi -e 's/REMUS_DEVOPS/CHECKPOINT_DEVOPS/g' tools/libxl/*.[ch] 
tools/libxl/*.idl
13. perl -pi -e 's/\bremus\b/checkpoint/g' 
tools/libxl/libxl_checkpoint_device.[ch]
14. perl -pi -e 's/\bremus device/checkpoint device/g' 
tools/libxl/libxl_internal.h
15. perl -pi -e 's/\bRemus device/checkpoint device/g' 
tools/libxl/libxl_internal.h
16. perl -pi -e 's/\bremus abstract/checkpoint abstract/g' 
tools/libxl/libxl_internal.h
17. perl -pi -e 's/\bremus invocation/checkpoint invocation/g' 
tools/libxl/libxl_internal.h
18. perl -pi -e 's/\blibxl__remus_device_\(/libxl__checkpoint_device_(/g' 
tools/libxl/libxl_internal.h

The patch also fixes the following backword compatibility:
  The error code ERROR_REMUS_XXX was introduced in Xen 4.5, and
  changed to ERROR_CHECKPOINT_XXX after previous renaming.

Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Reviewed-Lightly-by: Konrad Rzeszutek Wilk <konrad.w...@oracle.com>
Acked-by: Wei Liu <wei.l...@citrix.com>
---
 tools/libxl/Makefile   |   2 +-
 tools/libxl/libxl.h|  12 ++
 ...xl_remus_device.c => libxl_checkpoint_device.c} | 198 ++---
 tools/libxl/libxl_internal.h   | 112 ++--
 tools/libxl/libxl_netbuffer.c  | 108 +--
 tools/libxl/libxl_nonetbuffer.c|  10 +-
 tools/libxl/libxl_remus.c  |  76 
 tools/libxl/libxl_remus_disk_drbd.c|  52 +++---
 tools/libxl/libxl_types.idl|   4 +-
 9 files changed, 293 insertions(+), 281 deletions(-)
 rename tools/libxl/{libxl_remus_device.c => libxl_checkpoint_device.c} (52%)

diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 263ea0e..789a12e 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -64,7 +64,7 @@ else
 LIBXL_OBJS-y += libxl_no_convert_callout.o
 endif
 
-LIBXL_OBJS-y += libxl_remus.o libxl_remus_device.o libxl_remus_disk_drbd.o
+LIBXL_OBJS-y += libxl_remus.o libxl_checkpoint_device.o libxl_remus_disk_drbd.o
 
 LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o libxl_psr.o
 LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_arm.o libxl_libfdt_compat.o
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 6225db1..f9e3ef5 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -883,6 +883,18 @@ void libxl_mac_copy(libxl_ctx *ctx, libxl_mac *dst, 
libxl_mac *src);
  */
 #define LIBXL_HAVE_CHECKPOINTED_STREAM 1
 
+/*
+ * ERROR_REMUS_XXX error code only exists from Xen 4.5, Xen 4.6 and it
+ * is changed to ERROR_CHECKPOINT_XXX in Xen 4.7
+ */
+#if defined(LIBXL_API_VERSION) && LIBXL_API_VERSION >= 0x040500 \
+   && LIBXL_API_VERSION < 0x040700
+#define ERROR_REMUS_DEVOPS_DOES_NOT_MATCH \
+ERROR_CHECKPOINT_DEVOPS_DOES_NOT_MATCH
+#define ERROR_REMUS_DEVICE_NOT_SUPPORTED \
+ERROR_CHECKPOINT_DEVICE_NOT_SUPPORTED
+#endif
+
 typedef char **libxl_string_list;
 void libxl_string_list_dispose(libxl_string_list *sl);
 int libxl_string_list_length(const libxl_string_list *sl);
diff --git a/tools/libxl/libxl_remus_device.c 
b/tools/libxl/libxl_checkpoint_device.c
similarity index 52%
rename from tools/libxl/libxl_remus_device.c
rename to tools/libxl/libxl_checkpoint_device.c
index a6cb7f6..109cd23 100644
--- a/tools/libxl/libxl_remus_device.c
+++ b/tools/libxl/libxl_checkpoint_device.c
@@ -17,9 +17,9 @@
 
 #include "libxl_internal.h"
 
-extern const libxl__remus_device_instance_ops remus_device_nic;
-extern const libxl__remus_device_instance_ops remus_device_drbd_disk;
-static const libxl__remus_device_instance_ops *remus_ops[] = {
+extern const libxl__checkpoint_device_instance_ops remus_device_nic;
+extern const libxl__checkpoint_device_instance_ops remus_device_drbd_disk;

[Xen-devel] [PATCH v8 05/13] tools/libxc: support to resume uncooperative HVM guests

[Wen Congyang]

Before this patch:
1. suspend
a. PVHVM and PV: we use the same way to suspend the guest (send the suspend
   request to the guest). If the guest doesn't support evtchn, the xenstore
   variant will be used, suspending the guest via XenBus control node.
b. pure HVM: we call xc_domain_shutdown(..., SHUTDOWN_suspend) to suspend
   the guest

2. Resume:
a. fast path(fast=1)
   Do not change the guest state. We call libxl__domain_resume(.., 1) which
   calls xc_domain_resume(..., 1 /* fast=1*/) to resume the guest.
   PV:   modify the return code to 1, and than call the domctl:
 XEN_DOMCTL_resumedomain
   PVHVM:same with PV
   pure HVM: do nothing in modify_returncode, and than call the domctl:
 XEN_DOMCTL_resumedomain
b. slow
   Used when the guest's state have been changed. Will call
   libxl__domain_resume(..., 0) to resume the guest.
   PV:   update start info, and reset all secondary CPU states. Than call
 the domctl: XEN_DOMCTL_resumedomain
   PVHVM:can not be resumed. You will get the following error message:
 "Cannot resume uncooperative HVM guests"
   pure HVM: same with PVHVM

After this patch:
1. suspend
   unchanged

2. Resume
a. fast path:
   unchanged
b. slow
   PV:   unchanged
   PVHVM:call XEN_DOMCTL_resumedomain to resume the guest. Because we
 don't modify the return code, the PV driver will disconnect
 and reconnect.
 The guest ends up doing the XENMAPSPACE_shared_info
 XENMEM_add_to_physmap hypercall and resetting all of its CPU
 states to point to the shared_info(well except the ones past 32).
 That is the Linux kernel does that - regardless whether the
 SCHEDOP_shutdown:SHUTDOWN_suspend returns 1 or not.
   Pure HVM: call XEN_DOMCTL_resumedomain to resume the guest.

Under COLO, we will update the guest's state(modify memory, cpu's registers,
device status...). In this case, we cannot use the fast path to resume it.
Keep the return code 0, and use a slow path to resume the guest. While
resuming HVM using slow path is not supported currently, this patch is to
make the resume call to not fail.

Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.w...@oracle.com>
---
 tools/libxc/xc_resume.c | 25 +
 1 file changed, 21 insertions(+), 4 deletions(-)

diff --git a/tools/libxc/xc_resume.c b/tools/libxc/xc_resume.c
index e692b81..4eedf87 100644
--- a/tools/libxc/xc_resume.c
+++ b/tools/libxc/xc_resume.c
@@ -108,6 +108,26 @@ static int xc_domain_resume_cooperative(xc_interface *xch, 
uint32_t domid)
 return do_domctl(xch, );
 }
 
+static int xc_domain_resume_hvm(xc_interface *xch, uint32_t domid)
+{
+DECLARE_DOMCTL;
+
+/*
+ * The domctl XEN_DOMCTL_resumedomain unpause each vcpu. After
+ * the domctl, the guest will run.
+ *
+ * If it is PVHVM, the guest called the hypercall
+ *SCHEDOP_shutdown:SHUTDOWN_suspend
+ * to suspend itself. We don't modify the return code, so the PV driver
+ * will disconnect and reconnect.
+ *
+ * If it is a HVM, the guest will continue running.
+ */
+domctl.cmd = XEN_DOMCTL_resumedomain;
+domctl.domain = domid;
+return do_domctl(xch, );
+}
+
 static int xc_domain_resume_any(xc_interface *xch, uint32_t domid)
 {
 DECLARE_DOMCTL;
@@ -137,10 +157,7 @@ static int xc_domain_resume_any(xc_interface *xch, 
uint32_t domid)
  */
 #if defined(__i386__) || defined(__x86_64__)
 if ( info.hvm )
-{
-ERROR("Cannot resume uncooperative HVM guests");
-return rc;
-}
+return xc_domain_resume_hvm(xch, domid);
 
 if ( xc_domain_get_guest_width(xch, domid, >guest_width) != 0 )
 {
-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v8 11/13] tools/libxl: store remus_ops in checkpoint device state

[Wen Congyang]

Checkpoint device is an abstract layer to do checkpoint.
COLO can also use it to do checkpoint. But there are
still some codes in checkpoint device which touch remus.

This patch and:
 tools/libxl: move remus state into a seperate structure
 tools/libxl: seperate device init/cleanup from checkpoint device layer
will seperate remus from checkpoint device layer.

We use remus ops directly in checkpoint device. Store it
in checkpoint device state so that we do not aware of
remus_ops in the checkpoint device layer.

It is pure refactoring and no functional changes.

Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Acked-by:Ian Campbell <ian.campb...@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.w...@oracle.com>
Acked-by: Wei Liu <wei.l...@citrix.com>
---
 tools/libxl/libxl_checkpoint_device.c | 10 +-
 tools/libxl/libxl_internal.h  |  2 ++
 tools/libxl/libxl_remus.c |  9 +
 3 files changed, 12 insertions(+), 9 deletions(-)

diff --git a/tools/libxl/libxl_checkpoint_device.c 
b/tools/libxl/libxl_checkpoint_device.c
index 226f159..bbc6dc4 100644
--- a/tools/libxl/libxl_checkpoint_device.c
+++ b/tools/libxl/libxl_checkpoint_device.c
@@ -17,14 +17,6 @@
 
 #include "libxl_internal.h"
 
-extern const libxl__checkpoint_device_instance_ops remus_device_nic;
-extern const libxl__checkpoint_device_instance_ops remus_device_drbd_disk;
-static const libxl__checkpoint_device_instance_ops *remus_ops[] = {
-_device_nic,
-_device_drbd_disk,
-NULL,
-};
-
 /*- helper functions -*/
 
 static int init_device_subkind(libxl__checkpoint_devices_state *cds)
@@ -172,7 +164,7 @@ static void device_setup_iterate(libxl__egc *egc, 
libxl__ao_device *aodev)
 goto out;
 
 do {
-dev->ops = remus_ops[++dev->ops_index];
+dev->ops = dev->cds->ops[++dev->ops_index];
 if (!dev->ops) {
 libxl_device_nic * nic = NULL;
 libxl_device_disk * disk = NULL;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index bde7a15..2847d13 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2895,6 +2895,8 @@ struct libxl__checkpoint_devices_state {
 uint32_t domid;
 libxl__checkpoint_callback *callback;
 int device_kind_flags;
+/* The ops must be pointer array, and the last ops must be NULL. */
+const libxl__checkpoint_device_instance_ops **ops;
 
 /*- private for abstract layer only -*/
 
diff --git a/tools/libxl/libxl_remus.c b/tools/libxl/libxl_remus.c
index d41a439..86f81c3 100644
--- a/tools/libxl/libxl_remus.c
+++ b/tools/libxl/libxl_remus.c
@@ -18,6 +18,14 @@
 
 #include "libxl_internal.h"
 
+extern const libxl__checkpoint_device_instance_ops remus_device_nic;
+extern const libxl__checkpoint_device_instance_ops remus_device_drbd_disk;
+static const libxl__checkpoint_device_instance_ops *remus_ops[] = {
+_device_nic,
+_device_drbd_disk,
+NULL,
+};
+
 /* Remus setup and teardown -*/
 
 static void remus_setup_done(libxl__egc *egc,
@@ -55,6 +63,7 @@ void libxl__remus_setup(libxl__egc *egc,
 cds->ao = ao;
 cds->domid = dss->domid;
 cds->callback = remus_setup_done;
+cds->ops = remus_ops;
 
 dss->sws.checkpoint_callback = remus_checkpoint_stream_written;
 
-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v7 08/18] tools/libxl: introduce libxl__domain_restore_device_model to load qemu state

[Wen Congyang]

On 02/04/2016 03:40 AM, Wei Liu wrote:
> On Fri, Jan 29, 2016 at 01:27:24PM +0800, Wen Congyang wrote:
>> In normal migration, the qemu state is passed to qemu as a parameter.
>> With COLO, secondary vm is running. So we will do the following steps
>> at every checkpoint:
>> 1. suspend both primary vm and secondary vm
>> 2. sync the state
>> 3. resume both primary vm and secondary vm
>> Primary will send qemu's state in step2, and secondary's qemu should
>> read it and restore the state before it is resumed. We can not pass
>> the state to qemu as a parameter because secondary QEMU already started
>> at this point, so we introduce libxl__domain_restore_device_model() to
>> do it. This API MUST be called before resuming secondary vm.
>>
>> Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
>> Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
>> Cc: Anthony Perard <anthony.per...@citrix.com>
>> ---
>>  tools/libxl/libxl_dom_save.c | 20 
>>  tools/libxl/libxl_internal.h |  4 
>>  tools/libxl/libxl_qmp.c  | 10 ++
>>  3 files changed, 34 insertions(+)
>>
>> diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
>> index cd2e7de..7383d2d 100644
>> --- a/tools/libxl/libxl_dom_save.c
>> +++ b/tools/libxl/libxl_dom_save.c
>> @@ -518,6 +518,26 @@ int 
>> libxl__restore_emulator_xenstore_data(libxl__domain_create_state *dcs,
>>  return rc;
>>  }
>>  
>> +int libxl__domain_restore_device_model(libxl__gc *gc, uint32_t domid,
>> +   const char *restore_file)
>> +{
>> +int rc;
>> +
>> +switch (libxl__device_model_version_running(gc, domid)) {
>> +case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN_TRADITIONAL:
>> +/* Will never be supported. */
>> +rc = ERROR_INVAL;
>> +break;
> 
> I'm not entirely sure if this statement would be true. The function name
> is generic enough to indicate this case should be supported.
> 
> However, this function is not used anywhere in this series, so I don't
> know whether my comment makes sense.
> 
> One way of moving forward is to stick this patch to COLO series itself.
> Let's skip this in this prerequisite series.

OK, I will put it in the COLO series itself.
This API is used for COLO, and COLO requries the newest qemu with block 
replication.
The block replication is still in the way. The tranditional qemu doesn't support
block replication and it is hard to backport it.

> 
>> +case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN:
>> +rc = libxl__qmp_restore(gc, domid, restore_file);
>> +break;
>> +default:
>> +rc = ERROR_INVAL;
>> +}
>> +
>> +return rc;
>> +}
>> +
>>  /*
>>   * Local variables:
>>   * mode: C
>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>> index fbd1acb..896c119 100644
>> --- a/tools/libxl/libxl_internal.h
>> +++ b/tools/libxl/libxl_internal.h
>> @@ -1117,6 +1117,8 @@ _hidden int libxl__domain_rename(libxl__gc *gc, 
>> uint32_t domid,
>>   const char *old_name, const char *new_name,
>>   xs_transaction_t trans);
>>  
>> +_hidden int libxl__domain_restore_device_model(libxl__gc *gc, uint32_t 
>> domid,
>> +   const char *restore_file);
>>  _hidden int libxl__domain_resume_device_model(libxl__gc *gc, uint32_t 
>> domid);
>>  
>>  _hidden const char *libxl__userdata_path(libxl__gc *gc, uint32_t domid,
>> @@ -1760,6 +1762,8 @@ _hidden int libxl__qmp_stop(libxl__gc *gc, int domid);
>>  _hidden int libxl__qmp_resume(libxl__gc *gc, int domid);
>>  /* Save current QEMU state into fd. */
>>  _hidden int libxl__qmp_save(libxl__gc *gc, int domid, const char *filename);
>> +/* Load current QEMU state from fd. */
> 
> This comment is wrong, it loads QEMU state from file, not fd.

will fix it in the next version.

Thanks
Wen Congyang

> 
>> +_hidden int libxl__qmp_restore(libxl__gc *gc, int domid, const char 
>> *filename);
>>  /* Set dirty bitmap logging status */
>>  _hidden int libxl__qmp_set_global_dirty_log(libxl__gc *gc, int domid, bool 
>> enable);
>>  _hidden int libxl__qmp_insert_cdrom(libxl__gc *gc, int domid, const 
>> libxl_device_disk *disk);
>> diff --git a/tools/libxl/libxl_qmp.c b/tools/libxl/libxl_qmp.c
>> index 714038b..eec8a44 100644
>> --- a/tools/libxl/libxl_qmp.c
>> +++ b/tools/libxl/libxl_qmp.c
>&g

Re: [Xen-devel] [PATCH v7 01/18] libxl/remus: init checkpoint_callback in Remus setup callback

[Wen Congyang]

On 02/04/2016 03:39 AM, Wei Liu wrote:
> On Fri, Jan 29, 2016 at 01:27:17PM +0800, Wen Congyang wrote:
>> init stream {read/write} state checkpoint_callback in Remus setup callback.
>> There's no functional change, it's just refactoring so that we can move
>> all remus code into one file.
>>
>> Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
>> Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
>> CC: Andrew Cooper <andrew.coop...@citrix.com>
>> CC: Ian Campbell <ian.campb...@citrix.com>
>> CC: Ian Jackson <ian.jack...@eu.citrix.com>
>> CC: Wei Liu <wei.l...@citrix.com>
>> Reviewed-by: Konrad Rzeszutek Wilk <konrad.w...@oracle.com>
>> ---
>>  tools/libxl/libxl.c  |  2 ++
>>  tools/libxl/libxl_create.c   | 10 +-
>>  tools/libxl/libxl_dom.c  |  5 +
>>  tools/libxl/libxl_internal.h |  4 
>>  4 files changed, 16 insertions(+), 5 deletions(-)
>>
>> diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
>> index 94b5656..5346a0c 100644
>> --- a/tools/libxl/libxl.c
>> +++ b/tools/libxl/libxl.c
>> @@ -917,6 +917,8 @@ static void libxl__remus_setup(libxl__egc *egc,
>>  rds->domid = dss->domid;
>>  rds->callback = remus_setup_done;
>>  
>> +dss->sws.checkpoint_callback = remus_checkpoint_stream_written;
>> +
>>  libxl__remus_devices_setup(egc, rds);
>>  return;
>>  
>> diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
>> index e491d83..8b1efe5 100644
>> --- a/tools/libxl/libxl_create.c
>> +++ b/tools/libxl/libxl_create.c
>> @@ -718,6 +718,12 @@ static void remus_checkpoint_stream_done(
>>  libxl__xc_domain_saverestore_async_callback_done(egc, >shs, rc);
>>  }
>>  
>> +static void libxl__remus_restore_setup(libxl__egc *egc,
>> +   libxl__domain_create_state *dcs)
>> +{
>> +dcs->srs.checkpoint_callback = remus_checkpoint_stream_done;
>> +}
>> +
>>  /*- main domain creation -*/
>>  
>>  /* We have a linear control flow; only one event callback is
>> @@ -1004,6 +1010,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
>>  libxl__domain_build_state *const state = >build_state;
>>  libxl__srm_restore_autogen_callbacks *const callbacks =
>>  >srs.shs.callbacks.restore.a;
>> +const int checkpointed_stream = dcs->restore_params.checkpointed_stream;
>>  
>>  if (rc) {
>>  domcreate_rebuild_done(egc, dcs, rc);
>> @@ -1042,9 +1049,10 @@ static void domcreate_bootloader_done(libxl__egc *egc,
> 
> A few lines above in this function, there is a line like:
> 
> /* Restore */
> callbacks->checkpoint = libxl__remus_domain_restore_checkpoint_callback;
> 
> Do you not need to move this into libxl__remus_restore_setup as well? As
> far as I can tell that's only useful for remus.
> 
>>  dcs->srs.fd = restore_fd;
>>  dcs->srs.legacy = (dcs->restore_params.stream_version == 1);
>>  dcs->srs.completion_callback = domcreate_stream_done;
>> -dcs->srs.checkpoint_callback = remus_checkpoint_stream_done;
>>  
>>  if (restore_fd >= 0) {
>> +if (checkpointed_stream)
>> +libxl__remus_restore_setup(egc, dcs);
>>  libxl__stream_read_start(egc, >srs);
>>  return;
>>  }
>> diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
>> index 2269998..9e28bc4 100644
>> --- a/tools/libxl/libxl_dom.c
>> +++ b/tools/libxl/libxl_dom.c
>> @@ -1569,8 +1569,6 @@ out:
>>  
>>  /*- remus asynchronous checkpoint callback -*/
>>  
>> -static void remus_checkpoint_stream_written(
>> -libxl__egc *egc, libxl__stream_write_state *sws, int rc);
>>  static void remus_devices_commit_cb(libxl__egc *egc,
>>  libxl__remus_devices_state *rds,
>>  int rc);
>> @@ -1588,7 +1586,7 @@ static void 
>> libxl__remus_domain_save_checkpoint_callback(void *data)
>>  libxl__stream_write_start_checkpoint(egc, >sws);
>>  }
>>  
>> -static void remus_checkpoint_stream_written(
>> +void remus_checkpoint_stream_written(
>>  libxl__egc *egc, libxl__stream_write_state *sws, int rc)
>>  {
>>  libxl__domain_suspend_state *dss = CONTAINER_OF(sws, *dss, sws);
>> @@ -1761,7 +1759,6 @@ void libxl__domain_save(libxl__egc *egc, 
>> libxl__domain_suspend_state *dss)
>>  callbacks->suspend = libxl__remus_domain_suspend_callback;
>>  callbacks->postcopy = libxl__remus_domain_resume_callback;
>>  callbacks->checkpoint = 
>> libxl__remus_domain_save_checkpoint_callback;
> 
> Do you not want to move this to libxl__remus_setup?

I think so, and will fix these two in the next version.

Thanks
Wen Congyang

> 
> 
> Wei.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v7 12/18] tools/libx{l, c}: add back channel to libxc

[Wen Congyang]

On 02/04/2016 03:40 AM, Wei Liu wrote:
> On Fri, Jan 29, 2016 at 01:27:28PM +0800, Wen Congyang wrote:
>> In COLO mode, both VMs are running, and are considered in sync if the
>> visible network traffic is identical.  After some time, they fall out of
>> sync.
>>
>> At this point, the two VMs have definitely diverged.  Lets call the
>> primary dirty bitmap set A, while the secondary dirty bitmap set B.
>>
>> Sets A and B are different.
>>
>> Under normal migration, the page data for set A will be sent from the
>> primary to the secondary.
>>
>> However, the set difference B - A (the one in B but not in A, lets
>> call this C) is out-of-date on the secondary (with respect to the
>> primary) and will not be sent by the primary (to secondary), as it
>> was not memory dirtied by the primary. The secondary needs C page data
>> to reconstruct an exact copy of the primary at the checkpoint.
>>
>> The secondary cannot calculate C as it doesn't know A.  Instead, the
>> secondary must send B to the primary, at which point the primary
>> calculates the union of A and B (lets call this D) which is all the
>> pages dirtied by both the primary and the secondary, and sends all page
>> data covered by D.
>>
>> In the general case, D is a superset of both A and B.  Without the
>> backchannel dirty bitmap, a COLO checkpoint can't reconstruct a valid
>> copy of the primary.
>>
>> We transfer the dirty bitmap on libxc side, so we need to introduce back
>> channel to libxc.
>>
>> Note: it is different from the paper. We change the original design to
>> the current one, according to our following concerns:
>> 1. The original design needs extra memory on Secondary host. When there's
>>multiple backups on one host, the memory cost is high.
>> 2. The memory cache code will be another 1k+, it will make the review
>>more time consuming.
>>
>> Note: the back channel will be used in the patch
> 
> "will not be used" ?
> 
> I don't see any read / write to the newly introduced fd.

It is used in COLO series.

Some patches in this series just introduce an API. Thess APIs will be used
in COLO series. Do you mean that these patches should be put in COLO series?
If so, I will check all patches.

> 
>>  libxc/restore: send dirty pfn list to primary when checkpoint under COLO
>> to send dirty pfn list from secondary to primary. The patch is posted in
>> another series.
>>
>> Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
>> Signed-off-by: Andrew Cooper <andrew.coop...@citrix.com>
>> CC: Ian Campbell <ian.campb...@citrix.com>
>> CC: Ian Jackson <ian.jack...@eu.citrix.com>
>> CC: Wei Liu <wei.l...@citrix.com>
>> ---
> [...]
>>  
>>  /*- helper execution -*/
>> +static int dup_fd_helper(libxl__gc *gc, int fd, const char *what)
>> +{
>> +int dup_fd = fd;
>> +
>> +if (fd <= 2) {
>> +dup_fd = dup(fd);
>> +if (dup_fd < 0) {
>> +LOGE(ERROR,"dup %s", what);
>> +exit(-1);
>> +}
>> +}
>> +libxl_fd_set_cloexec(CTX, dup_fd, 0);
>> +
>> +return dup_fd;
>> +}
>>  
> 
> It would be better if introduction of this helper to be separated into a
> different patch.

OK, will fix it in the next version.

Thanks
Wen Congyang

> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v7 05/18] tools/libxc: support to resume uncooperative HVM guests

[Wen Congyang]

On 02/04/2016 03:40 AM, Wei Liu wrote:
> On Fri, Jan 29, 2016 at 01:27:21PM +0800, Wen Congyang wrote:
>> Before this patch:
>> 1. suspend
>> a. PVHVM and PV: we use the same way to suspend the guest (send the suspend
>>request to the guest). If the guest doesn't support evtchn, the xenstore
>>variant will be used, suspending the guest via XenBus control node.
>> b. pure HVM: we call xc_domain_shutdown(..., SHUTDOWN_suspend) to suspend
>>the guest
>>
>> 2. Resume:
>> a. fast path(fast=1)
>>Do not change the guest state. We call libxl__domain_resume(.., 1) which
>>calls xc_domain_resume(..., 1 /* fast=1*/) to resume the guest.
>>PV:   modify the return code to 1, and than call the domctl:
>>  XEN_DOMCTL_resumedomain
>>PVHVM:same with PV
>>pure HVM: do nothing in modify_returncode, and than call the domctl:
> 
> "then"
> 
>>  XEN_DOMCTL_resumedomain
>> b. slow
>>Used when the guest's state have been changed. Will call
>>libxl__domain_resume(..., 0) to resume the guest.
>>PV:   update start info, and reset all secondary CPU states. Than call
>>  the domctl: XEN_DOMCTL_resumedomain
>>PVHVM:can not be resumed. You will get the following error message:
>>  "Cannot resume uncooperative HVM guests"
>>purt HVM: same with PVHVM
> 
> "pure"
> 
>>
>> After this patch:
>> 1. suspend
>>unchanged
>>
>> 2. Resume
>> a. fast path:
>>unchanged
>> b. slow
>>PV:   unchanged
>>PVHVM:call XEN_DOMCTL_resumedomain to resume the guest. Because we
>>  don't modify the return code, the PV driver will disconnect
>>  and reconnect.
>>  The guest ends up doing the XENMAPSPACE_shared_info
>>  XENMEM_add_to_physmap hypercall and resetting all of its CPU
>>  states to point to the shared_info(well except the ones past 
>> 32).
>>  That is the Linux kernel does that - regardless whether the
>>  SCHEDOP_shutdown:SHUTDOWN_suspend returns 1 or not.
>>Pure HVM: call XEN_DOMCTL_resumedomain to resume the guest.
> 
> In summary, this patch only changes slow path resume. Further more, it
> only affects PVHVM and pure HVM variants.
> 
> With you patch, pure HVM is able to resume with effectively the same
> path via XEN_DOMCTL_resumedomain, albeit it is done in two functions
> (_cooperative and _any).
> 
> And according to the recently change in documentation, slow path is
> always safe.
> 
> I think the commit message can be simplified a bit. This is assuming
> using XEN_DOMCTL_resumedomain to resume (PV)HVM in slow path is safe.
> 
> ===
> 
> Use XEN_DOMCTL_resumedomain to resume (PV)HVM guest in slow path
> 
> Previously it was not possible to resume PVHVM or pure HVM guest in slow
> path because libxc didn't support that.
> 
> Using XEN_DOMCTL_resumedomain without modifying guest state  to resume a
> guest is considered to be always safe.  Introduce a function to do that
> for (PV)HVM guests in slow path resume.
> 
> This patch fixes a bug that denies (PV)HVM slow path resume.  This will
> enable COLO to work properly:  COLO requires HVM guest to start in the
> new context that has been set up by COLO, hence slow path resume is
> required.
> 
> ===
> 
> Does this sound right? Especially the wording about safety.

It sounds right.

Thanks
Wen Congyang

> 
> Ian and Ian, you seemed to have suggested Congyang to write the above
> commit message. What do you think about my updated one?
> 
>>
>> Under COLO, we will update the guest's state(modify memory, cpu's registers,
>> device status...). In this case, we cannot use the fast path to resume it.
>> Keep the return code 0, and use a slow path to resume the guest. While
>> resuming HVM using slow path is not supported currently, this patch is to
>> make the resume call to not fail.
>>
>> Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
>> Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
>> ---
>>  tools/libxc/xc_resume.c | 25 +
>>  1 file changed, 21 insertions(+), 4 deletions(-)
>>
>> diff --git a/tools/libxc/xc_resume.c b/tools/libxc/xc_resume.c
>> index 87d4324..4a9b035 100644
>> --- a/tools/libxc/xc_resume.c
>> +++ b/tools/libxc/xc_resume.c
>> @@ -108,6 +108,26 @@ static int xc_domain_resume_cooperative(xc_interface 
>> *xch, uint32_t domid)
>>  return do_d

Re: [Xen-devel] [PATCH v7 07/18] migration/save: pass checkpointed_stream from libxl to libxc

[Wen Congyang]

On 02/04/2016 03:40 AM, Wei Liu wrote:
> On Fri, Jan 29, 2016 at 01:27:23PM +0800, Wen Congyang wrote:
>> Pass checkpointed_stream from libxl to libxc.
>> It won't affact legacy migration because legacy migration
>> won't use this param.
>>
>> Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
>> Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
>> CC: Ian Campbell <ian.campb...@citrix.com>
>> CC: Ian Jackson <ian.jack...@eu.citrix.com>
>> CC: Wei Liu <wei.l...@citrix.com>
>> CC: Andrew Cooper <andrew.coop...@citrix.com>
> 
> Acked-by: Wei Liu <wei.l...@citrix.com>
> 
> With one nit below.
> 
>>  
>> -if ( ctx->save.debug && !ctx->save.checkpointed )
>> +if ( ctx->save.debug &&
>> + ctx->save.checkpointed != MIG_STREAM_NONE )
> 
> You can fold this line to previous one.

Will fix it in the next version.

Thanks
Wen Congyang

> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v7 12/18] tools/libx{l, c}: add back channel to libxc

[Wen Congyang]

On 01/30/2016 12:38 AM, Konrad Rzeszutek Wilk wrote:
> On Fri, Jan 29, 2016 at 01:27:28PM +0800, Wen Congyang wrote:
>> In COLO mode, both VMs are running, and are considered in sync if the
>> visible network traffic is identical.  After some time, they fall out of
>> sync.
>>
>> At this point, the two VMs have definitely diverged.  Lets call the
>> primary dirty bitmap set A, while the secondary dirty bitmap set B.
>>
>> Sets A and B are different.
>>
>> Under normal migration, the page data for set A will be sent from the
>> primary to the secondary.
>>
>> However, the set difference B - A (the one in B but not in A, lets
>> call this C) is out-of-date on the secondary (with respect to the
>> primary) and will not be sent by the primary (to secondary), as it
>> was not memory dirtied by the primary. The secondary needs C page data
>> to reconstruct an exact copy of the primary at the checkpoint.
>>
>> The secondary cannot calculate C as it doesn't know A.  Instead, the
>> secondary must send B to the primary, at which point the primary
>> calculates the union of A and B (lets call this D) which is all the
>> pages dirtied by both the primary and the secondary, and sends all page
>> data covered by D.
>>
>> In the general case, D is a superset of both A and B.  Without the
>> backchannel dirty bitmap, a COLO checkpoint can't reconstruct a valid
>> copy of the primary.
>>
>> We transfer the dirty bitmap on libxc side, so we need to introduce back
>> channel to libxc.
>>
>> Note: it is different from the paper. We change the original design to
>> the current one, according to our following concerns:
>> 1. The original design needs extra memory on Secondary host. When there's
>>multiple backups on one host, the memory cost is high.
>> 2. The memory cache code will be another 1k+, it will make the review
>>more time consuming.
>>
>> Note: the back channel will be used in the patch
>>  libxc/restore: send dirty pfn list to primary when checkpoint under COLO
>> to send dirty pfn list from secondary to primary. The patch is posted in
>> another series.
>>
>> Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
>> Signed-off-by: Andrew Cooper <andrew.coop...@citrix.com>
>> CC: Ian Campbell <ian.campb...@citrix.com>
>> CC: Ian Jackson <ian.jack...@eu.citrix.com>
>> CC: Wei Liu <wei.l...@citrix.com>
> 
> It is a bit confusing to have 'back_fd' and then 'send_fd'. 
> 
> Could you change the 'send_fd' (in this patch) to be called 
> 'send_back_fd' so that the connection between:
>  tools/libxl: Add back channel to allow migration target send data back
> and this patch is clear?
> 
> Or perhaps also add it in the commit description that you are using
> the 'send_fd' provided by ' tools/libxl: Add back channel to allow migration 
> target send data back'

Before this series:
In libxl:
we have send_fd/recv_fd(libxl_domain_remus_start()), and only have 
restore_fd(libxl_domain_create_restore())
In libxc:
We have io_fd(xc_domain_save()/xc_domain_restore())
The fd in libxc is provided by libxl.

I think after this series, we can add the following fd:
1. add a send_back_fd in libxl_domain_create_restore()
2. add a recv_fd in xc_domain_save()
3. add a send_back_fd in xc_domain_restore()

What about this?

Thanks
Wen Congyang

> 
> Otherwise: Reviewed-by: Konrad Rzeszutek Wilk <konrad.w...@oracle.com>
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v7 16/18] tools/libxl: store remus_ops in checkpoint device state

[Wen Congyang]

Checkpoint device is an abstract layer to do checkpoint.
COLO can also use it to do checkpoint. But there are
still some codes in checkpoint device which touch remus.

This patch and:
 tools/libxl: move remus state into a seperate structure
 tools/libxl: seperate device init/cleanup from checkpoint device layer
will seperate remus from checkpoint device layer.

We use remus ops directly in checkpoint device. Store it
in checkpoint device state so that we do not aware of
remus_ops in the checkpoint device layer.

It is pure refactoring and no functional changes.

Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
Acked-by:Ian Campbell <ian.campb...@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.w...@oracle.com>
---
 tools/libxl/libxl_checkpoint_device.c | 10 +-
 tools/libxl/libxl_internal.h  |  2 ++
 tools/libxl/libxl_remus.c |  9 +
 3 files changed, 12 insertions(+), 9 deletions(-)

diff --git a/tools/libxl/libxl_checkpoint_device.c 
b/tools/libxl/libxl_checkpoint_device.c
index 226f159..bbc6dc4 100644
--- a/tools/libxl/libxl_checkpoint_device.c
+++ b/tools/libxl/libxl_checkpoint_device.c
@@ -17,14 +17,6 @@
 
 #include "libxl_internal.h"
 
-extern const libxl__checkpoint_device_instance_ops remus_device_nic;
-extern const libxl__checkpoint_device_instance_ops remus_device_drbd_disk;
-static const libxl__checkpoint_device_instance_ops *remus_ops[] = {
-_device_nic,
-_device_drbd_disk,
-NULL,
-};
-
 /*- helper functions -*/
 
 static int init_device_subkind(libxl__checkpoint_devices_state *cds)
@@ -172,7 +164,7 @@ static void device_setup_iterate(libxl__egc *egc, 
libxl__ao_device *aodev)
 goto out;
 
 do {
-dev->ops = remus_ops[++dev->ops_index];
+dev->ops = dev->cds->ops[++dev->ops_index];
 if (!dev->ops) {
 libxl_device_nic * nic = NULL;
 libxl_device_disk * disk = NULL;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 0f2c96b..ee415fd 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2897,6 +2897,8 @@ struct libxl__checkpoint_devices_state {
 uint32_t domid;
 libxl__checkpoint_callback *callback;
 int device_kind_flags;
+/* The ops must be pointer array, and the last ops must be NULL. */
+const libxl__checkpoint_device_instance_ops **ops;
 
 /*- private for abstract layer only -*/
 
diff --git a/tools/libxl/libxl_remus.c b/tools/libxl/libxl_remus.c
index d088dad..3375331 100644
--- a/tools/libxl/libxl_remus.c
+++ b/tools/libxl/libxl_remus.c
@@ -18,6 +18,14 @@
 
 #include "libxl_internal.h"
 
+extern const libxl__checkpoint_device_instance_ops remus_device_nic;
+extern const libxl__checkpoint_device_instance_ops remus_device_drbd_disk;
+static const libxl__checkpoint_device_instance_ops *remus_ops[] = {
+_device_nic,
+_device_drbd_disk,
+NULL,
+};
+
 /* Remus setup and teardown -*/
 
 static void remus_setup_done(libxl__egc *egc,
@@ -50,6 +58,7 @@ void libxl__remus_setup(libxl__egc *egc,
 cds->ao = ao;
 cds->domid = dss->domid;
 cds->callback = remus_setup_done;
+cds->ops = remus_ops;
 
 dss->sws.checkpoint_callback = remus_checkpoint_stream_written;
 
-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v7 11/18] tools/libxl: Add back channel to allow migration target send data back

[Wen Congyang]

In COLO mode, secondary needs to send the following data to primary:
1. In libxl
   Secondary sends the following CHECKPOINT_CONTEXT to primary:
   CHECKPOINT_SVM_SUSPENDED, CHECKPOINT_SVM_READY and CHECKPOINT_SVM_RESUMED
2. In libxc
   Secondary sends the dirty pfn list to primary

But the io_fd only can be written in primary, and only can be read in
secondary. Save recv_fd in domain_suspend_state, and send_fd in
domain_create_state. Extend libxl_domain_create_restore API, add a
send_fd param to it. Add LIBXL_HAVE_CREATE_RESTORE_SEND_FD to indicate
the API change.

Signed-off-by: Wen Congyang <we...@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.y...@easystack.cn>
---
 tools/libxl/libxl.c  |  2 +-
 tools/libxl/libxl.h  | 30 --
 tools/libxl/libxl_create.c   |  9 +
 tools/libxl/libxl_internal.h |  2 ++
 tools/libxl/xl_cmdimpl.c |  8 +++-
 tools/ocaml/libs/xl/xenlight_stubs.c |  2 +-
 6 files changed, 44 insertions(+), 9 deletions(-)

diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index fc7844d..e286329 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -871,7 +871,7 @@ int libxl_domain_remus_start(libxl_ctx *ctx, 
libxl_domain_remus_info *info,
 dss->callback = remus_failover_cb;
 dss->domid = domid;
 dss->fd = send_fd;
-/* TODO do something with recv_fd */
+dss->recv_fd = recv_fd;
 dss->type = type;
 dss->live = 1;
 dss->debug = 0;
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 6225db1..5e4aede 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -639,6 +639,15 @@ typedef struct libxl__ctx libxl_ctx;
 #define LIBXL_HAVE_DOMAIN_CREATE_RESTORE_PARAMS 1
 
 /*
+ * LIBXL_HAVE_DOMAIN_CREATE_RESTORE_SEND_FD 1
+ *
+ * If this is defined, libxl_domain_create_restore()'s API has changed to
+ * include a send_fd param which used for libxl migration back channel
+ * during COLO.
+ */
+#define LIBXL_HAVE_DOMAIN_CREATE_RESTORE_SEND_FD 1
+
+/*
  * LIBXL_HAVE_CREATEINFO_PVH
  * If this is defined, then libxl supports creation of a PVH guest.
  */
@@ -1152,7 +1161,7 @@ int libxl_domain_create_new(libxl_ctx *ctx, 
libxl_domain_config *d_config,
 const libxl_asyncprogress_how *aop_console_how)
 LIBXL_EXTERNAL_CALLERS_ONLY;
 int libxl_domain_create_restore(libxl_ctx *ctx, libxl_domain_config *d_config,
-uint32_t *domid, int restore_fd,
+uint32_t *domid, int restore_fd, int send_fd,
 const libxl_domain_restore_params *params,
 const libxl_asyncop_how *ao_how,
 const libxl_asyncprogress_how *aop_console_how)
@@ -1173,7 +1182,7 @@ int static inline libxl_domain_create_restore_0x040200(
 libxl_domain_restore_params_init();
 
 ret = libxl_domain_create_restore(
-ctx, d_config, domid, restore_fd, , ao_how, aop_console_how);
+ctx, d_config, domid, restore_fd, -1, , ao_how, 
aop_console_how);
 
 libxl_domain_restore_params_dispose();
 return ret;
@@ -1181,6 +1190,23 @@ int static inline libxl_domain_create_restore_0x040200(
 
 #define libxl_domain_create_restore libxl_domain_create_restore_0x040200
 
+#elif defined(LIBXL_API_VERSION) && LIBXL_API_VERSION >= 0x040400 \
+ && LIBXL_API_VERSION < 0x040700
+
+int static inline libxl_domain_create_restore_0x040400(
+libxl_ctx *ctx, libxl_domain_config *d_config,
+uint32_t *domid, int restore_fd,
+const libxl_domain_restore_params *params,
+const libxl_asyncop_how *ao_how,
+const libxl_asyncprogress_how *aop_console_how)
+LIBXL_EXTERNAL_CALLERS_ONLY
+{
+return libxl_domain_create_restore(ctx, d_config, domid, restore_fd,
+   -1, params, ao_how, aop_console_how);
+}
+
+#define libxl_domain_create_restore libxl_domain_create_restore_0x040400
+
 #endif
 
 int libxl_domain_soft_reset(libxl_ctx *ctx,
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 0d20c2d..eb869ea 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1563,7 +1563,7 @@ static void domain_create_cb(libxl__egc *egc,
  int rc, uint32_t domid);
 
 static int do_domain_create(libxl_ctx *ctx, libxl_domain_config *d_config,
-uint32_t *domid, int restore_fd,
+uint32_t *domid, int restore_fd, int send_fd,
 const libxl_domain_restore_params *params,
 const libxl_asyncop_how *ao_how,
 const libxl_asyncprogress_how *aop_console_how)
@@ -1578,6 +1578,7 @@ static int do_domain_create(libxl_ctx *ctx, 
libxl_domain_config *d_config