Re: [libvirt] [PATCH v2 06/25] API: Add domain job operation for backups

2019-12-03 Thread Eric Blake 

On 12/3/19 11:17 AM, Peter Krempa wrote:

Introduce VIR_DOMAIN_JOB_OPERATION_BACKUP into virDomainJobOperation.

Signed-off-by: Peter Krempa 
---
  include/libvirt/libvirt-domain.h | 1 +
  tools/virsh-domain.c | 4 +++-
  2 files changed, 4 insertions(+), 1 deletion(-)


Reviewed-by: Eric Blake 



diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h
index 84c5492a7b..6d1c7f1a3b 100644
--- a/include/libvirt/libvirt-domain.h
+++ b/include/libvirt/libvirt-domain.h
@@ -3269,6 +3269,7 @@ typedef enum {
  VIR_DOMAIN_JOB_OPERATION_SNAPSHOT = 6,
  VIR_DOMAIN_JOB_OPERATION_SNAPSHOT_REVERT = 7,
  VIR_DOMAIN_JOB_OPERATION_DUMP = 8,
+VIR_DOMAIN_JOB_OPERATION_BACKUP = 9,

  # ifdef VIR_ENUM_SENTINELS
  VIR_DOMAIN_JOB_OPERATION_LAST
diff --git a/tools/virsh-domain.c b/tools/virsh-domain.c
index bb942267f0..5c313279d7 100644
--- a/tools/virsh-domain.c
+++ b/tools/virsh-domain.c
@@ -6068,7 +6068,9 @@ VIR_ENUM_IMPL(virshDomainJobOperation,
N_("Outgoing migration"),
N_("Snapshot"),
N_("Snapshot revert"),
-  N_("Dump"));
+  N_("Dump"),
+  N_("Backup"),
+);

  static const char *
  virshDomainJobOperationToString(int op)



--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v2 01/25] qemu: domain: Export qemuDomainGetImageIds

2019-12-03 Thread Eric Blake 

On 12/3/19 11:17 AM, Peter Krempa wrote:

Signed-off-by: Peter Krempa 
---
  src/qemu/qemu_domain.c | 2 +-
  src/qemu/qemu_domain.h | 7 +++
  2 files changed, 8 insertions(+), 1 deletion(-)



Reviewed-by: Eric Blake 

--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH 2/2] Add -mem-shared option

2019-12-03 Thread Eduardo Habkost 
On Tue, Dec 03, 2019 at 09:56:15AM +0100, Thomas Huth wrote:
> On 02/12/2019 22.00, Eduardo Habkost wrote:
> > On Mon, Dec 02, 2019 at 08:39:48AM +0100, Igor Mammedov wrote:
> >> On Fri, 29 Nov 2019 18:46:12 +0100
> >> Paolo Bonzini  wrote:
> >>
> >>> On 29/11/19 13:16, Igor Mammedov wrote:
>  As for "-m", I'd make it just an alias that translates
>   -m/mem-path/mem-prealloc  
> >>>
> >>> I think we should just deprecate -mem-path/-mem-prealloc in 5.0.  CCing
> >>> Thomas as mister deprecation. :)
> >>
> >> I'll add that to my series
> > 
> > Considering that the plan is to eventually reimplement those
> > options as syntactic sugar for memory backend options (hopefully
> > in less than 2 QEMU releases), what's the point of deprecating
> > them?
> 
> Well, it depends on the "classification" [1] of the parameter...
> 
> Let's ask: What's the main purpose of the option?
> 
> Is it easier to use than the "full" option, and thus likely to be used
> by a lot of people who run QEMU directly from the CLI? In that case it
> should stay as "convenience option" and not be deprecated.
> 
> Or is the option merely there to give the upper layers like libvirt or
> some few users and their scripts some more grace period to adapt their
> code, but we all agree that the options are rather ugly and should
> finally go away? Then it's rather a "legacy option" and the deprecation
> process is the right way to go. Our QEMU interface is still way to
> overcrowded, we should try to keep it as clean as possible.

That's a good way to describe the questions involved.  To me they
are clearly convenience options.

We could still replace them with new (more consistent and less
ugly) convenience options, though.

> 
>  Thomas
> 
> 
> [1] Using the terms from:
> https://www.youtube.com/watch?v=Oscjpkns7tM=8m

-- 
Eduardo

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH] tests: stop linking virt tests to secondary drivers

2019-12-03 Thread Laine Stump 

On 12/3/19 12:30 PM, Daniel P. Berrangé wrote:

The hard dependancy between the virt drivers and the network
or storage drivers was removed quite a while back now, so
the tests no longer need to link to these drivers.

Signed-off-by: Daniel P. Berrangé 
---



Reviewed-by: Laine Stump 



--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH] maint: Post-release version bump to 5.11.0

2019-12-03 Thread Jim Fehlig 
On 12/3/19 1:35 AM, Peter Krempa wrote:
> On Tue, Dec 03, 2019 at 04:24:59 +, Jim Fehlig wrote:
>> Signed-off-by: Jim Fehlig 
>> ---
>>
>> Pushing under the trivial rule.
>>
>>   configure.ac  | 2 +-
>>   docs/news.xml | 8 
>>   2 files changed, 9 insertions(+), 1 deletion(-)
>>
>> diff --git a/configure.ac b/configure.ac
>> index ef521e370c..d0ef7de750 100644
>> --- a/configure.ac
>> +++ b/configure.ac
>> @@ -16,7 +16,7 @@ dnl You should have received a copy of the GNU Lesser 
>> General Public
>>   dnl License along with this library.  If not, see
>>   dnl .
>>   
>> -AC_INIT([libvirt], [5.10.0], [libvir-list@redhat.com], [], 
>> [https://libvirt.org])
>> +AC_INIT([libvirt], [5.11.0], [libvir-list@redhat.com], [], 
>> [https://libvirt.org])
> 
> The next release is going to be 6.0.0

And I thought it would be impossible to screw this up - not so. I see Jan has 
already fixed it. Sorry.

Regards,
Jim

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH] tools: skip libvirt-guests fast if libvirtd is not active

2019-12-03 Thread Doug Smythies 
On 2019.12.03 00:56 Christian Ehrhardt wrote:

> The most common operation of libvirt-guests is to manage the local
> libvirtd. But users might have disabled that and while we are
> After=libvirtd for ordering we are not Requiring it..
> OTOH adding that or any harder dependency might affect our ordering.
>
> But if people have disabled libvirt they will do a full retry loop
> until timeout. Lets check if the local service is active at all and skip
> fast if it is not.
>
> Fixes: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1854653
>
> Reported-by: Doug Smythies 
> Signed-off-by: Christian Ehrhardt 

Tested-by: Doug Smythies 

Thanks.

> ---
>  tools/libvirt-guests.sh.in | 8 
>  1 file changed, 8 insertions(+)
>
> diff --git a/tools/libvirt-guests.sh.in b/tools/libvirt-guests.sh.in
> index 4bc6e866f0..5a9930ee2f 100644
> --- a/tools/libvirt-guests.sh.in
> +++ b/tools/libvirt-guests.sh.in
> @@ -90,6 +90,14 @@ test_connect()
>  {
>  uri=$1
>  
> +if [ "x$uri" = xdefault ]; then
> +# Default config is most common and for the local libvirtd
> +# Check if it is active before wasting time in connect loop
> +if ! systemctl -q is-active libvirtd; then
> +return 1
> +fi
> +fi
> +
>  i=${CONNECT_RETRIES}
>  while [ $i -gt 0 ]; do
>  run_virsh "$uri" connect 2>/dev/null
> -- 
> 2.24.0



--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH] tests: stop linking virt tests to secondary drivers

2019-12-03 Thread Daniel P . Berrangé 
The hard dependancy between the virt drivers and the network
or storage drivers was removed quite a while back now, so
the tests no longer need to link to these drivers.

Signed-off-by: Daniel P. Berrangé 
---
 tests/Makefile.am | 33 -
 1 file changed, 12 insertions(+), 21 deletions(-)

diff --git a/tests/Makefile.am b/tests/Makefile.am
index e009de830c..ac4c0e1a3a 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -500,11 +500,10 @@ sockettest_SOURCES = \
 sockettest_LDADD = $(LDADDS)
 
 if WITH_LIBXL
-libxl_LDADDS = ../src/libvirt_driver_libxl_impl.la
-if WITH_NETWORK
-libxl_LDADDS += ../src/libvirt_driver_network_impl.la
-endif WITH_NETWORK
-libxl_LDADDS += $(LDADDS)
+libxl_LDADDS = \
+   ../src/libvirt_driver_libxl_impl.la \
+   $(LDADDS) \
+   $(NULL)
 
 libxltestdriver_la_SOURCES =
 libxltestdriver_la_LDFLAGS = $(DRIVERLIB_LDFLAGS)
@@ -547,12 +546,6 @@ if WITH_QEMU
 libqemumonitortestutils_la_SOURCES = $(QEMUMONITORTESTUTILS_SOURCES)
 
 qemu_LDADDS = ../src/libvirt_driver_qemu_impl.la
-if WITH_NETWORK
-qemu_LDADDS += ../src/libvirt_driver_network_impl.la
-endif WITH_NETWORK
-if WITH_STORAGE
-qemu_LDADDS += ../src/libvirt_driver_storage_impl.la
-endif WITH_STORAGE
 if WITH_DTRACE_PROBES
 qemu_LDADDS += ../src/libvirt_qemu_probes.lo
 endif WITH_DTRACE_PROBES
@@ -733,11 +726,10 @@ endif ! WITH_QEMU
 
 if WITH_LXC
 
-lxc_LDADDS = ../src/libvirt_driver_lxc_impl.la
-if WITH_NETWORK
-lxc_LDADDS += ../src/libvirt_driver_network_impl.la
-endif WITH_NETWORK
-lxc_LDADDS += $(LDADDS)
+lxc_LDADDS = \
+   ../src/libvirt_driver_lxc_impl.la \
+   $(LDADDS) \
+   $(NULL)
 
 lxcxml2xmltest_SOURCES = \
lxcxml2xmltest.c testutilslxc.c testutilslxc.h \
@@ -818,11 +810,10 @@ libbhyveargv2xmlmock_la_SOURCES = \
 libbhyveargv2xmlmock_la_LDFLAGS = $(MOCKLIBS_LDFLAGS)
 libbhyveargv2xmlmock_la_LIBADD = $(MOCKLIBS_LIBS)
 
-bhyve_LDADDS = ../src/libvirt_driver_bhyve_impl.la
-if WITH_STORAGE
-bhyve_LDADDS += ../src/libvirt_driver_storage_impl.la
-endif WITH_STORAGE
-bhyve_LDADDS += $(LDADDS)
+bhyve_LDADDS = \
+   ../src/libvirt_driver_bhyve_impl.la \
+   $(LDADDS) \
+   $(NULL)
 bhyvexml2argvtest_SOURCES = \
bhyvexml2argvtest.c \
testutils.c testutils.h
-- 
2.21.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH v2 03/25] virsh: Implement VIR_DOMAIN_JOB_DISK_TEMP_(USED|TOTAL) in cmdDomjobinfo

2019-12-03 Thread Peter Krempa 
Signed-off-by: Peter Krempa 
Reviewed-by: Eric Blake 
Reviewed-by: Daniel P. Berrangé 
---
 tools/virsh-domain.c | 18 ++
 1 file changed, 18 insertions(+)

diff --git a/tools/virsh-domain.c b/tools/virsh-domain.c
index 21ea1a69ea..bb942267f0 100644
--- a/tools/virsh-domain.c
+++ b/tools/virsh-domain.c
@@ -6388,6 +6388,24 @@ cmdDomjobinfo(vshControl *ctl, const vshCmd *cmd)
 vshPrint(ctl, "%-17s %-13d\n", _("Auto converge throttle:"), ivalue);
 }

+if ((rc = virTypedParamsGetULLong(params, nparams,
+  VIR_DOMAIN_JOB_DISK_TEMP_USED,
+  )) < 0) {
+goto save_error;
+} else if (rc) {
+val = vshPrettyCapacity(value, );
+vshPrint(ctl, "%-17s %-.3lf %s\n", _("Temporary disk space use:"), 
val, unit);
+}
+
+if ((rc = virTypedParamsGetULLong(params, nparams,
+  VIR_DOMAIN_JOB_DISK_TEMP_TOTAL,
+  )) < 0) {
+goto save_error;
+} else if (rc) {
+val = vshPrettyCapacity(value, );
+vshPrint(ctl, "%-17s %-.3lf %s\n", _("Temporary disk space total:"), 
val, unit);
+}
+
 ret = true;

  cleanup:
-- 
2.23.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH v2 15/25] qemu: domain: Track backup job data in the status XML

2019-12-03 Thread Peter Krempa 
Store the data of a backup job along with the index counter for new
backup jobs in the status XML. Currently we will support only one
backup job and thus there's no necessity to add arrays of jobs.

Signed-off-by: Peter Krempa 
---
 src/qemu/qemu_domain.c | 58 ++
 src/qemu/qemu_domain.h |  3 +++
 2 files changed, 61 insertions(+)

diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c
index 980287d2a0..98d0dad861 100644
--- a/src/qemu/qemu_domain.c
+++ b/src/qemu/qemu_domain.c
@@ -62,6 +62,7 @@
 #include "locking/domain_lock.h"
 #include "virdomainsnapshotobjlist.h"
 #include "virdomaincheckpointobjlist.h"
+#include "backup_conf.h"

 #ifdef MAJOR_IN_MKDEV
 # include 
@@ -2236,6 +2237,9 @@ qemuDomainObjPrivateDataClear(qemuDomainObjPrivatePtr 
priv)
 priv->pflash0 = NULL;
 virObjectUnref(priv->pflash1);
 priv->pflash1 = NULL;
+
+virDomainBackupDefFree(priv->backup);
+priv->backup = NULL;
 }


@@ -2643,6 +2647,26 @@ qemuDomainObjPrivateXMLFormatBlockjobs(virBufferPtr buf,
 }


+static int
+qemuDomainObjPrivateXMLFormatBackups(virBufferPtr buf,
+ virDomainObjPtr vm)
+{
+qemuDomainObjPrivatePtr priv = vm->privateData;
+g_auto(virBuffer) attrBuf = VIR_BUFFER_INITIALIZER;
+g_auto(virBuffer) childBuf = VIR_BUFFER_INIT_CHILD(buf);
+
+if (!virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_INCREMENTAL_BACKUP))
+return 0;
+
+if (priv->backup &&
+virDomainBackupDefFormat(, priv->backup, true) < 0)
+return -1;
+
+virXMLFormatElement(buf, "backups", , );
+return 0;
+}
+
+
 void
 qemuDomainObjPrivateXMLFormatAllowReboot(virBufferPtr buf,
  virTristateBool allowReboot)
@@ -2938,6 +2962,9 @@ qemuDomainObjPrivateXMLFormat(virBufferPtr buf,

 virBufferAsprintf(buf, "%i\n", 
priv->agentTimeout);

+if (qemuDomainObjPrivateXMLFormatBackups(buf, vm) < 0)
+return -1;
+
 return 0;
 }

@@ -3311,6 +3338,34 @@ qemuDomainObjPrivateXMLParseBlockjobs(virDomainObjPtr vm,
 }


+static int
+qemuDomainObjPrivateXMLParseBackups(qemuDomainObjPrivatePtr priv,
+xmlXPathContextPtr ctxt)
+{
+g_autofree xmlNodePtr *nodes = NULL;
+ssize_t nnodes = 0;
+
+if ((nnodes = virXPathNodeSet("./backups/domainbackup", ctxt, )) < 0)
+return -1;
+
+if (nnodes > 1) {
+virReportError(VIR_ERR_INTERNAL_ERROR, "%s",
+   _("only one backup job is supported"));
+return -1;
+}
+
+if (nnodes == 0)
+return 0;
+
+if (!(priv->backup = virDomainBackupDefParseNode(ctxt->doc, nodes[0],
+ priv->driver->xmlopt,
+ 
VIR_DOMAIN_BACKUP_PARSE_INTERNAL)))
+return -1;
+
+return 0;
+}
+
+
 int
 qemuDomainObjPrivateXMLParseAllowReboot(xmlXPathContextPtr ctxt,
 virTristateBool *allowReboot)
@@ -3740,6 +3795,9 @@ qemuDomainObjPrivateXMLParse(xmlXPathContextPtr ctxt,
 if (qemuDomainObjPrivateXMLParseBlockjobs(vm, priv, ctxt) < 0)
 goto error;

+if (qemuDomainObjPrivateXMLParseBackups(priv, ctxt) < 0)
+goto error;
+
 qemuDomainStorageIdReset(priv);
 if (virXPathULongLong("string(./nodename/@index)", ctxt,
   >nodenameindex) == -2) {
diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h
index 4cd7eec4ce..e07c8aa58f 100644
--- a/src/qemu/qemu_domain.h
+++ b/src/qemu/qemu_domain.h
@@ -414,6 +414,9 @@ struct _qemuDomainObjPrivate {
  * commandline for pflash drives. */
 virStorageSourcePtr pflash0;
 virStorageSourcePtr pflash1;
+
+/* running backup job */
+virDomainBackupDefPtr backup;
 };

 #define QEMU_DOMAIN_PRIVATE(vm) \
-- 
2.23.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH v2 18/25] conf: backup: Add fields for tracking stats of completed sub-jobs

2019-12-03 Thread Peter Krempa 
We need a place to store stats of completed sub-jobs so that we can
later report accurate stats.

Signed-off-by: Peter Krempa 
---
 src/conf/backup_conf.h | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/src/conf/backup_conf.h b/src/conf/backup_conf.h
index c970e01920..5dfc42e297 100644
--- a/src/conf/backup_conf.h
+++ b/src/conf/backup_conf.h
@@ -70,6 +70,13 @@ struct _virDomainBackupDef {

 size_t ndisks; /* should not exceed dom->ndisks */
 virDomainBackupDiskDef *disks;
+
+/* internal data */
+/* statistic totals for completed diks */
+unsigned long long push_transferred;
+unsigned long long push_total;
+unsigned long long pull_tmp_used;
+unsigned long long pull_tmp_total;
 };

 typedef enum {
-- 
2.23.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH v2 08/25] backup: Parse and output backup XML

2019-12-03 Thread Peter Krempa 
From: Eric Blake 

Accept XML describing a generic block job, and output it again as
needed. This may still need a few tweaks to match the documented XML
and RNG schema.

Signed-off-by: Eric Blake 
---
 po/POTFILES.in   |   1 +
 src/conf/Makefile.inc.am |   2 +
 src/conf/backup_conf.c   | 499 +++
 src/conf/backup_conf.h   | 101 
 src/conf/virconftypes.h  |   3 +
 src/libvirt_private.syms |   8 +
 6 files changed, 614 insertions(+)
 create mode 100644 src/conf/backup_conf.c
 create mode 100644 src/conf/backup_conf.h

diff --git a/po/POTFILES.in b/po/POTFILES.in
index debb51cd70..0ff3beeb7e 100644
--- a/po/POTFILES.in
+++ b/po/POTFILES.in
@@ -19,6 +19,7 @@
 @SRCDIR@/src/bhyve/bhyve_monitor.c
 @SRCDIR@/src/bhyve/bhyve_parse_command.c
 @SRCDIR@/src/bhyve/bhyve_process.c
+@SRCDIR@/src/conf/backup_conf.c
 @SRCDIR@/src/conf/capabilities.c
 @SRCDIR@/src/conf/checkpoint_conf.c
 @SRCDIR@/src/conf/cpu_conf.c
diff --git a/src/conf/Makefile.inc.am b/src/conf/Makefile.inc.am
index 5035b9b524..debc6f4eef 100644
--- a/src/conf/Makefile.inc.am
+++ b/src/conf/Makefile.inc.am
@@ -12,6 +12,8 @@ NETDEV_CONF_SOURCES = \
$(NULL)

 DOMAIN_CONF_SOURCES = \
+   conf/backup_conf.c \
+   conf/backup_conf.h \
conf/capabilities.c \
conf/capabilities.h \
conf/checkpoint_conf.c \
diff --git a/src/conf/backup_conf.c b/src/conf/backup_conf.c
new file mode 100644
index 00..aaafdf12a2
--- /dev/null
+++ b/src/conf/backup_conf.c
@@ -0,0 +1,499 @@
+/*
+ * backup_conf.c: domain backup XML processing
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library.  If not, see
+ * .
+ */
+
+#include 
+
+#include "configmake.h"
+#include "internal.h"
+#include "virbuffer.h"
+#include "datatypes.h"
+#include "domain_conf.h"
+#include "virlog.h"
+#include "viralloc.h"
+#include "backup_conf.h"
+#include "virstoragefile.h"
+#include "virfile.h"
+#include "virerror.h"
+#include "virxml.h"
+#include "virstring.h"
+#include "virhash.h"
+#include "virenum.h"
+
+#define VIR_FROM_THIS VIR_FROM_DOMAIN
+
+VIR_LOG_INIT("conf.backup_conf");
+
+VIR_ENUM_DECL(virDomainBackup);
+VIR_ENUM_IMPL(virDomainBackup,
+  VIR_DOMAIN_BACKUP_TYPE_LAST,
+  "default",
+  "push",
+  "pull");
+
+/* following values appear in the status XML */
+VIR_ENUM_DECL(virDomainBackupDiskState);
+VIR_ENUM_IMPL(virDomainBackupDiskState,
+  VIR_DOMAIN_BACKUP_DISK_STATE_LAST,
+  "",
+  "running",
+  "complete",
+  "failed",
+  "cancelling",
+  "cancelled");
+
+void
+virDomainBackupDefFree(virDomainBackupDefPtr def)
+{
+size_t i;
+
+if (!def)
+return;
+
+g_free(def->incremental);
+virStorageNetHostDefFree(1, def->server);
+
+for (i = 0; i < def->ndisks; i++) {
+virDomainBackupDiskDefPtr disk = def->disks + i;
+
+g_free(disk->name);
+virObjectUnref(disk->store);
+}
+
+g_free(def->disks);
+g_free(def);
+}
+
+
+static int
+virDomainBackupDiskDefParseXML(xmlNodePtr node,
+   xmlXPathContextPtr ctxt,
+   virDomainBackupDiskDefPtr def,
+   bool push,
+   unsigned int flags,
+   virDomainXMLOptionPtr xmlopt)
+{
+VIR_XPATH_NODE_AUTORESTORE(ctxt);
+g_autofree char *type = NULL;
+g_autofree char *driver = NULL;
+g_autofree char *backup = NULL;
+g_autofree char *state = NULL;
+int tmp;
+xmlNodePtr srcNode;
+unsigned int storageSourceParseFlags = 0;
+bool internal = flags & VIR_DOMAIN_BACKUP_PARSE_INTERNAL;
+
+if (internal)
+storageSourceParseFlags = VIR_DOMAIN_DEF_PARSE_STATUS;
+
+ctxt->node = node;
+
+if (!(def->name = virXMLPropString(node, "name"))) {
+virReportError(VIR_ERR_XML_ERROR, "%s",
+   _("missing name from disk backup element"));
+return -1;
+}
+
+def->backup = VIR_TRISTATE_BOOL_YES;
+
+if ((backup = virXMLPropString(node, "backup"))) {
+if ((tmp = virTristateBoolTypeFromString(backup)) <= 0) {
+virReportError(VIR_ERR_XML_ERROR,
+   _("invalid disk 'backup' state '%s'"), backup);
+return -1;
+}
+
+

[libvirt] [PATCH v2 25/25] qemu: Add support for VIR_DOMAIN_CAPS_FEATURE_BACKUP

2019-12-03 Thread Peter Krempa 
Signed-off-by: Peter Krempa 
---
 src/qemu/qemu_capabilities.c  | 1 +
 tests/domaincapsdata/qemu_1.5.3-q35.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_1.5.3-tcg.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_1.5.3.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_1.6.0-q35.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_1.6.0-tcg.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_1.6.0.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_1.7.0-q35.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_1.7.0-tcg.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_1.7.0.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_2.1.1-q35.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_2.1.1-tcg.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_2.1.1.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_2.10.0-q35.x86_64.xml   | 1 +
 tests/domaincapsdata/qemu_2.10.0-tcg.x86_64.xml   | 1 +
 tests/domaincapsdata/qemu_2.10.0-virt.aarch64.xml | 1 +
 tests/domaincapsdata/qemu_2.10.0.aarch64.xml  | 1 +
 tests/domaincapsdata/qemu_2.10.0.ppc64.xml| 1 +
 tests/domaincapsdata/qemu_2.10.0.s390x.xml| 1 +
 tests/domaincapsdata/qemu_2.10.0.x86_64.xml   | 1 +
 tests/domaincapsdata/qemu_2.11.0-q35.x86_64.xml   | 1 +
 tests/domaincapsdata/qemu_2.11.0-tcg.x86_64.xml   | 1 +
 tests/domaincapsdata/qemu_2.11.0.s390x.xml| 1 +
 tests/domaincapsdata/qemu_2.11.0.x86_64.xml   | 1 +
 tests/domaincapsdata/qemu_2.12.0-q35.x86_64.xml   | 1 +
 tests/domaincapsdata/qemu_2.12.0-tcg.x86_64.xml   | 1 +
 tests/domaincapsdata/qemu_2.12.0-virt.aarch64.xml | 1 +
 tests/domaincapsdata/qemu_2.12.0.aarch64.xml  | 1 +
 tests/domaincapsdata/qemu_2.12.0.ppc64.xml| 1 +
 tests/domaincapsdata/qemu_2.12.0.s390x.xml| 1 +
 tests/domaincapsdata/qemu_2.12.0.x86_64.xml   | 1 +
 tests/domaincapsdata/qemu_2.4.0-q35.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_2.4.0-tcg.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_2.4.0.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_2.5.0-q35.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_2.5.0-tcg.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_2.5.0.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_2.6.0-q35.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_2.6.0-tcg.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_2.6.0-virt.aarch64.xml  | 1 +
 tests/domaincapsdata/qemu_2.6.0.aarch64.xml   | 1 +
 tests/domaincapsdata/qemu_2.6.0.ppc64.xml | 1 +
 tests/domaincapsdata/qemu_2.6.0.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_2.7.0-q35.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_2.7.0-tcg.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_2.7.0.s390x.xml | 1 +
 tests/domaincapsdata/qemu_2.7.0.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_2.8.0-q35.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_2.8.0-tcg.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_2.8.0.s390x.xml | 1 +
 tests/domaincapsdata/qemu_2.8.0.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_2.9.0-q35.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_2.9.0-tcg.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_2.9.0.ppc64.xml | 1 +
 tests/domaincapsdata/qemu_2.9.0.s390x.xml | 1 +
 tests/domaincapsdata/qemu_2.9.0.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_3.0.0-q35.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_3.0.0-tcg.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_3.0.0.ppc64.xml | 1 +
 tests/domaincapsdata/qemu_3.0.0.s390x.xml | 1 +
 tests/domaincapsdata/qemu_3.0.0.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_3.1.0-q35.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_3.1.0-tcg.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_3.1.0.ppc64.xml | 1 +
 tests/domaincapsdata/qemu_3.1.0.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_4.0.0-q35.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_4.0.0-tcg.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_4.0.0-virt.aarch64.xml  | 1 +
 tests/domaincapsdata/qemu_4.0.0.aarch64.xml   | 1 +
 tests/domaincapsdata/qemu_4.0.0.ppc64.xml | 1 +
 tests/domaincapsdata/qemu_4.0.0.s390x.xml | 1 +
 tests/domaincapsdata/qemu_4.0.0.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_4.1.0-q35.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_4.1.0-tcg.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_4.1.0.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_4.2.0-q35.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_4.2.0-tcg.x86_64.xml| 1 +
 tests/domaincapsdata/qemu_4.2.0-virt.aarch64.xml  | 1 +
 tests/domaincapsdata/qemu_4.2.0.aarch64.xml   | 1 +
 tests/domaincapsdata/qemu_4.2.0.ppc64.xml | 1 +
 tests/domaincapsdata/qemu_4.2.0.s390x.xml | 1 +
 tests/domaincapsdata/qemu_4.2.0.x86_64.xml| 1 +
 82 files changed, 82 insertions(+)

diff --git a/src/qemu/qemu_capabilities.c b/src/qemu/qemu_capabilities.c
index edb128c881..3b9e4561fa 100644
--- a/src/qemu/qemu_capabilities.c
+++ b/src/qemu/qemu_capabilities.c
@@ -5458,6 +5458,7 @@ static const struct

[libvirt] [PATCH v2 21/25] qemu: backup: Implement stats gathering while the job is running

2019-12-03 Thread Peter Krempa 
We can use the output of 'query-jobs' to figure out some useful
information about a backup job. That is progress in case of a push job
and scratch file use in case of a pull job.

Add a worker which will total up the data and call it from
qemuDomainGetJobStatsInternal.

Signed-off-by: Peter Krempa 
---
 src/qemu/qemu_backup.c | 98 ++
 src/qemu/qemu_backup.h |  5 +++
 src/qemu/qemu_driver.c |  3 +-
 3 files changed, 105 insertions(+), 1 deletion(-)

diff --git a/src/qemu/qemu_backup.c b/src/qemu/qemu_backup.c
index 8307a42e1c..452f1b0a4d 100644
--- a/src/qemu/qemu_backup.c
+++ b/src/qemu/qemu_backup.c
@@ -939,3 +939,101 @@ qemuBackupNotifyBlockjobEnd(virDomainObjPtr vm,

 /* otherwise we must wait for the jobs to end */
 }
+
+
+static void
+qemuBackupGetJobInfoStatsUpdateOne(virDomainObjPtr vm,
+   bool push,
+   const char *diskdst,
+   qemuDomainBackupStats *stats,
+   qemuMonitorJobInfoPtr *blockjobs,
+   size_t nblockjobs)
+{
+virDomainDiskDefPtr domdisk;
+qemuMonitorJobInfoPtr monblockjob = NULL;
+g_autoptr(qemuBlockJobData) diskblockjob = NULL;
+size_t i;
+
+/* it's just statistics so let's not worry so much about errors */
+if (!(domdisk = virDomainDiskByTarget(vm->def, diskdst)))
+return;
+
+if (!(diskblockjob = qemuBlockJobDiskGetJob(domdisk)))
+return;
+
+for (i = 0; i < nblockjobs; i++) {
+if (STREQ_NULLABLE(blockjobs[i]->id, diskblockjob->name)) {
+monblockjob = blockjobs[i];
+break;
+}
+}
+if (!monblockjob)
+return;
+
+if (push) {
+stats->total += monblockjob->progressTotal;
+stats->transferred += monblockjob->progressCurrent;
+} else {
+stats->tmp_used += monblockjob->progressCurrent;
+stats->tmp_total += monblockjob->progressTotal;
+}
+}
+
+
+int
+qemuBackupGetJobInfoStats(virQEMUDriverPtr driver,
+  virDomainObjPtr vm,
+  qemuDomainJobInfoPtr jobInfo)
+{
+qemuDomainBackupStats *stats = >stats.backup;
+qemuDomainObjPrivatePtr priv = vm->privateData;
+qemuMonitorJobInfoPtr *blockjobs = NULL;
+size_t nblockjobs = 0;
+size_t i;
+int rc;
+int ret = -1;
+
+if (!priv->backup) {
+virReportError(VIR_ERR_INTERNAL_ERROR, "%s",
+   _("backup job data missing"));
+return -1;
+}
+
+if (qemuDomainJobInfoUpdateTime(jobInfo) < 0)
+return -1;
+
+jobInfo->status = QEMU_DOMAIN_JOB_STATUS_ACTIVE;
+
+qemuDomainObjEnterMonitor(driver, vm);
+
+rc = qemuMonitorGetJobInfo(priv->mon, , );
+
+if (qemuDomainObjExitMonitor(driver, vm) < 0 || rc < 0)
+goto cleanup;
+
+/* count in completed jobs */
+stats->total = priv->backup->push_total;
+stats->transferred = priv->backup->push_transferred;
+stats->tmp_used = priv->backup->pull_tmp_used;
+stats->tmp_total = priv->backup->pull_tmp_total;
+
+for (i = 0; i < priv->backup->ndisks; i++) {
+if (priv->backup->disks[i].state != 
VIR_DOMAIN_BACKUP_DISK_STATE_RUNNING)
+continue;
+
+qemuBackupGetJobInfoStatsUpdateOne(vm,
+   priv->backup->type == 
VIR_DOMAIN_BACKUP_TYPE_PUSH,
+   priv->backup->disks[i].name,
+   stats,
+   blockjobs,
+   nblockjobs);
+}
+
+ret = 0;
+
+ cleanup:
+for (i = 0; i < nblockjobs; i++)
+qemuMonitorJobInfoFree(blockjobs[i]);
+g_free(blockjobs);
+return ret;
+}
diff --git a/src/qemu/qemu_backup.h b/src/qemu/qemu_backup.h
index 96297fc9e4..0f76abe067 100644
--- a/src/qemu/qemu_backup.h
+++ b/src/qemu/qemu_backup.h
@@ -39,3 +39,8 @@ qemuBackupNotifyBlockjobEnd(virDomainObjPtr vm,
 qemuBlockjobState state,
 unsigned long long cur,
 unsigned long long end);
+
+int
+qemuBackupGetJobInfoStats(virQEMUDriverPtr driver,
+  virDomainObjPtr vm,
+  qemuDomainJobInfoPtr jobInfo);
diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c
index 00359d37e9..95882d9d14 100644
--- a/src/qemu/qemu_driver.c
+++ b/src/qemu/qemu_driver.c
@@ -13907,7 +13907,8 @@ qemuDomainGetJobStatsInternal(virQEMUDriverPtr driver,
 break;

 case QEMU_DOMAIN_JOB_STATS_TYPE_BACKUP:
-/* TODO implement for backup job */
+if (qemuBackupGetJobInfoStats(driver, vm, jobInfo) < 0)
+goto cleanup;
 break;

 case QEMU_DOMAIN_JOB_STATS_TYPE_NONE:
-- 
2.23.0

--
libvir-list mailing list
libvir-list@redhat.com

Re: [libvirt] [PATCH] spec: mingw: add deps for genprotocol.pl usage

2019-12-03 Thread Daniel P . Berrangé 
On Tue, Dec 03, 2019 at 12:11:50PM -0500, Cole Robinson wrote:
> As of 0985a9597bb03 we no longer distribute generated files. mingw
> needs to call genprotocol.pl itself, which means it needs rpcgen
> and cpp installed.
> 
> Signed-off-by: Cole Robinson 
> ---
>  mingw-libvirt.spec.in | 4 
>  1 file changed, 4 insertions(+)

Reviewed-by: Daniel P. Berrangé 


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH v2 23/25] qemu: blockjob: Implement concluded blockjob handler for backup blockjobs

2019-12-03 Thread Peter Krempa 
After the individual sub-blockjobs of a backup libvirt job finish we
must detect it and notify the parent job, so that it can be properly
terminated.

Since we update job information to determine success of an blockjob we
can directly report back also statistics of the blockjob.

Signed-off-by: Peter Krempa 
---
 src/qemu/qemu_blockjob.c | 74 ++--
 1 file changed, 72 insertions(+), 2 deletions(-)

diff --git a/src/qemu/qemu_blockjob.c b/src/qemu/qemu_blockjob.c
index d434b8bddd..a6b0af182c 100644
--- a/src/qemu/qemu_blockjob.c
+++ b/src/qemu/qemu_blockjob.c
@@ -27,6 +27,7 @@
 #include "qemu_block.h"
 #include "qemu_domain.h"
 #include "qemu_alias.h"
+#include "qemu_backup.h"

 #include "conf/domain_conf.h"
 #include "conf/domain_event.h"
@@ -1272,11 +1273,71 @@ 
qemuBlockJobProcessEventConcludedCreate(virQEMUDriverPtr driver,
 }


+static void
+qemuBlockJobProcessEventConcludedBackup(virQEMUDriverPtr driver,
+virDomainObjPtr vm,
+qemuBlockJobDataPtr job,
+qemuDomainAsyncJob asyncJob,
+qemuBlockjobState newstate,
+unsigned long long progressCurrent,
+unsigned long long progressTotal)
+{
+g_autoptr(virQEMUDriverConfig) cfg = virQEMUDriverGetConfig(driver);
+uid_t uid;
+gid_t gid;
+g_autoptr(qemuBlockStorageSourceAttachData) backend = NULL;
+g_autoptr(virJSONValue) actions = NULL;
+
+qemuBackupNotifyBlockjobEnd(vm, job->disk, newstate, progressCurrent, 
progressTotal);
+
+if (job->data.backup.store &&
+!(backend = 
qemuBlockStorageSourceDetachPrepare(job->data.backup.store, NULL)))
+return;
+
+if (job->data.backup.bitmap) {
+if (!(actions = virJSONValueNewArray()))
+return;
+
+if (qemuMonitorTransactionBitmapRemove(actions,
+   job->disk->src->nodeformat,
+   job->data.backup.bitmap) < 0)
+return;
+}
+
+if (qemuDomainObjEnterMonitorAsync(driver, vm, asyncJob) < 0)
+return;
+
+if (backend)
+qemuBlockStorageSourceAttachRollback(qemuDomainGetMonitor(vm), 
backend);
+
+if (actions)
+qemuMonitorTransaction(qemuDomainGetMonitor(vm), );
+
+if (qemuDomainObjExitMonitor(driver, vm) < 0)
+return;
+
+if (job->data.backup.store) {
+qemuDomainStorageSourceAccessRevoke(driver, vm, 
job->data.backup.store);
+
+if (job->data.backup.deleteStore &&
+job->data.backup.store->type == VIR_STORAGE_TYPE_FILE) {
+qemuDomainGetImageIds(cfg, vm, job->data.backup.store, NULL, , 
);
+
+if (virFileRemove(job->data.backup.store->path, uid, gid) < 0)
+VIR_WARN("failed to remove scratch file '%s'",
+ job->data.backup.store->path);
+}
+}
+}
+
+
 static void
 qemuBlockJobEventProcessConcludedTransition(qemuBlockJobDataPtr job,
 virQEMUDriverPtr driver,
 virDomainObjPtr vm,
-qemuDomainAsyncJob asyncJob)
+qemuDomainAsyncJob asyncJob,
+unsigned long long progressCurrent,
+unsigned long long progressTotal)
 {
 bool success = job->newstate == QEMU_BLOCKJOB_STATE_COMPLETED;

@@ -1310,6 +1371,9 @@ 
qemuBlockJobEventProcessConcludedTransition(qemuBlockJobDataPtr job,
 break;

 case QEMU_BLOCKJOB_TYPE_BACKUP:
+qemuBlockJobProcessEventConcludedBackup(driver, vm, job, asyncJob,
+job->newstate, progressCurrent,
+progressTotal);
 break;

 case QEMU_BLOCKJOB_TYPE_BROKEN:
@@ -1336,6 +1400,8 @@ qemuBlockJobEventProcessConcluded(qemuBlockJobDataPtr job,
 size_t njobinfo = 0;
 size_t i;
 bool refreshed = false;
+unsigned long long progressCurrent = 0;
+unsigned long long progressTotal = 0;

 if (qemuDomainObjEnterMonitorAsync(driver, vm, asyncJob) < 0)
 goto cleanup;
@@ -1348,6 +1414,9 @@ qemuBlockJobEventProcessConcluded(qemuBlockJobDataPtr job,
 if (STRNEQ_NULLABLE(job->name, jobinfo[i]->id))
 continue;

+progressCurrent = jobinfo[i]->progressCurrent;
+progressTotal = jobinfo[i]->progressTotal;
+
 job->errmsg = g_strdup(jobinfo[i]->error);

 if (job->errmsg)
@@ -1380,7 +1449,8 @@ qemuBlockJobEventProcessConcluded(qemuBlockJobDataPtr job,

 VIR_DEBUG("handling job '%s' state '%d' newstate '%d'", job->name, 
job->state, job->newstate);

-

[libvirt] [PATCH v2 11/25] qemu: Add infrastructure for statistics of a backup job

2019-12-03 Thread Peter Krempa 
Introduce QEMU_DOMAIN_JOB_STATS_TYPE_BACKUP and the convertors and other
plumbing to be able to report statistics for the backup job.

Signed-off-by: Peter Krempa 
---
 src/qemu/qemu_domain.c | 62 ++
 src/qemu/qemu_domain.h | 10 +++
 src/qemu/qemu_driver.c |  4 +++
 3 files changed, 76 insertions(+)

diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c
index 8d2923300d..c1b0f81c81 100644
--- a/src/qemu/qemu_domain.c
+++ b/src/qemu/qemu_domain.c
@@ -519,6 +519,12 @@ qemuDomainJobInfoToInfo(qemuDomainJobInfoPtr jobInfo,
 info->memRemaining = info->memTotal - info->memProcessed;
 break;

+case QEMU_DOMAIN_JOB_STATS_TYPE_BACKUP:
+info->fileTotal = jobInfo->stats.backup.total;
+info->fileProcessed = jobInfo->stats.backup.transferred;
+info->fileRemaining = info->fileTotal - info->fileProcessed;
+break;
+
 case QEMU_DOMAIN_JOB_STATS_TYPE_NONE:
 break;
 }
@@ -751,6 +757,59 @@ qemuDomainDumpJobInfoToParams(qemuDomainJobInfoPtr jobInfo,
 }


+static int
+qemuDomainBackupJobInfoToParams(qemuDomainJobInfoPtr jobInfo,
+int *type,
+virTypedParameterPtr *params,
+int *nparams)
+{
+qemuDomainBackupStats *stats = >stats.backup;
+g_autoptr(virTypedParamList) par = g_new0(virTypedParamList, 1);
+
+if (virTypedParamListAddInt(par, jobInfo->operation,
+VIR_DOMAIN_JOB_OPERATION) < 0)
+return -1;
+
+if (virTypedParamListAddULLong(par, jobInfo->timeElapsed,
+   VIR_DOMAIN_JOB_TIME_ELAPSED) < 0)
+return -1;
+
+if (stats->transferred > 0 || stats->total > 0) {
+if (virTypedParamListAddULLong(par, stats->total,
+   VIR_DOMAIN_JOB_DISK_TOTAL) < 0)
+return -1;
+
+if (virTypedParamListAddULLong(par, stats->transferred,
+   VIR_DOMAIN_JOB_DISK_PROCESSED) < 0)
+return -1;
+
+if (virTypedParamListAddULLong(par, stats->total - stats->transferred,
+   VIR_DOMAIN_JOB_DISK_REMAINING) < 0)
+return -1;
+}
+
+if (stats->tmp_used > 0 || stats->tmp_total > 0) {
+if (virTypedParamListAddULLong(par, stats->tmp_used,
+   VIR_DOMAIN_JOB_DISK_TEMP_USED) < 0)
+return -1;
+
+if (virTypedParamListAddULLong(par, stats->tmp_total,
+   VIR_DOMAIN_JOB_DISK_TEMP_TOTAL) < 0)
+return -1;
+}
+
+if (jobInfo->status != QEMU_DOMAIN_JOB_STATUS_ACTIVE &&
+virTypedParamListAddBoolean(par,
+jobInfo->status == 
QEMU_DOMAIN_JOB_STATUS_COMPLETED,
+VIR_DOMAIN_JOB_SUCCESS) < 0)
+return -1;
+
+*nparams = virTypedParamListStealParams(par, params);
+*type = qemuDomainJobStatusToType(jobInfo->status);
+return 0;
+}
+
+
 int
 qemuDomainJobInfoToParams(qemuDomainJobInfoPtr jobInfo,
   int *type,
@@ -765,6 +824,9 @@ qemuDomainJobInfoToParams(qemuDomainJobInfoPtr jobInfo,
 case QEMU_DOMAIN_JOB_STATS_TYPE_MEMDUMP:
 return qemuDomainDumpJobInfoToParams(jobInfo, type, params, nparams);

+case QEMU_DOMAIN_JOB_STATS_TYPE_BACKUP:
+return qemuDomainBackupJobInfoToParams(jobInfo, type, params, nparams);
+
 case QEMU_DOMAIN_JOB_STATS_TYPE_NONE:
 virReportError(VIR_ERR_INTERNAL_ERROR, "%s",
_("invalid job statistics type"));
diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h
index 608546a27c..a552af6180 100644
--- a/src/qemu/qemu_domain.h
+++ b/src/qemu/qemu_domain.h
@@ -126,6 +126,7 @@ typedef enum {
 QEMU_DOMAIN_JOB_STATS_TYPE_MIGRATION,
 QEMU_DOMAIN_JOB_STATS_TYPE_SAVEDUMP,
 QEMU_DOMAIN_JOB_STATS_TYPE_MEMDUMP,
+QEMU_DOMAIN_JOB_STATS_TYPE_BACKUP,
 } qemuDomainJobStatsType;


@@ -136,6 +137,14 @@ struct _qemuDomainMirrorStats {
 unsigned long long total;
 };

+typedef struct _qemuDomainBackupStats qemuDomainBackupStats;
+struct _qemuDomainBackupStats {
+unsigned long long transferred;
+unsigned long long total;
+unsigned long long tmp_used;
+unsigned long long tmp_total;
+};
+
 typedef struct _qemuDomainJobInfo qemuDomainJobInfo;
 typedef qemuDomainJobInfo *qemuDomainJobInfoPtr;
 struct _qemuDomainJobInfo {
@@ -160,6 +169,7 @@ struct _qemuDomainJobInfo {
 union {
 qemuMonitorMigrationStats mig;
 qemuMonitorDumpStats dump;
+qemuDomainBackupStats backup;
 } stats;
 qemuDomainMirrorStats mirrorStats;
 };
diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c
index 1911073f3e..3ebc902d4f 100644
--- a/src/qemu/qemu_driver.c
+++ b/src/qemu/qemu_driver.c
@@ -13905,6 +13905,10 @@ qemuDomainGetJobStatsInternal(virQEMUDriverPtr

[libvirt] [PATCH v2 09/25] tests: genericxml2xml: Add testing of backup XML files

2019-12-03 Thread Peter Krempa 
Now that the parser and formatter are in place we can excercise it on
the test files.

Signed-off-by: Peter Krempa 
---
 tests/Makefile.am |  1 +
 .../backup-pull-seclabel.xml  | 18 
 tests/domainbackupxml2xmlout/backup-pull.xml  | 10 
 .../backup-push-seclabel.xml  | 17 +++
 tests/domainbackupxml2xmlout/backup-push.xml  | 10 
 tests/domainbackupxml2xmlout/empty.xml|  1 +
 tests/genericxml2xmltest.c| 46 +++
 tests/virschematest.c |  3 +-
 8 files changed, 105 insertions(+), 1 deletion(-)
 create mode 100644 tests/domainbackupxml2xmlout/backup-pull-seclabel.xml
 create mode 100644 tests/domainbackupxml2xmlout/backup-pull.xml
 create mode 100644 tests/domainbackupxml2xmlout/backup-push-seclabel.xml
 create mode 100644 tests/domainbackupxml2xmlout/backup-push.xml
 create mode 100644 tests/domainbackupxml2xmlout/empty.xml

diff --git a/tests/Makefile.am b/tests/Makefile.am
index ea9e2b2ad0..75eee0006c 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -92,6 +92,7 @@ EXTRA_DIST = \
cputestdata \
domaincapsdata \
domainbackupxml2xmlin \
+   domainbackupxml2xmlout \
domainconfdata \
domainschemadata \
fchostdata \
diff --git a/tests/domainbackupxml2xmlout/backup-pull-seclabel.xml 
b/tests/domainbackupxml2xmlout/backup-pull-seclabel.xml
new file mode 100644
index 00..c631c9b979
--- /dev/null
+++ b/tests/domainbackupxml2xmlout/backup-pull-seclabel.xml
@@ -0,0 +1,18 @@
+
+  1525889631
+  
+  
+
+  
+  
+
+  
+
+
+  
+  
+
+  
+
+  
+
diff --git a/tests/domainbackupxml2xmlout/backup-pull.xml 
b/tests/domainbackupxml2xmlout/backup-pull.xml
new file mode 100644
index 00..24fce9c0e7
--- /dev/null
+++ b/tests/domainbackupxml2xmlout/backup-pull.xml
@@ -0,0 +1,10 @@
+
+  1525889631
+  
+  
+
+  
+
+
+  
+
diff --git a/tests/domainbackupxml2xmlout/backup-push-seclabel.xml 
b/tests/domainbackupxml2xmlout/backup-push-seclabel.xml
new file mode 100644
index 00..9986889ba3
--- /dev/null
+++ b/tests/domainbackupxml2xmlout/backup-push-seclabel.xml
@@ -0,0 +1,17 @@
+
+  1525889631
+  
+
+  
+  
+
+  
+
+
+  
+  
+
+  
+
+  
+
diff --git a/tests/domainbackupxml2xmlout/backup-push.xml 
b/tests/domainbackupxml2xmlout/backup-push.xml
new file mode 100644
index 00..1997c814ae
--- /dev/null
+++ b/tests/domainbackupxml2xmlout/backup-push.xml
@@ -0,0 +1,10 @@
+
+  1525889631
+  
+
+  
+  
+
+
+  
+
diff --git a/tests/domainbackupxml2xmlout/empty.xml 
b/tests/domainbackupxml2xmlout/empty.xml
new file mode 100644
index 00..b1ba4953be
--- /dev/null
+++ b/tests/domainbackupxml2xmlout/empty.xml
@@ -0,0 +1 @@
+
diff --git a/tests/genericxml2xmltest.c b/tests/genericxml2xmltest.c
index 0d04413712..1376221ef8 100644
--- a/tests/genericxml2xmltest.c
+++ b/tests/genericxml2xmltest.c
@@ -8,6 +8,7 @@
 #include "testutils.h"
 #include "internal.h"
 #include "virstring.h"
+#include "conf/backup_conf.h"

 #define VIR_FROM_THIS VIR_FROM_NONE

@@ -44,6 +45,41 @@ testCompareXMLToXMLHelper(const void *data)
 }


+static int
+testCompareBackupXML(const void *data)
+{
+const char *testname = data;
+g_autofree char *xml_in = NULL;
+g_autofree char *file_in = NULL;
+g_autofree char *file_out = NULL;
+g_autoptr(virDomainBackupDef) backup = NULL;
+g_auto(virBuffer) buf = VIR_BUFFER_INITIALIZER;
+g_autofree char *actual = NULL;
+
+file_in = g_strdup_printf("%s/domainbackupxml2xmlin/%s.xml",
+  abs_srcdir, testname);
+file_out = g_strdup_printf("%s/domainbackupxml2xmlout/%s.xml",
+   abs_srcdir, testname);
+
+if (virFileReadAll(file_in, 1024 * 64, _in) < 0)
+return -1;
+
+if (!(backup = virDomainBackupDefParseString(xml_in, xmlopt, 0))) {
+VIR_TEST_VERBOSE("failed to parse backup def '%s'", file_in);
+return -1;
+}
+
+if (virDomainBackupDefFormat(, backup, false) < 0) {
+VIR_TEST_VERBOSE("failed to format backup def '%s'", file_in);
+return -1;
+}
+
+actual = virBufferContentAndReset();
+
+return virTestCompareToFile(actual, file_out);
+}
+
+
 static int
 mymain(void)
 {
@@ -149,6 +185,16 @@ mymain(void)

 DO_TEST_DIFFERENT("cputune");

+#define DO_TEST_BACKUP(name) \
+if (virTestRun("QEMU BACKUP XML-2-XML " name, testCompareBackupXML, name) 
< 0) \
+ret = -1;
+
+DO_TEST_BACKUP("empty");
+DO_TEST_BACKUP("backup-pull");
+DO_TEST_BACKUP("backup-pull-seclabel");
+DO_TEST_BACKUP("backup-push");
+DO_TEST_BACKUP("backup-push-seclabel");
+
 virObjectUnref(caps);
 virObjectUnref(xmlopt);

diff --git a/tests/virschematest.c b/tests/virschematest.c
index 5ae2d207d1..e4a440afb0 100644

[libvirt] [PATCH v2 20/25] qemu: Implement backup job APIs and qemu handling

2019-12-03 Thread Peter Krempa 
This allows to start and manage the backup job.

Signed-off-by: Peter Krempa 
---
 po/POTFILES.in   |   1 +
 src/qemu/Makefile.inc.am |   2 +
 src/qemu/qemu_backup.c   | 941 +++
 src/qemu/qemu_backup.h   |  41 ++
 src/qemu/qemu_driver.c   |  47 ++
 5 files changed, 1032 insertions(+)
 create mode 100644 src/qemu/qemu_backup.c
 create mode 100644 src/qemu/qemu_backup.h

diff --git a/po/POTFILES.in b/po/POTFILES.in
index 48f3f431ec..5afecf21ba 100644
--- a/po/POTFILES.in
+++ b/po/POTFILES.in
@@ -140,6 +140,7 @@
 @SRCDIR@/src/phyp/phyp_driver.c
 @SRCDIR@/src/qemu/qemu_agent.c
 @SRCDIR@/src/qemu/qemu_alias.c
+@SRCDIR@/src/qemu/qemu_backup.c
 @SRCDIR@/src/qemu/qemu_block.c
 @SRCDIR@/src/qemu/qemu_blockjob.c
 @SRCDIR@/src/qemu/qemu_capabilities.c
diff --git a/src/qemu/Makefile.inc.am b/src/qemu/Makefile.inc.am
index bf30f8a3c5..839b1cacb8 100644
--- a/src/qemu/Makefile.inc.am
+++ b/src/qemu/Makefile.inc.am
@@ -69,6 +69,8 @@ QEMU_DRIVER_SOURCES = \
qemu/qemu_vhost_user_gpu.h \
qemu/qemu_checkpoint.c \
qemu/qemu_checkpoint.h \
+   qemu/qemu_backup.c \
+   qemu/qemu_backup.h \
$(NULL)


diff --git a/src/qemu/qemu_backup.c b/src/qemu/qemu_backup.c
new file mode 100644
index 00..8307a42e1c
--- /dev/null
+++ b/src/qemu/qemu_backup.c
@@ -0,0 +1,941 @@
+/*
+ * qemu_backup.c: Implementation and handling of the backup jobs
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library.  If not, see
+ * .
+ */
+
+#include 
+
+#include "qemu_block.h"
+#include "qemu_conf.h"
+#include "qemu_capabilities.h"
+#include "qemu_monitor.h"
+#include "qemu_process.h"
+#include "qemu_backup.h"
+#include "qemu_monitor_json.h"
+#include "qemu_checkpoint.h"
+#include "qemu_command.h"
+
+#include "virerror.h"
+#include "virlog.h"
+#include "virbuffer.h"
+#include "viralloc.h"
+#include "virxml.h"
+#include "virstoragefile.h"
+#include "virstring.h"
+#include "backup_conf.h"
+#include "virdomaincheckpointobjlist.h"
+
+#define VIR_FROM_THIS VIR_FROM_QEMU
+
+VIR_LOG_INIT("qemu.qemu_backup");
+
+
+static virDomainBackupDefPtr
+qemuDomainGetBackup(virDomainObjPtr vm)
+{
+qemuDomainObjPrivatePtr priv = vm->privateData;
+
+if (!priv->backup) {
+virReportError(VIR_ERR_NO_DOMAIN_BACKUP, "%s",
+   _("no domain backup job present"));
+return NULL;
+}
+
+return priv->backup;
+}
+
+
+static int
+qemuBackupPrepare(virDomainBackupDefPtr def)
+{
+
+if (def->type == VIR_DOMAIN_BACKUP_TYPE_PULL) {
+if (!def->server) {
+def->server = g_new(virStorageNetHostDef, 1);
+
+def->server->transport = VIR_STORAGE_NET_HOST_TRANS_TCP;
+def->server->name = g_strdup("localhost");
+}
+
+switch ((virStorageNetHostTransport) def->server->transport) {
+case VIR_STORAGE_NET_HOST_TRANS_TCP:
+/* TODO: Update qemu.conf to provide a port range,
+ * probably starting at 10809, for obtaining automatic
+ * port via virPortAllocatorAcquire, as well as store
+ * somewhere if we need to call virPortAllocatorRelease
+ * during BackupEnd. Until then, user must provide port */
+if (!def->server->port) {
+virReportError(VIR_ERR_OPERATION_UNSUPPORTED, "%s",
+   _(" must specify TCP port for 
now"));
+return -1;
+}
+break;
+
+case VIR_STORAGE_NET_HOST_TRANS_UNIX:
+/* TODO: Do we need to mess with selinux? */
+break;
+
+case VIR_STORAGE_NET_HOST_TRANS_RDMA:
+case VIR_STORAGE_NET_HOST_TRANS_LAST:
+virReportError(VIR_ERR_INTERNAL_ERROR, "%s",
+   _("unexpected transport in "));
+return -1;
+}
+}
+
+return 0;
+}
+
+
+struct qemuBackupDiskData {
+virDomainBackupDiskDefPtr backupdisk;
+virDomainDiskDefPtr domdisk;
+qemuBlockJobDataPtr blockjob;
+virStorageSourcePtr store;
+char *incrementalBitmap;
+qemuBlockStorageSourceChainDataPtr crdata;
+bool labelled;
+bool initialized;
+bool created;
+bool added;
+bool started;
+bool done;
+};
+
+
+static void
+qemuBackupDiskDataCleanupOne(virDomainObjPtr vm,
+ struct qemuBackupDiskData *dd)
+{
+

[libvirt] [PATCH v2 16/25] qemu: blockjob: Track internal data for 'backup' blockjob

2019-12-03 Thread Peter Krempa 
A backup blockjob needs to be able to notify the parent backup job as
well as track all data to be able to clean up the bitmap and blockdev
used for the backup.

Add the data structure, job allocation function and status XML formatter
and parser.

Signed-off-by: Peter Krempa 
---
 src/qemu/qemu_blockjob.c | 34 +-
 src/qemu/qemu_blockjob.h | 18 ++
 src/qemu/qemu_domain.c   | 21 +
 3 files changed, 72 insertions(+), 1 deletion(-)

diff --git a/src/qemu/qemu_blockjob.c b/src/qemu/qemu_blockjob.c
index 5455eaba65..d434b8bddd 100644
--- a/src/qemu/qemu_blockjob.c
+++ b/src/qemu/qemu_blockjob.c
@@ -78,8 +78,12 @@ qemuBlockJobDataDisposeJobdata(qemuBlockJobDataPtr job)
 {
 if (job->type == QEMU_BLOCKJOB_TYPE_CREATE)
 virObjectUnref(job->data.create.src);
-}

+if (job->type == QEMU_BLOCKJOB_TYPE_BACKUP) {
+virObjectUnref(job->data.backup.store);
+g_free(job->data.backup.bitmap);
+}
+}

 static void
 qemuBlockJobDataDispose(void *obj)
@@ -370,6 +374,34 @@ qemuBlockJobDiskNewCopy(virDomainObjPtr vm,
 }


+qemuBlockJobDataPtr
+qemuBlockJobDiskNewBackup(virDomainObjPtr vm,
+  virDomainDiskDefPtr disk,
+  virStorageSourcePtr store,
+  bool deleteStore,
+  const char *bitmap)
+{
+g_autoptr(qemuBlockJobData) job = NULL;
+g_autofree char *jobname = NULL;
+
+jobname = g_strdup_printf("backup-%s-%s", disk->dst, 
disk->src->nodeformat);
+
+if (!(job = qemuBlockJobDataNew(QEMU_BLOCKJOB_TYPE_BACKUP, jobname)))
+return NULL;
+
+job->data.backup.bitmap = g_strdup(bitmap);
+job->data.backup.store = virObjectRef(store);
+job->data.backup.deleteStore = deleteStore;
+
+/* backup jobs are usually started in bulk by transaction so the caller
+ * shall save the status XML */
+if (qemuBlockJobRegister(job, vm, disk, false) < 0)
+return NULL;
+
+return g_steal_pointer();
+}
+
+
 /**
  * qemuBlockJobDiskGetJob:
  * @disk: disk definition
diff --git a/src/qemu/qemu_blockjob.h b/src/qemu/qemu_blockjob.h
index 4734984c99..52b03aaf9e 100644
--- a/src/qemu/qemu_blockjob.h
+++ b/src/qemu/qemu_blockjob.h
@@ -107,6 +107,16 @@ struct _qemuBlockJobCopyData {
 };


+typedef struct _qemuBlockJobBackupData qemuBlockJobBackupData;
+typedef qemuBlockJobBackupData *qemuBlockJobDataBackupPtr;
+
+struct _qemuBlockJobBackupData {
+virStorageSourcePtr store;
+bool deleteStore;
+char *bitmap;
+};
+
+
 typedef struct _qemuBlockJobData qemuBlockJobData;
 typedef qemuBlockJobData *qemuBlockJobDataPtr;

@@ -124,6 +134,7 @@ struct _qemuBlockJobData {
 qemuBlockJobCommitData commit;
 qemuBlockJobCreateData create;
 qemuBlockJobCopyData copy;
+qemuBlockJobBackupData backup;
 } data;

 int type; /* qemuBlockJobType */
@@ -184,6 +195,13 @@ qemuBlockJobDiskNewCopy(virDomainObjPtr vm,
 bool shallow,
 bool reuse);

+qemuBlockJobDataPtr
+qemuBlockJobDiskNewBackup(virDomainObjPtr vm,
+  virDomainDiskDefPtr disk,
+  virStorageSourcePtr store,
+  bool deleteStore,
+  const char *bitmap);
+
 qemuBlockJobDataPtr
 qemuBlockJobDiskGetJob(virDomainDiskDefPtr disk)
 ATTRIBUTE_NONNULL(1);
diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c
index 98d0dad861..f4f841526e 100644
--- a/src/qemu/qemu_domain.c
+++ b/src/qemu/qemu_domain.c
@@ -2608,6 +2608,18 @@ qemuDomainObjPrivateXMLFormatBlockjobIterator(void 
*payload,
 break;

 case QEMU_BLOCKJOB_TYPE_BACKUP:
+virBufferEscapeString(, "\n", 
job->data.backup.bitmap);
+if (job->data.backup.store) {
+if 
(qemuDomainObjPrivateXMLFormatBlockjobFormatSource(,
+  "store",
+  
job->data.backup.store,
+  
data->xmlopt,
+  false) < 
0)
+return -1;
+
+if (job->data.backup.deleteStore)
+virBufferAddLit(, "\n");
+}
 break;

 case QEMU_BLOCKJOB_TYPE_BROKEN:
@@ -3199,6 +3211,15 @@ 
qemuDomainObjPrivateXMLParseBlockjobDataSpecific(qemuBlockJobDataPtr job,
 break;

 case QEMU_BLOCKJOB_TYPE_BACKUP:
+job->data.backup.bitmap =  
virXPathString("string(./bitmap/@name)", ctxt);
+
+if (!(tmp = virXPathNode("./store", ctxt)) ||
+!(job->data.backup.store = 
qemuDomainObjPrivateXMLParseBlockjobChain(tmp, ctxt, xmlopt)))
+goto broken;
+
+if (virXPathNode("./deleteStore", ctxt))
+

[libvirt] [PATCH v2 14/25] qemu: monitor: Add support for blockdev-backup via 'transaction'

2019-12-03 Thread Peter Krempa 
Implement the transaction actions generator for blockdev-backup.

Signed-off-by: Peter Krempa 
---
 src/qemu/qemu_monitor.c  | 13 +
 src/qemu/qemu_monitor.h  | 15 +++
 src/qemu/qemu_monitor_json.c | 29 +
 src/qemu/qemu_monitor_json.h |  8 
 tests/qemumonitorjsontest.c  |  8 +++-
 5 files changed, 72 insertions(+), 1 deletion(-)

diff --git a/src/qemu/qemu_monitor.c b/src/qemu/qemu_monitor.c
index a48305b046..6e6678eb9b 100644
--- a/src/qemu/qemu_monitor.c
+++ b/src/qemu/qemu_monitor.c
@@ -4615,3 +4615,16 @@ qemuMonitorTransactionSnapshotBlockdev(virJSONValuePtr 
actions,
 {
 return qemuMonitorJSONTransactionSnapshotBlockdev(actions, node, overlay);
 }
+
+
+int
+qemuMonitorTransactionBackup(virJSONValuePtr actions,
+ const char *device,
+ const char *jobname,
+ const char *target,
+ const char *bitmap,
+ qemuMonitorTransactionBackupSyncMode syncmode)
+{
+return qemuMonitorJSONTransactionBackup(actions, device, jobname, target,
+bitmap, syncmode);
+}
diff --git a/src/qemu/qemu_monitor.h b/src/qemu/qemu_monitor.h
index e2bfc420bb..79e078fca4 100644
--- a/src/qemu/qemu_monitor.h
+++ b/src/qemu/qemu_monitor.h
@@ -1392,3 +1392,18 @@ int
 qemuMonitorTransactionSnapshotBlockdev(virJSONValuePtr actions,
const char *node,
const char *overlay);
+
+typedef enum {
+QEMU_MONITOR_TRANSACTION_BACKUP_SYNC_MODE_NONE = 0,
+QEMU_MONITOR_TRANSACTION_BACKUP_SYNC_MODE_INCREMENTAL,
+QEMU_MONITOR_TRANSACTION_BACKUP_SYNC_MODE_FULL,
+QEMU_MONITOR_TRANSACTION_BACKUP_SYNC_MODE_LAST,
+} qemuMonitorTransactionBackupSyncMode;
+
+int
+qemuMonitorTransactionBackup(virJSONValuePtr actions,
+ const char *device,
+ const char *jobname,
+ const char *target,
+ const char *bitmap,
+ qemuMonitorTransactionBackupSyncMode syncmode);
diff --git a/src/qemu/qemu_monitor_json.c b/src/qemu/qemu_monitor_json.c
index 391f39668a..00e1d3ce15 100644
--- a/src/qemu/qemu_monitor_json.c
+++ b/src/qemu/qemu_monitor_json.c
@@ -9198,6 +9198,35 @@ 
qemuMonitorJSONTransactionSnapshotBlockdev(virJSONValuePtr actions,
  NULL);
 }

+VIR_ENUM_DECL(qemuMonitorTransactionBackupSyncMode);
+VIR_ENUM_IMPL(qemuMonitorTransactionBackupSyncMode,
+  QEMU_MONITOR_TRANSACTION_BACKUP_SYNC_MODE_LAST,
+  "none",
+  "incremental",
+  "full");
+
+int
+qemuMonitorJSONTransactionBackup(virJSONValuePtr actions,
+ const char *device,
+ const char *jobname,
+ const char *target,
+ const char *bitmap,
+ qemuMonitorTransactionBackupSyncMode syncmode)
+{
+const char *syncmodestr = 
qemuMonitorTransactionBackupSyncModeTypeToString(syncmode);
+
+return qemuMonitorJSONTransactionAdd(actions,
+ "blockdev-backup",
+ "s:device", device,
+ "s:job-id", jobname,
+ "s:target", target,
+ "s:sync", syncmodestr,
+ "S:bitmap", bitmap,
+ "T:auto-finalize", 
VIR_TRISTATE_BOOL_YES,
+ "T:auto-dismiss", 
VIR_TRISTATE_BOOL_NO,
+ NULL);
+}
+

 static qemuMonitorJobInfoPtr
 qemuMonitorJSONGetJobInfoOne(virJSONValuePtr data)
diff --git a/src/qemu/qemu_monitor_json.h b/src/qemu/qemu_monitor_json.h
index 25b568d6b0..5d05772fa2 100644
--- a/src/qemu/qemu_monitor_json.h
+++ b/src/qemu/qemu_monitor_json.h
@@ -667,3 +667,11 @@ int
 qemuMonitorJSONTransactionSnapshotBlockdev(virJSONValuePtr actions,
const char *node,
const char *overlay);
+
+int
+qemuMonitorJSONTransactionBackup(virJSONValuePtr actions,
+ const char *device,
+ const char *jobname,
+ const char *target,
+ const char *bitmap,
+ qemuMonitorTransactionBackupSyncMode 
syncmode);
diff --git a/tests/qemumonitorjsontest.c b/tests/qemumonitorjsontest.c
index 21f17f42af..4f3bfad1d7 100644
--- a/tests/qemumonitorjsontest.c
+++ b/tests/qemumonitorjsontest.c
@@ -2962,7 +2962,13 @@ testQemuMonitorJSONTransaction(const void

[libvirt] [PATCH v2 24/25] conf: domaincaps: Add 'backup' feature flag

2019-12-03 Thread Peter Krempa 
This flag will allow figuring out whether the hypervisor supports the
incremental backup and checkpoint features.

Signed-off-by: Peter Krempa 
---
 docs/formatdomaincaps.html.in  | 8 
 docs/schemas/domaincaps.rng| 9 +
 src/conf/domain_capabilities.c | 1 +
 src/conf/domain_capabilities.h | 1 +
 4 files changed, 19 insertions(+)

diff --git a/docs/formatdomaincaps.html.in b/docs/formatdomaincaps.html.in
index 0bafb67705..85226328a8 100644
--- a/docs/formatdomaincaps.html.in
+++ b/docs/formatdomaincaps.html.in
@@ -517,6 +517,7 @@
 vmcoreinfo supported='yes'/
 genid supported='yes'/
 backingStoreInput supported='yes'/
+backup supported='yes'/
 sev
   cbitpos47/cbitpos
   reduced-phys-bits1/reduced-phys-bits
@@ -560,6 +561,13 @@
 the disk to a running guest, or similar.
 

+backup
+
+Reports whether the hypervisor supports the backup,checkpoint and 
related
+features. (virDomainBackupBegin,
+virDomainCheckpointCreateXML etc).
+
+
 SEV capabilities

 AMD Secure Encrypted Virtualization (SEV) capabilities are exposed under
diff --git a/docs/schemas/domaincaps.rng b/docs/schemas/domaincaps.rng
index 88b545ec2a..682cc82177 100644
--- a/docs/schemas/domaincaps.rng
+++ b/docs/schemas/domaincaps.rng
@@ -210,6 +210,9 @@
   
 
   
+  
+
+  
   
 
   
@@ -241,6 +244,12 @@
 
   

+  
+
+  
+
+  
+
   
 
   
diff --git a/src/conf/domain_capabilities.c b/src/conf/domain_capabilities.c
index ca208f2340..921d795630 100644
--- a/src/conf/domain_capabilities.c
+++ b/src/conf/domain_capabilities.c
@@ -41,6 +41,7 @@ VIR_ENUM_IMPL(virDomainCapsFeature,
   "vmcoreinfo",
   "genid",
   "backingStoreInput",
+  "backup",
 );

 static virClassPtr virDomainCapsClass;
diff --git a/src/conf/domain_capabilities.h b/src/conf/domain_capabilities.h
index 4ec9fe006c..9f4a23d015 100644
--- a/src/conf/domain_capabilities.h
+++ b/src/conf/domain_capabilities.h
@@ -163,6 +163,7 @@ typedef enum {
 VIR_DOMAIN_CAPS_FEATURE_VMCOREINFO,
 VIR_DOMAIN_CAPS_FEATURE_GENID,
 VIR_DOMAIN_CAPS_FEATURE_BACKING_STORE_INPUT,
+VIR_DOMAIN_CAPS_FEATURE_BACKUP,

 VIR_DOMAIN_CAPS_FEATURE_LAST
 } virDomainCapsFeature;
-- 
2.23.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH v2 05/25] backup: Introduce virDomainBackup APIs

2019-12-03 Thread Peter Krempa 
From: Eric Blake 

Introduce a few new public APIs related to incremental backups.  This
builds on the previous notion of a checkpoint (without an existing
checkpoint, the new API is a full backup, differing from
virDomainBlockCopy in the point of time chosen and in operation on
multiple disks at once); and also allows creation of a new checkpoint
at the same time as starting the backup (after all, an incremental
backup is only useful if it covers the state since the previous
backup).

A backup job also affects filtering a listing of domains, as well as
adding event reporting for signaling when a push model backup
completes (where the hypervisor creates the backup); note that the
pull model does not have an event (starting the backup lets a third
party access the data, and only the third party knows when it is
finished).

The full list of new APIs:
virDomainBackupBegin;
virDomainBackupGetXMLDesc;

Signed-off-by: Eric Blake 
Signed-off-by: Peter Krempa 
---
 include/libvirt/libvirt-domain.h |  19 +++-
 src/driver-hypervisor.h  |  12 +++
 src/libvirt-domain-checkpoint.c  |   7 +-
 src/libvirt-domain.c | 143 +++
 src/libvirt_public.syms  |   6 ++
 5 files changed, 183 insertions(+), 4 deletions(-)

diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h
index b9908fe7b0..84c5492a7b 100644
--- a/include/libvirt/libvirt-domain.h
+++ b/include/libvirt/libvirt-domain.h
@@ -4129,8 +4129,10 @@ typedef void 
(*virConnectDomainEventMigrationIterationCallback)(virConnectPtr co
  * @nparams: size of the params array
  * @opaque: application specific data
  *
- * This callback occurs when a job (such as migration) running on the domain
- * is completed. The params array will contain statistics of the just completed
+ * This callback occurs when a job (such as migration or backup) running on
+ * the domain is completed.
+ *
+ * The params array will contain statistics of the just completed
  * job as virDomainGetJobStats would return. The callback must not free @params
  * (the array will be freed once the callback finishes).
  *
@@ -4949,4 +4951,17 @@ int virDomainAgentSetResponseTimeout(virDomainPtr domain,
  int timeout,
  unsigned int flags);

+typedef enum {
+VIR_DOMAIN_BACKUP_BEGIN_REUSE_EXTERNAL = (1 << 0), /* reuse separately
+  provided images */
+} virDomainBackupBeginFlags;
+
+int virDomainBackupBegin(virDomainPtr domain,
+ const char *backupXML,
+ const char *checkpointXML,
+ unsigned int flags);
+
+char *virDomainBackupGetXMLDesc(virDomainPtr domain,
+unsigned int flags);
+
 #endif /* LIBVIRT_DOMAIN_H */
diff --git a/src/driver-hypervisor.h b/src/driver-hypervisor.h
index 4afd8f6ec5..bce023017d 100644
--- a/src/driver-hypervisor.h
+++ b/src/driver-hypervisor.h
@@ -1377,6 +1377,16 @@ typedef int
int timeout,
unsigned int flags);

+typedef int
+(*virDrvDomainBackupBegin)(virDomainPtr domain,
+   const char *backupXML,
+   const char *checkpointXML,
+   unsigned int flags);
+
+typedef char *
+(*virDrvDomainBackupGetXMLDesc)(virDomainPtr domain,
+unsigned int flags);
+
 typedef struct _virHypervisorDriver virHypervisorDriver;
 typedef virHypervisorDriver *virHypervisorDriverPtr;

@@ -1638,4 +1648,6 @@ struct _virHypervisorDriver {
 virDrvDomainCheckpointDelete domainCheckpointDelete;
 virDrvDomainGetGuestInfo domainGetGuestInfo;
 virDrvDomainAgentSetResponseTimeout domainAgentSetResponseTimeout;
+virDrvDomainBackupBegin domainBackupBegin;
+virDrvDomainBackupGetXMLDesc domainBackupGetXMLDesc;
 };
diff --git a/src/libvirt-domain-checkpoint.c b/src/libvirt-domain-checkpoint.c
index fa391f8a06..432c2d5a52 100644
--- a/src/libvirt-domain-checkpoint.c
+++ b/src/libvirt-domain-checkpoint.c
@@ -102,8 +102,11 @@ virDomainCheckpointGetConnect(virDomainCheckpointPtr 
checkpoint)
  * @flags: bitwise-OR of supported virDomainCheckpointCreateFlags
  *
  * Create a new checkpoint using @xmlDesc, with a top-level
- *  element, on a running @domain.  Note that @xmlDesc
- * must validate against the  XML schema.
+ *  element, on a running @domain.  Note that
+ * @xmlDesc must validate against the  XML schema.
+ * Typically, it is more common to create a new checkpoint as part of
+ * kicking off a backup job with virDomainBackupBegin(); however, it
+ * is also possible to start a checkpoint without a backup.
  *
  * See Checkpoint XML
  * for more details on @xmlDesc. In particular, some hypervisors may require
diff --git a/src/libvirt-domain.c b/src/libvirt-domain.c
index b9345804ea..f873246ace 100644

[libvirt] [PATCH v2 17/25] tests: qemustatusxml2xml: Add test for 'pull' type backup job

2019-12-03 Thread Peter Krempa 
Signed-off-by: Peter Krempa 
---
 .../qemustatusxml2xmldata/backup-pull-in.xml  | 608 ++
 .../qemustatusxml2xmldata/backup-pull-out.xml |   1 +
 tests/qemuxml2xmltest.c   |   2 +
 3 files changed, 611 insertions(+)
 create mode 100644 tests/qemustatusxml2xmldata/backup-pull-in.xml
 create mode 12 tests/qemustatusxml2xmldata/backup-pull-out.xml

diff --git a/tests/qemustatusxml2xmldata/backup-pull-in.xml 
b/tests/qemustatusxml2xmldata/backup-pull-in.xml
new file mode 100644
index 00..6ef4965bed
--- /dev/null
+++ b/tests/qemustatusxml2xmldata/backup-pull-in.xml
@@ -0,0 +1,608 @@
+
+  
+  
+  
+
+  
+  
+
+  
+  
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+  
+  
+
+
+
+
+
+
+
+
+
+
+  
+  
+  
+  
+  
+  
+  
+
+  
+  
+  
+
+  
+
+  
+  
+
+  
+
+  
+  
+
+  
+  -2
+  
+
+  12345
+  
+  
+
+  
+
+  
+
+  
+  
+copy
+0439a4a8-db56-4933-9183-d8681d7b0746
+1024000
+1024000
+1
+
+  /machine
+
+
+  hvm
+  
+  
+
+
+  
+  
+  
+
+
+  
+  
+  
+
+destroy
+restart
+restart
+
+  
+  
+
+
+  /usr/bin/qemu-system-x86_64
+  
+
+
+  
+
+  
+  
+
+  
+
+
+  
+  
+
+  
+
+
+  
+  pull3.qcow2
+
+  
+  
+
+
+  
+
+  
+  
+
+pull2.qcow2
+  
+
+
+  
+  
+
+  
+
+
+  
+  pull1.qcow2
+
+  
+  
+
+
+  
+
+  
+  
+
+pull0.qcow2
+  
+
+
+  
+
+  
+
+
+
+
+
+  
+
+  
+  
+
+
+  
+
+  
+  
+
+  
+
+
+  
+  
+
+  
+
+
+  
+  commit3.qcow2
+
+  
+  
+
+
+  
+
+  
+  
+
+commit2.qcow2
+  
+
+
+  
+  
+
+  
+
+
+  
+  commit1.qcow2
+
+  
+  
+
+
+  
+
+  
+  
+
+commit0.qcow2
+  
+
+
+  
+
+  
+
+
+
+
+
+  
+
+  
+  
+
+
+  
+
+  
+  
+
+  
+
+
+  
+  
+
+  
+
+
+  
+

[libvirt] [PATCH v2 19/25] doc: Document quirk of getting block job info for a 'backup' blockjob

2019-12-03 Thread Peter Krempa 
The stats reported for a blockjob which is member of a domain pull
backup refer to the utilization of the scratch file rather than the
progress of the backup as the progress of the backup depends on the
client. Note this quirk in the docs.

Signed-off-by: Peter Krempa 
---
 src/libvirt-domain.c | 4 
 tools/virsh.pod  | 4 
 2 files changed, 8 insertions(+)

diff --git a/src/libvirt-domain.c b/src/libvirt-domain.c
index f873246ace..793eceb39f 100644
--- a/src/libvirt-domain.c
+++ b/src/libvirt-domain.c
@@ -9949,6 +9949,10 @@ virDomainBlockJobAbort(virDomainPtr dom, const char 
*disk,
  * and was no-op. In this case libvirt reports cur = 1 and end = 1.
  * Since 2.3.0.
  *
+ * Note that the progress reported for blockjobs corresponding to a pull-mode
+ * backup don't report progress of the backup but rather usage of temporary
+ * space required for the backup.
+ *
  * Returns -1 in case of failure, 0 when nothing found, 1 when info was found.
  */
 int
diff --git a/tools/virsh.pod b/tools/virsh.pod
index b04f7c0fdc..244895ceb4 100644
--- a/tools/virsh.pod
+++ b/tools/virsh.pod
@@ -994,6 +994,10 @@ I<--bytes> with a scaled value permits a finer granularity 
to be selected.
 A scaled value used without I<--bytes> will be rounded down to MiB/s. Note
 that the I<--bytes> may be unsupported by the hypervisor.

+Note that the progress reported for blockjobs corresponding to a pull-mode
+backup don't report progress of the backup but rather usage of temporary
+space required for the backup.
+
 =item B I I [I] [I<--bytes>] [I]
 [I<--wait> [I<--verbose>] [I<--timeout> B] [I<--async>]]
 [I<--keep-relative>]
-- 
2.23.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH v2 10/25] backup: Implement virsh support for backup

2019-12-03 Thread Peter Krempa 
From: Eric Blake 

Introduce virsh commands for performing backup jobs.

Signed-off-by: Eric Blake 
Signed-off-by: Peter Krempa 
---
 po/POTFILES.in   |   1 +
 tools/Makefile.am|   1 +
 tools/virsh-backup.c | 151 +++
 tools/virsh-backup.h |  21 ++
 tools/virsh.c|   2 +
 tools/virsh.h|   1 +
 tools/virsh.pod  |  31 +
 7 files changed, 208 insertions(+)
 create mode 100644 tools/virsh-backup.c
 create mode 100644 tools/virsh-backup.h

diff --git a/po/POTFILES.in b/po/POTFILES.in
index 0ff3beeb7e..48f3f431ec 100644
--- a/po/POTFILES.in
+++ b/po/POTFILES.in
@@ -329,6 +329,7 @@
 @SRCDIR@/src/vz/vz_utils.h
 @SRCDIR@/tests/virpolkittest.c
 @SRCDIR@/tools/libvirt-guests.sh.in
+@SRCDIR@/tools/virsh-backup.c
 @SRCDIR@/tools/virsh-checkpoint.c
 @SRCDIR@/tools/virsh-completer-host.c
 @SRCDIR@/tools/virsh-console.c
diff --git a/tools/Makefile.am b/tools/Makefile.am
index 1a541a3984..b9d31838df 100644
--- a/tools/Makefile.am
+++ b/tools/Makefile.am
@@ -232,6 +232,7 @@ virt_login_shell_helper_CFLAGS = \

 virsh_SOURCES = \
virsh.c virsh.h \
+   virsh-backup.c virsh-backup.h\
virsh-checkpoint.c virsh-checkpoint.h \
virsh-completer.c virsh-completer.h \
virsh-completer-domain.c virsh-completer-domain.h \
diff --git a/tools/virsh-backup.c b/tools/virsh-backup.c
new file mode 100644
index 00..04464c6bff
--- /dev/null
+++ b/tools/virsh-backup.c
@@ -0,0 +1,151 @@
+/*
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library.  If not, see
+ * .
+ */
+
+#include 
+#include "virsh-backup.h"
+#include "virsh-util.h"
+
+#include "internal.h"
+#include "virfile.h"
+
+/*
+ * "backup-begin" command
+ */
+static const vshCmdInfo info_backup_begin[] = {
+{.name = "help",
+ .data = N_("Start a disk backup of a live domain")
+},
+{.name = "desc",
+ .data = N_("Use XML to start a full or incremental disk backup of a live "
+"domain, optionally creating a checkpoint")
+},
+{.name = NULL}
+};
+
+static const vshCmdOptDef opts_backup_begin[] = {
+VIRSH_COMMON_OPT_DOMAIN_FULL(0),
+{.name = "backupxml",
+ .type = VSH_OT_STRING,
+ .help = N_("domain backup XML"),
+},
+{.name = "checkpointxml",
+ .type = VSH_OT_STRING,
+ .help = N_("domain checkpoint XML"),
+},
+{.name = "reuse-external",
+ .type = VSH_OT_BOOL,
+ .help = N_("reuse files provided by caller"),
+},
+{.name = NULL}
+};
+
+static bool
+cmdBackupBegin(vshControl *ctl,
+   const vshCmd *cmd)
+{
+g_autoptr(virshDomain) dom = NULL;
+const char *backup_from = NULL;
+g_autofree char *backup_buffer = NULL;
+const char *check_from = NULL;
+g_autofree char *check_buffer = NULL;
+unsigned int flags = 0;
+
+if (vshCommandOptBool(cmd, "reuse-external"))
+flags |= VIR_DOMAIN_BACKUP_BEGIN_REUSE_EXTERNAL;
+
+if (!(dom = virshCommandOptDomain(ctl, cmd, NULL)))
+return false;
+
+if (vshCommandOptStringReq(ctl, cmd, "backupxml", _from) < 0)
+return false;
+
+if (!backup_from) {
+backup_buffer = g_strdup("");
+} else {
+if (virFileReadAll(backup_from, VSH_MAX_XML_FILE, _buffer) < 0) 
{
+vshSaveLibvirtError();
+return false;
+}
+}
+
+if (vshCommandOptStringReq(ctl, cmd, "checkpointxml", _from) < 0)
+return false;
+if (check_from) {
+if (virFileReadAll(check_from, VSH_MAX_XML_FILE, _buffer) < 0) {
+vshSaveLibvirtError();
+return false;
+}
+}
+
+if (virDomainBackupBegin(dom, backup_buffer, check_buffer, flags) < 0)
+return false;
+
+vshPrint(ctl, _("Backup started\n"));
+return true;
+}
+
+
+/*
+ * "backup-dumpxml" command
+ */
+static const vshCmdInfo info_backup_dumpxml[] = {
+{.name = "help",
+ .data = N_("Dump XML for an ongoing domain block backup job")
+},
+{.name = "desc",
+ .data = N_("Backup Dump XML")
+},
+{.name = NULL}
+};
+
+static const vshCmdOptDef opts_backup_dumpxml[] = {
+VIRSH_COMMON_OPT_DOMAIN_FULL(0),
+{.name = NULL}
+};
+
+static bool
+cmdBackupDumpXML(vshControl *ctl,
+ const vshCmd *cmd)
+{
+g_autoptr(virshDomain) dom = NULL;
+g_autofree char *xml = NULL;
+
+

[libvirt] [PATCH v2 22/25] qemu: driver: Allow cancellation of the backup job

2019-12-03 Thread Peter Krempa 
Use the helper which cancels all blockjobs to perform the backup job
cancellation in qemuDomainAbortJob.

Signed-off-by: Peter Krempa 
---
 src/qemu/qemu_driver.c | 15 ++-
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c
index 95882d9d14..2408b08106 100644
--- a/src/qemu/qemu_driver.c
+++ b/src/qemu/qemu_driver.c
@@ -14054,11 +14054,16 @@ static int qemuDomainAbortJob(virDomainPtr dom)
 }

 VIR_DEBUG("Cancelling job at client request");
-qemuDomainObjAbortAsyncJob(vm);
-qemuDomainObjEnterMonitor(driver, vm);
-ret = qemuMonitorMigrateCancel(priv->mon);
-if (qemuDomainObjExitMonitor(driver, vm) < 0)
-ret = -1;
+if (priv->job.asyncJob == QEMU_ASYNC_JOB_BACKUP) {
+qemuBackupJobCancelBlockjobs(vm, priv->backup, true);
+ret = 0;
+} else {
+qemuDomainObjAbortAsyncJob(vm);
+qemuDomainObjEnterMonitor(driver, vm);
+ret = qemuMonitorMigrateCancel(priv->mon);
+if (qemuDomainObjExitMonitor(driver, vm) < 0)
+ret = -1;
+}

  endjob:
 qemuDomainObjEndJob(driver, vm);
-- 
2.23.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH v2 00/25] Incremental backup support for qemu

2019-12-03 Thread Peter Krempa 
Next version which includes feedback from V1:

https://www.redhat.com/archives/libvir-list/2019-November/msg01315.html

and also few features and bugs fixed based on offline requests:
- The flag VIR_DOMAIN_BACKUP_BEGIN_REUSE_EXTERNAL was added to
facilitate users who wish to provide their own files.
- The schema was fixed as many legitimate uses were not described:
- format for the scratch file was not supported
- security labels for the scratch file were not supported
- tests were insufficient
- backup XML 2 XML testing was added
- scratch files created by libvirt are now removed after the job
finishes
- domain capability feature entry was added
- the code for determining bitmaps for incremental backup was slightly
 optimized
- documentation now documents our behaviour towards the scratch file and
the relationship to the new flag.

I might have forgotten to apply some reviewed-by tags though. I'm sorry
for that. Most patches changed though (including the API patches which
add the flag) so a review is welcome even there.

You can fetch the new version at:

  git fetch https://gitlab.com/pipo.sk/libvirt.git blockdev-backup-v2

Note that the branch also contains commit to enable block commit for
easier testing. With the posted code the following approach can be
used to enable it (new qemu required):


   ...
   
 
   


Eric Blake (5):
  backup: Document new XML for backups
  backup: Introduce virDomainBackup APIs
  backup: Implement backup APIs for remote driver
  backup: Parse and output backup XML
  backup: Implement virsh support for backup

Peter Krempa (20):
  qemu: domain: Export qemuDomainGetImageIds
  API: Introduce field for reporting temporary disk space usage of a
domain job
  virsh: Implement VIR_DOMAIN_JOB_DISK_TEMP_(USED|TOTAL) in
cmdDomjobinfo
  API: Add domain job operation for backups
  tests: genericxml2xml: Add testing of backup XML files
  qemu: Add infrastructure for statistics of a backup job
  qemu: domain: Introduce QEMU_ASYNC_JOB_BACKUP async job type
  Add 'backup' block job type
  qemu: monitor: Add support for blockdev-backup via 'transaction'
  qemu: domain: Track backup job data in the status XML
  qemu: blockjob: Track internal data for 'backup' blockjob
  tests: qemustatusxml2xml: Add test for 'pull' type backup job
  conf: backup: Add fields for tracking stats of completed sub-jobs
  doc: Document quirk of getting block job info for a 'backup' blockjob
  qemu: Implement backup job APIs and qemu handling
  qemu: backup: Implement stats gathering while the job is running
  qemu: driver: Allow cancellation of the backup job
  qemu: blockjob: Implement concluded blockjob handler for backup
blockjobs
  conf: domaincaps: Add 'backup' feature flag
  qemu: Add support for VIR_DOMAIN_CAPS_FEATURE_BACKUP

 docs/docs.html.in |3 +-
 docs/format.html.in   |1 +
 docs/formatbackup.html.in |  175 +++
 docs/formatcheckpoint.html.in |   12 +-
 docs/formatdomaincaps.html.in |8 +
 docs/index.html.in|3 +-
 docs/schemas/domainbackup.rng |  223 
 docs/schemas/domaincaps.rng   |9 +
 examples/c/misc/event-test.c  |3 +
 include/libvirt/libvirt-domain.h  |   37 +-
 libvirt.spec.in   |1 +
 mingw-libvirt.spec.in |2 +
 po/POTFILES.in|3 +
 src/conf/Makefile.inc.am  |2 +
 src/conf/backup_conf.c|  499 
 src/conf/backup_conf.h|  108 ++
 src/conf/domain_capabilities.c|1 +
 src/conf/domain_capabilities.h|1 +
 src/conf/domain_conf.c|2 +-
 src/conf/virconftypes.h   |3 +
 src/driver-hypervisor.h   |   12 +
 src/libvirt-domain-checkpoint.c   |7 +-
 src/libvirt-domain.c  |  147 +++
 src/libvirt_private.syms  |8 +
 src/libvirt_public.syms   |6 +
 src/qemu/Makefile.inc.am  |2 +
 src/qemu/qemu_backup.c| 1039 +
 src/qemu/qemu_backup.h|   46 +
 src/qemu/qemu_blockjob.c  |  111 +-
 src/qemu/qemu_blockjob.h  |   19 +
 src/qemu/qemu_capabilities.c  |1 +
 src/qemu/qemu_domain.c|  150 ++-
 src/qemu/qemu_domain.h|   21 +
 src/qemu/qemu_driver.c|   68 +-
 src/qemu/qemu_migration.c |2 +
 src/qemu/qemu_monitor.c   |   13 +
 src/qemu/qemu_monitor.h   |   15 +
 src/qemu/qemu_monitor_json.c  |   33 +

[libvirt] [PATCH v2 07/25] backup: Implement backup APIs for remote driver

2019-12-03 Thread Peter Krempa 
From: Eric Blake 

This one is fairly straightforward - the generator already does what
we need.

Signed-off-by: Eric Blake 
Signed-off-by: Peter Krempa 
---
 src/remote/remote_driver.c   |  2 ++
 src/remote/remote_protocol.x | 33 -
 src/remote_protocol-structs  | 15 +++
 3 files changed, 49 insertions(+), 1 deletion(-)

diff --git a/src/remote/remote_driver.c b/src/remote/remote_driver.c
index a1384fc655..ddb95914a6 100644
--- a/src/remote/remote_driver.c
+++ b/src/remote/remote_driver.c
@@ -8702,6 +8702,8 @@ static virHypervisorDriver hypervisor_driver = {
 .domainCheckpointDelete = remoteDomainCheckpointDelete, /* 5.6.0 */
 .domainGetGuestInfo = remoteDomainGetGuestInfo, /* 5.7.0 */
 .domainAgentSetResponseTimeout = remoteDomainAgentSetResponseTimeout, /* 
5.10.0 */
+.domainBackupBegin = remoteDomainBackupBegin, /* 6.0.0 */
+.domainBackupGetXMLDesc = remoteDomainBackupGetXMLDesc, /* 6.0.0 */
 };

 static virNetworkDriver network_driver = {
diff --git a/src/remote/remote_protocol.x b/src/remote/remote_protocol.x
index 23e42d17b1..c79cb98ae8 100644
--- a/src/remote/remote_protocol.x
+++ b/src/remote/remote_protocol.x
@@ -3754,6 +3754,23 @@ struct remote_domain_agent_set_response_timeout_ret {
 int result;
 };

+
+struct remote_domain_backup_begin_args {
+remote_nonnull_domain dom;
+remote_string backup_xml;
+remote_string checkpoint_xml;
+unsigned int flags;
+};
+
+struct remote_domain_backup_get_xml_desc_args {
+remote_nonnull_domain dom;
+unsigned int flags;
+};
+
+struct remote_domain_backup_get_xml_desc_ret {
+remote_nonnull_string xml;
+};
+
 /*- Protocol. -*/

 /* Define the program number, protocol version and procedure numbers here. */
@@ -6633,5 +6650,19 @@ enum remote_procedure {
  * @generate: both
  * @acl: domain:write
  */
-REMOTE_PROC_DOMAIN_AGENT_SET_RESPONSE_TIMEOUT = 420
+REMOTE_PROC_DOMAIN_AGENT_SET_RESPONSE_TIMEOUT = 420,
+
+/**
+ * @generate: both
+ * @acl: domain:checkpoint
+ * @acl: domain:block_write
+ */
+REMOTE_PROC_DOMAIN_BACKUP_BEGIN = 421,
+
+/**
+ * @generate: both
+ * @priority: high
+ * @acl: domain:read
+ */
+REMOTE_PROC_DOMAIN_BACKUP_GET_XML_DESC = 422
 };
diff --git a/src/remote_protocol-structs b/src/remote_protocol-structs
index 9ad7a857e0..abc5c5fd2c 100644
--- a/src/remote_protocol-structs
+++ b/src/remote_protocol-structs
@@ -3122,6 +3122,19 @@ struct remote_domain_agent_set_response_timeout_args {
 struct remote_domain_agent_set_response_timeout_ret {
 intresult;
 };
+struct remote_domain_backup_begin_args {
+remote_nonnull_domain  dom;
+remote_string  backup_xml;
+remote_string  checkpoint_xml;
+u_int  flags;
+};
+struct remote_domain_backup_get_xml_desc_args {
+remote_nonnull_domain  dom;
+u_int  flags;
+};
+struct remote_domain_backup_get_xml_desc_ret {
+remote_nonnull_string  xml;
+};
 enum remote_procedure {
 REMOTE_PROC_CONNECT_OPEN = 1,
 REMOTE_PROC_CONNECT_CLOSE = 2,
@@ -3543,4 +3556,6 @@ enum remote_procedure {
 REMOTE_PROC_DOMAIN_GET_GUEST_INFO = 418,
 REMOTE_PROC_CONNECT_SET_IDENTITY = 419,
 REMOTE_PROC_DOMAIN_AGENT_SET_RESPONSE_TIMEOUT = 420,
+REMOTE_PROC_DOMAIN_BACKUP_BEGIN = 421,
+REMOTE_PROC_DOMAIN_BACKUP_GET_XML_DESC = 422,
 };
-- 
2.23.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH v2 13/25] Add 'backup' block job type

2019-12-03 Thread Peter Krempa 
A backup job may consist of many backup sub-blockjobs. Add the new
blockjob type and add all type converter strings.

Signed-off-by: Peter Krempa 
---
 examples/c/misc/event-test.c | 3 +++
 include/libvirt/libvirt-domain.h | 3 +++
 src/conf/domain_conf.c   | 2 +-
 src/qemu/qemu_blockjob.c | 3 +++
 src/qemu/qemu_blockjob.h | 1 +
 src/qemu/qemu_domain.c   | 4 
 src/qemu/qemu_driver.c   | 1 +
 src/qemu/qemu_monitor_json.c | 4 
 tools/virsh-domain.c | 4 +++-
 9 files changed, 23 insertions(+), 2 deletions(-)

diff --git a/examples/c/misc/event-test.c b/examples/c/misc/event-test.c
index 5db572175d..ae282a5027 100644
--- a/examples/c/misc/event-test.c
+++ b/examples/c/misc/event-test.c
@@ -891,6 +891,9 @@ blockJobTypeToStr(int type)

 case VIR_DOMAIN_BLOCK_JOB_TYPE_ACTIVE_COMMIT:
 return "active layer block commit";
+
+case VIR_DOMAIN_BLOCK_JOB_TYPE_BACKUP:
+return "backup";
 }

 return "unknown";
diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h
index 6d1c7f1a3b..f40096af88 100644
--- a/include/libvirt/libvirt-domain.h
+++ b/include/libvirt/libvirt-domain.h
@@ -2446,6 +2446,9 @@ typedef enum {
  * exists as long as sync is active */
 VIR_DOMAIN_BLOCK_JOB_TYPE_ACTIVE_COMMIT = 4,

+/* Backup (virDomainBackupBegin) */
+VIR_DOMAIN_BLOCK_JOB_TYPE_BACKUP = 5,
+
 # ifdef VIR_ENUM_SENTINELS
 VIR_DOMAIN_BLOCK_JOB_TYPE_LAST
 # endif
diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c
index 9580884747..78436a89a2 100644
--- a/src/conf/domain_conf.c
+++ b/src/conf/domain_conf.c
@@ -1227,7 +1227,7 @@ VIR_ENUM_IMPL(virDomainOsDefFirmware,
 VIR_ENUM_DECL(virDomainBlockJob);
 VIR_ENUM_IMPL(virDomainBlockJob,
   VIR_DOMAIN_BLOCK_JOB_TYPE_LAST,
-  "", "", "copy", "", "active-commit",
+  "", "", "copy", "", "active-commit", "",
 );

 VIR_ENUM_IMPL(virDomainMemoryModel,
diff --git a/src/qemu/qemu_blockjob.c b/src/qemu/qemu_blockjob.c
index baa79ea80c..5455eaba65 100644
--- a/src/qemu/qemu_blockjob.c
+++ b/src/qemu/qemu_blockjob.c
@@ -65,6 +65,7 @@ VIR_ENUM_IMPL(qemuBlockjob,
   "copy",
   "commit",
   "active-commit",
+  "backup",
   "",
   "create",
   "broken");
@@ -1276,6 +1277,8 @@ 
qemuBlockJobEventProcessConcludedTransition(qemuBlockJobDataPtr job,
 qemuBlockJobProcessEventConcludedCopyAbort(driver, vm, job, 
asyncJob);
 break;

+case QEMU_BLOCKJOB_TYPE_BACKUP:
+break;

 case QEMU_BLOCKJOB_TYPE_BROKEN:
 case QEMU_BLOCKJOB_TYPE_NONE:
diff --git a/src/qemu/qemu_blockjob.h b/src/qemu/qemu_blockjob.h
index fdfe2c57ec..4734984c99 100644
--- a/src/qemu/qemu_blockjob.h
+++ b/src/qemu/qemu_blockjob.h
@@ -60,6 +60,7 @@ typedef enum {
 QEMU_BLOCKJOB_TYPE_COPY = VIR_DOMAIN_BLOCK_JOB_TYPE_COPY,
 QEMU_BLOCKJOB_TYPE_COMMIT = VIR_DOMAIN_BLOCK_JOB_TYPE_COMMIT,
 QEMU_BLOCKJOB_TYPE_ACTIVE_COMMIT = VIR_DOMAIN_BLOCK_JOB_TYPE_ACTIVE_COMMIT,
+QEMU_BLOCKJOB_TYPE_BACKUP = VIR_DOMAIN_BLOCK_JOB_TYPE_BACKUP,
 /* Additional enum values local to qemu */
 QEMU_BLOCKJOB_TYPE_INTERNAL,
 QEMU_BLOCKJOB_TYPE_CREATE,
diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c
index 4c8ffd60b0..980287d2a0 100644
--- a/src/qemu/qemu_domain.c
+++ b/src/qemu/qemu_domain.c
@@ -2603,6 +2603,8 @@ qemuDomainObjPrivateXMLFormatBlockjobIterator(void 
*payload,
 virBufferAddLit(, " shallownew='yes'");
 break;

+case QEMU_BLOCKJOB_TYPE_BACKUP:
+break;

 case QEMU_BLOCKJOB_TYPE_BROKEN:
 case QEMU_BLOCKJOB_TYPE_NONE:
@@ -3169,6 +3171,8 @@ 
qemuDomainObjPrivateXMLParseBlockjobDataSpecific(qemuBlockJobDataPtr job,
 }
 break;

+case QEMU_BLOCKJOB_TYPE_BACKUP:
+break;

 case QEMU_BLOCKJOB_TYPE_BROKEN:
 case QEMU_BLOCKJOB_TYPE_NONE:
diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c
index 3ebc902d4f..913ab18812 100644
--- a/src/qemu/qemu_driver.c
+++ b/src/qemu/qemu_driver.c
@@ -17427,6 +17427,7 @@ qemuDomainBlockPivot(virQEMUDriverPtr driver,

 case QEMU_BLOCKJOB_TYPE_PULL:
 case QEMU_BLOCKJOB_TYPE_COMMIT:
+case QEMU_BLOCKJOB_TYPE_BACKUP:
 case QEMU_BLOCKJOB_TYPE_INTERNAL:
 case QEMU_BLOCKJOB_TYPE_CREATE:
 case QEMU_BLOCKJOB_TYPE_BROKEN:
diff --git a/src/qemu/qemu_monitor_json.c b/src/qemu/qemu_monitor_json.c
index 9f3783ab70..391f39668a 100644
--- a/src/qemu/qemu_monitor_json.c
+++ b/src/qemu/qemu_monitor_json.c
@@ -1147,6 +1147,8 @@ qemuMonitorJSONHandleBlockJobImpl(qemuMonitorPtr mon,
 type = VIR_DOMAIN_BLOCK_JOB_TYPE_COMMIT;
 else if (STREQ(type_str, "mirror"))
 type = VIR_DOMAIN_BLOCK_JOB_TYPE_COPY;
+else if (STREQ(type_str, "backup"))
+type = VIR_DOMAIN_BLOCK_JOB_TYPE_BACKUP;

 switch ((virConnectDomainEventBlockJobStatus) event) {

[libvirt] [PATCH v2 04/25] backup: Document new XML for backups

2019-12-03 Thread Peter Krempa 
From: Eric Blake 

Prepare for new backup APIs by describing the XML that will represent
a backup.  The XML resembles snapshots and checkpoints in being able
to select actions for a set of disks, but has other differences.  It
can support both push model (the hypervisor does the backup directly
into the destination file) and pull model (the hypervisor exposes an
access port for a third party to grab what is necessary).  Add
testsuite coverage for some minimal uses of the XML.

The  element within  tries to model the same
elements as a  under , but sharing the RNG grammar
proved to be hairy. That is in part because while  use
 to describe a host resource in use by the guest, a backup job
is using a host resource that is not visible to the guest: a push
backup action is instead describing a  (which ultimately could
be a remote network resource, but for simplicity the RNG just
validates a local file for now), and a pull backup action is instead
describing a temporary local file  (which probably should not
be a remote resource).  A future refactoring may thus introduce some
way to parameterize RNG to accept ... so that
the name of the subelement can be  for domain, or  or
 as needed for backups. Future patches may improve this area
of code.

Signed-off-by: Eric Blake 
Signed-off-by: Peter Krempa 
---
 docs/docs.html.in |   3 +-
 docs/format.html.in   |   1 +
 docs/formatbackup.html.in | 175 ++
 docs/formatcheckpoint.html.in |  12 +-
 docs/index.html.in|   3 +-
 docs/schemas/domainbackup.rng | 223 ++
 libvirt.spec.in   |   1 +
 mingw-libvirt.spec.in |   2 +
 tests/Makefile.am |   1 +
 .../backup-pull-seclabel.xml  |  18 ++
 tests/domainbackupxml2xmlin/backup-pull.xml   |  10 +
 .../backup-push-seclabel.xml  |  17 ++
 tests/domainbackupxml2xmlin/backup-push.xml   |  10 +
 tests/domainbackupxml2xmlin/empty.xml |   1 +
 tests/virschematest.c |   1 +
 15 files changed, 470 insertions(+), 8 deletions(-)
 create mode 100644 docs/formatbackup.html.in
 create mode 100644 docs/schemas/domainbackup.rng
 create mode 100644 tests/domainbackupxml2xmlin/backup-pull-seclabel.xml
 create mode 100644 tests/domainbackupxml2xmlin/backup-pull.xml
 create mode 100644 tests/domainbackupxml2xmlin/backup-push-seclabel.xml
 create mode 100644 tests/domainbackupxml2xmlin/backup-push.xml
 create mode 100644 tests/domainbackupxml2xmlin/empty.xml

diff --git a/docs/docs.html.in b/docs/docs.html.in
index 268c16f3b3..f8a949bb53 100644
--- a/docs/docs.html.in
+++ b/docs/docs.html.in
@@ -82,7 +82,8 @@
   node devices,
   secrets,
   snapshots,
-  checkpoints
+  checkpoints,
+  backup jobs

 URI format
 The URI formats used for connecting to libvirt
diff --git a/docs/format.html.in b/docs/format.html.in
index 3be2237663..d013528fe0 100644
--- a/docs/format.html.in
+++ b/docs/format.html.in
@@ -27,6 +27,7 @@
   Secrets
   Snapshots
   Checkpoints
+  Backup jobs
 

 Command line validation
diff --git a/docs/formatbackup.html.in b/docs/formatbackup.html.in
new file mode 100644
index 00..d2e4609c1c
--- /dev/null
+++ b/docs/formatbackup.html.in
@@ -0,0 +1,175 @@
+
+
+http://www.w3.org/1999/xhtml;>
+  
+Backup XML format
+
+
+
+Backup XML
+
+
+  Creating a backup, whether full or incremental, is done
+  via virDomainBackupBegin(), which takes an XML
+  description of the actions to perform, as well as an optional
+  second XML document describing a
+  checkpoint to create at the same point in time. See
+  also a comparison between
+  the various state capture APIs.
+
+
+  There are two general modes for backups: a push mode (where the
+  hypervisor writes out the data to the destination file, which
+  may be local or remote), and a pull mode (where the hypervisor
+  creates an NBD server that a third-party client can then read as
+  needed, and which requires the use of temporary storage,
+  typically local, until the backup is complete).
+
+
+  The instructions for beginning a backup job are provided as
+  attributes and elements of the
+  top-level domainbackup element. This element
+  includes an optional attribute mode which can be
+  either "push" or "pull" (default
+  push). virDomainBackupGetXMLDesc() can be used to
+  see the actual values selected for elements omitted during
+  creation (for example, learning which port the NBD server is
+  using in the pull model or what file names libvirt generated
+  when none were supplied). The following child elements and attributes
+  are supported:
+
+
+  incremental
+

[libvirt] [PATCH v2 06/25] API: Add domain job operation for backups

2019-12-03 Thread Peter Krempa 
Introduce VIR_DOMAIN_JOB_OPERATION_BACKUP into virDomainJobOperation.

Signed-off-by: Peter Krempa 
---
 include/libvirt/libvirt-domain.h | 1 +
 tools/virsh-domain.c | 4 +++-
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h
index 84c5492a7b..6d1c7f1a3b 100644
--- a/include/libvirt/libvirt-domain.h
+++ b/include/libvirt/libvirt-domain.h
@@ -3269,6 +3269,7 @@ typedef enum {
 VIR_DOMAIN_JOB_OPERATION_SNAPSHOT = 6,
 VIR_DOMAIN_JOB_OPERATION_SNAPSHOT_REVERT = 7,
 VIR_DOMAIN_JOB_OPERATION_DUMP = 8,
+VIR_DOMAIN_JOB_OPERATION_BACKUP = 9,

 # ifdef VIR_ENUM_SENTINELS
 VIR_DOMAIN_JOB_OPERATION_LAST
diff --git a/tools/virsh-domain.c b/tools/virsh-domain.c
index bb942267f0..5c313279d7 100644
--- a/tools/virsh-domain.c
+++ b/tools/virsh-domain.c
@@ -6068,7 +6068,9 @@ VIR_ENUM_IMPL(virshDomainJobOperation,
   N_("Outgoing migration"),
   N_("Snapshot"),
   N_("Snapshot revert"),
-  N_("Dump"));
+  N_("Dump"),
+  N_("Backup"),
+);

 static const char *
 virshDomainJobOperationToString(int op)
-- 
2.23.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH v2 12/25] qemu: domain: Introduce QEMU_ASYNC_JOB_BACKUP async job type

2019-12-03 Thread Peter Krempa 
We will want to use the async job infrastructure along with all the APIs
and event for the backup job so add the backup job as a new async job
type.

Signed-off-by: Peter Krempa 
---
 src/qemu/qemu_domain.c|  3 +++
 src/qemu/qemu_domain.h|  1 +
 src/qemu/qemu_migration.c |  2 ++
 src/qemu/qemu_process.c   | 25 +
 4 files changed, 31 insertions(+)

diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c
index c1b0f81c81..4c8ffd60b0 100644
--- a/src/qemu/qemu_domain.c
+++ b/src/qemu/qemu_domain.c
@@ -114,6 +114,7 @@ VIR_ENUM_IMPL(qemuDomainAsyncJob,
   "dump",
   "snapshot",
   "start",
+  "backup",
 );

 VIR_ENUM_IMPL(qemuDomainNamespace,
@@ -210,6 +211,7 @@ qemuDomainAsyncJobPhaseToString(qemuDomainAsyncJob job,
 case QEMU_ASYNC_JOB_SNAPSHOT:
 case QEMU_ASYNC_JOB_START:
 case QEMU_ASYNC_JOB_NONE:
+case QEMU_ASYNC_JOB_BACKUP:
 G_GNUC_FALLTHROUGH;
 case QEMU_ASYNC_JOB_LAST:
 break;
@@ -235,6 +237,7 @@ qemuDomainAsyncJobPhaseFromString(qemuDomainAsyncJob job,
 case QEMU_ASYNC_JOB_SNAPSHOT:
 case QEMU_ASYNC_JOB_START:
 case QEMU_ASYNC_JOB_NONE:
+case QEMU_ASYNC_JOB_BACKUP:
 G_GNUC_FALLTHROUGH;
 case QEMU_ASYNC_JOB_LAST:
 break;
diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h
index a552af6180..4cd7eec4ce 100644
--- a/src/qemu/qemu_domain.h
+++ b/src/qemu/qemu_domain.h
@@ -104,6 +104,7 @@ typedef enum {
 QEMU_ASYNC_JOB_DUMP,
 QEMU_ASYNC_JOB_SNAPSHOT,
 QEMU_ASYNC_JOB_START,
+QEMU_ASYNC_JOB_BACKUP,

 QEMU_ASYNC_JOB_LAST
 } qemuDomainAsyncJob;
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c
index dabdda2715..d0e2b65d01 100644
--- a/src/qemu/qemu_migration.c
+++ b/src/qemu/qemu_migration.c
@@ -1415,6 +1415,8 @@ qemuMigrationJobName(virDomainObjPtr vm)
 return _("snapshot job");
 case QEMU_ASYNC_JOB_START:
 return _("start job");
+case QEMU_ASYNC_JOB_BACKUP:
+return _("backup job");
 case QEMU_ASYNC_JOB_LAST:
 default:
 return _("job");
diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c
index 75ee3893c6..cca280992f 100644
--- a/src/qemu/qemu_process.c
+++ b/src/qemu/qemu_process.c
@@ -89,6 +89,7 @@
 #include "virresctrl.h"
 #include "virvsock.h"
 #include "viridentity.h"
+#include "virthreadjob.h"

 #define VIR_FROM_THIS VIR_FROM_QEMU

@@ -3583,6 +3584,7 @@ qemuProcessRecoverJob(virQEMUDriverPtr driver,
 qemuDomainObjPrivatePtr priv = vm->privateData;
 virDomainState state;
 int reason;
+unsigned long long now;

 state = virDomainObjGetState(vm, );

@@ -3632,6 +3634,29 @@ qemuProcessRecoverJob(virQEMUDriverPtr driver,
 /* Already handled in VIR_DOMAIN_PAUSED_STARTING_UP check. */
 break;

+case QEMU_ASYNC_JOB_BACKUP:
+ignore_value(virTimeMillisNow());
+
+/* Restore the config of the async job which is not persisted */
+priv->jobs_queued++;
+priv->job.asyncJob = QEMU_ASYNC_JOB_BACKUP;
+priv->job.asyncOwnerAPI = virThreadJobGet();
+priv->job.asyncStarted = now;
+
+qemuDomainObjSetAsyncJobMask(vm, (QEMU_JOB_DEFAULT_MASK |
+  JOB_MASK(QEMU_JOB_SUSPEND) |
+  JOB_MASK(QEMU_JOB_MODIFY)));
+
+/* We reset the job parameters for backup so that the job will look
+ * active. This is possible because we are able to recover the state
+ * of blockjobs and also the backup job allows all sub-job types */
+priv->job.current = g_new0(qemuDomainJobInfo, 1);
+priv->job.current->operation = VIR_DOMAIN_JOB_OPERATION_BACKUP;
+priv->job.current->statsType = QEMU_DOMAIN_JOB_STATS_TYPE_BACKUP;
+priv->job.current->status = QEMU_DOMAIN_JOB_STATUS_ACTIVE;
+priv->job.current->started = now;
+break;
+
 case QEMU_ASYNC_JOB_NONE:
 case QEMU_ASYNC_JOB_LAST:
 break;
-- 
2.23.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH v2 01/25] qemu: domain: Export qemuDomainGetImageIds

2019-12-03 Thread Peter Krempa 
Signed-off-by: Peter Krempa 
---
 src/qemu/qemu_domain.c | 2 +-
 src/qemu/qemu_domain.h | 7 +++
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c
index 470d342afc..8d2923300d 100644
--- a/src/qemu/qemu_domain.c
+++ b/src/qemu/qemu_domain.c
@@ -10240,7 +10240,7 @@ qemuDomainCleanupRun(virQEMUDriverPtr driver,
 priv->ncleanupCallbacks_max = 0;
 }

-static void
+void
 qemuDomainGetImageIds(virQEMUDriverConfigPtr cfg,
   virDomainObjPtr vm,
   virStorageSourcePtr src,
diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h
index f626d3a54c..608546a27c 100644
--- a/src/qemu/qemu_domain.h
+++ b/src/qemu/qemu_domain.h
@@ -840,6 +840,13 @@ bool qemuDomainDiskChangeSupported(virDomainDiskDefPtr 
disk,
 const char *qemuDomainDiskNodeFormatLookup(virDomainObjPtr vm,
const char *disk);

+void qemuDomainGetImageIds(virQEMUDriverConfigPtr cfg,
+   virDomainObjPtr vm,
+   virStorageSourcePtr src,
+   virStorageSourcePtr parentSrc,
+   uid_t *uid,
+   gid_t *gid);
+
 int qemuDomainStorageFileInit(virQEMUDriverPtr driver,
   virDomainObjPtr vm,
   virStorageSourcePtr src,
-- 
2.23.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH v2 02/25] API: Introduce field for reporting temporary disk space usage of a domain job

2019-12-03 Thread Peter Krempa 
A pull mode backup job uses temporary disk images to hold the changed
parts of the disk while the client is copying the changes. Since usage
of the temporary space can be monitored but doesn't really fit any of
the existing stats fields introduce new fields for reporting this data.

Signed-off-by: Peter Krempa 
Reviewed-by: Daniel P. Berrangé 
---
 include/libvirt/libvirt-domain.h | 14 ++
 1 file changed, 14 insertions(+)

diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h
index 40c71091ec..b9908fe7b0 100644
--- a/include/libvirt/libvirt-domain.h
+++ b/include/libvirt/libvirt-domain.h
@@ -3586,6 +3586,20 @@ typedef enum {
  */
 # define VIR_DOMAIN_JOB_SUCCESS "success"

+/**
+ * VIR_DOMAIN_JOB_DISK_TEMP_USED:
+ * virDomainGetJobStats field: current usage of temporary disk space for the
+ * job in bytes as VIR_TYPED_PARAM_ULLONG.
+ */
+# define VIR_DOMAIN_JOB_DISK_TEMP_USED "disk_temp_used"
+
+/**
+ * VIR_DOMAIN_JOB_DISK_TEMP_TOTAL:
+ * virDomainGetJobStats field: possible total temporary disk space for the
+ * job in bytes as VIR_TYPED_PARAM_ULLONG.
+ */
+# define VIR_DOMAIN_JOB_DISK_TEMP_TOTAL "disk_temp_total"
+
 /**
  * virConnectDomainEventGenericCallback:
  * @conn: the connection pointer
-- 
2.23.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH] scripts: fix list indexing when printing mis-ordered symbols

2019-12-03 Thread Daniel P . Berrangé 
The python array slice syntax expects the first and last indexes,
not the first length and element count.

Signed-off-by: Daniel P. Berrangé 
---
 scripts/check-symsorting.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/check-symsorting.py b/scripts/check-symsorting.py
index 8e698c0657..006c42f1b3 100755
--- a/scripts/check-symsorting.py
+++ b/scripts/check-symsorting.py
@@ -59,8 +59,8 @@ def check_sorting(group, symfile, line, groupfile, lastgroup):
 issorted = False
 
 if not issorted:
-actual = group[first:(last - first + 1)]
-expect = sortedgroup[first:(last - first + 1)]
+actual = group[first:last]
+expect = sortedgroup[first:last]
 print("Symbol block at %s:%s: symbols not sorted" %
   (symfile, line), file=sys.stderr)
 for g in actual:
-- 
2.23.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH] spec: mingw: add deps for genprotocol.pl usage

2019-12-03 Thread Cole Robinson 
As of 0985a9597bb03 we no longer distribute generated files. mingw
needs to call genprotocol.pl itself, which means it needs rpcgen
and cpp installed.

Signed-off-by: Cole Robinson 
---
 mingw-libvirt.spec.in | 4 
 1 file changed, 4 insertions(+)

diff --git a/mingw-libvirt.spec.in b/mingw-libvirt.spec.in
index c29f3eeed2..fd9b729591 100644
--- a/mingw-libvirt.spec.in
+++ b/mingw-libvirt.spec.in
@@ -89,6 +89,10 @@ BuildRequires: mingw64-libssh2
 BuildRequires: mingw32-curl
 BuildRequires: mingw64-curl
 %endif
+BuildRequires: cpp
+%if 0%{?fedora} || 0%{?rhel} > 7
+BuildRequires: rpcgen
+%endif
 
 BuildArch:  noarch
 
-- 
2.23.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v2] driver: Include source as a flag to virDomainGetHostname

2019-12-03 Thread Michal Privoznik 

On 12/2/19 4:24 AM, Julio Faracco wrote:

There is a lots of possibilities to retrieve hostname information from
domain. Libvirt could use lease information from dnsmasq to get current
hostname too. QEMU supports QEMU-agent but it can use lease source. See
'domifaddr' as an example.

This commit still adds lease options for QEMU. It will get the first
hostname available from domain networks.

This case, every driver has a default section inside switch to keep
compatibility. So, if someone call 'domhostname' without specifying
source, it will get the default option.

Signed-off-by: Julio Faracco 
---
  include/libvirt/libvirt-domain.h |  5 +++
  src/lxc/lxc_driver.c | 71 
  src/openvz/openvz_driver.c   | 30 ++
  src/qemu/qemu_driver.c   | 69 ++-
  tools/virsh-domain.c | 25 ++-
  5 files changed, 178 insertions(+), 22 deletions(-)

diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h
index a2f007568c..b37f33d5d0 100644
--- a/include/libvirt/libvirt-domain.h
+++ b/include/libvirt/libvirt-domain.h
@@ -4790,6 +4790,11 @@ typedef struct _virTypedParameter virMemoryParameter;
   */
  typedef virMemoryParameter *virMemoryParameterPtr;
  
+typedef enum {

+VIR_DOMAIN_HOSTNAME_SRC_LEASE = (1 << 0), /* Parse DHCP lease file */
+VIR_DOMAIN_HOSTNAME_SRC_AGENT = (1 << 1), /* Query qemu guest agent */
+} virDomainHostnameSource;
+


You need to document that virDomainGetHostname() uses these new flags. 
And also virDomainGetHostnameFlags would probably be a better name for 
the enum then. And since the enum is changing its name its members 
should reflect that too.



  typedef enum {
  VIR_DOMAIN_INTERFACE_ADDRESSES_SRC_LEASE = 0, /* Parse DHCP lease file */
  VIR_DOMAIN_INTERFACE_ADDRESSES_SRC_AGENT = 1, /* Query qemu guest agent */
diff --git a/src/lxc/lxc_driver.c b/src/lxc/lxc_driver.c
index 826bf074e3..3221b06261 100644
--- a/src/lxc/lxc_driver.c
+++ b/src/lxc/lxc_driver.c
@@ -5321,6 +5321,76 @@ lxcDomainGetCPUStats(virDomainPtr dom,
  return ret;
  }
  
+static char *

+lxcDomainGetHostname(virDomainPtr dom,
+ unsigned int flags)
+{
+virDomainObjPtr vm = NULL;
+char macaddr[VIR_MAC_STRING_BUFLEN];
+g_autoptr(virNetwork) network = NULL;
+virNetworkDHCPLeasePtr *leases = NULL;
+int n_leases;
+size_t i, j;
+char *hostname = NULL;


A lot of these are used in the for() loop. Might be worth declaring them 
for better readability.



+
+virCheckFlags(VIR_DOMAIN_HOSTNAME_SRC_LEASE |
+  VIR_DOMAIN_HOSTNAME_SRC_AGENT, NULL);


Since AGENT is not supported by this API you can leave it out from here 
and virCheckFlags() will produce sensible error.



+
+if (!(vm = lxcDomObjFromDomain(dom)))
+return NULL;
+
+if (virDomainGetHostnameEnsureACL(dom->conn, vm->def) < 0)
+goto cleanup;
+


This looks like a good place to check if the domain is alive. Inactive 
domains don't have hostnames, do they?



+switch (flags) {


Huh, are these two flags mutually exclusive? I know that you will end up 
with just one flag here (and thus may remove the switch() completely), 
but usually flags are tested for like this:


  if (flags & FLAG1) { ... }
  if (flags & FLAG2) { ... }

If we want to make the flags exclusive (which is okay), then you'll also 
need VIR_EXCLUSIVE_FLAGS_RET() and it could go right after 
virCheckFlags() call. Again, I know it's not needed for lxc case, but 
looks like qemu driver will need it.



+default:
+case VIR_DOMAIN_HOSTNAME_SRC_LEASE:
+for (i = 0; i < vm->def->nnets; i++) {
+if (vm->def->nets[i]->type != VIR_DOMAIN_NET_TYPE_NETWORK)
+continue;
+
+virMacAddrFormat(&(vm->def->nets[i]->mac), macaddr);
+virObjectUnref(network);


If you'd declare @network here this unref won't be needed.


+network = virNetworkLookupByName(dom->conn,
+ 
vm->def->nets[i]->data.network.name);


There are two problems with this:

1) it breaks split daemon - dom->conn is not the correct connection 
object. You'll need to call virGetConnectNetwork() to obtain a 
connection object to the network driver and then pass it to this 
function (note, you don't have to obtain the connection object inside 
this loop, it can be done outside). I know we have a couple of bad 
examples in our code and I will be proposing a fix shortly.


2) The function can return NULL. Unlikely, but can happen.


+
+if ((n_leases = virNetworkGetDHCPLeases(network, macaddr,
+, 0)) < 0)
+virReportError(VIR_ERR_CONFIG_UNSUPPORTED,
+   _("There is no available hostname %d"),
+   flags);


No need to overwirte the sensible error message produced by the API.

[libvirt] [PATCH 0/4] Introduce VIR_MIGRATE_PARAM_TLS_DESTINATION migration param

2019-12-03 Thread Jiri Denemark 
Normally the TLS certificate from the destination host must match the
host's name for TLS verification to succeed. When the certificate does
not match the destination hostname and the expected cetificate's
hostname is known, this parameter can be used to pass this expected
hostname when starting the migration.

Jiri Denemark (4):
  qemu: Add support for setting string migration params
  Introduce VIR_MIGRATE_PARAM_TLS_DESTINATION migration param
  qemu: Implement VIR_MIGRATE_PARAM_TLS_DESTINATION
  virsh: Add --tls-destination option for migrate command

 include/libvirt/libvirt-domain.h | 14 +++
 src/qemu/qemu_migration.h|  1 +
 src/qemu/qemu_migration_params.c | 65 +++-
 tools/virsh-domain.c | 11 ++
 tools/virsh.pod  |  8 +++-
 5 files changed, 96 insertions(+), 3 deletions(-)

-- 
2.24.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH 3/4] qemu: Implement VIR_MIGRATE_PARAM_TLS_DESTINATION

2019-12-03 Thread Jiri Denemark 
Signed-off-by: Jiri Denemark 
---
 src/qemu/qemu_migration.h| 1 +
 src/qemu/qemu_migration_params.c | 9 -
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/src/qemu/qemu_migration.h b/src/qemu/qemu_migration.h
index d98fe9f80a..b6f88d3fd9 100644
--- a/src/qemu/qemu_migration.h
+++ b/src/qemu/qemu_migration.h
@@ -83,6 +83,7 @@
 VIR_MIGRATE_PARAM_AUTO_CONVERGE_INCREMENT,  VIR_TYPED_PARAM_INT, \
 VIR_MIGRATE_PARAM_BANDWIDTH_POSTCOPY, VIR_TYPED_PARAM_ULLONG, \
 VIR_MIGRATE_PARAM_PARALLEL_CONNECTIONS, VIR_TYPED_PARAM_INT, \
+VIR_MIGRATE_PARAM_TLS_DESTINATION, VIR_TYPED_PARAM_STRING, \
 NULL
 
 
diff --git a/src/qemu/qemu_migration_params.c b/src/qemu/qemu_migration_params.c
index 88c08528bc..e6cf6b4b0b 100644
--- a/src/qemu/qemu_migration_params.c
+++ b/src/qemu/qemu_migration_params.c
@@ -197,6 +197,10 @@ static const qemuMigrationParamsTPMapItem 
qemuMigrationParamsTPMap[] = {
 {.typedParam = VIR_MIGRATE_PARAM_PARALLEL_CONNECTIONS,
  .param = QEMU_MIGRATION_PARAM_MULTIFD_CHANNELS,
  .party = QEMU_MIGRATION_SOURCE | QEMU_MIGRATION_DESTINATION},
+
+{.typedParam = VIR_MIGRATE_PARAM_TLS_DESTINATION,
+ .param = QEMU_MIGRATION_PARAM_TLS_HOSTNAME,
+ .party = QEMU_MIGRATION_SOURCE},
 };
 
 static const qemuMigrationParamType qemuMigrationParamTypes[] = {
@@ -1007,7 +1011,10 @@ qemuMigrationParamsEnableTLS(virQEMUDriverPtr driver,
 
 if (qemuMigrationParamsSetString(migParams,
  QEMU_MIGRATION_PARAM_TLS_CREDS,
- *tlsAlias) < 0 ||
+ *tlsAlias) < 0)
+goto error;
+
+if (!migParams->params[QEMU_MIGRATION_PARAM_TLS_HOSTNAME].set &&
 qemuMigrationParamsSetString(migParams,
  QEMU_MIGRATION_PARAM_TLS_HOSTNAME,
  NULLSTR_EMPTY(hostname)) < 0)
-- 
2.24.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH 1/4] qemu: Add support for setting string migration params

2019-12-03 Thread Jiri Denemark 
The functions for converting migration typed parameters to QEMU
migration parameters and back were only implemented for integer types.
This patch adds support for string parameters.

Signed-off-by: Jiri Denemark 
---
 src/qemu/qemu_migration_params.c | 56 
 1 file changed, 56 insertions(+)

diff --git a/src/qemu/qemu_migration_params.c b/src/qemu/qemu_migration_params.c
index c28814bc5b..88c08528bc 100644
--- a/src/qemu/qemu_migration_params.c
+++ b/src/qemu/qemu_migration_params.c
@@ -412,6 +412,51 @@ qemuMigrationParamsSetTPULL(qemuMigrationParamsPtr 
migParams,
 }
 
 
+static int
+qemuMigrationParamsGetTPString(qemuMigrationParamsPtr migParams,
+   qemuMigrationParam param,
+   virTypedParameterPtr params,
+   int nparams,
+   const char *name)
+{
+const char *value = NULL;
+int rc;
+
+if (qemuMigrationParamsCheckType(param, QEMU_MIGRATION_PARAM_TYPE_STRING) 
< 0)
+return -1;
+
+if (!params)
+return 0;
+
+if ((rc = virTypedParamsGetString(params, nparams, name, )) < 0)
+return -1;
+
+migParams->params[param].value.s = g_strdup(value);
+migParams->params[param].set = !!rc;
+return 0;
+}
+
+
+static int
+qemuMigrationParamsSetTPString(qemuMigrationParamsPtr migParams,
+   qemuMigrationParam param,
+   virTypedParameterPtr *params,
+   int *nparams,
+   int *maxparams,
+   const char *name)
+{
+if (qemuMigrationParamsCheckType(param, QEMU_MIGRATION_PARAM_TYPE_STRING) 
< 0)
+return -1;
+
+if (!migParams->params[param].set)
+return 0;
+
+return virTypedParamsAddString(params, nparams, maxparams, name,
+   migParams->params[param].value.s);
+}
+
+
+
 static int
 qemuMigrationParamsSetCompression(virTypedParameterPtr params,
   int nparams,
@@ -536,7 +581,12 @@ qemuMigrationParamsFromFlags(virTypedParameterPtr params,
 break;
 
 case QEMU_MIGRATION_PARAM_TYPE_BOOL:
+break;
+
 case QEMU_MIGRATION_PARAM_TYPE_STRING:
+if (qemuMigrationParamsGetTPString(migParams, item->param, params,
+   nparams, item->typedParam) < 0)
+goto error;
 break;
 }
 }
@@ -612,7 +662,13 @@ qemuMigrationParamsDump(qemuMigrationParamsPtr migParams,
 break;
 
 case QEMU_MIGRATION_PARAM_TYPE_BOOL:
+break;
+
 case QEMU_MIGRATION_PARAM_TYPE_STRING:
+if (qemuMigrationParamsSetTPString(migParams, item->param,
+   params, nparams, maxparams,
+   item->typedParam) < 0)
+return -1;
 break;
 }
 }
-- 
2.24.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH 2/4] Introduce VIR_MIGRATE_PARAM_TLS_DESTINATION migration param

2019-12-03 Thread Jiri Denemark 
Normally the TLS certificate from the destination host must match the
host's name for TLS verification to succeed. When the certificate does
not match the destination hostname and the expected cetificate's
hostname is known, this parameter can be used to pass this expected
hostname when starting the migration.

Signed-off-by: Jiri Denemark 
---
 include/libvirt/libvirt-domain.h | 14 ++
 1 file changed, 14 insertions(+)

diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h
index 40c71091ec..a099b3d891 100644
--- a/include/libvirt/libvirt-domain.h
+++ b/include/libvirt/libvirt-domain.h
@@ -1051,6 +1051,20 @@ typedef enum {
  */
 # define VIR_MIGRATE_PARAM_PARALLEL_CONNECTIONS "parallel.connections"
 
+/**
+ * VIR_MIGRATE_PARAM_TLS_DESTINATION:
+ *
+ * virDomainMigrate* params field: override the destination host name used for
+ * TLS verification. As VIR_TYPED_PARAM_STRING.
+ *
+ * Normally the TLS certificate from the destination host must match the host's
+ * name for TLS verification to succeed. When the certificate does not match
+ * the destination hostname and the expected cetificate's hostname is known,
+ * this parameter can be used to pass this expected hostname when starting
+ * the migration.
+ */
+# define VIR_MIGRATE_PARAM_TLS_DESTINATION  "tls.destination"
+
 /* Domain migration. */
 virDomainPtr virDomainMigrate (virDomainPtr domain, virConnectPtr dconn,
unsigned long flags, const char *dname,
-- 
2.24.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH 4/4] virsh: Add --tls-destination option for migrate command

2019-12-03 Thread Jiri Denemark 
This option can be used to override the destination host name used for
TLS verification.

Signed-off-by: Jiri Denemark 
---
 tools/virsh-domain.c | 11 +++
 tools/virsh.pod  |  8 ++--
 2 files changed, 17 insertions(+), 2 deletions(-)

diff --git a/tools/virsh-domain.c b/tools/virsh-domain.c
index 21ea1a69ea..c2cfcf409d 100644
--- a/tools/virsh-domain.c
+++ b/tools/virsh-domain.c
@@ -10566,6 +10566,10 @@ static const vshCmdOptDef opts_migrate[] = {
  .type = VSH_OT_INT,
  .help = N_("migration bandwidth limit in MiB/s")
 },
+{.name = "tls-destination",
+ .type = VSH_OT_STRING,
+ .help = N_("override the destination host name used for TLS verification")
+},
 {.name = NULL}
 };
 
@@ -10789,6 +10793,13 @@ doMigrate(void *opaque)
 goto save_error;
 }
 
+if (vshCommandOptStringReq(ctl, cmd, "tls-destination", ) < 0)
+goto out;
+if (opt &&
+virTypedParamsAddString(, , ,
+VIR_MIGRATE_PARAM_TLS_DESTINATION, opt) < 0)
+goto save_error;
+
 if (vshCommandOptBool(cmd, "live"))
 flags |= VIR_MIGRATE_LIVE;
 if (vshCommandOptBool(cmd, "p2p"))
diff --git a/tools/virsh.pod b/tools/virsh.pod
index a8331154e1..aaf1eba825 100644
--- a/tools/virsh.pod
+++ b/tools/virsh.pod
@@ -2174,7 +2174,7 @@ I I [I] [I] 
[I] [I] [I<--persistent-xml> B] [I<--tls>]
 [I<--postcopy-bandwidth> B]
 [I<--parallel> [I<--parallel-connections> B]]
-[I<--bandwidth> B]
+[I<--bandwidth> B] [I<--tls-destination> B]
 
 Migrate domain to another host.  Add I<--live> for live migration; <--p2p>
 for peer-2-peer migration; I<--direct> for direct migration; or I<--tunnelled>
@@ -2267,7 +2267,11 @@ respectively. I<--comp-xbzrle-cache> sets size of page 
cache in bytes.
 Providing I<--tls> causes the migration to use the host configured TLS setup
 (see migrate_tls_x509_cert_dir in /etc/libvirt/qemu.conf) in order to perform
 the migration of the domain. Usage requires proper TLS setup for both source
-and target.
+and target. Normally the TLS certificate from the destination host must match
+the host's name for TLS verification to succeed. When the certificate does not
+match the destination hostname and the expected cetificate's hostname is
+known, I<--tls-destination> can be used to pass the expected B when
+starting the migration.
 
 I<--parallel> option will cause migration data to be sent over multiple
 parallel connections. The number of such connections can be set using
-- 
2.24.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH] tests: stop static linking to libvirt code in tests

2019-12-03 Thread Daniel P . Berrangé 
If we static link to libvirt_util.la then we can't override functions in
this file by simply implementing them in the test code. Any tests should
dynamic link to the main libvirt.la and ensure symbols are exported.

Signed-off-by: Daniel P. Berrangé 
---
 src/libvirt_private.syms |  2 ++
 tests/Makefile.am| 11 +++
 2 files changed, 5 insertions(+), 8 deletions(-)

diff --git a/src/libvirt_private.syms b/src/libvirt_private.syms
index 8fe0bf9365..0864de030b 100644
--- a/src/libvirt_private.syms
+++ b/src/libvirt_private.syms
@@ -915,6 +915,8 @@ virDomainSnapshotDefFormat;
 virDomainSnapshotDefIsExternal;
 virDomainSnapshotDefNew;
 virDomainSnapshotDefParseString;
+virDomainSnapshotDiskDefFree;
+virDomainSnapshotDiskDefParseXML;
 virDomainSnapshotFormatConvertXMLFlags;
 virDomainSnapshotIsExternal;
 virDomainSnapshotLocationTypeFromString;
diff --git a/tests/Makefile.am b/tests/Makefile.am
index e009de830c..af8fd69842 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -662,8 +662,7 @@ qemublocktest_SOURCES = \
$(NULL)
 qemublocktest_LDADD = \
libqemumonitortestutils.la \
-   ../src/libvirt_conf.la \
-   ../src/libvirt_util.la \
+   ../src/libvirt.la \
$(qemu_LDADDS) \
$(NULL)
 
@@ -939,8 +938,7 @@ storagevolxml2argvtest_SOURCES = \
 storagevolxml2argvtest_LDADD = \
$(LIBXML_LIBS) \
../src/libvirt_driver_storage_impl.la \
-   ../src/libvirt_conf.la \
-   ../src/libvirt_util.la \
+   ../src/libvirt.la \
$(LDADDS)
 
 storagepoolxml2argvtest_SOURCES = \
@@ -949,8 +947,7 @@ storagepoolxml2argvtest_SOURCES = \
 storagepoolxml2argvtest_LDADD = \
$(LIBXML_LIBS) \
../src/libvirt_driver_storage_impl.la \
-   ../src/libvirt_conf.la \
-   ../src/libvirt_util.la \
+   ../src/libvirt.la \
$(LDADDS)
 
 storagepoolxml2xmltest_SOURCES = \
@@ -1150,8 +1147,6 @@ virstoragetest_SOURCES = \
virstoragetest.c testutils.h testutils.c
 virstoragetest_LDADD = $(LDADDS) \
../src/libvirt.la \
-   ../src/libvirt_conf.la \
-   ../src/libvirt_util.la \
../src/libvirt_driver_storage_impl.la \
../gnulib/lib/libgnu.la \
$(NULL)
-- 
2.23.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v2 04/12] esx: implement connectListAllStoragePools

2019-12-03 Thread Ján Tomko 

On Fri, Nov 15, 2019 at 01:40:43PM +0100, Pino Toscano wrote:

Implement the .connectListAllStoragePools storage API in the esx storage
driver, and in all its subdrivers.

Signed-off-by: Pino Toscano 
---
src/esx/esx_storage_backend_iscsi.c | 72 +
src/esx/esx_storage_backend_vmfs.c  | 98 +
src/esx/esx_storage_driver.c| 68 
3 files changed, 238 insertions(+)

diff --git a/src/esx/esx_storage_backend_iscsi.c 
b/src/esx/esx_storage_backend_iscsi.c
index 72ab0d3cb0..4f5d8e5e24 100644
--- a/src/esx/esx_storage_backend_iscsi.c
+++ b/src/esx/esx_storage_backend_iscsi.c
@@ -779,6 +779,77 @@ esxStorageVolGetPath(virStorageVolPtr volume)



+#define MATCH(FLAG) (flags & (FLAG))
+static int
+esxConnectListAllStoragePools(virConnectPtr conn,
+  virStoragePoolPtr **pools,
+  unsigned int flags)
+{
+bool success = false;
+size_t count = 0;


size_t is unsigned... [0]


+esxPrivate *priv = conn->privateData;
+esxVI_HostInternetScsiHba *hostInternetScsiHba = NULL;
+esxVI_HostInternetScsiHbaStaticTarget *target;
+size_t i;
+
+/* this driver provides only iSCSI pools */
+if (MATCH(VIR_CONNECT_LIST_STORAGE_POOLS_FILTERS_POOL_TYPE) &&
+!(MATCH(VIR_CONNECT_LIST_STORAGE_POOLS_ISCSI)))
+return 0;
+
+if (esxVI_LookupHostInternetScsiHba(priv->primary,
+) < 0) {
+virReportError(VIR_ERR_INTERNAL_ERROR, "%s",
+   _("Unable to obtain iSCSI adapter"));
+goto cleanup;
+}
+
+/* FIXME: code looks for software iSCSI adapter only */
+if (!hostInternetScsiHba) {
+/* iSCSI adapter may not be enabled for this host */
+return 0;
+}
+
+/*
+ * ESX has two kind of targets:
+ * 1. staticIscsiTargets
+ * 2. dynamicIscsiTargets
+ * For each dynamic target if its reachable a static target is added.


it's


+ * return iSCSI names for all static targets to avoid duplicate names.
+ */
+for (target = hostInternetScsiHba->configuredStaticTarget;
+ target; target = target->_next) {
+virStoragePoolPtr pool;
+
+pool = targetToStoragePool(conn, target->iScsiName, target);
+if (!pool)
+goto cleanup;
+
+if (VIR_APPEND_ELEMENT(*pools, count, pool) < 0)
+goto cleanup;
+}
+
+success = true;
+
+ cleanup:
+if (! success) {
+if (*pools) {
+for (i = 0; i < count; ++i)
+VIR_FREE((*pools)[i]);
+VIR_FREE(*pools);
+}
+
+count = -1;


[0] so assigning negative value feels wrong.

using

   int ret = -1;
   ...

   ret = count;
cleanup:
  if (ret < 0) {
  }

gets rid of that assignment.


+}
+
+esxVI_HostInternetScsiHba_Free();
+
+return count;
+}
+#undef MATCH
+
+
+
virStorageDriver esxStorageBackendISCSI = {
.connectNumOfStoragePools = esxConnectNumOfStoragePools, /* 1.0.1 */
.connectListStoragePools = esxConnectListStoragePools, /* 1.0.1 */
@@ -799,4 +870,5 @@ virStorageDriver esxStorageBackendISCSI = {
.storageVolDelete = esxStorageVolDelete, /* 1.0.1 */
.storageVolWipe = esxStorageVolWipe, /* 1.0.1 */
.storageVolGetPath = esxStorageVolGetPath, /* 1.0.1 */
+.connectListAllStoragePools = esxConnectListAllStoragePools, /* 5.10.0 */


6.0.0


};
diff --git a/src/esx/esx_storage_backend_vmfs.c 
b/src/esx/esx_storage_backend_vmfs.c
index b890825a40..05b273aed7 100644
--- a/src/esx/esx_storage_backend_vmfs.c
+++ b/src/esx/esx_storage_backend_vmfs.c


[...]


+
+success = true;
+
+ cleanup:
+if (! success) {
+if (*pools) {
+for (i = 0; i < count; ++i)
+VIR_FREE((*pools)[i]);
+VIR_FREE(*pools);
+}
+
+count = -1;


same here


+}
+
+esxVI_String_Free();
+esxVI_ObjectContent_Free();
+
+return count;
+}
+#undef MATCH
+
+
+
virStorageDriver esxStorageBackendVMFS = {
.connectNumOfStoragePools = esxConnectNumOfStoragePools, /* 0.8.2 */
.connectListStoragePools = esxConnectListStoragePools, /* 0.8.2 */
@@ -1480,4 +1577,5 @@ virStorageDriver esxStorageBackendVMFS = {
.storageVolGetInfo = esxStorageVolGetInfo, /* 0.8.4 */
.storageVolGetXMLDesc = esxStorageVolGetXMLDesc, /* 0.8.4 */
.storageVolGetPath = esxStorageVolGetPath, /* 0.8.4 */
+.connectListAllStoragePools = esxConnectListAllStoragePools, /* 5.10.0 */


6.0.0


};
diff --git a/src/esx/esx_storage_driver.c b/src/esx/esx_storage_driver.c
index 8a34732b45..6d17ac28ea 100644
--- a/src/esx/esx_storage_driver.c
+++ b/src/esx/esx_storage_driver.c
@@ -517,6 +517,73 @@ esxStoragePoolIsPersistent(virStoragePoolPtr pool 
G_GNUC_UNUSED)



+#define MATCH(FLAG) (flags & (FLAG))
+static int
+esxConnectListAllStoragePools(virConnectPtr conn,
+  virStoragePoolPtr **pools,
+  unsigned int

Re: [libvirt] [PATCH 2/2] Add -mem-shared option

2019-12-03 Thread Igor Mammedov 
On Tue, 3 Dec 2019 09:56:15 +0100
Thomas Huth  wrote:

> On 02/12/2019 22.00, Eduardo Habkost wrote:
> > On Mon, Dec 02, 2019 at 08:39:48AM +0100, Igor Mammedov wrote:  
> >> On Fri, 29 Nov 2019 18:46:12 +0100
> >> Paolo Bonzini  wrote:
> >>  
> >>> On 29/11/19 13:16, Igor Mammedov wrote:  
>  As for "-m", I'd make it just an alias that translates
>   -m/mem-path/mem-prealloc
> >>>
> >>> I think we should just deprecate -mem-path/-mem-prealloc in 5.0.  CCing
> >>> Thomas as mister deprecation. :)  
> >>
> >> I'll add that to my series  
> > 
> > Considering that the plan is to eventually reimplement those
> > options as syntactic sugar for memory backend options (hopefully
> > in less than 2 QEMU releases), what's the point of deprecating
> > them?  
> 
> Well, it depends on the "classification" [1] of the parameter...
> 
> Let's ask: What's the main purpose of the option?
> 
> Is it easier to use than the "full" option, and thus likely to be used
> by a lot of people who run QEMU directly from the CLI? In that case it
> should stay as "convenience option" and not be deprecated.
> 
> Or is the option merely there to give the upper layers like libvirt or
> some few users and their scripts some more grace period to adapt their
> code, but we all agree that the options are rather ugly and should
> finally go away? Then it's rather a "legacy option" and the deprecation
> process is the right way to go. Our QEMU interface is still way 
> overcrowded, we should try to keep it as clean as possible.

After switching to memdev for main RAM, users could use relatively
short global options
 -global memory-backend.prealloc|share=on
and
 -global memory-backend-file.mem-path=X|prealloc|share=on

instead of us adding and maintaining slightly shorter
 -mem-shared/-mem-path/-mem-prealloc

>  Thomas
> 
> 
> [1] Using the terms from:
> https://www.youtube.com/watch?v=Oscjpkns7tM=8m

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH RFC V2] guests: Add support for openSUSE

2019-12-03 Thread Andrea Bolognani 
On Mon, 2019-12-02 at 20:33 +0100, Andrea Bolognani wrote:
> > On Tue, Nov 26, 2019 at 1:52 AM Jim Fehlig  wrote:
> > > +++ b/guests/host_vars/libvirt-opensuse-15/main.yml
> > > @@ -0,0 +1,22 @@
> > > +---
> > > +projects:
> > > +  - libosinfo
> > > +  - libvirt
> > > +  - libvirt-cim
> > > +  - libvirt-dbus
> > > +  - libvirt-glib
> > > +  - libvirt-perl
> > > +  - libvirt-python
> > > +  - libvirt-sandbox
> > > +  - libvirt-tck
> > > +  - osinfo-db
> > > +  - osinfo-db-tools
> > > +  - virt-manager
> > > +  - virt-viewer
> 
> Since right now we're only aiming at building libvirt, we should only
> list that project: it's a good starting point, and we can add more as
> we make sure we can actually 'lcitool build' them successfully.

I didn't mention this explicitly, but part of "focusing on libvirt
only as a first stab" means only adding the mappings necessary to
resolve the abstract package names listed in

  guests/vars/projects/base.yml
  guests/vars/projects/blacklist.yml
  guests/vars/projects/libvirt.yml

All other mappings should go in at the same time as additional
projects are enabled.

I think that we're basically good: once Fabiano has posted his
patches addressing CentOS 7.3 compatibility, you can respin on top
of them while addressing review comments, and unless we happen to
notice something that was missed until now we should be able to
finally merge openSUSE support :)

-- 
Andrea Bolognani / Red Hat / Virtualization

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v3 2/4] qemu: block: use the delete flag to delete snapshot images if requested

2019-12-03 Thread Peter Krempa 
On Thu, Nov 21, 2019 at 11:00:45 +0100, Pavel Mores wrote:
> When blockcommit finishes successfully, one of the
> qemuBlockJobProcessEventCompletedCommit() and
> qemuBlockJobProcessEventCompletedActiveCommit() event handlers is called.
> This is where the delete flag (stored in qemuBlockJobCommitData since the
> previous commit) can actually be used to delete the committed snapshot
> images if requested.
> 
> We use virFileRemove() instead of a simple unlink() to cover the case where
> the image to be removed is on an NFS volume.  To acquire the uid/gid arguments
> for the virFileRemove() call, we call qemuDomainGetImageIds() which was
> previously static in its file so we first have to export it.
> 
> Signed-off-by: Pavel Mores 
> ---
>  src/qemu/qemu_blockjob.c | 39 +++
>  src/qemu/qemu_domain.c   |  2 +-
>  src/qemu/qemu_domain.h   |  6 ++
>  3 files changed, 46 insertions(+), 1 deletion(-)
> 
> diff --git a/src/qemu/qemu_blockjob.c b/src/qemu/qemu_blockjob.c
> index 7d94a6ce38..1bf23dac3c 100644
> --- a/src/qemu/qemu_blockjob.c
> +++ b/src/qemu/qemu_blockjob.c
> @@ -965,6 +965,37 @@ qemuBlockJobProcessEventCompletedPull(virQEMUDriverPtr 
> driver,
>  }
>  
>  
> +/**
> + * Helper for qemuBlockJobProcessEventCompletedCommit() and
> + * qemuBlockJobProcessEventCompletedActiveCommit().  Relies on adjustments
> + * these functions perform on the 'backingStore' chain to function correctly.
> + *
> + * TODO look into removing backing store for non-local snapshots too
> + */
> +static void
> +qemuBlockJobUnlinkCommittedStorage(virQEMUDriverPtr driver,
> +   virDomainObjPtr vm,
> +   virDomainDiskDefPtr disk,
> +   virStorageSourcePtr top)
> +{
> +virStorageSourcePtr p = top;
> +virQEMUDriverConfigPtr cfg = virQEMUDriverGetConfig(driver);

The reference acquired here is leaked from this function. You can use
g_autoptr for it.


--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v3 4/4] qemu: block: enable the snapshot image deletion feature

2019-12-03 Thread Peter Krempa 
On Thu, Nov 21, 2019 at 11:00:47 +0100, Pavel Mores wrote:
> With all plumbing in place, we can now enable the new functionality.
> 
> Signed-off-by: Pavel Mores 
> ---
>  src/qemu/qemu_driver.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c
> index a3ff4b2177..75458f5c8a 100644
> --- a/src/qemu/qemu_driver.c
> +++ b/src/qemu/qemu_driver.c
> @@ -18496,10 +18496,10 @@ qemuDomainBlockCommit(virDomainPtr dom,
>  bool persistjob = false;
>  bool blockdev = false;
>  
> -/* XXX Add support for COMMIT_DELETE */
>  virCheckFlags(VIR_DOMAIN_BLOCK_COMMIT_SHALLOW |
>VIR_DOMAIN_BLOCK_COMMIT_ACTIVE |
>VIR_DOMAIN_BLOCK_COMMIT_RELATIVE |
> +  VIR_DOMAIN_BLOCK_COMMIT_DELETE |
>VIR_DOMAIN_BLOCK_COMMIT_BANDWIDTH_BYTES, -1);
>  
>  if (!(vm = qemuDomainObjFromDomain(dom)))
> @@ -18638,7 +18638,8 @@ qemuDomainBlockCommit(virDomainPtr dom,
>  goto endjob;

You must check that VIR_QEMU_CAPS_BLOCKDEV is enabled for this VM
otherwise the flag won't work. If blockdev isn't enabled you still must
produce an error message.

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v3 3/4] qemu: block: store the delete flag in libvirtd's status XML

2019-12-03 Thread Peter Krempa 
On Thu, Nov 21, 2019 at 11:00:46 +0100, Pavel Mores wrote:
> Since blockcommit is asynchronous, libvirtd can be restarted while the
> operation runs.  To ensure the information necessary to finish up the job
> is not lost, serialisation to and deserialisation from the status XML is
> added.
> 
> To unittest this, the new element was only added to the active commit test,
> the non-active commit test doesn't have the new element so as to test its
> absence.
> 
> Signed-off-by: Pavel Mores 
> ---
>  src/qemu/qemu_domain.c   | 4 
>  tests/qemustatusxml2xmldata/blockjob-blockdev-in.xml | 1 +
>  2 files changed, 5 insertions(+)

Reviewed-by: Peter Krempa 

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v3 2/4] qemu: block: use the delete flag to delete snapshot images if requested

2019-12-03 Thread Peter Krempa 
On Thu, Nov 21, 2019 at 11:00:45 +0100, Pavel Mores wrote:
> When blockcommit finishes successfully, one of the
> qemuBlockJobProcessEventCompletedCommit() and
> qemuBlockJobProcessEventCompletedActiveCommit() event handlers is called.
> This is where the delete flag (stored in qemuBlockJobCommitData since the
> previous commit) can actually be used to delete the committed snapshot
> images if requested.
> 
> We use virFileRemove() instead of a simple unlink() to cover the case where
> the image to be removed is on an NFS volume.  To acquire the uid/gid arguments
> for the virFileRemove() call, we call qemuDomainGetImageIds() which was
> previously static in its file so we first have to export it.
> 
> Signed-off-by: Pavel Mores 
> ---
>  src/qemu/qemu_blockjob.c | 39 +++
>  src/qemu/qemu_domain.c   |  2 +-
>  src/qemu/qemu_domain.h   |  6 ++
>  3 files changed, 46 insertions(+), 1 deletion(-)
> 
> diff --git a/src/qemu/qemu_blockjob.c b/src/qemu/qemu_blockjob.c
> index 7d94a6ce38..1bf23dac3c 100644
> --- a/src/qemu/qemu_blockjob.c
> +++ b/src/qemu/qemu_blockjob.c
> @@ -965,6 +965,37 @@ qemuBlockJobProcessEventCompletedPull(virQEMUDriverPtr 
> driver,
>  }
>  
>  
> +/**
> + * Helper for qemuBlockJobProcessEventCompletedCommit() and
> + * qemuBlockJobProcessEventCompletedActiveCommit().  Relies on adjustments
> + * these functions perform on the 'backingStore' chain to function correctly.

Preferably use the format of the comments we use elsewhere too.

> + *
> + * TODO look into removing backing store for non-local snapshots too
> + */
> +static void
> +qemuBlockJobUnlinkCommittedStorage(virQEMUDriverPtr driver,

You could name it more generic as this same operation may be added for
other block jobs in the future.

E.g. qemuBlockJobDeleteImages.

> +   virDomainObjPtr vm,
> +   virDomainDiskDefPtr disk,
> +   virStorageSourcePtr top)
> +{
> +virStorageSourcePtr p = top;
> +virQEMUDriverConfigPtr cfg = virQEMUDriverGetConfig(driver);
> +uid_t uid;
> +gid_t gid;
> +
> +for (; p != NULL; p = p->backingStore) {
> +if (virStorageSourceIsLocalStorage(p)) {

This also returns true for 'VIR_STORAGE_TYPE_FILE' but we certainly
don't want to unlink those.

> +
> +qemuDomainGetImageIds(cfg, vm, p, disk->src, , );
> +
> +if (virFileRemove(p->path, uid, gid) < 0) {
> +VIR_WARN("Unable to remove snapshot image file %s (%s)",

Add apostrophes around the first %s -> '%s'

> + p->path, g_strerror(errno));
> +}
> +}
> +}
> +}
> +
>  /**
>   * qemuBlockJobProcessEventCompletedCommit:
>   * @driver: qemu driver object
> @@ -1031,6 +1062,10 @@ 
> qemuBlockJobProcessEventCompletedCommit(virQEMUDriverPtr driver,
>  job->data.commit.topparent->backingStore = job->data.commit.base;
>  
>  qemuBlockJobEventProcessConcludedRemoveChain(driver, vm, asyncJob, 
> job->data.commit.top);
> +
> +if (job->data.commit.deleteCommittedImages)
> +qemuBlockJobUnlinkCommittedStorage(driver, vm, job->disk, 
> job->data.commit.top);
> +
>  virObjectUnref(job->data.commit.top);
>  job->data.commit.top = NULL;
>  
> @@ -1121,6 +1156,10 @@ 
> qemuBlockJobProcessEventCompletedActiveCommit(virQEMUDriverPtr driver,
>  job->disk->src->readonly = job->data.commit.top->readonly;
>  
>  qemuBlockJobEventProcessConcludedRemoveChain(driver, vm, asyncJob, 
> job->data.commit.top);
> +
> +if (job->data.commit.deleteCommittedImages)
> +qemuBlockJobUnlinkCommittedStorage(driver, vm, job->disk, 
> job->data.commit.top);
> +
>  virObjectUnref(job->data.commit.top);
>  job->data.commit.top = NULL;
>  /* the mirror element does not serve functional purpose for the commit 
> job */
> diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c
> index 262b74d1ab..07bf8f5a54 100644
> --- a/src/qemu/qemu_domain.c
> +++ b/src/qemu/qemu_domain.c
> @@ -10133,7 +10133,7 @@ qemuDomainCleanupRun(virQEMUDriverPtr driver,
>  priv->ncleanupCallbacks_max = 0;
>  }
>  
> -static void
> +void
>  qemuDomainGetImageIds(virQEMUDriverConfigPtr cfg,
>virDomainObjPtr vm,
>virStorageSourcePtr src,

Preferably export this in a separate commit.

> diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h
> index 98a9540275..f66610c7d3 100644
> --- a/src/qemu/qemu_domain.h
> +++ b/src/qemu/qemu_domain.h
> @@ -834,6 +834,12 @@ bool qemuDomainDiskChangeSupported(virDomainDiskDefPtr 
> disk,
>  const char *qemuDomainDiskNodeFormatLookup(virDomainObjPtr vm,
> const char *disk);
>  
> +void qemuDomainGetImageIds(virQEMUDriverConfigPtr cfg,
> +  virDomainObjPtr vm,
> +  virStorageSourcePtr src,
> +  virStorageSourcePtr parentSrc,
> +

Re: [libvirt] [PATCH v3 1/4] qemu: block: propagate the delete flag to where it can actually be used

2019-12-03 Thread Peter Krempa 
On Thu, Nov 21, 2019 at 11:00:44 +0100, Pavel Mores wrote:
> Propagate the delete flag from qemuDomainBlockCommit() (which was just
> ignoring it until now) to qemuBlockJobDiskNewCommit() where it can be
> stored in the qemuBlockJobCommitData structure which holds information
> necessary to finish the job asynchronously.
> 
> In the actual qemuBlockJobDiskNewCommit() in this commit, we temporarily
> pass a literal 'false' to preserve the current behaviour until the whole
> implementation of the feature is in place.
> 
> Signed-off-by: Pavel Mores 
> ---
>  src/qemu/qemu_blockjob.c | 4 +++-
>  src/qemu/qemu_blockjob.h | 4 +++-
>  src/qemu/qemu_driver.c   | 2 +-
>  3 files changed, 7 insertions(+), 3 deletions(-)

ACK

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [glib PATCH] glib,docs: Remove "object-tree" chapter

2019-12-03 Thread Michal Privoznik 

On 12/3/19 2:04 PM, Fabiano Fidêncio wrote:

gtk-doc-1.30+ does *not* create tree_index.sgml in case the library does
not provide GObjects, which is the case of LibvirtGLib. In previous
versions, though, an empty file would be generated.

This seems to be a regression which hit *a* *lot* of different projects,
all of them having to take the very same approach.

Signed-off-by: Fabiano Fidêncio 
---
  docs/libvirt-glib/Libvirt-glib-docs.xml | 4 
  1 file changed, 4 deletions(-)


Reviewed-by: Michal Privoznik 

Michal

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] Libvirt 5.9.0 Lags

2019-12-03 Thread Dincer Beken 
Hello Daniel,

Thanks for your reply.
Yes, the issue is all inside the guest, not between the host and guest or 
between guests.



Regards,
Dincer


Mit freundlichen Grüßen

Dincer Beken

E-Mail: dbe...@blackned.de

Tel.: +49 8331 9959-852


[cid:CD8F0C8F-87D2-4D00-BB49-89D99B0921F9]



blackned gmbh · www.blackned.de
zentrale: zugspitzstraße 1 · 87751 heimertingen

niederlassung münchen: zeppelinstraße 2 · 82205 gilching

niederlassung nürnberg: südwestpark 23 · 90449 nürnberg





geschäftsführer: timo haas · josef stadler · hrb memmingen 17319





Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.

Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten 
haben,

informieren Sie bitte sofort den Absender und löschen Sie diese E-Mail.

Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht 
gestattet.

This e-mail may contain confidential and/or privileged information.

If you are not the intended recipient (or have received this e-mail in error) 
please notify the

sender immediately and delete this e-mail. Any unauthorized copying, disclosure 
or

distribution of the contents in this e-mail is strictly forbidden.




Von: Daniel P. Berrangé 
Gesendet: Dienstag, 3. Dezember 2019 14:39
An: Dincer Beken 
Cc: libvir-list@redhat.com 
Betreff: Re: [libvirt] Libvirt 5.9.0 Lags

On Tue, Dec 03, 2019 at 11:28:37AM +, Dincer Beken wrote:
>
> Hello all,
>
> I am using Arch Linux 5.3.13-arch1-1 and libvirt 5.9.0.
>
> After I made a linux update (sudo pacman -Syu), I got the new, libvirt 
> version mentioned above.
>
> But soon after I realized, while programming in eclipse, that copy paste does 
> not work anymore. After couple of tries, when I try to copy some lines of 
> code, it freezes for a short time, and looses the copied text in the 
> clipboard.
>
> I converted the VM's into vdi and continuing to work with virtual box. There, 
> on the same laptop and the same VM, I don't have that issue!
>
> Is this a known issue? I really need a fix urgently, since I ve got other VMs 
> which are optimized for KVM.

Libvirt isn't involved in cut+paste inside the guest, it is just responsible
for configuring & launching QEMU. So I'd be surprised if a change in libvirt
version has had such an effect on the guest - more likely would be a change
in QEMU version having an impact.


Regards,
Daniel
--
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] Libvirt 5.9.0 Lags

2019-12-03 Thread Daniel P . Berrangé 
On Tue, Dec 03, 2019 at 11:28:37AM +, Dincer Beken wrote:
> 
> Hello all,
> 
> I am using Arch Linux 5.3.13-arch1-1 and libvirt 5.9.0.
> 
> After I made a linux update (sudo pacman -Syu), I got the new, libvirt 
> version mentioned above.
> 
> But soon after I realized, while programming in eclipse, that copy paste does 
> not work anymore. After couple of tries, when I try to copy some lines of 
> code, it freezes for a short time, and looses the copied text in the 
> clipboard.
> 
> I converted the VM's into vdi and continuing to work with virtual box. There, 
> on the same laptop and the same VM, I don't have that issue!
> 
> Is this a known issue? I really need a fix urgently, since I ve got other VMs 
> which are optimized for KVM.

Libvirt isn't involved in cut+paste inside the guest, it is just responsible
for configuring & launching QEMU. So I'd be surprised if a change in libvirt
version has had such an effect on the guest - more likely would be a change
in QEMU version having an impact.


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH RFC V2] guests: Add support for openSUSE

2019-12-03 Thread Andrea Bolognani 
On Tue, 2019-12-03 at 13:54 +0100, Fabiano Fidêncio wrote:
> > > > > 'lci update ... libvirt' fails in task "Configure hostname":
> > > > > 
> > > > > fatal: [libvirt-opensuse-tumbleweed]: FAILED! => {"changed": false, 
> > > > > "msg": "hostname module cannot be used on platform Linux 
> > > > > (Opensuse-tumbleweed)"}
> > > > 
> > > > It seems to be an Ansible bug: 
> > > > https://github.com/ansible/ansible/issues/42726
> 
> I've opened a PR for ansible with a fix for this issue:
> https://github.com/ansible/ansible/pull/65457
> 
> With this one fixed and backported to the host OS libvirt-jenkins-ci
> uses we will be able to also add tumbleweed.

Backporting the fix locally on the production host would be a bit
gross, but luckily we don't really need to worry about that since
we're not going to run openSUSE Tumbleweed there - just like we
don't have Debian sid or FreeBSD -CURRENT :)

Either way, great job!

-- 
Andrea Bolognani / Red Hat / Virtualization

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH 04/21] backup: Introduce virDomainBackup APIs

2019-12-03 Thread Peter Krempa 
On Wed, Nov 27, 2019 at 11:30:30 -0600, Eric Blake wrote:
> On 11/26/19 3:39 PM, Peter Krempa wrote:

[...]

> > + * For now, backup jobs are also mutually exclusive with any
> > + * other block job on the same device, although this restriction may
> > + * be lifted in a future release. Progress of the backup job can be
> > + * tracked via virDomainGetJobStats(). Completion of the job is also 
> > announced
> > + * asynchronously via VIR_DOMAIN_EVENT_ID_JOB_COMPLETED event.
> 
> Is that true for pull mode, or only for push mode?  With push mode, it's
> obvious when the event is needed - when qemu finishes pushing.  But in pull
> mode, the only time an event makes sense is when you finally abort the job
> and the NBD server goes away - but as that is always an explicit libvirt API
> call, does the event still make sense?

It's also emitted for the pull mode. The same way as we emit this event
for migration. A migration success is also reported by the migration API
return value, but the event is still emitted.

Given that the event also carries the statistics it might be of
interrest to the APP, or can be ignored.

> 
> > + *
> > + * There are two fundamental backup approaches. The first, called a
> > + * push model, instructs the hypervisor to copy the state of the guest
> > + * disk to the designated storage destination (which may be on the


[...]

> > + *
> > + * In some cases, a user can start a backup job without supplying all
> > + * details and rely on libvirt to fill in the rest (for example,
> > + * selecting the port used for an NBD export). This API can then be
> > + * used to learn what default values were chosen.
> > + *
> > + * Returns a NUL-terminated UTF-8 encoded XML instance or NULL in
> > + * case of error.  The caller must free() the returned value.
> > + */
> 
> Do we need any further tweaks to the virDomainAbortJob() API to mention that
> it is used to end a backup?

virDomainAbortJob documents that it aborts any current background job at
the soonest opportunity. The rest of the documentation mentions only
caveats for certain kinds of jobs. Since we don't have any for backup I
don't feel that it's necessary.

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [jenkins-ci PATCH] lcitool: Explain our use of the 'ks' kernel parameter

2019-12-03 Thread Erik Skultety 
On Tue, Dec 03, 2019 at 02:00:50PM +0100, Andrea Bolognani wrote:
> On Tue, 2019-12-03 at 12:30 +0100, Erik Skultety wrote:
> > On Tue, Dec 03, 2019 at 12:19:55PM +0100, Andrea Bolognani wrote:
> > >  extra_arg = "console=ttyS0 
> > > ks=file:/{}".format(install_config)
> >
> > On a side note - I'm pretty sure I've already mentioned it in the past, but 
> > we
> > should convert to the 'inst.ks' syntax instead of plain 'ks', anaconda
> > currently accepts both [1], but it documents that *all* installer options 
> > are
> > prefixed with 'inst.' and in future releases it may stop accepting the older
> > syntax.
>
> Is the new syntax supported by RHEL/CentOS 7? That would be the only
> obstacle to its adoption. If you have time to cook a patch and verify
> it doesn't cause regressions, I'll gladly take it :)

Yes, it is - well, truth to be told, it was introduced around 7.2, but we're
only considering the latest minor, so no, it shouldn't be an issue. Okay, I'll
have a look.

Erik

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [glib PATCH] glib,docs: Remove "object-tree" chapter

2019-12-03 Thread Fabiano Fidêncio 
gtk-doc-1.30+ does *not* create tree_index.sgml in case the library does
not provide GObjects, which is the case of LibvirtGLib. In previous
versions, though, an empty file would be generated.

This seems to be a regression which hit *a* *lot* of different projects,
all of them having to take the very same approach.

Signed-off-by: Fabiano Fidêncio 
---
 docs/libvirt-glib/Libvirt-glib-docs.xml | 4 
 1 file changed, 4 deletions(-)

diff --git a/docs/libvirt-glib/Libvirt-glib-docs.xml 
b/docs/libvirt-glib/Libvirt-glib-docs.xml
index f2f3572..505f138 100644
--- a/docs/libvirt-glib/Libvirt-glib-docs.xml
+++ b/docs/libvirt-glib/Libvirt-glib-docs.xml
@@ -16,10 +16,6 @@
 
 
   
-  
-Object Hierarchy
-
-  
   
 API Index
 
-- 
2.23.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [jenkins-ci PATCH] lcitool: Explain our use of the 'ks' kernel parameter

2019-12-03 Thread Andrea Bolognani 
On Tue, 2019-12-03 at 12:30 +0100, Erik Skultety wrote:
> On Tue, Dec 03, 2019 at 12:19:55PM +0100, Andrea Bolognani wrote:
> >  extra_arg = "console=ttyS0 ks=file:/{}".format(install_config)
> 
> On a side note - I'm pretty sure I've already mentioned it in the past, but we
> should convert to the 'inst.ks' syntax instead of plain 'ks', anaconda
> currently accepts both [1], but it documents that *all* installer options are
> prefixed with 'inst.' and in future releases it may stop accepting the older
> syntax.

Is the new syntax supported by RHEL/CentOS 7? That would be the only
obstacle to its adoption. If you have time to cook a patch and verify
it doesn't cause regressions, I'll gladly take it :)

> To the patch itself:
> Reviewed-by: Erik Skultety 

Thanks! Pushed now.

-- 
Andrea Bolognani / Red Hat / Virtualization

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH RFC V2] guests: Add support for openSUSE

2019-12-03 Thread Fabiano Fidêncio 
> > > > 'lci update ... libvirt' fails in task "Configure hostname":
> > > >
> > > > fatal: [libvirt-opensuse-tumbleweed]: FAILED! => {"changed": false, 
> > > > "msg": "hostname module cannot be used on platform Linux 
> > > > (Opensuse-tumbleweed)"}
> > >
> > > It seems to be an Ansible bug: 
> > > https://github.com/ansible/ansible/issues/42726
> >
> > Is it possible that this failed simply because the hostname command
> > is not available? You mention this below.
>
> No. When testing this patch I was already installing "hostname" in the box.
>

I've opened a PR for ansible with a fix for this issue:
https://github.com/ansible/ansible/pull/65457

With this one fixed and backported to the host OS libvirt-jenkins-ci
uses we will be able to also add tumbleweed.


--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH] maint: post-post-version-bump version bump to 6.0.0

2019-12-03 Thread Ján Tomko 
My commit 3bbe10205b79c8803e69c61897a0491d7a3f57c0 forgot
to update the news.

Signed-off-by: Ján Tomko 
---
Pushed under the trivial paper bag rule.

 docs/news.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/news.xml b/docs/news.xml
index e4533cc46e..43aa96f497 100644
--- a/docs/news.xml
+++ b/docs/news.xml
@@ -42,7 +42,7 @@
  -->
 
 
-  
+  
 
 
 
-- 
2.19.2

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH] spec: Fix printing test-suite.log when make check fails

2019-12-03 Thread Pavel Hrdina 
On Tue, Dec 03, 2019 at 11:53:41AM +0100, Jiri Denemark wrote:
> Commit v5.7.0-248-g03449e2504 removed "cd tests" without updating the
> patch to test-suite.log.
> 
> Signed-off-by: Jiri Denemark 
> ---
>  libvirt.spec.in | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Reviewed-by: Pavel Hrdina 


signature.asc
Description: PGP signature
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH] maint: post-version bump version bump to 6.0.0

2019-12-03 Thread Ján Tomko 
Signed-off-by: Ján Tomko 
---
Pushed as trivial.

 configure.ac | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/configure.ac b/configure.ac
index d0ef7de750..76d28d2f67 100644
--- a/configure.ac
+++ b/configure.ac
@@ -16,7 +16,7 @@ dnl You should have received a copy of the GNU Lesser General 
Public
 dnl License along with this library.  If not, see
 dnl .
 
-AC_INIT([libvirt], [5.11.0], [libvir-list@redhat.com], [], 
[https://libvirt.org])
+AC_INIT([libvirt], [6.0.0], [libvir-list@redhat.com], [], 
[https://libvirt.org])
 
 if test $srcdir = "."
 then
-- 
2.21.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [jenkins-ci PATCH] lcitool: Explain our use of the 'ks' kernel parameter

2019-12-03 Thread Erik Skultety 
On Tue, Dec 03, 2019 at 12:19:55PM +0100, Andrea Bolognani wrote:
> It's not immediately obvious why we're using it even when installing
> a distribution like Debian, so explain the motivation in a comment.
>
> Signed-off-by: Andrea Bolognani 
> ---
>  guests/lcitool | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/guests/lcitool b/guests/lcitool
> index a630971..0e27796 100755
> --- a/guests/lcitool
> +++ b/guests/lcitool
> @@ -540,7 +540,9 @@ class Application:
>  # preseed files must use a well-known name to be picked up by
>  # d-i; for kickstart files, we can use whatever name we please
>  # but we need to point anaconda in the right direction through
> -# a kernel argument
> +# the 'ks' kernel parameter. We can use 'ks' unconditionally
> +# for simplicity's sake, because distributions that don't use
> +# kickstart for unattended installation will simply ignore it
>  extra_arg = "console=ttyS0 ks=file:/{}".format(install_config)

On a side note - I'm pretty sure I've already mentioned it in the past, but we
should convert to the 'inst.ks' syntax instead of plain 'ks', anaconda
currently accepts both [1], but it documents that *all* installer options are
prefixed with 'inst.' and in future releases it may stop accepting the older
syntax.

To the patch itself:
Reviewed-by: Erik Skultety 

[1] https://anaconda-installer.readthedocs.io/en/latest/boot-options.html

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] Libvirt 5.9.0 Lags

2019-12-03 Thread Dincer Beken 

Hello all,

I am using Arch Linux 5.3.13-arch1-1 and libvirt 5.9.0.

After I made a linux update (sudo pacman -Syu), I got the new, libvirt version 
mentioned above.

But soon after I realized, while programming in eclipse, that copy paste does 
not work anymore. After couple of tries, when I try to copy some lines of code, 
it freezes for a short time, and looses the copied text in the clipboard.

I converted the VM's into vdi and continuing to work with virtual box. There, 
on the same laptop and the same VM, I don't have that issue!

Is this a known issue? I really need a fix urgently, since I ve got other VMs 
which are optimized for KVM.


I ve also post this issue to stackoverflow:

https://stackoverflow.com/questions/59155986/lags-in-libvirt-5-9-0

Many Thanks, Dincer


--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH] tests: fix name of 32-bit x86 QEMU binary

2019-12-03 Thread Erik Skultety 
On Tue, Dec 03, 2019 at 11:09:56AM +, Daniel P. Berrangé wrote:
> The 32-bit x86 binary is called qemu-system-i386, not
> qemu-system-i686. This mistake across many test XML files was
> not noticed because the mistake was also made in testutilsqemu.c
> when mocking the capabilities.
> 
> Signed-off-by: Daniel P. Berrangé 
Reviewed-by: Erik Skultety 

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH] cpu: add CLZERO CPUID support for AMD platforms (libvirt 4.5)

2019-12-03 Thread Ani Sinha 
Qemu commit e900135dcfb67 ("i386: Add CPUID bit for CLZERO and XSAVEERPTR")
adds support for CLZERO CPUID bit. This commit extends support for this
bit in libvirt.

This patch is based off of libvirt-4.5 tree.

Signed-off-by: Ani Sinha 

---
 src/cpu/cpu_map.xml | 5 +
 1 file changed, 5 insertions(+)

diff --git a/src/cpu/cpu_map.xml b/src/cpu/cpu_map.xml
index 088c820..c251cb7 100644
--- a/src/cpu/cpu_map.xml
+++ b/src/cpu/cpu_map.xml
@@ -431,6 +431,9 @@
 
   
 
+
+  
+
 
 
 
@@ -1685,6 +1688,7 @@
   
   
   
+  
   
   
   
@@ -1756,6 +1760,7 @@
   
   
   
+  
   
   
   
-- 
1.9.4


--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [jenkins-ci PATCH] lcitool: Explain our use of the 'ks' kernel parameter

2019-12-03 Thread Andrea Bolognani 
It's not immediately obvious why we're using it even when installing
a distribution like Debian, so explain the motivation in a comment.

Signed-off-by: Andrea Bolognani 
---
 guests/lcitool | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/guests/lcitool b/guests/lcitool
index a630971..0e27796 100755
--- a/guests/lcitool
+++ b/guests/lcitool
@@ -540,7 +540,9 @@ class Application:
 # preseed files must use a well-known name to be picked up by
 # d-i; for kickstart files, we can use whatever name we please
 # but we need to point anaconda in the right direction through
-# a kernel argument
+# the 'ks' kernel parameter. We can use 'ks' unconditionally
+# for simplicity's sake, because distributions that don't use
+# kickstart for unattended installation will simply ignore it
 extra_arg = "console=ttyS0 ks=file:/{}".format(install_config)
 
 virt_install = distutils.spawn.find_executable("virt-install")
-- 
2.23.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] CLZERO support for AMD

2019-12-03 Thread Ani Sinha 
Thanks. I have sent a patch based off the tip here:
https://www.redhat.com/archives/libvir-list/2019-December/msg00086.html

I also have a patch off libvirt-4.5 which I have tested on our HW. If needed, I 
can send that patch as well.

Ani

On Dec 2, 2019, 5:22 PM +0530, Daniel P. Berrangé , wrote:
On Wed, Nov 27, 2019 at 11:10:21AM +, Ani Sinha wrote:
Hi :

I see a commit in qemu that adds support for CLZERO but in CPUIDs. :

commit e900135dcfb676a4019dc206395e02db5d1de0d6
Author: Sebastian Andrzej Siewior 
Date: Wed Sep 25 23:49:48 2019 +0200

i386: Add CPUID bit for CLZERO and XSAVEERPTR

The CPUID bits CLZERO and XSAVEERPTR are availble on AMD's ZEN platform
and could be passed to the guest.

Signed-off-by: Sebastian Andrzej Siewior 
Signed-off-by: Paolo Bonzini 

Is there a corresponding change in libvirt adding this capability? Libvirt 
complains when I add it manually in the vm xml:

# virsh create newvm.xml
error: Failed to create domain from newvm.xml
error: unsupported configuration: unknown CPU feature: clzero

# grep clzero newvm.xml


Should we add it?

Yes, if it is supported by QEMU to expose to a guest, then it should be
added to libvirt too.

Presumably only needs add an addition to src/cpu_maps/x86_features.xml

Regards,
Daniel
--
|: 
https://urldefense.proofpoint.com/v2/url?u=https-3A__berrange.com=DwIFaQ=s883GpUCOChKOHiocYtGcg=IIUxIyRwG4RGy57y2nvMNYcDkqW-NHozZ2R38VYcg5U=PLIQbn3uvPc_Hvp-uU09tByBWn_UFE0wrSWE2MeXnLI=HIAImXOwBBjDjbozqHZOBn1uaSs-AU3LugXqoQrrLko=
 -o- 
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.flickr.com_photos_dberrange=DwIFaQ=s883GpUCOChKOHiocYtGcg=IIUxIyRwG4RGy57y2nvMNYcDkqW-NHozZ2R38VYcg5U=PLIQbn3uvPc_Hvp-uU09tByBWn_UFE0wrSWE2MeXnLI=6zWQAT7zkr1fFfhCvnSAN6tLwYa1vFb0m66XYH46Vgw=
 :|
|: 
https://urldefense.proofpoint.com/v2/url?u=https-3A__libvirt.org=DwIFaQ=s883GpUCOChKOHiocYtGcg=IIUxIyRwG4RGy57y2nvMNYcDkqW-NHozZ2R38VYcg5U=PLIQbn3uvPc_Hvp-uU09tByBWn_UFE0wrSWE2MeXnLI=mSj1Kh5NlO-NiBTedJkvXyreeEvNDLcEAkxVV8TxUmY=
 -o- 
https://urldefense.proofpoint.com/v2/url?u=https-3A__fstop138.berrange.com=DwIFaQ=s883GpUCOChKOHiocYtGcg=IIUxIyRwG4RGy57y2nvMNYcDkqW-NHozZ2R38VYcg5U=PLIQbn3uvPc_Hvp-uU09tByBWn_UFE0wrSWE2MeXnLI=mw-XirgI1BWzbKo6zZTTY_SMrUw0aD1j6CiHYAhTVJw=
 :|
|: 
https://urldefense.proofpoint.com/v2/url?u=https-3A__entangle-2Dphoto.org=DwIFaQ=s883GpUCOChKOHiocYtGcg=IIUxIyRwG4RGy57y2nvMNYcDkqW-NHozZ2R38VYcg5U=PLIQbn3uvPc_Hvp-uU09tByBWn_UFE0wrSWE2MeXnLI=gDp9hO9xGy4cPvntUrsFe_9iiycTQSeX7II4_RzQeA0=
 -o- 
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.instagram.com_dberrange=DwIFaQ=s883GpUCOChKOHiocYtGcg=IIUxIyRwG4RGy57y2nvMNYcDkqW-NHozZ2R38VYcg5U=PLIQbn3uvPc_Hvp-uU09tByBWn_UFE0wrSWE2MeXnLI=_vzLCn26_4Iv36BEhb8Et71nUZIB6-m6JTa97ffbqMM=
 :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH RFC V2] guests: Add support for openSUSE

2019-12-03 Thread Andrea Bolognani 
On Tue, 2019-12-03 at 11:50 +0100, Fabiano Fidêncio wrote:
> > It sounds like we can just provide install= unconditionally, same as
> > we currently do with ks=? We'd have to verify doing so doesn't cause
> > any unintended side-effects for existing guest types.
> 
> I'd rather have it done in a more explicit way, just for the sake of
> someone else trying to debug / understand the code later on.

A comment in the code will do just fine, in my opinion. And then,
when we can start relying on a new enough virt-install, we can drop
the workaround with minimal fuss.

Our unconditional use of ks= could also benefit from being explained
in a comment, by the way. I'll post a patch for that shortly.

-- 
Andrea Bolognani / Red Hat / Virtualization

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH] cpu: add CLZERO CPUID support for AMD platforms

2019-12-03 Thread Ani Sinha 
Qemu commit e900135dcfb67 ("i386: Add CPUID bit for CLZERO and XSAVEERPTR")
adds support for CLZERO CPUID bit.
This commit extends support for this CPUID bit into libvirt.

Signed-off-by: Ani Sinha 
---
 src/cpu_map/x86_EPYC-IBPB.xml | 1 +
 src/cpu_map/x86_EPYC.xml  | 1 +
 src/cpu_map/x86_features.xml  | 3 +++
 3 files changed, 5 insertions(+)

diff --git a/src/cpu_map/x86_EPYC-IBPB.xml b/src/cpu_map/x86_EPYC-IBPB.xml
index 283697e..a70fbd8 100644
--- a/src/cpu_map/x86_EPYC-IBPB.xml
+++ b/src/cpu_map/x86_EPYC-IBPB.xml
@@ -14,6 +14,7 @@
 
 
 
+
 
 
 
diff --git a/src/cpu_map/x86_EPYC.xml b/src/cpu_map/x86_EPYC.xml
index f060139..6c11d82 100644
--- a/src/cpu_map/x86_EPYC.xml
+++ b/src/cpu_map/x86_EPYC.xml
@@ -14,6 +14,7 @@
 
 
 
+
 
 
 
diff --git a/src/cpu_map/x86_features.xml b/src/cpu_map/x86_features.xml
index 2bed1e0..dd62755 100644
--- a/src/cpu_map/x86_features.xml
+++ b/src/cpu_map/x86_features.xml
@@ -473,6 +473,9 @@
   
 
   
+  
+
+  
   
 
   
-- 
1.9.4


--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH] spec: Fix printing test-suite.log when make check fails

2019-12-03 Thread Jiri Denemark 
Commit v5.7.0-248-g03449e2504 removed "cd tests" without updating the
patch to test-suite.log.

Signed-off-by: Jiri Denemark 
---
 libvirt.spec.in | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libvirt.spec.in b/libvirt.spec.in
index c7fac104d9..4c6161a26f 100644
--- a/libvirt.spec.in
+++ b/libvirt.spec.in
@@ -1307,7 +1307,7 @@ mv 
$RPM_BUILD_ROOT%{_datadir}/systemtap/tapset/libvirt_qemu_probes.stp \
 cd %{_vpath_builddir}
 if ! make %{?_smp_mflags} check VIR_TEST_DEBUG=1
 then
-  cat test-suite.log || true
+  cat tests/test-suite.log || true
   exit 1
 fi
 
-- 
2.24.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH RFC V2] guests: Add support for openSUSE

2019-12-03 Thread Fabiano Fidêncio 
Andrea,

> It sounds like we can just provide install= unconditionally, same as
> we currently do with ks=? We'd have to verify doing so doesn't cause
> any unintended side-effects for existing guest types.

I'd rather have it done in a more explicit way, just for the sake of
someone else trying to debug / understand the code later on.

>
> (Incidentally, didn't CentOS 8 also require passing the installation
> URL explicitly based on your tests, or something like that?

That's a different issue. :-)
CentOS 8 explicitly requires the URL to be set in the kickstart file.
That's another issue, to be solved another day ...

> We need
> to get around to adding that OS to CI too at some point O:-)

... most likely when we add CentOS 8 VMs. :-)

> >
> > An important question ... can't we have / use a CentOS 8 machine?
> > Would be lovely to not have to work those issues around on lcitool.
>
> I'd love that as well, but I'm afraid that's out of our hands since
> the CentOS CI machines are handled by the CentOS CI folks... We could
> try asking nicely and see what happens :) But in the short term
> having a workaround in lcitool is definitely the way to go.

I see.


--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH RFC V2] guests: Add support for openSUSE

2019-12-03 Thread Andrea Bolognani 
On Tue, 2019-12-03 at 11:05 +0100, Fabiano Fidêncio wrote:
> > Speaking of which: I've run into a fairly major problem with
> > the Leap 15.1 support enabled by this patch: while unattended
> > installation works as expected on my laptop (Fedora 31), when I try
> > the same steps on the CentOS CI production environment (CentOS 7.5)
> > it doesn't seem to find or recognize the autoinst.xml file: instead,
> > I get a "please make sure your installation medium is available"
> > message and the chance to input the information manually, which is
> > obviously not what we want :) I'll try to figure out what's going
> > wrong tomorrow.
> 
> So, the error you're facing is not because the autoinst.xml file is
> not recognised. It happens because the installation source is not
> passed.
> 
> virt-install 1.5.0, which is the version present on CentOS 7.5,
> doesn't seem to know the proper kernel URL argument to be passed to
> different distros. So far it worked because it uses "method=URL",
> which is what both CentOS 7 / Fedora (the ones we care about) use and
> Ubuntu / Debian are special with regard to this.
> 
> OpenSUSE, though, requires "install=URL" to be passed. As it doesn't
> happen, the installation asks for a specific installation source.

So that was the problem! Great detective work :)

> Okay, the way to go, IMHO, is patching lcitool to:
> - Have a kernel-url-argument as part of the distro definition;
> - Mount the extra-args based on that;
> 
> Mind that "method=" will always be appended but as long as we provide
> the "install=" to the kernel command line, we're fine.

It sounds like we can just provide install= unconditionally, same as
we currently do with ks=? We'd have to verify doing so doesn't cause
any unintended side-effects for existing guest types.

(Incidentally, didn't CentOS 8 also require passing the installation
URL explicitly based on your tests, or something like that? We need
to get around to adding that OS to CI too at some point O:-)

> I'll cook some patches in the afternoon and post them.
> 
> An important question ... can't we have / use a CentOS 8 machine?
> Would be lovely to not have to work those issues around on lcitool.

I'd love that as well, but I'm afraid that's out of our hands since
the CentOS CI machines are handled by the CentOS CI folks... We could
try asking nicely and see what happens :) But in the short term
having a workaround in lcitool is definitely the way to go.

-- 
Andrea Bolognani / Red Hat / Virtualization

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH] news: Fix XML validation

2019-12-03 Thread Andrea Bolognani 
On Tue, 2019-12-03 at 04:02 +, Jim Fehlig wrote:
> The news schema requires two digits for both month and day in the date
> attribute. s/2/02/ in the day value of date to fix the following
> 'make check' failure
> 
> 2165) Checking ../docs/news.xml against ../news.rng ...
> libvirt: XML Util error : XML document failed to validate against schema:
> Unable to validate doc against 
> /home/jfehlig/virt/upstream/libvirt/build/../docs/schemas/../news.rng
> Element release failed to validate attributes
> 
> Signed-off-by: Jim Fehlig 
> ---
> 
> Pushing under the build-breaker rule. One could argue the schema could
> be relaxed

Nah, it's fine as it is :)

Thanks for catching the issue and fixing it!

-- 
Andrea Bolognani / Red Hat / Virtualization

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH RFC V2] guests: Add support for openSUSE

2019-12-03 Thread Fabiano Fidêncio 
Andrea,

>
> Speaking of which: I've run into a fairly major problem with
> the Leap 15.1 support enabled by this patch: while unattended
> installation works as expected on my laptop (Fedora 31), when I try
> the same steps on the CentOS CI production environment (CentOS 7.5)
> it doesn't seem to find or recognize the autoinst.xml file: instead,
> I get a "please make sure your installation medium is available"
> message and the chance to input the information manually, which is
> obviously not what we want :) I'll try to figure out what's going
> wrong tomorrow.

So, the error you're facing is not because the autoinst.xml file is
not recognised. It happens because the installation source is not
passed.

virt-install 1.5.0, which is the version present on CentOS 7.5,
doesn't seem to know the proper kernel URL argument to be passed to
different distros. So far it worked because it uses "method=URL",
which is what both CentOS 7 / Fedora (the ones we care about) use and
Ubuntu / Debian are special with regard to this.

OpenSUSE, though, requires "install=URL" to be passed. As it doesn't
happen, the installation asks for a specific installation source.

Okay, the way to go, IMHO, is patching lcitool to:
- Have a kernel-url-argument as part of the distro definition;
- Mount the extra-args based on that;

Mind that "method=" will always be appended but as long as we provide
the "install=" to the kernel command line, we're fine.

I'll cook some patches in the afternoon and post them.

An important question ... can't we have / use a CentOS 8 machine?
Would be lovely to not have to work those issues around on lcitool.

Best Regards,
-- 
Fabiano Fidêncio

>
>
> [1] guests/playbooks/update/tasks/packages.yml
> --
> Andrea Bolognani / Red Hat / Virtualization
>


--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH 2/2] Add -mem-shared option

2019-12-03 Thread Thomas Huth 
On 02/12/2019 22.00, Eduardo Habkost wrote:
> On Mon, Dec 02, 2019 at 08:39:48AM +0100, Igor Mammedov wrote:
>> On Fri, 29 Nov 2019 18:46:12 +0100
>> Paolo Bonzini  wrote:
>>
>>> On 29/11/19 13:16, Igor Mammedov wrote:
 As for "-m", I'd make it just an alias that translates
  -m/mem-path/mem-prealloc  
>>>
>>> I think we should just deprecate -mem-path/-mem-prealloc in 5.0.  CCing
>>> Thomas as mister deprecation. :)
>>
>> I'll add that to my series
> 
> Considering that the plan is to eventually reimplement those
> options as syntactic sugar for memory backend options (hopefully
> in less than 2 QEMU releases), what's the point of deprecating
> them?

Well, it depends on the "classification" [1] of the parameter...

Let's ask: What's the main purpose of the option?

Is it easier to use than the "full" option, and thus likely to be used
by a lot of people who run QEMU directly from the CLI? In that case it
should stay as "convenience option" and not be deprecated.

Or is the option merely there to give the upper layers like libvirt or
some few users and their scripts some more grace period to adapt their
code, but we all agree that the options are rather ugly and should
finally go away? Then it's rather a "legacy option" and the deprecation
process is the right way to go. Our QEMU interface is still way to
overcrowded, we should try to keep it as clean as possible.

 Thomas


[1] Using the terms from:
https://www.youtube.com/watch?v=Oscjpkns7tM=8m

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH] tools: skip libvirt-guests fast if libvirtd is not active

2019-12-03 Thread Christian Ehrhardt 
The most common operation of libvirt-guests is to manage the local
libvirtd. But users might have disabled that and while we are
After=libvirtd for ordering we are not Requiring it..
OTOH adding that or any harder dependency might affect our ordering.

But if people have disabled libvirt they will do a full retry loop
until timeout. Lets check if the local service is active at all and skip
fast if it is not.

Fixes: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1854653

Reported-by: Doug Smythies 
Signed-off-by: Christian Ehrhardt 
---
 tools/libvirt-guests.sh.in | 8 
 1 file changed, 8 insertions(+)

diff --git a/tools/libvirt-guests.sh.in b/tools/libvirt-guests.sh.in
index 4bc6e866f0..5a9930ee2f 100644
--- a/tools/libvirt-guests.sh.in
+++ b/tools/libvirt-guests.sh.in
@@ -90,6 +90,14 @@ test_connect()
 {
 uri=$1
 
+if [ "x$uri" = xdefault ]; then
+# Default config is most common and for the local libvirtd
+# Check if it is active before wasting time in connect loop
+if ! systemctl -q is-active libvirtd; then
+return 1
+fi
+fi
+
 i=${CONNECT_RETRIES}
 while [ $i -gt 0 ]; do
 run_virsh "$uri" connect 2>/dev/null
-- 
2.24.0


--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH] maint: Post-release version bump to 5.11.0

2019-12-03 Thread Peter Krempa 
On Tue, Dec 03, 2019 at 04:24:59 +, Jim Fehlig wrote:
> Signed-off-by: Jim Fehlig 
> ---
> 
> Pushing under the trivial rule.
> 
>  configure.ac  | 2 +-
>  docs/news.xml | 8 
>  2 files changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/configure.ac b/configure.ac
> index ef521e370c..d0ef7de750 100644
> --- a/configure.ac
> +++ b/configure.ac
> @@ -16,7 +16,7 @@ dnl You should have received a copy of the GNU Lesser 
> General Public
>  dnl License along with this library.  If not, see
>  dnl .
>  
> -AC_INIT([libvirt], [5.10.0], [libvir-list@redhat.com], [], 
> [https://libvirt.org])
> +AC_INIT([libvirt], [5.11.0], [libvir-list@redhat.com], [], 
> [https://libvirt.org])

The next release is going to be 6.0.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH] news: Fix XML validation

2019-12-03 Thread Daniel Veillard 
  Dohhh, that's why :-) !
Thanks for chasing this Jim,

Daniel

On Tue, Dec 03, 2019 at 04:02:11AM +, Jim Fehlig wrote:
> The news schema requires two digits for both month and day in the date
> attribute. s/2/02/ in the day value of date to fix the following
> 'make check' failure
> 
> 2165) Checking ../docs/news.xml against ../news.rng ...
> libvirt: XML Util error : XML document failed to validate against schema:
> Unable to validate doc against 
> /home/jfehlig/virt/upstream/libvirt/build/../docs/schemas/../news.rng
> Element release failed to validate attributes
> 
> Signed-off-by: Jim Fehlig 
> ---
> 
> Pushing under the build-breaker rule. One could argue the schema could
> be relaxed, but let's fix the build first.
> 
>  docs/news.xml | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/docs/news.xml b/docs/news.xml
> index 55b227839e..46a3466238 100644
> --- a/docs/news.xml
> +++ b/docs/news.xml
> @@ -42,7 +42,7 @@
>   -->
>  
>  
> -  
> +  
>  
>
>  
> -- 
> 2.24.0
> 
> 
> --
> libvir-list mailing list
> libvir-list@redhat.com
> https://www.redhat.com/mailman/listinfo/libvir-list

-- 
Daniel Veillard  | Red Hat Developers Tools http://developer.redhat.com/
veill...@redhat.com  | libxml Gnome XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ | virtualization library  http://libvirt.org/

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list