[libvirt PATCH v7 22/35] tests: add tests for nbdkit invocation

[Jonathon Jongsma]

We were testing the arguments that were being passed to qemu when a disk
was being served by nbdkit, but the arguments used to start nbdkit
itself were not testable. This adds a test to ensure that we're invoking
nbdkit correctly for various disk source definitions.

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 build-aux/syntax-check.mk |   2 +-
 src/qemu/qemu_nbdkit.c|   4 +-
 src/qemu/qemu_nbdkitpriv.h|  31 ++
 tests/meson.build |   1 +
 .../disk-cdrom-network.args.disk0 |   6 +
 .../disk-cdrom-network.args.disk1 |   8 +
 .../disk-cdrom-network.args.disk1.pipe.778|   1 +
 .../disk-cdrom-network.args.disk2 |   8 +
 .../disk-cdrom-network.args.disk2.pipe.780|   1 +
 .../disk-network-http.args.disk0  |   6 +
 .../disk-network-http.args.disk1  |   5 +
 .../disk-network-http.args.disk2  |   6 +
 .../disk-network-http.args.disk2.pipe.778 |   1 +
 .../disk-network-http.args.disk3  |   7 +
 .../disk-network-http.args.disk3.pipe.780 |   1 +
 ...work-source-curl-nbdkit-backing.args.disk0 |   7 +
 ...ce-curl-nbdkit-backing.args.disk0.pipe.778 |   1 +
 .../disk-network-source-curl.args.disk0   |   7 +
 ...sk-network-source-curl.args.disk0.pipe.778 |   1 +
 .../disk-network-source-curl.args.disk1   |   7 +
 ...sk-network-source-curl.args.disk1.pipe.780 |   1 +
 .../disk-network-source-curl.args.disk2   |   7 +
 ...sk-network-source-curl.args.disk2.pipe.782 |   1 +
 .../disk-network-source-curl.args.disk3   |   6 +
 .../disk-network-source-curl.args.disk4   |   6 +
 .../disk-network-ssh.args.disk0   |   6 +
 tests/qemunbdkittest.c| 308 ++
 27 files changed, 444 insertions(+), 2 deletions(-)
 create mode 100644 src/qemu/qemu_nbdkitpriv.h
 create mode 100644 tests/qemunbdkitdata/disk-cdrom-network.args.disk0
 create mode 100644 tests/qemunbdkitdata/disk-cdrom-network.args.disk1
 create mode 100644 tests/qemunbdkitdata/disk-cdrom-network.args.disk1.pipe.778
 create mode 100644 tests/qemunbdkitdata/disk-cdrom-network.args.disk2
 create mode 100644 tests/qemunbdkitdata/disk-cdrom-network.args.disk2.pipe.780
 create mode 100644 tests/qemunbdkitdata/disk-network-http.args.disk0
 create mode 100644 tests/qemunbdkitdata/disk-network-http.args.disk1
 create mode 100644 tests/qemunbdkitdata/disk-network-http.args.disk2
 create mode 100644 tests/qemunbdkitdata/disk-network-http.args.disk2.pipe.778
 create mode 100644 tests/qemunbdkitdata/disk-network-http.args.disk3
 create mode 100644 tests/qemunbdkitdata/disk-network-http.args.disk3.pipe.780
 create mode 100644 
tests/qemunbdkitdata/disk-network-source-curl-nbdkit-backing.args.disk0
 create mode 100644 
tests/qemunbdkitdata/disk-network-source-curl-nbdkit-backing.args.disk0.pipe.778
 create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk0
 create mode 100644 
tests/qemunbdkitdata/disk-network-source-curl.args.disk0.pipe.778
 create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk1
 create mode 100644 
tests/qemunbdkitdata/disk-network-source-curl.args.disk1.pipe.780
 create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk2
 create mode 100644 
tests/qemunbdkitdata/disk-network-source-curl.args.disk2.pipe.782
 create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk3
 create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk4
 create mode 100644 tests/qemunbdkitdata/disk-network-ssh.args.disk0
 create mode 100644 tests/qemunbdkittest.c

diff --git a/build-aux/syntax-check.mk b/build-aux/syntax-check.mk
index 64c1e2773e..ec04402133 100644
--- a/build-aux/syntax-check.mk
+++ b/build-aux/syntax-check.mk
@@ -1370,7 +1370,7 @@ exclude_file_name_regexp--sc_prohibit_close = \
   
(\.p[yl]$$|\.spec\.in$$|^docs/|^(src/util/vir(file|event)\.c|src/libvirt-stream\.c|tests/(vir.+mock\.c|commandhelper\.c|qemusecuritymock\.c)|tools/nss/libvirt_nss_(leases|macs)\.c)|tools/virt-qemu-qmp-proxy$$)
 
 exclude_file_name_regexp--sc_prohibit_empty_lines_at_EOF = \
-  
(^tests/(nodedevmdevctl|viracpi|virhostcpu|virpcitest|virstoragetest)data/|docs/js/.*\.js|docs/fonts/.*\.woff|\.diff|tests/virconfdata/no-newline\.conf$$)
+  
(^tests/(nodedevmdevctl|viracpi|virhostcpu|virpcitest|virstoragetest|qemunbdkit)data/|docs/js/.*\.js|docs/fonts/.*\.woff|\.diff|tests/virconfdata/no-newline\.conf$$)
 
 exclude_file_name_regexp--sc_prohibit_fork_wrappers = \
   
(^(src/(util/(vircommand|virdaemon)|lxc/lxc_controller)|tests/testutils)\.c$$)
diff --git a/src/qemu/qemu_nbdkit.c b/src/qemu/qemu_nbdkit.c
index 299d8824f2..df638e99c0 100644
--- a/src/qemu/qemu_nbdkit.c
+++ b/src/qemu/qemu_nbdkit.c
@@ -31,6 +31,8 @@
 #include "qemu_domain.h"
 #include "qemu_extdevice.h"
 #include "qemu_nbdkit.h"
+#define LIBVIRT_QEMU_NBDKITPRIV_H_ALLOW
+#include 

[libvirt PATCH v7 04/35] util: Allow virFileCache data to be any GObject

[Jonathon Jongsma]

Since the libvirt documentation suggests to prefer GObject over
virObject, and since virObject is a GObject, change virFileCache to
allow GObjects as data.

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 src/util/virfilecache.c | 14 --
 src/util/virfilecache.h |  2 +-
 2 files changed, 9 insertions(+), 7 deletions(-)

diff --git a/src/util/virfilecache.c b/src/util/virfilecache.c
index c730de066e..6f698016a1 100644
--- a/src/util/virfilecache.c
+++ b/src/util/virfilecache.c
@@ -170,7 +170,7 @@ virFileCacheLoad(virFileCache *cache,
 *data = g_steal_pointer();
 
  cleanup:
-virObjectUnref(loadData);
+g_clear_pointer(, g_object_unref);
 return ret;
 }
 
@@ -207,7 +207,7 @@ virFileCacheNewData(virFileCache *cache,
 return NULL;
 
 if (virFileCacheSave(cache, name, data) < 0) {
-g_clear_pointer(, virObjectUnref);
+g_clear_object();
 }
 }
 
@@ -239,7 +239,7 @@ virFileCacheNew(const char *dir,
 if (!(cache = virObjectNew(virFileCacheClass)))
 return NULL;
 
-cache->table = virHashNew(virObjectUnref);
+cache->table = virHashNew(g_object_unref);
 
 cache->dir = g_strdup(dir);
 
@@ -270,7 +270,7 @@ virFileCacheValidate(virFileCache *cache,
 if (*data) {
 VIR_DEBUG("Caching data '%p' for '%s'", *data, name);
 if (virHashAddEntry(cache->table, name, *data) < 0) {
-g_clear_pointer(data, virObjectUnref);
+g_clear_pointer(data, g_object_unref);
 }
 }
 }
@@ -300,7 +300,8 @@ virFileCacheLookup(virFileCache *cache,
 data = virHashLookup(cache->table, name);
 virFileCacheValidate(cache, name, );
 
-virObjectRef(data);
+if (data)
+g_object_ref(data);
 virObjectUnlock(cache);
 
 return data;
@@ -331,7 +332,8 @@ virFileCacheLookupByFunc(virFileCache *cache,
 data = virHashSearch(cache->table, iter, iterData, );
 virFileCacheValidate(cache, name, );
 
-virObjectRef(data);
+if (data)
+g_object_ref(data);
 virObjectUnlock(cache);
 
 return data;
diff --git a/src/util/virfilecache.h b/src/util/virfilecache.h
index c3bc0f529c..944741c0a7 100644
--- a/src/util/virfilecache.h
+++ b/src/util/virfilecache.h
@@ -48,7 +48,7 @@ typedef bool
  * @priv: private data created together with cache
  *
  * Creates a new data based on the @name.  The returned data must be
- * an instance of virObject.
+ * an instance of GObject.
  *
  * Returns data object or NULL on error.
  */
-- 
2.41.0

[libvirt PATCH v7 27/35] schema: add password configuration for ssh disk

[Jonathon Jongsma]

Right now, ssh network disks are not usable. There is some basic support
in libvirt that is meant to support disk chains that have backing disks
located at ssh urls, but there is no real way for a user to configure a
ssh-based disk.  This commit allows users to configure an ssh disk with
password authentication. Implementation will follow.


  

  



Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 docs/formatdomain.rst | 27 ++-
 src/conf/schemas/domaincommon.rng | 23 ++-
 2 files changed, 36 insertions(+), 14 deletions(-)

diff --git a/docs/formatdomain.rst b/docs/formatdomain.rst
index 68f54ab3ed..39d4230ec0 100644
--- a/docs/formatdomain.rst
+++ b/docs/formatdomain.rst
@@ -2778,7 +2778,7 @@ paravirtualized driver is specified via the ``disk`` 
element.
``network``
   The ``protocol`` attribute specifies the protocol to access to the
   requested image. Possible values are "nbd", "iscsi", "rbd", "sheepdog",
-  "gluster", "vxhs", "nfs", "http", "https", "ftp", ftps", or "tftp".
+  "gluster", "vxhs", "nfs", "http", "https", "ftp", ftps", "tftp", or 
"ssh".
 
   For any ``protocol`` other than ``nbd`` an additional attribute ``name``
   is mandatory to specify which volume/image will be used.
@@ -2930,18 +2930,19 @@ paravirtualized driver is specified via the ``disk`` 
element.
``auth``
   :since:`Since libvirt 3.9.0` , the ``auth`` element is supported for a
   disk ``type`` "network" that is using a ``source`` element with the
-  ``protocol`` attributes "rbd" or "iscsi". If present, the ``auth`` 
element
-  provides the authentication credentials needed to access the source. It
-  includes a mandatory attribute ``username``, which identifies the 
username
-  to use during authentication, as well as a sub-element ``secret`` with
-  mandatory attribute ``type``, to tie back to a `libvirt secret
-  object `__ that holds the actual password or other
-  credentials (the domain XML intentionally does not expose the password,
-  only the reference to the object that does manage the password). Known
-  secret types are "ceph" for Ceph RBD network sources and "iscsi" for CHAP
-  authentication of iSCSI targets. Both will require either a ``uuid``
-  attribute with the UUID of the secret object or a ``usage`` attribute
-  matching the key that was specified in the secret object.
+  ``protocol`` attributes "rbd", "iscsi", or "ssh". If present, the
+  ``auth`` element provides the authentication credentials needed to access
+  the source. It includes a mandatory attribute ``username``, which
+  identifies the username to use during authentication, as well as a
+  sub-element ``secret`` with mandatory attribute ``type``, to tie back to
+  a `libvirt secret object `__ that holds the actual
+  password or other credentials (the domain XML intentionally does not
+  expose the password, only the reference to the object that does manage
+  the password). Known secret types are "ceph" for Ceph RBD network sources
+  and "iscsi" for CHAP authentication of iSCSI targets. Both will require
+  either a ``uuid`` attribute with the UUID of the secret object or a
+  ``usage`` attribute matching the key that was specified in the secret
+  object.
``encryption``
   :since:`Since libvirt 3.9.0` , the ``encryption`` can be a sub-element of
   the ``source`` element for encrypted storage sources. If present,
diff --git a/src/conf/schemas/domaincommon.rng 
b/src/conf/schemas/domaincommon.rng
index 4a475f5c36..cd838a475c 100644
--- a/src/conf/schemas/domaincommon.rng
+++ b/src/conf/schemas/domaincommon.rng
@@ -2172,6 +2172,27 @@
 
   
 
+  
+
+  
+
+  
+ssh
+  
+
+
+
+
+
+  
+
+
+
+  
+
+  
+
+  
   
 
   
@@ -2179,7 +2200,6 @@
   
 sheepdog
 tftp
-ssh
   
 
 
@@ -2289,6 +2309,7 @@
   
   
   
+  
   
   
   
-- 
2.41.0

[libvirt PATCH v7 32/35] qemu: implement keyfile auth for ssh disks with nbdkit

[Jonathon Jongsma]

For ssh disks that are served by nbdkit, we can support logging in with
an ssh key file. Pass the path to the configured key file and the
username to the nbdkit process.

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 src/conf/domain_conf.c| 30 +
 src/conf/storage_source_conf.c|  2 ++
 src/conf/storage_source_conf.h|  5 ++-
 src/qemu/qemu_nbdkit.c| 15 +++--
 .../disk-network-ssh-key.args.disk0   |  9 +
 .../disk-network-ssh.args.disk2   |  9 +
 tests/qemunbdkittest.c|  1 +
 .../qemuxml2argvdata/disk-network-ssh-key.xml | 33 +++
 8 files changed, 93 insertions(+), 11 deletions(-)
 create mode 100644 tests/qemunbdkitdata/disk-network-ssh-key.args.disk0
 create mode 100644 tests/qemunbdkitdata/disk-network-ssh.args.disk2
 create mode 100644 tests/qemuxml2argvdata/disk-network-ssh-key.xml

diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c
index 842b6404b5..929e115bce 100644
--- a/src/conf/domain_conf.c
+++ b/src/conf/domain_conf.c
@@ -7268,10 +7268,18 @@ virDomainDiskSourceNetworkParse(xmlNodePtr node,
 return -1;
 }
 }
-if (src->protocol == VIR_STORAGE_NET_PROTOCOL_SSH &&
-(tmpnode = virXPathNode("./knownHosts", ctxt))) {
-if (!(src->ssh_known_hosts_file = virXMLPropStringRequired(tmpnode, 
"path")))
-return -1;
+if (src->protocol == VIR_STORAGE_NET_PROTOCOL_SSH) {
+if ((tmpnode = virXPathNode("./knownHosts", ctxt))) {
+if (!(src->ssh_known_hosts_file = 
virXMLPropStringRequired(tmpnode, "path")))
+return -1;
+}
+if ((tmpnode = virXPathNode("./identity", ctxt))) {
+if (!(src->ssh_user = virXMLPropStringRequired(tmpnode, 
"username")))
+return -1;
+
+if (!(src->ssh_keyfile = virXMLPropStringRequired(tmpnode, 
"keyfile")))
+return -1;
+}
 }
 
 return 0;
@@ -22280,8 +22288,18 @@ virDomainDiskSourceFormatNetwork(virBuffer *attrBuf,
 if (src->timeout)
 virBufferAsprintf(childBuf, "\n", 
src->timeout);
 
-if (src->protocol == VIR_STORAGE_NET_PROTOCOL_SSH && 
src->ssh_known_hosts_file)
-virBufferEscapeString(childBuf, "\n", 
src->ssh_known_hosts_file);
+if (src->protocol == VIR_STORAGE_NET_PROTOCOL_SSH) {
+if (src->ssh_known_hosts_file)
+virBufferEscapeString(childBuf, "\n", 
src->ssh_known_hosts_file);
+if (src->ssh_keyfile) {
+virBufferAddLit(childBuf, "ssh_user);
+virBufferEscapeString(childBuf, " keyfile='%s'", src->ssh_keyfile);
+
+virBufferAddLit(childBuf, "/>\n");
+}
+}
 }
 
 
diff --git a/src/conf/storage_source_conf.c b/src/conf/storage_source_conf.c
index 906bc36a9b..ce9c1f66c2 100644
--- a/src/conf/storage_source_conf.c
+++ b/src/conf/storage_source_conf.c
@@ -896,6 +896,7 @@ virStorageSourceCopy(const virStorageSource *src,
 def->ssh_host_key_check_disabled = src->ssh_host_key_check_disabled;
 def->ssh_user = g_strdup(src->ssh_user);
 def->ssh_known_hosts_file = g_strdup(src->ssh_known_hosts_file);
+def->ssh_keyfile = g_strdup(src->ssh_keyfile);
 
 def->nfs_user = g_strdup(src->nfs_user);
 def->nfs_group = g_strdup(src->nfs_group);
@@ -1172,6 +1173,7 @@ virStorageSourceClear(virStorageSource *def)
 
 VIR_FREE(def->ssh_user);
 VIR_FREE(def->ssh_known_hosts_file);
+VIR_FREE(def->ssh_keyfile);
 
 VIR_FREE(def->nfs_user);
 VIR_FREE(def->nfs_group);
diff --git a/src/conf/storage_source_conf.h b/src/conf/storage_source_conf.h
index 8a9c7d07e2..8c805664af 100644
--- a/src/conf/storage_source_conf.h
+++ b/src/conf/storage_source_conf.h
@@ -406,12 +406,11 @@ struct _virStorageSource {
 
 bool hostcdrom; /* backing device is a cdrom */
 
-/* passthrough variables for the ssh driver which we don't handle properly 
*/
-/* these must not be used apart from formatting the output JSON in the 
qemu driver */
+/* ssh variables */
 char *ssh_user;
 bool ssh_host_key_check_disabled;
-/* additional ssh variables */
 char *ssh_known_hosts_file;
+char *ssh_keyfile;
 
 /* nfs_user and nfs_group store the strings passed in by the user for NFS 
params.
  * nfs_uid and nfs_gid represent the converted/looked up ID numbers which 
are used
diff --git a/src/qemu/qemu_nbdkit.c b/src/qemu/qemu_nbdkit.c
index d0eca1ab89..0393850ddc 100644
--- a/src/qemu/qemu_nbdkit.c
+++ b/src/qemu/qemu_nbdkit.c
@@ -1035,8 +1035,12 @@ qemuNbdkitProcessBuildCommandSSH(qemuNbdkitProcess *proc,
 if (proc->source->auth) {
 if (qemuNbdkitProcessBuildCommandAuth(proc->source->auth, cmd) < 0)
 return -1;
-} else if (proc->source->ssh_user) {
-virCommandAddArgPair(cmd, "user", proc->source->ssh_user);
+} else {
+if (proc->source->ssh_keyfile)
+

[libvirt PATCH v7 17/35] qemu: include nbdkit state in private xml

[Jonathon Jongsma]

Add xml to the private data for a disk source to represent the nbdkit
process so that the state can be re-created if the libvirt daemon is
restarted. Format:

   
 /path/to/nbdkit.pid
 /path/to/nbdkit.socket
   

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 src/qemu/qemu_domain.c| 52 +
 src/qemu/qemu_nbdkit.c| 71 +++
 src/qemu/qemu_nbdkit.h|  8 +++
 src/qemu/qemu_process.c   |  6 ++
 tests/qemustatusxml2xmldata/modern-in.xml |  4 ++
 5 files changed, 141 insertions(+)

diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c
index 951f3127d9..8429ce1028 100644
--- a/src/qemu/qemu_domain.c
+++ b/src/qemu/qemu_domain.c
@@ -1958,6 +1958,33 @@ 
qemuStorageSourcePrivateDataAssignSecinfo(qemuDomainSecretInfo **secinfo,
 }
 
 
+static int
+qemuStorageSourcePrivateDataParseNbdkit(xmlNodePtr node,
+xmlXPathContextPtr ctxt,
+virStorageSource *src)
+{
+g_autofree char *pidfile = NULL;
+g_autofree char *socketfile = NULL;
+VIR_XPATH_NODE_AUTORESTORE(ctxt);
+
+ctxt->node = node;
+
+if (!(pidfile = virXPathString("string(./pidfile)", ctxt))) {
+virReportError(VIR_ERR_INTERNAL_ERROR, "%s", _("missing nbdkit 
pidfile"));
+return -1;
+}
+
+if (!(socketfile = virXPathString("string(./socketfile)", ctxt))) {
+virReportError(VIR_ERR_INTERNAL_ERROR, "%s", _("missing nbdkit 
socketfile"));
+return -1;
+}
+
+qemuNbdkitReconnectStorageSource(src, pidfile, socketfile);
+
+return 0;
+}
+
+
 static int
 qemuStorageSourcePrivateDataParse(xmlXPathContextPtr ctxt,
   virStorageSource *src)
@@ -1971,6 +1998,7 @@ qemuStorageSourcePrivateDataParse(xmlXPathContextPtr ctxt,
 bool fdsetPresent = false;
 unsigned int fdSetID;
 int enccount;
+xmlNodePtr nbdkitnode = NULL;
 
 src->nodestorage = 
virXPathString("string(./nodenames/nodename[@type='storage']/@name)", ctxt);
 src->nodeformat = 
virXPathString("string(./nodenames/nodename[@type='format']/@name)", ctxt);
@@ -2036,6 +2064,10 @@ qemuStorageSourcePrivateDataParse(xmlXPathContextPtr 
ctxt,
 virTristateBoolTypeFromString(thresholdEventWithIndex) == 
VIR_TRISTATE_BOOL_YES)
 src->thresholdEventWithIndex = true;
 
+if ((nbdkitnode = virXPathNode("nbdkit", ctxt))) {
+if (qemuStorageSourcePrivateDataParseNbdkit(nbdkitnode, ctxt, src) < 0)
+return -1;
+}
 return 0;
 }
 
@@ -2053,6 +2085,23 @@ qemuStorageSourcePrivateDataFormatSecinfo(virBuffer *buf,
 }
 
 
+static void
+qemuStorageSourcePrivateDataFormatNbdkit(qemuNbdkitProcess *nbdkit,
+ virBuffer *buf)
+{
+g_auto(virBuffer) childBuf = VIR_BUFFER_INIT_CHILD(buf);
+
+if (!nbdkit)
+return;
+
+virBufferEscapeString(, "%s\n",
+  nbdkit->pidfile);
+virBufferEscapeString(, "%s\n",
+  nbdkit->socketfile);
+virXMLFormatElement(buf, "nbdkit", NULL, );
+}
+
+
 static int
 qemuStorageSourcePrivateDataFormat(virStorageSource *src,
virBuffer *buf)
@@ -2102,6 +2151,9 @@ qemuStorageSourcePrivateDataFormat(virStorageSource *src,
 if (src->thresholdEventWithIndex)
 virBufferAddLit(buf, "\n");
 
+if (srcPriv)
+qemuStorageSourcePrivateDataFormatNbdkit(srcPriv->nbdkitProcess, buf);
+
 return 0;
 }
 
diff --git a/src/qemu/qemu_nbdkit.c b/src/qemu/qemu_nbdkit.c
index 81861bae4a..e3923ab4f2 100644
--- a/src/qemu/qemu_nbdkit.c
+++ b/src/qemu/qemu_nbdkit.c
@@ -627,6 +627,77 @@ qemuNbdkitProcessNew(virStorageSource *source,
 return nbdkit;
 }
 
+/**
+ * qemuNbdkitReconnectStorageSource:
+ * @source: a storage source
+ * @pidfile: a pidfile for an nbdkit process
+ * @socketfile: the socket file associated with the nbdkit process
+ *
+ * This function constructs a new qemuNbdkitProcess object with the given 
values for @pidfile and
+ * @socketfile and stores it in @source. This is intended to be called when 
the libvirt daemon is
+ * restarted and tries to reconnect to all currently-running domains. Since 
this function is called
+ * from the code that parses the current daemon state, it should not perform 
any filesystem
+ * operations, or anything else that might fail. Additional initialization 
will be done later by
+ * calling qemuNbdkitStorageSourceManageProcess().
+ */
+void
+qemuNbdkitReconnectStorageSource(virStorageSource *source,
+ const char *pidfile,
+ const char *socketfile)
+{
+qemuDomainStorageSourcePrivate *srcpriv = 
qemuDomainStorageSourcePrivateFetch(source);
+
+if (srcpriv->nbdkitProcess) {
+VIR_WARN("source already has an nbdkit process");
+return;
+}
+
+srcpriv->nbdkitProcess = 

[libvirt PATCH v7 05/35] qemu: implement basic virFileCache for nbdkit caps

[Jonathon Jongsma]

Preparatory step for caching nbdkit capabilities. This patch implements
the newData and isValid virFileCacheHandlers callback functions.

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 src/qemu/qemu_nbdkit.c | 89 +-
 src/qemu/qemu_nbdkit.h |  4 ++
 2 files changed, 92 insertions(+), 1 deletion(-)

diff --git a/src/qemu/qemu_nbdkit.c b/src/qemu/qemu_nbdkit.c
index 58828dd89a..9828f562f7 100644
--- a/src/qemu/qemu_nbdkit.c
+++ b/src/qemu/qemu_nbdkit.c
@@ -201,7 +201,7 @@ qemuNbdkitGetDirMtime(const char *moddir)
 }
 
 
-G_GNUC_UNUSED static void
+static void
 qemuNbdkitCapsQuery(qemuNbdkitCaps *caps)
 {
 struct stat st;
@@ -240,3 +240,90 @@ qemuNbdkitCapsSet(qemuNbdkitCaps *nbdkitCaps,
 {
 ignore_value(virBitmapSetBit(nbdkitCaps->flags, flag));
 }
+
+
+static bool
+virNbkditCapsCheckModdir(const char *moddir,
+ time_t expectedMtime)
+{
+time_t mtime = qemuNbdkitGetDirMtime(moddir);
+
+if (mtime != expectedMtime) {
+VIR_DEBUG("Outdated capabilities for nbdkit: module directory '%s' 
changed (%lld vs %lld)",
+  moddir, (long long)mtime, (long long)expectedMtime);
+return false;
+}
+return true;
+}
+
+
+static bool
+virNbdkitCapsIsValid(void *data,
+ void *privData G_GNUC_UNUSED)
+{
+qemuNbdkitCaps *nbdkitCaps = data;
+struct stat st;
+
+if (!nbdkitCaps->path)
+return true;
+
+if (!virNbkditCapsCheckModdir(NBDKIT_PLUGINDIR, 
nbdkitCaps->pluginDirMtime))
+return false;
+if (!virNbkditCapsCheckModdir(NBDKIT_FILTERDIR, 
nbdkitCaps->filterDirMtime))
+return false;
+
+if (nbdkitCaps->libvirtCtime != virGetSelfLastChanged() ||
+nbdkitCaps->libvirtVersion != LIBVIR_VERSION_NUMBER) {
+VIR_DEBUG("Outdated capabilities for '%s': libvirt changed (%lld vs 
%lld, %lu vs %lu)",
+  nbdkitCaps->path,
+  (long long)nbdkitCaps->libvirtCtime,
+  (long long)virGetSelfLastChanged(),
+  (unsigned long)nbdkitCaps->libvirtVersion,
+  (unsigned long)LIBVIR_VERSION_NUMBER);
+return false;
+}
+
+if (stat(nbdkitCaps->path, ) < 0) {
+VIR_DEBUG("Failed to stat nbdkit binary '%s': %s",
+  nbdkitCaps->path,
+  g_strerror(errno));
+return false;
+}
+
+if (st.st_ctime != nbdkitCaps->ctime) {
+VIR_DEBUG("Outdated capabilities for '%s': nbdkit binary changed (%lld 
vs %lld)",
+  nbdkitCaps->path,
+  (long long)st.st_ctime, (long long)nbdkitCaps->ctime);
+return false;
+}
+
+return true;
+}
+
+
+static void*
+virNbdkitCapsNewData(const char *binary,
+ void *privData G_GNUC_UNUSED)
+{
+qemuNbdkitCaps *caps = qemuNbdkitCapsNew(binary);
+qemuNbdkitCapsQuery(caps);
+
+return caps;
+}
+
+
+virFileCacheHandlers nbdkitCapsCacheHandlers = {
+.isValid = virNbdkitCapsIsValid,
+.newData = virNbdkitCapsNewData,
+.loadFile = NULL,
+.saveFile = NULL,
+.privFree = NULL,
+};
+
+
+virFileCache*
+qemuNbdkitCapsCacheNew(const char *cachedir)
+{
+g_autofree char *dir = g_build_filename(cachedir, "nbdkitcapabilities", 
NULL);
+return virFileCacheNew(dir, "xml", );
+}
diff --git a/src/qemu/qemu_nbdkit.h b/src/qemu/qemu_nbdkit.h
index e191e1fdb4..4aba7c8455 100644
--- a/src/qemu/qemu_nbdkit.h
+++ b/src/qemu/qemu_nbdkit.h
@@ -21,6 +21,7 @@
 
 #include "internal.h"
 #include "virenum.h"
+#include "virfilecache.h"
 
 typedef struct _qemuNbdkitCaps qemuNbdkitCaps;
 
@@ -38,6 +39,9 @@ VIR_ENUM_DECL(qemuNbdkitCaps);
 qemuNbdkitCaps *
 qemuNbdkitCapsNew(const char *path);
 
+virFileCache *
+qemuNbdkitCapsCacheNew(const char *cachedir);
+
 bool
 qemuNbdkitCapsGet(qemuNbdkitCaps *nbdkitCaps,
   qemuNbdkitCapsFlags flag);
-- 
2.41.0

[libvirt PATCH v7 14/35] qemu: log error output from nbdkit

[Jonathon Jongsma]

log stderr and stdout from nbdkit into its own log so that
nbdkit-related issues can be debugged more easily.

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 src/qemu/qemu_nbdkit.c | 16 +++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/src/qemu/qemu_nbdkit.c b/src/qemu/qemu_nbdkit.c
index 6bf962d0f1..2d70e72c42 100644
--- a/src/qemu/qemu_nbdkit.c
+++ b/src/qemu/qemu_nbdkit.c
@@ -852,12 +852,23 @@ qemuNbdkitProcessStart(qemuNbdkitProcess *proc,
 virTimeBackOffVar timebackoff;
 g_autoptr(virURI) uri = NULL;
 g_autofree char *uristring = NULL;
+g_autofree char *basename = g_strdup_printf("%s-nbdkit-%i", vm->def->name, 
proc->source->id);
+int logfd = -1;
+g_autoptr(qemuLogContext) logContext = NULL;
 
 if (!(cmd = qemuNbdkitProcessBuildCommand(proc)))
 return -1;
 
+if (!(logContext = qemuLogContextNew(driver, vm, basename))) {
+virLastErrorPrefixMessage("%s", _("can't connect to virtlogd"));
+return -1;
+}
+
+logfd = qemuLogContextGetWriteFD(logContext);
+
 VIR_DEBUG("starting nbdkit process for %s", proc->source->nodestorage);
-virCommandSetErrorBuffer(cmd, );
+virCommandSetErrorFD(cmd, );
+virCommandSetOutputFD(cmd, );
 virCommandSetPidFile(cmd, proc->pidfile);
 
 if (qemuExtDeviceLogCommand(driver, vm, cmd, "nbdkit") < 0)
@@ -899,6 +910,9 @@ qemuNbdkitProcessStart(qemuNbdkitProcess *proc,
 if ((uri = qemuBlockStorageSourceGetURI(proc->source)))
 uristring = virURIFormat(uri);
 
+if (qemuLogContextReadFiltered(logContext, , 1024) < 0)
+VIR_WARN("Unable to read from nbdkit log");
+
 virReportError(VIR_ERR_OPERATION_FAILED,
_("Failed to connect to nbdkit for '%1$s': %2$s"),
NULLSTR(uristring), NULLSTR(errbuf));
-- 
2.41.0

[libvirt PATCH v7 33/35] schema: add ssh-agent configuration for ssh disks

[Jonathon Jongsma]

Add the ability to specify a path to a ssh-agent socket in order to use
the ssh-agent to authenticate to remote ssh disks. Example
configuration:




...

...


Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 docs/formatdomain.rst | 13 -
 src/conf/schemas/domaincommon.rng | 11 ---
 2 files changed, 16 insertions(+), 8 deletions(-)

diff --git a/docs/formatdomain.rst b/docs/formatdomain.rst
index 2ebd7bb1fc..720433617d 100644
--- a/docs/formatdomain.rst
+++ b/docs/formatdomain.rst
@@ -3007,11 +3007,14 @@ paravirtualized driver is specified via the ``disk`` 
element.
   are intended to be default, then the entire element may be omitted.
 
   When using an ``ssh`` protocol, this element is used to enable
-  authentication via ssh keys. In this configuration, the element has two
-  attributes. The ``username`` attribute specifies the name of the user on
-  the remote server and the ``keyfile`` attribute specifies the path to the
-  keyfile. Note that this only works for ssh keys that are not
-  password-protected.
+  authentication via ssh keys. In this configuration, the element has three
+  possible attributes. The ``username`` attribute is required and specifies
+  the name of the user on the remote server. ssh keys can be specified in
+  one of two ways. The first way is by adding them to an ssh-agent and
+  providing the path to the ssh-agent socket in the ``agentsock``
+  attribute. This method works for ssh keys with or without password
+  protection. Alternatively, for ssh keys without a password, the ssh key
+  can be specified directly by setting the ``keyfile`` attribute.
``reconnect``
   For disk type ``vhostuser`` configures reconnect timeout if the 
connection
   is lost. This is set with the two mandatory attributes ``enabled`` and
diff --git a/src/conf/schemas/domaincommon.rng 
b/src/conf/schemas/domaincommon.rng
index 47c5ee2a31..d8dd1b8c69 100644
--- a/src/conf/schemas/domaincommon.rng
+++ b/src/conf/schemas/domaincommon.rng
@@ -2186,9 +2186,14 @@
 
   
 
-
-  
-
+
+  
+
+  
+  
+
+  
+
   
 
   
-- 
2.41.0

[libvirt PATCH v7 23/35] qemu: add test for authenticating a https network disk

[Jonathon Jongsma]

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 tests/qemunbdkitdata/disk-network-source-curl.args.disk1  | 4 +++-
 .../disk-network-source-curl.args.disk1.pipe.780  | 2 +-
 .../disk-network-source-curl.args.disk1.pipe.782  | 1 +
 tests/qemunbdkitdata/disk-network-source-curl.args.disk2  | 2 +-
 .../disk-network-source-curl.args.disk2.pipe.784  | 1 +
 .../disk-network-source-curl.x86_64-latest.args   | 3 ++-
 tests/qemuxml2argvdata/disk-network-source-curl.xml   | 3 +++
 7 files changed, 12 insertions(+), 4 deletions(-)
 create mode 100644 
tests/qemunbdkitdata/disk-network-source-curl.args.disk1.pipe.782
 create mode 100644 
tests/qemunbdkitdata/disk-network-source-curl.args.disk2.pipe.784

diff --git a/tests/qemunbdkitdata/disk-network-source-curl.args.disk1 
b/tests/qemunbdkitdata/disk-network-source-curl.args.disk1
index fde6a4f533..d1288dd242 100644
--- a/tests/qemunbdkitdata/disk-network-source-curl.args.disk1
+++ b/tests/qemunbdkitdata/disk-network-source-curl.args.disk1
@@ -3,5 +3,7 @@ nbdkit \
 --foreground curl \
 protocols=https \
 'url=https://https.example.org:8443/path/to/disk5.iso?foo=bar' \
-cookie=-779 \
+user=myname \
+password=-779 \
+cookie=-781 \
 sslverify=false
diff --git a/tests/qemunbdkitdata/disk-network-source-curl.args.disk1.pipe.780 
b/tests/qemunbdkitdata/disk-network-source-curl.args.disk1.pipe.780
index 20af4ae383..ccdd4033fc 100644
--- a/tests/qemunbdkitdata/disk-network-source-curl.args.disk1.pipe.780
+++ b/tests/qemunbdkitdata/disk-network-source-curl.args.disk1.pipe.780
@@ -1 +1 @@
-cookie1=cookievalue1; cookie2=cookievalue2
\ No newline at end of file
+iscsi-mycluster_myname-secret
\ No newline at end of file
diff --git a/tests/qemunbdkitdata/disk-network-source-curl.args.disk1.pipe.782 
b/tests/qemunbdkitdata/disk-network-source-curl.args.disk1.pipe.782
new file mode 100644
index 00..20af4ae383
--- /dev/null
+++ b/tests/qemunbdkitdata/disk-network-source-curl.args.disk1.pipe.782
@@ -0,0 +1 @@
+cookie1=cookievalue1; cookie2=cookievalue2
\ No newline at end of file
diff --git a/tests/qemunbdkitdata/disk-network-source-curl.args.disk2 
b/tests/qemunbdkitdata/disk-network-source-curl.args.disk2
index 88c9fa35a1..f1d0e1929e 100644
--- a/tests/qemunbdkitdata/disk-network-source-curl.args.disk2
+++ b/tests/qemunbdkitdata/disk-network-source-curl.args.disk2
@@ -4,4 +4,4 @@ nbdkit \
 --readonly curl \
 protocols=http,https \
 url=http://http.example.org:8080/path/to/disk2.iso \
-cookie=-781
+cookie=-783
diff --git a/tests/qemunbdkitdata/disk-network-source-curl.args.disk2.pipe.784 
b/tests/qemunbdkitdata/disk-network-source-curl.args.disk2.pipe.784
new file mode 100644
index 00..5c035e84c5
--- /dev/null
+++ b/tests/qemunbdkitdata/disk-network-source-curl.args.disk2.pipe.784
@@ -0,0 +1 @@
+cookie1=cookievalue1; cookie2=cookievalue2; cookie3=cookievalue3
\ No newline at end of file
diff --git a/tests/qemuxml2argvdata/disk-network-source-curl.x86_64-latest.args 
b/tests/qemuxml2argvdata/disk-network-source-curl.x86_64-latest.args
index cb0e5a92ea..f6ab5532cc 100644
--- a/tests/qemuxml2argvdata/disk-network-source-curl.x86_64-latest.args
+++ b/tests/qemuxml2argvdata/disk-network-source-curl.x86_64-latest.args
@@ -32,9 +32,10 @@ 
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain--1-QEMUGuest1/.config \
 -blockdev 
'{"driver":"https","url":"https://https.example.org:8443/path/to/disk1.iso","cookie-secret":"libvirt-5-storage-httpcookie-secret0","node-name":"libvirt-5-storage","auto-read-only":true,"discard":"unmap"}'
 \
 -blockdev 
'{"node-name":"libvirt-5-format","read-only":true,"driver":"raw","file":"libvirt-5-storage"}'
 \
 -device 
'{"driver":"virtio-blk-pci","bus":"pci.0","addr":"0x3","drive":"libvirt-5-format","id":"virtio-disk0","bootindex":1}'
 \
+-object 
'{"qom-type":"secret","id":"libvirt-4-storage-auth-secret0","data":"9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1","keyid":"masterKey0","iv":"AAECAwQFBgcICQoLDA0ODw==","format":"base64"}'
 \
 -object 
'{"qom-type":"secret","id":"libvirt-4-storage-httpcookie-secret0","data":"BUU0KmnWfonHdjzhYhwVQZ5iTI1KweTJ22q8XWUVoBCVu1z70reDuczPBIabZtC3","keyid":"masterKey0","iv":"AAECAwQFBgcICQoLDA0ODw==","format":"base64"}'
 \
 -object 
'{"qom-type":"secret","id":"libvirt-4-format-encryption-secret0","data":"9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1","keyid":"masterKey0","iv":"AAECAwQFBgcICQoLDA0ODw==","format":"base64"}'
 \
--blockdev 
'{"driver":"https","url":"https://https.example.org:8443/path/to/disk5.iso?foo=bar","sslverify":false,"cookie-secret":"libvirt-4-storage-httpcookie-secret0","node-name":"libvirt-4-storage","auto-read-only":true,"discard":"unmap"}'
 \
+-blockdev 

[libvirt PATCH v7 18/35] util: secure erase virCommand send buffers

[Jonathon Jongsma]

All users of virCommandSetSendBuffer() are using it to send sensitive
data to a child process. So, since these buffers contain sensitive
information, clear it with virSecureErase().

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 src/util/vircommand.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/src/util/vircommand.c b/src/util/vircommand.c
index 5f094c625a..899d413dd2 100644
--- a/src/util/vircommand.c
+++ b/src/util/vircommand.c
@@ -54,6 +54,7 @@
 #include "virpidfile.h"
 #include "virprocess.h"
 #include "virbuffer.h"
+#include "virsecureerase.h"
 #include "virthread.h"
 #include "virstring.h"
 
@@ -1697,6 +1698,7 @@ virCommandFreeSendBuffers(virCommand *cmd)
 
 for (i = 0; i < virCommandGetNumSendBuffers(cmd); i++) {
 VIR_FORCE_CLOSE(cmd->sendBuffers[i].fd);
+virSecureErase(cmd->sendBuffers[i].buffer, cmd->sendBuffers[i].buflen);
 VIR_FREE(cmd->sendBuffers[i].buffer);
 }
 VIR_FREE(cmd->sendBuffers);
-- 
2.41.0

[libvirt PATCH v7 31/35] schema: add keyfile configuration for ssh disks

[Jonathon Jongsma]

Authenticating via key file to an ssh server is often preferable to
logging in via password. In order to support this functionality add a
new  xml element for ssh disks that allows the user to specify
a keyfile and username. Example configuration:


  

...
  
...


Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 docs/formatdomain.rst |  7 +++
 src/conf/schemas/domaincommon.rng | 19 ++-
 2 files changed, 25 insertions(+), 1 deletion(-)

diff --git a/docs/formatdomain.rst b/docs/formatdomain.rst
index 9230f540f4..2ebd7bb1fc 100644
--- a/docs/formatdomain.rst
+++ b/docs/formatdomain.rst
@@ -3005,6 +3005,13 @@ paravirtualized driver is specified via the ``disk`` 
element.
   of these attributes is omitted, then that field is assumed to be the
   default value for the current system. If both ``user`` and ``group``
   are intended to be default, then the entire element may be omitted.
+
+  When using an ``ssh`` protocol, this element is used to enable
+  authentication via ssh keys. In this configuration, the element has two
+  attributes. The ``username`` attribute specifies the name of the user on
+  the remote server and the ``keyfile`` attribute specifies the path to the
+  keyfile. Note that this only works for ssh keys that are not
+  password-protected.
``reconnect``
   For disk type ``vhostuser`` configures reconnect timeout if the 
connection
   is lost. This is set with the two mandatory attributes ``enabled`` and
diff --git a/src/conf/schemas/domaincommon.rng 
b/src/conf/schemas/domaincommon.rng
index ca43586323..47c5ee2a31 100644
--- a/src/conf/schemas/domaincommon.rng
+++ b/src/conf/schemas/domaincommon.rng
@@ -2180,6 +2180,19 @@
 
   
 
+  
+
+  
+
+  
+
+
+  
+
+  
+
+  
+
   
 
   
@@ -2199,11 +2212,15 @@
   
 
 
-  
+  
+
+
+  
 
   
 
   
+
   
 
   
-- 
2.41.0

[libvirt PATCH v7 19/35] qemu: pass sensitive data to nbdkit via pipe

[Jonathon Jongsma]

Rather than passing passwords and cookies (which could contain
passwords) to nbdkit via commandline arguments, use the alternate format
that nbdkit supports where we can specify a file descriptor which nbdkit
will read to get the password or cookies.

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 src/qemu/qemu_nbdkit.c | 54 ++
 1 file changed, 34 insertions(+), 20 deletions(-)

diff --git a/src/qemu/qemu_nbdkit.c b/src/qemu/qemu_nbdkit.c
index e3923ab4f2..22a67b0748 100644
--- a/src/qemu/qemu_nbdkit.c
+++ b/src/qemu/qemu_nbdkit.c
@@ -24,7 +24,6 @@
 #include "virerror.h"
 #include "virlog.h"
 #include "virpidfile.h"
-#include "virsecureerase.h"
 #include "virtime.h"
 #include "virutil.h"
 #include "qemu_block.h"
@@ -753,6 +752,29 @@ qemuNbdkitInitStorageSource(qemuNbdkitCaps *caps,
 }
 
 
+static int
+qemuNbdkitCommandPassDataByPipe(virCommand *cmd,
+const char *argName,
+unsigned char **buf,
+size_t buflen)
+{
+g_autofree char *fdfmt = NULL;
+int fd = virCommandSetSendBuffer(cmd, buf, buflen);
+
+if (fd < 0)
+return -1;
+
+/* some nbdkit arguments accept a variation where nbdkit will read the data
+ * from a file descriptor, e.g. password=-FD */
+fdfmt = g_strdup_printf("-%i", fd);
+virCommandAddArgPair(cmd, argName, fdfmt);
+
+virCommandDoAsyncIO(cmd);
+
+return 0;
+}
+
+
 static int
 qemuNbdkitProcessBuildCommandCurl(qemuNbdkitProcess *proc,
   virCommand *cmd)
@@ -775,7 +797,6 @@ qemuNbdkitProcessBuildCommandCurl(qemuNbdkitProcess *proc,
 g_autoptr(virConnect) conn = virGetConnectSecret();
 g_autofree uint8_t *secret = NULL;
 size_t secretlen = 0;
-g_autofree char *password = NULL;
 int secrettype;
 virStorageAuthDef *authdef = proc->source->auth;
 
@@ -799,26 +820,19 @@ qemuNbdkitProcessBuildCommandCurl(qemuNbdkitProcess *proc,
 return -1;
 }
 
-/* ensure that the secret is a NULL-terminated string */
-password = g_strndup((char*)secret, secretlen);
-virSecureErase(secret, secretlen);
-
-/* for now, just report an error rather than passing the password in
- * cleartext on the commandline */
-virReportError(VIR_ERR_INTERNAL_ERROR, "%s",
-   _("Password not yet supported for nbdkit sources"));
-
-virSecureEraseString(password);
-
-return -1;
+if (qemuNbdkitCommandPassDataByPipe(cmd, "password",
+, secretlen) < 0)
+return -1;
 }
 
-if (proc->source->ncookies > 0) {
-/* for now, just report an error rather than passing cookies in
- * cleartext on the commandline */
-virReportError(VIR_ERR_INTERNAL_ERROR, "%s",
-   _("Cookies not yet supported for nbdkit sources"));
-return -1;
+/* Create a pipe to send the cookies to the nbdkit process. */
+if (proc->source->ncookies) {
+g_autofree char *cookies = 
qemuBlockStorageSourceGetCookieString(proc->source);
+
+if (qemuNbdkitCommandPassDataByPipe(cmd, "cookie",
+(unsigned char**),
+strlen(cookies)) < 0)
+return -1;
 }
 
 if (proc->source->sslverify == VIR_TRISTATE_BOOL_NO) {
-- 
2.41.0

[libvirt PATCH v7 06/35] qemu: implement persistent file cache for nbdkit caps

[Jonathon Jongsma]

Implement the loadFile and saveFile virFileCacheHandlers callbacks so
that nbdkit capabilities are cached perstistently across daemon
restarts. The format and implementation is modeled on the qemu
capabilities, but simplified slightly.

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 po/POTFILES|   1 +
 src/qemu/qemu_nbdkit.c | 226 -
 2 files changed, 225 insertions(+), 2 deletions(-)

diff --git a/po/POTFILES b/po/POTFILES
index 5d6ec195b4..6167f98ac5 100644
--- a/po/POTFILES
+++ b/po/POTFILES
@@ -181,6 +181,7 @@ src/qemu/qemu_monitor.c
 src/qemu/qemu_monitor_json.c
 src/qemu/qemu_monitor_text.c
 src/qemu/qemu_namespace.c
+src/qemu/qemu_nbdkit.c
 src/qemu/qemu_passt.c
 src/qemu/qemu_process.c
 src/qemu/qemu_qapi.c
diff --git a/src/qemu/qemu_nbdkit.c b/src/qemu/qemu_nbdkit.c
index 9828f562f7..97612b637f 100644
--- a/src/qemu/qemu_nbdkit.c
+++ b/src/qemu/qemu_nbdkit.c
@@ -312,11 +312,233 @@ virNbdkitCapsNewData(const char *binary,
 }
 
 
+static int
+qemuNbdkitCapsValidateBinary(qemuNbdkitCaps *nbdkitCaps,
+ xmlXPathContextPtr ctxt)
+{
+g_autofree char *str = NULL;
+
+if (!(str = virXPathString("string(./path)", ctxt))) {
+virReportError(VIR_ERR_INTERNAL_ERROR, "%s",
+   _("missing path in nbdkit capabilities cache"));
+return -1;
+}
+
+if (STRNEQ(str, nbdkitCaps->path)) {
+virReportError(VIR_ERR_INTERNAL_ERROR,
+   _("Expected caps for '%1$s' but saw '%2$s'"),
+   nbdkitCaps->path, str);
+return -1;
+}
+
+return 0;
+}
+
+
+static int
+qemuNbdkitCapsParseFlags(qemuNbdkitCaps *nbdkitCaps,
+ xmlXPathContextPtr ctxt)
+{
+g_autofree xmlNodePtr *nodes = NULL;
+size_t i;
+int n;
+
+if ((n = virXPathNodeSet("./flag", ctxt, )) < 0) {
+virReportError(VIR_ERR_INTERNAL_ERROR, "%s",
+   _("failed to parse qemu capabilities flags"));
+return -1;
+}
+
+VIR_DEBUG("Got flags %d", n);
+for (i = 0; i < n; i++) {
+unsigned int flag;
+
+if (virXMLPropEnum(nodes[i], "name", qemuNbdkitCapsTypeFromString,
+   VIR_XML_PROP_REQUIRED, ) < 0)
+return -1;
+
+qemuNbdkitCapsSet(nbdkitCaps, flag);
+}
+
+return 0;
+}
+
+
+/*
+ * Parsing a doc that looks like
+ *
+ * 
+ *   /some/path
+ *   234235253
+ *   234235253
+ *   234235253
+ *   234235253
+ *   1002016
+ *   
+ *   
+ *   ...
+ * 
+ *
+ * Returns 0 on success, 1 if outdated, -1 on error
+ */
+static int
+qemuNbdkitCapsLoadCache(qemuNbdkitCaps *nbdkitCaps,
+const char *filename)
+{
+g_autoptr(xmlDoc) doc = NULL;
+g_autoptr(xmlXPathContext) ctxt = NULL;
+long long int l;
+
+if (!(doc = virXMLParse(filename, NULL, NULL, "nbdkitCaps", , NULL, 
false)))
+return -1;
+
+if (virXPathLongLong("string(./selfctime)", ctxt, ) < 0) {
+VIR_DEBUG("missing selfctime in nbdkit capabilities XML");
+return -1;
+}
+nbdkitCaps->libvirtCtime = (time_t)l;
+
+nbdkitCaps->libvirtVersion = 0;
+virXPathUInt("string(./selfvers)", ctxt, >libvirtVersion);
+
+if (nbdkitCaps->libvirtCtime != virGetSelfLastChanged() ||
+nbdkitCaps->libvirtVersion != LIBVIR_VERSION_NUMBER) {
+VIR_DEBUG("Outdated capabilities in %s: libvirt changed (%lld vs %lld, 
%lu vs %lu), stopping load",
+  nbdkitCaps->path,
+  (long long)nbdkitCaps->libvirtCtime,
+  (long long)virGetSelfLastChanged(),
+  (unsigned long)nbdkitCaps->libvirtVersion,
+  (unsigned long)LIBVIR_VERSION_NUMBER);
+return 1;
+}
+
+if (qemuNbdkitCapsValidateBinary(nbdkitCaps, ctxt) < 0)
+return -1;
+
+if (virXPathLongLong("string(./nbdkitctime)", ctxt, ) < 0) {
+VIR_DEBUG("missing nbdkitctime in nbdkit capabilities XML");
+return -1;
+}
+nbdkitCaps->ctime = (time_t)l;
+
+if (virXPathLongLong("string(./plugindirmtime)", ctxt, ) < 0) {
+VIR_DEBUG("missing plugindirmtime in nbdkit capabilities XML");
+return -1;
+}
+nbdkitCaps->pluginDirMtime = (time_t)l;
+
+if (virXPathLongLong("string(./filterdirmtime)", ctxt, ) < 0) {
+VIR_DEBUG("missing filterdirmtime in nbdkit capabilities XML");
+return -1;
+}
+nbdkitCaps->filterDirMtime = (time_t)l;
+
+if (qemuNbdkitCapsParseFlags(nbdkitCaps, ctxt) < 0)
+return -1;
+
+if ((nbdkitCaps->version = virXPathString("string(./version)", ctxt)) == 
NULL) {
+VIR_DEBUG("missing version in nbdkit capabilities cache");
+return -1;
+}
+
+return 0;
+}
+
+
+static void*
+virNbdkitCapsLoadFile(const char *filename,
+  const char *binary,
+  void *privData G_GNUC_UNUSED,
+  bool *outdated)
+{
+

[libvirt PATCH v7 21/35] util: make virCommandSetSendBuffer testable

[Jonathon Jongsma]

Add a private function to peek at the list of send buffers in virCommand
so that it is testable

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 src/libvirt_private.syms  |  1 +
 src/util/vircommand.c | 17 +
 src/util/vircommand.h |  8 
 src/util/vircommandpriv.h |  4 
 4 files changed, 22 insertions(+), 8 deletions(-)

diff --git a/src/libvirt_private.syms b/src/libvirt_private.syms
index 1e3e407097..e4da5388e7 100644
--- a/src/libvirt_private.syms
+++ b/src/libvirt_private.syms
@@ -2117,6 +2117,7 @@ virCommandNewArgs;
 virCommandNewVAList;
 virCommandNonblockingFDs;
 virCommandPassFD;
+virCommandPeekSendBuffers;
 virCommandRawStatus;
 virCommandRequireHandshake;
 virCommandRun;
diff --git a/src/util/vircommand.c b/src/util/vircommand.c
index 899d413dd2..7d7ce4297e 100644
--- a/src/util/vircommand.c
+++ b/src/util/vircommand.c
@@ -78,14 +78,6 @@ struct _virCommandFD {
 unsigned int flags;
 };
 
-typedef struct _virCommandSendBuffer virCommandSendBuffer;
-struct _virCommandSendBuffer {
-int fd;
-unsigned char *buffer;
-size_t buflen;
-off_t offset;
-};
-
 struct _virCommand {
 int has_error; /* 0 on success, -1 on error  */
 
@@ -3515,3 +3507,12 @@ virCommandSetRunAmong(virCommand *cmd,
 
 cmd->schedCore = pid;
 }
+
+void
+virCommandPeekSendBuffers(virCommand *cmd,
+  virCommandSendBuffer **buffers,
+  int *nbuffers)
+{
+*buffers = cmd->sendBuffers;
+*nbuffers = cmd->numSendBuffers;
+}
diff --git a/src/util/vircommand.h b/src/util/vircommand.h
index d51449ac90..9bcdce35b9 100644
--- a/src/util/vircommand.h
+++ b/src/util/vircommand.h
@@ -24,6 +24,14 @@
 #include "internal.h"
 #include "virbuffer.h"
 
+typedef struct _virCommandSendBuffer virCommandSendBuffer;
+struct _virCommandSendBuffer {
+int fd;
+unsigned char *buffer;
+size_t buflen;
+off_t offset;
+};
+
 typedef struct _virCommand virCommand;
 
 /* This will execute in the context of the first child
diff --git a/src/util/vircommandpriv.h b/src/util/vircommandpriv.h
index ff17fa5ded..d579810bb5 100644
--- a/src/util/vircommandpriv.h
+++ b/src/util/vircommandpriv.h
@@ -47,3 +47,7 @@ void virCommandSetDryRun(virCommandDryRunToken *tok,
  bool bufCommandStripPath,
  virCommandDryRunCallback cb,
  void *opaque);
+
+void virCommandPeekSendBuffers(virCommand *cmd,
+   virCommandSendBuffer **buffers,
+   int *nbuffers);
-- 
2.41.0

[libvirt PATCH v7 24/35] qemu: Monitor nbdkit process for exit

[Jonathon Jongsma]

Adds the ability to monitor the nbdkit process so that we can take
action in case the child exits unexpectedly.

When the nbdkit process exits, we pause the vm, restart nbdkit, and then
resume the vm. This allows the vm to continue working in the event of a
nbdkit failure.

Eventually we may want to generalize this functionality since we may
need something similar for e.g. qemu-storage-daemon, etc.

The process is monitored with the pidfd_open() syscall if it exists
(since linux 5.3). Otherwise it resorts to checking whether the process
is alive once a second. The one-second time period was chosen somewhat
arbitrarily.

Signed-off-by: Jonathon Jongsma 
---
 meson.build |   7 ++
 src/qemu/qemu_domain.c  |   1 +
 src/qemu/qemu_domain.h  |   1 +
 src/qemu/qemu_driver.c  |  18 ++
 src/qemu/qemu_nbdkit.c  | 137 ++--
 src/qemu/qemu_nbdkit.h  |   8 ++-
 src/qemu/qemu_process.c |  13 +++-
 src/qemu/qemu_process.h |   3 +
 8 files changed, 180 insertions(+), 8 deletions(-)

diff --git a/meson.build b/meson.build
index 965ada483b..8da3b59070 100644
--- a/meson.build
+++ b/meson.build
@@ -682,6 +682,13 @@ symbols = [
   [ 'sched.h', 'cpu_set_t' ],
 ]
 
+if host_machine.system() == 'linux'
+  symbols += [
+# process management
+[ 'sys/syscall.h', 'SYS_pidfd_open' ],
+  ]
+endif
+
 foreach symbol : symbols
   if cc.has_header_symbol(symbol[0], symbol[1], args: '-D_GNU_SOURCE', prefix: 
symbol.get(2, ''))
 conf.set('WITH_DECL_@0@'.format(symbol[1].to_upper()), 1)
diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c
index 46fe5a1cf4..2b8ece8f19 100644
--- a/src/qemu/qemu_domain.c
+++ b/src/qemu/qemu_domain.c
@@ -11511,6 +11511,7 @@ qemuProcessEventFree(struct qemuProcessEvent *event)
 case QEMU_PROCESS_EVENT_PR_DISCONNECT:
 case QEMU_PROCESS_EVENT_UNATTENDED_MIGRATION:
 case QEMU_PROCESS_EVENT_RESET:
+case QEMU_PROCESS_EVENT_NBDKIT_EXITED:
 case QEMU_PROCESS_EVENT_LAST:
 break;
 }
diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h
index ddd20e67b4..f018b45eb6 100644
--- a/src/qemu/qemu_domain.h
+++ b/src/qemu/qemu_domain.h
@@ -465,6 +465,7 @@ typedef enum {
 QEMU_PROCESS_EVENT_MEMORY_DEVICE_SIZE_CHANGE,
 QEMU_PROCESS_EVENT_UNATTENDED_MIGRATION,
 QEMU_PROCESS_EVENT_RESET,
+QEMU_PROCESS_EVENT_NBDKIT_EXITED,
 
 QEMU_PROCESS_EVENT_LAST
 } qemuProcessEventType;
diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c
index ad8428948b..46e089fe0f 100644
--- a/src/qemu/qemu_driver.c
+++ b/src/qemu/qemu_driver.c
@@ -4040,6 +4040,21 @@ processResetEvent(virQEMUDriver *driver,
 }
 
 
+
+
+static void
+processNbdkitExitedEvent(virDomainObj *vm,
+ qemuNbdkitProcess *nbdkit)
+{
+if (virDomainObjBeginJob(vm, VIR_JOB_MODIFY) < 0)
+return;
+
+qemuNbdkitProcessRestart(nbdkit, vm);
+
+virDomainObjEndJob(vm);
+}
+
+
 static void qemuProcessEventHandler(void *data, void *opaque)
 {
 struct qemuProcessEvent *processEvent = data;
@@ -4097,6 +4112,9 @@ static void qemuProcessEventHandler(void *data, void 
*opaque)
 case QEMU_PROCESS_EVENT_RESET:
 processResetEvent(driver, vm);
 break;
+case QEMU_PROCESS_EVENT_NBDKIT_EXITED:
+processNbdkitExitedEvent(vm, processEvent->data);
+break;
 case QEMU_PROCESS_EVENT_LAST:
 break;
 }
diff --git a/src/qemu/qemu_nbdkit.c b/src/qemu/qemu_nbdkit.c
index df638e99c0..c3fa349922 100644
--- a/src/qemu/qemu_nbdkit.c
+++ b/src/qemu/qemu_nbdkit.c
@@ -19,6 +19,7 @@
 
 #include 
 #include 
+#include 
 
 #include "vircommand.h"
 #include "virerror.h"
@@ -33,6 +34,7 @@
 #include "qemu_nbdkit.h"
 #define LIBVIRT_QEMU_NBDKITPRIV_H_ALLOW
 #include "qemu_nbdkitpriv.h"
+#include "qemu_process.h"
 #include "qemu_security.h"
 
 #include 
@@ -611,6 +613,106 @@ qemuNbdkitCapsCacheNew(const char *cachedir)
 }
 
 
+void
+qemuNbdkitProcessRestart(qemuNbdkitProcess *proc,
+ virDomainObj *vm)
+{
+qemuDomainObjPrivate *vmpriv = vm->privateData;
+virQEMUDriver *driver = vmpriv->driver;
+
+/* clean up resources associated with process */
+qemuNbdkitProcessStop(proc);
+
+if (qemuNbdkitProcessStart(proc, vm, driver) < 0)
+VIR_WARN("Unable to restart nbkdit process");
+}
+
+
+typedef struct {
+qemuNbdkitProcess *proc;
+virDomainObj *vm;
+} qemuNbdkitProcessEventData;
+
+
+static qemuNbdkitProcessEventData*
+qemuNbdkitProcessEventDataNew(qemuNbdkitProcess *proc,
+  virDomainObj *vm)
+{
+qemuNbdkitProcessEventData *d = g_new(qemuNbdkitProcessEventData, 1);
+d->proc = proc;
+d->vm = virObjectRef(vm);
+return d;
+}
+
+
+static void
+qemuNbdkitProcessEventDataFree(qemuNbdkitProcessEventData *d)
+{
+virObjectUnref(d->vm);
+g_free(d);
+}
+
+
+#if WITH_DECL_SYS_PIDFD_OPEN
+static void
+qemuNbdkitProcessPidfdCb(int watch G_GNUC_UNUSED,
+ int fd,
+ int 

[libvirt PATCH v7 35/35] rpm: update spec file for for nbdkit support

[Jonathon Jongsma]

Require libnbd-devel when building the qemu driver, recommend nbdkit
packages.

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 libvirt.spec.in | 8 
 1 file changed, 8 insertions(+)

diff --git a/libvirt.spec.in b/libvirt.spec.in
index 7157cfe3b4..94cae34496 100644
--- a/libvirt.spec.in
+++ b/libvirt.spec.in
@@ -312,6 +312,7 @@ BuildRequires: util-linux
 BuildRequires: libacl-devel
 # From QEMU RPMs, used by virstoragetest
 BuildRequires: /usr/bin/qemu-img
+BuildRequires: libnbd-devel
 %endif
 # For LVM drivers
 BuildRequires: lvm2
@@ -768,6 +769,9 @@ Requires: numad
 Recommends: passt
 Recommends: passt-selinux
 %endif
+Recommends: nbdkit
+Recommends: nbdkit-curl-plugin
+Recommends: nbdkit-ssh-plugin
 
 %description daemon-driver-qemu
 The qemu driver plugin for the libvirtd daemon, providing
@@ -1069,8 +1073,10 @@ exit 1
 
 %if %{with_qemu}
 %define arg_qemu -Ddriver_qemu=enabled
+%define arg_libnbd -Dlibndb=enabled
 %else
 %define arg_qemu -Ddriver_qemu=disabled
+%define arg_libnbd -Dlibndb=disabled
 %endif
 
 %if %{with_openvz}
@@ -1259,6 +1265,7 @@ export SOURCE_DATE_EPOCH=$(stat --printf='%Y' 
%{_specdir}/libvirt.spec)
-Dyajl=enabled \
%{?arg_sanlock} \
-Dlibpcap=enabled \
+   %{?arg_libnbd} \
-Dlibnl=enabled \
-Daudit=enabled \
-Ddtrace=enabled \
@@ -1322,6 +1329,7 @@ export SOURCE_DATE_EPOCH=$(stat --printf='%Y' 
%{_specdir}/libvirt.spec)
   -Dglusterfs=disabled \
   -Dhost_validate=disabled \
   -Dlibiscsi=disabled \
+  -Dlibnbd=disabled \
   -Dlibnl=disabled \
   -Dlibpcap=disabled \
   -Dlibssh2=disabled \
-- 
2.41.0

[libvirt PATCH v7 20/35] qemu: use nbdkit to serve network disks if available

[Jonathon Jongsma]

For virStorageSource objects that contain an nbdkitProcess, start that
nbdkit process to serve that network drive and then pass the nbdkit
socket to qemu rather than sending the network url to qemu directly.

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 src/qemu/qemu_block.c | 162 +++---
 src/qemu/qemu_domain.c|  13 +-
 src/qemu/qemu_extdevice.c |  62 +++
 src/qemu/qemu_hotplug.c   |   7 +
 src/qemu/qemu_nbdkit.c|  42 +
 src/qemu/qemu_nbdkit.h|  13 ++
 ...sk-cdrom-network-nbdkit.x86_64-latest.args |  42 +
 .../disk-cdrom-network-nbdkit.xml |   1 +
 ...isk-network-http-nbdkit.x86_64-latest.args |  44 +
 .../disk-network-http-nbdkit.xml  |   1 +
 ...rce-curl-nbdkit-backing.x86_64-latest.args |  37 
 ...isk-network-source-curl-nbdkit-backing.xml |  45 +
 ...work-source-curl-nbdkit.x86_64-latest.args |  49 ++
 .../disk-network-source-curl-nbdkit.xml   |   1 +
 ...isk-network-source-curl.x86_64-latest.args |  52 ++
 .../disk-network-source-curl.xml  |  71 
 ...disk-network-ssh-nbdkit.x86_64-latest.args |  35 
 .../disk-network-ssh-nbdkit.xml   |   1 +
 tests/qemuxml2argvtest.c  |   6 +
 19 files changed, 618 insertions(+), 66 deletions(-)
 create mode 100644 
tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.x86_64-latest.args
 create mode 12 tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.xml
 create mode 100644 
tests/qemuxml2argvdata/disk-network-http-nbdkit.x86_64-latest.args
 create mode 12 tests/qemuxml2argvdata/disk-network-http-nbdkit.xml
 create mode 100644 
tests/qemuxml2argvdata/disk-network-source-curl-nbdkit-backing.x86_64-latest.args
 create mode 100644 
tests/qemuxml2argvdata/disk-network-source-curl-nbdkit-backing.xml
 create mode 100644 
tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.x86_64-latest.args
 create mode 12 tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.xml
 create mode 100644 
tests/qemuxml2argvdata/disk-network-source-curl.x86_64-latest.args
 create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl.xml
 create mode 100644 
tests/qemuxml2argvdata/disk-network-ssh-nbdkit.x86_64-latest.args
 create mode 12 tests/qemuxml2argvdata/disk-network-ssh-nbdkit.xml

diff --git a/src/qemu/qemu_block.c b/src/qemu/qemu_block.c
index dcdf883926..1a2dc8ffb4 100644
--- a/src/qemu/qemu_block.c
+++ b/src/qemu/qemu_block.c
@@ -438,6 +438,32 @@ qemuBlockStorageSourceGetCURLProps(virStorageSource *src,
 }
 
 
+static virJSONValue *
+qemuBlockStorageSourceGetNbdkitProps(virStorageSource *src)
+{
+qemuDomainStorageSourcePrivate *srcPriv = 
QEMU_DOMAIN_STORAGE_SOURCE_PRIVATE(src);
+virJSONValue *ret = NULL;
+g_autoptr(virJSONValue) serverprops = NULL;
+virStorageNetHostDef host = { .transport = VIR_STORAGE_NET_HOST_TRANS_UNIX 
};
+
+/* srcPriv->nbdkitProcess will already be initialized if we can use nbdkit
+ * to proxy this storage source */
+if (!(srcPriv  && srcPriv->nbdkitProcess))
+return NULL;
+
+host.socket = srcPriv->nbdkitProcess->socketfile;
+serverprops = qemuBlockStorageSourceBuildJSONSocketAddress();
+
+if (!serverprops)
+return NULL;
+
+if (virJSONValueObjectAdd(, "a:server", , NULL) < 0)
+return NULL;
+
+return ret;
+}
+
+
 static virJSONValue *
 qemuBlockStorageSourceGetISCSIProps(virStorageSource *src,
 bool onlytarget)
@@ -890,69 +916,75 @@ qemuBlockStorageSourceGetBackendProps(virStorageSource 
*src,
 return NULL;
 
 case VIR_STORAGE_TYPE_NETWORK:
-switch ((virStorageNetProtocol) src->protocol) {
-case VIR_STORAGE_NET_PROTOCOL_GLUSTER:
-driver = "gluster";
-if (!(fileprops = qemuBlockStorageSourceGetGlusterProps(src, 
onlytarget)))
-return NULL;
-break;
-
-case VIR_STORAGE_NET_PROTOCOL_VXHS:
-driver = "vxhs";
-if (!(fileprops = qemuBlockStorageSourceGetVxHSProps(src, 
onlytarget)))
-return NULL;
-break;
-
-case VIR_STORAGE_NET_PROTOCOL_HTTP:
-case VIR_STORAGE_NET_PROTOCOL_HTTPS:
-case VIR_STORAGE_NET_PROTOCOL_FTP:
-case VIR_STORAGE_NET_PROTOCOL_FTPS:
-case VIR_STORAGE_NET_PROTOCOL_TFTP:
-driver = virStorageNetProtocolTypeToString(src->protocol);
-if (!(fileprops = qemuBlockStorageSourceGetCURLProps(src, 
onlytarget)))
-return NULL;
-break;
-
-case VIR_STORAGE_NET_PROTOCOL_ISCSI:
-driver = "iscsi";
-if (!(fileprops = qemuBlockStorageSourceGetISCSIProps(src, 
onlytarget)))
-return NULL;
-break;
-
-case VIR_STORAGE_NET_PROTOCOL_NBD:
+/* prefer using nbdkit for 

[libvirt PATCH v7 28/35] qemu: implement password auth for ssh disks with nbdkit

[Jonathon Jongsma]

For ssh disks that are served by nbdkit, lookup the password from the
configured secret and securely pass it to the nbdkit process using fd
passing.

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 src/qemu/qemu_nbdkit.c| 84 ++-
 .../disk-network-ssh-password.args.disk0  |  8 ++
 ...k-network-ssh-password.args.disk0.pipe.778 |  1 +
 .../disk-network-ssh.args.disk1   |  8 ++
 .../disk-network-ssh.args.disk1.pipe.778  |  1 +
 tests/qemunbdkittest.c|  1 +
 ...sk-network-ssh-password.x86_64-latest.args | 35 
 .../disk-network-ssh-password.xml | 34 
 tests/qemuxml2argvtest.c  |  1 +
 9 files changed, 134 insertions(+), 39 deletions(-)
 create mode 100644 tests/qemunbdkitdata/disk-network-ssh-password.args.disk0
 create mode 100644 
tests/qemunbdkitdata/disk-network-ssh-password.args.disk0.pipe.778
 create mode 100644 tests/qemunbdkitdata/disk-network-ssh.args.disk1
 create mode 100644 tests/qemunbdkitdata/disk-network-ssh.args.disk1.pipe.778
 create mode 100644 
tests/qemuxml2argvdata/disk-network-ssh-password.x86_64-latest.args
 create mode 100644 tests/qemuxml2argvdata/disk-network-ssh-password.xml

diff --git a/src/qemu/qemu_nbdkit.c b/src/qemu/qemu_nbdkit.c
index 3e82c90850..cedeaa880b 100644
--- a/src/qemu/qemu_nbdkit.c
+++ b/src/qemu/qemu_nbdkit.c
@@ -936,6 +936,43 @@ qemuNbdkitCommandPassDataByPipe(virCommand *cmd,
 }
 
 
+static int
+qemuNbdkitProcessBuildCommandAuth(virStorageAuthDef *authdef,
+  virCommand *cmd)
+{
+g_autoptr(virConnect) conn = NULL;
+g_autofree uint8_t *secret = NULL;
+size_t secretlen = 0;
+int secrettype;
+
+if (!authdef)
+return 0;
+
+if ((secrettype = virSecretUsageTypeFromString(authdef->secrettype)) < 0) {
+virReportError(VIR_ERR_CONFIG_UNSUPPORTED,
+   _("invalid secret type %1$s"),
+   authdef->secrettype);
+return -1;
+}
+
+conn = virGetConnectSecret();
+if (virSecretGetSecretString(conn,
+ >seclookupdef,
+ secrettype,
+ ,
+ ) < 0)
+return -1;
+
+virCommandAddArgPair(cmd, "user", authdef->username);
+
+if (qemuNbdkitCommandPassDataByPipe(cmd, "password",
+, secretlen) < 0)
+return -1;
+
+return 0;
+}
+
+
 static int
 qemuNbdkitProcessBuildCommandCurl(qemuNbdkitProcess *proc,
   virCommand *cmd)
@@ -954,37 +991,8 @@ qemuNbdkitProcessBuildCommandCurl(qemuNbdkitProcess *proc,
 }
 virCommandAddArgPair(cmd, "url", uristring);
 
-if (proc->source->auth) {
-g_autoptr(virConnect) conn = virGetConnectSecret();
-g_autofree uint8_t *secret = NULL;
-size_t secretlen = 0;
-int secrettype;
-virStorageAuthDef *authdef = proc->source->auth;
-
-virCommandAddArgPair(cmd, "user",
- proc->source->auth->username);
-
-if ((secrettype = 
virSecretUsageTypeFromString(proc->source->auth->secrettype)) < 0) {
-virReportError(VIR_ERR_CONFIG_UNSUPPORTED,
-   _("invalid secret type %1$s"),
-   proc->source->auth->secrettype);
-return -1;
-}
-
-if (virSecretGetSecretString(conn,
- >seclookupdef,
- secrettype,
- ,
- ) < 0) {
-virReportError(VIR_ERR_INTERNAL_ERROR, "%s",
-   _("failed to get auth secret for storage"));
-return -1;
-}
-
-if (qemuNbdkitCommandPassDataByPipe(cmd, "password",
-, secretlen) < 0)
-return -1;
-}
+if (proc->source->auth && 
qemuNbdkitProcessBuildCommandAuth(proc->source->auth, cmd) < 0)
+return -1;
 
 /* Create a pipe to send the cookies to the nbdkit process. */
 if (proc->source->ncookies) {
@@ -1013,7 +1021,6 @@ static int
 qemuNbdkitProcessBuildCommandSSH(qemuNbdkitProcess *proc,
  virCommand *cmd)
 {
-const char *user = NULL;
 virStorageNetHostDef *host = >source->hosts[0];
 g_autofree char *portstr = g_strdup_printf("%u", host->port);
 
@@ -1024,13 +1031,12 @@ qemuNbdkitProcessBuildCommandSSH(qemuNbdkitProcess 
*proc,
 virCommandAddArgPair(cmd, "port", portstr);
 virCommandAddArgPair(cmd, "path", proc->source->path);
 
-if (proc->source->auth)
-user = proc->source->auth->username;
-else if (proc->source->ssh_user)
-user = proc->source->ssh_user;
-
-if (user)
-virCommandAddArgPair(cmd, "user", user);
+if 

[libvirt PATCH v7 34/35] qemu: implement ssh-agent auth for ssh disks with nbdkit

[Jonathon Jongsma]

It's not possible to use password-protected ssh keys directly with
libvirt because libvirt doesn't have any way to prompt a user for the
password. To accomodate password-protected key files, an administrator
can add these keys to an ssh agent and then configure the domain with
the path to the ssh-agent socket.

Note that this requires an administrator or management app to
configure the ssh-agent with an appropriate socket path and add the
necessary keys to it. In addition, it does not currently work with
selinux enabled. The ssh-agent socket would need a label that libvirt
would be allowed to access rather than unconfined_t.

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 src/conf/domain_conf.c  | 14 --
 src/conf/storage_source_conf.c  |  2 ++
 src/conf/storage_source_conf.h  |  1 +
 src/qemu/qemu_nbdkit.c  | 10 ++
 .../disk-network-ssh-key.args.disk0 |  6 +++---
 .../disk-network-ssh-key.args.disk1 |  9 +
 tests/qemuxml2argvdata/disk-network-ssh-key.xml | 17 ++---
 7 files changed, 51 insertions(+), 8 deletions(-)
 create mode 100644 tests/qemunbdkitdata/disk-network-ssh-key.args.disk1

diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c
index 929e115bce..398f40d2be 100644
--- a/src/conf/domain_conf.c
+++ b/src/conf/domain_conf.c
@@ -7277,8 +7277,17 @@ virDomainDiskSourceNetworkParse(xmlNodePtr node,
 if (!(src->ssh_user = virXMLPropStringRequired(tmpnode, 
"username")))
 return -1;
 
-if (!(src->ssh_keyfile = virXMLPropStringRequired(tmpnode, 
"keyfile")))
+/* optional path to an ssh key file */
+src->ssh_keyfile = virXMLPropString(tmpnode, "keyfile");
+
+/* optional ssh-agent socket location */
+src->ssh_agent = virXMLPropString(tmpnode, "agentsock");
+if (!src->ssh_keyfile && !src->ssh_agent) {
+virReportError(VIR_ERR_XML_ERROR,
+   _("element '%1$s' requires either 'keyfile' or 
'agentsock' attribute"),
+   tmpnode->name);
 return -1;
+}
 }
 }
 
@@ -22291,11 +22300,12 @@ virDomainDiskSourceFormatNetwork(virBuffer *attrBuf,
 if (src->protocol == VIR_STORAGE_NET_PROTOCOL_SSH) {
 if (src->ssh_known_hosts_file)
 virBufferEscapeString(childBuf, "\n", 
src->ssh_known_hosts_file);
-if (src->ssh_keyfile) {
+if (src->ssh_keyfile || src->ssh_agent) {
 virBufferAddLit(childBuf, "ssh_user);
 virBufferEscapeString(childBuf, " keyfile='%s'", src->ssh_keyfile);
+virBufferEscapeString(childBuf, " agentsock='%s'", src->ssh_agent);
 
 virBufferAddLit(childBuf, "/>\n");
 }
diff --git a/src/conf/storage_source_conf.c b/src/conf/storage_source_conf.c
index ce9c1f66c2..cafa031dfe 100644
--- a/src/conf/storage_source_conf.c
+++ b/src/conf/storage_source_conf.c
@@ -897,6 +897,7 @@ virStorageSourceCopy(const virStorageSource *src,
 def->ssh_user = g_strdup(src->ssh_user);
 def->ssh_known_hosts_file = g_strdup(src->ssh_known_hosts_file);
 def->ssh_keyfile = g_strdup(src->ssh_keyfile);
+def->ssh_agent = g_strdup(src->ssh_agent);
 
 def->nfs_user = g_strdup(src->nfs_user);
 def->nfs_group = g_strdup(src->nfs_group);
@@ -1174,6 +1175,7 @@ virStorageSourceClear(virStorageSource *def)
 VIR_FREE(def->ssh_user);
 VIR_FREE(def->ssh_known_hosts_file);
 VIR_FREE(def->ssh_keyfile);
+VIR_FREE(def->ssh_agent);
 
 VIR_FREE(def->nfs_user);
 VIR_FREE(def->nfs_group);
diff --git a/src/conf/storage_source_conf.h b/src/conf/storage_source_conf.h
index 8c805664af..061faa66cb 100644
--- a/src/conf/storage_source_conf.h
+++ b/src/conf/storage_source_conf.h
@@ -411,6 +411,7 @@ struct _virStorageSource {
 bool ssh_host_key_check_disabled;
 char *ssh_known_hosts_file;
 char *ssh_keyfile;
+char *ssh_agent;
 
 /* nfs_user and nfs_group store the strings passed in by the user for NFS 
params.
  * nfs_uid and nfs_gid represent the converted/looked up ID numbers which 
are used
diff --git a/src/qemu/qemu_nbdkit.c b/src/qemu/qemu_nbdkit.c
index 0393850ddc..a417146426 100644
--- a/src/qemu/qemu_nbdkit.c
+++ b/src/qemu/qemu_nbdkit.c
@@ -1043,6 +1043,9 @@ qemuNbdkitProcessBuildCommandSSH(qemuNbdkitProcess *proc,
 virCommandAddArgPair(cmd, "user", proc->source->ssh_user);
 }
 
+if (proc->source->ssh_agent)
+virCommandAddEnvPair(cmd, "SSH_AUTH_SOCK", proc->source->ssh_agent);
+
 if (proc->source->ssh_host_key_check_disabled)
 virCommandAddArgPair(cmd, "verify-remote-host", "false");
 
@@ -1165,6 +1168,10 @@ qemuNbdkitProcessStart(qemuNbdkitProcess *proc,
 qemuSecurityDomainSetPathLabel(driver, vm, proc->source->ssh_keyfile, 
false) < 0)
 goto error;
 
+if 

[libvirt PATCH v7 25/35] qemu: Taint domain if nbdkit restart fails

[Jonathon Jongsma]

Since the restart handler will trigger at an arbitrary time (when the
nbdkit process crashes, for instance), it's difficult to provide
feedback to the user if the restart is unsuccessful. Rather than just
relying on a warning in the log, taint the domain so that there will be
a slightly more user-visible notification.

Signed-off-by: Jonathon Jongsma 
---
 src/conf/domain_conf.c | 2 ++
 src/conf/domain_conf.h | 1 +
 src/qemu/qemu_nbdkit.c | 4 +++-
 3 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c
index bb4f1fdb94..8feaf5d055 100644
--- a/src/conf/domain_conf.c
+++ b/src/conf/domain_conf.c
@@ -87,6 +87,7 @@ VIR_ENUM_IMPL(virDomainTaint,
   "custom-hypervisor-feature",
   "deprecated-config",
   "custom-device",
+  "nbdkit-restart",
 );
 
 VIR_ENUM_IMPL(virDomainTaintMessage,
@@ -105,6 +106,7 @@ VIR_ENUM_IMPL(virDomainTaintMessage,
   N_("hypervisor feature autodetection override"),
   N_("use of deprecated configuration settings"),
   N_("custom device configuration"),
+  N_("nbdkit restart failed"),
 );
 
 VIR_ENUM_IMPL(virDomainVirt,
diff --git a/src/conf/domain_conf.h b/src/conf/domain_conf.h
index ca195a52d2..c0729905a8 100644
--- a/src/conf/domain_conf.h
+++ b/src/conf/domain_conf.h
@@ -3194,6 +3194,7 @@ typedef enum {
 VIR_DOMAIN_TAINT_CUSTOM_HYPERVISOR_FEATURE, /* custom hypervisor feature 
control */
 VIR_DOMAIN_TAINT_DEPRECATED_CONFIG,  /* Configuration that is marked 
deprecated */
 VIR_DOMAIN_TAINT_CUSTOM_DEVICE, /* hypervisor device config customized */
+VIR_DOMAIN_TAINT_NBDKIT_RESTART,/* nbdkit could not be restarted */
 
 VIR_DOMAIN_TAINT_LAST
 } virDomainTaintFlags;
diff --git a/src/qemu/qemu_nbdkit.c b/src/qemu/qemu_nbdkit.c
index c3fa349922..ff5f1c0d05 100644
--- a/src/qemu/qemu_nbdkit.c
+++ b/src/qemu/qemu_nbdkit.c
@@ -623,8 +623,10 @@ qemuNbdkitProcessRestart(qemuNbdkitProcess *proc,
 /* clean up resources associated with process */
 qemuNbdkitProcessStop(proc);
 
-if (qemuNbdkitProcessStart(proc, vm, driver) < 0)
+if (qemuNbdkitProcessStart(proc, vm, driver) < 0) {
 VIR_WARN("Unable to restart nbkdit process");
+virDomainObjTaint(vm, VIR_DOMAIN_TAINT_NBDKIT_RESTART);
+}
 }
 
 
-- 
2.41.0

[libvirt PATCH v7 13/35] qemu: move qemuProcessReadLog() to qemuLogContext

[Jonathon Jongsma]

This code can be used by the nbdkit implementation for reading back
filtered log data for error reporting. Move it to qemuLogContext so that
it can be shared. Renamed to qemuLogContextReadFiltered().

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 src/qemu/qemu_logcontext.c | 65 
 src/qemu/qemu_logcontext.h |  3 ++
 src/qemu/qemu_process.c| 67 +-
 3 files changed, 69 insertions(+), 66 deletions(-)

diff --git a/src/qemu/qemu_logcontext.c b/src/qemu/qemu_logcontext.c
index 0121ae5173..6e20f58bfa 100644
--- a/src/qemu/qemu_logcontext.c
+++ b/src/qemu/qemu_logcontext.c
@@ -21,6 +21,7 @@
 #include "qemu_logcontext.h"
 #include "viralloc.h"
 #include "virlog.h"
+#include "virstring.h"
 #include "virutil.h"
 
 #include 
@@ -236,6 +237,70 @@ qemuLogContextRead(qemuLogContext *ctxt,
 }
 
 
+/**
+ * qemuLogContextFilter: Read and filter log for relevant messages
+ * @ctxt: the domain log context
+ * @msg: pointer to buffer to store the read messages in
+ * @max: maximum length of the message returned in @msg after filtering
+ *
+ * Reads log output from @ctxt and filters it. Skips messages not produced by
+ * the target executable or irrelevant messages. If @max is not zero, @buf will
+ * contain at most @max characters from the end of the log and @buf will start
+ * after a new line if possible.
+ */
+int
+qemuLogContextReadFiltered(qemuLogContext *ctxt,
+   char **msg,
+   size_t max)
+{
+char *buf;
+char *eol;
+char *filter_next;
+size_t skip;
+ssize_t got;
+
+if ((got = qemuLogContextRead(ctxt, )) < 0)
+return -1;
+
+/* Filter out debug messages from intermediate libvirt process */
+filter_next = buf;
+while ((eol = strchr(filter_next, '\n'))) {
+*eol = '\0';
+if (virLogProbablyLogMessage(filter_next) ||
+strstr(filter_next, "char device redirected to")) {
+skip = (eol + 1) - filter_next;
+memmove(filter_next, eol + 1, buf + got - eol);
+got -= skip;
+} else {
+filter_next = eol + 1;
+*eol = '\n';
+}
+}
+
+if (got > 0 &&
+buf[got - 1] == '\n') {
+buf[got - 1] = '\0';
+got--;
+}
+
+if (max > 0 && got > max) {
+skip = got - max;
+
+if (buf[skip - 1] != '\n' &&
+(eol = strchr(buf + skip, '\n')) &&
+!virStringIsEmpty(eol + 1))
+skip = eol + 1 - buf;
+
+memmove(buf, buf + skip, got - skip + 1);
+got -= skip;
+}
+
+buf = g_renew(char, buf, got + 1);
+*msg = buf;
+return 0;
+}
+
+
 int
 qemuLogContextGetWriteFD(qemuLogContext *ctxt)
 {
diff --git a/src/qemu/qemu_logcontext.h b/src/qemu/qemu_logcontext.h
index 6ad60b7b4a..738e908bc3 100644
--- a/src/qemu/qemu_logcontext.h
+++ b/src/qemu/qemu_logcontext.h
@@ -32,6 +32,9 @@ int qemuLogContextWrite(qemuLogContext *ctxt,
 const char *fmt, ...) G_GNUC_PRINTF(2, 3);
 ssize_t qemuLogContextRead(qemuLogContext *ctxt,
char **msg);
+int qemuLogContextReadFiltered(qemuLogContext *ctxt,
+   char **msg,
+   size_t max);
 int qemuLogContextGetWriteFD(qemuLogContext *ctxt);
 void qemuLogContextMarkPosition(qemuLogContext *ctxt);
 
diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c
index e03d0d8b4d..a77d2ba7de 100644
--- a/src/qemu/qemu_process.c
+++ b/src/qemu/qemu_process.c
@@ -1908,71 +1908,6 @@ qemuConnectMonitor(virQEMUDriver *driver,
 }
 
 
-/**
- * qemuProcessReadLog: Read log file of a qemu VM
- * @logCtxt: the domain log context
- * @msg: pointer to buffer to store the read messages in
- * @max: maximum length of the message returned in @msg
- *
- * Reads log of a qemu VM. Skips messages not produced by qemu or irrelevant
- * messages. If @max is not zero, @msg will contain at most @max characters
- * from the end of the log and @msg will start after a new line if possible.
- *
- * Returns 0 on success or -1 on error
- */
-static int
-qemuProcessReadLog(qemuLogContext *logCtxt,
-   char **msg,
-   size_t max)
-{
-char *buf;
-ssize_t got;
-char *eol;
-char *filter_next;
-size_t skip;
-
-if ((got = qemuLogContextRead(logCtxt, )) < 0)
-return -1;
-
-/* Filter out debug messages from intermediate libvirt process */
-filter_next = buf;
-while ((eol = strchr(filter_next, '\n'))) {
-*eol = '\0';
-if (virLogProbablyLogMessage(filter_next) ||
-strstr(filter_next, "char device redirected to")) {
-skip = (eol + 1) - filter_next;
-memmove(filter_next, eol + 1, buf + got - eol);
-got -= skip;
-} else {
-filter_next = eol + 1;
-*eol = '\n';
-}
-}
-
-if (got > 0 &&
-buf[got 

[libvirt PATCH v7 16/35] qemu: split qemuDomainSecretStorageSourcePrepare

[Jonathon Jongsma]

This prepares encryption secrets and authentication secrets. When we add
nbdkit-backed network storage sources, we will not need to send
authentication secrets to qemu, since they will be sent to nbdkit
instead. So split this into two different functions.

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 src/qemu/qemu_domain.c | 103 +
 1 file changed, 62 insertions(+), 41 deletions(-)

diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c
index 23608f95bd..951f3127d9 100644
--- a/src/qemu/qemu_domain.c
+++ b/src/qemu/qemu_domain.c
@@ -1398,38 +1398,70 @@ 
qemuDomainSecretStorageSourcePrepareCookies(qemuDomainObjPrivate *priv,
 
 
 /**
- * qemuDomainSecretStorageSourcePrepare:
+ * qemuDomainSecretStorageSourcePrepareEncryption:
  * @priv: domain private object
  * @src: storage source struct to setup
- * @authalias: prefix of the alias for secret holding authentication data
- * @encalias: prefix of the alias for secret holding encryption password
+ * @alias: prefix of the alias for secret holding encryption password
  *
- * Prepares data necessary for encryption and authentication of @src. The two
- * alias prefixes are provided since in the backing chain authentication 
belongs
- * to the storage protocol data whereas encryption is relevant to the format
- * driver in qemu. The two will have different node names.
+ * Prepares data necessary for encryption of @src.
  *
  * Returns 0 on success; -1 on error while reporting an libvirt error.
  */
 static int
-qemuDomainSecretStorageSourcePrepare(qemuDomainObjPrivate *priv,
- virStorageSource *src,
- const char *aliasprotocol,
- const char *aliasformat)
+qemuDomainSecretStorageSourcePrepareEncryption(qemuDomainObjPrivate *priv,
+   virStorageSource *src,
+   const char *alias)
 {
 qemuDomainStorageSourcePrivate *srcPriv;
-bool hasEnc = src->encryption && src->encryption->nsecrets > 0;
+size_t nsecrets = 0;
+size_t i;
 
-if (virStorageSourceIsEmpty(src))
+if (!(src->encryption && src->encryption->nsecrets > 0))
 return 0;
 
-if (!src->auth && !hasEnc && src->ncookies == 0)
+if (virStorageSourceIsEmpty(src))
 return 0;
 
-if (!(src->privateData = qemuDomainStorageSourcePrivateNew()))
-return -1;
+nsecrets = src->encryption->nsecrets;
+
+srcPriv = qemuDomainStorageSourcePrivateFetch(src);
+
+srcPriv->enccount = nsecrets;
+srcPriv->encinfo = g_new0(qemuDomainSecretInfo *, nsecrets);
+for (i = 0; i < nsecrets; ++i) {
+if (!(srcPriv->encinfo[i] = qemuDomainSecretInfoSetupFromSecret(priv, 
alias,
+
"encryption", i,
+
VIR_SECRET_USAGE_TYPE_VOLUME,
+NULL,
+
>encryption->secrets[i]->seclookupdef)))
+return -1;
+}
+
+return 0;
+}
+
+
+/**
+ * qemuDomainSecretStorageSourcePrepareAuth:
+ * @priv: domain private object
+ * @src: storage source struct to setup
+ * @alias: prefix of the alias for secret holding authentication data
+ *
+ * Prepares data necessary for authentication of @src.
+ *
+ * Returns 0 on success; -1 on error while reporting an libvirt error.
+ */
+static int
+qemuDomainSecretStorageSourcePrepareAuth(qemuDomainObjPrivate *priv,
+ virStorageSource *src,
+ const char *alias)
+{
+qemuDomainStorageSourcePrivate *srcPriv;
+
+if (virStorageSourceIsEmpty(src))
+return 0;
 
-srcPriv = QEMU_DOMAIN_STORAGE_SOURCE_PRIVATE(src);
+srcPriv = qemuDomainStorageSourcePrivateFetch(src);
 
 if (src->auth) {
 virSecretUsageType usageType = VIR_SECRET_USAGE_TYPE_ISCSI;
@@ -1437,7 +1469,7 @@ qemuDomainSecretStorageSourcePrepare(qemuDomainObjPrivate 
*priv,
 if (src->protocol == VIR_STORAGE_NET_PROTOCOL_RBD)
 usageType = VIR_SECRET_USAGE_TYPE_CEPH;
 
-if (!(srcPriv->secinfo = qemuDomainSecretInfoSetupFromSecret(priv, 
aliasprotocol,
+if (!(srcPriv->secinfo = qemuDomainSecretInfoSetupFromSecret(priv, 
alias,
  "auth", 0,
  usageType,
  
src->auth->username,
@@ -1445,26 +1477,10 @@ 
qemuDomainSecretStorageSourcePrepare(qemuDomainObjPrivate *priv,
 return -1;
 }
 
-if (hasEnc) {
-size_t nsecrets = src->encryption->nsecrets;
-size_t i;
-
-

[libvirt PATCH v7 30/35] qemu: implement knownHosts for ssh disks with nbdkit

[Jonathon Jongsma]

For ssh disks that are served by nbdkit, use the configured value for
knownHosts and pass it to the nbdkit process.

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 src/conf/domain_conf.c|  8 ++
 src/conf/storage_source_conf.c|  2 ++
 src/conf/storage_source_conf.h|  2 ++
 src/qemu/qemu_extdevice.c |  4 +--
 src/qemu/qemu_hotplug.c   |  4 +--
 src/qemu/qemu_nbdkit.c| 25 +++
 src/qemu/qemu_nbdkit.h|  6 +++--
 .../disk-network-ssh-password.args.disk0  |  3 ++-
 .../disk-network-ssh.args.disk0   |  3 ++-
 .../disk-network-ssh-password.xml |  1 +
 tests/qemuxml2argvdata/disk-network-ssh.xml   |  1 +
 11 files changed, 46 insertions(+), 13 deletions(-)

diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c
index 8feaf5d055..842b6404b5 100644
--- a/src/conf/domain_conf.c
+++ b/src/conf/domain_conf.c
@@ -7268,6 +7268,11 @@ virDomainDiskSourceNetworkParse(xmlNodePtr node,
 return -1;
 }
 }
+if (src->protocol == VIR_STORAGE_NET_PROTOCOL_SSH &&
+(tmpnode = virXPathNode("./knownHosts", ctxt))) {
+if (!(src->ssh_known_hosts_file = virXMLPropStringRequired(tmpnode, 
"path")))
+return -1;
+}
 
 return 0;
 }
@@ -22274,6 +22279,9 @@ virDomainDiskSourceFormatNetwork(virBuffer *attrBuf,
 
 if (src->timeout)
 virBufferAsprintf(childBuf, "\n", 
src->timeout);
+
+if (src->protocol == VIR_STORAGE_NET_PROTOCOL_SSH && 
src->ssh_known_hosts_file)
+virBufferEscapeString(childBuf, "\n", 
src->ssh_known_hosts_file);
 }
 
 
diff --git a/src/conf/storage_source_conf.c b/src/conf/storage_source_conf.c
index dcac3a8ff6..906bc36a9b 100644
--- a/src/conf/storage_source_conf.c
+++ b/src/conf/storage_source_conf.c
@@ -895,6 +895,7 @@ virStorageSourceCopy(const virStorageSource *src,
 /* ssh config passthrough for libguestfs */
 def->ssh_host_key_check_disabled = src->ssh_host_key_check_disabled;
 def->ssh_user = g_strdup(src->ssh_user);
+def->ssh_known_hosts_file = g_strdup(src->ssh_known_hosts_file);
 
 def->nfs_user = g_strdup(src->nfs_user);
 def->nfs_group = g_strdup(src->nfs_group);
@@ -1170,6 +1171,7 @@ virStorageSourceClear(virStorageSource *def)
 VIR_FREE(def->tlsHostname);
 
 VIR_FREE(def->ssh_user);
+VIR_FREE(def->ssh_known_hosts_file);
 
 VIR_FREE(def->nfs_user);
 VIR_FREE(def->nfs_group);
diff --git a/src/conf/storage_source_conf.h b/src/conf/storage_source_conf.h
index f13e7c756a..8a9c7d07e2 100644
--- a/src/conf/storage_source_conf.h
+++ b/src/conf/storage_source_conf.h
@@ -410,6 +410,8 @@ struct _virStorageSource {
 /* these must not be used apart from formatting the output JSON in the 
qemu driver */
 char *ssh_user;
 bool ssh_host_key_check_disabled;
+/* additional ssh variables */
+char *ssh_known_hosts_file;
 
 /* nfs_user and nfs_group store the strings passed in by the user for NFS 
params.
  * nfs_uid and nfs_gid represent the converted/looked up ID numbers which 
are used
diff --git a/src/qemu/qemu_extdevice.c b/src/qemu/qemu_extdevice.c
index 42ecdf13d5..3cf3867056 100644
--- a/src/qemu/qemu_extdevice.c
+++ b/src/qemu/qemu_extdevice.c
@@ -297,11 +297,11 @@ qemuExtDevicesStop(virQEMUDriver *driver,
 
 for (i = 0; i < def->ndisks; i++) {
 virDomainDiskDef *disk = def->disks[i];
-qemuNbdkitStopStorageSource(disk->src);
+qemuNbdkitStopStorageSource(disk->src, vm);
 }
 
 if (def->os.loader && def->os.loader->nvram)
-qemuNbdkitStopStorageSource(def->os.loader->nvram);
+qemuNbdkitStopStorageSource(def->os.loader->nvram, vm);
 }
 
 
diff --git a/src/qemu/qemu_hotplug.c b/src/qemu/qemu_hotplug.c
index dc06486922..bbc5177206 100644
--- a/src/qemu/qemu_hotplug.c
+++ b/src/qemu/qemu_hotplug.c
@@ -1036,7 +1036,7 @@ qemuDomainAttachDeviceDiskLiveInternal(virQEMUDriver 
*driver,
 if (virStorageSourceChainHasManagedPR(disk->src))
 ignore_value(qemuHotplugRemoveManagedPR(vm, VIR_ASYNC_JOB_NONE));
 
-qemuNbdkitStopStorageSource(disk->src);
+qemuNbdkitStopStorageSource(disk->src, vm);
 }
 qemuDomainSecretDiskDestroy(disk);
 qemuDomainCleanupStorageSourceFD(disk->src);
@@ -4496,7 +4496,7 @@ qemuDomainRemoveDiskDevice(virQEMUDriver *driver,
 qemuHotplugRemoveManagedPR(vm, VIR_ASYNC_JOB_NONE) < 0)
 goto cleanup;
 
-qemuNbdkitStopStorageSource(disk->src);
+qemuNbdkitStopStorageSource(disk->src, vm);
 
 if (disk->transient) {
 VIR_DEBUG("Removing transient overlay '%s' of disk '%s'",
diff --git a/src/qemu/qemu_nbdkit.c b/src/qemu/qemu_nbdkit.c
index cedeaa880b..d0eca1ab89 100644
--- a/src/qemu/qemu_nbdkit.c
+++ b/src/qemu/qemu_nbdkit.c
@@ -624,7 +624,7 @@ qemuNbdkitProcessRestart(qemuNbdkitProcess *proc,
 virQEMUDriver *driver = 

[libvirt PATCH v7 26/35] qemu: try to connect to nbdkit early to detect errors

[Jonathon Jongsma]

When using nbdkit to serve a network disk source, the nbdkit process
will start and wait for an nbd connection before actually attempting to
connect to the (remote) disk location. Because of this, nbdkit will not
report an error until after qemu is launched and tries to read from the
disk. This results in a fairly user-unfriendly error saying that qemu
was unable to start because "Requested export not available".

Ideally we'd like to be able to tell the user *why* the export is not
available, but this sort of information is only available to nbdkit, not
qemu. It could be because the url was incorrect, or because of an
authentication failure, or one of many other possibilities.

To make this friendlier for users and easier to detect
misconfigurations, try to connect to nbdkit immediately after starting
nbdkit and before we try to start qemu. This requires adding a
dependency on libnbd. If an error occurs when connecting to nbdkit, read
back from the nbdkit error log and provide that information in the error
report from qemuNbdkitProcessStart().

User-visible change demonstrated below:
Previous error:

$ virsh start nbdkit-test
2023-01-18 19:47:45.778+: 30895: error : 
virNetClientProgramDispatchError:172 : internal
error: process exited while connecting to monitor: 
2023-01-18T19:47:45.704658Z
qemu-system-x86_64: -blockdev {"driver":"nbd","server":{"type":"unix",

"path":"/var/lib/libvirt/qemu/domain-1-nbdkit-test/nbdkit-libvirt-1-storage.socket"},
"node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}: 
Requested export not
available
error: Failed to start domain 'nbdkit-test'
error: internal error: process exited while connecting to monitor: 
2023-01-18T19:47:45.704658Z
qemu-system-x86_64: -blockdev {"driver":"nbd","server":{"type":"unix",

"path":"/var/lib/libvirt/qemu/domain-1-nbdkit-test/nbdkit-libvirt-1-storage.socket"},
"node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}: 
Requested export not
available

After this change:

$ virsh start nbdkit-test
2023-01-18 19:44:36.242+: 30895: error : 
virNetClientProgramDispatchError:172 : internal
error: Failed to connect to nbdkit for 
'http://localhost:/nonexistent.iso': nbdkit: curl[1]:
error: problem doing HEAD request to fetch size of URL 
[http://localhost:/nonexistent.iso]:
HTTP response code said error: The requested URL returned error: 404
error: Failed to start domain 'nbdkit-test'
error: internal error: Failed to connect to nbdkit for 
'http://localhost:/nonexistent.iso]:
error: problem doing HEAD request to fetch size of URL 
[http://localhost:/nonexistent.iso]:
HTTP response code said error: The requested URL returned error: 404

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 meson.build|  7 +++
 meson_options.txt  |  1 +
 src/qemu/meson.build   |  1 +
 src/qemu/qemu_nbdkit.c | 23 +++
 4 files changed, 32 insertions(+)

diff --git a/meson.build b/meson.build
index 8da3b59070..a5ab94c0ba 100644
--- a/meson.build
+++ b/meson.build
@@ -1002,6 +1002,12 @@ endif
 libiscsi_version = '1.18.0'
 libiscsi_dep = dependency('libiscsi', version: '>=' + libiscsi_version, 
required: get_option('libiscsi'))
 
+libnbd_version = '1.0'
+libnbd_dep = dependency('libnbd', version: '>=' + libnbd_version, required: 
get_option('libnbd'))
+if libnbd_dep.found()
+  conf.set('WITH_LIBNBD', 1)
+endif
+
 libnl_version = '3.0'
 if not get_option('libnl').disabled() and host_machine.system() == 'linux'
   libnl_dep = dependency('libnl-3.0', version: '>=' + libnl_version, required: 
get_option('libnl'))
@@ -2216,6 +,7 @@ libs_summary = {
   'glusterfs': glusterfs_dep.found(),
   'libiscsi': libiscsi_dep.found(),
   'libkvm': libkvm_dep.found(),
+  'libnbd': libnbd_dep.found(),
   'libnl': libnl_dep.found(),
   'libparted': libparted_dep.found(),
   'libpcap': libpcap_dep.found(),
diff --git a/meson_options.txt b/meson_options.txt
index 9174c4021c..ba6e49afc5 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -25,6 +25,7 @@ option('curl', type: 'feature', value: 'auto', description: 
'curl support')
 option('fuse', type: 'feature', value: 'auto', description: 'fuse support')
 option('glusterfs', type: 'feature', value: 'auto', description: 'glusterfs 
support')
 option('libiscsi', type: 'feature', value: 'auto', description: 'libiscsi 
support')
+option('libnbd', type: 'feature', value: 'auto', description: 'libnbd support')
 option('libnl', type: 'feature', value: 'auto', description: 'libnl support')
 option('libpcap', type: 'feature', value: 'auto', description: 'libpcap 
support')
 option('libssh', type: 'feature', value: 'auto', description: 'libssh support')
diff --git a/src/qemu/meson.build b/src/qemu/meson.build
index 6d7a1bfbb0..607b597c8c 100644
--- a/src/qemu/meson.build
+++ b/src/qemu/meson.build
@@ -99,6 +99,7 @@ if conf.has('WITH_QEMU')
   

[libvirt PATCH v7 07/35] qemu: use file cache for nbdkit caps

[Jonathon Jongsma]

Add the virFileCache implementation for nbdkit capabilities to the qemu
driver. This allows us to determine whether nbdkit is installed and
which plugins are supported. it also has persistent caching and the
capabilities are re-queried whenever something changes.

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 src/qemu/qemu_conf.h   | 3 +++
 src/qemu/qemu_driver.c | 3 +++
 2 files changed, 6 insertions(+)

diff --git a/src/qemu/qemu_conf.h b/src/qemu/qemu_conf.h
index 4f610d86a1..a44985fb8b 100644
--- a/src/qemu/qemu_conf.h
+++ b/src/qemu/qemu_conf.h
@@ -318,6 +318,9 @@ struct _virQEMUDriver {
 
 /* Immutable pointer, self-locking APIs */
 virHashAtomic *migrationErrors;
+
+/* Immutable pointer, self-locking APIs */
+virFileCache *nbdkitCapsCache;
 };
 
 virQEMUDriverConfig *virQEMUDriverConfigNew(bool privileged,
diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c
index 5db42f0753..ad8428948b 100644
--- a/src/qemu/qemu_driver.c
+++ b/src/qemu/qemu_driver.c
@@ -845,6 +845,8 @@ qemuStateInitialize(bool privileged,
defsecmodel)))
 goto error;
 
+qemu_driver->nbdkitCapsCache = qemuNbdkitCapsCacheNew(cfg->cacheDir);
+
 /* If hugetlbfs is present, then we need to create a sub-directory within
  * it, since we can't assume the root mount point has permissions that
  * will let our spawned QEMU instances use it. */
@@ -1078,6 +1080,7 @@ qemuStateCleanup(void)
 ebtablesContextFree(qemu_driver->ebtables);
 VIR_FREE(qemu_driver->qemuImgBinary);
 virObjectUnref(qemu_driver->domains);
+virObjectUnref(qemu_driver->nbdkitCapsCache);
 
 if (qemu_driver->lockFD != -1)
 virPidFileRelease(qemu_driver->config->stateDir, "driver", 
qemu_driver->lockFD);
-- 
2.41.0

[libvirt PATCH v7 29/35] schema: add configuration for host verification of ssh disks

[Jonathon Jongsma]

In order to make ssh disks usable, we need to be able to validate a
remote host. To do this, add a  xml element for ssh disks to
allow the user to specify a location for a file that contains known host
keys. Implementation to follow.

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 docs/formatdomain.rst |  8 
 src/conf/schemas/domaincommon.rng | 11 +++
 2 files changed, 19 insertions(+)

diff --git a/docs/formatdomain.rst b/docs/formatdomain.rst
index 39d4230ec0..9230f540f4 100644
--- a/docs/formatdomain.rst
+++ b/docs/formatdomain.rst
@@ -3021,6 +3021,14 @@ paravirtualized driver is specified via the ``disk`` 
element.
  paused and will be rerun after a successful reconnect. After that 
time, any
  delayed requests and all future requests before a successful reconnect
  will immediately fail. If not set the default QEMU value is 0.
+   ``knownHosts``
+  For storage accessed via the ``ssh`` protocol, this element configures a
+  path to a file that will be used to verify the remote host. This file
+  must contain the expected host key for the remote host or the connection
+  will fail. The location of the file is specified via the ``path``
+  attribute.
+  :since:`Since 9.6.0`
+
 
For a "file" or "volume" disk type which represents a cdrom or floppy (the
``device`` attribute), it is possible to define policy what to do with the
diff --git a/src/conf/schemas/domaincommon.rng 
b/src/conf/schemas/domaincommon.rng
index cd838a475c..ca43586323 100644
--- a/src/conf/schemas/domaincommon.rng
+++ b/src/conf/schemas/domaincommon.rng
@@ -2172,6 +2172,14 @@
 
   
 
+  
+
+  
+
+  
+
+  
+
   
 
   
@@ -2187,6 +2195,9 @@
   
 
 
+
+  
+
 
   
 
-- 
2.41.0

[libvirt PATCH v7 10/35] qemu: add functions to start and stop nbdkit

[Jonathon Jongsma]

Add some helper functions to build a virCommand object and run the
nbdkit process for a given virStorageSource.

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 src/qemu/qemu_nbdkit.c | 250 +
 src/qemu/qemu_nbdkit.h |  10 ++
 2 files changed, 260 insertions(+)

diff --git a/src/qemu/qemu_nbdkit.c b/src/qemu/qemu_nbdkit.c
index 9a2a89224d..6bf962d0f1 100644
--- a/src/qemu/qemu_nbdkit.c
+++ b/src/qemu/qemu_nbdkit.c
@@ -24,6 +24,8 @@
 #include "virerror.h"
 #include "virlog.h"
 #include "virpidfile.h"
+#include "virsecureerase.h"
+#include "virtime.h"
 #include "virutil.h"
 #include "qemu_block.h"
 #include "qemu_conf.h"
@@ -666,6 +668,168 @@ qemuNbdkitInitStorageSource(qemuNbdkitCaps *caps,
 }
 
 
+static int
+qemuNbdkitProcessBuildCommandCurl(qemuNbdkitProcess *proc,
+  virCommand *cmd)
+{
+g_autoptr(virURI) uri = qemuBlockStorageSourceGetURI(proc->source);
+g_autofree char *uristring = virURIFormat(uri);
+
+/* nbdkit plugin name */
+virCommandAddArg(cmd, "curl");
+if (proc->source->protocol == VIR_STORAGE_NET_PROTOCOL_HTTP) {
+/* allow http to be upgraded to https via e.g. redirect */
+virCommandAddArgPair(cmd, "protocols", "http,https");
+} else {
+virCommandAddArgPair(cmd, "protocols",
+ 
virStorageNetProtocolTypeToString(proc->source->protocol));
+}
+virCommandAddArgPair(cmd, "url", uristring);
+
+if (proc->source->auth) {
+g_autoptr(virConnect) conn = virGetConnectSecret();
+g_autofree uint8_t *secret = NULL;
+size_t secretlen = 0;
+g_autofree char *password = NULL;
+int secrettype;
+virStorageAuthDef *authdef = proc->source->auth;
+
+virCommandAddArgPair(cmd, "user",
+ proc->source->auth->username);
+
+if ((secrettype = 
virSecretUsageTypeFromString(proc->source->auth->secrettype)) < 0) {
+virReportError(VIR_ERR_CONFIG_UNSUPPORTED,
+   _("invalid secret type %1$s"),
+   proc->source->auth->secrettype);
+return -1;
+}
+
+if (virSecretGetSecretString(conn,
+ >seclookupdef,
+ secrettype,
+ ,
+ ) < 0) {
+virReportError(VIR_ERR_INTERNAL_ERROR, "%s",
+   _("failed to get auth secret for storage"));
+return -1;
+}
+
+/* ensure that the secret is a NULL-terminated string */
+password = g_strndup((char*)secret, secretlen);
+virSecureErase(secret, secretlen);
+
+/* for now, just report an error rather than passing the password in
+ * cleartext on the commandline */
+virReportError(VIR_ERR_INTERNAL_ERROR, "%s",
+   _("Password not yet supported for nbdkit sources"));
+
+virSecureEraseString(password);
+
+return -1;
+}
+
+if (proc->source->ncookies > 0) {
+/* for now, just report an error rather than passing cookies in
+ * cleartext on the commandline */
+virReportError(VIR_ERR_INTERNAL_ERROR, "%s",
+   _("Cookies not yet supported for nbdkit sources"));
+return -1;
+}
+
+if (proc->source->sslverify == VIR_TRISTATE_BOOL_NO) {
+virCommandAddArgPair(cmd, "sslverify", "false");
+}
+
+if (proc->source->timeout > 0) {
+g_autofree char *timeout = g_strdup_printf("%llu", 
proc->source->timeout);
+virCommandAddArgPair(cmd, "timeout", timeout);
+}
+
+return 0;
+}
+
+
+static int
+qemuNbdkitProcessBuildCommandSSH(qemuNbdkitProcess *proc,
+ virCommand *cmd)
+{
+const char *user = NULL;
+virStorageNetHostDef *host = >source->hosts[0];
+g_autofree char *portstr = g_strdup_printf("%u", host->port);
+
+/* nbdkit plugin name */
+virCommandAddArg(cmd, "ssh");
+
+virCommandAddArgPair(cmd, "host", host->name);
+virCommandAddArgPair(cmd, "port", portstr);
+virCommandAddArgPair(cmd, "path", proc->source->path);
+
+if (proc->source->auth)
+user = proc->source->auth->username;
+else if (proc->source->ssh_user)
+user = proc->source->ssh_user;
+
+if (user)
+virCommandAddArgPair(cmd, "user", user);
+
+if (proc->source->ssh_host_key_check_disabled)
+virCommandAddArgPair(cmd, "verify-remote-host", "false");
+
+return 0;
+}
+
+
+static virCommand *
+qemuNbdkitProcessBuildCommand(qemuNbdkitProcess *proc)
+{
+g_autoptr(virCommand) cmd = virCommandNewArgList(proc->caps->path,
+ "--unix",
+ proc->socketfile,
+ "--foreground",
+ 

[libvirt PATCH v7 15/35] tests: add ability to test various nbdkit capabilities

[Jonathon Jongsma]

Add new DO_TEST_CAPS_LATEST_NBDKIT macro to test xml2argv for various
nbdkit capability scenarios.

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 src/qemu/qemu_nbdkit.c   | 20 +---
 tests/qemuxml2argvtest.c | 11 +++
 tests/testutilsqemu.c| 26 ++
 tests/testutilsqemu.h|  4 
 4 files changed, 58 insertions(+), 3 deletions(-)

diff --git a/src/qemu/qemu_nbdkit.c b/src/qemu/qemu_nbdkit.c
index 2d70e72c42..81861bae4a 100644
--- a/src/qemu/qemu_nbdkit.c
+++ b/src/qemu/qemu_nbdkit.c
@@ -290,10 +290,16 @@ virNbkditCapsCheckModdir(const char *moddir,
 
 static bool
 virNbdkitCapsIsValid(void *data,
- void *privData G_GNUC_UNUSED)
+ void *privData)
 {
 qemuNbdkitCaps *nbdkitCaps = data;
 struct stat st;
+/* when run under test, we will use privData as a signal to indicate that
+ * we shouldn't touch the filesystem */
+bool skipValidation = (privData != NULL);
+
+if (skipValidation)
+return true;
 
 if (!nbdkitCaps->path)
 return true;
@@ -334,9 +340,17 @@ virNbdkitCapsIsValid(void *data,
 
 static void*
 virNbdkitCapsNewData(const char *binary,
- void *privData G_GNUC_UNUSED)
+ void *privData)
 {
-qemuNbdkitCaps *caps = qemuNbdkitCapsNew(binary);
+/* when run under test, we will use privData as a signal to indicate that
+ * we shouldn't touch the filesystem */
+bool skipNewData = (privData != NULL);
+qemuNbdkitCaps *caps = NULL;
+
+if (skipNewData)
+return NULL;
+
+caps = qemuNbdkitCapsNew(binary);
 qemuNbdkitCapsQuery(caps);
 
 return caps;
diff --git a/tests/qemuxml2argvtest.c b/tests/qemuxml2argvtest.c
index 1b76b32812..d64c21ae17 100644
--- a/tests/qemuxml2argvtest.c
+++ b/tests/qemuxml2argvtest.c
@@ -611,6 +611,14 @@ testCompareXMLToArgv(const void *data)
 if (qemuTestCapsCacheInsert(driver.qemuCapsCache, info->qemuCaps) < 0)
 goto cleanup;
 
+if (info->nbdkitCaps) {
+if (virFileCacheInsertData(driver.nbdkitCapsCache, TEST_NBDKIT_PATH,
+   g_object_ref(info->nbdkitCaps)) < 0) {
+g_object_unref(info->nbdkitCaps);
+goto cleanup;
+}
+}
+
 if (info->migrateFrom &&
 !(migrateURI = qemuMigrationDstGetURI(info->migrateFrom,
   info->migrateFd)))
@@ -831,6 +839,9 @@ mymain(void)
 # define DO_TEST_CAPS_ARCH_VER(name, arch, ver) \
 DO_TEST_CAPS_ARCH_VER_FULL(name, arch, ver, ARG_END)
 
+# define DO_TEST_CAPS_LATEST_NBDKIT(name, ...) \
+DO_TEST_CAPS_ARCH_LATEST_FULL(name, "x86_64", ARG_NBDKIT_CAPS, 
__VA_ARGS__, QEMU_NBDKIT_CAPS_LAST, ARG_END)
+
 # define DO_TEST_CAPS_LATEST(name) \
 DO_TEST_CAPS_ARCH_LATEST(name, "x86_64")
 
diff --git a/tests/testutilsqemu.c b/tests/testutilsqemu.c
index fdbad16abe..9a607ab5a3 100644
--- a/tests/testutilsqemu.c
+++ b/tests/testutilsqemu.c
@@ -50,6 +50,10 @@ virFindFileInPath(const char *file)
 return g_strdup_printf("/usr/bin/%s", file);
 }
 
+if (g_str_equal(file, "nbdkit")) {
+return g_strdup(TEST_NBDKIT_PATH);
+}
+
 /* Nothing in tests should be relying on real files
  * in host OS, so we return NULL to try to force
  * an error in such a case
@@ -288,6 +292,7 @@ void qemuTestDriverFree(virQEMUDriver *driver)
 virObjectUnref(driver->caps);
 virObjectUnref(driver->config);
 virObjectUnref(driver->securityManager);
+g_clear_object(>nbdkitCapsCache);
 
 virCPUDefFree(cpuDefault);
 virCPUDefFree(cpuHaswell);
@@ -487,6 +492,12 @@ int qemuTestDriverInit(virQEMUDriver *driver)
 if (!driver->qemuCapsCache)
 goto error;
 
+driver->nbdkitCapsCache = qemuNbdkitCapsCacheNew("/dev/null");
+/* the nbdkitCapsCache just interprets the presence of a non-null private
+ * data pointer as a signal to skip cache validation. This prevents the
+ * cache from trying to validate the plugindir mtime, etc during test */
+virFileCacheSetPriv(driver->nbdkitCapsCache, GUINT_TO_POINTER(1));
+
 driver->xmlopt = virQEMUDriverCreateXMLConf(driver, "none");
 if (!driver->xmlopt)
 goto error;
@@ -780,6 +791,14 @@ testQemuInfoSetArgs(struct testQemuInfo *info,
 ignore_value(virBitmapSetBit(info->args.fakeCapsDel, flag));
 break;
 
+case ARG_NBDKIT_CAPS:
+if (!(info->args.fakeNbdkitCaps))
+info->args.fakeNbdkitCaps = 
virBitmapNew(QEMU_NBDKIT_CAPS_LAST);
+
+while ((flag = va_arg(argptr, int)) < QEMU_NBDKIT_CAPS_LAST)
+ignore_value(virBitmapSetBit(info->args.fakeNbdkitCaps, flag));
+break;
+
 case ARG_GIC:
 info->args.gic = va_arg(argptr, int);
 break;
@@ -1054,6 +1073,11 @@ testQemuInfoInitArgs(struct testQemuInfo *info)
 for (cap = -1; (cap = 

[libvirt PATCH v7 11/35] Generalize qemuDomainLogContextNew()

[Jonathon Jongsma]

Allow to specify a basename for the log file so that
qemuDomainLogContextNew() can be used to create log contexts for
secondary loggers.

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 src/qemu/qemu_domain.c  | 5 +++--
 src/qemu/qemu_domain.h  | 3 ++-
 src/qemu/qemu_process.c | 2 +-
 3 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c
index 5a2eb4868a..d79f9879df 100644
--- a/src/qemu/qemu_domain.c
+++ b/src/qemu/qemu_domain.c
@@ -7119,7 +7119,8 @@ void qemuDomainObjCheckNetTaint(virQEMUDriver *driver,
 
 
 qemuDomainLogContext *qemuDomainLogContextNew(virQEMUDriver *driver,
-  virDomainObj *vm)
+  virDomainObj *vm,
+  const char *basename)
 {
 g_autoptr(virQEMUDriverConfig) cfg = virQEMUDriverGetConfig(driver);
 qemuDomainLogContext *ctxt = 
QEMU_DOMAIN_LOG_CONTEXT(g_object_new(QEMU_TYPE_DOMAIN_LOG_CONTEXT, NULL));
@@ -7128,7 +7129,7 @@ qemuDomainLogContext 
*qemuDomainLogContextNew(virQEMUDriver *driver,
 ctxt->writefd = -1;
 ctxt->readfd = -1;
 
-ctxt->path = g_strdup_printf("%s/%s.log", cfg->logDir, vm->def->name);
+ctxt->path = g_strdup_printf("%s/%s.log", cfg->logDir, basename);
 
 if (cfg->stdioLogD) {
 ctxt->manager = virLogManagerNew(driver->privileged);
diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h
index 89edc75fcf..a262555c8c 100644
--- a/src/qemu/qemu_domain.h
+++ b/src/qemu/qemu_domain.h
@@ -657,7 +657,8 @@ void qemuDomainObjCheckNetTaint(virQEMUDriver *driver,
 qemuDomainLogContext *logCtxt);
 
 qemuDomainLogContext *qemuDomainLogContextNew(virQEMUDriver *driver,
-  virDomainObj *vm);
+  virDomainObj *vm,
+  const char *basename);
 int qemuDomainLogContextWrite(qemuDomainLogContext *ctxt,
   const char *fmt, ...) G_GNUC_PRINTF(2, 3);
 ssize_t qemuDomainLogContextRead(qemuDomainLogContext *ctxt,
diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c
index a6ed69cfe2..e0385d11be 100644
--- a/src/qemu/qemu_process.c
+++ b/src/qemu/qemu_process.c
@@ -7616,7 +7616,7 @@ qemuProcessLaunch(virConnectPtr conn,
 hookData.cfg = cfg;
 
 VIR_DEBUG("Creating domain log file");
-if (!(logCtxt = qemuDomainLogContextNew(driver, vm))) {
+if (!(logCtxt = qemuDomainLogContextNew(driver, vm, vm->def->name))) {
 virLastErrorPrefixMessage("%s", _("can't connect to virtlogd"));
 goto cleanup;
 }
-- 
2.41.0

[libvirt PATCH v7 12/35] qemu: Extract qemuDomainLogContext into a new file

[Jonathon Jongsma]

This will allow us to use it for nbdkit logging in upcoming commits.

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 po/POTFILES|   1 +
 src/qemu/meson.build   |   1 +
 src/qemu/qemu_domain.c | 247 ++
 src/qemu/qemu_domain.h |  27 +---
 src/qemu/qemu_logcontext.c | 264 +
 src/qemu/qemu_logcontext.h |  38 ++
 src/qemu/qemu_process.c|  44 +++
 7 files changed, 346 insertions(+), 276 deletions(-)
 create mode 100644 src/qemu/qemu_logcontext.c
 create mode 100644 src/qemu/qemu_logcontext.h

diff --git a/po/POTFILES b/po/POTFILES
index 6167f98ac5..3a51aea5cb 100644
--- a/po/POTFILES
+++ b/po/POTFILES
@@ -174,6 +174,7 @@ src/qemu/qemu_hostdev.c
 src/qemu/qemu_hotplug.c
 src/qemu/qemu_interface.c
 src/qemu/qemu_interop_config.c
+src/qemu/qemu_logcontext.c
 src/qemu/qemu_migration.c
 src/qemu/qemu_migration_cookie.c
 src/qemu/qemu_migration_params.c
diff --git a/src/qemu/meson.build b/src/qemu/meson.build
index 9be6996195..6d7a1bfbb0 100644
--- a/src/qemu/meson.build
+++ b/src/qemu/meson.build
@@ -21,6 +21,7 @@ qemu_driver_sources = [
   'qemu_hotplug.c',
   'qemu_interface.c',
   'qemu_interop_config.c',
+  'qemu_logcontext.c',
   'qemu_migration.c',
   'qemu_migration_cookie.c',
   'qemu_migration_params.c',
diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c
index d79f9879df..23608f95bd 100644
--- a/src/qemu/qemu_domain.c
+++ b/src/qemu/qemu_domain.c
@@ -455,21 +455,8 @@ qemuDomainObjFromDomain(virDomainPtr domain)
 }
 
 
-struct _qemuDomainLogContext {
-GObject parent;
-
-int writefd;
-int readfd; /* Only used if manager == NULL */
-off_t pos;
-ino_t inode; /* Only used if manager != NULL */
-char *path;
-virLogManager *manager;
-};
-
-G_DEFINE_TYPE(qemuDomainLogContext, qemu_domain_log_context, G_TYPE_OBJECT);
 static virClass *qemuDomainSaveCookieClass;
 
-static void qemuDomainLogContextFinalize(GObject *obj);
 static void qemuDomainSaveCookieDispose(void *obj);
 
 
@@ -482,32 +469,8 @@ qemuDomainOnceInit(void)
 return 0;
 }
 
-static void qemu_domain_log_context_init(qemuDomainLogContext *logctxt 
G_GNUC_UNUSED)
-{
-}
-
-static void qemu_domain_log_context_class_init(qemuDomainLogContextClass 
*klass)
-{
-GObjectClass *obj = G_OBJECT_CLASS(klass);
-
-obj->finalize = qemuDomainLogContextFinalize;
-}
-
 VIR_ONCE_GLOBAL_INIT(qemuDomain);
 
-static void
-qemuDomainLogContextFinalize(GObject *object)
-{
-qemuDomainLogContext *ctxt = QEMU_DOMAIN_LOG_CONTEXT(object);
-VIR_DEBUG("ctxt=%p", ctxt);
-
-virLogManagerFree(ctxt->manager);
-VIR_FREE(ctxt->path);
-VIR_FORCE_CLOSE(ctxt->writefd);
-VIR_FORCE_CLOSE(ctxt->readfd);
-G_OBJECT_CLASS(qemu_domain_log_context_parent_class)->finalize(object);
-}
-
 /* qemuDomainGetMasterKeyFilePath:
  * @libDir: Directory path to domain lib files
  *
@@ -6882,7 +6845,7 @@ static void G_GNUC_PRINTF(5, 6)
 qemuDomainObjTaintMsg(virQEMUDriver *driver,
   virDomainObj *obj,
   virDomainTaintFlags taint,
-  qemuDomainLogContext *logCtxt,
+  qemuLogContext *logCtxt,
   const char *fmt, ...)
 {
 virErrorPtr orig_err = NULL;
@@ -6935,12 +6898,12 @@ qemuDomainObjTaintMsg(virQEMUDriver *driver,
 goto cleanup;
 
 if (logCtxt) {
-rc = qemuDomainLogContextWrite(logCtxt,
-   "%s: Domain id=%d is tainted: 
%s%s%s%s\n",
-   timestamp,
-   obj->def->id,
-   virDomainTaintTypeToString(taint),
-   extraprefix, extramsg, extrasuffix);
+rc = qemuLogContextWrite(logCtxt,
+ "%s: Domain id=%d is tainted: %s%s%s%s\n",
+ timestamp,
+ obj->def->id,
+ virDomainTaintTypeToString(taint),
+ extraprefix, extramsg, extrasuffix);
 } else {
 rc = qemuDomainLogAppendMessage(driver, obj,
 "%s: Domain id=%d is tainted: 
%s%s%s%s\n",
@@ -6961,7 +6924,7 @@ qemuDomainObjTaintMsg(virQEMUDriver *driver,
 void qemuDomainObjTaint(virQEMUDriver *driver,
 virDomainObj *obj,
 virDomainTaintFlags taint,
-qemuDomainLogContext *logCtxt)
+qemuLogContext *logCtxt)
 {
 qemuDomainObjTaintMsg(driver, obj, taint, logCtxt, NULL);
 qemuDomainSaveStatus(obj);
@@ -6970,7 +6933,7 @@ void qemuDomainObjTaint(virQEMUDriver *driver,
 static void
 qemuDomainObjCheckMachineTaint(virQEMUDriver *driver,
virDomainObj *obj,
-   qemuDomainLogContext *logCtxt)
+

[libvirt PATCH v7 08/35] qemu: Add qemuNbdkitProcess

[Jonathon Jongsma]

An object for storing information about a nbdkit process that is serving
a specific virStorageSource. At the moment, this information is just
stored in the private data of virStorageSource and not used at all.
Future commits will use this data to actually start a nbdkit process.

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 src/qemu/qemu_conf.c   | 22 
 src/qemu/qemu_conf.h   |  2 ++
 src/qemu/qemu_domain.c | 31 
 src/qemu/qemu_domain.h |  4 +++
 src/qemu/qemu_nbdkit.c | 82 ++
 src/qemu/qemu_nbdkit.h | 26 ++
 6 files changed, 167 insertions(+)

diff --git a/src/qemu/qemu_conf.c b/src/qemu/qemu_conf.c
index 3f811d064f..a0360e8d1b 100644
--- a/src/qemu/qemu_conf.c
+++ b/src/qemu/qemu_conf.c
@@ -1656,3 +1656,25 @@ qemuHugepageMakeBasedir(virQEMUDriver *driver,
 
 return 0;
 }
+
+
+/*
+ * qemuGetNbdkitCaps:
+ * @driver: the qemu driver
+ *
+ * Gets the capabilities for Nbdkit for the specified driver. These can be used
+ * to determine whether a particular disk source can be served by nbdkit or
+ * not.
+ *
+ * Returns: a reference to qemuNbdkitCaps or NULL
+ */
+qemuNbdkitCaps*
+qemuGetNbdkitCaps(virQEMUDriver *driver)
+{
+char *nbdkitBinary = virFindFileInPath("nbdkit");
+
+if (!nbdkitBinary)
+return NULL;
+
+return virFileCacheLookup(driver->nbdkitCapsCache, nbdkitBinary);
+}
diff --git a/src/qemu/qemu_conf.h b/src/qemu/qemu_conf.h
index a44985fb8b..1a3ba3a0fb 100644
--- a/src/qemu/qemu_conf.h
+++ b/src/qemu/qemu_conf.h
@@ -377,3 +377,5 @@ int qemuGetMemoryBackingPath(virQEMUDriver *driver,
 
 int qemuHugepageMakeBasedir(virQEMUDriver *driver,
 virHugeTLBFS *hugepage);
+
+qemuNbdkitCaps* qemuGetNbdkitCaps(virQEMUDriver *driver);
diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c
index bfeddc7746..5a2eb4868a 100644
--- a/src/qemu/qemu_domain.c
+++ b/src/qemu/qemu_domain.c
@@ -882,6 +882,7 @@ qemuDomainStorageSourcePrivateDispose(void *obj)
 g_clear_pointer(>httpcookie, qemuDomainSecretInfoFree);
 g_clear_pointer(>tlsKeySecret, qemuDomainSecretInfoFree);
 g_clear_pointer(>fdpass, qemuFDPassFree);
+g_clear_pointer(>nbdkitProcess, qemuNbdkitProcessFree);
 }
 
 
@@ -10471,6 +10472,34 @@ qemuDomainPrepareStorageSourceNFS(virStorageSource 
*src)
 }
 
 
+/* qemuPrepareStorageSourceNbdkit:
+ * @src: source for a disk
+ *
+ * If src is an network source that is managed by nbdkit, prepare data so that
+ * nbdkit can be launched before the domain is started
+ *
+ * Returns true if nbdkit will be used for this source,
+ */
+static bool
+qemuDomainPrepareStorageSourceNbdkit(virStorageSource *src,
+ virQEMUDriverConfig *cfg,
+ const char *alias,
+ qemuDomainObjPrivate *priv)
+{
+g_autoptr(qemuNbdkitCaps) nbdkit = NULL;
+
+if (virStorageSourceGetActualType(src) != VIR_STORAGE_TYPE_NETWORK)
+return false;
+
+nbdkit = qemuGetNbdkitCaps(priv->driver);
+if (!nbdkit)
+return false;
+
+return qemuNbdkitInitStorageSource(nbdkit, src, priv->libDir,
+   alias, cfg->user, cfg->group);
+}
+
+
 /* qemuProcessPrepareStorageSourceTLS:
  * @source: source for a disk
  * @cfg: driver configuration
@@ -11300,6 +11329,8 @@ 
qemuDomainPrepareStorageSourceBlockdevNodename(virDomainDiskDef *disk,
 if (qemuDomainPrepareStorageSourceFDs(src, priv) < 0)
 return -1;
 
+qemuDomainPrepareStorageSourceNbdkit(src, cfg, src->nodestorage, priv);
+
 return 0;
 }
 
diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h
index 786f239495..89edc75fcf 100644
--- a/src/qemu/qemu_domain.h
+++ b/src/qemu/qemu_domain.h
@@ -33,6 +33,7 @@
 #include "qemu_conf.h"
 #include "qemu_capabilities.h"
 #include "qemu_migration_params.h"
+#include "qemu_nbdkit.h"
 #include "qemu_slirp.h"
 #include "qemu_fd.h"
 #include "virchrdev.h"
@@ -308,6 +309,9 @@ struct _qemuDomainStorageSourcePrivate {
 
 /* file descriptors if user asks for FDs to be passed */
 qemuFDPass *fdpass;
+
+/* an nbdkit process for serving network storage sources */
+qemuNbdkitProcess *nbdkitProcess;
 };
 
 virObject *qemuDomainStorageSourcePrivateNew(void);
diff --git a/src/qemu/qemu_nbdkit.c b/src/qemu/qemu_nbdkit.c
index 97612b637f..12c721f7f1 100644
--- a/src/qemu/qemu_nbdkit.c
+++ b/src/qemu/qemu_nbdkit.c
@@ -549,3 +549,85 @@ qemuNbdkitCapsCacheNew(const char *cachedir)
 g_autofree char *dir = g_build_filename(cachedir, "nbdkitcapabilities", 
NULL);
 return virFileCacheNew(dir, "xml", );
 }
+
+
+static qemuNbdkitProcess *
+qemuNbdkitProcessNew(virStorageSource *source,
+ const char *pidfile,
+ const char *socketfile)
+{
+qemuNbdkitProcess *nbdkit = g_new0(qemuNbdkitProcess, 1);
+/* weak reference -- source owns this object, so it will always outlive us 

[libvirt PATCH v7 09/35] qemu: query nbdkit module dir from binary

[Jonathon Jongsma]

Rather than hard-coding the nbdkit module directory, query the nbdkit
binary for the location to these directories. nbdkit provides a
--dump-config optiont that outputs this information and can be easily
parsed. We can also get the version from this output rather than
executing `nbdkit --version` separately.

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 src/qemu/qemu_nbdkit.c | 77 --
 1 file changed, 60 insertions(+), 17 deletions(-)

diff --git a/src/qemu/qemu_nbdkit.c b/src/qemu/qemu_nbdkit.c
index 12c721f7f1..9a2a89224d 100644
--- a/src/qemu/qemu_nbdkit.c
+++ b/src/qemu/qemu_nbdkit.c
@@ -20,7 +20,6 @@
 #include 
 #include 
 
-#include "configmake.h"
 #include "vircommand.h"
 #include "virerror.h"
 #include "virlog.h"
@@ -39,10 +38,6 @@
 
 VIR_LOG_INIT("qemu.nbdkit");
 
-#define NBDKIT_MODDIR LIBDIR "/nbdkit"
-#define NBDKIT_PLUGINDIR NBDKIT_MODDIR "/plugins"
-#define NBDKIT_FILTERDIR NBDKIT_MODDIR "/filters"
-
 VIR_ENUM_IMPL(qemuNbdkitCaps,
 QEMU_NBDKIT_CAPS_LAST,
 /* 0 */
@@ -56,6 +51,9 @@ struct _qemuNbdkitCaps {
 
 char *path;
 char *version;
+char *filterDir;
+char *pluginDir;
+
 time_t ctime;
 time_t libvirtCtime;
 time_t pluginDirMtime;
@@ -129,18 +127,47 @@ qemuNbdkitCapsQueryFilters(qemuNbdkitCaps *nbdkit)
 
 
 static int
-qemuNbdkitCapsQueryVersion(qemuNbdkitCaps *nbdkit)
+qemuNbdkitCapsQueryBuildConfig(qemuNbdkitCaps *nbdkit)
 {
+size_t i;
+g_autofree char *output = NULL;
+g_auto(GStrv) lines = NULL;
+const char *line;
 g_autoptr(virCommand) cmd = virCommandNewArgList(nbdkit->path,
- "--version",
+ "--dump-config",
  NULL);
 
-virCommandSetOutputBuffer(cmd, >version);
+virCommandSetOutputBuffer(cmd, );
 
 if (virCommandRun(cmd, NULL) != 0)
 return -1;
 
-VIR_DEBUG("Got nbdkit version %s", nbdkit->version);
+lines = g_strsplit(output, "\n", 0);
+if (!lines)
+return -1;
+
+for (i = 0; (line = lines[i]); i++) {
+const char *key;
+const char *val;
+char *p;
+
+p = strchr(line, '=');
+if (!p)
+continue;
+
+*p = '\0';
+key = line;
+val = p + 1;
+
+VIR_DEBUG("Got nbdkit config value %s=%s", key, val);
+
+if (STREQ(key, "version"))
+nbdkit->version = g_strdup(val);
+else if (STREQ(key, "filterdir"))
+nbdkit->filterDir = g_strdup(val);
+else if (STREQ(key, "plugindir"))
+nbdkit->pluginDir = g_strdup(val);
+}
 return 0;
 }
 
@@ -152,6 +179,8 @@ qemuNbdkitCapsFinalize(GObject *object)
 
 g_clear_pointer(>path, g_free);
 g_clear_pointer(>version, g_free);
+g_clear_pointer(>filterDir, g_free);
+g_clear_pointer(>pluginDir, g_free);
 g_clear_pointer(>flags, virBitmapFree);
 
 G_OBJECT_CLASS(qemu_nbdkit_caps_parent_class)->finalize(object);
@@ -214,15 +243,15 @@ qemuNbdkitCapsQuery(qemuNbdkitCaps *caps)
 return;
 }
 
+qemuNbdkitCapsQueryBuildConfig(caps);
+qemuNbdkitCapsQueryPlugins(caps);
+qemuNbdkitCapsQueryFilters(caps);
+
 caps->ctime = st.st_ctime;
-caps->filterDirMtime = qemuNbdkitGetDirMtime(NBDKIT_FILTERDIR);
-caps->pluginDirMtime = qemuNbdkitGetDirMtime(NBDKIT_PLUGINDIR);
+caps->filterDirMtime = qemuNbdkitGetDirMtime(caps->filterDir);
+caps->pluginDirMtime = qemuNbdkitGetDirMtime(caps->pluginDir);
 caps->libvirtCtime = virGetSelfLastChanged();
 caps->libvirtVersion = LIBVIR_VERSION_NUMBER;
-
-qemuNbdkitCapsQueryPlugins(caps);
-qemuNbdkitCapsQueryFilters(caps);
-qemuNbdkitCapsQueryVersion(caps);
 }
 
 
@@ -267,9 +296,9 @@ virNbdkitCapsIsValid(void *data,
 if (!nbdkitCaps->path)
 return true;
 
-if (!virNbkditCapsCheckModdir(NBDKIT_PLUGINDIR, 
nbdkitCaps->pluginDirMtime))
+if (!virNbkditCapsCheckModdir(nbdkitCaps->pluginDir, 
nbdkitCaps->pluginDirMtime))
 return false;
-if (!virNbkditCapsCheckModdir(NBDKIT_FILTERDIR, 
nbdkitCaps->filterDirMtime))
+if (!virNbkditCapsCheckModdir(nbdkitCaps->filterDir, 
nbdkitCaps->filterDirMtime))
 return false;
 
 if (nbdkitCaps->libvirtCtime != virGetSelfLastChanged() ||
@@ -421,12 +450,22 @@ qemuNbdkitCapsLoadCache(qemuNbdkitCaps *nbdkitCaps,
 }
 nbdkitCaps->ctime = (time_t)l;
 
+if ((nbdkitCaps->pluginDir = virXPathString("string(./plugindir)", ctxt)) 
== NULL) {
+VIR_DEBUG("missing plugindir in nbdkit capabilities cache");
+return -1;
+}
+
 if (virXPathLongLong("string(./plugindirmtime)", ctxt, ) < 0) {
 VIR_DEBUG("missing plugindirmtime in nbdkit capabilities XML");
 return -1;
 }
 nbdkitCaps->pluginDirMtime = (time_t)l;
 
+if ((nbdkitCaps->filterDir = virXPathString("string(./filterdir)", ctxt)) 
== NULL) {

[libvirt PATCH v7 03/35] qemu: expand nbdkit capabilities

[Jonathon Jongsma]

In order to add caching of the nbdkit capabilities, we will need to
compare against file modification times, etc. So look up this
information when creating the nbdkit caps.

Add a nbdkit_moddir build option to allow the builder to specify the
location to look for nbdkit plugins and filters.

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 src/qemu/qemu_nbdkit.c | 42 ++
 1 file changed, 42 insertions(+)

diff --git a/src/qemu/qemu_nbdkit.c b/src/qemu/qemu_nbdkit.c
index e4e8fd568e..58828dd89a 100644
--- a/src/qemu/qemu_nbdkit.c
+++ b/src/qemu/qemu_nbdkit.c
@@ -20,6 +20,7 @@
 #include 
 #include 
 
+#include "configmake.h"
 #include "vircommand.h"
 #include "virerror.h"
 #include "virlog.h"
@@ -38,6 +39,10 @@
 
 VIR_LOG_INIT("qemu.nbdkit");
 
+#define NBDKIT_MODDIR LIBDIR "/nbdkit"
+#define NBDKIT_PLUGINDIR NBDKIT_MODDIR "/plugins"
+#define NBDKIT_FILTERDIR NBDKIT_MODDIR "/filters"
+
 VIR_ENUM_IMPL(qemuNbdkitCaps,
 QEMU_NBDKIT_CAPS_LAST,
 /* 0 */
@@ -51,6 +56,11 @@ struct _qemuNbdkitCaps {
 
 char *path;
 char *version;
+time_t ctime;
+time_t libvirtCtime;
+time_t pluginDirMtime;
+time_t filterDirMtime;
+unsigned int libvirtVersion;
 
 virBitmap *flags;
 };
@@ -175,9 +185,41 @@ qemuNbdkitCapsNew(const char *path)
 }
 
 
+static time_t
+qemuNbdkitGetDirMtime(const char *moddir)
+{
+struct stat st;
+
+if (stat(moddir, ) < 0) {
+VIR_DEBUG("Failed to stat nbdkit module directory '%s': %s",
+  moddir,
+  g_strerror(errno));
+return 0;
+}
+
+return st.st_mtime;
+}
+
+
 G_GNUC_UNUSED static void
 qemuNbdkitCapsQuery(qemuNbdkitCaps *caps)
 {
+struct stat st;
+
+if (stat(caps->path, ) < 0) {
+VIR_DEBUG("Failed to stat nbdkit binary '%s': %s",
+  caps->path,
+  g_strerror(errno));
+caps->ctime  = 0;
+return;
+}
+
+caps->ctime = st.st_ctime;
+caps->filterDirMtime = qemuNbdkitGetDirMtime(NBDKIT_FILTERDIR);
+caps->pluginDirMtime = qemuNbdkitGetDirMtime(NBDKIT_PLUGINDIR);
+caps->libvirtCtime = virGetSelfLastChanged();
+caps->libvirtVersion = LIBVIR_VERSION_NUMBER;
+
 qemuNbdkitCapsQueryPlugins(caps);
 qemuNbdkitCapsQueryFilters(caps);
 qemuNbdkitCapsQueryVersion(caps);
-- 
2.41.0

[libvirt PATCH v7 01/35] schema: allow 'ssh' as a protocol for network disks

[Jonathon Jongsma]

There was support in the code for parsing protocol='ssh' on network disk
sources, but it was not present in the xml schema. Add this to the
schema.

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 src/conf/schemas/domaincommon.rng |  1 +
 tests/qemublocktest.c |  2 +-
 ...w2-invalid.json => network-ssh-qcow2.json} |  0
 ...cow2-invalid.xml => network-ssh-qcow2.xml} |  0
 .../disk-network-ssh.x86_64-latest.args   | 35 +++
 tests/qemuxml2argvdata/disk-network-ssh.xml   | 31 
 tests/qemuxml2argvtest.c  |  1 +
 7 files changed, 69 insertions(+), 1 deletion(-)
 rename tests/qemublocktestdata/imagecreate/{network-ssh-qcow2-invalid.json => 
network-ssh-qcow2.json} (100%)
 rename tests/qemublocktestdata/imagecreate/{network-ssh-qcow2-invalid.xml => 
network-ssh-qcow2.xml} (100%)
 create mode 100644 tests/qemuxml2argvdata/disk-network-ssh.x86_64-latest.args
 create mode 100644 tests/qemuxml2argvdata/disk-network-ssh.xml

diff --git a/src/conf/schemas/domaincommon.rng 
b/src/conf/schemas/domaincommon.rng
index de3bd1c35c..4a475f5c36 100644
--- a/src/conf/schemas/domaincommon.rng
+++ b/src/conf/schemas/domaincommon.rng
@@ -2179,6 +2179,7 @@
   
 sheepdog
 tftp
+ssh
   
 
 
diff --git a/tests/qemublocktest.c b/tests/qemublocktest.c
index 9a968477d7..8bad69e7ac 100644
--- a/tests/qemublocktest.c
+++ b/tests/qemublocktest.c
@@ -1213,7 +1213,7 @@ mymain(void)
 
 TEST_IMAGE_CREATE("network-gluster-qcow2", NULL);
 TEST_IMAGE_CREATE("network-rbd-qcow2", NULL);
-TEST_IMAGE_CREATE("network-ssh-qcow2-invalid", NULL);
+TEST_IMAGE_CREATE("network-ssh-qcow2", NULL);
 
 #define TEST_BITMAP_DETECT(testname) \
 do { \
diff --git a/tests/qemublocktestdata/imagecreate/network-ssh-qcow2-invalid.json 
b/tests/qemublocktestdata/imagecreate/network-ssh-qcow2.json
similarity index 100%
rename from tests/qemublocktestdata/imagecreate/network-ssh-qcow2-invalid.json
rename to tests/qemublocktestdata/imagecreate/network-ssh-qcow2.json
diff --git a/tests/qemublocktestdata/imagecreate/network-ssh-qcow2-invalid.xml 
b/tests/qemublocktestdata/imagecreate/network-ssh-qcow2.xml
similarity index 100%
rename from tests/qemublocktestdata/imagecreate/network-ssh-qcow2-invalid.xml
rename to tests/qemublocktestdata/imagecreate/network-ssh-qcow2.xml
diff --git a/tests/qemuxml2argvdata/disk-network-ssh.x86_64-latest.args 
b/tests/qemuxml2argvdata/disk-network-ssh.x86_64-latest.args
new file mode 100644
index 00..b7fd30032b
--- /dev/null
+++ b/tests/qemuxml2argvdata/disk-network-ssh.x86_64-latest.args
@@ -0,0 +1,35 @@
+LC_ALL=C \
+PATH=/bin \
+HOME=/var/lib/libvirt/qemu/domain--1-QEMUGuest1 \
+USER=test \
+LOGNAME=test \
+XDG_DATA_HOME=/var/lib/libvirt/qemu/domain--1-QEMUGuest1/.local/share \
+XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain--1-QEMUGuest1/.cache \
+XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain--1-QEMUGuest1/.config \
+/usr/bin/qemu-system-x86_64 \
+-name guest=QEMUGuest1,debug-threads=on \
+-S \
+-object 
'{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain--1-QEMUGuest1/master-key.aes"}'
 \
+-machine pc,usb=off,dump-guest-core=off,memory-backend=pc.ram,acpi=off \
+-accel kvm \
+-cpu qemu64 \
+-m size=219136k \
+-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":224395264}' \
+-overcommit mem-lock=off \
+-smp 1,sockets=1,cores=1,threads=1 \
+-uuid c7a5fdbd-edaf-9455-926a-d65c16db1809 \
+-display none \
+-no-user-config \
+-nodefaults \
+-chardev socket,id=charmonitor,fd=1729,server=on,wait=off \
+-mon chardev=charmonitor,id=monitor,mode=control \
+-rtc base=utc \
+-no-shutdown \
+-boot strict=on \
+-device 
'{"driver":"piix3-usb-uhci","id":"usb","bus":"pci.0","addr":"0x1.0x2"}' \
+-blockdev 
'{"driver":"ssh","path":"test.img","server":{"host":"example.org","port":""},"node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}'
 \
+-blockdev 
'{"node-name":"libvirt-1-format","read-only":false,"driver":"raw","file":"libvirt-1-storage"}'
 \
+-device 
'{"driver":"virtio-blk-pci","bus":"pci.0","addr":"0x2","drive":"libvirt-1-format","id":"virtio-disk0","bootindex":1}'
 \
+-audiodev '{"id":"audio1","driver":"none"}' \
+-sandbox 
on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
+-msg timestamp=on
diff --git a/tests/qemuxml2argvdata/disk-network-ssh.xml 
b/tests/qemuxml2argvdata/disk-network-ssh.xml
new file mode 100644
index 00..355add4fea
--- /dev/null
+++ b/tests/qemuxml2argvdata/disk-network-ssh.xml
@@ -0,0 +1,31 @@
+
+  QEMUGuest1
+  c7a5fdbd-edaf-9455-926a-d65c16db1809
+  219136
+  219136
+  1
+  
+hvm
+
+  
+  
+  destroy
+  restart
+  destroy
+  
+
+  
+  
+
+
+
+  
+  
+
+
+
+
+
+
+  
+
diff --git a/tests/qemuxml2argvtest.c b/tests/qemuxml2argvtest.c
index 

[libvirt PATCH v7 02/35] qemu: Add functions for determining nbdkit availability

[Jonathon Jongsma]

In future commits, we will optionally use nbdkit to serve some remote
disk sources. This patch queries to see whether nbdkit is installed on
the host and queries it for capabilities. The data will be used in later
commits.

Signed-off-by: Jonathon Jongsma 
Reviewed-by: Peter Krempa 
---
 src/qemu/meson.build   |   1 +
 src/qemu/qemu_conf.h   |   1 +
 src/qemu/qemu_nbdkit.c | 200 +
 src/qemu/qemu_nbdkit.h |  50 +++
 4 files changed, 252 insertions(+)
 create mode 100644 src/qemu/qemu_nbdkit.c
 create mode 100644 src/qemu/qemu_nbdkit.h

diff --git a/src/qemu/meson.build b/src/qemu/meson.build
index c8806bbc36..9be6996195 100644
--- a/src/qemu/meson.build
+++ b/src/qemu/meson.build
@@ -28,6 +28,7 @@ qemu_driver_sources = [
   'qemu_monitor_json.c',
   'qemu_monitor_text.c',
   'qemu_namespace.c',
+  'qemu_nbdkit.c',
   'qemu_passt.c',
   'qemu_process.c',
   'qemu_qapi.c',
diff --git a/src/qemu/qemu_conf.h b/src/qemu/qemu_conf.h
index 11c740d28f..4f610d86a1 100644
--- a/src/qemu/qemu_conf.h
+++ b/src/qemu/qemu_conf.h
@@ -36,6 +36,7 @@
 #include "virthreadpool.h"
 #include "locking/lock_manager.h"
 #include "qemu_capabilities.h"
+#include "qemu_nbdkit.h"
 #include "virclosecallbacks.h"
 #include "virhostdev.h"
 #include "virfile.h"
diff --git a/src/qemu/qemu_nbdkit.c b/src/qemu/qemu_nbdkit.c
new file mode 100644
index 00..e4e8fd568e
--- /dev/null
+++ b/src/qemu/qemu_nbdkit.c
@@ -0,0 +1,200 @@
+/*
+ * qemu_nbdkit.c: helpers for using nbdkit
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library.  If not, see
+ * .
+ *
+ */
+
+#include 
+#include 
+
+#include "vircommand.h"
+#include "virerror.h"
+#include "virlog.h"
+#include "virpidfile.h"
+#include "virutil.h"
+#include "qemu_block.h"
+#include "qemu_conf.h"
+#include "qemu_domain.h"
+#include "qemu_extdevice.h"
+#include "qemu_nbdkit.h"
+#include "qemu_security.h"
+
+#include 
+
+#define VIR_FROM_THIS VIR_FROM_QEMU
+
+VIR_LOG_INIT("qemu.nbdkit");
+
+VIR_ENUM_IMPL(qemuNbdkitCaps,
+QEMU_NBDKIT_CAPS_LAST,
+/* 0 */
+"plugin-curl", /* QEMU_NBDKIT_CAPS_PLUGIN_CURL */
+"plugin-ssh", /* QEMU_NBDKIT_CAPS_PLUGIN_SSH */
+"filter-readahead", /* QEMU_NBDKIT_CAPS_FILTER_READAHEAD */
+);
+
+struct _qemuNbdkitCaps {
+GObject parent;
+
+char *path;
+char *version;
+
+virBitmap *flags;
+};
+G_DEFINE_TYPE(qemuNbdkitCaps, qemu_nbdkit_caps, G_TYPE_OBJECT);
+
+
+static void
+qemuNbdkitCheckCommandCap(qemuNbdkitCaps *nbdkit,
+  virCommand *cmd,
+  qemuNbdkitCapsFlags cap)
+{
+if (virCommandRun(cmd, NULL) != 0)
+return;
+
+VIR_DEBUG("Setting nbdkit capability %i", cap);
+ignore_value(virBitmapSetBit(nbdkit->flags, cap));
+}
+
+
+static void
+qemuNbdkitQueryFilter(qemuNbdkitCaps *nbdkit,
+  const char *filter,
+  qemuNbdkitCapsFlags cap)
+{
+g_autoptr(virCommand) cmd = virCommandNewArgList(nbdkit->path,
+ "--version",
+ NULL);
+
+virCommandAddArgPair(cmd, "--filter", filter);
+
+/* use null plugin to check for filter */
+virCommandAddArg(cmd, "null");
+
+qemuNbdkitCheckCommandCap(nbdkit, cmd, cap);
+}
+
+
+static void
+qemuNbdkitQueryPlugin(qemuNbdkitCaps *nbdkit,
+  const char *plugin,
+  qemuNbdkitCapsFlags cap)
+{
+g_autoptr(virCommand) cmd = virCommandNewArgList(nbdkit->path,
+ plugin,
+ "--version",
+ NULL);
+
+qemuNbdkitCheckCommandCap(nbdkit, cmd, cap);
+}
+
+
+static void
+qemuNbdkitCapsQueryPlugins(qemuNbdkitCaps *nbdkit)
+{
+qemuNbdkitQueryPlugin(nbdkit, "curl", QEMU_NBDKIT_CAPS_PLUGIN_CURL);
+qemuNbdkitQueryPlugin(nbdkit, "ssh", QEMU_NBDKIT_CAPS_PLUGIN_SSH);
+}
+
+
+static void
+qemuNbdkitCapsQueryFilters(qemuNbdkitCaps *nbdkit)
+{
+qemuNbdkitQueryFilter(nbdkit, "readahead",
+  QEMU_NBDKIT_CAPS_FILTER_READAHEAD);
+}
+
+
+static int
+qemuNbdkitCapsQueryVersion(qemuNbdkitCaps *nbdkit)
+{
+g_autoptr(virCommand) cmd = virCommandNewArgList(nbdkit->path,
+ 

[libvirt PATCH v7 00/35] Use nbdkit for http/ftp/ssh network drives in libvirt

[Jonathon Jongsma]

This is the seventh version of this patch series. See
https://bugzilla.redhat.com/show_bug.cgi?id=2016527 for more information.

Note that testing this requires selinux policy changes which are not fully
done, but there is a new policy in development that has allowed me to run with
selinux in enforcing mode for the common cases. See
https://bugzilla.redhat.com/show_bug.cgi?id=2182505 for more information. The
following scenarios should work now with selinux enabled using the selinux
policy from that bug:
 - http/https disks
 - ssh disks with password authentication
 - ssh disks with passwordless keyfile

The one major thing that doesn't work and is difficult to get working with
selinux enabled is the ssh-agent. This is because there doesn't seem to be any
selinux policy for ssh-agent, so by default the ssh-agent socket is labeled
unconfined_t. We cannot allow access from the libvirt/qemu to unconfined_t
because that would open up access to just about anything on the host. So
additional work will likely be necessary for ssh-agent/libvirt interaction in
the future. Fortunately ssh-agent is something that never was really supported
with the old qemu block driver either, so I think we could potentially merge
this patchset either without the ssh-agent patches or with a note that
ssh-agent won't work with selinux enabled.

Note also that gitlab CI will not work for this series without changes to the
ci definitions due to the addition of libnbd dependency.

Changes in v7:
 - rebased to latest master
 - moved restarting of nbdkit process to per-domain event thread
 - a few other smaller changes suggested by Peter in v6

Jonathon Jongsma (35):
  schema: allow 'ssh' as a protocol for network disks
  qemu: Add functions for determining nbdkit availability
  qemu: expand nbdkit capabilities
  util: Allow virFileCache data to be any GObject
  qemu: implement basic virFileCache for nbdkit caps
  qemu: implement persistent file cache for nbdkit caps
  qemu: use file cache for nbdkit caps
  qemu: Add qemuNbdkitProcess
  qemu: query nbdkit module dir from binary
  qemu: add functions to start and stop nbdkit
  Generalize qemuDomainLogContextNew()
  qemu: Extract qemuDomainLogContext into a new file
  qemu: move qemuProcessReadLog() to qemuLogContext
  qemu: log error output from nbdkit
  tests: add ability to test various nbdkit capabilities
  qemu: split qemuDomainSecretStorageSourcePrepare
  qemu: include nbdkit state in private xml
  util: secure erase virCommand send buffers
  qemu: pass sensitive data to nbdkit via pipe
  qemu: use nbdkit to serve network disks if available
  util: make virCommandSetSendBuffer testable
  tests: add tests for nbdkit invocation
  qemu: add test for authenticating a https network disk
  qemu: Monitor nbdkit process for exit
  qemu: Taint domain if nbdkit restart fails
  qemu: try to connect to nbdkit early to detect errors
  schema: add password configuration for ssh disk
  qemu: implement password auth for ssh disks with nbdkit
  schema: add configuration for host verification of ssh disks
  qemu: implement knownHosts for ssh disks with nbdkit
  schema: add keyfile configuration for ssh disks
  qemu: implement keyfile auth for ssh disks with nbdkit
  schema: add ssh-agent configuration for ssh disks
  qemu: implement ssh-agent auth for ssh disks with nbdkit
  rpm: update spec file for for nbdkit support

 build-aux/syntax-check.mk |2 +-
 docs/formatdomain.rst |   45 +-
 libvirt.spec.in   |8 +
 meson.build   |   14 +
 meson_options.txt |1 +
 po/POTFILES   |2 +
 src/conf/domain_conf.c|   38 +
 src/conf/domain_conf.h|1 +
 src/conf/schemas/domaincommon.rng |   55 +
 src/conf/storage_source_conf.c|6 +
 src/conf/storage_source_conf.h|6 +-
 src/libvirt_private.syms  |1 +
 src/qemu/meson.build  |3 +
 src/qemu/qemu_block.c |  162 ++-
 src/qemu/qemu_conf.c  |   22 +
 src/qemu/qemu_conf.h  |6 +
 src/qemu/qemu_domain.c|  436 +++---
 src/qemu/qemu_domain.h|   31 +-
 src/qemu/qemu_driver.c|   21 +
 src/qemu/qemu_extdevice.c |   62 +
 src/qemu/qemu_hotplug.c   |7 +
 src/qemu/qemu_logcontext.c|  329 +
 src/qemu/qemu_logcontext.h|   41 +
 src/qemu/qemu_nbdkit.c| 1277 +
 src/qemu/qemu_nbdkit.h|  119 ++
 src/qemu/qemu_nbdkitpriv.h|   31 +
 src/qemu/qemu_process.c   |  122 +-
 src/qemu/qemu_process.h   |3 +
 

Re: CPU pinning implementation through libvirt

[Martin Kletzander]

On Wed, Aug 16, 2023 at 05:41:37PM -0600, x...@trimaso.com.mx wrote:

I'll be direct.

After learning that QEMU itself has never officially implemented options
such as "-vcpu", "-vcpupin", or similar, remaining only as unimplemented
commits at their mailing lists, I felt the need of asking here once and
for all:

How does libvirt actually implements CPU pinning under the hood?
There's the xml with the cputune section and all the vcpupins, but does
libvirt translates this to QEMU native commands, or what else does it
do?



It uses whatever the underlying OS provides, cgroups, setting affinity,
etc. on QEMU's threads.


Thanks.



signature.asc
Description: PGP signature

[PATCH 5/5] virDomainFeaturesXENDefParse: Use virXMLNodeGetSubelementList

[Peter Krempa]

Rewrite the old-style parser to use virXMLNodeGetSubelementList

Signed-off-by: Peter Krempa 
---
 src/conf/domain_conf.c | 17 +
 1 file changed, 9 insertions(+), 8 deletions(-)

diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c
index 3a08034b9d..02bba77768 100644
--- a/src/conf/domain_conf.c
+++ b/src/conf/domain_conf.c
@@ -16409,22 +16409,25 @@ static int
 virDomainFeaturesXENDefParse(virDomainDef *def,
  xmlNodePtr node)
 {
+g_autofree xmlNodePtr *feats = NULL;
+size_t nfeats = virXMLNodeGetSubelementList(node, NULL, );
+size_t i;
+
 def->features[VIR_DOMAIN_FEATURE_XEN] = VIR_TRISTATE_SWITCH_ON;

-node = xmlFirstElementChild(node);
-while (node) {
+for (i = 0; i < nfeats; i++) {
 int feature;
 virTristateSwitch value;

-feature = virDomainXenTypeFromString((const char *)node->name);
+feature = virDomainXenTypeFromString((const char *)feats[i]->name);
 if (feature < 0) {
 virReportError(VIR_ERR_CONFIG_UNSUPPORTED,
_("unsupported Xen feature: %1$s"),
-   node->name);
+   feats[i]->name);
 return -1;
 }

-if (virXMLPropTristateSwitch(node, "state",
+if (virXMLPropTristateSwitch(feats[i], "state",
  VIR_XML_PROP_REQUIRED, ) < 0)
 return -1;

@@ -16438,7 +16441,7 @@ virDomainFeaturesXENDefParse(virDomainDef *def,
 if (value != VIR_TRISTATE_SWITCH_ON)
 break;

-if (virXMLPropEnum(node, "mode",
+if (virXMLPropEnum(feats[i], "mode",
virDomainXenPassthroughModeTypeFromString,
VIR_XML_PROP_NONZERO,
>xen_passthrough_mode) < 0)
@@ -16448,8 +16451,6 @@ virDomainFeaturesXENDefParse(virDomainDef *def,
 case VIR_DOMAIN_XEN_LAST:
 break;
 }
-
-node = xmlNextElementSibling(node);
 }

 return 0;
-- 
2.41.0

[PATCH 3/5] virDomainFeaturesCapabilitiesDefParse: Use virXMLNodeGetSubelementList

[Peter Krempa]

Rewrite the old-style parser to use virXMLNodeGetSubelementList

Signed-off-by: Peter Krempa 
---
 src/conf/domain_conf.c | 13 +++--
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c
index bb4f1fdb94..d510279472 100644
--- a/src/conf/domain_conf.c
+++ b/src/conf/domain_conf.c
@@ -16462,7 +16462,10 @@ static int
 virDomainFeaturesCapabilitiesDefParse(virDomainDef *def,
   xmlNodePtr node)
 {
+g_autofree xmlNodePtr *caps = NULL;
+size_t ncaps = virXMLNodeGetSubelementList(node, NULL, );
 virDomainCapabilitiesPolicy policy;
+size_t i;

 if (virXMLPropEnumDefault(node, "policy",
   virDomainCapabilitiesPolicyTypeFromString,
@@ -16472,25 +16475,23 @@ virDomainFeaturesCapabilitiesDefParse(virDomainDef 
*def,

 def->features[VIR_DOMAIN_FEATURE_CAPABILITIES] = policy;

-node = xmlFirstElementChild(node);
-while (node) {
+for (i = 0; i < ncaps; i++) {
 virTristateSwitch state;
-int val = virDomainProcessCapsFeatureTypeFromString((const char 
*)node->name);
+int val = virDomainProcessCapsFeatureTypeFromString((const char 
*)caps[i]->name);
 if (val < 0) {
 virReportError(VIR_ERR_CONFIG_UNSUPPORTED,
-   _("unexpected capability feature '%1$s'"), 
node->name);
+   _("unexpected capability feature '%1$s'"), 
caps[i]->name);
 return -1;
 }


-if (virXMLPropTristateSwitch(node, "state", VIR_XML_PROP_NONE, ) 
< 0)
+if (virXMLPropTristateSwitch(caps[i], "state", VIR_XML_PROP_NONE, 
) < 0)
 return -1;

 if (state == VIR_TRISTATE_SWITCH_ABSENT)
 state = VIR_TRISTATE_SWITCH_ON;

 def->caps_features[val] = state;
-node = xmlNextElementSibling(node);
 }

 return 0;
-- 
2.41.0

[PATCH 0/5] Refactor few old-style XML node lookup loops

[Peter Krempa]

Peter Krempa (5):
  virNetworkDNSHostDefParseXML: Refactor parsing
  virsh: domain: Refactor XML handling for disk changes
  virDomainFeaturesCapabilitiesDefParse: Use virXMLNodeGetSubelementList
  virDomainFeaturesKVMDefParse: Use virXMLNodeGetSubelementList
  virDomainFeaturesXENDefParse: Use virXMLNodeGetSubelementList

 src/conf/domain_conf.c  | 50 +++---
 src/conf/network_conf.c | 94 -
 tools/virsh-domain.c| 63 +++
 3 files changed, 78 insertions(+), 129 deletions(-)

-- 
2.41.0

[PATCH 1/5] virNetworkDNSHostDefParseXML: Refactor parsing

[Peter Krempa]

Use 'virXMLNodeGetSubelementList' instead of looping through XML nodes
and modernize the code.

Signed-off-by: Peter Krempa 
---
 src/conf/network_conf.c | 94 -
 1 file changed, 46 insertions(+), 48 deletions(-)

diff --git a/src/conf/network_conf.c b/src/conf/network_conf.c
index e4c8c5fd4d..9dd6fb4ce9 100644
--- a/src/conf/network_conf.c
+++ b/src/conf/network_conf.c
@@ -656,63 +656,61 @@ virNetworkDNSHostDefParseXML(const char *networkName,
  virNetworkDNSHostDef *def,
  bool partialOkay)
 {
-xmlNodePtr cur;
-g_autofree char *ip = NULL;
-
-if (!(ip = virXMLPropString(node, "ip")) && !partialOkay) {
-virReportError(VIR_ERR_XML_DETAIL,
-   _("Missing IP address in network '%1$s' DNS HOST 
record"),
-   networkName);
-goto error;
-}
-
-if (ip && (virSocketAddrParse(>ip, ip, AF_UNSPEC) < 0)) {
-virReportError(VIR_ERR_XML_DETAIL,
-   _("Invalid IP address in network '%1$s' DNS HOST 
record"),
-   networkName);
-goto error;
-}
+g_autofree xmlNodePtr *hostnameNodes = NULL;
+size_t nhostnameNodes = virXMLNodeGetSubelementList(node, "hostname", 
);
+size_t i;
+g_auto(GStrv) hostnames = NULL;
+g_autofree char *ip = virXMLPropString(node, "ip");

-cur = node->children;
-while (cur != NULL) {
-if (cur->type == XML_ELEMENT_NODE &&
-virXMLNodeNameEqual(cur, "hostname")) {
-  if (cur->children != NULL) {
-  g_autofree char *name = virXMLNodeContentString(cur);
+if (nhostnameNodes > 0) {
+hostnames = g_new0(char *, nhostnameNodes + 1);

-  if (!name)
-  goto error;
+for (i = 0; i < nhostnameNodes; i++) {
+if (!(hostnames[i] = virXMLNodeContentString(hostnameNodes[i])))
+return -1;

-  if (!name[0]) {
-  virReportError(VIR_ERR_XML_DETAIL,
- _("Missing hostname in network '%1$s' DNS 
HOST record"),
- networkName);
-  goto error;
-  }
-  VIR_APPEND_ELEMENT(def->names, def->nnames, name);
-  }
+if (*hostnames[i] == '\0') {
+virReportError(VIR_ERR_XML_DETAIL,
+   _("Missing hostname in network '%1$s' DNS HOST 
record"),
+   networkName);
+return -1;
+}
+}
+} else {
+if (!partialOkay) {
+virReportError(VIR_ERR_XML_DETAIL,
+   _("Missing hostname in network '%1$s' DNS HOST 
record"),
+   networkName);
+return -1;
 }
-cur = cur->next;
-}
-if (def->nnames == 0 && !partialOkay) {
-virReportError(VIR_ERR_XML_DETAIL,
-   _("Missing hostname in network '%1$s' DNS HOST record"),
-   networkName);
-goto error;
 }

-if (!VIR_SOCKET_ADDR_VALID(>ip) && def->nnames == 0) {
-virReportError(VIR_ERR_XML_DETAIL,
-   _("Missing ip and hostname in network '%1$s' DNS HOST 
record"),
-   networkName);
-goto error;
+if (ip) {
+if (virSocketAddrParse(>ip, ip, AF_UNSPEC) < 0) {
+virReportError(VIR_ERR_XML_DETAIL,
+   _("Invalid IP address in network '%1$s' DNS HOST 
record"),
+   networkName);
+return -1;
+}
+} else {
+if (!partialOkay) {
+virReportError(VIR_ERR_XML_DETAIL,
+   _("Missing IP address in network '%1$s' DNS HOST 
record"),
+   networkName);
+return -1;
+}
+
+if (nhostnameNodes == 0) {
+virReportError(VIR_ERR_XML_DETAIL,
+   _("Missing ip and hostname in network '%1$s' DNS 
HOST record"),
+   networkName);
+return -1;
+}
 }

+def->names = g_steal_pointer();
+def->nnames = nhostnameNodes;
 return 0;
-
- error:
-virNetworkDNSHostDefClear(def);
-return -1;
 }


-- 
2.41.0

[PATCH 2/5] virsh: domain: Refactor XML handling for disk changes

[Peter Krempa]

Use virXMLNodeGetSubelement to find needed subelements.

In virshUpdateDiskXML this commit removes the code which keeps XML
formatting tidy, but that is not needed for the code to format proper
XMLs.

Signed-off-by: Peter Krempa 
---
 tools/virsh-domain.c | 63 +---
 1 file changed, 7 insertions(+), 56 deletions(-)

diff --git a/tools/virsh-domain.c b/tools/virsh-domain.c
index e0776c991f..541a799aaf 100644
--- a/tools/virsh-domain.c
+++ b/tools/virsh-domain.c
@@ -12641,19 +12641,13 @@ cmdDetachInterface(vshControl *ctl, const vshCmd *cmd)
 static void
 virshDiskDropBackingStore(xmlNodePtr disk_node)
 {
-xmlNodePtr tmp;
+xmlNodePtr tmp = virXMLNodeGetSubelement(disk_node, "backingStore");

-for (tmp = disk_node->children; tmp; tmp = tmp->next) {
-if (tmp->type != XML_ELEMENT_NODE)
-continue;
-
-if (virXMLNodeNameEqual(tmp, "backingStore")) {
-xmlUnlinkNode(tmp);
-xmlFreeNode(tmp);
+if (!tmp)
+return;

-return;
-}
-}
+xmlUnlinkNode(tmp);
+xmlFreeNode(tmp);
 }


@@ -12753,10 +12747,7 @@ virshUpdateDiskXML(xmlNodePtr disk_node,
const char *target,
virshUpdateDiskXMLType type)
 {
-xmlNodePtr tmp = NULL;
 xmlNodePtr source = NULL;
-xmlNodePtr target_node = NULL;
-xmlNodePtr text_node = NULL;
 g_autofree char *device_type = NULL;
 char *ret = NULL;
 g_autofree char *startupPolicy = NULL;
@@ -12773,33 +12764,7 @@ virshUpdateDiskXML(xmlNodePtr disk_node,
 return NULL;
 }

-/* find the current source subelement */
-for (tmp = disk_node->children; tmp; tmp = tmp->next) {
-/*
- * Save the last text node before the .  The
- * reasoning behind this is that the target node will be
- * present in this case and also has a proper indentation.
- */
-if (!target_node && tmp->type == XML_TEXT_NODE)
-text_node = tmp;
-
-/*
- * We need only element nodes from now on.
- */
-if (tmp->type != XML_ELEMENT_NODE)
-continue;
-
-if (!source && virXMLNodeNameEqual(tmp, "source"))
-source = tmp;
-else if (!target_node && virXMLNodeNameEqual(tmp, "target"))
-target_node = tmp;
-
-/*
- * We've found all we needed.
- */
-if (source && target_node)
-break;
-}
+source = virXMLNodeGetSubelement(disk_node, "source");

 if (type == VIRSH_UPDATE_DISK_XML_EJECT) {
 if (!source) {
@@ -12852,21 +12817,7 @@ virshUpdateDiskXML(xmlNodePtr disk_node,
 if (startupPolicy)
 xmlNewProp(source, BAD_CAST "startupPolicy", BAD_CAST 
startupPolicy);

-/*
- * So that the output XML looks nice in case anyone calls
- * 'change-media' with '--print-xml', let's attach the source
- * before target...
- */
-xmlAddPrevSibling(target_node, source);
-
-/*
- * ... and duplicate the text node doing the indentation just
- * so it's more easily readable.  And don't make it fatal.
- */
-if ((tmp = xmlCopyNode(text_node, 0))) {
-if (!xmlAddPrevSibling(target_node, tmp))
-xmlFreeNode(tmp);
-}
+xmlAddChild(disk_node, source);
 }

 if (!(ret = virXMLNodeToString(NULL, disk_node))) {
-- 
2.41.0

[PATCH 4/5] virDomainFeaturesKVMDefParse: Use virXMLNodeGetSubelementList

[Peter Krempa]

Rewrite the old-style parser to use virXMLNodeGetSubelementList

Signed-off-by: Peter Krempa 
---
 src/conf/domain_conf.c | 20 +---
 1 file changed, 9 insertions(+), 11 deletions(-)

diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c
index d510279472..3a08034b9d 100644
--- a/src/conf/domain_conf.c
+++ b/src/conf/domain_conf.c
@@ -16355,24 +16355,24 @@ static int
 virDomainFeaturesKVMDefParse(virDomainDef *def,
  xmlNodePtr node)
 {
-g_autofree virDomainFeatureKVM *kvm = NULL;
-
-kvm = g_new0(virDomainFeatureKVM, 1);
+g_autofree virDomainFeatureKVM *kvm = g_new0(virDomainFeatureKVM, 1);
+g_autofree xmlNodePtr *feats = NULL;
+size_t nfeats = virXMLNodeGetSubelementList(node, NULL, );
+size_t i;

-node = xmlFirstElementChild(node);
-while (node) {
+for (i = 0; i < nfeats; i++) {
 int feature;
 virTristateSwitch value;

-feature = virDomainKVMTypeFromString((const char *)node->name);
+feature = virDomainKVMTypeFromString((const char *)feats[i]->name);
 if (feature < 0) {
 virReportError(VIR_ERR_CONFIG_UNSUPPORTED,
_("unsupported KVM feature: %1$s"),
-   node->name);
+   feats[i]->name);
 return -1;
 }

-if (virXMLPropTristateSwitch(node, "state", VIR_XML_PROP_REQUIRED,
+if (virXMLPropTristateSwitch(feats[i], "state", VIR_XML_PROP_REQUIRED,
  ) < 0)
 return -1;

@@ -16382,7 +16382,7 @@ virDomainFeaturesKVMDefParse(virDomainDef *def,
 if (feature == VIR_DOMAIN_KVM_DIRTY_RING &&
 value == VIR_TRISTATE_SWITCH_ON) {

-if (virXMLPropUInt(node, "size", 0, VIR_XML_PROP_REQUIRED,
+if (virXMLPropUInt(feats[i], "size", 0, VIR_XML_PROP_REQUIRED,
>dirty_ring_size) < 0) {
 return -1;
 }
@@ -16396,8 +16396,6 @@ virDomainFeaturesKVMDefParse(virDomainDef *def,
 return -1;
 }
 }
-
-node = xmlNextElementSibling(node);
 }

 def->features[VIR_DOMAIN_FEATURE_KVM] = VIR_TRISTATE_SWITCH_ON;
-- 
2.41.0

[PATCH 6/9] qemuhotplugtest: Remove 'modern' field for cpu hotplug tests

[Peter Krempa]

Nowadays all tests were considered 'modern' so it makes no longer sense
to have that field.

Signed-off-by: Peter Krempa 
---
 tests/qemuhotplugtest.c | 40 ++--
 1 file changed, 14 insertions(+), 26 deletions(-)

diff --git a/tests/qemuhotplugtest.c b/tests/qemuhotplugtest.c
index 5666e87f75..98f68ce3f4 100644
--- a/tests/qemuhotplugtest.c
+++ b/tests/qemuhotplugtest.c
@@ -276,7 +276,6 @@ struct testQemuHotplugCpuData {

 virDomainObj *vm;
 qemuMonitorTest *mon;
-bool modern;
 };


@@ -317,7 +316,6 @@ struct testQemuHotplugCpuParams {
 int newcpus;
 const char *cpumap;
 bool state;
-bool modern;
 bool fail;
 const char *arch;
 GHashTable *capsLatestFiles;
@@ -338,8 +336,6 @@ testQemuHotplugCpuPrepare(const struct 
testQemuHotplugCpuParams *params)

 data = g_new0(struct testQemuHotplugCpuData, 1);

-data->modern = params->modern;
-
 data->file_xml_dom = g_strdup_printf("%s-domain.xml", prefix);
 data->file_xml_res_live = g_strdup_printf("%s-result-live.xml", prefix);
 data->file_xml_res_conf = g_strdup_printf("%s-result-conf.xml", prefix);
@@ -360,9 +356,6 @@ testQemuHotplugCpuPrepare(const struct 
testQemuHotplugCpuParams *params)

 priv = data->vm->privateData;

-if (data->modern)
-virQEMUCapsSet(priv->qemuCaps, QEMU_CAPS_QUERY_HOTPLUGGABLE_CPUS);
-
 if (!(data->mon = qemuMonitorTestNewFromFileFull(data->file_json_monitor,
  , data->vm, 
schema)))
 goto error;
@@ -370,9 +363,6 @@ testQemuHotplugCpuPrepare(const struct 
testQemuHotplugCpuParams *params)
 if (params->fail)
 qemuMonitorTestAllowUnusedCommands(data->mon);

-if (!data->modern)
-qemuMonitorTestSkipDeprecatedValidation(data->mon, true);
-
 priv->mon = qemuMonitorTestGetMonitor(data->mon);
 virObjectUnlock(priv->mon);

@@ -429,7 +419,7 @@ testQemuHotplugCpuGroup(const void *opaque)

 rc = qemuDomainSetVcpusInternal(, data->vm, data->vm->def,
 data->vm->newDef, params->newcpus,
-params->modern);
+true);

 if (params->fail) {
 if (rc == 0)
@@ -799,44 +789,42 @@ mymain(void)
"blockdev-del", QMP_OK,
"blockdev-del", QMP_OK);

-#define DO_TEST_CPU_GROUP(archname, prefix, vcpus, modernhp, expectfail) \
+#define DO_TEST_CPU_GROUP(archname, prefix, vcpus, expectfail) \
 do { \
 cpudata.test = prefix; \
 cpudata.arch = archname; \
 cpudata.newcpus = vcpus; \
-cpudata.modern = modernhp; \
 cpudata.fail = expectfail; \
 if (virTestRun("hotplug vcpus group " prefix, \
testQemuHotplugCpuGroup, ) < 0) \
 ret = -1; \
 } while (0)

-DO_TEST_CPU_GROUP("x86_64", "x86-modern-bulk", 7, true, false);
-DO_TEST_CPU_GROUP("ppc64", "ppc64-modern-bulk", 24, true, false);
-DO_TEST_CPU_GROUP("ppc64", "ppc64-modern-bulk", 15, true, true);
-DO_TEST_CPU_GROUP("ppc64", "ppc64-modern-bulk", 23, true, true);
-DO_TEST_CPU_GROUP("ppc64", "ppc64-modern-bulk", 25, true, true);
+DO_TEST_CPU_GROUP("x86_64", "x86-modern-bulk", 7, false);
+DO_TEST_CPU_GROUP("ppc64", "ppc64-modern-bulk", 24, false);
+DO_TEST_CPU_GROUP("ppc64", "ppc64-modern-bulk", 15, true);
+DO_TEST_CPU_GROUP("ppc64", "ppc64-modern-bulk", 23, true);
+DO_TEST_CPU_GROUP("ppc64", "ppc64-modern-bulk", 25, true);

-#define DO_TEST_CPU_INDIVIDUAL(archname, prefix, mapstr, statefl, modernhp, 
expectfail) \
+#define DO_TEST_CPU_INDIVIDUAL(archname, prefix, mapstr, statefl, expectfail) \
 do { \
 cpudata.test = prefix; \
 cpudata.arch = archname; \
 cpudata.cpumap = mapstr; \
 cpudata.state = statefl; \
-cpudata.modern = modernhp; \
 cpudata.fail = expectfail; \
 if (virTestRun("hotplug vcpus group " prefix, \
testQemuHotplugCpuIndividual, ) < 0) \
 ret = -1; \
 } while (0)

-DO_TEST_CPU_INDIVIDUAL("x86_64", "x86-modern-individual-add", "7", true, 
true, false);
-DO_TEST_CPU_INDIVIDUAL("x86_64", "x86-modern-individual-add", "6,7", true, 
true, true);
-DO_TEST_CPU_INDIVIDUAL("x86_64", "x86-modern-individual-add", "7", false, 
true, true);
+DO_TEST_CPU_INDIVIDUAL("x86_64", "x86-modern-individual-add", "7", true, 
false);
+DO_TEST_CPU_INDIVIDUAL("x86_64", "x86-modern-individual-add", "6,7", true, 
true);
+DO_TEST_CPU_INDIVIDUAL("x86_64", "x86-modern-individual-add", "7", false, 
true);

-DO_TEST_CPU_INDIVIDUAL("ppc64", "ppc64-modern-individual", "16-23", true, 
true, false);
-DO_TEST_CPU_INDIVIDUAL("ppc64", "ppc64-modern-individual", "16-22", true, 
true, true);
-DO_TEST_CPU_INDIVIDUAL("ppc64", "ppc64-modern-individual", "17", true, 
true, true);
+DO_TEST_CPU_INDIVIDUAL("ppc64", 

[PATCH 2/9] qemu: capabilities: Export functions necessary for probing machine types

[Peter Krempa]

Upcoming patch will re-probe machines from the current qemu instance to
populate the private copy of qemuCaps after reconnecting to a running
instance. This is needed to be able to access the machine type data,
while storing them in the status XML seems to be an overkill, for
information which can be easily reprobed.

Export 'virQEMUCapsInitQMPArch' needed to populate the 'arch' field and
'virQEMUCapsProbeQMPMachineTypes'.

Signed-off-by: Peter Krempa 
---
 src/qemu/qemu_capabilities.c | 6 +++---
 src/qemu/qemu_capabilities.h | 9 +
 2 files changed, 12 insertions(+), 3 deletions(-)

diff --git a/src/qemu/qemu_capabilities.c b/src/qemu/qemu_capabilities.c
index 05cc11218a..86619ee92f 100644
--- a/src/qemu/qemu_capabilities.c
+++ b/src/qemu/qemu_capabilities.c
@@ -2798,7 +2798,7 @@ virQEMUCapsHasMachines(virQEMUCaps *qemuCaps)
 }


-static int
+int
 virQEMUCapsProbeQMPMachineTypes(virQEMUCaps *qemuCaps,
 virDomainVirtType virtType,
 qemuMonitor *mon)
@@ -5407,9 +5407,9 @@ virQEMUCapsIsValid(void *data,
  *
  * Returns: 0 on success, <0 on failure
  */
-static int
+int
 virQEMUCapsInitQMPArch(virQEMUCaps *qemuCaps,
-qemuMonitor *mon)
+   qemuMonitor *mon)
 {
 g_autofree char *archstr = NULL;

diff --git a/src/qemu/qemu_capabilities.h b/src/qemu/qemu_capabilities.h
index 2198b94cf2..620ebf779f 100644
--- a/src/qemu/qemu_capabilities.h
+++ b/src/qemu/qemu_capabilities.h
@@ -895,3 +895,12 @@ virQEMUCapsProbeHVF(virQEMUCaps *qemuCaps) G_NO_INLINE;

 virArch virQEMUCapsArchFromString(const char *arch);
 const char *virQEMUCapsArchToString(virArch arch);
+
+int
+virQEMUCapsInitQMPArch(virQEMUCaps *qemuCaps,
+   qemuMonitor *mon);
+
+int
+virQEMUCapsProbeQMPMachineTypes(virQEMUCaps *qemuCaps,
+virDomainVirtType virtType,
+qemuMonitor *mon);
-- 
2.41.0

[PATCH 8/9] qemu: capabilities: Retire QEMU_CAPS_QUERY_HOTPLUGGABLE_CPUS

[Peter Krempa]

All qemu versions have that command and cpu hotplug code now directly
probes the machine type.

Signed-off-by: Peter Krempa 
---
 src/qemu/qemu_capabilities.c  | 8 ++--
 src/qemu/qemu_capabilities.h  | 2 +-
 tests/qemucapabilitiesdata/caps_4.2.0_aarch64.xml | 1 -
 tests/qemucapabilitiesdata/caps_4.2.0_ppc64.xml   | 1 -
 tests/qemucapabilitiesdata/caps_4.2.0_s390x.xml   | 1 -
 tests/qemucapabilitiesdata/caps_4.2.0_x86_64.xml  | 1 -
 tests/qemucapabilitiesdata/caps_5.0.0_aarch64.xml | 1 -
 tests/qemucapabilitiesdata/caps_5.0.0_ppc64.xml   | 1 -
 tests/qemucapabilitiesdata/caps_5.0.0_riscv64.xml | 1 -
 tests/qemucapabilitiesdata/caps_5.0.0_x86_64.xml  | 1 -
 tests/qemucapabilitiesdata/caps_5.1.0_sparc.xml   | 1 -
 tests/qemucapabilitiesdata/caps_5.1.0_x86_64.xml  | 1 -
 tests/qemucapabilitiesdata/caps_5.2.0_aarch64.xml | 1 -
 tests/qemucapabilitiesdata/caps_5.2.0_ppc64.xml   | 1 -
 tests/qemucapabilitiesdata/caps_5.2.0_riscv64.xml | 1 -
 tests/qemucapabilitiesdata/caps_5.2.0_s390x.xml   | 1 -
 tests/qemucapabilitiesdata/caps_5.2.0_x86_64.xml  | 1 -
 tests/qemucapabilitiesdata/caps_6.0.0_aarch64.xml | 1 -
 tests/qemucapabilitiesdata/caps_6.0.0_s390x.xml   | 1 -
 tests/qemucapabilitiesdata/caps_6.0.0_x86_64.xml  | 1 -
 tests/qemucapabilitiesdata/caps_6.1.0_x86_64.xml  | 1 -
 tests/qemucapabilitiesdata/caps_6.2.0_aarch64.xml | 1 -
 tests/qemucapabilitiesdata/caps_6.2.0_ppc64.xml   | 1 -
 tests/qemucapabilitiesdata/caps_6.2.0_x86_64.xml  | 1 -
 tests/qemucapabilitiesdata/caps_7.0.0_aarch64+hvf.xml | 1 -
 tests/qemucapabilitiesdata/caps_7.0.0_aarch64.xml | 1 -
 tests/qemucapabilitiesdata/caps_7.0.0_ppc64.xml   | 1 -
 tests/qemucapabilitiesdata/caps_7.0.0_x86_64.xml  | 1 -
 tests/qemucapabilitiesdata/caps_7.1.0_ppc64.xml   | 1 -
 tests/qemucapabilitiesdata/caps_7.1.0_x86_64.xml  | 1 -
 tests/qemucapabilitiesdata/caps_7.2.0_ppc.xml | 1 -
 tests/qemucapabilitiesdata/caps_7.2.0_x86_64+hvf.xml  | 1 -
 tests/qemucapabilitiesdata/caps_7.2.0_x86_64.xml  | 1 -
 tests/qemucapabilitiesdata/caps_8.0.0_riscv64.xml | 1 -
 tests/qemucapabilitiesdata/caps_8.0.0_x86_64.xml  | 1 -
 tests/qemucapabilitiesdata/caps_8.1.0_s390x.xml   | 1 -
 tests/qemucapabilitiesdata/caps_8.1.0_x86_64.xml  | 1 -
 37 files changed, 3 insertions(+), 42 deletions(-)

diff --git a/src/qemu/qemu_capabilities.c b/src/qemu/qemu_capabilities.c
index 86619ee92f..0538acd05d 100644
--- a/src/qemu/qemu_capabilities.c
+++ b/src/qemu/qemu_capabilities.c
@@ -390,7 +390,7 @@ VIR_ENUM_IMPL(virQEMUCaps,
   /* 230 */
   "smm", /* X_QEMU_CAPS_MACHINE_SMM_OPT */
   "virtio-pci-disable-legacy", /* 
X_QEMU_CAPS_VIRTIO_PCI_DISABLE_LEGACY */
-  "query-hotpluggable-cpus", /* QEMU_CAPS_QUERY_HOTPLUGGABLE_CPUS 
*/
+  "query-hotpluggable-cpus", /* 
X_QEMU_CAPS_QUERY_HOTPLUGGABLE_CPUS */
   "virtio-net.rx_queue_size", /* 
X_QEMU_CAPS_VIRTIO_NET_RX_QUEUE_SIZE */
   "machine-iommu", /* X_QEMU_CAPS_MACHINE_IOMMU */

@@ -1209,7 +1209,6 @@ struct virQEMUCapsStringFlags {
 struct virQEMUCapsStringFlags virQEMUCapsCommands[] = {
 { "query-vnc", QEMU_CAPS_VNC },
 { "rtc-reset-reinjection", QEMU_CAPS_RTC_RESET_REINJECTION },
-{ "query-hotpluggable-cpus", QEMU_CAPS_QUERY_HOTPLUGGABLE_CPUS },
 { "query-cpu-model-expansion", QEMU_CAPS_QUERY_CPU_MODEL_EXPANSION },
 { "query-cpu-definitions", QEMU_CAPS_QUERY_CPU_DEFINITIONS },
 { "query-cpu-model-baseline", QEMU_CAPS_QUERY_CPU_MODEL_BASELINE },
@@ -5852,7 +5851,7 @@ static const struct virQEMUCapsMachineTypeFilter 
virQEMUCapsMachineFilter[] = {

 void
 virQEMUCapsFilterByMachineType(virQEMUCaps *qemuCaps,
-   virDomainVirtType virtType,
+   virDomainVirtType virtType G_GNUC_UNUSED,
const char *machineType)
 {
 size_t i;
@@ -5870,9 +5869,6 @@ virQEMUCapsFilterByMachineType(virQEMUCaps *qemuCaps,
 for (j = 0; j < filter->nflags; j++)
 virQEMUCapsClear(qemuCaps, filter->flags[j]);
 }
-
-if (!virQEMUCapsGetMachineHotplugCpus(qemuCaps, virtType, machineType))
-virQEMUCapsClear(qemuCaps, QEMU_CAPS_QUERY_HOTPLUGGABLE_CPUS);
 }


diff --git a/src/qemu/qemu_capabilities.h b/src/qemu/qemu_capabilities.h
index 620ebf779f..d68a819442 100644
--- a/src/qemu/qemu_capabilities.h
+++ b/src/qemu/qemu_capabilities.h
@@ -369,7 +369,7 @@ typedef enum { /* virQEMUCapsFlags grouping marker for 
syntax-check */
 /* 230 */
 X_QEMU_CAPS_MACHINE_SMM_OPT, /* -machine xxx,smm=on/off/auto */
 X_QEMU_CAPS_VIRTIO_PCI_DISABLE_LEGACY, /* virtio-*pci.disable-legacy */
-QEMU_CAPS_QUERY_HOTPLUGGABLE_CPUS, /* qmp command query-hotpluggable-cpus 
*/
+X_QEMU_CAPS_QUERY_HOTPLUGGABLE_CPUS, /* qmp command 
query-hotpluggable-cpus */
 

[PATCH 9/9] qemu: capabilities: Remove unused 'virQEMUCapsFilterByMachineType'

[Peter Krempa]

The filtering of qemu capabilities by machine type doesn't seem to be
ever used, remove it and adjust callers.

Signed-off-by: Peter Krempa 
---
 src/qemu/qemu_capabilities.c | 41 +---
 src/qemu/qemu_capabilities.h |  7 +-
 src/qemu/qemu_domain.c   |  4 +---
 src/qemu/qemu_process.c  |  4 +---
 4 files changed, 4 insertions(+), 52 deletions(-)

diff --git a/src/qemu/qemu_capabilities.c b/src/qemu/qemu_capabilities.c
index 0538acd05d..751702649f 100644
--- a/src/qemu/qemu_capabilities.c
+++ b/src/qemu/qemu_capabilities.c
@@ -5836,42 +5836,6 @@ virQEMUCapsLoadFile(const char *filename,
 }


-struct virQEMUCapsMachineTypeFilter {
-const char *machineType;
-virQEMUCapsFlags *flags;
-size_t nflags;
-};
-
-static const struct virQEMUCapsMachineTypeFilter virQEMUCapsMachineFilter[] = {
-/* { "blah", virQEMUCapsMachineBLAHFilter,
- G_N_ELEMENTS(virQEMUCapsMachineBLAHFilter) }, */
-{ "", NULL, 0 },
-};
-
-
-void
-virQEMUCapsFilterByMachineType(virQEMUCaps *qemuCaps,
-   virDomainVirtType virtType G_GNUC_UNUSED,
-   const char *machineType)
-{
-size_t i;
-
-if (!machineType)
-return;
-
-for (i = 0; i < G_N_ELEMENTS(virQEMUCapsMachineFilter); i++) {
-const struct virQEMUCapsMachineTypeFilter *filter = 
[i];
-size_t j;
-
-if (STRNEQ(filter->machineType, machineType))
-continue;
-
-for (j = 0; j < filter->nflags; j++)
-virQEMUCapsClear(qemuCaps, filter->flags[j]);
-}
-}
-
-
 virFileCacheHandlers qemuCapsCacheHandlers = {
 .isValid = virQEMUCapsIsValid,
 .newData = virQEMUCapsNewData,
@@ -5941,9 +5905,7 @@ virQEMUCapsCacheLookup(virFileCache *cache,

 virQEMUCaps *
 virQEMUCapsCacheLookupCopy(virFileCache *cache,
-   virDomainVirtType virtType,
-   const char *binary,
-   const char *machineType)
+   const char *binary)
 {
 virQEMUCaps *qemuCaps = virQEMUCapsCacheLookup(cache, binary);
 virQEMUCaps *ret;
@@ -5954,7 +5916,6 @@ virQEMUCapsCacheLookupCopy(virFileCache *cache,
 ret = virQEMUCapsNewCopy(qemuCaps);
 virObjectUnref(qemuCaps);

-virQEMUCapsFilterByMachineType(ret, virtType, machineType);
 return ret;
 }

diff --git a/src/qemu/qemu_capabilities.h b/src/qemu/qemu_capabilities.h
index d68a819442..ba289ba6b9 100644
--- a/src/qemu/qemu_capabilities.h
+++ b/src/qemu/qemu_capabilities.h
@@ -783,9 +783,6 @@ const char *virQEMUCapsGetMachineDefaultRAMid(virQEMUCaps 
*qemuCaps,
   virDomainVirtType virtType,
   const char *name);

-void virQEMUCapsFilterByMachineType(virQEMUCaps *qemuCaps,
-virDomainVirtType virtType,
-const char *machineType);
 char * virQEMUCapsGetDefaultEmulator(virArch hostarch,
  virArch guestarch);

@@ -796,9 +793,7 @@ virFileCache *virQEMUCapsCacheNew(const char *libDir,
 virQEMUCaps *virQEMUCapsCacheLookup(virFileCache *cache,
   const char *binary);
 virQEMUCaps *virQEMUCapsCacheLookupCopy(virFileCache *cache,
-  virDomainVirtType virtType,
-  const char *binary,
-  const char *machineType);
+  const char *binary);
 virQEMUCaps *virQEMUCapsCacheLookupDefault(virFileCache *cache,
  const char *binary,
  const char *archStr,
diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c
index 39ad97f1b1..3ef8ca33b6 100644
--- a/src/qemu/qemu_domain.c
+++ b/src/qemu/qemu_domain.c
@@ -6622,9 +6622,7 @@ qemuDomainDefFormatBufInternal(virQEMUDriver *driver,
 qCaps = virObjectRef(qemuCaps);
 } else {
 if (!(qCaps = virQEMUCapsCacheLookupCopy(driver->qemuCapsCache,
- def->virtType,
- def->emulator,
- def->os.machine)))
+ def->emulator)))
 return -1;
 }

diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c
index 3e1198a564..14d582e77f 100644
--- a/src/qemu/qemu_process.c
+++ b/src/qemu/qemu_process.c
@@ -5659,9 +5659,7 @@ qemuProcessPrepareQEMUCaps(virDomainObj *vm,

 virObjectUnref(priv->qemuCaps);
 if (!(priv->qemuCaps = virQEMUCapsCacheLookupCopy(qemuCapsCache,
-  vm->def->virtType,
-  vm->def->emulator,
-   

[PATCH 3/9] qemu: process: Probe machine type data on reconnect to qemu

[Peter Krempa]

When reconnecting we populate only the capability flags from the XML as
we need to know the exact flags that were present when starting the VM.

On the other hand the machine type data is not stored as it wasn't
really used after startup. While storing all of the data into the status
XML would be theoretically possible, with machine-type specific data it
makes no sense to do so, and thus the data can be re-probed from the
current instance.

Signed-off-by: Peter Krempa 
---
 src/qemu/qemu_process.c | 37 +
 1 file changed, 37 insertions(+)

diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c
index a6ed69cfe2..3e1198a564 100644
--- a/src/qemu/qemu_process.c
+++ b/src/qemu/qemu_process.c
@@ -8816,6 +8816,38 @@ qemuProcessRefreshCPU(virQEMUDriver *driver,
 }


+/**
+ * qemuProcessReloadMachineTypes:
+ *
+ * Reload machine type information into the 'qemuCaps' object from the current
+ * qemu.
+ */
+static int
+qemuProcessReloadMachineTypes(virDomainObj *vm)
+{
+qemuDomainObjPrivate *priv = vm->privateData;
+bool fail = false;
+
+qemuDomainObjEnterMonitor(vm);
+
+if (virQEMUCapsInitQMPArch(priv->qemuCaps, priv->mon) < 0)
+fail = true;
+
+if (!fail &&
+virQEMUCapsProbeQMPMachineTypes(priv->qemuCaps,
+vm->def->virtType,
+priv->mon) < 0)
+fail = true;
+
+qemuDomainObjExitMonitor(vm);
+
+if (fail)
+return -1;
+
+return 0;
+}
+
+
 struct qemuProcessReconnectData {
 virQEMUDriver *driver;
 virDomainObj *obj;
@@ -8950,6 +8982,11 @@ qemuProcessReconnect(void *opaque)
 goto error;
 }

+/* Reload and populate machine type data into 'qemuCaps' as that is not
+ * serialized into the status XML. */
+if (qemuProcessReloadMachineTypes(obj) < 0)
+goto error;
+
 if (qemuDomainAssignAddresses(obj->def, priv->qemuCaps,
   driver, obj, false) < 0) {
 goto error;
-- 
2.41.0

[PATCH 5/9] tests: qemuhotplugtest: Fix arch-specific parts of 'ppc64' test XMLs

[Peter Krempa]

The tests were using a copy of a x86_64 based XML and thus
'qemuhotplugtest' was selecting wrong capabilities to use for that
specific test.

Signed-off-by: Peter Krempa 
---
 .../ppc64-modern-bulk-domain.xml  |  4 ++--
 .../ppc64-modern-bulk-result-conf.xml | 18 ++
 .../ppc64-modern-bulk-result-live.xml | 19 ---
 .../ppc64-modern-individual-domain.xml|  4 ++--
 .../ppc64-modern-individual-result-conf.xml   | 18 ++
 .../ppc64-modern-individual-result-live.xml   | 19 ---
 6 files changed, 40 insertions(+), 42 deletions(-)

diff --git a/tests/qemuhotplugtestcpus/ppc64-modern-bulk-domain.xml 
b/tests/qemuhotplugtestcpus/ppc64-modern-bulk-domain.xml
index cc0322762a..f7c754ccf7 100644
--- a/tests/qemuhotplugtestcpus/ppc64-modern-bulk-domain.xml
+++ b/tests/qemuhotplugtestcpus/ppc64-modern-bulk-domain.xml
@@ -5,7 +5,7 @@
   219100
   32
   
-hvm
+hvm
 
   
   
@@ -15,6 +15,6 @@
   restart
   destroy
   
-  /usr/bin/qemu-system-x86_64
+  /usr/bin/qemu-system-ppc64
   
 
diff --git a/tests/qemuhotplugtestcpus/ppc64-modern-bulk-result-conf.xml 
b/tests/qemuhotplugtestcpus/ppc64-modern-bulk-result-conf.xml
index afb2c85bc2..ad11b2f8a6 100644
--- a/tests/qemuhotplugtestcpus/ppc64-modern-bulk-result-conf.xml
+++ b/tests/qemuhotplugtestcpus/ppc64-modern-bulk-result-conf.xml
@@ -39,11 +39,11 @@
 
   
   
-hvm
+hvm
 
   
   
-qemu64
+POWER9
 
   
   
@@ -51,16 +51,18 @@
   restart
   destroy
   
-/usr/bin/qemu-system-x86_64
-
-  
+/usr/bin/qemu-system-ppc64
+
+  
+
+
+  
+  
 
-
-
-
 
 
   
 
+
   
 
diff --git a/tests/qemuhotplugtestcpus/ppc64-modern-bulk-result-live.xml 
b/tests/qemuhotplugtestcpus/ppc64-modern-bulk-result-live.xml
index d86247d70c..2a3b4a495f 100644
--- a/tests/qemuhotplugtestcpus/ppc64-modern-bulk-result-live.xml
+++ b/tests/qemuhotplugtestcpus/ppc64-modern-bulk-result-live.xml
@@ -39,11 +39,11 @@
 
   
   
-hvm
+hvm
 
   
   
-qemu64
+POWER9
 
   
   
@@ -51,24 +51,21 @@
   restart
   destroy
   
-/usr/bin/qemu-system-x86_64
-
+/usr/bin/qemu-system-ppc64
+
   
-  
+  
 
 
+  
+  
   
 
-
-  
-
-
-  
-
 
 
   
   
 
+
   
 
diff --git a/tests/qemuhotplugtestcpus/ppc64-modern-individual-domain.xml 
b/tests/qemuhotplugtestcpus/ppc64-modern-individual-domain.xml
index cc0322762a..f7c754ccf7 100644
--- a/tests/qemuhotplugtestcpus/ppc64-modern-individual-domain.xml
+++ b/tests/qemuhotplugtestcpus/ppc64-modern-individual-domain.xml
@@ -5,7 +5,7 @@
   219100
   32
   
-hvm
+hvm
 
   
   
@@ -15,6 +15,6 @@
   restart
   destroy
   
-  /usr/bin/qemu-system-x86_64
+  /usr/bin/qemu-system-ppc64
   
 
diff --git a/tests/qemuhotplugtestcpus/ppc64-modern-individual-result-conf.xml 
b/tests/qemuhotplugtestcpus/ppc64-modern-individual-result-conf.xml
index e8d30f7ddd..34aec9b965 100644
--- a/tests/qemuhotplugtestcpus/ppc64-modern-individual-result-conf.xml
+++ b/tests/qemuhotplugtestcpus/ppc64-modern-individual-result-conf.xml
@@ -39,11 +39,11 @@
 
   
   
-hvm
+hvm
 
   
   
-qemu64
+POWER9
 
   
   
@@ -51,16 +51,18 @@
   restart
   destroy
   
-/usr/bin/qemu-system-x86_64
-
-  
+/usr/bin/qemu-system-ppc64
+
+  
+
+
+  
+  
 
-
-
-
 
 
   
 
+
   
 
diff --git a/tests/qemuhotplugtestcpus/ppc64-modern-individual-result-live.xml 
b/tests/qemuhotplugtestcpus/ppc64-modern-individual-result-live.xml
index afc500657c..5ce2cfd0b0 100644
--- a/tests/qemuhotplugtestcpus/ppc64-modern-individual-result-live.xml
+++ b/tests/qemuhotplugtestcpus/ppc64-modern-individual-result-live.xml
@@ -39,11 +39,11 @@
 
   
   
-hvm
+hvm
 
   
   
-qemu64
+POWER9
 
   
   
@@ -51,24 +51,21 @@
   restart
   destroy
   
-/usr/bin/qemu-system-x86_64
-
+/usr/bin/qemu-system-ppc64
+
   
-  
+  
 
 
+  
+  
   
 
-
-  
-
-
-  
-
 
 
   
   
 
+
   
 
-- 
2.41.0

[PATCH 7/9] qemuDomainSupportsVcpuHotplug: Base return value on virQEMUCapsGetMachineHotplugCpus

[Peter Krempa]

The QEMU_CAPS_QUERY_HOTPLUGGABLE_CPUS flag is always asserted as all
qemu versions support the command and selectively cleared when copying
the capabilities for VM use if given machine type does not support cpu
hotplug.

Rework this to directly probe the machine as we now populate the data
also when re-connecting to a qemu instance after daemon restart, so that
the capability can be removed.

Signed-off-by: Peter Krempa 
---
 src/qemu/qemu_domain.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c
index 49ae44fcbf..39ad97f1b1 100644
--- a/src/qemu/qemu_domain.c
+++ b/src/qemu/qemu_domain.c
@@ -10002,7 +10002,9 @@ qemuDomainSupportsVcpuHotplug(virDomainObj *vm)
 {
 qemuDomainObjPrivate *priv = vm->privateData;

-return virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_QUERY_HOTPLUGGABLE_CPUS);
+return virQEMUCapsGetMachineHotplugCpus(priv->qemuCaps,
+vm->def->virtType,
+vm->def->os.machine);
 }


-- 
2.41.0

[PATCH 4/9] qemuxml2argvtest: Modernize 'cpu-hotplug-startup' case

[Peter Krempa]

Signed-off-by: Peter Krempa 
---
 ...args => cpu-hotplug-startup.x86_64-latest.args} | 14 --
 tests/qemuxml2argvtest.c   |  2 +-
 2 files changed, 9 insertions(+), 7 deletions(-)
 rename tests/qemuxml2argvdata/{cpu-hotplug-startup.args => 
cpu-hotplug-startup.x86_64-latest.args} (55%)

diff --git a/tests/qemuxml2argvdata/cpu-hotplug-startup.args 
b/tests/qemuxml2argvdata/cpu-hotplug-startup.x86_64-latest.args
similarity index 55%
rename from tests/qemuxml2argvdata/cpu-hotplug-startup.args
rename to tests/qemuxml2argvdata/cpu-hotplug-startup.x86_64-latest.args
index a870d56792..009c08d71a 100644
--- a/tests/qemuxml2argvdata/cpu-hotplug-startup.args
+++ b/tests/qemuxml2argvdata/cpu-hotplug-startup.x86_64-latest.args
@@ -9,12 +9,14 @@ 
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain--1-QEMUGuest1/.config \
 /usr/bin/qemu-system-x86_64 \
 -name guest=QEMUGuest1,debug-threads=on \
 -S \
--object 
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain--1-QEMUGuest1/master-key.aes
 \
--machine pc,usb=off,dump-guest-core=off \
+-object 
'{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain--1-QEMUGuest1/master-key.aes"}'
 \
+-machine pc,usb=off,dump-guest-core=off,memory-backend=pc.ram,acpi=off \
 -accel tcg \
+-cpu qemu64 \
 -m size=219136k \
+-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":224395264}' \
 -overcommit mem-lock=off \
--smp 1,maxcpus=6,sockets=3,cores=2,threads=1 \
+-smp 1,maxcpus=6,sockets=3,dies=1,cores=2,threads=1 \
 -uuid c7a5fdbd-edaf-9455-926a-d65c16db1809 \
 -display none \
 -no-user-config \
@@ -23,9 +25,9 @@ 
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain--1-QEMUGuest1/.config \
 -mon chardev=charmonitor,id=monitor,mode=control \
 -rtc base=utc \
 -no-shutdown \
--no-acpi \
 -boot strict=on \
--usb \
+-device 
'{"driver":"piix3-usb-uhci","id":"usb","bus":"pci.0","addr":"0x1.0x2"}' \
 -audiodev '{"id":"audio1","driver":"none"}' \
--device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x2 \
+-device 
'{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.0","addr":"0x2"}' \
+-sandbox 
on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
 -msg timestamp=on
diff --git a/tests/qemuxml2argvtest.c b/tests/qemuxml2argvtest.c
index 6e63f8a2eb..1e6cbbf028 100644
--- a/tests/qemuxml2argvtest.c
+++ b/tests/qemuxml2argvtest.c
@@ -2513,7 +2513,7 @@ mymain(void)
 DO_TEST_CAPS_LATEST_PARSE_ERROR("virtio-iommu-invalid-address-type");
 DO_TEST_CAPS_LATEST_PARSE_ERROR("virtio-iommu-invalid-address");

-DO_TEST("cpu-hotplug-startup", QEMU_CAPS_QUERY_HOTPLUGGABLE_CPUS);
+DO_TEST_CAPS_LATEST("cpu-hotplug-startup");
 DO_TEST_CAPS_ARCH_LATEST_PARSE_ERROR("cpu-hotplug-granularity", "ppc64");

 DO_TEST_CAPS_LATEST("virtio-options");
-- 
2.41.0

[PATCH 1/9] qemu: Rename qemuDomainSupportsNewVcpuHotplug to qemuDomainSupportsVcpuHotplug

[Peter Krempa]

Support for legacy cpu hotplug was removed a long time ago. At this
point this function only checks whether the current machine type
supports cpu hotplug.

Signed-off-by: Peter Krempa 
---
 src/qemu/qemu_domain.c  | 4 ++--
 src/qemu/qemu_domain.h  | 2 +-
 src/qemu/qemu_hotplug.c | 4 ++--
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c
index bfeddc7746..49ae44fcbf 100644
--- a/src/qemu/qemu_domain.c
+++ b/src/qemu/qemu_domain.c
@@ -9998,7 +9998,7 @@ qemuDomainValidateVcpuInfo(virDomainObj *vm)


 bool
-qemuDomainSupportsNewVcpuHotplug(virDomainObj *vm)
+qemuDomainSupportsVcpuHotplug(virDomainObj *vm)
 {
 qemuDomainObjPrivate *priv = vm->privateData;

@@ -10035,7 +10035,7 @@ qemuDomainRefreshVcpuInfo(virDomainObj *vm,
 int rc;
 int ret = -1;

-hotplug = qemuDomainSupportsNewVcpuHotplug(vm);
+hotplug = qemuDomainSupportsVcpuHotplug(vm);
 VIR_DEBUG("Maxvcpus %zu hotplug %d", maxvcpus, hotplug);

 if (qemuDomainObjEnterMonitorAsync(vm, asyncJob) < 0)
diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h
index 786f239495..15d1bb4900 100644
--- a/src/qemu/qemu_domain.h
+++ b/src/qemu/qemu_domain.h
@@ -860,7 +860,7 @@ int qemuDomainSetMaxMemLock(virDomainObj *vm,
 int qemuDomainDefValidateMemoryHotplug(const virDomainDef *def,
const virDomainMemoryDef *mem);

-bool qemuDomainSupportsNewVcpuHotplug(virDomainObj *vm);
+bool qemuDomainSupportsVcpuHotplug(virDomainObj *vm);
 bool qemuDomainHasVcpuPids(virDomainObj *vm);
 pid_t qemuDomainGetVcpuPid(virDomainObj *vm, unsigned int vcpuid);
 int qemuDomainValidateVcpuInfo(virDomainObj *vm);
diff --git a/src/qemu/qemu_hotplug.c b/src/qemu/qemu_hotplug.c
index d7d1db8ecc..d845731aee 100644
--- a/src/qemu/qemu_hotplug.c
+++ b/src/qemu/qemu_hotplug.c
@@ -6400,7 +6400,7 @@ qemuDomainHotplugAddVcpu(virQEMUDriver *driver,
 size_t i;
 bool vcpuTidMissing = false;

-if (!qemuDomainSupportsNewVcpuHotplug(vm)) {
+if (!qemuDomainSupportsVcpuHotplug(vm)) {
 virReportError(VIR_ERR_OPERATION_UNSUPPORTED, "%s",
_("cpu hotplug is not supported"));
 return -1;
@@ -6846,7 +6846,7 @@ qemuDomainSetVcpuInternal(virQEMUDriver *driver,
 g_autoptr(virBitmap) livevcpus = NULL;

 if (def) {
-if (!qemuDomainSupportsNewVcpuHotplug(vm)) {
+if (!qemuDomainSupportsVcpuHotplug(vm)) {
 virReportError(VIR_ERR_OPERATION_UNSUPPORTED, "%s",
_("this qemu version does not support specific "
  "vCPU hotplug"));
-- 
2.41.0

[PATCH 0/9] qemu: Rework probing of vCPU hotplug support

[Peter Krempa]

Nowadays all qemu's support the command which was used as witness, but
was gated on machine type's support of vCPU hotplug.  Directly probe the
machine type.


Peter Krempa (9):
  qemu: Rename qemuDomainSupportsNewVcpuHotplug to
qemuDomainSupportsVcpuHotplug
  qemu: capabilities: Export functions necessary for probing machine
types
  qemu: process: Probe machine type data on reconnect to qemu
  qemuxml2argvtest: Modernize 'cpu-hotplug-startup' case
  tests: qemuhotplugtest: Fix arch-specific parts of 'ppc64' test XMLs
  qemuhotplugtest: Remove 'modern' field for cpu hotplug tests
  qemuDomainSupportsVcpuHotplug: Base return value on
virQEMUCapsGetMachineHotplugCpus
  qemu: capabilities: Retire QEMU_CAPS_QUERY_HOTPLUGGABLE_CPUS
  qemu: capabilities: Remove unused 'virQEMUCapsFilterByMachineType'

 src/qemu/qemu_capabilities.c  | 53 ++-
 src/qemu/qemu_capabilities.h  | 18 ---
 src/qemu/qemu_domain.c| 12 ++---
 src/qemu/qemu_domain.h|  2 +-
 src/qemu/qemu_hotplug.c   |  4 +-
 src/qemu/qemu_process.c   | 41 --
 .../caps_4.2.0_aarch64.xml|  1 -
 .../qemucapabilitiesdata/caps_4.2.0_ppc64.xml |  1 -
 .../qemucapabilitiesdata/caps_4.2.0_s390x.xml |  1 -
 .../caps_4.2.0_x86_64.xml |  1 -
 .../caps_5.0.0_aarch64.xml|  1 -
 .../qemucapabilitiesdata/caps_5.0.0_ppc64.xml |  1 -
 .../caps_5.0.0_riscv64.xml|  1 -
 .../caps_5.0.0_x86_64.xml |  1 -
 .../qemucapabilitiesdata/caps_5.1.0_sparc.xml |  1 -
 .../caps_5.1.0_x86_64.xml |  1 -
 .../caps_5.2.0_aarch64.xml|  1 -
 .../qemucapabilitiesdata/caps_5.2.0_ppc64.xml |  1 -
 .../caps_5.2.0_riscv64.xml|  1 -
 .../qemucapabilitiesdata/caps_5.2.0_s390x.xml |  1 -
 .../caps_5.2.0_x86_64.xml |  1 -
 .../caps_6.0.0_aarch64.xml|  1 -
 .../qemucapabilitiesdata/caps_6.0.0_s390x.xml |  1 -
 .../caps_6.0.0_x86_64.xml |  1 -
 .../caps_6.1.0_x86_64.xml |  1 -
 .../caps_6.2.0_aarch64.xml|  1 -
 .../qemucapabilitiesdata/caps_6.2.0_ppc64.xml |  1 -
 .../caps_6.2.0_x86_64.xml |  1 -
 .../caps_7.0.0_aarch64+hvf.xml|  1 -
 .../caps_7.0.0_aarch64.xml|  1 -
 .../qemucapabilitiesdata/caps_7.0.0_ppc64.xml |  1 -
 .../caps_7.0.0_x86_64.xml |  1 -
 .../qemucapabilitiesdata/caps_7.1.0_ppc64.xml |  1 -
 .../caps_7.1.0_x86_64.xml |  1 -
 tests/qemucapabilitiesdata/caps_7.2.0_ppc.xml |  1 -
 .../caps_7.2.0_x86_64+hvf.xml |  1 -
 .../caps_7.2.0_x86_64.xml |  1 -
 .../caps_8.0.0_riscv64.xml|  1 -
 .../caps_8.0.0_x86_64.xml |  1 -
 .../qemucapabilitiesdata/caps_8.1.0_s390x.xml |  1 -
 .../caps_8.1.0_x86_64.xml |  1 -
 tests/qemuhotplugtest.c   | 40 +-
 .../ppc64-modern-bulk-domain.xml  |  4 +-
 .../ppc64-modern-bulk-result-conf.xml | 18 ---
 .../ppc64-modern-bulk-result-live.xml | 19 +++
 .../ppc64-modern-individual-domain.xml|  4 +-
 .../ppc64-modern-individual-result-conf.xml   | 18 ---
 .../ppc64-modern-individual-result-live.xml   | 19 +++
 ...=> cpu-hotplug-startup.x86_64-latest.args} | 14 ++---
 tests/qemuxml2argvtest.c  |  2 +-
 50 files changed, 126 insertions(+), 177 deletions(-)
 rename tests/qemuxml2argvdata/{cpu-hotplug-startup.args => 
cpu-hotplug-startup.x86_64-latest.args} (55%)

-- 
2.41.0

[libvirt PATCH] ci: lcitool: Add libvirt-tck+runtime deps list

[Erik Skultety]

This change was supposed to be part of commit 120a674f , but was
proposed against the libvirt TCK project instead. Since we're running
the TCK test suite as part of this project, this is the right place for
the TCK runtime deps list config.

Signed-off-by: Erik Skultety 
---
 ci/lcitool/projects/libvirt-tck+runtime.yml | 32 +
 1 file changed, 32 insertions(+)
 create mode 100644 ci/lcitool/projects/libvirt-tck+runtime.yml

diff --git a/ci/lcitool/projects/libvirt-tck+runtime.yml 
b/ci/lcitool/projects/libvirt-tck+runtime.yml
new file mode 100644
index 00..6872cb8c2c
--- /dev/null
+++ b/ci/lcitool/projects/libvirt-tck+runtime.yml
@@ -0,0 +1,32 @@
+---
+packages:
+  - bc
+  - libguestfs-tools
+  - perl
+  - perl-Archive-Tar
+  - perl-Digest
+  - perl-Digest-MD5
+  - perl-File-Slurp
+  - perl-IO-Compress-Bzip2
+  - perl-IO-Interface
+  - perl-IO-String
+  - perl-Module-Build
+  - perl-Net-OpenSSH
+  - perl-NetAddr-IP
+  - perl-Sub-Uplevel
+  - perl-TAP-Formatter-HTML
+  - perl-TAP-Formatter-JUnit
+  - perl-TAP-Harness-Archive
+  - perl-Test-Exception
+  - perl-Test-LWP-UserAgent
+  - perl-Test-Pod
+  - perl-Test-Pod-Coverage
+  - perl-Test-Simple
+  - perl-XML-Twig
+  - perl-XML-Writer
+  - perl-XML-XPath
+  - perl-YAML
+  - perl-accessors
+  - perl-generators
+  - rpmbuild
+  - tcpdump
-- 
2.41.0

Entering freeze for libvirt-9.7.0

[Jiri Denemark]

I have just tagged v9.7.0-rc1 in the repository and pushed signed
tarballs and source RPMs to https://download.libvirt.org/

Please give the release candidate some testing and in case you find a
serious issue which should have a fix in the upcoming release, feel
free to reply to this thread to make sure the issue is more visible.

If you have not done so yet, please update NEWS.rst to document any
significant change you made since the last release.

Thanks,

Jirka