This is the eighth version of this patch series. See
https://bugzilla.redhat.com/show_bug.cgi?id=2016527 for more information.

Note that testing this requires selinux policy changes which are not fully
done, but there is a new policy in development that has allowed me to run with
selinux in enforcing mode for the common cases. See
https://bugzilla.redhat.com/show_bug.cgi?id=2182505 for more information. The
following scenarios should work now with selinux enabled using the selinux
policy from that bug:
 - http/https disks
 - ssh disks with password authentication
 - ssh disks with passwordless keyfile

The one major thing that doesn't work and is difficult to get working with
selinux enabled is the ssh-agent. This is because there doesn't seem to be any
selinux policy for ssh-agent, so by default the ssh-agent socket is labeled
unconfined_t. We cannot allow access from the libvirt/qemu to unconfined_t
because that would open up access to just about anything on the host. So
additional work will likely be necessary for ssh-agent/libvirt interaction in
the future. Fortunately ssh-agent is something that never was really supported
with the old qemu block driver either, so I think we could potentially merge
this patchset either without the ssh-agent patches or with a note that
ssh-agent won't work with selinux enabled.

Changes in v8:
 - Hopefully addressed all of Peter's issues, in addition to:
 - updated documentation to say 9.8.0, since 9.7.0 is currently in freeze
 - used WITH_NBDKIT instead of WITH_DECL_SYS_PIDFD_OPEN to make the code a bit
   more concise and understandable
 - enabled ci by adding libnbd to the dependencies, which uncovered a couple
   additional minor issues with those platforms that don't support the
   pidfd_open syscall
   - don't run nbdkit tests when WITH_NBDKIT is not defined
   - avoid warnings with unused function arguments
   - note that the ubuntu containers are currently failing due to a
     LeakSanitizer error, but I haven't reproduced it locally and can't figure
     out how to get better information from the leak sanitizer. Pointers
     appreciated: https://gitlab.com/jjongsma/libvirt/-/jobs/4991631193
 - One change of note is a new patch "qemu: improve error handling when
   restarting nbdkit". In order to provide better error reporting to the
   user and avoid VIR_WARN as suggested by Peter, some functions now return an
   error and this error is propagated up to qemuProcessReconnect(). This could
   potentially result in running domains being killed upon a libvirt restart,
   but only if they were in a state where they were was not a running nbdkit
   backend or libvirt couldn't monitor the process nbdkit.

Jonathon Jongsma (37):
  schema: allow 'ssh' as a protocol for network disks
  qemu: Add functions for determining nbdkit availability
  qemu: expand nbdkit capabilities
  util: Allow virFileCache data to be any GObject
  qemu: implement basic virFileCache for nbdkit caps
  qemu: implement persistent file cache for nbdkit caps
  qemu: use file cache for nbdkit caps
  qemu: Add qemuNbdkitProcess
  qemu: query nbdkit module dir from binary
  qemu: add functions to start and stop nbdkit
  Generalize qemuDomainLogContextNew()
  qemu: Extract qemuDomainLogContext into a new file
  qemu: move qemuProcessReadLog() to qemuLogContext
  qemu: log error output from nbdkit
  tests: add ability to test various nbdkit capabilities
  qemu: split qemuDomainSecretStorageSourcePrepare
  qemu: include nbdkit state in private xml
  util: secure erase virCommand send buffers
  qemu: pass sensitive data to nbdkit via pipe
  qemu: use nbdkit to serve network disks if available
  util: make virCommandSetSendBuffer testable
  tests: add tests for nbdkit invocation
  qemu: add test for authenticating a https network disk
  qemu: Add Taint for nbdkit restart failure
  qemu: Monitor nbdkit process for exit
  qemu: improve error handling when restarting nbdkit
  qemu: try to connect to nbdkit early to detect errors
  schema: add password configuration for ssh disk
  qemu: implement password auth for ssh disks with nbdkit
  schema: add configuration for host verification of ssh disks
  qemu: implement knownHosts for ssh disks with nbdkit
  schema: add keyfile configuration for ssh disks
  qemu: implement keyfile auth for ssh disks with nbdkit
  schema: add ssh-agent configuration for ssh disks
  qemu: implement ssh-agent auth for ssh disks with nbdkit
  rpm: update spec file for for nbdkit support
  ci: add libnbd to build

 build-aux/syntax-check.mk                     |    2 +-
 ci/buildenv/almalinux-8.sh                    |    1 +
 ci/buildenv/centos-stream-8.sh                |    1 +
 ci/buildenv/centos-stream-9.sh                |    1 +
 ci/buildenv/debian-12-cross-aarch64.sh        |    1 +
 ci/buildenv/debian-12-cross-armv6l.sh         |    1 +
 ci/buildenv/debian-12-cross-armv7l.sh         |    1 +
 ci/buildenv/debian-12-cross-i686.sh           |    1 +
 ci/buildenv/debian-12-cross-mips64el.sh       |    1 +
 ci/buildenv/debian-12-cross-mipsel.sh         |    1 +
 ci/buildenv/debian-12-cross-ppc64le.sh        |    1 +
 ci/buildenv/debian-12-cross-s390x.sh          |    1 +
 ci/buildenv/debian-12.sh                      |    1 +
 ci/buildenv/debian-sid-cross-aarch64.sh       |    1 +
 ci/buildenv/debian-sid-cross-armv6l.sh        |    1 +
 ci/buildenv/debian-sid-cross-armv7l.sh        |    1 +
 ci/buildenv/debian-sid-cross-i686.sh          |    1 +
 ci/buildenv/debian-sid-cross-mips64el.sh      |    1 +
 ci/buildenv/debian-sid-cross-mipsel.sh        |    1 +
 ci/buildenv/debian-sid-cross-ppc64le.sh       |    1 +
 ci/buildenv/debian-sid-cross-s390x.sh         |    1 +
 ci/buildenv/debian-sid.sh                     |    1 +
 ci/buildenv/fedora-37.sh                      |    1 +
 ci/buildenv/fedora-38-cross-mingw32.sh        |    1 +
 ci/buildenv/fedora-38-cross-mingw64.sh        |    1 +
 ci/buildenv/fedora-38.sh                      |    1 +
 ci/buildenv/fedora-rawhide-cross-mingw32.sh   |    1 +
 ci/buildenv/fedora-rawhide-cross-mingw64.sh   |    1 +
 ci/buildenv/fedora-rawhide.sh                 |    1 +
 ci/buildenv/opensuse-leap-15.sh               |    1 +
 ci/buildenv/opensuse-tumbleweed.sh            |    1 +
 ci/buildenv/ubuntu-2204.sh                    |    1 +
 ci/containers/almalinux-8.Dockerfile          |    1 +
 ci/containers/centos-stream-8.Dockerfile      |    1 +
 ci/containers/centos-stream-9.Dockerfile      |    1 +
 .../debian-12-cross-aarch64.Dockerfile        |    1 +
 .../debian-12-cross-armv6l.Dockerfile         |    1 +
 .../debian-12-cross-armv7l.Dockerfile         |    1 +
 ci/containers/debian-12-cross-i686.Dockerfile |    1 +
 .../debian-12-cross-mips64el.Dockerfile       |    1 +
 .../debian-12-cross-mipsel.Dockerfile         |    1 +
 .../debian-12-cross-ppc64le.Dockerfile        |    1 +
 .../debian-12-cross-s390x.Dockerfile          |    1 +
 ci/containers/debian-12.Dockerfile            |    1 +
 .../debian-sid-cross-aarch64.Dockerfile       |    1 +
 .../debian-sid-cross-armv6l.Dockerfile        |    1 +
 .../debian-sid-cross-armv7l.Dockerfile        |    1 +
 .../debian-sid-cross-i686.Dockerfile          |    1 +
 .../debian-sid-cross-mips64el.Dockerfile      |    1 +
 .../debian-sid-cross-mipsel.Dockerfile        |    1 +
 .../debian-sid-cross-ppc64le.Dockerfile       |    1 +
 .../debian-sid-cross-s390x.Dockerfile         |    1 +
 ci/containers/debian-sid.Dockerfile           |    1 +
 ci/containers/fedora-37.Dockerfile            |    1 +
 .../fedora-38-cross-mingw32.Dockerfile        |    1 +
 .../fedora-38-cross-mingw64.Dockerfile        |    1 +
 ci/containers/fedora-38.Dockerfile            |    1 +
 .../fedora-rawhide-cross-mingw32.Dockerfile   |    1 +
 .../fedora-rawhide-cross-mingw64.Dockerfile   |    1 +
 ci/containers/fedora-rawhide.Dockerfile       |    1 +
 ci/containers/opensuse-leap-15.Dockerfile     |    1 +
 ci/containers/opensuse-tumbleweed.Dockerfile  |    1 +
 ci/containers/ubuntu-2204.Dockerfile          |    1 +
 ci/lcitool/projects/libvirt.yml               |    1 +
 docs/formatdomain.rst                         |   45 +-
 libvirt.spec.in                               |    8 +
 meson.build                                   |   18 +
 meson_options.txt                             |    1 +
 po/POTFILES                                   |    2 +
 src/conf/domain_conf.c                        |   38 +
 src/conf/domain_conf.h                        |    1 +
 src/conf/schemas/domaincommon.rng             |   55 +
 src/conf/storage_source_conf.c                |    6 +
 src/conf/storage_source_conf.h                |    6 +-
 src/libvirt_private.syms                      |    1 +
 src/qemu/meson.build                          |    3 +
 src/qemu/qemu_block.c                         |  162 ++-
 src/qemu/qemu_conf.c                          |   22 +
 src/qemu/qemu_conf.h                          |    6 +
 src/qemu/qemu_domain.c                        |  436 +++---
 src/qemu/qemu_domain.h                        |   31 +-
 src/qemu/qemu_driver.c                        |   20 +
 src/qemu/qemu_extdevice.c                     |   62 +
 src/qemu/qemu_hotplug.c                       |    7 +
 src/qemu/qemu_logcontext.c                    |  329 +++++
 src/qemu/qemu_logcontext.h                    |   41 +
 src/qemu/qemu_nbdkit.c                        | 1291 +++++++++++++++++
 src/qemu/qemu_nbdkit.h                        |  119 ++
 src/qemu/qemu_nbdkitpriv.h                    |   31 +
 src/qemu/qemu_process.c                       |  126 +-
 src/qemu/qemu_process.h                       |    3 +
 src/util/vircommand.c                         |   19 +-
 src/util/vircommand.h                         |    8 +
 src/util/vircommandpriv.h                     |    4 +
 src/util/virfilecache.c                       |   14 +-
 src/util/virfilecache.h                       |    2 +-
 tests/meson.build                             |    5 +
 tests/qemublocktest.c                         |    2 +-
 ...w2-invalid.json => network-ssh-qcow2.json} |    0
 ...cow2-invalid.xml => network-ssh-qcow2.xml} |    0
 .../disk-cdrom-network.args.disk0             |    6 +
 .../disk-cdrom-network.args.disk1             |    8 +
 .../disk-cdrom-network.args.disk1.pipe.778    |    1 +
 .../disk-cdrom-network.args.disk2             |    8 +
 .../disk-cdrom-network.args.disk2.pipe.780    |    1 +
 .../disk-network-http.args.disk0              |    6 +
 .../disk-network-http.args.disk1              |    5 +
 .../disk-network-http.args.disk2              |    6 +
 .../disk-network-http.args.disk2.pipe.778     |    1 +
 .../disk-network-http.args.disk3              |    7 +
 .../disk-network-http.args.disk3.pipe.780     |    1 +
 ...work-source-curl-nbdkit-backing.args.disk0 |    7 +
 ...ce-curl-nbdkit-backing.args.disk0.pipe.778 |    1 +
 .../disk-network-source-curl.args.disk0       |    7 +
 ...sk-network-source-curl.args.disk0.pipe.778 |    1 +
 .../disk-network-source-curl.args.disk1       |    9 +
 ...sk-network-source-curl.args.disk1.pipe.780 |    1 +
 ...sk-network-source-curl.args.disk1.pipe.782 |    1 +
 .../disk-network-source-curl.args.disk2       |    7 +
 ...sk-network-source-curl.args.disk2.pipe.782 |    1 +
 ...sk-network-source-curl.args.disk2.pipe.784 |    1 +
 .../disk-network-source-curl.args.disk3       |    6 +
 .../disk-network-source-curl.args.disk4       |    6 +
 .../disk-network-ssh-key.args.disk0           |    9 +
 .../disk-network-ssh-key.args.disk1           |    9 +
 .../disk-network-ssh-password.args.disk0      |    9 +
 ...k-network-ssh-password.args.disk0.pipe.778 |    1 +
 .../disk-network-ssh.args.disk0               |    7 +
 .../disk-network-ssh.args.disk1               |    8 +
 .../disk-network-ssh.args.disk1.pipe.778      |    1 +
 .../disk-network-ssh.args.disk2               |    9 +
 tests/qemunbdkittest.c                        |  310 ++++
 tests/qemustatusxml2xmldata/modern-in.xml     |    4 +
 ...sk-cdrom-network-nbdkit.x86_64-latest.args |   42 +
 .../disk-cdrom-network-nbdkit.xml             |    1 +
 ...isk-network-http-nbdkit.x86_64-latest.args |   44 +
 .../disk-network-http-nbdkit.xml              |    1 +
 ...rce-curl-nbdkit-backing.x86_64-latest.args |   37 +
 ...isk-network-source-curl-nbdkit-backing.xml |   45 +
 ...work-source-curl-nbdkit.x86_64-latest.args |   49 +
 .../disk-network-source-curl-nbdkit.xml       |    1 +
 ...isk-network-source-curl.x86_64-latest.args |   53 +
 .../disk-network-source-curl.xml              |   74 +
 .../qemuxml2argvdata/disk-network-ssh-key.xml |   44 +
 ...disk-network-ssh-nbdkit.x86_64-latest.args |   35 +
 .../disk-network-ssh-nbdkit.xml               |    1 +
 ...sk-network-ssh-password.x86_64-latest.args |   35 +
 .../disk-network-ssh-password.xml             |   35 +
 .../disk-network-ssh.x86_64-latest.args       |   35 +
 tests/qemuxml2argvdata/disk-network-ssh.xml   |   32 +
 tests/qemuxml2argvtest.c                      |   23 +
 tests/testutilsqemu.c                         |   26 +
 tests/testutilsqemu.h                         |    4 +
 153 files changed, 3599 insertions(+), 472 deletions(-)
 create mode 100644 src/qemu/qemu_logcontext.c
 create mode 100644 src/qemu/qemu_logcontext.h
 create mode 100644 src/qemu/qemu_nbdkit.c
 create mode 100644 src/qemu/qemu_nbdkit.h
 create mode 100644 src/qemu/qemu_nbdkitpriv.h
 rename tests/qemublocktestdata/imagecreate/{network-ssh-qcow2-invalid.json => 
network-ssh-qcow2.json} (100%)
 rename tests/qemublocktestdata/imagecreate/{network-ssh-qcow2-invalid.xml => 
network-ssh-qcow2.xml} (100%)
 create mode 100644 tests/qemunbdkitdata/disk-cdrom-network.args.disk0
 create mode 100644 tests/qemunbdkitdata/disk-cdrom-network.args.disk1
 create mode 100644 tests/qemunbdkitdata/disk-cdrom-network.args.disk1.pipe.778
 create mode 100644 tests/qemunbdkitdata/disk-cdrom-network.args.disk2
 create mode 100644 tests/qemunbdkitdata/disk-cdrom-network.args.disk2.pipe.780
 create mode 100644 tests/qemunbdkitdata/disk-network-http.args.disk0
 create mode 100644 tests/qemunbdkitdata/disk-network-http.args.disk1
 create mode 100644 tests/qemunbdkitdata/disk-network-http.args.disk2
 create mode 100644 tests/qemunbdkitdata/disk-network-http.args.disk2.pipe.778
 create mode 100644 tests/qemunbdkitdata/disk-network-http.args.disk3
 create mode 100644 tests/qemunbdkitdata/disk-network-http.args.disk3.pipe.780
 create mode 100644 
tests/qemunbdkitdata/disk-network-source-curl-nbdkit-backing.args.disk0
 create mode 100644 
tests/qemunbdkitdata/disk-network-source-curl-nbdkit-backing.args.disk0.pipe.778
 create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk0
 create mode 100644 
tests/qemunbdkitdata/disk-network-source-curl.args.disk0.pipe.778
 create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk1
 create mode 100644 
tests/qemunbdkitdata/disk-network-source-curl.args.disk1.pipe.780
 create mode 100644 
tests/qemunbdkitdata/disk-network-source-curl.args.disk1.pipe.782
 create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk2
 create mode 100644 
tests/qemunbdkitdata/disk-network-source-curl.args.disk2.pipe.782
 create mode 100644 
tests/qemunbdkitdata/disk-network-source-curl.args.disk2.pipe.784
 create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk3
 create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk4
 create mode 100644 tests/qemunbdkitdata/disk-network-ssh-key.args.disk0
 create mode 100644 tests/qemunbdkitdata/disk-network-ssh-key.args.disk1
 create mode 100644 tests/qemunbdkitdata/disk-network-ssh-password.args.disk0
 create mode 100644 
tests/qemunbdkitdata/disk-network-ssh-password.args.disk0.pipe.778
 create mode 100644 tests/qemunbdkitdata/disk-network-ssh.args.disk0
 create mode 100644 tests/qemunbdkitdata/disk-network-ssh.args.disk1
 create mode 100644 tests/qemunbdkitdata/disk-network-ssh.args.disk1.pipe.778
 create mode 100644 tests/qemunbdkitdata/disk-network-ssh.args.disk2
 create mode 100644 tests/qemunbdkittest.c
 create mode 100644 
tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.x86_64-latest.args
 create mode 120000 tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.xml
 create mode 100644 
tests/qemuxml2argvdata/disk-network-http-nbdkit.x86_64-latest.args
 create mode 120000 tests/qemuxml2argvdata/disk-network-http-nbdkit.xml
 create mode 100644 
tests/qemuxml2argvdata/disk-network-source-curl-nbdkit-backing.x86_64-latest.args
 create mode 100644 
tests/qemuxml2argvdata/disk-network-source-curl-nbdkit-backing.xml
 create mode 100644 
tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.x86_64-latest.args
 create mode 120000 tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.xml
 create mode 100644 
tests/qemuxml2argvdata/disk-network-source-curl.x86_64-latest.args
 create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl.xml
 create mode 100644 tests/qemuxml2argvdata/disk-network-ssh-key.xml
 create mode 100644 
tests/qemuxml2argvdata/disk-network-ssh-nbdkit.x86_64-latest.args
 create mode 120000 tests/qemuxml2argvdata/disk-network-ssh-nbdkit.xml
 create mode 100644 
tests/qemuxml2argvdata/disk-network-ssh-password.x86_64-latest.args
 create mode 100644 tests/qemuxml2argvdata/disk-network-ssh-password.xml
 create mode 100644 tests/qemuxml2argvdata/disk-network-ssh.x86_64-latest.args
 create mode 100644 tests/qemuxml2argvdata/disk-network-ssh.xml

-- 
2.41.0

Reply via email to