Re: [libvirt PATCH v8 00/37] Use nbdkit for http/ftp/ssh network drives in libvirt

2023-09-19 Thread Jonathon Jongsma

On 9/19/23 2:44 AM, Peter Krempa wrote:

On Thu, Aug 31, 2023 at 16:39:40 -0500, Jonathon Jongsma wrote:

[...]


Jonathon Jongsma (37):
   schema: allow 'ssh' as a protocol for network disks
   qemu: Add functions for determining nbdkit availability
   qemu: expand nbdkit capabilities
   util: Allow virFileCache data to be any GObject
   qemu: implement basic virFileCache for nbdkit caps
   qemu: implement persistent file cache for nbdkit caps
   qemu: use file cache for nbdkit caps
   qemu: Add qemuNbdkitProcess
   qemu: query nbdkit module dir from binary
   qemu: add functions to start and stop nbdkit
   Generalize qemuDomainLogContextNew()
   qemu: Extract qemuDomainLogContext into a new file
   qemu: move qemuProcessReadLog() to qemuLogContext
   qemu: log error output from nbdkit
   tests: add ability to test various nbdkit capabilities
   qemu: split qemuDomainSecretStorageSourcePrepare
   qemu: include nbdkit state in private xml
   util: secure erase virCommand send buffers
   qemu: pass sensitive data to nbdkit via pipe
   qemu: use nbdkit to serve network disks if available
   util: make virCommandSetSendBuffer testable
   tests: add tests for nbdkit invocation
   qemu: add test for authenticating a https network disk
   qemu: Add Taint for nbdkit restart failure
   qemu: Monitor nbdkit process for exit
   qemu: improve error handling when restarting nbdkit
   qemu: try to connect to nbdkit early to detect errors
   schema: add password configuration for ssh disk
   qemu: implement password auth for ssh disks with nbdkit
   schema: add configuration for host verification of ssh disks
   qemu: implement knownHosts for ssh disks with nbdkit
   schema: add keyfile configuration for ssh disks
   qemu: implement keyfile auth for ssh disks with nbdkit
   schema: add ssh-agent configuration for ssh disks
   qemu: implement ssh-agent auth for ssh disks with nbdkit


Please push these patches. I plan working on qemu storage daemon
integration which has the potential to conflict with this.


   rpm: update spec file for for nbdkit support


This one can be omitted if you're still waiting for the selinux policy.


   ci: add libnbd to build


This one probably should work even now.



Sorry for the delay. It should be pushed now.

Jonathon



Re: [libvirt PATCH v8 00/37] Use nbdkit for http/ftp/ssh network drives in libvirt

2023-09-19 Thread Peter Krempa
On Thu, Aug 31, 2023 at 16:39:40 -0500, Jonathon Jongsma wrote:

[...]

> Jonathon Jongsma (37):
>   schema: allow 'ssh' as a protocol for network disks
>   qemu: Add functions for determining nbdkit availability
>   qemu: expand nbdkit capabilities
>   util: Allow virFileCache data to be any GObject
>   qemu: implement basic virFileCache for nbdkit caps
>   qemu: implement persistent file cache for nbdkit caps
>   qemu: use file cache for nbdkit caps
>   qemu: Add qemuNbdkitProcess
>   qemu: query nbdkit module dir from binary
>   qemu: add functions to start and stop nbdkit
>   Generalize qemuDomainLogContextNew()
>   qemu: Extract qemuDomainLogContext into a new file
>   qemu: move qemuProcessReadLog() to qemuLogContext
>   qemu: log error output from nbdkit
>   tests: add ability to test various nbdkit capabilities
>   qemu: split qemuDomainSecretStorageSourcePrepare
>   qemu: include nbdkit state in private xml
>   util: secure erase virCommand send buffers
>   qemu: pass sensitive data to nbdkit via pipe
>   qemu: use nbdkit to serve network disks if available
>   util: make virCommandSetSendBuffer testable
>   tests: add tests for nbdkit invocation
>   qemu: add test for authenticating a https network disk
>   qemu: Add Taint for nbdkit restart failure
>   qemu: Monitor nbdkit process for exit
>   qemu: improve error handling when restarting nbdkit
>   qemu: try to connect to nbdkit early to detect errors
>   schema: add password configuration for ssh disk
>   qemu: implement password auth for ssh disks with nbdkit
>   schema: add configuration for host verification of ssh disks
>   qemu: implement knownHosts for ssh disks with nbdkit
>   schema: add keyfile configuration for ssh disks
>   qemu: implement keyfile auth for ssh disks with nbdkit
>   schema: add ssh-agent configuration for ssh disks
>   qemu: implement ssh-agent auth for ssh disks with nbdkit

Please push these patches. I plan working on qemu storage daemon
integration which has the potential to conflict with this.

>   rpm: update spec file for for nbdkit support

This one can be omitted if you're still waiting for the selinux policy.

>   ci: add libnbd to build

This one probably should work even now.



[libvirt PATCH v8 00/37] Use nbdkit for http/ftp/ssh network drives in libvirt

2023-08-31 Thread Jonathon Jongsma
This is the eighth version of this patch series. See
https://bugzilla.redhat.com/show_bug.cgi?id=2016527 for more information.

Note that testing this requires selinux policy changes which are not fully
done, but there is a new policy in development that has allowed me to run with
selinux in enforcing mode for the common cases. See
https://bugzilla.redhat.com/show_bug.cgi?id=2182505 for more information. The
following scenarios should work now with selinux enabled using the selinux
policy from that bug:
 - http/https disks
 - ssh disks with password authentication
 - ssh disks with passwordless keyfile

The one major thing that doesn't work and is difficult to get working with
selinux enabled is the ssh-agent. This is because there doesn't seem to be any
selinux policy for ssh-agent, so by default the ssh-agent socket is labeled
unconfined_t. We cannot allow access from the libvirt/qemu to unconfined_t
because that would open up access to just about anything on the host. So
additional work will likely be necessary for ssh-agent/libvirt interaction in
the future. Fortunately ssh-agent is something that never was really supported
with the old qemu block driver either, so I think we could potentially merge
this patchset either without the ssh-agent patches or with a note that
ssh-agent won't work with selinux enabled.

Changes in v8:
 - Hopefully addressed all of Peter's issues, in addition to:
 - updated documentation to say 9.8.0, since 9.7.0 is currently in freeze
 - used WITH_NBDKIT instead of WITH_DECL_SYS_PIDFD_OPEN to make the code a bit
   more concise and understandable
 - enabled ci by adding libnbd to the dependencies, which uncovered a couple
   additional minor issues with those platforms that don't support the
   pidfd_open syscall
   - don't run nbdkit tests when WITH_NBDKIT is not defined
   - avoid warnings with unused function arguments
   - note that the ubuntu containers are currently failing due to a
 LeakSanitizer error, but I haven't reproduced it locally and can't figure
 out how to get better information from the leak sanitizer. Pointers
 appreciated: https://gitlab.com/jjongsma/libvirt/-/jobs/4991631193
 - One change of note is a new patch "qemu: improve error handling when
   restarting nbdkit". In order to provide better error reporting to the
   user and avoid VIR_WARN as suggested by Peter, some functions now return an
   error and this error is propagated up to qemuProcessReconnect(). This could
   potentially result in running domains being killed upon a libvirt restart,
   but only if they were in a state where they were was not a running nbdkit
   backend or libvirt couldn't monitor the process nbdkit.

Jonathon Jongsma (37):
  schema: allow 'ssh' as a protocol for network disks
  qemu: Add functions for determining nbdkit availability
  qemu: expand nbdkit capabilities
  util: Allow virFileCache data to be any GObject
  qemu: implement basic virFileCache for nbdkit caps
  qemu: implement persistent file cache for nbdkit caps
  qemu: use file cache for nbdkit caps
  qemu: Add qemuNbdkitProcess
  qemu: query nbdkit module dir from binary
  qemu: add functions to start and stop nbdkit
  Generalize qemuDomainLogContextNew()
  qemu: Extract qemuDomainLogContext into a new file
  qemu: move qemuProcessReadLog() to qemuLogContext
  qemu: log error output from nbdkit
  tests: add ability to test various nbdkit capabilities
  qemu: split qemuDomainSecretStorageSourcePrepare
  qemu: include nbdkit state in private xml
  util: secure erase virCommand send buffers
  qemu: pass sensitive data to nbdkit via pipe
  qemu: use nbdkit to serve network disks if available
  util: make virCommandSetSendBuffer testable
  tests: add tests for nbdkit invocation
  qemu: add test for authenticating a https network disk
  qemu: Add Taint for nbdkit restart failure
  qemu: Monitor nbdkit process for exit
  qemu: improve error handling when restarting nbdkit
  qemu: try to connect to nbdkit early to detect errors
  schema: add password configuration for ssh disk
  qemu: implement password auth for ssh disks with nbdkit
  schema: add configuration for host verification of ssh disks
  qemu: implement knownHosts for ssh disks with nbdkit
  schema: add keyfile configuration for ssh disks
  qemu: implement keyfile auth for ssh disks with nbdkit
  schema: add ssh-agent configuration for ssh disks
  qemu: implement ssh-agent auth for ssh disks with nbdkit
  rpm: update spec file for for nbdkit support
  ci: add libnbd to build

 build-aux/syntax-check.mk |2 +-
 ci/buildenv/almalinux-8.sh|1 +
 ci/buildenv/centos-stream-8.sh|1 +
 ci/buildenv/centos-stream-9.sh|1 +
 ci/buildenv/debian-12-cross-aarch64.sh|1 +
 ci/buildenv/debian-12-cross-armv6l.sh |1 +
 ci/buildenv/debian-12-cross-armv7l.sh |1 +
 ci/buildenv/debian-12-cross-i686.sh   |1 +