Re: [Libguestfs] [PATCH v2v 0/2] output: Remove -o json mode [in stages]

2022-04-11 Thread Richard W.M. Jones
On Mon, Apr 11, 2022 at 12:34:30PM +0200, Laszlo Ersek wrote: > On 04/11/22 12:11, Richard W.M. Jones wrote: > > I'd like to remove the unused -o json mode. The implementation is > > over-complicated and the only potential user (KubeVirt) isn't using it > > now an

[Libguestfs] [PATCH v2v 2/2] output: Remove -o json mode

2022-04-11 Thread Richard W.M. Jones
This removes the -o json mode completely. --- docs/virt-v2v.pod| 4 - output/Makefile.am | 51 +- output/create_json.ml| 338 --- output/create_json.mli | 27 --- output/output_json.ml| 153

[Libguestfs] [PATCH v2v 0/2] output: Remove -o json mode [in stages]

2022-04-11 Thread Richard W.M. Jones
I'd like to remove the unused -o json mode. The implementation is over-complicated and the only potential user (KubeVirt) isn't using it now and would be better off with a proper -o kubevirt mode, which I'm going to implement later. My plan would be to add the patches in two stages. The first

[Libguestfs] [PATCH v2v 1/2] docs: Remove documentation for -o json in preparation for removal

2022-04-11 Thread Richard W.M. Jones
I would like to remove the -o json option in virt-v2v 2.2. It was added to support KubeVirt, but never actually used. It is better for specific targets to implement a proper output mode (as we will eventually do for KubeVirt) rather than relying on a custom templating language which knows

Re: [Libguestfs] [v2v PATCH] convert: warn about "--mac" options that don't match any source NICs

2022-04-11 Thread Richard W.M. Jones
On Sun, Apr 10, 2022 at 03:49:51PM +0200, Laszlo Ersek wrote: > Just before we call "Networks.map", collect all "--mac" references (from > "options.network_map" and "options.static_ips"), and check each against > the list of MACs available on the source ("source.s_nics"). Collect and > report

Re: [Libguestfs] [guestfs-tools PATCH] sysprep: remove lvm2's default "system.devices" file

2022-04-11 Thread Richard W.M. Jones
; ^ system_devices_file ^ ">. When cloning VMs, WWIDs may \ > + change, breaking C. Remove \ > + C<" ^ system_devices_file ^ ">."); > +perform_on_filesystems = Some lvm_system_devices_perform; > +} > + > +let () = register

Re: [Libguestfs] nbdkit test failures in test-curl-cookie-script & test-curl-header-script

2022-04-09 Thread Richard W.M. Jones
On Sat, Apr 09, 2022 at 07:35:57AM +0100, Richard W.M. Jones wrote: > > I get occasional failures in test-curl-header-script locally. I am > unable to reproduce it. > > The tests also failed a few days ago in test-curl-cookie-script: > > https://gitlab.com/nbdkit/nbd

Re: [Libguestfs] [v2v PATCH] input-xen: sync PasswordAuthentication hint from input-vmware manual

2022-04-09 Thread Richard W.M. Jones
irectly). > +does I take the password directly). You may need to adjust > +F on the Xen server to set > +C. > > If you are not using password authentication, an alternative is to use > ssh-agent, and add your ssh public key to Reviewed-by: Richard W.M. Jones Rich. -- Richard

[Libguestfs] nbdkit test failures in test-curl-cookie-script & test-curl-header-script

2022-04-09 Thread Richard W.M. Jones
I get occasional failures in test-curl-header-script locally. I am unable to reproduce it. The tests also failed a few days ago in test-curl-cookie-script: https://gitlab.com/nbdkit/nbdkit/-/jobs/2284052519 In both cases the tests fail (rarely) with the message. I don't know exactly where

Re: [Libguestfs] [v2v PATCH v2] input_vmx: cleanly reject guests with snapshots when using "-it ssh"

2022-04-09 Thread Richard W.M. Jones
;-ic esx://...> B<-it vddk>. Refer ... and here too. > +to L<https://bugzilla.redhat.com/1774386>. > + > =head1 INPUT FROM VMWARE VMX > > Virt-v2v is able to import guests from VMware’s vmx files. > @@ -106,7 +121,8 @@ a C folder containing the virtual machine

Re: [Libguestfs] [PATCH 1/2] spec: Recommend cap on NBD_REPLY_TYPE_BLOCK_STATUS length

2022-04-08 Thread Richard W.M. Jones
On Fri, Apr 08, 2022 at 08:45:56PM +0300, Nir Soffer wrote: > On Fri, Apr 8, 2022 at 6:47 PM Eric Blake wrote: > > > > On Fri, Apr 08, 2022 at 04:48:59PM +0300, Nir Soffer wrote: > ... > > > > BTW attached is an nbdkit plugin that creates an NBD server that > > > > responds with massive numbers

Re: [Libguestfs] [PATCH guestfs-tools] builder: templates: Enable Fedora image update again

2022-04-08 Thread Richard W.M. Jones
On Fri, Apr 08, 2022 at 05:36:45PM +0200, Laszlo Ersek wrote: > On 04/08/22 16:59, Richard W.M. Jones wrote: > > Fixes: commit 3fe941767042bf83d9a252b0819fa6d5a48059d0 > > --- > > builder/templates/make-template.ml | 6 -- > > 1 file changed, 4 insertions(+), 2

Re: [Libguestfs] [PATCH virt-v2v v2 2/2] input: -i vmx: Add support for NVMe devices

2022-04-08 Thread Richard W.M. Jones
On Fri, Apr 08, 2022 at 05:00:07PM +0200, Laszlo Ersek wrote: > On 04/08/22 13:11, Richard W.M. Jones wrote: > > +(* Find all NVMe hard disks. > > + * > > + * In the VMX file: > > + * nvme0.pcislotnumber = "192" > > + * nvme0:0.fileName = "g

Re: [Libguestfs] [PATCH 2/2] tests: fix rhbz1370424 to use proper path

2022-04-08 Thread Richard W.M. Jones
Thanks, I pushed both. The reason these test failures were not picked up is because I had them set to skip locally because of older errors in other components which have now been fixed. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming

[Libguestfs] [PATCH guestfs-tools] builder: templates: Enable Fedora image update again

2022-04-08 Thread Richard W.M. Jones
Fixes: commit 3fe941767042bf83d9a252b0819fa6d5a48059d0 --- builder/templates/make-template.ml | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/builder/templates/make-template.ml b/builder/templates/make-template.ml index fe3039aa06..51f72da969 100755 ---

Re: [Libguestfs] [v2v PATCH] input_vmx: cleanly reject guests with snapshots when using "-it ssh"

2022-04-08 Thread Richard W.M. Jones
On Fri, Apr 08, 2022 at 03:16:39PM +0200, Laszlo Ersek wrote: > +(* RHBZ#1774386 *) > +if not (remote_file_exists uri flat_vmdk) then > + failwith "this transport does not support guests with > snapshots"; This message isn't translated; use error (f_"...")

Re: [Libguestfs] [PATCH 1/2] spec: Recommend cap on NBD_REPLY_TYPE_BLOCK_STATUS length

2022-04-08 Thread Richard W.M. Jones
On Fri, Apr 08, 2022 at 04:48:59PM +0300, Nir Soffer wrote: > This is a malicious server. A good client will drop the connection when > receiving the first 1 byte chunk. > > The real issue here is not enforcing or suggesting a limit on the number of > extent the server returns, but enforcing a

Re: [Libguestfs] [PATCH 1/2] spec: Recommend cap on NBD_REPLY_TYPE_BLOCK_STATUS length

2022-04-08 Thread Richard W.M. Jones
On Fri, Apr 08, 2022 at 12:52:44PM +0100, Richard W.M. Jones wrote: > From experimentation I found this really hurts my laptop :-( ... because of a memory leak in the nbdkit Python bindings as it turned out. This function: > def extents(h, count, offset, flags): > e = [(i, 1, typ(

Re: [Libguestfs] [PATCH 1/2] spec: Recommend cap on NBD_REPLY_TYPE_BLOCK_STATUS length

2022-04-08 Thread Richard W.M. Jones
On Fri, Apr 08, 2022 at 09:25:01AM +0200, Wouter Verhelst wrote: > Hi Eric, > > On Thu, Apr 07, 2022 at 04:37:19PM -0500, Eric Blake wrote: > > The spec was silent on how many extents a server could reply with. > > However, both qemu and nbdkit (the two server implementations known to > > have

[Libguestfs] [PATCH virt-v2v v2 0/2] input: -i vmx: Add support for NVMe devices

2022-04-08 Thread Richard W.M. Jones
https://bugzilla.redhat.com/show_bug.cgi?id=2070530 Version 1 was here: https://listman.redhat.com/archives/libguestfs/2022-April/028560.html Version 2 fixes the bogus mapping of source device names as pointed out in Laszlo's review (and no longer touches target device names). The rest of the

[Libguestfs] [PATCH virt-v2v v2 2/2] input: -i vmx: Add support for NVMe devices

2022-04-08 Thread Richard W.M. Jones
We model NVMe devices in the source hypervisor. We currently assume that no one is using the namespaces feature of NVMe, ie. that each source device will appear in a Linux guest as /dev/nvme0n1, /dev/nvme1n1, etc. We could fix this if it is a problem, but it requires adjusting the current

[Libguestfs] [PATCH virt-v2v v2 1/2] convert/convert_linux.ml: Refactor source device name calculation

2022-04-08 Thread Richard W.M. Jones
This neutral refactoring changes the existing source device name calculation so that it can handle (in the next commit) NVMe device names. Existing source device names such as "hda" or "sda" are composed of a prefix ("hd", "sd") + alphabetic device index ("a"). NVMe devices have a namespace

Re: [Libguestfs] [PATCH v2v] input: -i vmx: Add support for NVMe devices

2022-04-08 Thread Richard W.M. Jones
On Tue, Apr 05, 2022 at 04:26:53PM +0200, Laszlo Ersek wrote: > On 04/04/22 17:15, Richard W.M. Jones wrote: > > We model NVMe devices in the source hypervisor. > > > > We currently assume that no one is using the namespaces feature of > > NVMe, ie. that each source de

Re: [Libguestfs] [PATCH nbdkit v2 5/5] fuzzing: Recommend combining fuzzing with ASAN

2022-04-07 Thread Richard W.M. Jones
Thanks - this series pushed as: b59380e0..c0779edb master -> master Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with

Re: [Libguestfs] [supermin PATCH v2 1/1] Fix for missing /lib/modules.

2022-04-07 Thread Richard W.M. Jones
On Wed, Apr 06, 2022 at 04:15:15PM -0500, George Prekas wrote: > Even when supplied with $SUPERMIN_MODULES, supermin tries to access > /lib/modules. This directory does not exist by default in all the > environments. > --- > src/format_ext2.ml | 14 -- > 1 file changed, 12

Re: [Libguestfs] [supermin PATCH v2 1/1] Fix for missing /lib/modules.

2022-04-07 Thread Richard W.M. Jones
On Thu, Apr 07, 2022 at 11:03:36AM +0200, Laszlo Ersek wrote: > On 04/06/22 23:15, George Prekas wrote: > > Even when supplied with $SUPERMIN_MODULES, supermin tries to access > > /lib/modules. This directory does not exist by default in all the > > environments. > > --- > > src/format_ext2.ml |

[Libguestfs] [PATCH nbdkit v2 3/5] configure: Disable linker script for server when using libfuzzer

2022-04-07 Thread Richard W.M. Jones
Libfuzzer used its own conditional flag for this. Instead use the USE_LINKER_SCRIPT_FOR_SERVER flag. --- configure.ac | 4 server/Makefile.am | 5 - 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/configure.ac b/configure.ac index 81c75fda..ba194781 100644 ---

[Libguestfs] [PATCH nbdkit v2 0/5] fuzzing: Recommend combining fuzzing with ASAN

2022-04-07 Thread Richard W.M. Jones
v1 was here: https://listman.redhat.com/archives/libguestfs/2022-April/028568.html v2: - Picked up Reviewed-by line for first patch. - Split second patch into 3 parts. Add ASAN_OPTIONS which is necessary for the tests to run successfully under ASAN. - Third patch has been extensively

[Libguestfs] [PATCH nbdkit v2 4/5] configure: Add --disable-linker-script flag

2022-04-07 Thread Richard W.M. Jones
This flag allows you to disable the linker script (for the server only). This is necessary for ASAN to work, and the documentation has been updated to explain how to run nbdkit under ASAN. Thanks Eric Blake for helping me with working out the root cause. --- configure.ac | 7 +++ README

[Libguestfs] [PATCH nbdkit v2 1/5] configure: Report --enable-libfuzzer flag in summary output

2022-04-07 Thread Richard W.M. Jones
This changes the summary output to report whether or not the ./configure --enable-libfuzzer flag was set: libfuzzer .. yes or for --disable-libfuzzer / the default of not set: libfuzzer .. no Reviewed-by: Laszlo Ersek ---

[Libguestfs] [PATCH nbdkit v2 5/5] fuzzing: Recommend combining fuzzing with ASAN

2022-04-07 Thread Richard W.M. Jones
ASAN (ie. Address Sanitizer or -fsanitize=address) adds extra checks for out-of-bounds memory access, use-after-free and other memory checks. It's useful to combine this with fuzzing. Fuzzing can normally only detect paths which cause the binary to crash. But some serious, latent bugs might not

[Libguestfs] [PATCH nbdkit v2 2/5] configure: Reorganise USE_LINKER_SCRIPT_FOR_SERVER conditional

2022-04-07 Thread Richard W.M. Jones
There will be several reasons for disabling the linker script for nbdkit (in forthcoming patches). Make the use_linker_script_for_server variable into a "global" (top level). Move the AM_CONDITIONAL that checks this variable later. This is just refactoring. --- configure.ac | 19

Re: [Libguestfs] [PATCH nbdkit 2/3] configure: Add --disable-linker-script flag

2022-04-07 Thread Richard W.M. Jones
I have split this into three patches in my local copy, but ... On Thu, Apr 07, 2022 at 11:20:47AM +0200, Laszlo Ersek wrote: > The second patch should be related to libfuzzer. It should contain hunks > #2 and #6. It is also a refactoring. Notably though, the comment in hunk > #2 should *not*

[Libguestfs] [PATCH nbdkit 2/3] configure: Add --disable-linker-script flag

2022-04-06 Thread Richard W.M. Jones
This flag allows you to disable the linker script (for the server only). This is necessary for ASAN to work as explained in the documentation. Thanks Eric Blake for helping me with working out the root cause. --- configure.ac | 30 -- server/Makefile.am | 5

[Libguestfs] [PATCH nbdkit 3/3] fuzzing: Recommend combining fuzzing with ASAN

2022-04-06 Thread Richard W.M. Jones
ASAN (ie. Address Sanitizer or -fsanitize=address) adds extra checks for out-of-bounds memory access, use-after-free and other memory checks. It's useful to combine this with fuzzing. Fuzzing can normally only detect paths which cause the binary to crash. But some serious, latent bugs might not

[Libguestfs] [PATCH nbdkit 1/3] configure: Report --enable-libfuzzer flag in summary output

2022-04-06 Thread Richard W.M. Jones
This changes the summary output to report whether or not the ./configure --enable-libfuzzer flag was set: libfuzzer .. yes or for --disable-libfuzzer / the default of not set: libfuzzer .. no --- configure.ac | 2 ++ 1 file

Re: [Libguestfs] xfs protofiles for libguestfs

2022-04-04 Thread Richard W.M. Jones
On Sat, Apr 02, 2022 at 08:18:52AM -0700, Alli wrote: > Hi all, > I hope I am in the right place, I found this email by following the > link in the project README. We are collecting use cases for xfs dev > projects, one of which would be enhancing xfs protofiles, which is a > type of script that

[Libguestfs] [PATCH v2v] input: -i vmx: Add support for NVMe devices

2022-04-04 Thread Richard W.M. Jones
We model NVMe devices in the source hypervisor. We currently assume that no one is using the namespaces feature of NVMe, ie. that each source device will appear in a Linux guest as /dev/nvme0n1, /dev/nvme1n1, etc. We could fix this if it is a problem, but it requires adjusting the current

[Libguestfs] [PATCH v2v] input: -i vmx: Add support for NVMe devices

2022-04-04 Thread Richard W.M. Jones
https://bugzilla.redhat.com/show_bug.cgi?id=2070530 This patch touches some very dark corners of virt-v2v, namely target bus assignment and Linux configuration file device name rewriting. I have only lightly tested it by converting mxie's own NVMe guest (which worked). Rich.

Re: [Libguestfs] [PATCH v3] spec: Clarify BLOCK_STATUS reply details

2022-04-04 Thread Richard W.M. Jones
or > + extent information at the first offset not covered by a > + reduced-length reply. > > All error chunk types have bit 15 set, and begin with the same > *error*, *message length*, and optional *message* fields as This seems clearer now, thanks. Reviewed-by: Richard W.M. Jone

[Libguestfs] [PATCH v2v] input: -i ova: Handle OVAs which contain user/group names with spaces

2022-04-04 Thread Richard W.M. Jones
If importing an OVA that has user/group names with spaces then the plain tar -tRvf command would print them without any quoting. Our simple strategy of splitting on spaces resulted in an "extra field" being parsed. This is an example from a real OVA (note "Domain Users" is the group name): $

[Libguestfs] [PATCH v2v] docs: Mention pnp_wait.exe and refine pvvxsvc.exe documentation

2022-04-04 Thread Richard W.M. Jones
Virt-v2v can use pnp_wait.exe to wait for new Windows devices to activate. However this was not mentioned in the existing documentation. Also refine the documentation for pvvxsvc.exe (used only by SUSE hosts) so that we don't mention the non-existent --firstboot* options. This was incorrect

Re: [Libguestfs] [p2v PATCH v2 11/11] do not connect to nbdkit after starting it

2022-04-01 Thread Richard W.M. Jones
o_start() -- stops reading from the socket > right after seeing "NBDMAGIC" (8 bytes). So that's a "broken pipe" > situation (RST) for the sending party (the server). > > Ref: https://listman.redhat.com/archives/libguestfs/2022-March/028528.html > Suggested-by:

Re: [Libguestfs] [p2v PATCH v2 09/11] nbd.c: pass "--exit-with-parent" to nbdkit if nbdkit supports it

2022-04-01 Thread Richard W.M. Jones
re -- thus, we'd need to > introduce a wrapper structure just for this, which is overkill. > > Ref: https://listman.redhat.com/archives/libguestfs/2022-March/028475.html > Suggested-by: Richard W.M. Jones > Signed-off-by: Laszlo Ersek > --- > > Notes: > v2: >

Re: [Libguestfs] [p2v PATCH v2 01/11] remove qemu-nbd support

2022-04-01 Thread Richard W.M. Jones
On Fri, Apr 01, 2022 at 12:27:58PM +0200, Laszlo Ersek wrote: > On 03/24/22 10:13, Richard W.M. Jones wrote: > > virt-p2v needs an NBD server running on the p2v machine to export the > > disks. Originally we used qemu-nbd since that was the only choice. > > In 2017, I add

Re: [Libguestfs] [PATCH nbdkit] tests: Skip vsock tests unless the vsock_loopback module is loaded

2022-04-01 Thread Richard W.M. Jones
On Fri, Apr 01, 2022 at 01:52:13PM +0200, Laszlo Ersek wrote: > On 04/01/22 12:46, Richard W.M. Jones wrote: > > It's unfortunate that vsock in Linux can be broken if some but not all > > vsock modules needed are loaded. As we are testing loopback > > (VMADD

[Libguestfs] [PATCH nbdkit] tests: Skip vsock tests unless the vsock_loopback module is loaded

2022-04-01 Thread Richard W.M. Jones
It's unfortunate that vsock in Linux can be broken if some but not all vsock modules needed are loaded. As we are testing loopback (VMADDR_CID_LOCAL), require that vsock_loopback is loaded. Linux is able to autoload the correct module(s) if vsock.ko is _not_ loaded. We could in theory enhance

Re: [Libguestfs] [p2v PATCH 10/10] nbd.c: bind listening socket without AI_ADDRCONFIG

2022-04-01 Thread Richard W.M. Jones
On Fri, Apr 01, 2022 at 12:07:06PM +0200, Laszlo Ersek wrote: > Now the question is if we are bothered by this "RST" in the conversion > log, as it's expected. If conversion is successful then no one will be looking at the logs, and if it fails then hopefully a search will bring up this thread.

Re: [Libguestfs] [p2v PATCH 01/10] remove qemu-nbd support

2022-03-31 Thread Richard W.M. Jones
On Thu, Mar 31, 2022 at 03:24:44PM +0200, Laszlo Ersek wrote: > On 03/31/22 14:27, Richard W.M. Jones wrote: > > On Thu, Mar 31, 2022 at 09:22:02AM +0200, Laszlo Ersek wrote: > >> diff --git a/dependencies.m4 b/dependencies.m4 > >> index 40b3786cb0b7..20d068028dbe 100644

Re: [Libguestfs] [p2v PATCH 10/10] nbd.c: bind listening socket without AI_ADDRCONFIG

2022-03-31 Thread Richard W.M. Jones
On Thu, Mar 31, 2022 at 03:38:40PM +0200, Laszlo Ersek wrote: > Thanks! BTW: do you have any comments about the following serial log > message (see the cover letter): [https://listman.redhat.com/archives/libguestfs/2022-March/028498.html] > > nbdkit: file[1]: error: reading initial client flags:

Re: [Libguestfs] [p2v PATCH 09/10] rely on Linux for killing nbdkit, when the parent thread exits

2022-03-31 Thread Richard W.M. Jones
On Thu, Mar 31, 2022 at 03:35:55PM +0200, Laszlo Ersek wrote: > And yes, that's the crux: if we need to keep the PID tracking / killing > / reaping (and I agree that we do!), then what does "--exit-with-parent" > buy us? I don't think it improves anything. So if I can't remove the PID > tracking

Re: [Libguestfs] [v2v PATCH] convert/linux_kernels: fix virtio-socket check

2022-03-31 Thread Richard W.M. Jones
virtio_blk ki.ki_supports_virtio_net >ki.ki_supports_virtio_rng ki.ki_supports_virtio_balloon; > - fpf "pvpanic=%b xen=%b debug=%b\n" > - ki.ki_supports_isa_pvpanic ki.ki_is_xen_pv_only_kernel ki.ki_is_debug > + fpf "pvpanic=%b vsock=%b xen=%b

Re: [Libguestfs] [p2v PATCH 10/10] nbd.c: bind listening socket without AI_ADDRCONFIG

2022-03-31 Thread Richard W.M. Jones
457/nbdkit > > tcp6 0 0 ::1:54145 ::1:57432 > > ESTABLISHED 49303/nbdkit > > The two listening nbdkit processes (PIDs 49310 and 49303) are now doing so > over both TCPv4 and TCPv6, and the third one (PID 49457) actually

Re: [Libguestfs] [p2v PATCH 09/10] rely on Linux for killing nbdkit, when the parent thread exits

2022-03-31 Thread Richard W.M. Jones
So this patch unfortunately isn't correct for a few reasons: (1) Non-Linux/BSD OSes don't support a concept like --exit-with-parent. nbdkit will exit with an error if you try to use this feature on such OSes. You can test if --exit-with-parent is supported before trying to use it by: nbdkit

Re: [Libguestfs] [p2v PATCH 08/10] nbd.c: remove "enum nbd_server"

2022-03-31 Thread Richard W.M. Jones
For patches 4-8: Reviewed-by: Richard W.M. Jones I would be tempted to squash a few of these together before pushing, but whatever you wish. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http

Re: [Libguestfs] [p2v PATCH 03/10] nbd.c: simplify start_nbdkit()

2022-03-31 Thread Richard W.M. Jones
beyond logging). > > Remove the branch in start_nbdkit() that starts nbdkit without socket > activation, and trim the start_nbdkit() prototype too. > > This patch is best viewed with "git show -b" due to the unindentation in > it. > > Ref: https://listman.redhat.com/arch

Re: [Libguestfs] [p2v PATCH 02/10] remove "--nbd=nbdkit-no-sa" option parsing

2022-03-31 Thread Richard W.M. Jones
too, with the rest > of this patch, virt-p2v refuses to start up -- as no usable nbd server is > found. > > This patch permits further simplification, which will be done subsequently > in this series. > > Ref: https://listman.redhat.com/archives/libguestfs/2022-March/028475.

Re: [Libguestfs] [p2v PATCH 01/10] remove qemu-nbd support

2022-03-31 Thread Richard W.M. Jones
On Thu, Mar 31, 2022 at 09:22:02AM +0200, Laszlo Ersek wrote: > diff --git a/dependencies.m4 b/dependencies.m4 > index 40b3786cb0b7..20d068028dbe 100644 > --- a/dependencies.m4 > +++ b/dependencies.m4 > @@ -23,39 +23,40 @@ dnl only runs on the virt-v2v conversion server. > ifelse(REDHAT,1, >

Re: [Libguestfs] [p2v PATCH 00/10] Simplify NBD server management

2022-03-31 Thread Richard W.M. Jones
One thing I realised over the weekend is that if we're removing qemu-nbd support, then we'd better have an nbdkit available on older RHEL. I pushed a few small fixes to nbdkit this morning, and I'm happy to report that current nbdkit from git compiles fine on RHEL 6, albeit with a reduced set

Re: [Libguestfs] Regression with LDM devices

2022-03-28 Thread Richard W.M. Jones
On Mon, Mar 28, 2022 at 01:25:42PM +0300, Nikolay Ivanets wrote: > Hello > > This commit caused a regression with LDM devices: > https://github.com/libguestfs/libguestfs/commit/86577ee3883836c1c4fff258c05261bd3858e22b > > 1. list-filesystems double all LDM volumes: > > > list-filesystems >

Re: [Libguestfs] [PATCH v2 3/3] spec: Clarify BLOCK_STATUS reply details

2022-03-25 Thread Richard W.M. Jones
On Fri, Mar 25, 2022 at 07:41:02AM -0500, Eric Blake wrote: > Our docs were inconsistent on whether a NBD_REPLY_TYPE_BLOCK_STATUS > reply chunk can exceed the client's requested length, and silent on > whether the lengths must be consistent whem multiple contexts were "when" > negotiated.

Re: [Libguestfs] [PATCH v2 1/3] docs: Clarify NBD_REPLY_TYPE_ERROR lengths

2022-03-25 Thread Richard W.M. Jones
On Fri, Mar 25, 2022 at 07:41:00AM -0500, Eric Blake wrote: > Add explicit mention that human-readable error message strings must > comply with the overall NBD string limits. > > Signed-off-by: Eric Blake > --- > doc/proto.md | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) > > diff

Re: [Libguestfs] [PATCH v2 2/3] docs: Clarify structured reads vs. block size constraints

2022-03-25 Thread Richard W.M. Jones
On Fri, Mar 25, 2022 at 07:41:01AM -0500, Eric Blake wrote: > The text for structured reads mentioned a mandatory split of certain > large reads, without also mentioning that large reads are generally > not possible when block constraints are in play. > > Signed-off-by: Eric Blake > --- >

Re: [Libguestfs] [p2v PATCH 2/2] Makefile.am: add "run-virt-p2v-in-an-nvme-vm"

2022-03-25 Thread Richard W.M. Jones
ACK series. I don't see any issues with adding more run-* targets (second patch), as long as they are useful which this one is. Thanks, Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog:

Re: [Libguestfs] Virt-v2v binary size

2022-03-24 Thread Richard W.M. Jones
On Thu, Mar 24, 2022 at 10:12:39AM +0100, Laszlo Ersek wrote: > On 03/23/22 16:39, Richard W.M. Jones wrote: > > ... is about 5MB stripped, which is not terrible, but also not great. > > Why does its size matter? Containers :-( The truth is it doesn't really matter, but I was int

Re: [Libguestfs] [PATCH p2v 2/2] tests: Increase memory for "make run-virt-p2v-in-a-vm"

2022-03-24 Thread Richard W.M. Jones
Thanks - pushed these as commits 764abefaa & 07a346ceb. BTW the earlier bug with version detection will mean that RHEL 7 virt-p2v ISO will definitely not be able to talk to RHEL 9.1 virt-v2v :-( Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my

Re: [Libguestfs] [PATCH p2v] ssh: Allow interop with virt-v2v 2.0

2022-03-24 Thread Richard W.M. Jones
Thanks - pushed that as commit 546f3a6. While I remember, there's another odd thing about virt-p2v that could be simplified if you're looking for some general clean-up work to do to get used to the code. virt-p2v needs an NBD server running on the p2v machine to export the disks. Originally we

[Libguestfs] [PATCH p2v 1/2] make-disk: Don't upgrade the kernel to avoid kernel command line bug

2022-03-23 Thread Richard W.M. Jones
There is some bug in recent Fedora which prevents the kernel from being upgraded correctly from libguestfs. As an example of what happens and how to reproduce it see: https://bugzilla.redhat.com/show_bug.cgi?id=1945835#c24 This problem causes the kernel command line to be messed up (including

[Libguestfs] [PATCH p2v 2/2] tests: Increase memory for "make run-virt-p2v-in-a-vm"

2022-03-23 Thread Richard W.M. Jones
With the general growth of Fedora, especially use of zram, it seems prudent to increase the amount of memory used in this test. Physical machines that we are emulating would have a lot more RAM than 1G. --- Makefile.am | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git

[Libguestfs] [PATCH p2v] ssh: Allow interop with virt-v2v 2.0

2022-03-23 Thread Richard W.M. Jones
The code previously checked that the remote version of virt-v2v was 1.x and >= 1.28. Since virt-v2v 2.0 has been released and is broadly compatible, allow 2.0 as well. --- ssh.c | 12 ++-- 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/ssh.c b/ssh.c index ce0caaa..7ac43eb

[Libguestfs] Virt-v2v binary size

2022-03-23 Thread Richard W.M. Jones
... is about 5MB stripped, which is not terrible, but also not great. I wrote a trivial Perl script to try to find out what's using the space and the report is attached. Also: $ size -d v2v/virt-v2v text data bss dec hex filename 3580677 165438436944 5272005

Re: [Libguestfs] [v2v PATCH] nbdkit, qemuNBD: run_unix: formally require externally provided socket

2022-03-23 Thread Richard W.M. Jones
On Wed, Mar 23, 2022 at 11:43:30AM +0100, Laszlo Ersek wrote: > At this point, virt-v2v never relies on the Unix domain sockets created > inside the "run_unix" implementations. Simplify the code by removing this > option. > > Consequently, the internally created temporary directory only holds the

Re: [Libguestfs] [PATCH v2v v3] lib: Improve security of in/out sockets when running virt-v2v as root

2022-03-22 Thread Richard W.M. Jones
On Tue, Mar 22, 2022 at 09:21:26PM +, Richard W.M. Jones wrote: > +and libvirt_qemu_user () = > + let uid = > +lazy ( ... > +) in > + Lazy.force uid Bleah, this but without the obviously bogus use of lazy(). I've fixed this in my local copy. Rich. --

[Libguestfs] [PATCH v2v v3] lib: Improve security of in/out sockets when running virt-v2v as root

2022-03-22 Thread Richard W.M. Jones
When using the libvirt backend and running as root, libvirt will run qemu as a non-root user (eg. qemu:qemu). The v2v directory stores NBD endpoints that qemu must be able to open and so we set the directory to mode 0711. Unfortunately this permits any non-root user to open the sockets (since,

Re: [Libguestfs] [PATCH v2v v2] lib: Use an ACL to allow qemu to access the v2v directory

2022-03-22 Thread Richard W.M. Jones
On Tue, Mar 22, 2022 at 05:46:18PM +, Richard W.M. Jones wrote: > On Tue, Mar 22, 2022 at 05:34:25PM +0100, Laszlo Ersek wrote: > > Is it OK if we pass "+107" to "setfacl" here? > > Yes it works fine. The "+" is a surprise feature of getpwnam(

Re: [Libguestfs] [PATCH v2v v2] lib: Use an ACL to allow qemu to access the v2v directory

2022-03-22 Thread Richard W.M. Jones
On Tue, Mar 22, 2022 at 05:34:25PM +0100, Laszlo Ersek wrote: > Is it OK if we pass "+107" to "setfacl" here? Yes it works fine. The "+" is a surprise feature of getpwnam(3) in fact, albeit one which doesn't appear to be documented in the man page. Rich. -- Richard Jones, Virtualization

Re: [Libguestfs] [PATCH v2v] lib: Use an ACL to allow qemu to access the v2v directory

2022-03-22 Thread Richard W.M. Jones
On Tue, Mar 22, 2022 at 05:10:07PM +0100, Laszlo Ersek wrote: > On 03/22/22 15:35, Richard W.M. Jones wrote: > > When using the libvirt backend and running as root, libvirt will run > > qemu as a non-root user (eg. qemu:qemu). The v2v directory stores NBD > > endpoints t

[Libguestfs] [PATCH v2v v2] lib: Use an ACL to allow qemu to access the v2v directory

2022-03-22 Thread Richard W.M. Jones
When using the libvirt backend and running as root, libvirt will run qemu as a non-root user (eg. qemu:qemu). The v2v directory stores NBD endpoints that qemu must be able to open and so we set the directory to mode 0711. Unfortunately this permits any non-root user to open the sockets (since,

[Libguestfs] [PATCH v2v] lib: Use an ACL to allow qemu to access the v2v directory

2022-03-22 Thread Richard W.M. Jones
When using the libvirt backend and running as root, libvirt will run qemu as a non-root user (eg. qemu:qemu). The v2v directory stores NBD endpoints that qemu must be able to open and so we set the directory to mode 0711. Unfortunately this permits any non-root user to open the sockets (since,

[Libguestfs] [PATCH v2v] lib: Use an ACL to allow qemu to access the v2v directory

2022-03-22 Thread Richard W.M. Jones
For fuller explanation see: https://bugzilla.redhat.com/show_bug.cgi?id=2066773#c1 I'm not very happy with this patch for a few reasons: - Does every distro use "qemu" as the user that runs qemu? - Having to run an external process (not a big deal, but a bit clumsy) - Aren't ACLs actually

Re: [Libguestfs] [p2v PATCH] build: remove gnulib

2022-03-21 Thread Richard W.M. Jones
On Mon, Mar 21, 2022 at 03:24:46PM +, Daniel P. Berrangé wrote: > On Mon, Mar 21, 2022 at 04:16:18PM +0100, Laszlo Ersek wrote: > > It turns out that all we need from gnulib @ 253f29d8b391 is xstrtoull(), > > ignore_value(), and assure(), when building on Fedora 35 anyway. > > I guess the

Re: [Libguestfs] [p2v PATCH] virt-p2v.pod: fix typo in paragraph about virt-v2v log capture

2022-03-19 Thread Richard W.M. Jones
On Sat, Mar 19, 2022 at 08:08:12AM +0100, Laszlo Ersek wrote: > Commit 9b057cc1ac5f ("p2v: Don't display debugging messages in the run > dialog.", 2016-06-18) contained the following hunk: > > > @@ -749,8 +749,9 @@ file (see above), which in turn references the NBD > > listening port(s) > > of

Re: [Libguestfs] [PATCH] daemon/rpm-c.c: Disable signature checking in librpm

2022-03-17 Thread Richard W.M. Jones
On Thu, Mar 17, 2022 at 03:33:32PM +0100, Laszlo Ersek wrote: > On 03/15/22 11:31, Richard W.M. Jones wrote: > > + rpmVSFlags oflags; > > + > >CAMLparam1 (unitv); > >ts = rpmtsCreate (); > > + > > + /* Disable signature checking (RHBZ#206418

[Libguestfs] [PATCH] daemon/rpm-c.c: Disable signature checking in librpm

2022-03-15 Thread Richard W.M. Jones
/show_bug.cgi?id=2064182 Signed-off-by: Richard W.M. Jones --- daemon/rpm-c.c | 7 +++ 1 file changed, 7 insertions(+) diff --git a/daemon/rpm-c.c b/daemon/rpm-c.c index be0e81e22..f6a7067e1 100644 --- a/daemon/rpm-c.c +++ b/daemon/rpm-c.c @@ -89,8 +89,15 @@ static rpmdbMatchIterator iter; value

Re: [Libguestfs] Real size vs virtual size image question?

2022-03-14 Thread Richard W.M. Jones
On Mon, Mar 14, 2022 at 01:14:27PM -0400, Lonnie Cumberland wrote: > Hello All, > > I have been doing some reading and it seems that for some images created that > the "real" size is not the same as the reported "virtual" size to the > filesystem.  There actually is a name for this, but I cannot

[Libguestfs] ANNOUNCE: virt-v2v 2.0 released

2022-03-14 Thread Richard W.M. Jones
is Plotnikov Kevin Locke Laszlo Ersek Martin Kletzander Mikhail Gordeev Nir Soffer Pino Toscano Richard W.M. Jones Valeriy Vdovin COPYRIGHT Copyright (C) 2009-2022 Red Hat Inc. LICENSE This program is free software; you can redis

[Libguestfs] ANNOUNCE: libguestfs 1.48 and guestfs-tools 1.48 released

2022-03-14 Thread Richard W.M. Jones
on whole block device https://bugzilla.redhat.com/1451665 RFE: Virt-v2v can't convert the guest which has encrypted partition SEE ALSO guestfs-examples(1), guestfs-faq(1), guestfs-performance(1), guestfs-recipes(1), guestfs-testing(1), gue

Re: [Libguestfs] Running lcitool in nbdkit

2022-03-14 Thread Richard W.M. Jones
On Mon, Mar 14, 2022 at 12:57:13PM +0100, Martin Kletzander wrote: > You can also set if for all Alpine versions with Alpine: perl-dev or all > distros using apk package manager with apk: perl-dev. I was wondering about that :-) How about this change?

Re: [Libguestfs] Running lcitool in nbdkit

2022-03-14 Thread Richard W.M. Jones
On Mon, Mar 14, 2022 at 11:33:16AM +, Richard W.M. Jones wrote: > Pipeline is currently running: > > https://gitlab.com/nbdkit/nbdkit/-/pipelines/491516164 I had previously fixed this failure in Alpine: perl.c:45:10: fatal error: XSUB.h: No such file or directory 45 |

Re: [Libguestfs] Running lcitool in nbdkit

2022-03-14 Thread Richard W.M. Jones
To sum it up, I added these commits to nbdkit which converge the generated files with what lcitool now generates: https://gitlab.com/nbdkit/nbdkit/-/commit/8e99576b89b58ad09c634b96cff579dd9a352fc6 https://gitlab.com/nbdkit/nbdkit/-/commit/200a147d6a99c593204b476d3cbb08cc794d834b Pipeline is

Re: [Libguestfs] Running lcitool in nbdkit

2022-03-14 Thread Richard W.M. Jones
On Mon, Mar 14, 2022 at 10:41:05AM +, Richard W.M. Jones wrote: > (b) lcitool wants to remove my perl conditionalization changes: > > @@ -312,11 +309,10 @@ mingw32-fedora-35: > - mingw32-fedora-35-container >allow_failure: false >variables: > -NAME: f

Re: [Libguestfs] Running lcitool in nbdkit

2022-03-14 Thread Richard W.M. Jones
I rebased Martin's tree[1] on top of libvirt-ci head, and reran lcitool. There are several places where I changed the generated files in nbdkit.git and where those changes are overwritten (I didn't realise those files were generated). I think I can fix most of them. But a couple of larger

Re: [Libguestfs] Running lcitool in nbdkit

2022-03-14 Thread Richard W.M. Jones
On Mon, Mar 14, 2022 at 10:01:46AM +, Richard W.M. Jones wrote: > Martin - I guess you must have created that nbdkit.yml file at some > point, but I can't find it in your fork. Found it: https://gitlab.com/nertpinx/libvirt-ci/-/commits/_nbdkit2 I guess we're still missing almalinux s

Re: [Libguestfs] Running lcitool in nbdkit

2022-03-14 Thread Richard W.M. Jones
On Mon, Mar 14, 2022 at 09:18:04AM +, Daniel P. Berrangé wrote: > On Sat, Mar 12, 2022 at 10:50:21AM +0000, Richard W.M. Jones wrote: > > Am I doing this wrong? [see log below] This is using the upstream > > libvirt-ci checked out just now from gitlab, but it seems as if

Re: [Libguestfs] guestfish with support Microsoft ReFS

2022-03-14 Thread Richard W.M. Jones
[Please always CC the mailing list] On Mon, Mar 14, 2022 at 09:24:08AM +0100, Gottschalk wrote: > Hello Rich, > > i installed the ReFS driver on a Testsystem. > > But guestfish doesn't find the ReFS driver which is a Kernelmodul. > > To mount a ReFS Partition i have to type the following

[Libguestfs] Running lcitool in nbdkit

2022-03-12 Thread Richard W.M. Jones
Am I doing this wrong? [see log below] This is using the upstream libvirt-ci checked out just now from gitlab, but it seems as if nbdkit has to be listed as a project under that repo. "./lcitool projects" shows libnbd but not nbdkit. A second question, since the libvirt-ci project does a bunch

[Libguestfs] Coming soon - libguestfs 1.48, guestfs-tools 1.48, virt-v2v 2.0

2022-03-11 Thread Richard W.M. Jones
I've prepared release notes for all three here: https://github.com/libguestfs/libguestfs/blob/master/docs/guestfs-release-notes-1.48.pod https://github.com/rwmjones/guestfs-tools/blob/master/docs/guestfs-tools-release-notes-1.48.pod

Re: [Libguestfs] libguestfs question

2022-03-10 Thread Richard W.M. Jones
On Thu, Mar 10, 2022 at 06:26:54AM -0500, Lonnie Cumberland wrote: > Hi Richard, > > Thanks for getting back to me so quickly on this question and I was afraid > that > this was the case.  > > I will probably definitely still use libguestfs as it does some things that I > really need as well if

Re: [Libguestfs] libguestfs question

2022-03-10 Thread Richard W.M. Jones
On Wed, Mar 09, 2022 at 08:24:27PM -0500, Lonnie Cumberland wrote: [...] > The second part of this is that once the application creates the > virtual disk and adds the database files, I want to keep the handle > open to those files so that as they are used in the application then > the database

Re: [Libguestfs] [PATCH 0/4] Remove some experimental features from libguestfs 1.48

2022-03-09 Thread Richard W.M. Jones
I pushed a more complete fix which removes libguestfs live everywhere, and makes the --live flag on tools print an error. Fixed in libguestfs-common: https://github.com/libguestfs/libguestfs-common/commit/8eaad00c59f3a258b37dd5167ae02d51095a9991 In libguestfs:

<    9   10   11   12   13   14   15   16   17   18   >