Re: [Libguestfs] [PATCH v2v] New virt-v2v-inspector tool
Thanks for the review. That's upstream now in commit 0805ea93796b8b57e7c9f0bc04f83ea76a9820a5. I fixed up the commit message and added a non-mandatory -O option to control output. We can keep it experimental status for now and decide before virt-v2v 2.2 (next stable) if we want to keep it or modify it. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH v2v] New virt-v2v-inspector tool
On Thu, Nov 24, 2022 at 10:54:42AM +0100, Laszlo Ersek wrote: > On 11/22/22 16:47, Richard W.M. Jones wrote: > > In Kubernetes and tools like Kubevirt, it's not possible to create > > some disks and then attach to them (in order to populate them with > > data) in one step. This makes virt-v2v conversions awkward because > > ideally we would like the output mode (-o kubevirt) to both create the > > target disks and populate them at the same time. > > > > So to work around this problem, we need a tool which can inspect the > > virt-v2v source hypervisor before we do the conversion in order to > > find out how many disks are needed and their sizes. Then we can > > create the target disks, and then we can run a second container with > > virt-v2v attached to the disks to do the conversion and populate the > > output. > > So, is the population of the *pre-created* disk images (aka volumes) a > feature of the new kubevirt output module? When I get around to finishing it, yes. Basically they will be presented to virt-v2v in the container as /dev devices (PVC "volumeMode: Block"). When running virt-v2v you have to give the names of the devices to virt-v2v, since that is only known by whoever creates the virt-v2v container. Note they are pre-created but not pre-populated, ie. virt-v2v sees correctly sized but empty /dev devices initially. I should also note that raw block device PVCs are very sparsely documented. I wonder if they are not widely used? In particular every worked example I've found copies from the same example in the Kubernetes upstream documentation. Guess I'll find out if it works soon ... > > This is a proposed tool to do this. It essentially uses the same -i* > > options as virt-v2v (and no -o* options) and outputs various useful > > metadata. Example: > > > > $ ./run virt-v2v-inspector --quiet -i disk /var/tmp/fedora-32.img > > virt-v2v-inspector: The QEMU Guest Agent will be installed for this > > guest> > at first boot. > > virt-v2v-inspector: warning: /files/boot/grub2/device.map/hd0 references > > unknown device "vda". You may have to fix this entry manually after > > conversion. > > > > > > > > virt-v2v-inspector > > virt-v2v > > 2.1.9 > > > > > > 6442450944 > > 1400897536 > > > > > > > > linux > > fedora > > fedora32 > > x86_64 > > 32 > > 0 > > rpm > > dnf > > Fedora 32 (Thirty Two) > > > > > > > > There should be sufficient information in the section to > > allocate target disks, plus additional information is printed which > > might be useful. > > > > Note that we do a full conversion in order to generate this > > information. In particular it's not possible to generate the > > estimate without this. It's plausible we could have a > > --no-convert option, but I'm not sure it's worthwhile: it would only > > save a little time, but would make everything less accurate, plus > > maybe it is a good idea to find out if conversion is going to work > > before we create the target disks? > > I think this is a great approach. The current problem from the > "recipient" (kubevirt) side is that temporary storage for the disk > images "in the middle" is really not wanted. This approach prevents just > that. All other logic from virt-v2v remains useful and should be kept IMO. > > > I chose XML instead of JSON for output. XML allows us to annotate > > elements with attributes like "estimated='true'". It also lets us > > represent 64 bit number accurately, where JSON cannot represent such > > numbers. > > > > One obvious problem is that (without --quiet) the program mixes up > > informational output with the final document, which is a bit of a > > pain. Ideas here? > > I think *small* regular files (fitting under $TMPDIR for example) should > be in order, as output. Add an "-o desc.xml" or "-O desc.xml" option to > the new tool, for saving the XML in "desc.xml" at once? (I hope I > understood the "mixing" issue correctly.) Yeah, I was just hoping to avoid having to specify an output file, as outputing to stdout is quite nice. But perhaps it's unavoidable. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH v2v] New virt-v2v-inspector tool
On 11/22/22 16:47, Richard W.M. Jones wrote: > This tool can be used to estimate the disk space needed before doing a > virt-v2v conversion. > > It is a replacement for the old --print-estimate option which was > dropped in virt-v2v 2.0 (commit 5828c9c7d5 "v2v: Remove > --print-estimate option"). > --- > docs/Makefile.am| 15 ++ > docs/test-v2v-docs.sh | 9 + > docs/virt-v2v-inspector.pod | 252 +++ > docs/virt-v2v.pod | 4 + > configure.ac| 1 + > Makefile.am | 3 +- > inspector/Makefile.am | 129 ++ > tests/Makefile.am | 2 + > inspector/inspector.mli | 19 ++ > inspector/inspector.ml | 472 > inspector/dummy.c | 2 + > tests/test-v2v-inspector.sh | 76 ++ > .gitignore | 3 + > run.in | 3 +- > 14 files changed, 988 insertions(+), 2 deletions(-) I've skimmed this. It seems very useful and mostly okay, where "mostly okay" qualifies *my level of understanding* of the code, and not the code itself. There is a bit of code duplication with other parts of the tree, but I think that's entirely OK: this is a new (experimental?) tool, for further feature enablement, so intrusive refactorings are not called for at this point, IMO. I suggest that we merge this ASAP and let people start testing it. One suggestion: migrate the nice info from the cover letter into this actual commit message. Acked-by: Laszlo Ersek Laszlo > > diff --git a/docs/Makefile.am b/docs/Makefile.am > index 3668fd4f0c..012c672294 100644 > --- a/docs/Makefile.am > +++ b/docs/Makefile.am > @@ -24,6 +24,7 @@ EXTRA_DIST = \ > virt-v2v-in-place.pod \ > virt-v2v-input-vmware.pod \ > virt-v2v-input-xen.pod \ > + virt-v2v-inspector.pod \ > virt-v2v-output-local.pod \ > virt-v2v-output-openstack.pod \ > virt-v2v-output-rhv.pod \ > @@ -40,6 +41,7 @@ man_MANS = \ > virt-v2v-in-place.1 \ > virt-v2v-input-vmware.1 \ > virt-v2v-input-xen.1 \ > + virt-v2v-inspector.1 \ > virt-v2v-output-local.1 \ > virt-v2v-output-openstack.1 \ > virt-v2v-output-rhv.1 \ > @@ -53,6 +55,7 @@ noinst_DATA = \ > $(top_builddir)/website/virt-v2v-in-place.1.html \ > $(top_builddir)/website/virt-v2v-input-vmware.1.html \ > $(top_builddir)/website/virt-v2v-input-xen.1.html \ > + $(top_builddir)/website/virt-v2v-inspector.1.html \ > $(top_builddir)/website/virt-v2v-output-local.1.html \ > $(top_builddir)/website/virt-v2v-output-openstack.1.html \ > $(top_builddir)/website/virt-v2v-output-rhv.1.html \ > @@ -117,6 +120,18 @@ stamp-virt-v2v-input-xen.pod: virt-v2v-input-xen.pod > $< > touch $@ > > +virt-v2v-inspector.1 $(top_builddir)/website/virt-v2v-inspector.1.html: > stamp-virt-v2v-inspector.pod > + > +stamp-virt-v2v-inspector.pod: virt-v2v-inspector.pod > + $(PODWRAPPER) \ > + --man virt-v2v-inspector.1 \ > + --html $(top_builddir)/website/virt-v2v-inspector.1.html \ > + --path $(top_srcdir)/common/options \ > + --license GPLv2+ \ > + --warning safe \ > + $< > + touch $@ > + > virt-v2v-output-local.1 > $(top_builddir)/website/virt-v2v-output-local.1.html: > stamp-virt-v2v-output-local.pod > > stamp-virt-v2v-output-local.pod: virt-v2v-output-local.pod > diff --git a/docs/test-v2v-docs.sh b/docs/test-v2v-docs.sh > index 92ae39ee57..c0de5a20ce 100755 > --- a/docs/test-v2v-docs.sh > +++ b/docs/test-v2v-docs.sh > @@ -75,3 +75,12 @@ $srcdir/../podcheck.pl virt-v2v-in-place.pod > virt-v2v-in-place \ > --oo,\ > --op,\ > --os > + > +$srcdir/../podcheck.pl virt-v2v-inspector.pod virt-v2v-inspector \ > + --path $srcdir/../common/options \ > + --ignore=\ > +--ic,\ > +--if,\ > +--io,\ > +--ip,\ > +--it > diff --git a/docs/virt-v2v-inspector.pod b/docs/virt-v2v-inspector.pod > new file mode 100644 > index 00..d2f0b66e4f > --- /dev/null > +++ b/docs/virt-v2v-inspector.pod > @@ -0,0 +1,252 @@ > +=head1 NAME > + > +virt-v2v-inspector - Estimate disk space needed before virt-v2v conversion > + > +=head1 SYNOPSIS > + > + virt-v2v-inspector [-i* options] guest > + > +=head1 DESCRIPTION > + > +Virt-v2v-inspector is a companion tool for L which can be > +used before conversion to estimate the number of output disks and disk > +space that will be required to complete the virt-v2v conversion. The > +common use for this is to preallocate target disks on management > +systems that need this (like Kubevirt). > + > +This manual page only documents the estimation feature, not all of the > +I<-i*> options which are the same as virt-v2v. You should read > +L first. > + > +=head2 Selecting the input guest > + > +You can run virt-v2v-inspector with the same I<-i*> options as > +virt-v2v. (Don't use any I<-o*> options). This will select the guest > +that you want to estimate. > + > +For
Re: [Libguestfs] [PATCH v2v] New virt-v2v-inspector tool
On 11/22/22 16:47, Richard W.M. Jones wrote: > In Kubernetes and tools like Kubevirt, it's not possible to create > some disks and then attach to them (in order to populate them with > data) in one step. This makes virt-v2v conversions awkward because > ideally we would like the output mode (-o kubevirt) to both create the > target disks and populate them at the same time. > > So to work around this problem, we need a tool which can inspect the > virt-v2v source hypervisor before we do the conversion in order to > find out how many disks are needed and their sizes. Then we can > create the target disks, and then we can run a second container with > virt-v2v attached to the disks to do the conversion and populate the > output. So, is the population of the *pre-created* disk images (aka volumes) a feature of the new kubevirt output module? > n > This is a proposed tool to do this. It essentially uses the same -i* > options as virt-v2v (and no -o* options) and outputs various useful > metadata. Example: > > $ ./run virt-v2v-inspector --quiet -i disk /var/tmp/fedora-32.img > virt-v2v-inspector: The QEMU Guest Agent will be installed for this guest > at first boot. > virt-v2v-inspector: warning: /files/boot/grub2/device.map/hd0 references > unknown device "vda". You may have to fix this entry manually after > conversion. > > > > virt-v2v-inspector > virt-v2v > 2.1.9 > > > 6442450944 > 1400897536 > > > > linux > fedora > fedora32 > x86_64 > 32 > 0 > rpm > dnf > Fedora 32 (Thirty Two) > > > > There should be sufficient information in the section to > allocate target disks, plus additional information is printed which > might be useful. > > Note that we do a full conversion in order to generate this > information. In particular it's not possible to generate the > estimate without this. It's plausible we could have a > --no-convert option, but I'm not sure it's worthwhile: it would only > save a little time, but would make everything less accurate, plus > maybe it is a good idea to find out if conversion is going to work > before we create the target disks? I think this is a great approach. The current problem from the "recipient" (kubevirt) side is that temporary storage for the disk images "in the middle" is really not wanted. This approach prevents just that. All other logic from virt-v2v remains useful and should be kept IMO. > I chose XML instead of JSON for output. XML allows us to annotate > elements with attributes like "estimated='true'". It also lets us > represent 64 bit number accurately, where JSON cannot represent such > numbers. > > One obvious problem is that (without --quiet) the program mixes up > informational output with the final document, which is a bit of a > pain. Ideas here? I think *small* regular files (fitting under $TMPDIR for example) should be in order, as output. Add an "-o desc.xml" or "-O desc.xml" option to the new tool, for saving the XML in "desc.xml" at once? (I hope I understood the "mixing" issue correctly.) Laszlo ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
[Libguestfs] [PATCH v2v] New virt-v2v-inspector tool
In Kubernetes and tools like Kubevirt, it's not possible to create some disks and then attach to them (in order to populate them with data) in one step. This makes virt-v2v conversions awkward because ideally we would like the output mode (-o kubevirt) to both create the target disks and populate them at the same time. So to work around this problem, we need a tool which can inspect the virt-v2v source hypervisor before we do the conversion in order to find out how many disks are needed and their sizes. Then we can create the target disks, and then we can run a second container with virt-v2v attached to the disks to do the conversion and populate the output. n This is a proposed tool to do this. It essentially uses the same -i* options as virt-v2v (and no -o* options) and outputs various useful metadata. Example: $ ./run virt-v2v-inspector --quiet -i disk /var/tmp/fedora-32.img virt-v2v-inspector: The QEMU Guest Agent will be installed for this guest at first boot. virt-v2v-inspector: warning: /files/boot/grub2/device.map/hd0 references unknown device "vda". You may have to fix this entry manually after conversion. virt-v2v-inspector virt-v2v 2.1.9 6442450944 1400897536 linux fedora fedora32 x86_64 32 0 rpm dnf Fedora 32 (Thirty Two) There should be sufficient information in the section to allocate target disks, plus additional information is printed which might be useful. Note that we do a full conversion in order to generate this information. In particular it's not possible to generate the estimate without this. It's plausible we could have a --no-convert option, but I'm not sure it's worthwhile: it would only save a little time, but would make everything less accurate, plus maybe it is a good idea to find out if conversion is going to work before we create the target disks? I chose XML instead of JSON for output. XML allows us to annotate elements with attributes like "estimated='true'". It also lets us represent 64 bit number accurately, where JSON cannot represent such numbers. One obvious problem is that (without --quiet) the program mixes up informational output with the final document, which is a bit of a pain. Ideas here? Rich. ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
[Libguestfs] [PATCH v2v] New virt-v2v-inspector tool
This tool can be used to estimate the disk space needed before doing a virt-v2v conversion. It is a replacement for the old --print-estimate option which was dropped in virt-v2v 2.0 (commit 5828c9c7d5 "v2v: Remove --print-estimate option"). --- docs/Makefile.am| 15 ++ docs/test-v2v-docs.sh | 9 + docs/virt-v2v-inspector.pod | 252 +++ docs/virt-v2v.pod | 4 + configure.ac| 1 + Makefile.am | 3 +- inspector/Makefile.am | 129 ++ tests/Makefile.am | 2 + inspector/inspector.mli | 19 ++ inspector/inspector.ml | 472 inspector/dummy.c | 2 + tests/test-v2v-inspector.sh | 76 ++ .gitignore | 3 + run.in | 3 +- 14 files changed, 988 insertions(+), 2 deletions(-) diff --git a/docs/Makefile.am b/docs/Makefile.am index 3668fd4f0c..012c672294 100644 --- a/docs/Makefile.am +++ b/docs/Makefile.am @@ -24,6 +24,7 @@ EXTRA_DIST = \ virt-v2v-in-place.pod \ virt-v2v-input-vmware.pod \ virt-v2v-input-xen.pod \ + virt-v2v-inspector.pod \ virt-v2v-output-local.pod \ virt-v2v-output-openstack.pod \ virt-v2v-output-rhv.pod \ @@ -40,6 +41,7 @@ man_MANS = \ virt-v2v-in-place.1 \ virt-v2v-input-vmware.1 \ virt-v2v-input-xen.1 \ + virt-v2v-inspector.1 \ virt-v2v-output-local.1 \ virt-v2v-output-openstack.1 \ virt-v2v-output-rhv.1 \ @@ -53,6 +55,7 @@ noinst_DATA = \ $(top_builddir)/website/virt-v2v-in-place.1.html \ $(top_builddir)/website/virt-v2v-input-vmware.1.html \ $(top_builddir)/website/virt-v2v-input-xen.1.html \ + $(top_builddir)/website/virt-v2v-inspector.1.html \ $(top_builddir)/website/virt-v2v-output-local.1.html \ $(top_builddir)/website/virt-v2v-output-openstack.1.html \ $(top_builddir)/website/virt-v2v-output-rhv.1.html \ @@ -117,6 +120,18 @@ stamp-virt-v2v-input-xen.pod: virt-v2v-input-xen.pod $< touch $@ +virt-v2v-inspector.1 $(top_builddir)/website/virt-v2v-inspector.1.html: stamp-virt-v2v-inspector.pod + +stamp-virt-v2v-inspector.pod: virt-v2v-inspector.pod + $(PODWRAPPER) \ + --man virt-v2v-inspector.1 \ + --html $(top_builddir)/website/virt-v2v-inspector.1.html \ + --path $(top_srcdir)/common/options \ + --license GPLv2+ \ + --warning safe \ + $< + touch $@ + virt-v2v-output-local.1 $(top_builddir)/website/virt-v2v-output-local.1.html: stamp-virt-v2v-output-local.pod stamp-virt-v2v-output-local.pod: virt-v2v-output-local.pod diff --git a/docs/test-v2v-docs.sh b/docs/test-v2v-docs.sh index 92ae39ee57..c0de5a20ce 100755 --- a/docs/test-v2v-docs.sh +++ b/docs/test-v2v-docs.sh @@ -75,3 +75,12 @@ $srcdir/../podcheck.pl virt-v2v-in-place.pod virt-v2v-in-place \ --oo,\ --op,\ --os + +$srcdir/../podcheck.pl virt-v2v-inspector.pod virt-v2v-inspector \ + --path $srcdir/../common/options \ + --ignore=\ +--ic,\ +--if,\ +--io,\ +--ip,\ +--it diff --git a/docs/virt-v2v-inspector.pod b/docs/virt-v2v-inspector.pod new file mode 100644 index 00..d2f0b66e4f --- /dev/null +++ b/docs/virt-v2v-inspector.pod @@ -0,0 +1,252 @@ +=head1 NAME + +virt-v2v-inspector - Estimate disk space needed before virt-v2v conversion + +=head1 SYNOPSIS + + virt-v2v-inspector [-i* options] guest + +=head1 DESCRIPTION + +Virt-v2v-inspector is a companion tool for L which can be +used before conversion to estimate the number of output disks and disk +space that will be required to complete the virt-v2v conversion. The +common use for this is to preallocate target disks on management +systems that need this (like Kubevirt). + +This manual page only documents the estimation feature, not all of the +I<-i*> options which are the same as virt-v2v. You should read +L first. + +=head2 Selecting the input guest + +You can run virt-v2v-inspector with the same I<-i*> options as +virt-v2v. (Don't use any I<-o*> options). This will select the guest +that you want to estimate. + +For example to estimate the space required for a guest in a stored +local disk called F you could do: + + virt-v2v-inspector -i disk filename.img + +=head2 Output + +The output from this tool is an XML document (written to stdout). + +=over 4 + +=item * + +Fields which are annotated with an C attribute are +estimated. Virt-v2v cannot always know exactly the final size of some +things, such as the exact real size of the output disk, since there +might be small perturbations between runs. Estimates are usually very +close to the final values. + +=item * + +Elements (including sub-trees) which are annotated with an +C attribute are for information only. These +elements might be changed or removed in future versions. If you would +like to rely on this data in your program please contact the +developers. +
