Re: [Libguestfs] [PATCH v2] v2v/v2v.ml: Use larger request size for -o rhv-upload

2022-02-15 Thread Richard W.M. Jones
On Tue, Feb 15, 2022 at 12:14:11PM +0100, Laszlo Ersek wrote: > On 02/15/22 11:43, Richard W.M. Jones wrote: > > On Mon, Feb 14, 2022 at 04:08:21PM +, Richard W.M. Jones wrote: > >> On Mon, Feb 14, 2022 at 04:52:17PM +0100, Laszlo Ersek wrote: > >>> On 02/14/22 1

Re: [Libguestfs] libguestfs "test-mdadm.sh" and php tests failing?

2022-02-15 Thread Richard W.M. Jones
On Tue, Feb 15, 2022 at 10:33:18AM +0100, Laszlo Ersek wrote: > Hi, > > on up-to-date Fedora 35, "test-mdadm.sh" in the libguestfs test suite is > failing like this: > > *stdin*:32: libguestfs: error: md_create: mdadm: r1t2: mdadm: specifying > chunk size is forbidden for this level > > I've

Re: [Libguestfs] [PATCH v2] v2v/v2v.ml: Use larger request size for -o rhv-upload

2022-02-15 Thread Richard W.M. Jones
On Mon, Feb 14, 2022 at 04:08:21PM +, Richard W.M. Jones wrote: > On Mon, Feb 14, 2022 at 04:52:17PM +0100, Laszlo Ersek wrote: > > On 02/14/22 14:01, Richard W.M. Jones wrote: > > > But nbdcopy needs to be reworked to make the input and output requests > > >

Re: [Libguestfs] [PATCH v2] v2v/v2v.ml: Use larger request size for -o rhv-upload

2022-02-14 Thread Richard W.M. Jones
On Mon, Feb 14, 2022 at 04:08:21PM +, Richard W.M. Jones wrote: > There is a limit on the number of parallel requests in flight > (nbdcopy --requests, default 64). This limits the implicit buffer to > max_requests * request_size. That's 16MB in the default > configuration.

Re: [Libguestfs] [PATCH v2] v2v/v2v.ml: Use larger request size for -o rhv-upload

2022-02-14 Thread Richard W.M. Jones
On Mon, Feb 14, 2022 at 04:52:17PM +0100, Laszlo Ersek wrote: > On 02/14/22 14:01, Richard W.M. Jones wrote: > > But nbdcopy needs to be reworked to make the input and output requests > > separate, so that nbdcopy will coalesce and split blocks as it copies. > > This is diffic

Re: [Libguestfs] [PATCH v2] v2v/v2v.ml: Use larger request size for -o rhv-upload

2022-02-14 Thread Richard W.M. Jones
On Mon, Feb 14, 2022 at 03:59:34PM +0200, Nir Soffer wrote: > On Mon, Feb 14, 2022 at 3:01 PM Richard W.M. Jones wrote: > > > > On Mon, Feb 14, 2022 at 12:53:01PM +0100, Laszlo Ersek wrote: > > > On 02/14/22 10:56, Richard W.M. Jones wrote: > > > > This change

Re: [Libguestfs] [PATCH v2] v2v/v2v.ml: Use larger request size for -o rhv-upload

2022-02-14 Thread Richard W.M. Jones
On Mon, Feb 14, 2022 at 12:53:01PM +0100, Laszlo Ersek wrote: > On 02/14/22 10:56, Richard W.M. Jones wrote: > > This change slowed things down (slightly) for me, although the change > > is within the margin of error so it probably made no difference. > > > > Before:

Re: [Libguestfs] [PATCH v2] v2v/v2v.ml: Use larger request size for -o rhv-upload

2022-02-14 Thread Richard W.M. Jones
On Mon, Feb 14, 2022 at 01:11:52PM +0200, Nir Soffer wrote: > On Mon, Feb 14, 2022 at 11:56 AM Richard W.M. Jones wrote: > > > > This change slowed things down (slightly) for me, although the change > > is within the margin of error so it probably made no diffe

Re: [Libguestfs] [PATCH v2] v2v/v2v.ml: Use larger request size for -o rhv-upload

2022-02-14 Thread Richard W.M. Jones
This change slowed things down (slightly) for me, although the change is within the margin of error so it probably made no difference. Before: $ time ./run virt-v2v -i disk /var/tmp/fedora-35.qcow2 -o rhv-upload -oc https://ovirt4410/ovirt-engine/api -op /tmp/ovirt-passwd -oo rhv-direct -os

Re: [Libguestfs] [PATCH] v2v/v2v.ml: Use larger request size for -o rhv-upload

2022-02-14 Thread Richard W.M. Jones
On Sun, Feb 13, 2022 at 05:13:37PM +0200, Nir Soffer wrote: > This can be done in nbdcopy by adding http-ops. The pipeline in > rhv-upload case will be: > > nbdkit (any input) -> nbdcopy (nbd-ops/http-ops) -> imageio server > > Another way is to drop nbdcopy for rhv-upload - if we provide

Re: [Libguestfs] [PATCH] v2v/v2v.ml: Use larger request size for -o rhv-upload

2022-02-13 Thread Richard W.M. Jones
On Sun, Feb 13, 2022 at 05:13:37PM +0200, Nir Soffer wrote: > On Sun, Feb 13, 2022 at 11:41 AM Richard W.M. Jones wrote: > > > > On Sat, Feb 12, 2022 at 10:49:42PM +0200, Nir Soffer wrote: > > > rhv-upload plugin is translating every NBD command to HTTP request, >

Re: [Libguestfs] [PATCH] v2v/v2v.ml: Use larger request size for -o rhv-upload

2022-02-13 Thread Richard W.M. Jones
On Sat, Feb 12, 2022 at 10:49:42PM +0200, Nir Soffer wrote: > rhv-upload plugin is translating every NBD command to HTTP request, > translated back to NBD command on imageio server. The HTTP client and > server, and the NBD client on the imageio server side are synchronous > and implemented in

Re: [Libguestfs] [libguestfs-common PATCH 3/3] add common ("standard") option -w / --wrap

2022-02-12 Thread Richard W.M. Jones
etopt.sh b/mltools/test-getopt.sh > index d9919a9a915a..1d7bb7b1dc4f 100755 > --- a/mltools/test-getopt.sh > +++ b/mltools/test-getopt.sh > @@ -68,6 +68,7 @@ $t --short-options | grep '^-is' > $t --short-options | grep '^-t' > $t --short-options | grep '^-V' > $t --short-optio

Re: [Libguestfs] [libguestfs-common PATCH 2/3] Tools_utils.wrap: only wrap text for TTYs

2022-02-12 Thread Richard W.M. Jones
_spaces chan indent; > + output_string chan str > + ) > > and _wrap chan indent column i len str = >if i < len then ( Reviewed-by: Richard W.M. Jones (Now it's irritating me that the function is called istty :-) Rich. -- Richard Jones, Virtualization Group, Red Hat

Re: [Libguestfs] [libguestfs-common PATCH 1/3] Demote "Std_utils.wrap" to an internal function in Tools_utils

2022-02-12 Thread Richard W.M. Jones
-i vmx to read VMware vmx files directly > (RHBZ#1441197).", 2017-04-11). "output_spaces" is generally useful for > formatting indented (not wrapped) debug output. > > Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1820221 > Suggested-by: Richard

Re: [Libguestfs] [PATCH libnbd v2 1/9] golang: tests: Add test for AioBuffer

2022-02-11 Thread Richard W.M. Jones
On Fri, Feb 11, 2022 at 03:21:21AM +0200, Nir Soffer wrote: > Add unit tests and benchmarks for AioBuffer. The tests are trivial but > they server as running documentation, and they point out important > details about the type. > > The benchmarks show the efficiency of allocating a new buffer,

Re: [Libguestfs] [libnbd PATCH 3/3] api: Add new API nbd_set_pread_initialize()

2022-02-10 Thread Richard W.M. Jones
On Thu, Feb 10, 2022 at 03:09:46PM +, Richard W.M. Jones wrote: > For Fedora, I won't do anything now, just let this go out in the next > release. It'll need a small adjustment if we backport it to the > stable-1.10 branch. > > For RHEL, I'm going to put this in RHEL 9.0, b

Re: [Libguestfs] [libnbd PATCH 3/3] api: Add new API nbd_set_pread_initialize()

2022-02-10 Thread Richard W.M. Jones
On Thu, Feb 10, 2022 at 08:50:07AM -0600, Eric Blake wrote: > On Thu, Feb 10, 2022 at 09:38:30AM +0000, Richard W.M. Jones wrote: > > On Wed, Feb 09, 2022 at 04:07:26PM -0600, Eric Blake wrote: > > > + "set_pread_initialize", { > > > +default_call with &g

Re: [Libguestfs] [libnbd PATCH 3/3] api: Add new API nbd_set_pread_initialize()

2022-02-10 Thread Richard W.M. Jones
Link "pread"; Link "pread_structured"; Link "aio_pread"; > +Link "aio_pread_structured"]; > + }; Could it be worth mentioning CVE-2022-0485 by name in the text here? And/or linking to: https://listman.redhat.com/archives/libguestfs/2022

Re: [Libguestfs] [PATCH v2v v2] output: -o rhv-upload: Kill nbdkit instances before finalizing

2022-02-09 Thread Richard W.M. Jones
On Wed, Feb 09, 2022 at 03:51:24PM +0200, Nir Soffer wrote: > On Wed, Feb 9, 2022 at 12:32 PM Richard W.M. Jones wrote: > > + (* We must kill all our nbdkit instances before finalizing the > > + * transfer. See: > > + * > > https://listman.redhat.com/archiv

Re: [Libguestfs] [guestfs-tools PATCH] virt-resize: replace "wrap" calls with calls to "info"

2022-02-09 Thread Richard W.M. Jones
On Wed, Feb 09, 2022 at 02:17:38PM +0100, Laszlo Ersek wrote: > Utilities shouldn't directly call "Std_utils.wrap" for printing > informative messages; the "Tools_utils.info" function handles that better. > > Because "info" prints a trailing newline automatically, strip one newline > from the

Re: [Libguestfs] Performance regression in -o rhv-upload questions

2022-02-09 Thread Richard W.M. Jones
Hi Nir, With my version 2 patch, it now takes about the same time to do a conversion as it did with old virt-v2v. However one thing I noticed is that we lose about 16 seconds between creating the disk and creating the transfer. Notice the timestamps here: 00:00:41 python3

[Libguestfs] [PATCH v2v v2] output: -o rhv-upload: Kill nbdkit instances before finalizing

2022-02-09 Thread Richard W.M. Jones
This addresses points raised in Laszlo's review: https://listman.redhat.com/archives/libguestfs/2022-February/msg00127.html ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs

[Libguestfs] [PATCH v2v v2] output: -o rhv-upload: Kill nbdkit instances before finalizing

2022-02-09 Thread Richard W.M. Jones
If nbdkit instance(s) are still running then they will hold open some http connections to imageio. In some versions of imageio, starting finalization in this state causes a 60 second timeout. See-also: https://listman.redhat.com/archives/libguestfs/2022-February/msg00111.html Thanks: Nir Soffer

Re: [Libguestfs] [nbdkit PATCH] RFC: noextents: Add extentmode=data config parameter

2022-02-08 Thread Richard W.M. Jones
> + > +=item B > + > +(nbdkit E 1.30) > + > +Extent support is advertised, but extent requests from the client will > +be answered with a claim that the entire disk forms a single allocated > +data extent. > + > +=back > + > +All other parameters are passed through

Re: [Libguestfs] [PATCH libnbd] golang: make-dist.sh: Generate the list file

2022-02-08 Thread Richard W.M. Jones
On Mon, Feb 07, 2022 at 01:28:30AM +0200, Nir Soffer wrote: > On Sun, Feb 6, 2022 at 9:13 PM Richard W.M. Jones wrote: > > > > On Sun, Feb 06, 2022 at 07:21:02PM +0200, Nir Soffer wrote: > > > Generated the list file when creating the distribution. Since the Go > &

Re: [Libguestfs] [PATCH 1/5] output/rhv-upload-plugin: Fix flush and close

2022-02-08 Thread Richard W.M. Jones
On Sat, Dec 18, 2021 at 10:36:29PM +0200, Nir Soffer wrote: > When locking the http pool, we wait until all connections are idle, and > take them from the pool. But since we used pool.qsize(), which is the > number of items currently in the queue, we did not wait for all > connections. > > This

Re: [Libguestfs] [PATCH 5/5] output/rhv-upload-plugin: Keep connections alive

2022-02-08 Thread Richard W.M. Jones
On Sat, Dec 18, 2021 at 10:36:33PM +0200, Nir Soffer wrote: > When importing from vddk, nbdcopy may be blocked for few minutes(!) > trying to get extents. While nbdcopy is blocked, imageio server closes > the idle connections. When we finally get a request from nbdcopy, we > fail to detect that

Re: [Libguestfs] [PATCH 3/5] output/rhv-upload-plugin: Extract send_flush() helper

2022-02-08 Thread Richard W.M. Jones
On Sat, Dec 18, 2021 at 10:36:31PM +0200, Nir Soffer wrote: > Extract a helper for sending flush request for single connection, and > inline the iter_http_pool() helper into flush(), its only user. > --- > output/rhv-upload-plugin.py | 54 - > 1 file changed,

Re: [Libguestfs] Performance regression in -o rhv-upload questions

2022-02-08 Thread Richard W.M. Jones
On Tue, Feb 08, 2022 at 04:44:33PM +0200, Nir Soffer wrote: > This happens because virt-v2v try to finalize the transfer *before* closing > the > connections to imageio server. > > Current imageio release mark the a ticket as canceled, but will not remove it > if the ticket has open connections

[Libguestfs] [PATCH v2v] output: -o rhv-upload: Kill nbdkit instances before finalizing

2022-02-08 Thread Richard W.M. Jones
If nbdkit instance(s) are still running then they will hold open some http connections to imageio. In some versions of imageio, starting finalization in this state causes a 60 second timeout. See-also: https://listman.redhat.com/archives/libguestfs/2022-February/msg00111.html Thanks: Nir Soffer

Re: [Libguestfs] Performance regression in -o rhv-upload questions

2022-02-08 Thread Richard W.M. Jones
On Tue, Feb 08, 2022 at 04:44:33PM +0200, Nir Soffer wrote: > On Tue, Feb 8, 2022 at 3:27 PM Richard W.M. Jones wrote: > > virt-v2v 1.44.2: > > Complete log: http://oirase.annexia.org/tmp/virt-v2v-1.44.2-rhv-upload.log > > > > [ 63.1] Copying disk 1/1 > >

[Libguestfs] Performance regression in -o rhv-upload questions

2022-02-08 Thread Richard W.M. Jones
Hi Nir, https://bugzilla.redhat.com/show_bug.cgi?id=2039255#c4 I'm looking for some advice/help with a performance regression in virt-v2v between 1.44.2 and the latest version. It's very obvious and reproducible when I do a conversion from a local disk to local RHV using -o rhv-upload,

Re: [Libguestfs] [libnbd PATCH] copy: Fail nbdcopy if NBD read or write fails

2022-02-08 Thread Richard W.M. Jones
On Fri, Feb 04, 2022 at 09:18:07AM +0100, Laszlo Ersek wrote: > (And there was no easy way to skip connect-tcp6.) Wwe use this in libnbd.spec: https://src.fedoraproject.org/rpms/libnbd/blob/rawhide/f/libnbd.spec#_214 In libguestfs we have a whole infrastructure for skipping tests by setting

Re: [Libguestfs] [PATCH libnbd] golang: make-dist.sh: Generate the list file

2022-02-06 Thread Richard W.M. Jones
On Sun, Feb 06, 2022 at 07:21:02PM +0200, Nir Soffer wrote: > Generated the list file when creating the distribution. Since the Go > tool treat the list file on the proxy server as the source of truth, we > do the same. The new list file is created by downloading the current > list file, sorting

Re: [Libguestfs] [PATCH libnbd] generator/Go.ml: Simplify copy_uint32_array

2022-02-06 Thread Richard W.M. Jones
On Sun, Feb 06, 2022 at 07:45:20PM +0200, Nir Soffer wrote: > Create a slice backed up by the entries pointer, and copy the data with > builtin copy(). This can be 3x times faster but I did not measure it. > > Eric posted a similar patch[1] last year, but the patch seems to be > stuck with

Re: [Libguestfs] [libnbd PATCH v2 5/5] copy: CVE-2022-0485: Fail nbdcopy if NBD read or write fails

2022-02-04 Thread Richard W.M. Jones
On Fri, Feb 04, 2022 at 08:44:00AM -0600, Eric Blake wrote: > On Fri, Feb 04, 2022 at 10:09:26AM +0100, Laszlo Ersek wrote: > > On 02/03/22 21:25, Eric Blake wrote: > > > nbdcopy has a nasty bug when performing multi-threaded copies using > > > asynchronous nbd calls - it was blindly treating the

Re: [Libguestfs] [libnbd PATCH v2 2/5] ocaml: tests: Fix error handling

2022-02-04 Thread Richard W.M. Jones
On Fri, Feb 04, 2022 at 09:41:33AM +0100, Laszlo Ersek wrote: > On 02/03/22 21:25, Eric Blake wrote: > > Like a lot of the C examples, the aio copy test ignores read and write > > errors in the completion callback, which can cause silent data > > corruption. The failure in the test is not

Re: [Libguestfs] [libnbd PATCH v2 5/5] copy: CVE-2022-0485: Fail nbdcopy if NBD read or write fails

2022-02-04 Thread Richard W.M. Jones
On Fri, Feb 04, 2022 at 02:08:21AM +0200, Nir Soffer wrote: > On Thu, Feb 3, 2022 at 10:26 PM Eric Blake wrote: > > + if (*error) { > > +fprintf (stderr, "read at offset 0x%" PRIx64 "failed: %s\n", > > Good error message, but there is missing space before "failed". > > Why use hex offset?

Re: [Libguestfs] [libnbd PATCH v2 5/5] copy: CVE-2022-0485: Fail nbdcopy if NBD read or write fails

2022-02-03 Thread Richard W.M. Jones
On Thu, Feb 03, 2022 at 02:25:58PM -0600, Eric Blake wrote: ... > +# Tests several scenarios of handling NBD server errors > +# Serves as a regression test for the CVE-2022-0485 fix. > + > +. ../tests/functions.sh > + > +set -e > +set -x > + > +requires nbdkit --exit-with-parent --version It's

Re: [Libguestfs] [libnbd PATCH v2 4/5] copy: Pass in dummy variable rather than to callback

2022-02-03 Thread Richard W.M. Jones
On Thu, Feb 03, 2022 at 02:25:57PM -0600, Eric Blake wrote: > In several places where asynch handlers manually call the provided > nbd_completion_callback, the value of errno is indeterminate (for > example, in file-ops.c:file_asynch_read(), the previous call to > file_synch_read() already

Re: [Libguestfs] [PATCH libnbd] golang: tests: Fix error handling

2022-02-03 Thread Richard W.M. Jones
On Thu, Feb 03, 2022 at 09:39:24PM +0200, Nir Soffer wrote: > On Thu, Feb 3, 2022 at 9:24 PM Eric Blake wrote: > > > > On Thu, Feb 03, 2022 at 08:07:52PM +0200, Nir Soffer wrote: > > > Like lot of the C examples, the aio copy test ignores read and write > > > errors in the completion callback,

Re: [Libguestfs] [libnbd PATCH] copy: Fail nbdcopy if NBD read or write fails

2022-02-03 Thread Richard W.M. Jones
On Thu, Feb 03, 2022 at 10:41:10AM +, Richard W.M. Jones wrote: > [Adding some other distro maintainers for awareness. > Note the patch is not complete and upstream yet] > > On Wed, Feb 02, 2022 at 07:50:55PM -0600, Eric Blake wrote: > > FIXME: This is CVE-2022-XXX

Re: [Libguestfs] [libnbd PATCH] copy: Fail nbdcopy if NBD read or write fails

2022-02-03 Thread Richard W.M. Jones
[Adding some other distro maintainers for awareness. Note the patch is not complete and upstream yet] On Wed, Feb 02, 2022 at 07:50:55PM -0600, Eric Blake wrote: > FIXME: This is CVE-2022-X (still awaiting assignment of the CVE number). > > nbdcopy has a nasty bug when performing

Re: [Libguestfs] [libnbd PATCH] docs: Clarify how callbacks should handle errors

2022-02-03 Thread Richard W.M. Jones
On Wed, Feb 02, 2022 at 10:58:10AM -0600, Eric Blake wrote: > Recent patches have demonstrated confusion on which order callbacks > are reached, when it is safe or dangerous to ignore *error, and what a > completion callback should do when auto-retirement is in use. Add > wording to make it more

Re: [Libguestfs] [libnbd PATCH 4/4] copy: Ignore extents if we encountered earlier error

2022-02-03 Thread Richard W.M. Jones
The patch series looks fine, with or without Laszlo's small suggested change. ACK Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual

Re: [Libguestfs] [libnbd PATCH 0/4] Work towards fixing nbdcopy silent corruption

2022-02-02 Thread Richard W.M. Jones
On Wed, Feb 02, 2022 at 04:34:09PM +0100, Laszlo Ersek wrote: > OK -- so end-to-end (IIUC), all I could learn, with reading !err in the > OCaml callback, is whether an earlier invocation of the callback raised > an exception (possibly indirectly, i.e. by returning -1 to the wrapper, > or directly

Re: [Libguestfs] [libnbd PATCH 0/4] Work towards fixing nbdcopy silent corruption

2022-02-02 Thread Richard W.M. Jones
On Wed, Feb 02, 2022 at 09:59:04AM +0100, Laszlo Ersek wrote: > However, in extent_wrapper() -> extent_wrapper_locked(), there is more > confusion for me: > > metacontextv = caml_copy_string (metacontext); > offsetv = caml_copy_int64 (offset); > entriesv =

Re: [Libguestfs] [libnbd PATCH 0/4] Work towards fixing nbdcopy silent corruption

2022-02-02 Thread Richard W.M. Jones
On Wed, Feb 02, 2022 at 09:23:09AM +0100, Laszlo Ersek wrote: > On 02/01/22 20:44, Eric Blake wrote: > > These are the trivial patches modeled after Nir's commit 7ba6ef67, > > fixing further places that fail to check *error during callbacks. > > I find these interfaces very difficult to use. > >

Re: [Libguestfs] [v2v PATCH v2 7/7] convert/convert_windows: consult "best driver"'s dev list for virtio-1.0

2022-02-01 Thread Richard W.M. Jones
ACK series Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported.

Re: [Libguestfs] [PATCH nbdkit v2 4/4] common/allocators: Always align mlock buffer

2022-02-01 Thread Richard W.M. Jones
On Tue, Feb 01, 2022 at 08:22:47AM +0100, Laszlo Ersek wrote: > On 01/31/22 17:14, Richard W.M. Jones wrote: > > > > Thanks - I pushed this as: > > > >521e69f7..24217480 master -> master > > > > I changed patch 3 to include your recommendations.

Re: [Libguestfs] [PATCH nbdkit v2 4/4] common/allocators: Always align mlock buffer

2022-01-31 Thread Richard W.M. Jones
Thanks - I pushed this as: 521e69f7..24217480 master -> master I changed patch 3 to include your recommendations. But also I noticed there was a mistake in the memcpy which I hope I fixed and it's worth reading the comment about this:

Re: [Libguestfs] [PATCH libnbd] copy: Implement destination preferred block size

2022-01-31 Thread Richard W.M. Jones
On Sun, Jan 30, 2022 at 10:42:01PM +0200, Nir Soffer wrote: > On Sun, Jan 30, 2022 at 8:29 PM Richard W.M. Jones wrote: > > > > On Sun, Jan 30, 2022 at 06:18:03PM +0200, Nir Soffer wrote: > > > On Sun, Jan 30, 2022 at 11:10 AM Richard W.M. Jones > > > wrote: >

Re: [Libguestfs] [PATCH nbdkit] plugins: python: Add error example

2022-01-31 Thread Richard W.M. Jones
On Mon, Jan 31, 2022 at 03:17:21PM +0200, Nir Soffer wrote: > On Mon, Jan 31, 2022 at 3:10 PM Richard W.M. Jones wrote: > > > > On Mon, Jan 31, 2022 at 12:17:36AM +0200, Nir Soffer wrote: > > > On Mon, Jan 31, 2022 at 12:14 AM Nir Soffer wrote: > > > >

Re: [Libguestfs] [PATCH nbdkit] plugins: python: Add error example

2022-01-31 Thread Richard W.M. Jones
raise RuntimeError(f"extents error offset={offset} count={count}") > > + > > +# We don't really support extents, so we report the entire file as > > +# data. > > +return [(offset, count, 0)] > > + > > + > > +def pread(h, buf, o

Re: [Libguestfs] [PATCH libnbd] copy: Implement destination preferred block size

2022-01-31 Thread Richard W.M. Jones
On Sun, Jan 30, 2022 at 10:42:01PM +0200, Nir Soffer wrote: > On Sun, Jan 30, 2022 at 8:29 PM Richard W.M. Jones wrote: > > > > On Sun, Jan 30, 2022 at 06:18:03PM +0200, Nir Soffer wrote: > > > On Sun, Jan 30, 2022 at 11:10 AM Richard W.M. Jones > > > wrote: >

Re: [Libguestfs] [PATCH libnbd] copy: Implement destination preferred block size

2022-01-31 Thread Richard W.M. Jones
The way nbdcopy works now is it reads the extent map from the source (in pieces). This extent map is a flat, contiguous array describing data + holes. The size of those regions is whatever the source gives us, and we don't do any further processing on it[1]. Importantly it is unrelated to the

Re: [Libguestfs] [PATCH libnbd] copy: Implement destination preferred block size

2022-01-30 Thread Richard W.M. Jones
On Sun, Jan 30, 2022 at 06:18:03PM +0200, Nir Soffer wrote: > On Sun, Jan 30, 2022 at 11:10 AM Richard W.M. Jones wrote: > > > > On Sun, Jan 30, 2022 at 12:45:37AM +0200, Nir Soffer wrote: > > > On Fri, Jan 28, 2022 at 10:37 PM Richard

Re: [Libguestfs] [PATCH libnbd 9/9] golang: examples: aio_copy: Simplify using AioBuffer

2022-01-30 Thread Richard W.M. Jones
I quickly looked over this and it all looks sensible, ACK. It also looks as if you only added new functions / accessors -- apologies if I missed something - I only gave it a brief scan -- so it should be backwards compatible with existing code. If there's something that is not backwards

Re: [Libguestfs] [PATCH libnbd] copy: Implement destination preferred block size

2022-01-30 Thread Richard W.M. Jones
On Sun, Jan 30, 2022 at 12:45:37AM +0200, Nir Soffer wrote: > On Fri, Jan 28, 2022 at 10:37 PM Richard W.M. Jones wrote: > > + .get_preferred_block_size = nbd_ops_get_preferred_block_size, > > Why preferred block size and not minimum block size? For example if we > write 256

Re: [Libguestfs] [PATCH libnbd] copy: Implement destination preferred block size

2022-01-30 Thread Richard W.M. Jones
On Sun, Jan 30, 2022 at 12:27:53AM +0200, Nir Soffer wrote: > On Fri, Jan 28, 2022 at 10:39 PM Richard W.M. Jones wrote: > Doing read-modify-write in nbdcopy sounds like a bad idea. This is best done > closer to the storage, avoiding reading blocks from qemu-nbd to nbdcopy just > t

[Libguestfs] [PATCH libnbd] copy: Implement destination preferred block size

2022-01-28 Thread Richard W.M. Jones
For destinations, especially NBD, which have a preferred block size, adjust our write strategy so we always write whole blocks / zeroes of the preferred size. --- copy/file-ops.c | 8 copy/main.c | 82 -

[Libguestfs] [PATCH libnbd] copy: Implement destination preferred block size

2022-01-28 Thread Richard W.M. Jones
[NB: I think this is a failed attempt, so shoudn't go upstream, and doesn't need to be reviewed.] When nbdcopy writes to an NBD server it ignores the server's minimum/preferred block size. This hasn't caused a problem til now, but it turns out that in at least one case we care about (writing to

Re: [Libguestfs] [v2v PATCH 6/7] convert/libosinfo_utils: extract "best_driver" from "windows_virtio.ml"

2022-01-28 Thread Richard W.M. Jones
On Fri, Jan 28, 2022 at 05:12:17PM +0100, Laszlo Ersek wrote: > The "copy_from_libosinfo" function in "windows_virtio.ml" filters and > sorts the driver list from libosinfo in order to find the best driver. > Move this logic to a separate function (called "best_driver") in the > Libosinfo_utils

Re: [Libguestfs] [v2v PATCH 5/7] convert/libosinfo_utils: debug-log the devices supported by a driver

2022-01-28 Thread Richard W.M. Jones
On Fri, Jan 28, 2022 at 05:12:16PM +0100, Laszlo Ersek wrote: > While at it, put each file of a driver "pack" on a separate line, and > print "Driver:", "Files:" and "Devices:" headers as well. > > Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=204 > Signed-off-by: Laszlo Ersek > ---

Re: [Libguestfs] [v2v PATCH 4/7] convert/libosinfo: retrieve the device list for OsinfoDeviceDriver

2022-01-28 Thread Richard W.M. Jones
On Fri, Jan 28, 2022 at 05:12:15PM +0100, Laszlo Ersek wrote: > The OsinfoDeviceDriver class supports the > osinfo_device_driver_get_devices() method, we've just had no use for that > thus far. > > Using the previously extracted v2v_osinfo_device_list_to_value_list() > function, we can now

Re: [Libguestfs] [v2v PATCH 3/7] convert/libosinfo: factor out v2v_osinfo_device_list_to_value_list()

2022-01-28 Thread Richard W.M. Jones
On Fri, Jan 28, 2022 at 05:12:14PM +0100, Laszlo Ersek wrote: > Move the guts of v2v_osinfo_os_get_all_devices() to a new static function > called v2v_osinfo_device_list_to_value_list(). > > Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=204 > Signed-off-by: Laszlo Ersek > --- >

Re: [Libguestfs] [v2v PATCH 2/7] convert/windows_virtio: map 32-bit arch name from libguestfs to osinfo

2022-01-28 Thread Richard W.M. Jones
On Fri, Jan 28, 2022 at 05:12:13PM +0100, Laszlo Ersek wrote: > For Windows guests, the "inspect.i_arch" field is ultimately determined in > libguestfs, on the following call path: > > inspect_os [daemon/inspect.ml] > check_for_filesystem_on

[Libguestfs] [PATCH nbdkit v2 3/4] common/utils: Add vector_reserve_page_aligned

2022-01-27 Thread Richard W.M. Jones
Allow vectors to be page aligned (eg. for performance or page protection reasons). To create a page aligned vector you have to use _reserve_page_aligned instead of plain _reserve. Note that _duplicate creates unaligned duplicates. --- configure.ac | 4 ++- common/utils/vector.h | 12

[Libguestfs] [PATCH nbdkit v2 0/4] common/allocators: Always align mlock buffer

2022-01-27 Thread Richard W.M. Jones
Earlier patch was reviewed here: https://listman.redhat.com/archives/libguestfs/2022-January/msg00172.html For this series I pulled out two smaller changes into separate commits (patches 1 & 2). I then decided to modify the vector library to add a new _reserve_page_aligned function which always

[Libguestfs] [PATCH nbdkit v2 4/4] common/allocators: Always align mlock buffer

2022-01-27 Thread Richard W.M. Jones
On platforms with 64K page sizes (RHEL + aarch64 or ppc64le) we have hit a problem in test-memory-allocator-malloc-mlock.sh which appears to be caused by: 1. The test requests a 2K mlocked buffer. 2. The allocated buffer straddles a page boundary sometimes. 3. Linux expands the mlock

[Libguestfs] [PATCH nbdkit v2 1/4] common/allocators/malloc.c: Avoid calling munlock (NULL, ...)

2022-01-27 Thread Richard W.M. Jones
I'm not certain if it is wrong to ever call munlock(2) with the first parameter NULL (second parameter 0). Linux permits it. POSIX doesn't say either way. In any case it's not necessary to do it, so add an extra check to avoid it here. --- common/allocators/malloc.c | 2 +- 1 file changed, 1

[Libguestfs] [PATCH nbdkit v2 2/4] tests: Use a larger buffer for mlock test

2022-01-27 Thread Richard W.M. Jones
This test fails sometimes on machines with 64K page sizes — see following commits — and by using a larger buffer size we can make that a little bit more likely. I increased the buffer size to 10K so that on machines with a 4K page size we are deliberately allocating a partial page which should be

Re: [Libguestfs] [PATCH libnbd] .gitignore: Ignore the new Go examples executables

2022-01-27 Thread Richard W.M. Jones
On Thu, Jan 27, 2022 at 05:17:22PM +0200, Nir Soffer wrote: > Fixes 1dbe4cf5fac3b9455462495fac230c21f1e44049. > > Signed-off-by: Nir Soffer > --- > .gitignore | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/.gitignore b/.gitignore > index 3ecdceaf..395d137b 100644 > --- a/.gitignore

Re: [Libguestfs] [libguestfs/libguestfs] guestfs_copy_in fails with error: copy_in: tar subprocess failed: tar: .: file changed as we read it: errno 0 (Issue #75)

2022-01-27 Thread Richard W.M. Jones
On Wed, Jan 26, 2022 at 09:31:14PM -0800, anemade wrote: > I am using libguestfs Golang binding APIs(version 1.44) > Followed this document https://libguestfs.org/guestfs-golang.3.html to create > the disk, add the disk, format the partition, launch the appliance and > performing some copy_in and

Re: [Libguestfs] [PATCH libnbd] Add .editorconfig file

2022-01-27 Thread Richard W.M. Jones
On Thu, Jan 27, 2022 at 02:22:40AM +0200, Nir Soffer wrote: > This project uses non standard 2 spaces indent in some parts (lib, copy, > info) and tabs in others (e.g. golang/*.go). This can be solved by local > configuration or adding metadata comments to files, but there is a > better solution -

Re: [Libguestfs] [PATCH libnbd 2/2] examples: copy-libev.c: Fix error handling

2022-01-27 Thread Richard W.M. Jones
ACK series Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and

Re: [Libguestfs] [PATCH libnbd 1/2] examples: copy-libev.c: Clear buffers before use

2022-01-27 Thread Richard W.M. Jones
On Thu, Jan 27, 2022 at 01:49:30AM +0200, Nir Soffer wrote: > The example uses a buffer pool for all requests, but it did not clear > the buffers before they were used. If we failed to handle a read error, > this could lead to leaking sensitive data to the destination server. > > Signed-off-by:

Re: [Libguestfs] [PATCH libnbd v2] golang: examples: Add simple_copy and aio_copy examples

2022-01-27 Thread Richard W.M. Jones
On Thu, Jan 27, 2022 at 12:28:48AM +0200, Nir Soffer wrote: > On Wed, Jan 26, 2022 at 11:53 PM Richard W.M. Jones wrote: > > > > > > ACK > > Pushed as 1dbe4cf5fac3b9455462495fac230c21f1e44049 > > Should I mention the new examples in the release notes for 1.1

Re: [Libguestfs] [PATCH libnbd v2] golang: examples: Add simple_copy and aio_copy examples

2022-01-26 Thread Richard W.M. Jones
ACK -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported.

Re: [Libguestfs] [PATCH nbdkit] common/allocators: Always align mlock buffer

2022-01-26 Thread Richard W.M. Jones
On Wed, Jan 26, 2022 at 11:26:25AM +0100, Laszlo Ersek wrote: > On 01/25/22 12:26, Richard W.M. Jones wrote: > > On platforms with 64K page sizes (RHEL + aarch64 or ppc64le) we have > > hit a problem in test-memory-allocator-malloc-mlock.sh which appears > > to be caused by: &

Re: [Libguestfs] [PATCH v2v 2/2] v2v: Send nbdinfo debug information to stderr not stdout

2022-01-26 Thread Richard W.M. Jones
Thanks, these are upstream in: To github.com:libguestfs/virt-v2v.git 0d880dc288..68af35d48c master -> master Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you

[Libguestfs] [PATCH nbdkit] common/allocators: Always align mlock buffer

2022-01-25 Thread Richard W.M. Jones
On platforms with 64K page sizes (RHEL + aarch64 or ppc64le) we have hit a problem in test-memory-allocator-malloc-mlock.sh which appears to be caused by: 1. The test requests a 2K mlocked buffer. 2. The allocated buffer straddles a page boundary sometimes. 3. Linux expands the mlock

[Libguestfs] [PATCH v2v 2/2] v2v: Send nbdinfo debug information to stderr not stdout

2022-01-25 Thread Richard W.M. Jones
When running in verbose mode (virt-v2v -vx) debug messages are supposed to only be written to stderr. This allows virt-p2v and other wrappers to separate ordinary progress messages and warnings, from debugging information. However when we added nbdinfo output (only printed in verbose mode), the

[Libguestfs] [PATCH v2v 1/2] input: libvirt: Share a single connection to the source NBD server

2022-01-25 Thread Richard W.M. Jones
When using virt-p2v from RHEL 7, it starts a very old qemu-nbd server (probably 1.5.3) which required the --shared parameter to enable sharing even in read-only mode. Since it doesn't pass this parameter only a single connection at a time is allowed, and further connections will deadlock. Note

Re: [Libguestfs] [PATCH libnbd] golang: examples: Add simple_copy and aio_copy examples

2022-01-24 Thread Richard W.M. Jones
On Mon, Jan 24, 2022 at 03:35:57PM +0200, Nir Soffer wrote: > Show how to read entire image using the simple synchronous API and how > and the high performance asynchronous API. [...] Looks fine, thanks. ACK Rich. -- Richard Jones, Virtualization Group, Red Hat

Re: [Libguestfs] [PATCH v2v 2/3] lib/qemuNBD.ml: Use qemu-nbd --shared=0 flag to allow multiple connections

2022-01-20 Thread Richard W.M. Jones
On Thu, Jan 20, 2022 at 11:22:28AM -0600, Eric Blake wrote: > On Wed, Jan 19, 2022 at 06:57:48PM +0000, Richard W.M. Jones wrote: > > qemu-nbd --shared (-e) flag controls how many clients can connect > > concurrently. 0 means unlimited. > > As of qemu commit 3dcf56e625 (v6

Re: [Libguestfs] [PATCH v2v 6/7] tests: Add newer RHEL versions to the tests

2022-01-20 Thread Richard W.M. Jones
On Thu, Jan 20, 2022 at 01:28:10PM +0100, Laszlo Ersek wrote: > On 01/19/22 19:03, Richard W.M. Jones wrote: > > --- > > tests/Makefile.am | 3 +++ > > 1 file changed, 3 insertions(+) > > > > diff --git a/tests/Makefile.am b/tests/Makefile.am > &

[Libguestfs] [PATCH v2v 0/3] Fix test-v2v-trim.sh test

2022-01-19 Thread Richard W.M. Jones
Final 3 patches needed to fix "make check-slow", specifically the test-v2v-trim.sh test. This revealed a nasty bug in our sparsification if the input disk format is qcow2. Rich. ___ Libguestfs mailing list Libguestfs@redhat.com

[Libguestfs] [PATCH v2v 1/3] tests/test-v2v-trim.sh: Use -of qcow2 to preserve output format

2022-01-19 Thread Richard W.M. Jones
Old virt-v2v used the same output format as input format when doing disk to disk copies. Modular virt-v2v doesn't do this, it requires using -of to set the output format to the same as the input format if that is desired. --- tests/test-v2v-trim.sh | 2 +- 1 file changed, 1 insertion(+), 1

[Libguestfs] [PATCH v2v 3/3] lib/qemuNBD.ml: Use qemu-nbd --discard=unmap

2022-01-19 Thread Richard W.M. Jones
The default for qemu-nbd is to ignore discard requests. This meant that for input files in qcow2 format the "Mapping ..." (ie. fstrim) step did nothing, all the work was ignored and we copied deleted data over to the destination. This was detected by the test-v2v-trim.sh test. Fixes: commit

[Libguestfs] [PATCH v2v 2/3] lib/qemuNBD.ml: Use qemu-nbd --shared=0 flag to allow multiple connections

2022-01-19 Thread Richard W.M. Jones
qemu-nbd --shared (-e) flag controls how many clients can connect concurrently. 0 means unlimited. We want to allow the sockets to be queried by other processes while virt-v2v is running and it should be safe to do this. The default configuration of qemu-nbd doesn't allow this so add

[Libguestfs] [PATCH v2v 5/7] tests: Drop ancient Ubuntu versions, add Ubuntu 20.04 LTS

2022-01-19 Thread Richard W.M. Jones
Ubuntu 14.04 failed in the --update command. These versions are out of standard support (5 years) so drop them from testing. Add Ubuntu 20.04. --- tests/Makefile.am | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/tests/Makefile.am b/tests/Makefile.am index

[Libguestfs] [PATCH v2v 7/7] tests: Don't use virt-builder --update when testing Windows conversions

2022-01-19 Thread Richard W.M. Jones
The error was: virt-builder: error: cannot use ‘--update’ because no package manager has been detected for this guest OS. If this guest OS is a common one with ordinary package management then this may have been caused by a failure of libguestfs inspection. For OSes such as Windows that lack

[Libguestfs] [PATCH v2v 6/7] tests: Add newer RHEL versions to the tests

2022-01-19 Thread Richard W.M. Jones
--- tests/Makefile.am | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/Makefile.am b/tests/Makefile.am index 3e828558d4..da57e786e9 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -149,6 +149,9 @@ real_guests_scripts = \ test-v2v-conversion-of-rhel-6.8.sh \

[Libguestfs] [PATCH v2v 2/7] output: -o libvirt, qemu: Use correct device name for vsock

2022-01-19 Thread Richard W.M. Jones
According to https://libvirt.org/formatdomain.html#vsock it's not . There is also no model="none", we just omit it if not present. According to https://wiki.qemu.org/Features/VirtioVsock the qemu parameter is -device vhost-vsock-pci. Fixes: 05f780c16f0135c657615520c2245b42de1efc3e ---

[Libguestfs] [PATCH v2v 4/7] tests: Drop test of centos-6

2022-01-19 Thread Richard W.M. Jones
The image cannot be updated, see error below. In any case we are testing a RHEL 6 image so it is not really necessary to continue to test CentOS 6. Loaded plugins: fastestmirror, security Setting up Update Process Error: Cannot retrieve repository metadata (repomd.xml) for repository: base.

[Libguestfs] [PATCH v2v 3/7] tests: Drop some obsolete Fedora versions, start testing F35

2022-01-19 Thread Richard W.M. Jones
For very old Fedora versions, it's no longer possible to download the packages (required by virt-builder --update). We get this error: Error: Cannot retrieve metalink for repository: fedora/20/x86_64. Please verify its path and try again I also dropped Fedora 23 and 29. Although those can be

[Libguestfs] [PATCH v2v 0/7] Mostly fix "make check-slow"

2022-01-19 Thread Richard W.M. Jones
$ make && make check-slow I hadn't run this set of tests in a while and they were all failing. Luckily that was easy to fix (see patches 1 and 2). However we need to change the mix of guests being tested, dropping some old ones which are no longer testable, and adding some newer ones. There is

[Libguestfs] [PATCH v2v 1/7] output: -o libvirt: Fix element port/autoport

2022-01-19 Thread Richard W.M. Jones
According to https://libvirt.org/formatdomain.html only vnc and sdl graphics types support port/autoport. In addition, autoport='yes' is the non-deprecated replacement for port='-1'. Since we don't need to support ancient libvirt, we don't need to keep adding port='-1' in the autoport case.

<    11   12   13   14   15   16   17   18   19   20   >