Re: [Libguestfs] [libnbd PATCH v2 5/5] copy: CVE-2022-0485: Fail nbdcopy if NBD read or write fails

2022-02-03 Thread Nir Soffer
and others which > showcase ways in which sparse images can bypass errors. > > Once backports are complete, a followup patch on the main branch will > edit docs/libnbd-security.pod with the mailing list announcement of > the stable branch commit ids and release versions that incorpor

Re: [Libguestfs] [libnbd PATCH v2 4/5] copy: Pass in dummy variable rather than &errno to callback

2022-02-03 Thread Nir Soffer
On Thu, Feb 3, 2022 at 11:46 PM Richard W.M. Jones wrote: > > On Thu, Feb 03, 2022 at 02:25:57PM -0600, Eric Blake wrote: > > In several places where asynch handlers manually call the provided > > nbd_completion_callback, the value of errno is indeterminate (for > > example, in file-ops.c:file_asy

Re: [Libguestfs] [libnbd PATCH v2 3/5] docs: Clarify how callbacks should handle errors

2022-02-03 Thread Nir Soffer
()On Thu, Feb 3, 2022 at 10:28 PM Eric Blake wrote: > > Recent patches have demonstrated confusion on the order in which > callbacks are reached, when it is safe or dangerous to ignore *error, > and what a completion callback should do when auto-retirement is in > use. Add wording to make it more

Re: [Libguestfs] [libnbd PATCH v2 1/5] python: tests: Fix error handling

2022-02-03 Thread Nir Soffer
On Thu, Feb 3, 2022 at 10:26 PM Eric Blake wrote: > > Like a lot of the C examples, the aio copy test ignores read and write > errors in the completion callback, which can cause silent data > corruption. The failure in the test is not critical, but this is a bad > example that may be copied by dev

Re: [Libguestfs] [PATCH libnbd] golang: tests: Fix error handling

2022-02-03 Thread Nir Soffer
On Thu, Feb 3, 2022 at 10:52 PM Eric Blake wrote: > > On Thu, Feb 03, 2022 at 10:42:09PM +0200, Nir Soffer wrote: > > > > I wondered why we are using these numbers. I think using same pattern > > > > like {lang}/tests/{name}_test.{suffix} would be nicer. > >

Re: [Libguestfs] [PATCH libnbd] golang: tests: Fix error handling

2022-02-03 Thread Nir Soffer
On Thu, Feb 3, 2022 at 9:55 PM Richard W.M. Jones wrote: > > On Thu, Feb 03, 2022 at 09:39:24PM +0200, Nir Soffer wrote: > > On Thu, Feb 3, 2022 at 9:24 PM Eric Blake wrote: > > > > > > On Thu, Feb 03, 2022 at 08:07:52PM +0200, Nir Soffer wrote: > > > >

Re: [Libguestfs] [PATCH libnbd] golang: tests: Fix error handling

2022-02-03 Thread Nir Soffer
On Thu, Feb 3, 2022 at 9:24 PM Eric Blake wrote: > > On Thu, Feb 03, 2022 at 08:07:52PM +0200, Nir Soffer wrote: > > Like lot of the C examples, the aio copy test ignores read and write > > errors in the completion callback, which can cause silent data > > corruption. Th

Re: [Libguestfs] [PATCH libnbd 8/9] golang: aio_buffer.go: Benchmark copy flows

2022-02-03 Thread Nir Soffer
On Tue, Feb 1, 2022 at 3:15 PM Eric Blake wrote: > > On Sun, Jan 30, 2022 at 01:33:36AM +0200, Nir Soffer wrote: > > Add benchmark for coping a buffer using 3 strategies - reusing same > > buffer, making a new uninitialized buffer per copy, and using a zeroed > > buffer

[Libguestfs] [PATCH libnbd] golang: tests: Fix error handling

2022-02-03 Thread Nir Soffer
completion callback fails, similar to other Go examples. Signed-off-by: Nir Soffer --- golang/libnbd_590_aio_copy_test.go | 12 ++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/golang/libnbd_590_aio_copy_test.go b/golang/libnbd_590_aio_copy_test.go index 570635c9..8be0ad2d

[ovirt-users] Re: Unclear how to add local (POSIX) storage

2022-02-03 Thread Nir Soffer
On Thu, Feb 3, 2022 at 3:42 PM Richard W.M. Jones wrote: > > On Thu, Feb 03, 2022 at 03:07:20PM +0200, Nir Soffer wrote: > > On Thu, Feb 3, 2022 at 2:30 PM Richard W.M. Jones wrote: > > > > > > > > > I'm following the instructions here: > &g

[ovirt-users] Re: Unclear how to add local (POSIX) storage

2022-02-03 Thread Nir Soffer
On Thu, Feb 3, 2022 at 2:30 PM Richard W.M. Jones wrote: > > > I'm following the instructions here: > https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/html/administration_guide/sect-preparing_and_adding_posix_compliant_file_system_storage > > I've also added an extra virtua

Re: [Libguestfs] [libnbd PATCH] copy: Fail nbdcopy if NBD read or write fails

2022-02-03 Thread Nir Soffer
ust be careful that we do not leak an unknown value of errno on paths > where we did not encounter a failure. > > Reported-by: Nir Soffer > Fixes: bc896eec4d ("copy: Implement multi-conn, multiple threads, multiple > requests in flight.", v1.5.6) > Fixes: https://bugzilla.re

Re: [Libguestfs] [libnbd PATCH] docs: Clarify how callbacks should handle errors

2022-02-03 Thread Nir Soffer
On Wed, Feb 2, 2022 at 7:00 PM Eric Blake wrote: > > Recent patches have demonstrated confusion on which order callbacks > are reached, when it is safe or dangerous to ignore *error, and what a > completion callback should do when auto-retirement is in use. Add > wording to make it more obvious t

[ovirt-users] Re: Ovirt host error [Cannot configure LVM filter]

2022-02-01 Thread Nir Soffer
# multipath -ll It is strange that you don't have any multipath device - but "vdsm-tool config-lvm-filter" found the wwid 362cea7f0c4c75d0028def9c015471c79 and want to blacklist it. > [root@iondelsvr14 ~]# > > > On Tue, Feb 1, 2022 at 7:55 PM Nir Soffer wrote: >> >> On Tue

[ovirt-users] Re: Ovirt host error [Cannot configure LVM filter]

2022-02-01 Thread Nir Soffer
On Tue, Feb 1, 2022 at 4:17 PM Ayansh Rocks wrote: > > Hello All, > > I was adding few new hosts into the ovirt cluster but a couple of them gave > this error. > > Ovirt Release - 4.4.10 > > Installing Host iondelsvr14.iontrading.com. Check for LVM filter > configuration error: Cannot configure

[Libguestfs] [PATCH nbdkit] plugins: python: Make threading model explicit

2022-02-01 Thread Nir Soffer
The default threading model is already serializing all requests, but implementing thread_model() gives us good place to explain why the code works and why we don't use os.preadv and os.pwritev. --- plugins/python/examples/error.py | 8 1 file changed, 8 insertions(+) diff --git a/plugins

Re: [Libguestfs] [PATCH nbdkit] plugins: python: Add error example

2022-02-01 Thread Nir Soffer
On Mon, Jan 31, 2022 at 3:10 PM Richard W.M. Jones wrote: > > On Mon, Jan 31, 2022 at 12:17:36AM +0200, Nir Soffer wrote: > > On Mon, Jan 31, 2022 at 12:14 AM Nir Soffer wrote: > > > > > > This plugin simulates errors in pread, pwrite, and extents. This is > >

Re: [Libguestfs] [PATCH libnbd 8/9] golang: aio_buffer.go: Benchmark copy flows

2022-02-01 Thread Nir Soffer
On Tue, Feb 1, 2022 at 3:15 PM Eric Blake wrote: > > On Sun, Jan 30, 2022 at 01:33:36AM +0200, Nir Soffer wrote: > > Add benchmark for coping a buffer using 3 strategies - reusing same > > buffer, making a new uninitialized buffer per copy, and using a zeroed > > buffer

Re: [Libguestfs] [PATCH nbdkit] plugins: python: Add error example

2022-01-31 Thread Nir Soffer
On Mon, Jan 31, 2022 at 3:10 PM Richard W.M. Jones wrote: > > On Mon, Jan 31, 2022 at 12:17:36AM +0200, Nir Soffer wrote: > > On Mon, Jan 31, 2022 at 12:14 AM Nir Soffer wrote: > > > > > > This plugin simulates errors in pread, pwrite, and extents. This is > >

Re: [Libguestfs] [PATCH libnbd] copy: Implement destination preferred block size

2022-01-31 Thread Nir Soffer
On Mon, Jan 31, 2022 at 11:49 AM Laszlo Ersek wrote: > > On 01/28/22 21:36, Richard W.M. Jones wrote: > > [NB: I think this is a failed attempt, so shoudn't go upstream, and > > doesn't need to be reviewed.] > > > > When nbdcopy writes to an NBD server it ignores the server's > > minimum/preferred

Re: [Libguestfs] [PATCH nbdkit] plugins: python: Add error example

2022-01-30 Thread Nir Soffer
On Mon, Jan 31, 2022 at 12:14 AM Nir Soffer wrote: > > This plugin simulates errors in pread, pwrite, and extents. This is > useful for testing error handling in NBD clients, and understanding how > plugin exceptions are reported to the NBD client. > --- > plugins/python/Make

[Libguestfs] [PATCH nbdkit] plugins: python: Add error example

2022-01-30 Thread Nir Soffer
This plugin simulates errors in pread, pwrite, and extents. This is useful for testing error handling in NBD clients, and understanding how plugin exceptions are reported to the NBD client. --- plugins/python/Makefile.am | 1 + plugins/python/examples/error.py | 77 +

Re: [Libguestfs] [PATCH libnbd] copy: Implement destination preferred block size

2022-01-30 Thread Nir Soffer
On Sun, Jan 30, 2022 at 8:29 PM Richard W.M. Jones wrote: > > On Sun, Jan 30, 2022 at 06:18:03PM +0200, Nir Soffer wrote: > > On Sun, Jan 30, 2022 at 11:10 AM Richard W.M. Jones > > wrote: > > > > > > On Sun, Jan 30, 2022 at 12:45:37AM +0200, Nir Soffer wrote

Re: [Libguestfs] [PATCH libnbd 2/2] examples: copy-libev.c: Fix error handling

2022-01-30 Thread Nir Soffer
On Thu, Jan 27, 2022 at 8:36 PM Eric Blake wrote: > > On Thu, Jan 27, 2022 at 01:49:31AM +0200, Nir Soffer wrote: > > This example failed to check the *error parameter to the completion and > > extent callbacks. > > > > - If the source NBD server failed a read

Re: [Libguestfs] [PATCH libnbd] copy: Implement destination preferred block size

2022-01-30 Thread Nir Soffer
On Sun, Jan 30, 2022 at 11:10 AM Richard W.M. Jones wrote: > > On Sun, Jan 30, 2022 at 12:45:37AM +0200, Nir Soffer wrote: > > On Fri, Jan 28, 2022 at 10:37 PM Richard W.M. Jones > > wrote: > > > + .get_preferred_block_size = nbd_ops_get_preferred_block_size, >

Re: [Libguestfs] [PATCH libnbd 9/9] golang: examples: aio_copy: Simplify using AioBuffer

2022-01-30 Thread Nir Soffer
On Sun, Jan 30, 2022 at 11:15 AM Richard W.M. Jones wrote: > > > I quickly looked over this and it all looks sensible, ACK. > > It also looks as if you only added new functions / accessors -- > apologies if I missed something - I only gave it a brief scan -- so it > should be backwards compatible

[Libguestfs] [PATCH libnbd 8/9] golang: aio_buffer.go: Benchmark copy flows

2022-01-29 Thread Nir Soffer
5320 ns/op BenchmarkAioBufferCopyMakeZero-12 728940 8218 ns/op Signed-off-by: Nir Soffer --- golang/libnbd_620_aio_buffer_test.go | 32 1 file changed, 32 insertions(+) diff --git a/golang/libnbd_620_aio_buffer_test.go b/golang

[Libguestfs] [PATCH libnbd 9/9] golang: examples: aio_copy: Simplify using AioBuffer

2022-01-29 Thread Nir Soffer
mmand, and free it when the request is finished. Signed-off-by: Nir Soffer --- golang/examples/aio_copy/aio_copy.go | 29 +--- 1 file changed, 5 insertions(+), 24 deletions(-) diff --git a/golang/examples/aio_copy/aio_copy.go b/golang/examples/aio_copy/aio_copy.go

[Libguestfs] [PATCH libnbd 7/9] golang: aio_buffer.go: Speed up FromBytes()

2022-01-29 Thread Nir Soffer
: Nir Soffer --- golang/aio_buffer.go | 7 ++- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/golang/aio_buffer.go b/golang/aio_buffer.go index 788fe1a8..7960e554 100644 --- a/golang/aio_buffer.go +++ b/golang/aio_buffer.go @@ -47,25 +47,22 @@ func MakeAioBuffer(size uint

[Libguestfs] [PATCH libnbd 6/9] golang: tests: Use AioBuffer.Slice()

2022-01-29 Thread Nir Soffer
Slice() is easier to use and faster than Get() or Bytes(). Lets use the new way. Signed-off-by: Nir Soffer --- golang/libnbd_500_aio_pread_test.go | 2 +- golang/libnbd_510_aio_pwrite_test.go | 8 +--- golang/libnbd_620_aio_buffer_test.go | 8 +--- 3 files changed, 11 insertions(+), 7

[Libguestfs] [PATCH libnbd 5/9] golang: aio_buffer.go: Add Slice()

2022-01-29 Thread Nir Soffer
[User: 394.2 ms, System: 76.6 ms] Range (min … max): 450.6 ms … 476.0 ms3 runs Summary './aio_copy-slice nbd+unix:///?socket=/tmp/src.sock >/dev/null' ran 2.62 ± 0.09 times faster than './aio_copy-bytes nbd+unix:///?socket=/tmp/src.sock >/dev/null' Signed-

[Libguestfs] [PATCH libnbd 4/9] golang: aio_buffer.go: Add MakeAioBufferZero()

2022-01-29 Thread Nir Soffer
[]byte slice: BenchmarkMakeAioBufferZero-12 247710 4440 ns/op BenchmarkMakeByteSlice-12 84117 13733 ns/op Signed-off-by: Nir Soffer --- golang/aio_buffer.go | 6 ++ golang/libnbd_620_aio_buffer_test.go | 16

[Libguestfs] [PATCH libnbd 3/9] golang: aio_buffer.go: Add missing documentation

2022-01-29 Thread Nir Soffer
Add standard function documentation comments. The documentation should be available here: https://pkg.go.dev/libguestfs.org/libnbd#AioBuffer Signed-off-by: Nir Soffer --- golang/aio_buffer.go | 12 1 file changed, 12 insertions(+) diff --git a/golang/aio_buffer.go b/golang

[Libguestfs] [PATCH libnbd 2/9] golang: aio_buffer.go: Make it safer to use

2022-01-29 Thread Nir Soffer
in the program. Trying to use AioBuffer with libnbd API will likely segfault and panic. I did not try to test this. Signed-off-by: Nir Soffer --- golang/aio_buffer.go | 5 +++- golang/libnbd_620_aio_buffer_test.go | 41 2 files changed, 45 insertions

[Libguestfs] [PATCH libnbd 1/9] golang: tests: Add test for AioBuffer

2022-01-29 Thread Nir Soffer
ro-12 17551 69552 ns/op BenchmarkFromBytes-12 9632139112 ns/op BenchmarkAioBufferBytes-12 69375 16410 ns/op PASS ok libguestfs.org/libnbd 5.843s Signed-off-by: Nir Soffer --- golang/Makefile.am | 1 + g

[Libguestfs] [PATCH libnbd 0/9] golang: Safer, easier to use, and faster AioBuffer

2022-01-29 Thread Nir Soffer
is fully documnted now. Nir Soffer (9): golang: tests: Add test for AioBuffer golang: aio_buffer.go: Make it safer to use golang: aio_buffer.go: Add missing documentation golang: aio_buffer.go: Add MakeAioBufferZero() golang: aio_buffer.go: Add Slice() golang: tests: Use AioBuffer.Slice

Re: [Libguestfs] [PATCH libnbd] copy: Implement destination preferred block size

2022-01-29 Thread Nir Soffer
On Fri, Jan 28, 2022 at 10:37 PM Richard W.M. Jones wrote: > > For destinations, especially NBD, which have a preferred block size, > adjust our write strategy so we always write whole blocks / zeroes of > the preferred size. > --- > copy/file-ops.c | 8 > copy/main.c

Re: [Libguestfs] [PATCH libnbd] copy: Implement destination preferred block size

2022-01-29 Thread Nir Soffer
On Fri, Jan 28, 2022 at 10:39 PM Richard W.M. Jones wrote: > > [NB: I think this is a failed attempt, so shoudn't go upstream, and > doesn't need to be reviewed.] > > When nbdcopy writes to an NBD server it ignores the server's > minimum/preferred block size. This hasn't caused a problem til now,

Re: [Libguestfs] [PATCH libnbd] .gitignore: Ignore the new Go examples executables

2022-01-27 Thread Nir Soffer
On Thu, Jan 27, 2022 at 6:22 PM Richard W.M. Jones wrote: > > On Thu, Jan 27, 2022 at 05:17:22PM +0200, Nir Soffer wrote: > > Fixes 1dbe4cf5fac3b9455462495fac230c21f1e44049. > > > > Signed-off-by: Nir Soffer > > --- > > .gitignore | 2 ++ > > 1 file cha

[Libguestfs] [PATCH libnbd] .gitignore: Ignore the new Go examples executables

2022-01-27 Thread Nir Soffer
Fixes 1dbe4cf5fac3b9455462495fac230c21f1e44049. Signed-off-by: Nir Soffer --- .gitignore | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.gitignore b/.gitignore index 3ecdceaf..395d137b 100644 --- a/.gitignore +++ b/.gitignore @@ -72,22 +72,24 @@ Makefile.in /examples/copy-libev /fuse

Re: [Libguestfs] [PATCH libnbd] Add .editorconfig file

2022-01-27 Thread Nir Soffer
On Thu, Jan 27, 2022 at 10:51 AM Richard W.M. Jones wrote: > > On Thu, Jan 27, 2022 at 02:22:40AM +0200, Nir Soffer wrote: > > This project uses non standard 2 spaces indent in some parts (lib, copy, > > info) and tabs in others (e.g. golang/*.go). This can be solved by local &g

Re: [Libguestfs] [PATCH libnbd 2/2] examples: copy-libev.c: Fix error handling

2022-01-27 Thread Nir Soffer
On Thu, Jan 27, 2022 at 10:50 AM Richard W.M. Jones wrote: > > > ACK series Pushed as 7ba6ef679ff28d234a2f16d4a330d5a42dbdf98e ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs

[Libguestfs] [PATCH libnbd] Add .editorconfig file

2022-01-26 Thread Nir Soffer
://editorconfig.org/ [2] https://github.com/editorconfig/editorconfig-vim#readme [3] https://github.com/editorconfig/editorconfig-emacs#readme Signed-off-by: Nir Soffer --- .editorconfig | 19 +++ Makefile.am | 1 + 2 files changed, 20 insertions(+) create mode 100644

[Libguestfs] [PATCH libnbd 1/2] examples: copy-libev.c: Clear buffers before use

2022-01-26 Thread Nir Soffer
The example uses a buffer pool for all requests, but it did not clear the buffers before they were used. If we failed to handle a read error, this could lead to leaking sensitive data to the destination server. Signed-off-by: Nir Soffer --- examples/copy-libev.c | 9 - 1 file changed, 8

[Libguestfs] [PATCH libnbd 2/2] examples: copy-libev.c: Fix error handling

2022-01-26 Thread Nir Soffer
don't need to check it, since we already handle a NULL extents array by disabling can_extents. Here is an example failure: $ ./copy-libev nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock copy-libev: r0: read failed: Input/output error $ echo $? 1 Signed-off-by:

[Libguestfs] [PATCH libnbd 0/2] Fix error handling in copy-libev example

2022-01-26 Thread Nir Soffer
sesitive data the the destinatino server. Nir Soffer (2): examples: copy-libev.c: Clear buffers before use examples: copy-libev.c: Fix error handling examples/copy-libev.c | 28 +++- 1 file changed, 27 insertions(+), 1 deletion(-) -- 2.34.1

Re: [Libguestfs] [PATCH libnbd v2] golang: examples: Add simple_copy and aio_copy examples

2022-01-26 Thread Nir Soffer
On Wed, Jan 26, 2022 at 11:53 PM Richard W.M. Jones wrote: > > > ACK Pushed as 1dbe4cf5fac3b9455462495fac230c21f1e44049 Should I mention the new examples in the release notes for 1.12? ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.

[Libguestfs] [PATCH libnbd v2] golang: examples: Add simple_copy and aio_copy examples

2022-01-26 Thread Nir Soffer
ran 1.00 ± 0.02 times faster than 'nbdcopy --connections 1 --requests 4 --allocated nbd+unix:///?socket=/tmp/nbd.sock null:' 1.20 ± 0.02 times faster than 'nbdcopy --allocated nbd+unix:///?socket=/tmp/nbd.sock null:' 2.24 ± 0.04 times faster than 'nbdcopy --all

Re: [Libguestfs] [PATCH libnbd] golang: examples: Add simple_copy and aio_copy examples

2022-01-26 Thread Nir Soffer
On Mon, Jan 24, 2022 at 6:13 PM Richard W.M. Jones wrote: > > On Mon, Jan 24, 2022 at 03:35:57PM +0200, Nir Soffer wrote: > > Show how to read entire image using the simple synchronous API and how > > and the high performance asynchronous API. > [...] > > Looks fine, th

[ovirt-users] Re: What happened to oVirt engine-setup?

2022-01-26 Thread Nir Soffer
On Wed, Jan 26, 2022 at 8:32 AM Yedidyah Bar David wrote: > > > Do we recommend RHEL or CentOS as the initial OS? > > > > For 4.4.10 your best option is RHEL 8.5. > > At least in theory. Do you do this yourself? Heard of others doing it? I've been running oVirt on RHEL for years for development.

[ovirt-users] Re: What happened to oVirt engine-setup?

2022-01-25 Thread Nir Soffer
On Tue, Jan 25, 2022 at 7:21 PM Richard W.M. Jones wrote: > > On Tue, Jan 25, 2022 at 06:37:24PM +0200, Nir Soffer wrote: > > On Tue, Jan 25, 2022 at 5:46 PM Richard W.M. Jones > > wrote: > > > > > > A while back I had oVirt 4.4.7 installed which I used for

[ovirt-users] Re: What happened to oVirt engine-setup?

2022-01-25 Thread Nir Soffer
On Tue, Jan 25, 2022 at 5:46 PM Richard W.M. Jones wrote: > > A while back I had oVirt 4.4.7 installed which I used for testing. > For some reason that installation has died in some way, so I'm trying > to install a fresh new oVirt 4.4.10. > > Last time I installed ovirt, it was very easy - I prov

[Libguestfs] [PATCH libnbd] golang: examples: Add simple_copy and aio_copy examples

2022-01-24 Thread Nir Soffer
ran 1.00 ± 0.02 times faster than 'nbdcopy --connections 1 --requests 4 --allocated nbd+unix:///?socket=/tmp/nbd.sock null:' 1.20 ± 0.02 times faster than 'nbdcopy --allocated nbd+unix:///?socket=/tmp/nbd.sock null:' 2.24 ± 0.04 times faster than 'nbdcop

Re: [PATCH v2 2/2] iotests/block-status-cache: New test

2022-01-18 Thread Nir Soffer
we can only use the raw format > +iotests.main(supported_fmts=['raw'], > + supported_protocols=['file']) > diff --git a/tests/qemu-iotests/tests/block-status-cache.out > b/tests/qemu-iotests/tests/block-status-cache.out > new file mode 100644 > index 00..ae1213e6f8 > --- /dev/null > +++ b/tests/qemu-iotests/tests/block-status-cache.out > @@ -0,0 +1,5 @@ > +. > +-- > +Ran 1 tests > + > +OK > -- > 2.33.1 > The out file is not very useful, and even fragile - if the test framework will change the output format, the test will fail. Ideally we depend only on the relevant output of our tools, and using a different version of the test framework on replacing it (e.g pytest) will not require modifying the out files. Regardless I would like to see this fix merged and this issue already exists in other tests. Some tests in tests/ do have useful output that can make debugging failures easier. Reviewed-by: Nir Soffer

Re: [PATCH v2 2/2] iotests/block-status-cache: New test

2022-01-18 Thread Nir Soffer
we can only use the raw format > +iotests.main(supported_fmts=['raw'], > + supported_protocols=['file']) > diff --git a/tests/qemu-iotests/tests/block-status-cache.out > b/tests/qemu-iotests/tests/block-status-cache.out > new file mode 100644 > index 00..ae1213e6f8 > --- /dev/null > +++ b/tests/qemu-iotests/tests/block-status-cache.out > @@ -0,0 +1,5 @@ > +. > +-- > +Ran 1 tests > + > +OK > -- > 2.33.1 > The out file is not very useful, and even fragile - if the test framework will change the output format, the test will fail. Ideally we depend only on the relevant output of our tools, and using a different version of the test framework on replacing it (e.g pytest) will not require modifying the out files. Regardless I would like to see this fix merged and this issue already exists in other tests. Some tests in tests/ do have useful output that can make debugging failures easier. Reviewed-by: Nir Soffer

Re: [PATCH 1/2] block/io: Update BSC only if want_zero is true

2022-01-17 Thread Nir Soffer
data, and so we would update the block-status cache with wrong > information. > > Therefore, we should not update the cache with want_zero=false. > > Reported-by: Nir Soffer > Fixes: 0bc329fbb009f8601cec23bf2bc48ead0c5a5fa2 >("block: block-status cache for da

Re: [PATCH 1/2] block/io: Update BSC only if want_zero is true

2022-01-17 Thread Nir Soffer
data, and so we would update the block-status cache with wrong > information. > > Therefore, we should not update the cache with want_zero=false. > > Reported-by: Nir Soffer > Fixes: 0bc329fbb009f8601cec23bf2bc48ead0c5a5fa2 >("block: block-status cache for da

Re: [PATCH 2/2] iotests/block-status-cache: New test

2022-01-17 Thread Nir Soffer
nd after querying > ' + > + 'qemu:allocation-depth') > +print('Before:') > + print(map_pre) > +print('After:') > +print(map_post) > + > +self.fail("Map information differs") > + > + > +if __name__ == '__main__': > +# The block-status cache only works on the protocol layer, so to test it, > +# we can only use the raw format > +iotests.main(supported_fmts=['raw'], > + supported_protocols=['file']) > diff --git a/tests/qemu-iotests/tests/block-status-cache.out > b/tests/qemu-iotests/tests/block-status-cache.out > new file mode 100644 > index 00..ae1213e6f8 > --- /dev/null > +++ b/tests/qemu-iotests/tests/block-status-cache.out > @@ -0,0 +1,5 @@ > +. > +-- > +Ran 1 tests > + > +OK > -- > 2.33.1 > Reviewed-by: Nir Soffer

Re: [PATCH 2/2] iotests/block-status-cache: New test

2022-01-17 Thread Nir Soffer
nd after querying > ' + > + 'qemu:allocation-depth') > +print('Before:') > + print(map_pre) > +print('After:') > +print(map_post) > + > +self.fail("Map information differs") > + > + > +if __name__ == '__main__': > +# The block-status cache only works on the protocol layer, so to test it, > +# we can only use the raw format > +iotests.main(supported_fmts=['raw'], > + supported_protocols=['file']) > diff --git a/tests/qemu-iotests/tests/block-status-cache.out > b/tests/qemu-iotests/tests/block-status-cache.out > new file mode 100644 > index 00..ae1213e6f8 > --- /dev/null > +++ b/tests/qemu-iotests/tests/block-status-cache.out > @@ -0,0 +1,5 @@ > +. > +-- > +Ran 1 tests > + > +OK > -- > 2.33.1 > Reviewed-by: Nir Soffer

Re: Inconsistent block status reply in qemu-nbd in qemu-img-6.2.0

2022-01-17 Thread Nir Soffer
On Mon, Jan 17, 2022 at 10:46 AM Hanna Reitz wrote: > > On 16.01.22 19:09, Nir Soffer wrote: > > On Sun, Jan 16, 2022 at 1:17 PM Nir Soffer wrote: > >> Some of our tests started to fail since qemu-img 6.2.0 released. > >> > >> Here is an example failure:

Re: Inconsistent block status reply in qemu-nbd in qemu-img-6.2.0

2022-01-16 Thread Nir Soffer
On Sun, Jan 16, 2022 at 1:17 PM Nir Soffer wrote: > > Some of our tests started to fail since qemu-img 6.2.0 released. > > Here is an example failure: > > $ truncate -s1g /data/scratch/empty-1g.raw > > $ qemu-nbd -r -t -e8 -k /tmp/nbd.sock -A -f raw /data/scratch/empty-1g.

[ovirt-devel] Re: imageio failure (was: [oVirt/ovirt-release] Failed repoclosure job (Issue #71))

2022-01-16 Thread Nir Soffer
On Sun, Jan 16, 2022 at 12:18 PM Yedidyah Bar David wrote: > > On Sun, Jan 16, 2022 at 10:50 AM Nir Soffer wrote: > > > > On Sun, Jan 16, 2022 at 10:03 AM Yedidyah Bar David wrote: > >> > >> On Sun, Jan 16, 2022 at 9:11 AM github-actions[bot] > >>

Inconsistent block status reply in qemu-nbd in qemu-img-6.2.0

2022-01-16 Thread Nir Soffer
Some of our tests started to fail since qemu-img 6.2.0 released. Here is an example failure: $ truncate -s1g /data/scratch/empty-1g.raw $ qemu-nbd -r -t -e8 -k /tmp/nbd.sock -A -f raw /data/scratch/empty-1g.raw & $ nbdinfo --map nbd+unix:///?socket=/tmp/nbd.sock 0 10737418243 hol

[ovirt-devel] Re: imageio failure (was: [oVirt/ovirt-release] Failed repoclosure job (Issue #71))

2022-01-16 Thread Nir Soffer
On Sun, Jan 16, 2022 at 10:03 AM Yedidyah Bar David wrote: > On Sun, Jan 16, 2022 at 9:11 AM github-actions[bot] > wrote: > > > > ❌ The repoclosure CI job is still failing. Please investigate. > > > https://github.com/oVirt/ovirt-release/runs/4830975790?check_suite_focus=true > > Copr repo for o

Re: [Libguestfs] Virt-v2v performance benchmarking part 3

2022-01-12 Thread Nir Soffer
On Tue, Jan 11, 2022 at 1:00 PM Richard W.M. Jones wrote: > > On Tue, Jan 11, 2022 at 08:07:39AM +0100, Laszlo Ersek wrote: > > h, not necessarily; according to the manual, "qemu-img convert" uses > > (by default) 8 co-routines. There's also the -W flag ("out of order > > writes"), which I don

Re: [Libguestfs] Virt-v2v performance benchmarking part 3

2022-01-12 Thread Nir Soffer
On Wed, Jan 12, 2022 at 1:12 PM Laszlo Ersek wrote: > > On 01/11/22 11:56, Richard W.M. Jones wrote: > > On Tue, Jan 11, 2022 at 08:07:39AM +0100, Laszlo Ersek wrote: > >> h, not necessarily; according to the manual, "qemu-img convert" uses > >> (by default) 8 co-routines. There's also the -W

[ovirt-devel] Re: git backup

2022-01-12 Thread Nir Soffer
On Wed, Dec 29, 2021 at 12:37 PM Yedidyah Bar David wrote: > > Hi all, > > With the decision and on-going process to migrate from gerrit to > github, we do not anymore have a backup - github used to be a backup > for gerrit, automatically synced. > > Do we want a backup for github? Some options: >

[PATCH] nbd/server.c: Remove unused field

2022-01-11 Thread Nir Soffer
NBDRequestData struct has unused QSIMPLEQ_ENTRY filed. It seems that this field exists since the first git commit and was never used. Signed-off-by: Nir Soffer --- nbd/server.c | 1 - 1 file changed, 1 deletion(-) diff --git a/nbd/server.c b/nbd/server.c index 3927f7789d..ce5b2a1d02 100644

[PATCH] nbd/server.c: Remove unused field

2022-01-11 Thread Nir Soffer
NBDRequestData struct has unused QSIMPLEQ_ENTRY filed. It seems that this field exists since the first git commit and was never used. Signed-off-by: Nir Soffer --- nbd/server.c | 1 - 1 file changed, 1 deletion(-) diff --git a/nbd/server.c b/nbd/server.c index 3927f7789d..ce5b2a1d02 100644

[ovirt-users] Re: Error while removing snapshot: Unable to get volume info

2022-01-10 Thread Nir Soffer
On Mon, Jan 10, 2022 at 5:22 PM Francesco Lorenzini via Users < users@ovirt.org> wrote: > My problem should be the same as the one filed here: > https://bugzilla.redhat.com/show_bug.cgi?id=1948599 > > So, if I'm correct I must edit DB entries to fix the situations. Although > I don't like to opera

[ovirt-devel] VDSM have moved to github

2022-01-09 Thread Nir Soffer
We are not using the gerrit project now, and it was changed to read-only. We are accepting now pull requests on github: https://github.com/oVirt/vdsm/pulls To contribute, use the standard github flow: 1. Fork vdsm on github Example: https://github.com/nirs/vdsm (you need to create a github

Re: [Libguestfs] [PATCH 5/5] output/rhv-upload-plugin: Keep connections alive

2022-01-04 Thread Nir Soffer
On Tue, Jan 4, 2022 at 6:29 PM Nir Soffer wrote: > > On Tue, Jan 4, 2022 at 6:02 PM Richard W.M. Jones wrote: > > > > On Sat, Dec 18, 2021 at 10:36:33PM +0200, Nir Soffer wrote: > > > When importing from vddk, nbdcopy may be blocked for few minutes(!) > > >

Re: [Libguestfs] [PATCH 0/5] Fix rhv-upload output

2022-01-04 Thread Nir Soffer
On Tue, Jan 4, 2022 at 6:06 PM Richard W.M. Jones wrote: > > On Tue, Jan 04, 2022 at 03:56:03PM +, Richard W.M. Jones wrote: > > Patch 2 is definitely wrong. > > I should add > > ... and no longer necessary since Laszlo's fix went upstream: > > https://github.com/libguestfs/virt-v2v/commit/07b

Re: [Libguestfs] [PATCH 5/5] output/rhv-upload-plugin: Keep connections alive

2022-01-04 Thread Nir Soffer
On Tue, Jan 4, 2022 at 6:02 PM Richard W.M. Jones wrote: > > On Sat, Dec 18, 2021 at 10:36:33PM +0200, Nir Soffer wrote: > > When importing from vddk, nbdcopy may be blocked for few minutes(!) > > trying to get extents. While nbdcopy is blocked, imageio server closes > &

Re: [Libguestfs] [PATCH 0/5] Fix rhv-upload output

2022-01-04 Thread Nir Soffer
On Sat, Dec 18, 2021 at 10:36 PM Nir Soffer wrote: > > Fix problems in new rhv-upload implementation: > > - The plugin does not flush to all connections in flush() > - The plugin does not close all connections in cleanup() > - Idle connections are closed in imageio server, a

Re: [Libguestfs] [PATCH nbdkit 1/2] tests/test-python-plugin.py: Allow test to use large disks

2021-12-21 Thread Nir Soffer
On Wed, Dec 22, 2021 at 12:36 AM Richard W.M. Jones wrote: > > On Tue, Dec 21, 2021 at 10:25:35PM +, Richard W.M. Jones wrote: > > On Wed, Dec 22, 2021 at 12:21:25AM +0200, Nir Soffer wrote: > > > It would be help also if we can test extents argument, currently the >

Re: [Libguestfs] [PATCH nbdkit 2/2] python: More precise Python parameter passing

2021-12-21 Thread Nir Soffer
On Wed, Dec 22, 2021 at 12:24 AM Richard W.M. Jones wrote: > > On Wed, Dec 22, 2021 at 12:03:44AM +0200, Nir Soffer wrote: > > This is not needed. I think the accepted way to pass boolean values > > to python is "i". Internally python True and False are 1 and 0, and

Re: [Libguestfs] [PATCH nbdkit 1/2] tests/test-python-plugin.py: Allow test to use large disks

2021-12-21 Thread Nir Soffer
On Tue, Dec 21, 2021 at 11:21 PM Richard W.M. Jones wrote: > > The Python test harness uses a plugin which always created a fully > allocated disk backed by an in-memory bytearray. This prevented us > from testing very large disks (since we could run out of memory > easily). > > Add a feature all

Re: [Libguestfs] [PATCH nbdkit 2/2] python: More precise Python parameter passing

2021-12-21 Thread Nir Soffer
On Tue, Dec 21, 2021 at 11:21 PM Richard W.M. Jones wrote: > > Nir Soffer's commit 715e9c2386 ("plugins/python: Fix extents() count > format string") fixed the signedness of the count parameter of the > extents call. > > Following this commit I looked at a few other places where we pass > paramete

[ovirt-devel] Re: [ovirt-users] using stop_reason as a vdsm hook trigger into the UI

2021-12-20 Thread Nir Soffer
On Mon, Dec 20, 2021 at 9:59 PM Nathanaël Blanchet wrote: Adding the devel list since question is more about extending oVirt ... > The idea is to use the stop_reason element into the vm xml definition. But > after hours, I realized that this element is writed to the vm definition file > only af

[ovirt-users] Re: using stop_reason as a vdsm hook trigger into the UI

2021-12-20 Thread Nir Soffer
On Mon, Dec 20, 2021 at 9:59 PM Nathanaël Blanchet wrote: Adding the devel list since question is more about extending oVirt ... > The idea is to use the stop_reason element into the vm xml definition. But > after hours, I realized that this element is writed to the vm definition file > only af

[ovirt-users] Re: vdsm-client delete_checkpoints

2021-12-20 Thread Nir Soffer
On Mon, Dec 20, 2021 at 7:42 PM Tommaso - Shellrent via Users wrote: > > Hi, someone can give to use us an exemple of the command > > vdsm-client VM delete_checkpoints > > ? > > we have tried a lot of combinations like: > > vdsm-client VM delete_checkpoints vmID="ce5d0251-e971-4d89-be1b-4bc2828361

Re: [Libguestfs] [v2v PATCH] output_rhv: restrict block status collection to the old RHV output

2021-12-20 Thread Nir Soffer
On Mon, Dec 20, 2021 at 6:20 PM Nir Soffer wrote: Attached also output of another run using -v v2v.out.xz Description: application/xz ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs

Re: [Libguestfs] [v2v PATCH] output_rhv: restrict block status collection to the old RHV output

2021-12-20 Thread Nir Soffer
ch suggests that we explicitly restrict the "get_disk_allocated" call > with a new optional boolean parameter to the one output plugin that really > needs it, namely the old RHV one. > > Accordingly, revert the VDSM test case to its state at (a2a4f7a09996^). > > Cc: Nir Soff

Re: [Libguestfs] [PATCH 2/5] v2v/lib/util.ml: Get disk allocation from input

2021-12-20 Thread Nir Soffer
On Mon, Dec 20, 2021 at 12:04 PM Laszlo Ersek wrote: > > On 12/19/21 00:30, Nir Soffer wrote: > > > Laszlo, can you explain why we need the number of allocated bytes > > after the disk was already converted to the target storage? > > It's all described in comm

Re: [Libguestfs] [PATCH 2/5] v2v/lib/util.ml: Get disk allocation from input

2021-12-18 Thread Nir Soffer
On Sun, Dec 19, 2021 at 1:30 AM Nir Soffer wrote: > > On Sun, Dec 19, 2021 at 12:40 AM Nir Soffer wrote: > > > > On Sat, Dec 18, 2021 at 11:33 PM Richard W.M. Jones > > wrote: > > > > > > On Sat, Dec 18, 2021 at 10:36:30PM +0200, Nir Soffer wrote: &

[Libguestfs] [PATCH] output/rhv-upload-plugin: Support extents

2021-12-18 Thread Nir Soffer
Add extents support using imageio /extents API: https://ovirt.github.io/ovirt-imageio/images.html#extents Imageio does not support yet partial extents, so we cache entire image extents, and search for the requested range. The search can probably be faster using binary search, but it seems good eno

[Libguestfs] [PATCH nbdkit] plugins/python: Fix extents() count format string

2021-12-18 Thread Nir Soffer
The plugin used "i" (int32) instead of "I" (uint32) for the count, so when the client asks for 4294966784 bytes, the python plugin got -512. nbdkit: python.0: debug: python: extents count=4294966784 offset=0 req_one=0 ... nbdkit: python.0: debug: extents: count=-512 offset=0 flags=0 W

Re: [Libguestfs] [PATCH 2/5] v2v/lib/util.ml: Get disk allocation from input

2021-12-18 Thread Nir Soffer
On Sun, Dec 19, 2021 at 12:40 AM Nir Soffer wrote: > > On Sat, Dec 18, 2021 at 11:33 PM Richard W.M. Jones wrote: > > > > On Sat, Dec 18, 2021 at 10:36:30PM +0200, Nir Soffer wrote: > > > After finalizing the transfer, virt-v2v try to connect to the output > > &

Re: [Libguestfs] [PATCH 2/5] v2v/lib/util.ml: Get disk allocation from input

2021-12-18 Thread Nir Soffer
On Sat, Dec 18, 2021 at 11:33 PM Richard W.M. Jones wrote: > > On Sat, Dec 18, 2021 at 10:36:30PM +0200, Nir Soffer wrote: > > After finalizing the transfer, virt-v2v try to connect to the output > > socket and query disk allocation. This may work for some outputs > > supp

[Libguestfs] [PATCH 2/5] v2v/lib/util.ml: Get disk allocation from input

2021-12-18 Thread Nir Soffer
After finalizing the transfer, virt-v2v try to connect to the output socket and query disk allocation. This may work for some outputs supporting block status, but for rhv_upload output this cannot work for 2 reasons: - The rhv-upload-plugin does not support extents - The transfer was finalized befo

[Libguestfs] [PATCH 5/5] output/rhv-upload-plugin: Keep connections alive

2021-12-18 Thread Nir Soffer
When importing from vddk, nbdcopy may be blocked for few minutes(!) trying to get extents. While nbdcopy is blocked, imageio server closes the idle connections. When we finally get a request from nbdcopy, we fail to detect that the connection was closed. Detecting a closed connection is hard and r

[Libguestfs] [PATCH 3/5] output/rhv-upload-plugin: Extract send_flush() helper

2021-12-18 Thread Nir Soffer
Extract a helper for sending flush request for single connection, and inline the iter_http_pool() helper into flush(), its only user. --- output/rhv-upload-plugin.py | 54 - 1 file changed, 23 insertions(+), 31 deletions(-) diff --git a/output/rhv-upload-plugin

[Libguestfs] [PATCH 4/5] output/rhv-upload-plugin: Track http last request time

2021-12-18 Thread Nir Soffer
Track the last time a connection was used. This will be used to detect idle connections. --- output/rhv-upload-plugin.py | 30 -- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/output/rhv-upload-plugin.py b/output/rhv-upload-plugin.py index f7e5950f..8d

[Libguestfs] [PATCH 1/5] output/rhv-upload-plugin: Fix flush and close

2021-12-18 Thread Nir Soffer
When locking the http pool, we wait until all connections are idle, and take them from the pool. But since we used pool.qsize(), which is the number of items currently in the queue, we did not wait for all connections. This leads to following issues: - We send flush request only for some connecti

[Libguestfs] [PATCH 0/5] Fix rhv-upload output

2021-12-18 Thread Nir Soffer
e output. Tested on RHEL 8.6 with upstream nbdkit and libnbd. [1] https://github.com/oVirt/ovirt-imageio/pull/15 Fixes https://bugzilla.redhat.com/2032324 Nir Soffer (5): output/rhv-upload-plugin: Fix flush and close v2v/lib/util.ml: Get disk allocation from input output/rhv-upload-plugin:

[ovirt-devel] Re: Vdsm: Failing storage tests when run by non-root in a container

2021-12-16 Thread Nir Soffer
On Thu, Dec 16, 2021 at 10:47 PM Nir Soffer wrote: > On Thu, Dec 16, 2021 at 10:14 PM Milan Zamazal > wrote: > > > > Hi, > > > > when I run Vdsm tests as a non-root user in vdsm-test-centos-8 > > container, they systematically fail on several st

[ovirt-devel] Re: Vdsm: Failing storage tests when run by non-root in a container

2021-12-16 Thread Nir Soffer
On Thu, Dec 16, 2021 at 10:14 PM Milan Zamazal wrote: > > Hi, > > when I run Vdsm tests as a non-root user in vdsm-test-centos-8 > container, they systematically fail on several storage tests. I run the > tests as > > podman run -it -v $HOME:$HOME --userns=keep-id vdsm-test-centos-8 $HOME/test.

[ovirt-devel] Re: Vdsm: Failing storage test

2021-12-16 Thread Nir Soffer
On Thu, Dec 16, 2021 at 1:36 PM Milan Zamazal wrote: > > Hi, > > it seems a recent commit causes a storage test failure on GitHub: > > == FAILURES > === > test_prepare_connection_without_initiator_name > ___

Re: CI not working again

2021-12-14 Thread Nir Soffer
On Tue, Dec 14, 2021 at 1:09 PM Nir Soffer wrote: > > On Tue, Dec 14, 2021 at 12:13 PM Nir Soffer wrote: > ... > > > > Should we disable all vdsm tests on gerrit, and enable only x64_64 el8 > > > > build artifacts > > > > so we can at least g

<    1   2   3   4   5   6   7   8   9   10   >