Re: [PATCH 1/1] block: improve alignment detection and fix 271 test

2023-10-14 Thread Nir Soffer
the test to reflect real world usage? _reset_img 2083k I guess it works with: _reset_img 2084k Commits > commit a6b257a08e3d72219f03e461a52152672fec0612 > Author: Nir Soffer > Date: Tue Aug 13 21:21:03 2019 +0300 > file-posix: Handle undetectable alignme

Re: [Libguestfs] [PATCH v2 1/6] spec: Recommend cap on NBD_REPLY_TYPE_BLOCK_STATUS length

2023-03-06 Thread Nir Soffer
On Sun, Mar 5, 2023 at 10:42 AM Wouter Verhelst wrote: > > On Fri, Mar 03, 2023 at 04:17:40PM -0600, Eric Blake wrote: > > On Fri, Dec 16, 2022 at 10:32:01PM +0300, Vladimir Sementsov-Ogievskiy > > wrote: > > > s-o-b line missed. > > > > I'm not sure if the NBD project has a strict policy on

Re: [Libguestfs] [PATCH] docs: Prefer 'cookie' over 'handle'

2023-03-04 Thread Nir Soffer
On Sat, Mar 4, 2023 at 12:15 AM Eric Blake wrote: > > In libnbd, we quickly learned that distinguishing between 'handle' > (verb for acting on an object) and 'handle' (noun describing which > object to act on) could get confusing; we solved it by renaming the > latter to 'cookie'. Copy that

Re: [PATCH v2 2/5] Support format or cache specific out file

2022-12-13 Thread Nir Soffer
On Tue, Dec 13, 2022 at 8:09 PM Hanna Reitz wrote: > > On 13.12.22 16:56, Nir Soffer wrote: > > On Mon, Dec 12, 2022 at 12:38 PM Hanna Reitz wrote: > >> On 28.11.22 15:15, Nir Soffer wrote: > >>> Extend the test finder to find tests with format (*.o

Re: [PATCH v2 2/5] Support format or cache specific out file

2022-12-13 Thread Nir Soffer
On Mon, Dec 12, 2022 at 12:38 PM Hanna Reitz wrote: > > On 28.11.22 15:15, Nir Soffer wrote: > > Extend the test finder to find tests with format (*.out.qcow2) or cache > > specific (*.out.nocache) out file. This worked before only for the > > numbered tests. > >

[PATCH v2 5/5] qemu-img: Speed up checksum

2022-11-28 Thread Nir Soffer
5s ±0.183s | 1.808s ±0.016s | x1.36 | | nbd | 6g | direct| 3.540s ±0.020s | 1.749s ±0.018s | x2.02 | [1] raw image full of zeroes [2] raw fedora 35 image with additional random data, 50% full [3] image [2] exported by qemu-nbd via unix socket Signed-off-by: Nir Soffer --- qemu-i

[PATCH v2 1/5] qemu-img.c: Move IO_BUF_SIZE to the top of the file

2022-11-28 Thread Nir Soffer
This macro is used by various commands (compare, convert, rebase) but it is defined somewhere in the middle of the file. I'm going to use it in the new checksum command so lets clean up a bit before that. --- qemu-img.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git

[PATCH v2 4/5] iotests: Test qemu-img checksum

2022-11-28 Thread Nir Soffer
debugging in case of failures, the output includes a json map of the test image. Signed-off-by: Nir Soffer --- tests/qemu-iotests/tests/qemu-img-checksum| 63 +++ .../tests/qemu-img-checksum.out.qcow2 | 11 .../tests/qemu-img-checksum.out.raw | 10 +++ 3

[PATCH v2 2/5] Support format or cache specific out file

2022-11-28 Thread Nir Soffer
Extend the test finder to find tests with format (*.out.qcow2) or cache specific (*.out.nocache) out file. This worked before only for the numbered tests. --- tests/qemu-iotests/findtests.py | 10 -- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git

[PATCH v2 0/5] Add qemu-img checksum command using blkhash

2022-11-28 Thread Nir Soffer
tml - https://lists.nongnu.org/archive/html/qemu-block/2022-11/msg00171.html - https://lists.nongnu.org/archive/html/qemu-block/2022-11/msg00173.html Nir Soffer (5): qemu-img.c: Move IO_BUF_SIZE to the top of the file Support format or cache specific out file qemu-img: Add checksum command iote

[PATCH v2 3/5] qemu-img: Add checksum command

2022-11-28 Thread Nir Soffer
ng checksum for 8T empty image: qemu-img checksum: 3.7s, sha256sum (estimate): 17,749s Signed-off-by: Nir Soffer --- docs/tools/qemu-img.rst | 24 ++ meson.build | 10 ++- meson_options.txt | 2 + qemu-img-cmds.hx| 8 ++ qemu-img.c | 183 +

Re: [PATCH 2/3] iotests: Test qemu-img checksum

2022-11-28 Thread Nir Soffer
On Mon, Nov 7, 2022 at 1:41 PM Hanna Reitz wrote: > On 30.10.22 18:38, Nir Soffer wrote: > > On Wed, Oct 26, 2022 at 4:31 PM Hanna Reitz wrote: > > > > On 01.09.22 16:32, Nir Soffer wrote: > > > Add simple tests creating an image with all kinds

Re: [PATCH 1/3] qemu-img: Add checksum command

2022-11-28 Thread Nir Soffer
On Mon, Nov 7, 2022 at 12:20 PM Hanna Reitz wrote: > On 30.10.22 18:37, Nir Soffer wrote: > > On Wed, Oct 26, 2022 at 4:00 PM Hanna Reitz wrote: > > > > On 01.09.22 16:32, Nir Soffer wrote: > [...] > > > --- > > > docs/tools/qemu-i

Re: [PATCH 3/3] qemu-img: Speed up checksum

2022-10-30 Thread Nir Soffer
On Sun, Oct 30, 2022 at 7:38 PM Nir Soffer wrote: > On Wed, Oct 26, 2022 at 4:54 PM Hanna Reitz wrote: > >> On 01.09.22 16:32, Nir Soffer wrote: >> > [...] > > +/* The current chunk. */ >> > +int64_t offset; >> > +int64_t length; >>

Re: [PATCH 3/3] qemu-img: Speed up checksum

2022-10-30 Thread Nir Soffer
On Wed, Oct 26, 2022 at 4:54 PM Hanna Reitz wrote: > On 01.09.22 16:32, Nir Soffer wrote: > > Add coroutine based loop inspired by `qemu-img convert` design. > > > > Changes compared to `qemu-img convert`: > > > > - State for the entire image is kept in

Re: [PATCH 2/3] iotests: Test qemu-img checksum

2022-10-30 Thread Nir Soffer
On Wed, Oct 26, 2022 at 4:31 PM Hanna Reitz wrote: > On 01.09.22 16:32, Nir Soffer wrote: > > Add simple tests creating an image with all kinds of extents, different > > formats, different backing chain, different protocol, and different > > image options. Since all image

Re: [PATCH 1/3] qemu-img: Add checksum command

2022-10-30 Thread Nir Soffer
On Wed, Oct 26, 2022 at 4:00 PM Hanna Reitz wrote: > On 01.09.22 16:32, Nir Soffer wrote: > > The checksum command compute a checksum for disk image content using the > > blkhash library[1]. The blkhash library is not packaged yet, but it is > > available via copr[2]

Re: [PATCH 0/3] Add qemu-img checksum command using blkhash

2022-10-18 Thread Nir Soffer
On Sun, Sep 18, 2022 at 12:35 PM Nir Soffer wrote: > ping > > Kevin, Hanna, I hope you have time to take a look. > > https://lists.nongnu.org/archive/html/qemu-block/2022-09/msg00021.html Ping again, hopefully someone has time to look at this :-) > > > > On Thu,

Re: [PATCH 0/3] Add qemu-img checksum command using blkhash

2022-09-18 Thread Nir Soffer
ping Kevin, Hanna, I hope you have time to take a look. https://lists.nongnu.org/archive/html/qemu-block/2022-09/msg00021.html On Thu, Sep 1, 2022 at 5:32 PM Nir Soffer wrote: > > Since blkhash is available only via copr now, the new command is added as > optional feature, b

[PATCH 3/3] qemu-img: Speed up checksum

2022-09-01 Thread Nir Soffer
5s ±0.183s | 1.808s ±0.016s | x1.36 | | nbd | 6g | direct| 3.540s ±0.020s | 1.749s ±0.018s | x2.02 | [1] raw image full of zeroes [2] raw fedora 35 image with additional random data, 50% full [3] image [2] exported by qemu-nbd via unix socket Signed-off-by: Nir Soffer --- qemu-i

[PATCH 2/3] iotests: Test qemu-img checksum

2022-09-01 Thread Nir Soffer
includes a json map of every test image. Signed-off-by: Nir Soffer --- tests/qemu-iotests/tests/qemu-img-checksum| 149 ++ .../qemu-iotests/tests/qemu-img-checksum.out | 74 + 2 files changed, 223 insertions(+) create mode 100755 tests/qemu-iotests/tests/qemu-img

[PATCH 1/3] qemu-img: Add checksum command

2022-09-01 Thread Nir Soffer
ng checksum for 8T empty image: qemu-img checksum: 3.7s, sha256sum (estimate): 17,749s Signed-off-by: Nir Soffer --- docs/tools/qemu-img.rst | 22 + meson.build | 10 ++- meson_options.txt | 2 + qemu-img-cmds.hx| 8 ++ qemu-img.c | 191 +

[PATCH 0/3] Add qemu-img checksum command using blkhash

2022-09-01 Thread Nir Soffer
Since blkhash is available only via copr now, the new command is added as optional feature, built only if blkhash-devel package is installed. Nir Soffer (3): qemu-img: Add checksum command iotests: Test qemu-img checksum qemu-img: Speed up checksum docs/tools/qemu-img.rst

Re: [Libguestfs] [PATCH 1/2] spec: Recommend cap on NBD_REPLY_TYPE_BLOCK_STATUS length

2022-04-08 Thread Nir Soffer
On Fri, Apr 8, 2022 at 6:47 PM Eric Blake wrote: > > On Fri, Apr 08, 2022 at 04:48:59PM +0300, Nir Soffer wrote: ... > > > BTW attached is an nbdkit plugin that creates an NBD server that > > > responds with massive numbers of byte-granularity extents, in case > &

Re: [Libguestfs] [PATCH 1/2] spec: Recommend cap on NBD_REPLY_TYPE_BLOCK_STATUS length

2022-04-08 Thread Nir Soffer
On Fri, Apr 8, 2022 at 2:52 PM Richard W.M. Jones wrote: > > On Fri, Apr 08, 2022 at 09:25:01AM +0200, Wouter Verhelst wrote: > > Hi Eric, > > > > On Thu, Apr 07, 2022 at 04:37:19PM -0500, Eric Blake wrote: > > > The spec was silent on how many extents a server could reply with. > > > However,

Re: [PATCH v2] nbd/server: Allow MULTI_CONN for shared writable exports

2022-02-16 Thread Nir Soffer
On Wed, Feb 16, 2022 at 10:08 AM Vladimir Sementsov-Ogievskiy wrote: > > 16.02.2022 02:24, Eric Blake wrote: > > On Tue, Feb 15, 2022 at 09:23:36PM +0200, Nir Soffer wrote: > >> On Tue, Feb 15, 2022 at 7:22 PM Eric Blake wrote: > >> > >>> Accor

Re: [PATCH v2] nbd/server: Allow MULTI_CONN for shared writable exports

2022-02-16 Thread Nir Soffer
On Wed, Feb 16, 2022 at 12:13 PM Richard W.M. Jones wrote: > On Tue, Feb 15, 2022 at 05:24:14PM -0600, Eric Blake wrote: > > Oh. The QMP command (which is immediately visible through > > nbd-server-add/block-storage-add to qemu and qemu-storage-daemon) > > gains "multi-conn":"on", but you may be

Re: [PATCH v2] nbd/server: Allow MULTI_CONN for shared writable exports

2022-02-15 Thread Nir Soffer
On Tue, Feb 15, 2022 at 7:22 PM Eric Blake wrote: > According to the NBD spec, a server advertising > NBD_FLAG_CAN_MULTI_CONN promises that multiple client connections will > not see any cache inconsistencies: when properly separated by a single > flush, actions performed by one client will be

Re: [PATCH v2 2/2] iotests/block-status-cache: New test

2022-01-18 Thread Nir Soffer
644 > index 00..ae1213e6f8 > --- /dev/null > +++ b/tests/qemu-iotests/tests/block-status-cache.out > @@ -0,0 +1,5 @@ > +. > +-- > +Ran 1 tests > + > +OK > -- > 2.33.1 > The out file is not very useful, and even fragile - if the test framework will change the output format, the test will fail. Ideally we depend only on the relevant output of our tools, and using a different version of the test framework on replacing it (e.g pytest) will not require modifying the out files. Regardless I would like to see this fix merged and this issue already exists in other tests. Some tests in tests/ do have useful output that can make debugging failures easier. Reviewed-by: Nir Soffer

Re: [PATCH 1/2] block/io: Update BSC only if want_zero is true

2022-01-17 Thread Nir Soffer
data, and so we would update the block-status cache with wrong > information. > > Therefore, we should not update the cache with want_zero=false. > > Reported-by: Nir Soffer > Fixes: 0bc329fbb009f8601cec23bf2bc48ead0c5a5fa2 >("block: block-status cache for da

Re: [PATCH 2/2] iotests/block-status-cache: New test

2022-01-17 Thread Nir Soffer
self.fail("Map information differs") > + > + > +if __name__ == '__main__': > +# The block-status cache only works on the protocol layer, so to test it, > +# we can only use the raw format > +iotests.main(supported_fmts=['raw'], > + supported_protocols=['file']) > diff --git a/tests/qemu-iotests/tests/block-status-cache.out > b/tests/qemu-iotests/tests/block-status-cache.out > new file mode 100644 > index 00..ae1213e6f8 > --- /dev/null > +++ b/tests/qemu-iotests/tests/block-status-cache.out > @@ -0,0 +1,5 @@ > +. > +-- > +Ran 1 tests > + > +OK > -- > 2.33.1 > Reviewed-by: Nir Soffer

Re: Inconsistent block status reply in qemu-nbd in qemu-img-6.2.0

2022-01-17 Thread Nir Soffer
On Mon, Jan 17, 2022 at 10:46 AM Hanna Reitz wrote: > > On 16.01.22 19:09, Nir Soffer wrote: > > On Sun, Jan 16, 2022 at 1:17 PM Nir Soffer wrote: > >> Some of our tests started to fail since qemu-img 6.2.0 released. > >> > >> Here is an example failure:

Re: Inconsistent block status reply in qemu-nbd in qemu-img-6.2.0

2022-01-16 Thread Nir Soffer
On Sun, Jan 16, 2022 at 1:17 PM Nir Soffer wrote: > > Some of our tests started to fail since qemu-img 6.2.0 released. > > Here is an example failure: > > $ truncate -s1g /data/scratch/empty-1g.raw > > $ qemu-nbd -r -t -e8 -k /tmp/nbd.sock -A -f raw /data/scratch/empty-1g.

Inconsistent block status reply in qemu-nbd in qemu-img-6.2.0

2022-01-16 Thread Nir Soffer
Some of our tests started to fail since qemu-img 6.2.0 released. Here is an example failure: $ truncate -s1g /data/scratch/empty-1g.raw $ qemu-nbd -r -t -e8 -k /tmp/nbd.sock -A -f raw /data/scratch/empty-1g.raw & $ nbdinfo --map nbd+unix:///?socket=/tmp/nbd.sock 0 10737418243

[PATCH] nbd/server.c: Remove unused field

2022-01-11 Thread Nir Soffer
NBDRequestData struct has unused QSIMPLEQ_ENTRY filed. It seems that this field exists since the first git commit and was never used. Signed-off-by: Nir Soffer --- nbd/server.c | 1 - 1 file changed, 1 deletion(-) diff --git a/nbd/server.c b/nbd/server.c index 3927f7789d..ce5b2a1d02 100644

Re: [PATCH 3/4] qemu-img: add --shallow option for qemu-img compare --stat

2021-09-29 Thread Nir Soffer
On Wed, Sep 29, 2021 at 7:28 PM Vladimir Sementsov-Ogievskiy wrote: > > 29.09.2021 19:00, Nir Soffer wrote: > > On Wed, Sep 29, 2021 at 4:37 PM Vladimir Sementsov-Ogievskiy > > wrote: > >> > >> Allow compare only top images of backing chains. That's useful fo

Re: [PATCH 3/4] qemu-img: add --shallow option for qemu-img compare --stat

2021-09-29 Thread Nir Soffer
On Wed, Sep 29, 2021 at 4:37 PM Vladimir Sementsov-Ogievskiy wrote: > > Allow compare only top images of backing chains. That's useful for > comparing two increments from the same chain of incremental backups. > > Signed-off-by: Vladimir Sementsov-Ogievskiy > --- > docs/tools/qemu-img.rst | 8

Re: [PATCH] qemu-nbd: Change default cache mode to writeback

2021-09-19 Thread Nir Soffer
On Mon, Aug 16, 2021 at 6:50 PM Eric Blake wrote: > > On Fri, Aug 13, 2021 at 11:55:19PM +0300, Nir Soffer wrote: > > Both qemu and qemu-img use writeback cache mode by default, which is > > already documented in qemu(1). qemu-nbd uses writethrough cache mode by > > defaul

[PATCH] qemu-nbd: Change default cache mode to writeback

2021-08-13 Thread Nir Soffer
common use case. [1] https://bugzilla.redhat.com/1990656 Signed-off-by: Nir Soffer --- docs/tools/qemu-nbd.rst | 6 -- qemu-nbd.c | 6 -- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/tools/qemu-nbd.rst b/docs/tools/qemu-nbd.rst index ee862fa0bc..5643da26

Re: [PATCH for-6.1?] iotest: Further enhance qemu-img-bitmaps

2021-07-21 Thread Nir Soffer
On 7/21/21 11:46 PM, Eric Blake wrote: Add a regression test to make sure we detect attempts to use 'qemu-img bitmap' to modify an in-use local file. Suggested-by: Nir Soffer Signed-off-by: Eric Blake --- Sadly, this missed my bitmaps pull request today. If there's any reason to respin

Re: [PATCH v2 3/3] qemu-img: Add --skip-broken-bitmaps for 'convert --bitmaps'

2021-07-13 Thread Nir Soffer
On Tue, Jul 13, 2021 at 8:53 PM Eric Blake wrote: > > On Sat, Jul 10, 2021 at 09:37:35PM +0300, Nir Soffer wrote: > > > We don't want to delete inconsistent bitmaps by default: although a > > > corrupt bitmap is only a loss of optimization rather than a corruption &g

Re: [PATCH v2 2/3] qemu-img: Fail fast on convert --bitmaps with inconsistent bitmap

2021-07-13 Thread Nir Soffer
On Tue, Jul 13, 2021 at 8:48 PM Eric Blake wrote: > > On Sat, Jul 10, 2021 at 09:06:24PM +0300, Nir Soffer wrote: > > On 7/9/21 6:39 PM, Eric Blake wrote: > > > Waiting until the end of the convert operation (a potentially > > > time-consuming task) to finally dete

Re: [PATCH v2 3/3] qemu-img: Add --skip-broken-bitmaps for 'convert --bitmaps'

2021-07-10 Thread Nir Soffer
On 7/9/21 6:39 PM, Eric Blake wrote: The point of 'qemu-img convert --bitmaps' is to be a convenience for actions that are already possible through a string of smaller 'qemu-img bitmap' sub-commands. One situation not accounted for already is that if a source image contains an inconsistent

Re: [PATCH v2 2/3] qemu-img: Fail fast on convert --bitmaps with inconsistent bitmap

2021-07-10 Thread Nir Soffer
On 7/9/21 6:39 PM, Eric Blake wrote: Waiting until the end of the convert operation (a potentially time-consuming task) to finally detect that we can't copy a bitmap is bad, comparing to failing fast up front. Furthermore, this prevents us from leaving a file behind with a bitmap that is not

Re: [PATCH v2 1/3] iotests: Improve and rename test 291 to qemu-img-bitmap

2021-07-10 Thread Nir Soffer
from disk In this context a more useful error message would be: Try "qemu-img bitmap --remove" ... but this is not a new issue. +image: TEST_DIR/t.IMGFMT.copy +file format: IMGFMT +virtual size: 10 MiB (10485760 bytes) +cluster_size: 65536 +Format specific information: +bitmaps: +[0]: +flags: +name: b0 +granularity: 65536 +[1]: +flags: +[0]: auto +name: b4 +granularity: 65536 +corrupt: false *** done Reviewed-by: Nir Soffer

Re: [PATCH v3 3/2] qemu-img: Reword 'qemu-img map --output=json' docs

2021-07-07 Thread Nir Soffer
is optional; it is absent in >cases where ``human`` format would omit the entry or exit with an error. > -- > 2.31.1 Would be nice if this could be generated from the json schema instead of repeating the type and description of the fields, but this is a nice improvement. Reviewed-by: Nir Soffer

Re: [ovirt-users] Re: Any way to terminate stuck export task

2021-07-06 Thread Nir Soffer
On Tue, Jul 6, 2021 at 5:55 PM Gianluca Cecchi wrote: > > On Tue, Jul 6, 2021 at 2:52 PM Nir Soffer wrote: > >> >> >> Too bad. >> >> You can evaluate how ovirt 4.4. will work with this appliance using >> this dd command: >> >> dd if=

Re: [ovirt-users] Re: Any way to terminate stuck export task

2021-07-06 Thread Nir Soffer
On Tue, Jul 6, 2021 at 10:21 AM Gianluca Cecchi wrote: > > On Mon, Jul 5, 2021 at 5:06 PM Nir Soffer wrote: > >> >> >> qemu-img is busy in posix_fallocate(), wiring one byte to every 4k block. >> >> If you add -tt -T (as I suggested), we can see how much

Re: [ovirt-users] Re: Any way to terminate stuck export task

2021-07-05 Thread Nir Soffer
On Mon, Jul 5, 2021 at 3:36 PM Gianluca Cecchi wrote: > > On Mon, Jul 5, 2021 at 2:13 PM Nir Soffer wrote: >> >> >> > >> > vdsm 14342 3270 0 11:17 ?00:00:03 /usr/bin/qemu-img convert >> > -p -t none -T none -f raw >> >

Re: [ovirt-users] Re: Any way to terminate stuck export task

2021-07-05 Thread Nir Soffer
On Mon, Jul 5, 2021 at 4:06 PM Strahil Nikolov wrote: > > >Disks on the export domain are never used by a running VM so there is > no reason to > preallocate them. The system should always use sparse disks when > copying to export > domain. > > >When importing disks from export domain, the system

Re: [ovirt-users] Re: Any way to terminate stuck export task

2021-07-05 Thread Nir Soffer
On Mon, Jul 5, 2021 at 12:50 PM Gianluca Cecchi wrote: > > On Sun, Jul 4, 2021 at 1:01 PM Nir Soffer wrote: >> >> On Sun, Jul 4, 2021 at 11:30 AM Strahil Nikolov >> wrote: >> > >> > Isn't it better to strace it before killing qemu-img . >> >

Re: [ovirt-users] Re: Any way to terminate stuck export task

2021-07-04 Thread Nir Soffer
On Sun, Jul 4, 2021 at 11:30 AM Strahil Nikolov wrote: > > Isn't it better to strace it before killing qemu-img . It may be too late, but it may help to understand why this qemu-img run got stuck. > Best Regards, > Strahil Nikolov > > On Sun, Jul 4, 2021 at 0:15, Nir Soffer

Re: [ovirt-users] Any way to terminate stuck export task

2021-07-03 Thread Nir Soffer
On Sat, Jul 3, 2021 at 3:46 PM Gianluca Cecchi wrote: > > Hello, > in oVirt 4.3.10 an export job to export domain takes too long, probably due > to the NFS server slow. > How can I stop in a clean way the task? > I see the exported file remains always at 4,5Gb of size. > Command vmstat on host

Re: [PATCH v2] docs: document file-posix locking protocol

2021-07-03 Thread Nir Soffer
On Sat, Jul 3, 2021 at 4:51 PM Vladimir Sementsov-Ogievskiy wrote: > > Let's document how we use file locks in file-posix driver, to allow > external programs to "communicate" in this way with Qemu. This makes the locking implementation public, so qemu can never change it without breaking

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

2021-06-29 Thread Nir Soffer
On Tue, Jun 29, 2021 at 5:40 PM Kevin Wolf wrote: > > Am 29.06.2021 um 09:23 hat Vladimir Sementsov-Ogievskiy geschrieben: > > 28.06.2021 20:42, Eric Blake wrote: > > > On Wed, Jun 23, 2021 at 06:04:19PM +0200, Kevin Wolf wrote: > > > > > This is fine, but it means that this flag will present in

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

2021-06-23 Thread Nir Soffer
On Wed, Jun 23, 2021 at 7:04 PM Kevin Wolf wrote: > > Am 23.06.2021 um 15:58 hat Nir Soffer geschrieben: > > On Wed, Jun 23, 2021 at 11:58 AM Kevin Wolf wrote: > > > > > > Am 22.06.2021 um 18:56 hat Nir Soffer geschrieben: > > > > On Tue,

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

2021-06-23 Thread Nir Soffer
On Wed, Jun 23, 2021 at 11:58 AM Kevin Wolf wrote: > > Am 22.06.2021 um 18:56 hat Nir Soffer geschrieben: > > On Tue, Jun 22, 2021 at 6:38 PM Kevin Wolf wrote: > > > > > > Am 11.06.2021 um 21:03 hat Eric Blake geschrieben: > > > > To save the user fro

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

2021-06-22 Thread Nir Soffer
On Fri, Jun 11, 2021 at 10:03 PM Eric Blake wrote: > > To save the user from having to check 'qemu-img info --backing-chain' > or other followup command to determine which "depth":n goes beyond the > chain, add a boolean field "backing" that is set only for unallocated > portions of the disk. > >

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

2021-06-22 Thread Nir Soffer
On Tue, Jun 22, 2021 at 6:38 PM Kevin Wolf wrote: > > Am 11.06.2021 um 21:03 hat Eric Blake geschrieben: > > To save the user from having to check 'qemu-img info --backing-chain' > > or other followup command to determine which "depth":n goes beyond the > > chain, add a boolean field "backing"

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

2021-06-15 Thread Nir Soffer
On Tue, Jun 15, 2021 at 11:54 AM Vladimir Sementsov-Ogievskiy wrote: > > 11.06.2021 22:03, Eric Blake wrote: > > To save the user from having to check 'qemu-img info --backing-chain' > > or other followup command to determine which "depth":n goes beyond the > > chain, add a boolean field

Re: Reporting absent extents in qcow2 image

2021-06-15 Thread Nir Soffer
On Tue, Jun 15, 2021 at 2:23 PM Vladimir Sementsov-Ogievskiy wrote: > > 15.06.2021 01:29, Nir Soffer wrote: > > oVirt started to use now qemu:allocation-depth meta context. > > In the past, we use base:allocation and reported NBD_STATE_HOLE > > as a hole, and this broke in

Reporting absent extents in qcow2 image

2021-06-14 Thread Nir Soffer
oVirt started to use now qemu:allocation-depth meta context. In the past, we use base:allocation and reported NBD_STATE_HOLE as a hole, and this broke in qemu 6.0.0. Now we have NBD_STATE_HOLE from base:allocation, and flags == 0 from qemu:allocation-depth. We map flags == 0 to EXTENT_BACKING

Re: [PATCH 2/2] nbd: Add new qemu:joint-allocation metadata context

2021-06-14 Thread Nir Soffer
On Mon, Jun 14, 2021 at 4:56 PM Eric Blake wrote: > > On Sat, Jun 12, 2021 at 02:39:44AM +0300, Nir Soffer wrote: > > Since this change is not simple, and the chance that we also get the dirty > > bitmap included in the result seems to be very low, I decided to check the > &

Re: [PATCH 2/2] nbd: Add new qemu:joint-allocation metadata context

2021-06-11 Thread Nir Soffer
On Wed, Jun 9, 2021 at 9:01 PM Eric Blake wrote: > > When trying to reconstruct a qcow2 chain using information provided > over NBD, ovirt had been relying on an unsafe assumption that any > portion of the qcow2 file advertised as sparse would defer to the > backing image; this worked with what

Re: [PATCH] qemu-{img,nbd}: Don't report zeroed cluster as a hole

2021-06-11 Thread Nir Soffer
On Fri, Jun 11, 2021 at 9:34 PM Eric Blake wrote: > > On Fri, Jun 11, 2021 at 08:35:01PM +0300, Nir Soffer wrote: > > On Fri, Jun 11, 2021 at 4:28 PM Eric Blake wrote: > > > > > > On Fri, Jun 11, 2021 at 10:09:09AM +0200, Kevin Wolf wrote: > > > > >

Re: [PATCH v2] qemu-img: Make unallocated part of backing chain obvious in map

2021-06-11 Thread Nir Soffer
On Fri, Jun 11, 2021 at 5:59 PM Eric Blake wrote: > > On Fri, Jun 11, 2021 at 05:35:12PM +0300, Vladimir Sementsov-Ogievskiy wrote: > > > An obvious solution is to make 'qemu-img map --output=json' > > > distinguish between clusters that have a local allocation from those > > > that are found

Re: [PATCH] qemu-{img,nbd}: Don't report zeroed cluster as a hole

2021-06-11 Thread Nir Soffer
On Fri, Jun 11, 2021 at 4:28 PM Eric Blake wrote: > > On Fri, Jun 11, 2021 at 10:09:09AM +0200, Kevin Wolf wrote: > > > Yes, that might work as well. But we didn't previously document > > > depth to be optional. Removing something from output risks breaking > > > more downstream tools that

Re: [PATCH] qemu-{img,nbd}: Don't report zeroed cluster as a hole

2021-06-11 Thread Nir Soffer
> ‫ב-11 ביוני 2021, בשעה 11:14, ‏‏Vladimir Sementsov-Ogievskiy > ‏ כתב/ה:‬ > > 11.06.2021 11:09, Kevin Wolf wrote: >> Am 10.06.2021 um 22:46 hat Eric Blake geschrieben: >>>> On Thu, Jun 10, 2021 at 11:09:05PM +0300, Nir Soffer wrote: >>>>>> But

Re: [PATCH] qemu-{img,nbd}: Don't report zeroed cluster as a hole

2021-06-10 Thread Nir Soffer
On Thu, Jun 10, 2021 at 9:35 PM Eric Blake wrote: > > On Tue, Jun 08, 2021 at 07:38:10PM +0300, Nir Soffer wrote: > > The example I provided was not detailed enough, what we actually do is: > > > > qemu-nbd .. 'json:{"driver": "qcow2", "backing

Re: [libnbd PATCH] info: Avoid ambiguous 'allocated' terminology in mapping

2021-06-10 Thread Nir Soffer
cluster is known to come > from the backing chain (either locally, or from a given backing > depth), or it is completely absent from the chain (not allocated in > any of the layers). Instead of calling it "unallocated", we can call > it "absent". > > Adjust the testsuite

Re: [RFC libnbd PATCH] info: Add support for new qemu:joint-allocation

2021-06-10 Thread Nir Soffer
On Thu, Jun 10, 2021 at 4:06 PM Eric Blake wrote: > > On Thu, Jun 10, 2021 at 01:20:13AM +0300, Nir Soffer wrote: > > > + else if (strcmp (metacontext, "qemu:joint-allocation") == 0) { > > > +/* Combo of base:allocation and stripped-down qemu:allocation-dep

Re: [PATCH 2/2] nbd: Add new qemu:joint-allocation metadata context

2021-06-10 Thread Nir Soffer
On Thu, Jun 10, 2021 at 2:52 AM Nir Soffer wrote: > > On Wed, Jun 9, 2021 at 9:01 PM Eric Blake wrote: I posted a work in progress patch implementing support for qemu:joint-allocaition in oVirt: https://gerrit.ovirt.org/c/ovirt-imageio/+/115197 The most important part is the nbd client:

Re: [PATCH 2/2] nbd: Add new qemu:joint-allocation metadata context

2021-06-09 Thread Nir Soffer
be better to report: $ ./nbdinfo --map="qemu:joint-allocation" nbd://localhost 0 10737418243 hole,zero,unallocated This is the output for the empty qcow2 image. And this also affects --allocation-depth, for raw empty image we get: $ ./nbdinfo --map="qemu:all

Re: [RFC libnbd PATCH] info: Add support for new qemu:joint-allocation

2021-06-09 Thread Nir Soffer
On Thu, Jun 10, 2021 at 12:32 AM Eric Blake wrote: > > Qemu is adding qemu:joint-allocation as a single context combining the > two bits of base:allocation and a compression of qemu:allocation-depth > into two bits [1]. Decoding the bits makes it easier for humans to > see the result of that

Re: [PATCH] qemu-{img,nbd}: Don't report zeroed cluster as a hole

2021-06-08 Thread Nir Soffer
On Tue, Jun 8, 2021 at 9:46 PM Eric Blake wrote: > > On Tue, Jun 08, 2021 at 07:38:10PM +0300, Nir Soffer wrote: > > On Tue, Jun 8, 2021 at 12:22 AM Eric Blake wrote: > > > > > > On Mon, Jun 07, 2021 at 11:22:04PM +0300, Nir Soffer wrote: > > > > When

Re: [PATCH] qemu-{img,nbd}: Don't report zeroed cluster as a hole

2021-06-08 Thread Nir Soffer
On Tue, Jun 8, 2021 at 12:22 AM Eric Blake wrote: > > On Mon, Jun 07, 2021 at 11:22:04PM +0300, Nir Soffer wrote: > > When zeroing a cluster in an image with backing file, qemu-img and > > qemu-nbd reported the area as a hole. This does not affect the guest > > since

[PATCH] qemu-{img,nbd}: Don't report zeroed cluster as a hole

2021-06-07 Thread Nir Soffer
quot;: false, "offset": 196608}] $ build/qemu-nbd -r -t -f qcow2 top.qcow2 & $ qemu-img map --output json nbd://localhost [{ "start": 0, "length": 131072, "depth": 0, "zero": false, "data": true, "offset": 0},

Re: [PATCH] block/file-posix: Fix problem with fallocate(PUNCH_HOLE) on GPFS

2021-04-16 Thread Nir Soffer
On Fri, Apr 16, 2021 at 8:23 AM Thomas Huth wrote: > > A customer reported that running > > qemu-img convert -t none -O qcow2 -f qcow2 input.qcow2 output.qcow2 > > fails for them with the following error message when the images are > stored on a GPFS file system: > > qemu-img: error while

[PATCH] qemu-iotest: Test NBD hole reporting for qcow2

2021-04-13 Thread Nir Soffer
]. Add the missing tests for single qcow2 image and qcow2 image with a backing file. [1] https://listman.redhat.com/archives/libguestfs/2021-April/msg00050.html Signed-off-by: Nir Soffer --- tests/qemu-iotests/314 | 96 tests/qemu-iotests/314.out | 34

Re: [ovirt-users] Q: Sparsify/fstrim don't reduce actual disk image size

2021-04-08 Thread Nir Soffer
ment an image (best online): [---] ovirt can shrink the logical volume after that. [] Adding qemu-block since this is an interesting use case that may be useful to more qemu users. Nir > > On 7 Apr 2021, at 18:43,

Re: [PATCH V4] file-posix: allow -EBUSY error during ioctl(fd, BLKZEROOUT, range) on block

2021-03-26 Thread Nir Soffer
On Fri, Mar 26, 2021 at 3:21 AM ChangLimin wrote: > >On Thu, Mar 25, 2021 at 8:07 AM ChangLimin > wrote: > >>On Wed, Mar 24, 2021 at 4:52 PM Max Reitz wrote: > >>On 22.03.21 10:25, ChangLimin wrote: > >>> For Linux 5.10/5.11, qemu write zeros to a multipath device using > >>> ioctl(fd,

Re: [PATCH V4] file-posix: allow -EBUSY error during ioctl(fd, BLKZEROOUT, range) on block

2021-03-25 Thread Nir Soffer
On Thu, Mar 25, 2021 at 8:07 AM ChangLimin wrote: > >On Wed, Mar 24, 2021 at 4:52 PM Max Reitz wrote: > >On 22.03.21 10:25, ChangLimin wrote: > >> For Linux 5.10/5.11, qemu write zeros to a multipath device using > >> ioctl(fd, BLKZEROOUT, range) with cache none or directsync return -EBUSY > >>

Re: [PATCH V4] file-posix: allow -EBUSY error during ioctl(fd, BLKZEROOUT, range) on block

2021-03-24 Thread Nir Soffer
On Wed, Mar 24, 2021 at 4:52 PM Max Reitz wrote: > On 22.03.21 10:25, ChangLimin wrote: > > For Linux 5.10/5.11, qemu write zeros to a multipath device using > > ioctl(fd, BLKZEROOUT, range) with cache none or directsync return -EBUSY > > permanently. > > So as far as I can track back the

Re: [PATCH] nbd: server: Report holes for raw images

2021-03-03 Thread Nir Soffer
On Thu, Feb 25, 2021 at 8:51 PM Vladimir Sementsov-Ogievskiy < vsement...@virtuozzo.com> wrote: > 19.02.2021 19:58, Eric Blake wrote: > > On 2/19/21 10:42 AM, Eric Blake wrote: > > > >>> To me, data=false looks compatible with NBD_STATE_HOLE. From user point > >>> of view, getting same results

Re: [PATCH] file-posix: allow -EBUSY errors during write zeros on block

2021-03-02 Thread Nir Soffer
On Tue, Mar 2, 2021 at 4:08 AM ChangLimin wrote: > > After Linux 5.10, write zeros to a multipath device using > ioctl(fd, BLKZEROOUT, range) with cache none or directsync will return EBUSY. > > Similar to handle_aiocb_write_zeroes_unmap, handle_aiocb_write_zeroes_block > allow -EBUSY errors

[PATCH] nbd: server: Report holes for raw images

2021-02-19 Thread Nir Soffer
ing same results from qemu-nbd and qemu-img is more important than being more correct about allocation status. Changing nbd server to report holes using BDRV_BLOCK_DATA makes qemu-nbd results compatible with qemu-img map: $ qemu-img map --output json nbd://localhost [{ "start&quo

Re: [PATCH v3 2/2] qemu-nbd: Permit --shared=0 for unlimited clients

2021-02-10 Thread Nir Soffer
rror_report("Invalid shared device number '%s'", optarg); > exit(EXIT_FAILURE); > } > @@ -966,7 +965,7 @@ int main(int argc, char **argv) > if (socket_activation == 0) { > int backlog; > > -if (persistent) { > +if (persistent || shared == 0) { > backlog = SOMAXCONN; > } else { > backlog = MIN(shared, SOMAXCONN); > -- > 2.30.0 > Reviewed-by: Nir Soffer

Re: [PATCH v3 1/2] qemu-nbd: Use SOMAXCONN for socket listen() backlog

2021-02-10 Thread Nir Soffer
On Tue, Feb 9, 2021 at 5:28 PM Eric Blake wrote: > > Our default of a backlog of 1 connection is rather puny; it gets in > the way when we are explicitly allowing multiple clients (such as > qemu-nbd -e N [--shared], or nbd-server-start with its default > "max-connections":0 for unlimited), but

Re: [PATCH v2] qemu-nbd: Use SOMAXCONN for socket listen() backlog

2021-02-05 Thread Nir Soffer
On Fri, Feb 5, 2021 at 8:57 PM Eric Blake wrote: > > Our default of a backlog of 1 connection is rather puny, particularly > for scenarios where we expect multiple listeners to connect (such as > qemu-nbd -e X). This is especially important for Unix sockets, as a > definite benefit to clients:

Re: Potential regression in 'qemu-img convert' to LVM

2021-01-07 Thread Nir Soffer
On Tue, Sep 15, 2020 at 2:51 PM Stefan Reiter wrote: > > On 9/15/20 11:08 AM, Nir Soffer wrote: > > On Mon, Sep 14, 2020 at 3:25 PM Stefan Reiter wrote: > >> > >> Hi list, > >> > >> following command fails since 5.1 (tested on kernel 5.4.60): &g

Re: Libvirt driver iothread property for virtio-scsi disks

2020-11-04 Thread Nir Soffer
On Wed, Nov 4, 2020 at 6:54 PM Daniel P. Berrangé wrote: > > On Wed, Nov 04, 2020 at 05:48:40PM +0200, Nir Soffer wrote: > > The docs[1] say: > > > > - The optional iothread attribute assigns the disk to an IOThread as > > defined by > > the range for the do

Re: Libvirt driver iothread property for virtio-scsi disks

2020-11-04 Thread Nir Soffer
On Wed, Nov 4, 2020 at 6:42 PM Sergio Lopez wrote: > > On Wed, Nov 04, 2020 at 05:48:40PM +0200, Nir Soffer wrote: > > The docs[1] say: > > > > - The optional iothread attribute assigns the disk to an IOThread as > > defined by > > the range for the domain

Libvirt driver iothread property for virtio-scsi disks

2020-11-04 Thread Nir Soffer
The docs[1] say: - The optional iothread attribute assigns the disk to an IOThread as defined by the range for the domain iothreads value. Multiple disks may be assigned to the same IOThread and are numbered from 1 to the domain iothreads value. Available for a disk device target configured

Re: Allocation change in qemu-img convert (since 5.1?)

2020-10-26 Thread Nir Soffer
On Sun, Oct 25, 2020 at 5:55 PM Nir Soffer wrote: > > We have tests for vdsm qemuimg wrapper module converting > images and verifying the operation using "qemu-img compare". > > I found that tests using "qemu-img compare -s" started to fail > recently on R

Allocation change in qemu-img convert (since 5.1?)

2020-10-25 Thread Nir Soffer
We have tests for vdsm qemuimg wrapper module converting images and verifying the operation using "qemu-img compare". I found that tests using "qemu-img compare -s" started to fail recently on RHEL 8.3.0 nightly (qemu-img-5.1.0-10.module+el8.3.0+8254+568ca30d.x86_6) and Fedora 32 using

Re: [PULL 20/21] python/qemu/qmp.py: re-raise OSError when encountered

2020-10-20 Thread Nir Soffer
On Tue, Oct 20, 2020 at 8:52 PM John Snow wrote: > > Nested if conditions don't change when the exception block fires; we > need to explicitly re-raise the error if we didn't intend to capture and > suppress it. > > Signed-off-by: John Snow > Reviewed-by: Philippe Mathieu-Daudé > Message-id:

Re: [ovirt-devel] Disk sizes not updated on unmap/discard

2020-10-01 Thread Nir Soffer
On Wed, Sep 30, 2020 at 1:49 PM Tomáš Golembiovský wrote: > > Hi, > > currently, when we run virt-sparsify on VM or user runs VM with discard > enabled and when the disk is on block storage in qcow, the results are > not reflected in oVirt. The blocks get discarded, storage can reuse them > and

Re: [PATCH] docs: Better mention of qemu-img amend limitations

2020-09-23 Thread Nir Soffer
On Wed, Sep 23, 2020 at 11:38 PM Eric Blake wrote: > > Missed during merge resolution of commit bc5ee6da71. > > Signed-off-by: Eric Blake > --- > docs/tools/qemu-img.rst | 4 > 1 file changed, 4 insertions(+) > > diff --git a/docs/tools/qemu-img.rst b/docs/tools/qemu-img.rst > index

Re: Potential regression in 'qemu-img convert' to LVM

2020-09-15 Thread Nir Soffer
On Mon, Sep 14, 2020 at 3:25 PM Stefan Reiter wrote: > > Hi list, > > following command fails since 5.1 (tested on kernel 5.4.60): > > # qemu-img convert -p -f raw -O raw /dev/zvol/pool/disk-1 /dev/vg/disk-1 > qemu-img: error while writing at byte 2157968896: Device or resource busy > > (source

Re: [PATCH 2/2] block: file-posix: Replace posix_fallocate with fallocate

2020-09-15 Thread Nir Soffer
On Mon, Sep 14, 2020 at 8:32 PM Daniel P. Berrangé wrote: > > On Mon, Aug 31, 2020 at 05:01:27PM +0300, Nir Soffer wrote: > > If fallocate() is not supported, posix_fallocate() falls back to > > inefficient allocation, writing one byte for every 4k bytes[1]. This is >

  1   2   3   4   >