Re: [PATCH v2] scripts/qcow2-to-stdout.py: Add script to write qcow2 images to stdout

2024-07-27 Thread Nir Soffer
On Mon, Jul 1, 2024 at 6:13 PM Alberto Garcia wrote: > > This tool converts a disk image to qcow2, writing the result directly > to stdout. This can be used for example to send the generated file > over the network. > > This is equivalent to using qemu-img to convert a file to qcow2 and > then wri

Re: Strange behavior of qemu-img map: zero/data status depends on fallocated image page cache content

2024-07-21 Thread Nir Soffer
On Sun, Jun 30, 2024 at 5:31 PM Nir Soffer wrote: > > I found a strange behavior in qemu-img map - zero/data status depends on page > cache content. It looks like a kernel issue since qemu-img map is using > SEEK_HOLE/DATA (block/file-posix.c line 3111). > > Tested with lat

Re: backup-discard-source iotest fails on xfs, pass on ext4

2024-07-21 Thread Nir Soffer
On Sat, Jun 29, 2024 at 12:17 AM Nir Soffer wrote: > > While testing > https://lists.nongnu.org/archive/html/qemu-block/2024-06/msg00644.html > I noticed that > this tests is failing also with master > (28b8a57ad63670aa0ce90334523dc552b13b4336). > > The test succeeds with

Re: [PATCH 3/4] iotests: Change imports for Python 3.13

2024-07-02 Thread Nir Soffer
> On 2 Jul 2024, at 17:44, John Snow wrote: > > > > On Tue, Jul 2, 2024 at 7:52 AM Nir Soffer <mailto:nsof...@redhat.com>> wrote: >> On Thu, Jun 27, 2024 at 2:23 AM John Snow > <mailto:js...@redhat.com>> wrote: >> > >> > Python 3.13

Re: [PATCH 3/4] iotests: Change imports for Python 3.13

2024-07-02 Thread Nir Soffer
On Thu, Jun 27, 2024 at 2:23 AM John Snow wrote: > > Python 3.13 isn't out yet, but it's in beta and Fedora is ramping up to > make it the default system interpreter for Fedora 41. > > They moved our cheese for where ContextManager lives; add a conditional > to locate it while we support both pre-

Strange behavior of qemu-img map: zero/data status depends on fallocated image page cache content

2024-06-30 Thread Nir Soffer
I found a strange behavior in qemu-img map - zero/data status depends on page cache content. It looks like a kernel issue since qemu-img map is using SEEK_HOLE/DATA (block/file-posix.c line 3111). Tested with latest qemu on kernel 6.9.6-100.fc39.x86_64. I see similar behavior in xfs and ex4 files

backup-discard-source iotest fails on xfs, pass on ext4

2024-06-28 Thread Nir Soffer
While testing https://lists.nongnu.org/archive/html/qemu-block/2024-06/msg00644.html I noticed that this tests is failing also with master (28b8a57ad63670aa0ce90334523dc552b13b4336). The test succeeds with ext4, and fail with xfs. Nir --- Success on ext4: $ TEST_DIR=/data/tetests/qemu-iotests

[PATCH v3 1/2] qemu-iotest/245: Add missing discard=unmap

2024-06-28 Thread Nir Soffer
The test works since we punch holes by default even when opening the image without discard=on or discard=unmap. Fix the test to enable discard. --- tests/qemu-iotests/245 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/qemu-iotests/245 b/tests/qemu-iotests/245 index a934c

[PATCH v3 2/2] Consider discard option when writing zeros

2024-06-28 Thread Nir Soffer
u.org/archive/html/qemu-discuss/2024-06/msg3.html Signed-off-by: Nir Soffer --- block/io.c| 9 +- tests/qemu-iotests/tests/write-zeroes-unmap | 127 ++ .../qemu-iotests/tests/write-zeroes-unmap.out | 81 +++ 3 files changed

[PATCH v3 0/2] Consider discard option when writing zeros

2024-06-28 Thread Nir Soffer
was here: https://lists.nongnu.org/archive/html/qemu-block/2024-06/msg00231.html Nir Soffer (2): qemu-iotest/245: Add missing discard=unmap Consider discard option when writing zeros block/io.c| 9 +- tests/qemu-iotests/245| 2

Re: [PATCH v2] Consider discard option when writing zeros

2024-06-28 Thread Nir Soffer
On Thu, Jun 27, 2024 at 2:42 PM Kevin Wolf wrote: > Am 26.06.2024 um 18:27 hat Nir Soffer geschrieben: > > On Wed, Jun 26, 2024 at 12:17 PM Daniel P. Berrangé > > > wrote: > > > > > On Mon, Jun 24, 2024 at 06:08:26PM +0200, Kevin Wolf wrote: > > > &g

Re: [PATCH v2] Consider discard option when writing zeros

2024-06-26 Thread Nir Soffer
On Wed, Jun 26, 2024 at 12:17 PM Daniel P. Berrangé wrote: > On Mon, Jun 24, 2024 at 06:08:26PM +0200, Kevin Wolf wrote: > > Am 24.06.2024 um 17:23 hat Stefan Hajnoczi geschrieben: > > > On Wed, Jun 19, 2024 at 08:43:25PM +0300, Nir Soffer wrote: > > > > Te

Re: [PATCH v2] Consider discard option when writing zeros

2024-06-26 Thread Nir Soffer
On Wed, Jun 26, 2024 at 11:42 AM Kevin Wolf wrote: > Am 24.06.2024 um 23:12 hat Nir Soffer geschrieben: > > On Mon, Jun 24, 2024 at 7:08 PM Kevin Wolf wrote: > > > > > Am 24.06.2024 um 17:23 hat Stefan Hajnoczi geschrieben: > > > > On Wed, Jun 19, 2024 a

Re: [PATCH v2] Consider discard option when writing zeros

2024-06-24 Thread Nir Soffer
On Mon, Jun 24, 2024 at 7:08 PM Kevin Wolf wrote: > Am 24.06.2024 um 17:23 hat Stefan Hajnoczi geschrieben: > > On Wed, Jun 19, 2024 at 08:43:25PM +0300, Nir Soffer wrote: > > > Tested using: > > > > Hi Nir, > > This looks like a good candidate for the qemu-

Re: [PATCH v2] Consider discard option when writing zeros

2024-06-19 Thread Nir Soffer
On Wed, Jun 19, 2024 at 8:40 PM Nir Soffer wrote: > - Need to run all block tests > Stale note, make check pass

Re: [PATCH v2] Consider discard option when writing zeros

2024-06-19 Thread Nir Soffer
est.raw discard=unmap - write zeroes unmap 0 /tmp/test.raw Differences: $ diff -u before.out after.out --- before.out 2024-06-19 20:24:09.234083713 +0300 +++ after.out 2024-06-19 20:24:20.526165573 +0300 @@ -3,13 +3,13 @@ 1.0M /tmp/test.raw defaults - write zeroes unmap -0 /tmp/test.raw +

[PATCH v2] Consider discard option when writing zeros

2024-06-19 Thread Nir Soffer
his change we punched holes by default. I'm not sure this is a good change in behavior. - Need to run all block tests - Not sure that we have tests covering unmapping, we may need new tests - We may need new tests to cover this change Signed-off-by: Nir Soffer --- Changes since v1: - Replac

Re: [PATCH] block/file-posix: Consider discard flag when opening

2024-06-19 Thread Nir Soffer
> On 19 Jun 2024, at 11:16, Kevin Wolf wrote: > > Am 18.06.2024 um 23:24 hat Nir Soffer geschrieben: >> Set has_discard only when BDRV_O_UNMAP is not set. With this users that >> want to keep their images fully allocated can disable hole punching >> when writin

[PATCH] block/file-posix: Consider discard flag when opening

2024-06-18 Thread Nir Soffer
un by default. We needs tests for keeping images non-sparse. Signed-off-by: Nir Soffer --- block/file-posix.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/block/file-posix.c b/block/file-posix.c index be25e35ff6..acac2abadc 100644 --- a/block/file-posix.c +++ b/block

Re: [PATCH 1/1] block: improve alignment detection and fix 271 test

2023-10-14 Thread Nir Soffer
e test to reflect real world usage? _reset_img 2083k I guess it works with: _reset_img 2084k Commits > commit a6b257a08e3d72219f03e461a52152672fec0612 > Author: Nir Soffer > Date: Tue Aug 13 21:21:03 2019 +0300 > file-posix: Handle undetectable alignme

Re: [Libguestfs] [PATCH v2 1/6] spec: Recommend cap on NBD_REPLY_TYPE_BLOCK_STATUS length

2023-03-06 Thread Nir Soffer
On Sun, Mar 5, 2023 at 10:42 AM Wouter Verhelst wrote: > > On Fri, Mar 03, 2023 at 04:17:40PM -0600, Eric Blake wrote: > > On Fri, Dec 16, 2022 at 10:32:01PM +0300, Vladimir Sementsov-Ogievskiy > > wrote: > > > s-o-b line missed. > > > > I'm not sure if the NBD project has a strict policy on incl

Re: [Libguestfs] [PATCH] docs: Prefer 'cookie' over 'handle'

2023-03-04 Thread Nir Soffer
On Sat, Mar 4, 2023 at 12:15 AM Eric Blake wrote: > > In libnbd, we quickly learned that distinguishing between 'handle' > (verb for acting on an object) and 'handle' (noun describing which > object to act on) could get confusing; we solved it by renaming the > latter to 'cookie'. Copy that appro

Re: [PATCH v2 2/5] Support format or cache specific out file

2022-12-13 Thread Nir Soffer
On Tue, Dec 13, 2022 at 8:09 PM Hanna Reitz wrote: > > On 13.12.22 16:56, Nir Soffer wrote: > > On Mon, Dec 12, 2022 at 12:38 PM Hanna Reitz wrote: > >> On 28.11.22 15:15, Nir Soffer wrote: > >>> Extend the test finder to find tests with format (*.o

Re: [PATCH v2 2/5] Support format or cache specific out file

2022-12-13 Thread Nir Soffer
On Mon, Dec 12, 2022 at 12:38 PM Hanna Reitz wrote: > > On 28.11.22 15:15, Nir Soffer wrote: > > Extend the test finder to find tests with format (*.out.qcow2) or cache > > specific (*.out.nocache) out file. This worked before only for the > > numbered tests. > >

[PATCH v2 5/5] qemu-img: Speed up checksum

2022-11-28 Thread Nir Soffer
5s ±0.183s | 1.808s ±0.016s | x1.36 | | nbd | 6g | direct| 3.540s ±0.020s | 1.749s ±0.018s | x2.02 | [1] raw image full of zeroes [2] raw fedora 35 image with additional random data, 50% full [3] image [2] exported by qemu-nbd via unix socket Signed-off-by: Nir Soffer --- qemu-i

[PATCH v2 1/5] qemu-img.c: Move IO_BUF_SIZE to the top of the file

2022-11-28 Thread Nir Soffer
This macro is used by various commands (compare, convert, rebase) but it is defined somewhere in the middle of the file. I'm going to use it in the new checksum command so lets clean up a bit before that. --- qemu-img.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/qemu-img

[PATCH v2 4/5] iotests: Test qemu-img checksum

2022-11-28 Thread Nir Soffer
help debugging in case of failures, the output includes a json map of the test image. Signed-off-by: Nir Soffer --- tests/qemu-iotests/tests/qemu-img-checksum| 63 +++ .../tests/qemu-img-checksum.out.qcow2 | 11 .../tests/qemu-img-checksum.out.raw | 10 +

[PATCH v2 2/5] Support format or cache specific out file

2022-11-28 Thread Nir Soffer
Extend the test finder to find tests with format (*.out.qcow2) or cache specific (*.out.nocache) out file. This worked before only for the numbered tests. --- tests/qemu-iotests/findtests.py | 10 -- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/tests/qemu-iotests/findtests

[PATCH v2 0/5] Add qemu-img checksum command using blkhash

2022-11-28 Thread Nir Soffer
tml - https://lists.nongnu.org/archive/html/qemu-block/2022-11/msg00171.html - https://lists.nongnu.org/archive/html/qemu-block/2022-11/msg00173.html Nir Soffer (5): qemu-img.c: Move IO_BUF_SIZE to the top of the file Support format or cache specific out file qemu-img: Add checksum command iote

[PATCH v2 3/5] qemu-img: Add checksum command

2022-11-28 Thread Nir Soffer
blkhash/ [3] Computing checksum for 8T empty image: qemu-img checksum: 3.7s, sha256sum (estimate): 17,749s Signed-off-by: Nir Soffer --- docs/tools/qemu-img.rst | 24 ++ meson.build | 10 ++- meson_options.txt | 2 + qemu-img-cmds.hx| 8 ++ qemu-img.c

Re: [PATCH 2/3] iotests: Test qemu-img checksum

2022-11-28 Thread Nir Soffer
On Mon, Nov 7, 2022 at 1:41 PM Hanna Reitz wrote: > On 30.10.22 18:38, Nir Soffer wrote: > > On Wed, Oct 26, 2022 at 4:31 PM Hanna Reitz wrote: > > > > On 01.09.22 16:32, Nir Soffer wrote: > > > Add simple tests creating an image with all kinds

Re: [PATCH 1/3] qemu-img: Add checksum command

2022-11-28 Thread Nir Soffer
On Mon, Nov 7, 2022 at 12:20 PM Hanna Reitz wrote: > On 30.10.22 18:37, Nir Soffer wrote: > > On Wed, Oct 26, 2022 at 4:00 PM Hanna Reitz wrote: > > > > On 01.09.22 16:32, Nir Soffer wrote: > [...] > > > --- > > > docs/tools/qemu-i

Re: [PATCH 3/3] qemu-img: Speed up checksum

2022-10-30 Thread Nir Soffer
On Sun, Oct 30, 2022 at 7:38 PM Nir Soffer wrote: > On Wed, Oct 26, 2022 at 4:54 PM Hanna Reitz wrote: > >> On 01.09.22 16:32, Nir Soffer wrote: >> > [...] > > +/* The current chunk. */ >> > +int64_t offset; >> > +int64_t length; >>

Re: [PATCH 3/3] qemu-img: Speed up checksum

2022-10-30 Thread Nir Soffer
On Wed, Oct 26, 2022 at 4:54 PM Hanna Reitz wrote: > On 01.09.22 16:32, Nir Soffer wrote: > > Add coroutine based loop inspired by `qemu-img convert` design. > > > > Changes compared to `qemu-img convert`: > > > > - State for the entire image is kept in Img

Re: [PATCH 2/3] iotests: Test qemu-img checksum

2022-10-30 Thread Nir Soffer
On Wed, Oct 26, 2022 at 4:31 PM Hanna Reitz wrote: > On 01.09.22 16:32, Nir Soffer wrote: > > Add simple tests creating an image with all kinds of extents, different > > formats, different backing chain, different protocol, and different > > image options. Since all image

Re: [PATCH 1/3] qemu-img: Add checksum command

2022-10-30 Thread Nir Soffer
On Wed, Oct 26, 2022 at 4:00 PM Hanna Reitz wrote: > On 01.09.22 16:32, Nir Soffer wrote: > > The checksum command compute a checksum for disk image content using the > > blkhash library[1]. The blkhash library is not packaged yet, but it is > > available via copr[2]

Re: [PATCH 0/3] Add qemu-img checksum command using blkhash

2022-10-18 Thread Nir Soffer
On Sun, Sep 18, 2022 at 12:35 PM Nir Soffer wrote: > ping > > Kevin, Hanna, I hope you have time to take a look. > > https://lists.nongnu.org/archive/html/qemu-block/2022-09/msg00021.html Ping again, hopefully someone has time to look at this :-) > > > > On Thu,

Re: [PATCH 0/3] Add qemu-img checksum command using blkhash

2022-09-18 Thread Nir Soffer
ping Kevin, Hanna, I hope you have time to take a look. https://lists.nongnu.org/archive/html/qemu-block/2022-09/msg00021.html On Thu, Sep 1, 2022 at 5:32 PM Nir Soffer wrote: > > Since blkhash is available only via copr now, the new command is added as > optional feature, buil

[PATCH 3/3] qemu-img: Speed up checksum

2022-09-01 Thread Nir Soffer
5s ±0.183s | 1.808s ±0.016s | x1.36 | | nbd | 6g | direct| 3.540s ±0.020s | 1.749s ±0.018s | x2.02 | [1] raw image full of zeroes [2] raw fedora 35 image with additional random data, 50% full [3] image [2] exported by qemu-nbd via unix socket Signed-off-by: Nir Soffer --- qemu-i

[PATCH 2/3] iotests: Test qemu-img checksum

2022-09-01 Thread Nir Soffer
includes a json map of every test image. Signed-off-by: Nir Soffer --- tests/qemu-iotests/tests/qemu-img-checksum| 149 ++ .../qemu-iotests/tests/qemu-img-checksum.out | 74 + 2 files changed, 223 insertions(+) create mode 100755 tests/qemu-iotests/tests/qemu-img

[PATCH 1/3] qemu-img: Add checksum command

2022-09-01 Thread Nir Soffer
blkhash/ [3] Computing checksum for 8T empty image: qemu-img checksum: 3.7s, sha256sum (estimate): 17,749s Signed-off-by: Nir Soffer --- docs/tools/qemu-img.rst | 22 + meson.build | 10 ++- meson_options.txt | 2 + qemu-img-cmds.hx| 8 ++ qemu-img.c

[PATCH 0/3] Add qemu-img checksum command using blkhash

2022-09-01 Thread Nir Soffer
Since blkhash is available only via copr now, the new command is added as optional feature, built only if blkhash-devel package is installed. Nir Soffer (3): qemu-img: Add checksum command iotests: Test qemu-img checksum qemu-img: Speed up checksum docs/tools/qemu-img.rst

Re: [Libguestfs] [PATCH 1/2] spec: Recommend cap on NBD_REPLY_TYPE_BLOCK_STATUS length

2022-04-08 Thread Nir Soffer
On Fri, Apr 8, 2022 at 6:47 PM Eric Blake wrote: > > On Fri, Apr 08, 2022 at 04:48:59PM +0300, Nir Soffer wrote: ... > > > BTW attached is an nbdkit plugin that creates an NBD server that > > > responds with massive numbers of byte-granularity extents, in case > &

Re: [Libguestfs] [PATCH 1/2] spec: Recommend cap on NBD_REPLY_TYPE_BLOCK_STATUS length

2022-04-08 Thread Nir Soffer
On Fri, Apr 8, 2022 at 2:52 PM Richard W.M. Jones wrote: > > On Fri, Apr 08, 2022 at 09:25:01AM +0200, Wouter Verhelst wrote: > > Hi Eric, > > > > On Thu, Apr 07, 2022 at 04:37:19PM -0500, Eric Blake wrote: > > > The spec was silent on how many extents a server could reply with. > > > However, bot

Re: [PATCH v2] nbd/server: Allow MULTI_CONN for shared writable exports

2022-02-16 Thread Nir Soffer
On Wed, Feb 16, 2022 at 10:08 AM Vladimir Sementsov-Ogievskiy wrote: > > 16.02.2022 02:24, Eric Blake wrote: > > On Tue, Feb 15, 2022 at 09:23:36PM +0200, Nir Soffer wrote: > >> On Tue, Feb 15, 2022 at 7:22 PM Eric Blake wrote: > >> > >>> Accor

Re: [PATCH v2] nbd/server: Allow MULTI_CONN for shared writable exports

2022-02-16 Thread Nir Soffer
On Wed, Feb 16, 2022 at 12:13 PM Richard W.M. Jones wrote: > On Tue, Feb 15, 2022 at 05:24:14PM -0600, Eric Blake wrote: > > Oh. The QMP command (which is immediately visible through > > nbd-server-add/block-storage-add to qemu and qemu-storage-daemon) > > gains "multi-conn":"on", but you may be

Re: [PATCH v2] nbd/server: Allow MULTI_CONN for shared writable exports

2022-02-15 Thread Nir Soffer
On Tue, Feb 15, 2022 at 7:22 PM Eric Blake wrote: > According to the NBD spec, a server advertising > NBD_FLAG_CAN_MULTI_CONN promises that multiple client connections will > not see any cache inconsistencies: when properly separated by a single > flush, actions performed by one client will be vi

Re: [PATCH v2 2/2] iotests/block-status-cache: New test

2022-01-18 Thread Nir Soffer
we can only use the raw format > +iotests.main(supported_fmts=['raw'], > + supported_protocols=['file']) > diff --git a/tests/qemu-iotests/tests/block-status-cache.out > b/tests/qemu-iotests/tests/block-status-cache.out > new file mode 100644 > index 00..ae1213e6f8 > --- /dev/null > +++ b/tests/qemu-iotests/tests/block-status-cache.out > @@ -0,0 +1,5 @@ > +. > +-- > +Ran 1 tests > + > +OK > -- > 2.33.1 > The out file is not very useful, and even fragile - if the test framework will change the output format, the test will fail. Ideally we depend only on the relevant output of our tools, and using a different version of the test framework on replacing it (e.g pytest) will not require modifying the out files. Regardless I would like to see this fix merged and this issue already exists in other tests. Some tests in tests/ do have useful output that can make debugging failures easier. Reviewed-by: Nir Soffer

Re: [PATCH 1/2] block/io: Update BSC only if want_zero is true

2022-01-17 Thread Nir Soffer
data, and so we would update the block-status cache with wrong > information. > > Therefore, we should not update the cache with want_zero=false. > > Reported-by: Nir Soffer > Fixes: 0bc329fbb009f8601cec23bf2bc48ead0c5a5fa2 >("block: block-status cache for da

Re: [PATCH 2/2] iotests/block-status-cache: New test

2022-01-17 Thread Nir Soffer
nd after querying > ' + > + 'qemu:allocation-depth') > +print('Before:') > + print(map_pre) > +print('After:') > +print(map_post) > + > +self.fail("Map information differs") > + > + > +if __name__ == '__main__': > +# The block-status cache only works on the protocol layer, so to test it, > +# we can only use the raw format > +iotests.main(supported_fmts=['raw'], > + supported_protocols=['file']) > diff --git a/tests/qemu-iotests/tests/block-status-cache.out > b/tests/qemu-iotests/tests/block-status-cache.out > new file mode 100644 > index 00..ae1213e6f8 > --- /dev/null > +++ b/tests/qemu-iotests/tests/block-status-cache.out > @@ -0,0 +1,5 @@ > +. > +-- > +Ran 1 tests > + > +OK > -- > 2.33.1 > Reviewed-by: Nir Soffer

Re: Inconsistent block status reply in qemu-nbd in qemu-img-6.2.0

2022-01-17 Thread Nir Soffer
On Mon, Jan 17, 2022 at 10:46 AM Hanna Reitz wrote: > > On 16.01.22 19:09, Nir Soffer wrote: > > On Sun, Jan 16, 2022 at 1:17 PM Nir Soffer wrote: > >> Some of our tests started to fail since qemu-img 6.2.0 released. > >> > >> Here is an example failure:

Re: Inconsistent block status reply in qemu-nbd in qemu-img-6.2.0

2022-01-16 Thread Nir Soffer
On Sun, Jan 16, 2022 at 1:17 PM Nir Soffer wrote: > > Some of our tests started to fail since qemu-img 6.2.0 released. > > Here is an example failure: > > $ truncate -s1g /data/scratch/empty-1g.raw > > $ qemu-nbd -r -t -e8 -k /tmp/nbd.sock -A -f raw /data/scratch/empty-1g.

Inconsistent block status reply in qemu-nbd in qemu-img-6.2.0

2022-01-16 Thread Nir Soffer
Some of our tests started to fail since qemu-img 6.2.0 released. Here is an example failure: $ truncate -s1g /data/scratch/empty-1g.raw $ qemu-nbd -r -t -e8 -k /tmp/nbd.sock -A -f raw /data/scratch/empty-1g.raw & $ nbdinfo --map nbd+unix:///?socket=/tmp/nbd.sock 0 10737418243 hol

[PATCH] nbd/server.c: Remove unused field

2022-01-11 Thread Nir Soffer
NBDRequestData struct has unused QSIMPLEQ_ENTRY filed. It seems that this field exists since the first git commit and was never used. Signed-off-by: Nir Soffer --- nbd/server.c | 1 - 1 file changed, 1 deletion(-) diff --git a/nbd/server.c b/nbd/server.c index 3927f7789d..ce5b2a1d02 100644

Re: [PATCH 3/4] qemu-img: add --shallow option for qemu-img compare --stat

2021-09-29 Thread Nir Soffer
On Wed, Sep 29, 2021 at 7:28 PM Vladimir Sementsov-Ogievskiy wrote: > > 29.09.2021 19:00, Nir Soffer wrote: > > On Wed, Sep 29, 2021 at 4:37 PM Vladimir Sementsov-Ogievskiy > > wrote: > >> > >> Allow compare only top images of backing chains. That's usef

Re: [PATCH 3/4] qemu-img: add --shallow option for qemu-img compare --stat

2021-09-29 Thread Nir Soffer
On Wed, Sep 29, 2021 at 4:37 PM Vladimir Sementsov-Ogievskiy wrote: > > Allow compare only top images of backing chains. That's useful for > comparing two increments from the same chain of incremental backups. > > Signed-off-by: Vladimir Sementsov-Ogievskiy > --- > docs/tools/qemu-img.rst | 8 +

Re: [PATCH] qemu-nbd: Change default cache mode to writeback

2021-09-19 Thread Nir Soffer
On Mon, Aug 16, 2021 at 6:50 PM Eric Blake wrote: > > On Fri, Aug 13, 2021 at 11:55:19PM +0300, Nir Soffer wrote: > > Both qemu and qemu-img use writeback cache mode by default, which is > > already documented in qemu(1). qemu-nbd uses writethrough cache mode by > > defaul

[PATCH] qemu-nbd: Change default cache mode to writeback

2021-08-13 Thread Nir Soffer
the common use case. [1] https://bugzilla.redhat.com/1990656 Signed-off-by: Nir Soffer --- docs/tools/qemu-nbd.rst | 6 -- qemu-nbd.c | 6 -- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/tools/qemu-nbd.rst b/docs/tools/qemu-nbd.rst index ee862fa0bc..564

Re: [PATCH for-6.1?] iotest: Further enhance qemu-img-bitmaps

2021-07-21 Thread Nir Soffer
On 7/21/21 11:46 PM, Eric Blake wrote: Add a regression test to make sure we detect attempts to use 'qemu-img bitmap' to modify an in-use local file. Suggested-by: Nir Soffer Signed-off-by: Eric Blake --- Sadly, this missed my bitmaps pull request today. If there's any reason

Re: [PATCH v2 3/3] qemu-img: Add --skip-broken-bitmaps for 'convert --bitmaps'

2021-07-13 Thread Nir Soffer
On Tue, Jul 13, 2021 at 8:53 PM Eric Blake wrote: > > On Sat, Jul 10, 2021 at 09:37:35PM +0300, Nir Soffer wrote: > > > We don't want to delete inconsistent bitmaps by default: although a > > > corrupt bitmap is only a loss of optimization rather than a corruption &

Re: [PATCH v2 2/3] qemu-img: Fail fast on convert --bitmaps with inconsistent bitmap

2021-07-13 Thread Nir Soffer
On Tue, Jul 13, 2021 at 8:48 PM Eric Blake wrote: > > On Sat, Jul 10, 2021 at 09:06:24PM +0300, Nir Soffer wrote: > > On 7/9/21 6:39 PM, Eric Blake wrote: > > > Waiting until the end of the convert operation (a potentially > > > time-consuming task) to finally dete

Re: [PATCH v2 3/3] qemu-img: Add --skip-broken-bitmaps for 'convert --bitmaps'

2021-07-10 Thread Nir Soffer
On 7/9/21 6:39 PM, Eric Blake wrote: The point of 'qemu-img convert --bitmaps' is to be a convenience for actions that are already possible through a string of smaller 'qemu-img bitmap' sub-commands. One situation not accounted for already is that if a source image contains an inconsistent bitma

Re: [PATCH v2 2/3] qemu-img: Fail fast on convert --bitmaps with inconsistent bitmap

2021-07-10 Thread Nir Soffer
On 7/9/21 6:39 PM, Eric Blake wrote: Waiting until the end of the convert operation (a potentially time-consuming task) to finally detect that we can't copy a bitmap is bad, comparing to failing fast up front. Furthermore, this prevents us from leaving a file behind with a bitmap that is not mar

Re: [PATCH v2 1/3] iotests: Improve and rename test 291 to qemu-img-bitmap

2021-07-10 Thread Nir Soffer
itmap-remove to delete this bitmap from disk In this context a more useful error message would be: Try "qemu-img bitmap --remove" ... but this is not a new issue. +image: TEST_DIR/t.IMGFMT.copy +file format: IMGFMT +virtual size: 10 MiB (10485760 bytes) +cluster_size: 65536 +Format specific information: +bitmaps: +[0]: +flags: +name: b0 +granularity: 65536 +[1]: +flags: +[0]: auto +name: b4 +granularity: 65536 +corrupt: false *** done Reviewed-by: Nir Soffer

Re: [PATCH v3 3/2] qemu-img: Reword 'qemu-img map --output=json' docs

2021-07-07 Thread Nir Soffer
is optional; it is absent in >cases where ``human`` format would omit the entry or exit with an error. > -- > 2.31.1 Would be nice if this could be generated from the json schema instead of repeating the type and description of the fields, but this is a nice improvement. Reviewed-by: Nir Soffer

Re: [ovirt-users] Re: Any way to terminate stuck export task

2021-07-06 Thread Nir Soffer
On Tue, Jul 6, 2021 at 5:55 PM Gianluca Cecchi wrote: > > On Tue, Jul 6, 2021 at 2:52 PM Nir Soffer wrote: > >> >> >> Too bad. >> >> You can evaluate how ovirt 4.4. will work with this appliance using >> this dd command: >> >> dd if=

Re: [ovirt-users] Re: Any way to terminate stuck export task

2021-07-06 Thread Nir Soffer
On Tue, Jul 6, 2021 at 10:21 AM Gianluca Cecchi wrote: > > On Mon, Jul 5, 2021 at 5:06 PM Nir Soffer wrote: > >> >> >> qemu-img is busy in posix_fallocate(), wiring one byte to every 4k block. >> >> If you add -tt -T (as I suggested), we can see how much

Re: [ovirt-users] Re: Any way to terminate stuck export task

2021-07-05 Thread Nir Soffer
On Mon, Jul 5, 2021 at 3:36 PM Gianluca Cecchi wrote: > > On Mon, Jul 5, 2021 at 2:13 PM Nir Soffer wrote: >> >> >> > >> > vdsm 14342 3270 0 11:17 ?00:00:03 /usr/bin/qemu-img convert >> > -p -t none -T none -f raw >> >

Re: [ovirt-users] Re: Any way to terminate stuck export task

2021-07-05 Thread Nir Soffer
On Mon, Jul 5, 2021 at 4:06 PM Strahil Nikolov wrote: > > >Disks on the export domain are never used by a running VM so there is > no reason to > preallocate them. The system should always use sparse disks when > copying to export > domain. > > >When importing disks from export domain, the system

Re: [ovirt-users] Re: Any way to terminate stuck export task

2021-07-05 Thread Nir Soffer
On Mon, Jul 5, 2021 at 12:50 PM Gianluca Cecchi wrote: > > On Sun, Jul 4, 2021 at 1:01 PM Nir Soffer wrote: >> >> On Sun, Jul 4, 2021 at 11:30 AM Strahil Nikolov >> wrote: >> > >> > Isn't it better to strace it before killing qemu-img . >>

Re: [ovirt-users] Re: Any way to terminate stuck export task

2021-07-04 Thread Nir Soffer
On Sun, Jul 4, 2021 at 11:30 AM Strahil Nikolov wrote: > > Isn't it better to strace it before killing qemu-img . It may be too late, but it may help to understand why this qemu-img run got stuck. > Best Regards, > Strahil Nikolov > > On Sun, Jul 4, 2021 at 0:15, Nir So

Re: [ovirt-users] Any way to terminate stuck export task

2021-07-03 Thread Nir Soffer
On Sat, Jul 3, 2021 at 3:46 PM Gianluca Cecchi wrote: > > Hello, > in oVirt 4.3.10 an export job to export domain takes too long, probably due > to the NFS server slow. > How can I stop in a clean way the task? > I see the exported file remains always at 4,5Gb of size. > Command vmstat on host wi

Re: [PATCH v2] docs: document file-posix locking protocol

2021-07-03 Thread Nir Soffer
On Sat, Jul 3, 2021 at 4:51 PM Vladimir Sementsov-Ogievskiy wrote: > > Let's document how we use file locks in file-posix driver, to allow > external programs to "communicate" in this way with Qemu. This makes the locking implementation public, so qemu can never change it without breaking externa

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

2021-06-29 Thread Nir Soffer
On Tue, Jun 29, 2021 at 5:40 PM Kevin Wolf wrote: > > Am 29.06.2021 um 09:23 hat Vladimir Sementsov-Ogievskiy geschrieben: > > 28.06.2021 20:42, Eric Blake wrote: > > > On Wed, Jun 23, 2021 at 06:04:19PM +0200, Kevin Wolf wrote: > > > > > This is fine, but it means that this flag will present in a

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

2021-06-23 Thread Nir Soffer
On Wed, Jun 23, 2021 at 7:04 PM Kevin Wolf wrote: > > Am 23.06.2021 um 15:58 hat Nir Soffer geschrieben: > > On Wed, Jun 23, 2021 at 11:58 AM Kevin Wolf wrote: > > > > > > Am 22.06.2021 um 18:56 hat Nir Soffer geschrieben: > > > > On Tue,

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

2021-06-23 Thread Nir Soffer
On Wed, Jun 23, 2021 at 11:58 AM Kevin Wolf wrote: > > Am 22.06.2021 um 18:56 hat Nir Soffer geschrieben: > > On Tue, Jun 22, 2021 at 6:38 PM Kevin Wolf wrote: > > > > > > Am 11.06.2021 um 21:03 hat Eric Blake geschrieben: > > > > To save the user fro

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

2021-06-22 Thread Nir Soffer
On Fri, Jun 11, 2021 at 10:03 PM Eric Blake wrote: > > To save the user from having to check 'qemu-img info --backing-chain' > or other followup command to determine which "depth":n goes beyond the > chain, add a boolean field "backing" that is set only for unallocated > portions of the disk. > >

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

2021-06-22 Thread Nir Soffer
On Tue, Jun 22, 2021 at 6:38 PM Kevin Wolf wrote: > > Am 11.06.2021 um 21:03 hat Eric Blake geschrieben: > > To save the user from having to check 'qemu-img info --backing-chain' > > or other followup command to determine which "depth":n goes beyond the > > chain, add a boolean field "backing" tha

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

2021-06-15 Thread Nir Soffer
On Tue, Jun 15, 2021 at 11:54 AM Vladimir Sementsov-Ogievskiy wrote: > > 11.06.2021 22:03, Eric Blake wrote: > > To save the user from having to check 'qemu-img info --backing-chain' > > or other followup command to determine which "depth":n goes beyond the > > chain, add a boolean field "backing"

Re: Reporting absent extents in qcow2 image

2021-06-15 Thread Nir Soffer
On Tue, Jun 15, 2021 at 2:23 PM Vladimir Sementsov-Ogievskiy wrote: > > 15.06.2021 01:29, Nir Soffer wrote: > > oVirt started to use now qemu:allocation-depth meta context. > > In the past, we use base:allocation and reported NBD_STATE_HOLE > > as a hole, and this broke in

Reporting absent extents in qcow2 image

2021-06-14 Thread Nir Soffer
oVirt started to use now qemu:allocation-depth meta context. In the past, we use base:allocation and reported NBD_STATE_HOLE as a hole, and this broke in qemu 6.0.0. Now we have NBD_STATE_HOLE from base:allocation, and flags == 0 from qemu:allocation-depth. We map flags == 0 to EXTENT_BACKING (1

Re: [PATCH 2/2] nbd: Add new qemu:joint-allocation metadata context

2021-06-14 Thread Nir Soffer
On Mon, Jun 14, 2021 at 4:56 PM Eric Blake wrote: > > On Sat, Jun 12, 2021 at 02:39:44AM +0300, Nir Soffer wrote: > > Since this change is not simple, and the chance that we also get the dirty > > bitmap included in the result seems to be very low, I decided to check the > &

Re: [PATCH 2/2] nbd: Add new qemu:joint-allocation metadata context

2021-06-11 Thread Nir Soffer
On Wed, Jun 9, 2021 at 9:01 PM Eric Blake wrote: > > When trying to reconstruct a qcow2 chain using information provided > over NBD, ovirt had been relying on an unsafe assumption that any > portion of the qcow2 file advertised as sparse would defer to the > backing image; this worked with what qe

Re: [PATCH] qemu-{img,nbd}: Don't report zeroed cluster as a hole

2021-06-11 Thread Nir Soffer
On Fri, Jun 11, 2021 at 9:34 PM Eric Blake wrote: > > On Fri, Jun 11, 2021 at 08:35:01PM +0300, Nir Soffer wrote: > > On Fri, Jun 11, 2021 at 4:28 PM Eric Blake wrote: > > > > > > On Fri, Jun 11, 2021 at 10:09:09AM +0200, Kevin Wolf wrote: > > > > >

Re: [PATCH v2] qemu-img: Make unallocated part of backing chain obvious in map

2021-06-11 Thread Nir Soffer
On Fri, Jun 11, 2021 at 5:59 PM Eric Blake wrote: > > On Fri, Jun 11, 2021 at 05:35:12PM +0300, Vladimir Sementsov-Ogievskiy wrote: > > > An obvious solution is to make 'qemu-img map --output=json' > > > distinguish between clusters that have a local allocation from those > > > that are found nowh

Re: [PATCH] qemu-{img,nbd}: Don't report zeroed cluster as a hole

2021-06-11 Thread Nir Soffer
On Fri, Jun 11, 2021 at 4:28 PM Eric Blake wrote: > > On Fri, Jun 11, 2021 at 10:09:09AM +0200, Kevin Wolf wrote: > > > Yes, that might work as well. But we didn't previously document > > > depth to be optional. Removing something from output risks breaking > > > more downstream tools that expec

Re: [PATCH] qemu-{img,nbd}: Don't report zeroed cluster as a hole

2021-06-11 Thread Nir Soffer
> ‫ב-11 ביוני 2021, בשעה 11:14, ‏‏Vladimir Sementsov-Ogievskiy > ‏ כתב/ה:‬ > > 11.06.2021 11:09, Kevin Wolf wrote: >> Am 10.06.2021 um 22:46 hat Eric Blake geschrieben: >>>> On Thu, Jun 10, 2021 at 11:09:05PM +0300, Nir Soffer wrote: >>>>>> But

Re: [PATCH] qemu-{img,nbd}: Don't report zeroed cluster as a hole

2021-06-10 Thread Nir Soffer
On Thu, Jun 10, 2021 at 9:35 PM Eric Blake wrote: > > On Tue, Jun 08, 2021 at 07:38:10PM +0300, Nir Soffer wrote: > > The example I provided was not detailed enough, what we actually do is: > > > > qemu-nbd .. 'json:{"driver": "qcow2", "ba

Re: [libnbd PATCH] info: Avoid ambiguous 'allocated' terminology in mapping

2021-06-10 Thread Nir Soffer
in qemu:allocation-depth, either a cluster is known to come > from the backing chain (either locally, or from a given backing > depth), or it is completely absent from the chain (not allocated in > any of the layers). Instead of calling it "unallocated", we can call > it "

Re: [RFC libnbd PATCH] info: Add support for new qemu:joint-allocation

2021-06-10 Thread Nir Soffer
On Thu, Jun 10, 2021 at 4:06 PM Eric Blake wrote: > > On Thu, Jun 10, 2021 at 01:20:13AM +0300, Nir Soffer wrote: > > > + else if (strcmp (metacontext, "qemu:joint-allocation") == 0) { > > > +/* Combo of base:allocation and stripped-down qemu:allocation-dep

Re: [PATCH 2/2] nbd: Add new qemu:joint-allocation metadata context

2021-06-10 Thread Nir Soffer
On Thu, Jun 10, 2021 at 2:52 AM Nir Soffer wrote: > > On Wed, Jun 9, 2021 at 9:01 PM Eric Blake wrote: I posted a work in progress patch implementing support for qemu:joint-allocaition in oVirt: https://gerrit.ovirt.org/c/ovirt-imageio/+/115197 The most important part is the nbd client:

Re: [PATCH 2/2] nbd: Add new qemu:joint-allocation metadata context

2021-06-09 Thread Nir Soffer
not a practical issue for oVirt, but it would be better to report: $ ./nbdinfo --map="qemu:joint-allocation" nbd://localhost 0 10737418243 hole,zero,unallocated This is the output for the empty qcow2 image. And this also affects --allocation-depth, for raw empty image

Re: [RFC libnbd PATCH] info: Add support for new qemu:joint-allocation

2021-06-09 Thread Nir Soffer
On Thu, Jun 10, 2021 at 12:32 AM Eric Blake wrote: > > Qemu is adding qemu:joint-allocation as a single context combining the > two bits of base:allocation and a compression of qemu:allocation-depth > into two bits [1]. Decoding the bits makes it easier for humans to > see the result of that cont

Re: [PATCH] qemu-{img,nbd}: Don't report zeroed cluster as a hole

2021-06-08 Thread Nir Soffer
On Tue, Jun 8, 2021 at 9:46 PM Eric Blake wrote: > > On Tue, Jun 08, 2021 at 07:38:10PM +0300, Nir Soffer wrote: > > On Tue, Jun 8, 2021 at 12:22 AM Eric Blake wrote: > > > > > > On Mon, Jun 07, 2021 at 11:22:04PM +0300, Nir Soffer wrote: > > > > When

Re: [PATCH] qemu-{img,nbd}: Don't report zeroed cluster as a hole

2021-06-08 Thread Nir Soffer
On Tue, Jun 8, 2021 at 12:22 AM Eric Blake wrote: > > On Mon, Jun 07, 2021 at 11:22:04PM +0300, Nir Soffer wrote: > > When zeroing a cluster in an image with backing file, qemu-img and > > qemu-nbd reported the area as a hole. This does not affect the guest > > since the a

[PATCH] qemu-{img,nbd}: Don't report zeroed cluster as a hole

2021-06-07 Thread Nir Soffer
quot;: false, "offset": 196608}] $ build/qemu-nbd -r -t -f qcow2 top.qcow2 & $ qemu-img map --output json nbd://localhost [{ "start": 0, "length": 131072, "depth": 0, "zero": false, "data": true, "offset": 0},

Re: [PATCH] block/file-posix: Fix problem with fallocate(PUNCH_HOLE) on GPFS

2021-04-16 Thread Nir Soffer
On Fri, Apr 16, 2021 at 8:23 AM Thomas Huth wrote: > > A customer reported that running > > qemu-img convert -t none -O qcow2 -f qcow2 input.qcow2 output.qcow2 > > fails for them with the following error message when the images are > stored on a GPFS file system: > > qemu-img: error while writin

[PATCH] qemu-iotest: Test NBD hole reporting for qcow2

2021-04-13 Thread Nir Soffer
]. Add the missing tests for single qcow2 image and qcow2 image with a backing file. [1] https://listman.redhat.com/archives/libguestfs/2021-April/msg00050.html Signed-off-by: Nir Soffer --- tests/qemu-iotests/314 | 96 tests/qemu-iotests/314.out | 34

Re: [ovirt-users] Q: Sparsify/fstrim don't reduce actual disk image size

2021-04-08 Thread Nir Soffer
efragment an image (best online): [---] ovirt can shrink the logical volume after that. [] Adding qemu-block since this is an interesting use case that may be useful to more qemu users. Nir > > On 7 Apr 2021, at 18

Re: [PATCH V4] file-posix: allow -EBUSY error during ioctl(fd, BLKZEROOUT, range) on block

2021-03-26 Thread Nir Soffer
On Fri, Mar 26, 2021 at 3:21 AM ChangLimin wrote: > >On Thu, Mar 25, 2021 at 8:07 AM ChangLimin > wrote: > >>On Wed, Mar 24, 2021 at 4:52 PM Max Reitz wrote: > >>On 22.03.21 10:25, ChangLimin wrote: > >>> For Linux 5.10/5.11, qemu write zeros to a multipath device using > >>> ioctl(fd, BLKZEROO

  1   2   3   4   >