Re: [PATCH v2 2/2] iotests/block-status-cache: New test

[Eric Blake]

On Fri, Jan 28, 2022 at 02:50:51PM -0600, Eric Blake wrote:
> On Tue, Jan 18, 2022 at 06:00:00PM +0100, Hanna Reitz wrote:
> > Add a new test to verify that want_zero=false block-status calls do not
> > pollute the block-status cache for want_zero=true calls.
> > 
> > We check want_zero=true calls and their results using `qemu-img map`
> > (over NBD), and want_zero=false calls also using `qemu-img map` over
> > NBD, but using the qemu:allocation-depth context.
> > 
> > (This test case cannot be integrated into nbd-qemu-allocation, because
> > that is a qcow2 test, and this is a raw test.)
> > 
> > Signed-off-by: Hanna Reitz 
> > ---
> >  tests/qemu-iotests/tests/block-status-cache   | 139 ++
> >  .../qemu-iotests/tests/block-status-cache.out |   5 +
> >  2 files changed, 144 insertions(+)
> >  create mode 100755 tests/qemu-iotests/tests/block-status-cache
> >  create mode 100644 tests/qemu-iotests/tests/block-status-cache.out
> 
> Reviewed-by: Eric Blake 

I also tested that applying this patch but not 1/2 reproduces the
failure for me, so I'm adding:

Tested-by: Eric Blake 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [PATCH v2 2/2] iotests/block-status-cache: New test

[Eric Blake]

On Tue, Jan 18, 2022 at 06:00:00PM +0100, Hanna Reitz wrote:
> Add a new test to verify that want_zero=false block-status calls do not
> pollute the block-status cache for want_zero=true calls.
> 
> We check want_zero=true calls and their results using `qemu-img map`
> (over NBD), and want_zero=false calls also using `qemu-img map` over
> NBD, but using the qemu:allocation-depth context.
> 
> (This test case cannot be integrated into nbd-qemu-allocation, because
> that is a qcow2 test, and this is a raw test.)
> 
> Signed-off-by: Hanna Reitz 
> ---
>  tests/qemu-iotests/tests/block-status-cache   | 139 ++
>  .../qemu-iotests/tests/block-status-cache.out |   5 +
>  2 files changed, 144 insertions(+)
>  create mode 100755 tests/qemu-iotests/tests/block-status-cache
>  create mode 100644 tests/qemu-iotests/tests/block-status-cache.out

Reviewed-by: Eric Blake 

As this was detected as a regression using NBD, I'm happy to add this
to my NBD queue, and will send a pull request soon.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [PATCH v2 2/2] iotests/block-status-cache: New test

[Nir Soffer]

On Tue, Jan 18, 2022 at 7:12 PM Hanna Reitz  wrote:
>
> Add a new test to verify that want_zero=false block-status calls do not
> pollute the block-status cache for want_zero=true calls.
>
> We check want_zero=true calls and their results using `qemu-img map`
> (over NBD), and want_zero=false calls also using `qemu-img map` over
> NBD, but using the qemu:allocation-depth context.
>
> (This test case cannot be integrated into nbd-qemu-allocation, because
> that is a qcow2 test, and this is a raw test.)
>
> Signed-off-by: Hanna Reitz 
> ---
>  tests/qemu-iotests/tests/block-status-cache   | 139 ++
>  .../qemu-iotests/tests/block-status-cache.out |   5 +
>  2 files changed, 144 insertions(+)
>  create mode 100755 tests/qemu-iotests/tests/block-status-cache
>  create mode 100644 tests/qemu-iotests/tests/block-status-cache.out
>
> diff --git a/tests/qemu-iotests/tests/block-status-cache 
> b/tests/qemu-iotests/tests/block-status-cache
> new file mode 100755
> index 00..6fa10bb8f8
> --- /dev/null
> +++ b/tests/qemu-iotests/tests/block-status-cache
> @@ -0,0 +1,139 @@
> +#!/usr/bin/env python3
> +# group: rw quick
> +#
> +# Test cases for the block-status cache.
> +#
> +# Copyright (C) 2022 Red Hat, Inc.
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program.  If not, see .
> +#
> +
> +import os
> +import signal
> +import iotests
> +from iotests import qemu_img_create, qemu_img_pipe, qemu_nbd
> +
> +
> +image_size = 1 * 1024 * 1024
> +test_img = os.path.join(iotests.test_dir, 'test.img')
> +
> +nbd_pidfile = os.path.join(iotests.test_dir, 'nbd.pid')
> +nbd_sock = os.path.join(iotests.sock_dir, 'nbd.sock')
> +
> +
> +class TestBscWithNbd(iotests.QMPTestCase):
> +def setUp(self) -> None:
> +"""Just create an empty image with a read-only NBD server on it"""
> +assert qemu_img_create('-f', iotests.imgfmt, test_img,
> +   str(image_size)) == 0
> +
> +# Pass --allocation-depth to enable the qemu:allocation-depth 
> context,
> +# which we are going to query to provoke a block-status inquiry with
> +# want_zero=false.
> +assert qemu_nbd(f'--socket={nbd_sock}',
> +f'--format={iotests.imgfmt}',
> +'--persistent',
> +'--allocation-depth',
> +'--read-only',
> +f'--pid-file={nbd_pidfile}',
> +test_img) \
> +== 0
> +
> +def tearDown(self) -> None:
> +with open(nbd_pidfile, encoding='utf-8') as f:
> +pid = int(f.read())
> +os.kill(pid, signal.SIGTERM)
> +os.remove(nbd_pidfile)
> +os.remove(test_img)
> +
> +def test_with_zero_bug(self) -> None:
> +"""
> +Verify that the block-status cache is not corrupted by a
> +want_zero=false call.
> +We can provoke a want_zero=false call with `qemu-img map` over NBD 
> with
> +x-dirty-bitmap=qemu:allocation-depth, so we first run a normal `map`
> +(which results in want_zero=true), then using said
> +qemu:allocation-depth context, and finally another normal `map` to
> +verify that the cache has not been corrupted.
> +"""
> +
> +nbd_img_opts = f'driver=nbd,server.type=unix,server.path={nbd_sock}'
> +nbd_img_opts_alloc_depth = nbd_img_opts + \
> +',x-dirty-bitmap=qemu:allocation-depth'
> +
> +# Normal map, results in want_zero=true.
> +# This will probably detect an allocated data sector first (qemu 
> likes
> +# to allocate the first sector to facilitate alignment probing), and
> +# then the rest to be zero.  The BSC will thus contain (if anything)
> +# one range covering the first sector.
> +map_pre = qemu_img_pipe('map', '--output=json', '--image-opts',
> +nbd_img_opts)
> +
> +# qemu:allocation-depth maps for want_zero=false.
> +# want_zero=false should (with the file driver, which the server is
> +# using) report everything as data.  While this is sufficient for
> +# want_zero=false, this is nothing that should end up in the
> +# block-status cache.
> +# Due to a bug, this information did end up in the cache, though, and
> +# this would lead to wrong informati

[PATCH v2 2/2] iotests/block-status-cache: New test

[Hanna Reitz]

Add a new test to verify that want_zero=false block-status calls do not
pollute the block-status cache for want_zero=true calls.

We check want_zero=true calls and their results using `qemu-img map`
(over NBD), and want_zero=false calls also using `qemu-img map` over
NBD, but using the qemu:allocation-depth context.

(This test case cannot be integrated into nbd-qemu-allocation, because
that is a qcow2 test, and this is a raw test.)

Signed-off-by: Hanna Reitz 
---
 tests/qemu-iotests/tests/block-status-cache   | 139 ++
 .../qemu-iotests/tests/block-status-cache.out |   5 +
 2 files changed, 144 insertions(+)
 create mode 100755 tests/qemu-iotests/tests/block-status-cache
 create mode 100644 tests/qemu-iotests/tests/block-status-cache.out

diff --git a/tests/qemu-iotests/tests/block-status-cache 
b/tests/qemu-iotests/tests/block-status-cache
new file mode 100755
index 00..6fa10bb8f8
--- /dev/null
+++ b/tests/qemu-iotests/tests/block-status-cache
@@ -0,0 +1,139 @@
+#!/usr/bin/env python3
+# group: rw quick
+#
+# Test cases for the block-status cache.
+#
+# Copyright (C) 2022 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see .
+#
+
+import os
+import signal
+import iotests
+from iotests import qemu_img_create, qemu_img_pipe, qemu_nbd
+
+
+image_size = 1 * 1024 * 1024
+test_img = os.path.join(iotests.test_dir, 'test.img')
+
+nbd_pidfile = os.path.join(iotests.test_dir, 'nbd.pid')
+nbd_sock = os.path.join(iotests.sock_dir, 'nbd.sock')
+
+
+class TestBscWithNbd(iotests.QMPTestCase):
+def setUp(self) -> None:
+"""Just create an empty image with a read-only NBD server on it"""
+assert qemu_img_create('-f', iotests.imgfmt, test_img,
+   str(image_size)) == 0
+
+# Pass --allocation-depth to enable the qemu:allocation-depth context,
+# which we are going to query to provoke a block-status inquiry with
+# want_zero=false.
+assert qemu_nbd(f'--socket={nbd_sock}',
+f'--format={iotests.imgfmt}',
+'--persistent',
+'--allocation-depth',
+'--read-only',
+f'--pid-file={nbd_pidfile}',
+test_img) \
+== 0
+
+def tearDown(self) -> None:
+with open(nbd_pidfile, encoding='utf-8') as f:
+pid = int(f.read())
+os.kill(pid, signal.SIGTERM)
+os.remove(nbd_pidfile)
+os.remove(test_img)
+
+def test_with_zero_bug(self) -> None:
+"""
+Verify that the block-status cache is not corrupted by a
+want_zero=false call.
+We can provoke a want_zero=false call with `qemu-img map` over NBD with
+x-dirty-bitmap=qemu:allocation-depth, so we first run a normal `map`
+(which results in want_zero=true), then using said
+qemu:allocation-depth context, and finally another normal `map` to
+verify that the cache has not been corrupted.
+"""
+
+nbd_img_opts = f'driver=nbd,server.type=unix,server.path={nbd_sock}'
+nbd_img_opts_alloc_depth = nbd_img_opts + \
+',x-dirty-bitmap=qemu:allocation-depth'
+
+# Normal map, results in want_zero=true.
+# This will probably detect an allocated data sector first (qemu likes
+# to allocate the first sector to facilitate alignment probing), and
+# then the rest to be zero.  The BSC will thus contain (if anything)
+# one range covering the first sector.
+map_pre = qemu_img_pipe('map', '--output=json', '--image-opts',
+nbd_img_opts)
+
+# qemu:allocation-depth maps for want_zero=false.
+# want_zero=false should (with the file driver, which the server is
+# using) report everything as data.  While this is sufficient for
+# want_zero=false, this is nothing that should end up in the
+# block-status cache.
+# Due to a bug, this information did end up in the cache, though, and
+# this would lead to wrong information being returned on subsequent
+# want_zero=true calls.
+#
+# We need to run this map twice: On the first call, we probably still
+# have the first sector in the cache, and so this will be served from
+# the cache; and only the subsequent range w