Re: [PATCH V6 0/6] hw/block/nvme: support multi-path for ctrl/ns

[Minwoo Im]

On 21-01-27 05:39:29, Keith Busch wrote:
> This came out looking cleaner than I had initially expected. Thanks for
> seeing this feature through!
> 
> Reviewed-by: Keith Busch 

Thanks Keith for the review!

Re: [PATCH v2 8/9] tests/docker: Add dockerfile for Alpine Linux

[John Snow]

On 1/19/21 8:41 AM, Thomas Huth wrote:

On 18/01/2021 11.33, Daniel P. Berrangé wrote:

On Mon, Jan 18, 2021 at 02:38:07PM +0800, Jiaxun Yang wrote:

Alpine Linux[1] is a security-oriented, lightweight Linux distribution
based on musl libc and busybox.

It it popular among Docker guests and embedded applications.

Adding it to test against different libc.

[1]: https://alpinelinux.org/

Signed-off-by: Jiaxun Yang 
---
  tests/docker/dockerfiles/alpine.docker | 57 ++
  1 file changed, 57 insertions(+)
  create mode 100644 tests/docker/dockerfiles/alpine.docker

diff --git a/tests/docker/dockerfiles/alpine.docker 
b/tests/docker/dockerfiles/alpine.docker

new file mode 100644
index 00..5be5198d00
--- /dev/null
+++ b/tests/docker/dockerfiles/alpine.docker
@@ -0,0 +1,57 @@
+
+FROM alpine:edge
+
+RUN apk update
+RUN apk upgrade
+
+# Please keep this list sorted alphabetically
+ENV PACKAGES \
+    alsa-lib-dev \
+    bash \
+    bison \


This shouldn't be required.


bison and flex were required to avoid some warnings in the past while 
compiling the dtc submodule ... but I thought we got rid of the problem 
at one point in time, so this can be removed now, indeed.



+    build-base \


This seems to be a meta packae that pulls in other
misc toolchain packages. Please list the pieces we
need explicitly instead.


Looking at the "Depends" list on 
https://pkgs.alpinelinux.org/package/v3.3/main/x86/build-base there are 
only 6 dependencies and we need most of those for QEMU anyway, so I 
think it is ok to keep build-base here.



+    coreutils \
+    curl-dev \
+    flex \


This shouldn't be needed.


+    git \
+    glib-dev \
+    glib-static \
+    gnutls-dev \
+    gtk+3.0-dev \
+    libaio-dev \
+    libcap-dev \


Should not be required, as we use cap-ng.


Right.


+    libcap-ng-dev \
+    libjpeg-turbo-dev \
+    libnfs-dev \
+    libpng-dev \
+    libseccomp-dev \
+    libssh-dev \
+    libusb-dev \
+    libxml2-dev \
+    linux-headers \


Is this really needed ? We don't install kernel-headers on other
distros AFAICT.


I tried a build without this package, and it works fine indeed.


+    lzo-dev \
+    mesa-dev \
+    mesa-egl \
+    mesa-gbm \
+    meson \
+    ncurses-dev \
+    ninja \
+    paxmark \


What is this needed for ?


Seems like it also can be dropped.


+    perl \
+    pulseaudio-dev \
+    python3 \
+    py3-sphinx \
+    shadow \


Is this really needed ?


See:
https://www.spinics.net/lists/kvm/msg231556.html

I can remove the superfluous packages when picking up the patch, no need 
to respin just because of this.


  Thomas




You can refer to my post earlier this January for a "minimal" Alpine 
Linux build, if you wish.


My goal was to find the smallest set of packages possible without 
passing any explicit configure flags.


I wonder if it's worth having layered "core build" and "test build" 
images so that we can smoke test the minimalistic build from time to 
time -- I seem to recall Dan posting information about a dependency 
management tool for Dockerfiles, but I admit I didn't look too closely 
at what problem that solves, exactly.


--js

Re: [PATCH v5 03/10] iotests: Move try_remove to iotests.py

[John Snow]

On 1/18/21 5:57 AM, Max Reitz wrote:

Signed-off-by: Max Reitz 
Reviewed-by: Eric Blake 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Willian Rampazzo 
---
  tests/qemu-iotests/124|  8 +---
  tests/qemu-iotests/iotests.py | 11 +++
  2 files changed, 8 insertions(+), 11 deletions(-)

diff --git a/tests/qemu-iotests/124 b/tests/qemu-iotests/124
index 3705cbb6b3..e40eeb50b9 100755
--- a/tests/qemu-iotests/124
+++ b/tests/qemu-iotests/124
@@ -22,6 +22,7 @@
  
  import os

  import iotests
+from iotests import try_remove
  
  
  def io_write_patterns(img, patterns):

@@ -29,13 +30,6 @@ def io_write_patterns(img, patterns):
  iotests.qemu_io('-c', 'write -P%s %s %s' % pattern, img)
  
  
-def try_remove(img):

-try:
-os.remove(img)
-except OSError:
-pass
-
-
  def transaction_action(action, **kwargs):
  return {
  'type': action,
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 52facb8e04..a69b4cdc4e 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -523,12 +523,15 @@ class FilePath:
  return False
  
  
+def try_remove(img):

+try:
+os.remove(img)
+except OSError:
+pass
+
  def file_path_remover():
  for path in reversed(file_path_remover.paths):
-try:
-os.remove(path)
-except OSError:
-pass
+try_remove(path)
  
  
  def file_path(*names, base_dir=test_dir):




For what it's worth, at the time I wrote this I barely knew Python. I'd 
use FileNotFoundError instead now, which is a subclass of OSError.


Not your baby, not your problem.

Reviewed-by: John Snow 

Re: [PATCH v5 02/10] iotests/297: Rewrite in Python and extend reach

[John Snow]

On 1/18/21 5:57 AM, Max Reitz wrote:

Instead of checking iotests.py only, check all Python files in the
qemu-iotests/ directory.  Of course, most of them do not pass, so there
is an extensive skip list for now.  (The only files that do pass are
209, 254, 283, and iotests.py.)



Chiming in to say that I tried to tackle this before; I wrote some 
preliminary cleanups and sent to the list as an "WIP RFC" or something 
like that in earlyish 2020. I paid attention to qed.py and the other 
non-numerical files.


Maybe badly rotted by now, I don't know.


(Alternatively, we could have the opposite, i.e. an explicit list of
files that we do want to check, but I think it is better to check files
by default.)



I agree. Stop the bleeding first and worry about the rest after.


Unless started in debug mode (./check -d), the output has no information
on which files are tested, so we will not have a problem e.g. with
backports, where some files may be missing when compared to upstream.

Besides the technical rewrite, some more things are changed:

- For the pylint invocation, PYTHONPATH is adjusted.  This mirrors
   setting MYPYPATH for mypy.

- Also, MYPYPATH is now derived from PYTHONPATH, so that we include
   paths set by the environment.  Maybe at some point we want to let the
   check script add '../../python/' to PYTHONPATH so that iotests.py does
   not need to do that.



Does this solve an observed problem, or is it preventative? I ran into 
trouble once by pointing mypy to my system python libraries; it seemed 
to have a check that explicitly warned me against such tricks.


I guess for now, if it works, it works. :o)


- Passing --notes=FIXME,XXX to pylint suppresses warnings for TODO
   comments.  TODO is fine, we do not need 297 to complain about such
   comments.



Agreed. You can also edit pylintrc to choose which keywords trigger the 
check -- "TODO" is probably fine, but "FIXME" is maybe a shade worse. 
Season to taste.



- The "Success" line from mypy's output is suppressed, because (A) it
   does not add useful information, and (B) it would leak information
   about the files having been tested to the reference output, which we
   decidedly do not want.

Suggested-by: Vladimir Sementsov-Ogievskiy 
Signed-off-by: Max Reitz 
---
  tests/qemu-iotests/297 | 112 +
  tests/qemu-iotests/297.out |   5 +-
  2 files changed, 92 insertions(+), 25 deletions(-)

diff --git a/tests/qemu-iotests/297 b/tests/qemu-iotests/297
index 5c5420712b..e3db3e061e 100755
--- a/tests/qemu-iotests/297
+++ b/tests/qemu-iotests/297
@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/usr/bin/env python3
  #
  # Copyright (C) 2020 Red Hat, Inc.


You could bump it up, if you wanted.


  #
@@ -15,30 +15,98 @@
  # You should have received a copy of the GNU General Public License
  # along with this program.  If not, see .
  
-seq=$(basename $0)

-echo "QA output created by $seq"
+import os
+import re
+import shutil
+import subprocess
+import sys
  
-status=1	# failure is the default!

+import iotests
  
-# get standard environment

-. ./common.rc
  
-if ! type -p "pylint-3" > /dev/null; then

-_notrun "pylint-3 not found"
-fi
-if ! type -p "mypy" > /dev/null; then
-_notrun "mypy not found"
-fi
+# TODO: Empty this list!
+SKIP_FILES = (
+'030', '040', '041', '044', '045', '055', '056', '057', '065', '093',
+'096', '118', '124', '129', '132', '136', '139', '147', '148', '149',
+'151', '152', '155', '163', '165', '169', '194', '196', '199', '202',
+'203', '205', '206', '207', '208', '210', '211', '212', '213', '216',
+'218', '219', '222', '224', '228', '234', '235', '236', '237', '238',
+'240', '242', '245', '246', '248', '255', '256', '257', '258', '260',
+'262', '264', '266', '274', '277', '280', '281', '295', '296', '298',
+'299', '300', '302', '303', '304', '307',
+'nbd-fault-injector.py', 'qcow2.py', 'qcow2_format.py', 'qed.py'
+)
  
-pylint-3 --score=n iotests.py
  
-MYPYPATH=../../python/ mypy --warn-unused-configs --disallow-subclassing-any \

---disallow-any-generics --disallow-incomplete-defs \
---disallow-untyped-decorators --no-implicit-optional \
---warn-redundant-casts --warn-unused-ignores \
---no-implicit-reexport iotests.py
+def is_python_file(filename):
+if not os.path.isfile(filename):
+return False
  
-# success, all done

-echo "*** done"
-rm -f $seq.full
-status=0
+if filename.endswith('.py'):
+return True
+
+with open(filename) as f:
+try:
+first_line = f.readline()
+return re.match('^#!.*python', first_line) is not None
+except UnicodeDecodeError:  # Ignore binary files
+return False
+
+




+def run_linters():
+files = [filename for filename in (set(os.listdir('.')) - set(SKIP_FILES))
+ if is_python_file(filename)]
+
+iotests.logger.debug('Files to be checked:')
+iotests.logger.debug(', 

Re: [PATCH v5 01/10] iotests.py: Assume a couple of variables as given

[John Snow]

On 1/18/21 5:57 AM, Max Reitz wrote:

There are a couple of environment variables that we fetch with
os.environ.get() without supplying a default.  Clearly they are required
and expected to be set by the ./check script (as evidenced by
execute_setup_common(), which checks for test_dir and
qemu_default_machine to be set, and aborts if they are not).

Using .get() this way has the disadvantage of returning an Optional[str]
type, which mypy will complain about when tests just assume these values
to be str.

Use [] instead, which raises a KeyError for environment variables that
are not set.  When this exception is raised, catch it and move the abort
code from execute_setup_common() there.



Good idea.


Drop the 'assert iotests.sock_dir is not None' from iotest 300, because
that sort of thing is precisely what this patch wants to prevent.

Signed-off-by: Max Reitz 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Willian Rampazzo 


Reviewed-by: John Snow 

Re: [PATCH V6 4/6] hw/block/nvme: support for multi-controller in subsystem

[Minwoo Im]

On 21-01-26 09:57:23, Keith Busch wrote:
> On Tue, Jan 26, 2021 at 09:52:48AM +0900, Minwoo Im wrote:
> > On 21-01-25 10:11:43, Keith Busch wrote:
> > > On Mon, Jan 25, 2021 at 07:03:32PM +0100, Klaus Jensen wrote:
> > > > On Jan 24 11:54, Minwoo Im wrote:
> > > > > We have nvme-subsys and nvme devices mapped together.  To support
> > > > > multi-controller scheme to this setup, controller identifier(id) has 
> > > > > to
> > > > > be managed.  Earlier, cntlid(controller id) used to be always 0 
> > > > > because
> > > > > we didn't have any subsystem scheme that controller id matters.
> > > > > 
> > > > > This patch introduced 'cntlid' attribute to the nvme controller
> > > > > instance(NvmeCtrl) and make it allocated by the nvme-subsys device
> > > > > mapped to the controller.  If nvme-subsys is not given to the
> > > > > controller, then it will always be 0 as it was.
> > > > > 
> > > > > Added 'ctrls' array in the nvme-subsys instance to manage attached
> > > > > controllers to the subsystem with a limit(32).  This patch didn't take
> > > > > list for the controllers to make it seamless with nvme-ns device.
> > > > > 
> > > > > Signed-off-by: Minwoo Im 
> > > > > ---
> > > > >  hw/block/nvme-subsys.c | 21 +
> > > > >  hw/block/nvme-subsys.h |  4 
> > > > >  hw/block/nvme.c| 29 +
> > > > >  hw/block/nvme.h|  1 +
> > > > >  4 files changed, 55 insertions(+)
> > > > > 
> > > > > diff --git a/hw/block/nvme.c b/hw/block/nvme.c
> > > > > index b525fca14103..7138389be4bd 100644
> > > > > --- a/hw/block/nvme.c
> > > > > +++ b/hw/block/nvme.c
> > > > > @@ -4481,6 +4484,10 @@ static void nvme_init_ctrl(NvmeCtrl *n, 
> > > > > PCIDevice *pci_dev)
> > > > >  id->psd[0].enlat = cpu_to_le32(0x10);
> > > > >  id->psd[0].exlat = cpu_to_le32(0x4);
> > > > >  
> > > > > +if (n->subsys) {
> > > > > +id->cmic |= NVME_CMIC_MULTI_CTRL;
> > > > > +}
> > > > 
> > > > Since multiple controllers show up with a PCIe port of their own, do we
> > > > need to set bit 0 (NVME_CMIC_MULTI_PORT?) as well? Or am I
> > > > misunderstanding that bit?
> > > 
> > > AIUI, if you report this MULTI_PORT bit, then each PCI device in the
> > > subsystem needs to report a different "Port Number" in their PCIe Link
> > > Capabilities register. I don't think we can manipulate that value from
> > > the nvme "device", but I also don't know what a host should do with this
> > > information even if we could. So, I think it's safe to leave it at 0.
> > 
> > AFAIK, If we leave it to 0, kernel will not allocate disk for multi-path
> > case (e.g., nvmeXcYnZ).
> 
> Kernel only checks for MULTI_CTRL. It doesn't do anything with MULTI_PORT.

Please forgive me that I took this discussion as MULTI_CTRL rather than
MULTI_PORT.  Please ignore this noise ;)

Thanks!

Re: [PATCH V6 0/6] hw/block/nvme: support multi-path for ctrl/ns

[Keith Busch]

This came out looking cleaner than I had initially expected. Thanks for
seeing this feature through!

Reviewed-by: Keith Busch 

Re: [PATCH v2 1/2] tests/qtest: Only run fuzz-megasas-test if megasas device is available

[Thomas Huth]

On 26/01/2021 19.01, Alexander Bulekov wrote:

On 210126 1851, Thomas Huth wrote:

On 26/01/2021 12.16, Philippe Mathieu-Daudé wrote:

This test fails when QEMU is built without the megasas device,
restrict it to its availability.

Signed-off-by: Philippe Mathieu-Daudé 
---
   tests/qtest/fuzz-megasas-test.c | 49 +
   tests/qtest/fuzz-test.c | 25 -
   MAINTAINERS |  1 +
   tests/qtest/meson.build |  4 ++-
   4 files changed, 53 insertions(+), 26 deletions(-)
   create mode 100644 tests/qtest/fuzz-megasas-test.c

diff --git a/tests/qtest/fuzz-megasas-test.c b/tests/qtest/fuzz-megasas-test.c
new file mode 100644
index 000..940a76bf25a
--- /dev/null
+++ b/tests/qtest/fuzz-megasas-test.c
@@ -0,0 +1,49 @@
+/*
+ * QTest fuzzer-generated testcase for megasas device
+ *
+ * Copyright (c) 2020 Li Qiang 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+
+#include "libqos/libqtest.h"
+
+/*
+ * This used to trigger the assert in scsi_dma_complete
+ * https://bugs.launchpad.net/qemu/+bug/1878263
+ */
+static void test_lp1878263_megasas_zero_iov_cnt(void)
+{
+QTestState *s;
+
+s = qtest_init("-nographic -monitor none -serial none "
+   "-M q35 -device megasas -device scsi-cd,drive=null0 "
+   "-blockdev driver=null-co,read-zeroes=on,node-name=null0");
+qtest_outl(s, 0xcf8, 0x80001818);
+qtest_outl(s, 0xcfc, 0xc101);
+qtest_outl(s, 0xcf8, 0x8000181c);
+qtest_outl(s, 0xcf8, 0x80001804);
+qtest_outw(s, 0xcfc, 0x7);
+qtest_outl(s, 0xcf8, 0x8000186a);
+qtest_writeb(s, 0x14, 0xfe);
+qtest_writeb(s, 0x0, 0x02);
+qtest_outb(s, 0xc1c0, 0x17);
+qtest_quit(s);
+}
+
+int main(int argc, char **argv)
+{
+const char *arch = qtest_get_arch();
+
+g_test_init(, , NULL);
+
+if (strcmp(arch, "i386") == 0 || strcmp(arch, "x86_64") == 0) {
+qtest_add_func("fuzz/test_lp1878263_megasas_zero_iov_cnt",
+   test_lp1878263_megasas_zero_iov_cnt);
+}
+
+return g_test_run();
+}
diff --git a/tests/qtest/fuzz-test.c b/tests/qtest/fuzz-test.c
index cdb1100a0b8..6188fbb8e96 100644
--- a/tests/qtest/fuzz-test.c
+++ b/tests/qtest/fuzz-test.c
@@ -11,29 +11,6 @@
   #include "libqos/libqtest.h"
-/*
- * This used to trigger the assert in scsi_dma_complete
- * https://bugs.launchpad.net/qemu/+bug/1878263
- */
-static void test_lp1878263_megasas_zero_iov_cnt(void)
-{
-QTestState *s;
-
-s = qtest_init("-nographic -monitor none -serial none "
-   "-M q35 -device megasas -device scsi-cd,drive=null0 "
-   "-blockdev driver=null-co,read-zeroes=on,node-name=null0");
-qtest_outl(s, 0xcf8, 0x80001818);
-qtest_outl(s, 0xcfc, 0xc101);
-qtest_outl(s, 0xcf8, 0x8000181c);
-qtest_outl(s, 0xcf8, 0x80001804);
-qtest_outw(s, 0xcfc, 0x7);
-qtest_outl(s, 0xcf8, 0x8000186a);
-qtest_writeb(s, 0x14, 0xfe);
-qtest_writeb(s, 0x0, 0x02);
-qtest_outb(s, 0xc1c0, 0x17);
-qtest_quit(s);
-}
-
   static void test_lp1878642_pci_bus_get_irq_level_assert(void)
   {
   QTestState *s;
@@ -104,8 +81,6 @@ int main(int argc, char **argv)
   g_test_init(, , NULL);
   if (strcmp(arch, "i386") == 0 || strcmp(arch, "x86_64") == 0) {
-qtest_add_func("fuzz/test_lp1878263_megasas_zero_iov_cnt",
-   test_lp1878263_megasas_zero_iov_cnt);
   qtest_add_func("fuzz/test_lp1878642_pci_bus_get_irq_level_assert",
  test_lp1878642_pci_bus_get_irq_level_assert);
   qtest_add_func("fuzz/test_mmio_oob_from_memory_region_cache",
diff --git a/MAINTAINERS b/MAINTAINERS
index 34359a99b8e..44cd74b03cd 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1925,6 +1925,7 @@ S: Supported
   F: hw/scsi/megasas.c
   F: hw/scsi/mfi.h
   F: tests/qtest/megasas-test.c
+F: tests/qtest/fuzz-megasas-test.c
   Network packet abstractions
   M: Dmitry Fleytman 
diff --git a/tests/qtest/meson.build b/tests/qtest/meson.build
index 16d04625b8b..85682d0dfce 100644
--- a/tests/qtest/meson.build
+++ b/tests/qtest/meson.build
@@ -4,7 +4,9 @@
 subdir_done()
   endif
-qtests_generic = [
+qtests_generic = \
+  (config_all_devices.has_key('CONFIG_MEGASAS_SCSI_PCI') ? 
['fuzz-megasas-test'] : []) + \
+  [
 'cdrom-test',
 'device-introspect-test',
 'machine-none-test',



Reviewed-by: Thomas Huth 

I assume Alexander will take this patch through his fuzzer branch now? Or
shall I take it via the qtest branch?


I can take take this through my branch, unless thats somehow inconvenient.


That's perfectly fine!

 Thanks,
  Thomas

Re: [PATCH v2 1/2] tests/qtest: Only run fuzz-megasas-test if megasas device is available

[Alexander Bulekov]

On 210126 1851, Thomas Huth wrote:
> On 26/01/2021 12.16, Philippe Mathieu-Daudé wrote:
> > This test fails when QEMU is built without the megasas device,
> > restrict it to its availability.
> > 
> > Signed-off-by: Philippe Mathieu-Daudé 
> > ---
> >   tests/qtest/fuzz-megasas-test.c | 49 +
> >   tests/qtest/fuzz-test.c | 25 -
> >   MAINTAINERS |  1 +
> >   tests/qtest/meson.build |  4 ++-
> >   4 files changed, 53 insertions(+), 26 deletions(-)
> >   create mode 100644 tests/qtest/fuzz-megasas-test.c
> > 
> > diff --git a/tests/qtest/fuzz-megasas-test.c 
> > b/tests/qtest/fuzz-megasas-test.c
> > new file mode 100644
> > index 000..940a76bf25a
> > --- /dev/null
> > +++ b/tests/qtest/fuzz-megasas-test.c
> > @@ -0,0 +1,49 @@
> > +/*
> > + * QTest fuzzer-generated testcase for megasas device
> > + *
> > + * Copyright (c) 2020 Li Qiang 
> > + *
> > + * This work is licensed under the terms of the GNU GPL, version 2 or 
> > later.
> > + * See the COPYING file in the top-level directory.
> > + */
> > +
> > +#include "qemu/osdep.h"
> > +
> > +#include "libqos/libqtest.h"
> > +
> > +/*
> > + * This used to trigger the assert in scsi_dma_complete
> > + * https://bugs.launchpad.net/qemu/+bug/1878263
> > + */
> > +static void test_lp1878263_megasas_zero_iov_cnt(void)
> > +{
> > +QTestState *s;
> > +
> > +s = qtest_init("-nographic -monitor none -serial none "
> > +   "-M q35 -device megasas -device scsi-cd,drive=null0 "
> > +   "-blockdev 
> > driver=null-co,read-zeroes=on,node-name=null0");
> > +qtest_outl(s, 0xcf8, 0x80001818);
> > +qtest_outl(s, 0xcfc, 0xc101);
> > +qtest_outl(s, 0xcf8, 0x8000181c);
> > +qtest_outl(s, 0xcf8, 0x80001804);
> > +qtest_outw(s, 0xcfc, 0x7);
> > +qtest_outl(s, 0xcf8, 0x8000186a);
> > +qtest_writeb(s, 0x14, 0xfe);
> > +qtest_writeb(s, 0x0, 0x02);
> > +qtest_outb(s, 0xc1c0, 0x17);
> > +qtest_quit(s);
> > +}
> > +
> > +int main(int argc, char **argv)
> > +{
> > +const char *arch = qtest_get_arch();
> > +
> > +g_test_init(, , NULL);
> > +
> > +if (strcmp(arch, "i386") == 0 || strcmp(arch, "x86_64") == 0) {
> > +qtest_add_func("fuzz/test_lp1878263_megasas_zero_iov_cnt",
> > +   test_lp1878263_megasas_zero_iov_cnt);
> > +}
> > +
> > +return g_test_run();
> > +}
> > diff --git a/tests/qtest/fuzz-test.c b/tests/qtest/fuzz-test.c
> > index cdb1100a0b8..6188fbb8e96 100644
> > --- a/tests/qtest/fuzz-test.c
> > +++ b/tests/qtest/fuzz-test.c
> > @@ -11,29 +11,6 @@
> >   #include "libqos/libqtest.h"
> > -/*
> > - * This used to trigger the assert in scsi_dma_complete
> > - * https://bugs.launchpad.net/qemu/+bug/1878263
> > - */
> > -static void test_lp1878263_megasas_zero_iov_cnt(void)
> > -{
> > -QTestState *s;
> > -
> > -s = qtest_init("-nographic -monitor none -serial none "
> > -   "-M q35 -device megasas -device scsi-cd,drive=null0 "
> > -   "-blockdev 
> > driver=null-co,read-zeroes=on,node-name=null0");
> > -qtest_outl(s, 0xcf8, 0x80001818);
> > -qtest_outl(s, 0xcfc, 0xc101);
> > -qtest_outl(s, 0xcf8, 0x8000181c);
> > -qtest_outl(s, 0xcf8, 0x80001804);
> > -qtest_outw(s, 0xcfc, 0x7);
> > -qtest_outl(s, 0xcf8, 0x8000186a);
> > -qtest_writeb(s, 0x14, 0xfe);
> > -qtest_writeb(s, 0x0, 0x02);
> > -qtest_outb(s, 0xc1c0, 0x17);
> > -qtest_quit(s);
> > -}
> > -
> >   static void test_lp1878642_pci_bus_get_irq_level_assert(void)
> >   {
> >   QTestState *s;
> > @@ -104,8 +81,6 @@ int main(int argc, char **argv)
> >   g_test_init(, , NULL);
> >   if (strcmp(arch, "i386") == 0 || strcmp(arch, "x86_64") == 0) {
> > -qtest_add_func("fuzz/test_lp1878263_megasas_zero_iov_cnt",
> > -   test_lp1878263_megasas_zero_iov_cnt);
> >   qtest_add_func("fuzz/test_lp1878642_pci_bus_get_irq_level_assert",
> >  test_lp1878642_pci_bus_get_irq_level_assert);
> >   qtest_add_func("fuzz/test_mmio_oob_from_memory_region_cache",
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index 34359a99b8e..44cd74b03cd 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -1925,6 +1925,7 @@ S: Supported
> >   F: hw/scsi/megasas.c
> >   F: hw/scsi/mfi.h
> >   F: tests/qtest/megasas-test.c
> > +F: tests/qtest/fuzz-megasas-test.c
> >   Network packet abstractions
> >   M: Dmitry Fleytman 
> > diff --git a/tests/qtest/meson.build b/tests/qtest/meson.build
> > index 16d04625b8b..85682d0dfce 100644
> > --- a/tests/qtest/meson.build
> > +++ b/tests/qtest/meson.build
> > @@ -4,7 +4,9 @@
> > subdir_done()
> >   endif
> > -qtests_generic = [
> > +qtests_generic = \
> > +  (config_all_devices.has_key('CONFIG_MEGASAS_SCSI_PCI') ? 
> > ['fuzz-megasas-test'] : []) + \
> > +  [
> > 'cdrom-test',
> > 'device-introspect-test',
> > 'machine-none-test',
> > 
> 
> 

Re: [PATCH V6 4/6] hw/block/nvme: support for multi-controller in subsystem

[Keith Busch]

On Tue, Jan 26, 2021 at 09:52:48AM +0900, Minwoo Im wrote:
> On 21-01-25 10:11:43, Keith Busch wrote:
> > On Mon, Jan 25, 2021 at 07:03:32PM +0100, Klaus Jensen wrote:
> > > On Jan 24 11:54, Minwoo Im wrote:
> > > > We have nvme-subsys and nvme devices mapped together.  To support
> > > > multi-controller scheme to this setup, controller identifier(id) has to
> > > > be managed.  Earlier, cntlid(controller id) used to be always 0 because
> > > > we didn't have any subsystem scheme that controller id matters.
> > > > 
> > > > This patch introduced 'cntlid' attribute to the nvme controller
> > > > instance(NvmeCtrl) and make it allocated by the nvme-subsys device
> > > > mapped to the controller.  If nvme-subsys is not given to the
> > > > controller, then it will always be 0 as it was.
> > > > 
> > > > Added 'ctrls' array in the nvme-subsys instance to manage attached
> > > > controllers to the subsystem with a limit(32).  This patch didn't take
> > > > list for the controllers to make it seamless with nvme-ns device.
> > > > 
> > > > Signed-off-by: Minwoo Im 
> > > > ---
> > > >  hw/block/nvme-subsys.c | 21 +
> > > >  hw/block/nvme-subsys.h |  4 
> > > >  hw/block/nvme.c| 29 +
> > > >  hw/block/nvme.h|  1 +
> > > >  4 files changed, 55 insertions(+)
> > > > 
> > > > diff --git a/hw/block/nvme.c b/hw/block/nvme.c
> > > > index b525fca14103..7138389be4bd 100644
> > > > --- a/hw/block/nvme.c
> > > > +++ b/hw/block/nvme.c
> > > > @@ -4481,6 +4484,10 @@ static void nvme_init_ctrl(NvmeCtrl *n, 
> > > > PCIDevice *pci_dev)
> > > >  id->psd[0].enlat = cpu_to_le32(0x10);
> > > >  id->psd[0].exlat = cpu_to_le32(0x4);
> > > >  
> > > > +if (n->subsys) {
> > > > +id->cmic |= NVME_CMIC_MULTI_CTRL;
> > > > +}
> > > 
> > > Since multiple controllers show up with a PCIe port of their own, do we
> > > need to set bit 0 (NVME_CMIC_MULTI_PORT?) as well? Or am I
> > > misunderstanding that bit?
> > 
> > AIUI, if you report this MULTI_PORT bit, then each PCI device in the
> > subsystem needs to report a different "Port Number" in their PCIe Link
> > Capabilities register. I don't think we can manipulate that value from
> > the nvme "device", but I also don't know what a host should do with this
> > information even if we could. So, I think it's safe to leave it at 0.
> 
> AFAIK, If we leave it to 0, kernel will not allocate disk for multi-path
> case (e.g., nvmeXcYnZ).

Kernel only checks for MULTI_CTRL. It doesn't do anything with MULTI_PORT.

Re: [PATCH v2 2/2] tests/qtest: Only run fuzz-virtio-scsi when virtio-scsi is available

[Thomas Huth]

On 26/01/2021 12.16, Philippe Mathieu-Daudé wrote:

This test fails when QEMU is built without the virtio-scsi device,
restrict it to its availability.

Reviewed-by: Michael S. Tsirkin 
Signed-off-by: Philippe Mathieu-Daudé 
---
  tests/qtest/fuzz-test.c | 51 
  tests/qtest/fuzz-virtio-scsi-test.c | 75 +
  MAINTAINERS |  1 +
  tests/qtest/meson.build |  1 +
  4 files changed, 77 insertions(+), 51 deletions(-)
  create mode 100644 tests/qtest/fuzz-virtio-scsi-test.c

diff --git a/tests/qtest/fuzz-test.c b/tests/qtest/fuzz-test.c
index 6188fbb8e96..d112798afe3 100644
--- a/tests/qtest/fuzz-test.c
+++ b/tests/qtest/fuzz-test.c
@@ -25,55 +25,6 @@ static void test_lp1878642_pci_bus_get_irq_level_assert(void)
  qtest_quit(s);
  }
  
-/*

- * Here a MemoryRegionCache pointed to an MMIO region but had a
- * larger size than the underlying region.
- */
-static void test_mmio_oob_from_memory_region_cache(void)
-{
-QTestState *s;
-
-s = qtest_init("-M pc-q35-5.2 -display none -m 512M "
-  "-device virtio-scsi,num_queues=8,addr=03.0 ");
-
-qtest_outl(s, 0xcf8, 0x80001811);
-qtest_outb(s, 0xcfc, 0x6e);
-qtest_outl(s, 0xcf8, 0x80001824);
-qtest_outl(s, 0xcf8, 0x80001813);
-qtest_outl(s, 0xcfc, 0xa08);
-qtest_outl(s, 0xcf8, 0x80001802);
-qtest_outl(s, 0xcfc, 0x5a175a63);
-qtest_outb(s, 0x6e08, 0x9e);
-qtest_writeb(s, 0x9f003, 0xff);
-qtest_writeb(s, 0x9f004, 0x01);
-qtest_writeb(s, 0x9e012, 0x0e);
-qtest_writeb(s, 0x9e01b, 0x0e);
-qtest_writeb(s, 0x9f006, 0x01);
-qtest_writeb(s, 0x9f008, 0x01);
-qtest_writeb(s, 0x9f00a, 0x01);
-qtest_writeb(s, 0x9f00c, 0x01);
-qtest_writeb(s, 0x9f00e, 0x01);
-qtest_writeb(s, 0x9f010, 0x01);
-qtest_writeb(s, 0x9f012, 0x01);
-qtest_writeb(s, 0x9f014, 0x01);
-qtest_writeb(s, 0x9f016, 0x01);
-qtest_writeb(s, 0x9f018, 0x01);
-qtest_writeb(s, 0x9f01a, 0x01);
-qtest_writeb(s, 0x9f01c, 0x01);
-qtest_writeb(s, 0x9f01e, 0x01);
-qtest_writeb(s, 0x9f020, 0x01);
-qtest_writeb(s, 0x9f022, 0x01);
-qtest_writeb(s, 0x9f024, 0x01);
-qtest_writeb(s, 0x9f026, 0x01);
-qtest_writeb(s, 0x9f028, 0x01);
-qtest_writeb(s, 0x9f02a, 0x01);
-qtest_writeb(s, 0x9f02c, 0x01);
-qtest_writeb(s, 0x9f02e, 0x01);
-qtest_writeb(s, 0x9f030, 0x01);
-qtest_outb(s, 0x6e10, 0x00);
-qtest_quit(s);
-}
-
  int main(int argc, char **argv)
  {
  const char *arch = qtest_get_arch();
@@ -83,8 +34,6 @@ int main(int argc, char **argv)
  if (strcmp(arch, "i386") == 0 || strcmp(arch, "x86_64") == 0) {
  qtest_add_func("fuzz/test_lp1878642_pci_bus_get_irq_level_assert",
 test_lp1878642_pci_bus_get_irq_level_assert);
-qtest_add_func("fuzz/test_mmio_oob_from_memory_region_cache",
-   test_mmio_oob_from_memory_region_cache);
  }
  
  return g_test_run();

diff --git a/tests/qtest/fuzz-virtio-scsi-test.c 
b/tests/qtest/fuzz-virtio-scsi-test.c
new file mode 100644
index 000..aaf6d10e189
--- /dev/null
+++ b/tests/qtest/fuzz-virtio-scsi-test.c
@@ -0,0 +1,75 @@
+/*
+ * QTest fuzzer-generated testcase for virtio-scsi device
+ *
+ * Copyright (c) 2020 Li Qiang 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+
+#include "libqos/libqtest.h"
+
+/*
+ * Here a MemoryRegionCache pointed to an MMIO region but had a
+ * larger size than the underlying region.
+ */
+static void test_mmio_oob_from_memory_region_cache(void)
+{
+QTestState *s;
+
+s = qtest_init("-M pc-q35-5.2 -display none -m 512M "
+   "-device virtio-scsi,num_queues=8,addr=03.0 ");
+
+qtest_outl(s, 0xcf8, 0x80001811);
+qtest_outb(s, 0xcfc, 0x6e);
+qtest_outl(s, 0xcf8, 0x80001824);
+qtest_outl(s, 0xcf8, 0x80001813);
+qtest_outl(s, 0xcfc, 0xa08);
+qtest_outl(s, 0xcf8, 0x80001802);
+qtest_outl(s, 0xcfc, 0x5a175a63);
+qtest_outb(s, 0x6e08, 0x9e);
+qtest_writeb(s, 0x9f003, 0xff);
+qtest_writeb(s, 0x9f004, 0x01);
+qtest_writeb(s, 0x9e012, 0x0e);
+qtest_writeb(s, 0x9e01b, 0x0e);
+qtest_writeb(s, 0x9f006, 0x01);
+qtest_writeb(s, 0x9f008, 0x01);
+qtest_writeb(s, 0x9f00a, 0x01);
+qtest_writeb(s, 0x9f00c, 0x01);
+qtest_writeb(s, 0x9f00e, 0x01);
+qtest_writeb(s, 0x9f010, 0x01);
+qtest_writeb(s, 0x9f012, 0x01);
+qtest_writeb(s, 0x9f014, 0x01);
+qtest_writeb(s, 0x9f016, 0x01);
+qtest_writeb(s, 0x9f018, 0x01);
+qtest_writeb(s, 0x9f01a, 0x01);
+qtest_writeb(s, 0x9f01c, 0x01);
+qtest_writeb(s, 0x9f01e, 0x01);
+qtest_writeb(s, 0x9f020, 0x01);
+qtest_writeb(s, 0x9f022, 0x01);
+qtest_writeb(s, 0x9f024, 0x01);
+qtest_writeb(s, 0x9f026, 0x01);
+qtest_writeb(s, 0x9f028, 0x01);
+qtest_writeb(s, 

Re: [PATCH v2 1/2] tests/qtest: Only run fuzz-megasas-test if megasas device is available

[Thomas Huth]

On 26/01/2021 12.16, Philippe Mathieu-Daudé wrote:

This test fails when QEMU is built without the megasas device,
restrict it to its availability.

Signed-off-by: Philippe Mathieu-Daudé 
---
  tests/qtest/fuzz-megasas-test.c | 49 +
  tests/qtest/fuzz-test.c | 25 -
  MAINTAINERS |  1 +
  tests/qtest/meson.build |  4 ++-
  4 files changed, 53 insertions(+), 26 deletions(-)
  create mode 100644 tests/qtest/fuzz-megasas-test.c

diff --git a/tests/qtest/fuzz-megasas-test.c b/tests/qtest/fuzz-megasas-test.c
new file mode 100644
index 000..940a76bf25a
--- /dev/null
+++ b/tests/qtest/fuzz-megasas-test.c
@@ -0,0 +1,49 @@
+/*
+ * QTest fuzzer-generated testcase for megasas device
+ *
+ * Copyright (c) 2020 Li Qiang 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+
+#include "libqos/libqtest.h"
+
+/*
+ * This used to trigger the assert in scsi_dma_complete
+ * https://bugs.launchpad.net/qemu/+bug/1878263
+ */
+static void test_lp1878263_megasas_zero_iov_cnt(void)
+{
+QTestState *s;
+
+s = qtest_init("-nographic -monitor none -serial none "
+   "-M q35 -device megasas -device scsi-cd,drive=null0 "
+   "-blockdev driver=null-co,read-zeroes=on,node-name=null0");
+qtest_outl(s, 0xcf8, 0x80001818);
+qtest_outl(s, 0xcfc, 0xc101);
+qtest_outl(s, 0xcf8, 0x8000181c);
+qtest_outl(s, 0xcf8, 0x80001804);
+qtest_outw(s, 0xcfc, 0x7);
+qtest_outl(s, 0xcf8, 0x8000186a);
+qtest_writeb(s, 0x14, 0xfe);
+qtest_writeb(s, 0x0, 0x02);
+qtest_outb(s, 0xc1c0, 0x17);
+qtest_quit(s);
+}
+
+int main(int argc, char **argv)
+{
+const char *arch = qtest_get_arch();
+
+g_test_init(, , NULL);
+
+if (strcmp(arch, "i386") == 0 || strcmp(arch, "x86_64") == 0) {
+qtest_add_func("fuzz/test_lp1878263_megasas_zero_iov_cnt",
+   test_lp1878263_megasas_zero_iov_cnt);
+}
+
+return g_test_run();
+}
diff --git a/tests/qtest/fuzz-test.c b/tests/qtest/fuzz-test.c
index cdb1100a0b8..6188fbb8e96 100644
--- a/tests/qtest/fuzz-test.c
+++ b/tests/qtest/fuzz-test.c
@@ -11,29 +11,6 @@
  
  #include "libqos/libqtest.h"
  
-/*

- * This used to trigger the assert in scsi_dma_complete
- * https://bugs.launchpad.net/qemu/+bug/1878263
- */
-static void test_lp1878263_megasas_zero_iov_cnt(void)
-{
-QTestState *s;
-
-s = qtest_init("-nographic -monitor none -serial none "
-   "-M q35 -device megasas -device scsi-cd,drive=null0 "
-   "-blockdev driver=null-co,read-zeroes=on,node-name=null0");
-qtest_outl(s, 0xcf8, 0x80001818);
-qtest_outl(s, 0xcfc, 0xc101);
-qtest_outl(s, 0xcf8, 0x8000181c);
-qtest_outl(s, 0xcf8, 0x80001804);
-qtest_outw(s, 0xcfc, 0x7);
-qtest_outl(s, 0xcf8, 0x8000186a);
-qtest_writeb(s, 0x14, 0xfe);
-qtest_writeb(s, 0x0, 0x02);
-qtest_outb(s, 0xc1c0, 0x17);
-qtest_quit(s);
-}
-
  static void test_lp1878642_pci_bus_get_irq_level_assert(void)
  {
  QTestState *s;
@@ -104,8 +81,6 @@ int main(int argc, char **argv)
  g_test_init(, , NULL);
  
  if (strcmp(arch, "i386") == 0 || strcmp(arch, "x86_64") == 0) {

-qtest_add_func("fuzz/test_lp1878263_megasas_zero_iov_cnt",
-   test_lp1878263_megasas_zero_iov_cnt);
  qtest_add_func("fuzz/test_lp1878642_pci_bus_get_irq_level_assert",
 test_lp1878642_pci_bus_get_irq_level_assert);
  qtest_add_func("fuzz/test_mmio_oob_from_memory_region_cache",
diff --git a/MAINTAINERS b/MAINTAINERS
index 34359a99b8e..44cd74b03cd 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1925,6 +1925,7 @@ S: Supported
  F: hw/scsi/megasas.c
  F: hw/scsi/mfi.h
  F: tests/qtest/megasas-test.c
+F: tests/qtest/fuzz-megasas-test.c
  
  Network packet abstractions

  M: Dmitry Fleytman 
diff --git a/tests/qtest/meson.build b/tests/qtest/meson.build
index 16d04625b8b..85682d0dfce 100644
--- a/tests/qtest/meson.build
+++ b/tests/qtest/meson.build
@@ -4,7 +4,9 @@
subdir_done()
  endif
  
-qtests_generic = [

+qtests_generic = \
+  (config_all_devices.has_key('CONFIG_MEGASAS_SCSI_PCI') ? 
['fuzz-megasas-test'] : []) + \
+  [
'cdrom-test',
'device-introspect-test',
'machine-none-test',



Reviewed-by: Thomas Huth 

I assume Alexander will take this patch through his fuzzer branch now? Or 
shall I take it via the qtest branch?


 Thomas

Re: [PATCH] hw/block/nvme: add broadcast nsid support flush command

[Stefan Hajnoczi]

On Mon, Jan 25, 2021 at 09:42:31PM +0100, Klaus Jensen wrote:
> From: Gollu Appalanaidu 
> 
> Add support for using the broadcast nsid to issue a flush on all
> namespaces through a single command.
> 
> Signed-off-by: Gollu Appalanaidu 
> Reviewed-by: Klaus Jensen 
> ---
>  include/block/nvme.h  |   8 +++
>  hw/block/nvme.c   | 123 +++---
>  hw/block/trace-events |   2 +
>  3 files changed, 126 insertions(+), 7 deletions(-)

Acked-by: Stefan Hajnoczi 


signature.asc
Description: PGP signature

Re: [PATCH v9 09/11] block: check availablity for preadv/pwritev on mac

[Peter Maydell]

On Tue, 26 Jan 2021 at 01:38, Joelle van Dyne  wrote:
>
> macOS 11/iOS 14 added preadv/pwritev APIs. Due to weak linking, configure
> will succeed with CONFIG_PREADV even when targeting a lower OS version.

I just ran into this this afternoon. It turns out that all our OSX
CI configs pass --enable-werror or equivalent to configure, which
means that when the configure test provokes the warning that
"'preadv' is only available on macOS 11.0 or newer", that is a
compile error due to -Werror, and configure decides preadv is
not available. But if you do a configure for the default setup that
doesn't add -Werror then the test passes and then the binaries
fail at runtime... and this is the first time I'd happened to do
a build with the newer XCode SDK and without -Werror enabled.

So I think that leaves two points for this patch:

(1) we need to fix the configure test so that it either
succeeds without warnings or fails, so that --enable-werror
and --disable-werror configures make the same decision about
preadv support.

(2) we need to decide whether we want to support the OSX idea
that you can compile in support for a function like preadv()
and then find that it's not present at runtime, or if we just
want to make the choice at configure time. I'm on the fence about
this.

I'm going to send out a patch which converts the configure
test to a meson.build one-liner -- this fixes (1) and
(by default) leaves the answer to (2) at "no" (you get preadv()
only if you built on macOS 11 for macOS 11; if you build with
10.x support then you dont' get it).

I'm agnostic about the final answer to (2) -- we do have the
support for the runtime preadv_present flag in this file already,
after all -- so I guess I'll leave that to the block maintainers.
In the meantime we can fix the non-controversial part.

thanks
-- PMM

Re: [PATCH v9 0/6] Rework iotests/check

[Vladimir Sementsov-Ogievskiy]

26.01.2021 18:36, Kevin Wolf wrote:

Am 26.01.2021 um 16:15 hat Vladimir Sementsov-Ogievskiy geschrieben:

OK, thanks for handling it!


You're welcome.

Only problem now: Max sent a conflicting pull request that touches
'group'. He suggested that we could split the deletion of 'group' from
the 'check' rewrite and merge it only later when nobody touches 'group'
any more (because it's unused).


I think it's OK.. Nothing really wrong in forgetting to remove unused file, and 
remove it later :)



The other option is that I wait a bit or speculatively merge his tree
(with a lot more patches) before doing my pull request in the hope that
it doesn't fail.


When will we move to python 3.7?


I seem to remember that 3.6 is used by more or less all of the current
enterprise distributions, so I'm afraid it will be a while.

Kevin




--
Best regards,
Vladimir

Re: [PATCH v9 0/6] Rework iotests/check

[Kevin Wolf]

Am 26.01.2021 um 16:15 hat Vladimir Sementsov-Ogievskiy geschrieben:
> OK, thanks for handling it!

You're welcome.

Only problem now: Max sent a conflicting pull request that touches
'group'. He suggested that we could split the deletion of 'group' from
the 'check' rewrite and merge it only later when nobody touches 'group'
any more (because it's unused).

The other option is that I wait a bit or speculatively merge his tree
(with a lot more patches) before doing my pull request in the hope that
it doesn't fail.

> When will we move to python 3.7?

I seem to remember that 3.6 is used by more or less all of the current
enterprise distributions, so I'm afraid it will be a while.

Kevin

Re: [PATCH v9 0/6] Rework iotests/check

[Kevin Wolf]

Am 26.01.2021 um 14:19 hat Vladimir Sementsov-Ogievskiy geschrieben:
> 26.01.2021 15:53, Kevin Wolf wrote:
> > Am 25.01.2021 um 19:50 hat Vladimir Sementsov-Ogievskiy geschrieben:
> > > v9:
> > > 01: new, one more whitespace-fixing patch
> > > testenv: allow case when we don't have system-arch emulator, but have 
> > > several for another architectures
> > >   change direct os.access(..., os.X_OK) calls to new helper 
> > > function which also check that path is a file
> > > testrunner: s/fail/not run/ for 'No qualified output'
> > >  drop elapsed time arg for one of 'not run' results (now no 
> > > elapsed time for any 'not run' result)
> > 
> > More CI fun:
> > 
> > Traceback (most recent call last):
> >File "./check", line 117, in 
> >  testfinder = TestFinder(test_dir=env.source_iotests)
> >File "/builds/.../qemu/tests/qemu-iotests/findtests.py", line 53, in 
> > __init__
> >  for line in f:
> >File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
> >  return codecs.ascii_decode(input, self.errors)[0]
> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1591: 
> > ordinal not in range(128)
> 
> Can be solved by adding encoding='utf8' to the open().. But.. Adding it 
> everywhere is not fun.
> 
> So, what about just define PYTHONUTF8=1 for running check in check-block.sh?
> 
> https://docs.python.org/3/using/cmdline.html#envvar-PYTHONUTF8

That looked nice, but we both missed the important line:

"New in version 3.7: See PEP 540 for more details."

So I'm back to explicitly requesting utf-8 encoding everywhere and that
seems to finally make it pass in the CI.

Kevin


diff --git a/tests/qemu-iotests/findtests.py b/tests/qemu-iotests/findtests.py
index d0c72efd6a..dd77b453b8 100644
--- a/tests/qemu-iotests/findtests.py
+++ b/tests/qemu-iotests/findtests.py
@@ -49,7 +49,7 @@ class TestFinder:
os.path.isfile(f + '.out')]
 
 for t in self.all_tests:
-with open(t) as f:
+with open(t, encoding="utf-8") as f:
 for line in f:
 if line.startswith('# group: '):
 for g in line.split()[2:]:
@@ -57,7 +57,7 @@ class TestFinder:
 break
 
 def add_group_file(self, fname: str) -> None:
-with open(fname) as f:
+with open(fname, encoding="utf-8") as f:
 for line in f:
 line = line.strip()
 
diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
index 046f9ce38f..a581be6a29 100644
--- a/tests/qemu-iotests/testrunner.py
+++ b/tests/qemu-iotests/testrunner.py
@@ -41,7 +41,8 @@ def silent_unlink(path: Path) -> None:
 
 
 def file_diff(file1: str, file2: str) -> List[str]:
-with open(file1) as f1, open(file2) as f2:
+with open(file1, encoding="utf-8") as f1, \
+ open(file2, encoding="utf-8") as f2:
 # We want to ignore spaces at line ends. There are a lot of mess about
 # it in iotests.
 # TODO: fix all tests to not produce extra spaces, fix all .out files
@@ -81,7 +82,7 @@ class LastElapsedTime(ContextManager['LastElapsedTime']):
 self.cache: Dict[str, Dict[str, Dict[str, float]]]
 
 try:
-with open(cache_file) as f:
+with open(cache_file, encoding="utf-8") as f:
 self.cache = json.load(f)
 except (OSError, ValueError):
 self.cache = {}
@@ -102,7 +103,7 @@ class LastElapsedTime(ContextManager['LastElapsedTime']):
 d.setdefault(self.env.imgproto, {})[self.env.imgfmt] = elapsed
 
 def save(self) -> None:
-with open(self.cache_file, 'w') as f:
+with open(self.cache_file, 'w', encoding="utf-8") as f:
 json.dump(self.cache, f)
 
 def __enter__(self) -> 'LastElapsedTime':
@@ -245,7 +246,7 @@ class TestRunner(ContextManager['TestRunner']):
 if self.env.debug:
 args.append('-d')
 
-with f_test.open() as f:
+with f_test.open(encoding="utf-8") as f:
 try:
 if f.readline() == '#!/usr/bin/env python3':
 args.insert(0, self.env.python)
@@ -256,7 +257,7 @@ class TestRunner(ContextManager['TestRunner']):
 env.update(self.test_run_env)
 
 t0 = time.time()
-with f_bad.open('w') as f:
+with f_bad.open('w', encoding="utf-8") as f:
 proc = subprocess.Popen(args, cwd=str(f_test.parent), env=env,
 stdout=f, stderr=subprocess.STDOUT)
 try:

Re: [PATCH v9 0/6] Rework iotests/check

[Vladimir Sementsov-Ogievskiy]

26.01.2021 18:07, Kevin Wolf wrote:

Am 26.01.2021 um 14:19 hat Vladimir Sementsov-Ogievskiy geschrieben:

26.01.2021 15:53, Kevin Wolf wrote:

Am 25.01.2021 um 19:50 hat Vladimir Sementsov-Ogievskiy geschrieben:

v9:
01: new, one more whitespace-fixing patch
testenv: allow case when we don't have system-arch emulator, but have several 
for another architectures
   change direct os.access(..., os.X_OK) calls to new helper function 
which also check that path is a file
testrunner: s/fail/not run/ for 'No qualified output'
  drop elapsed time arg for one of 'not run' results (now no 
elapsed time for any 'not run' result)


More CI fun:

Traceback (most recent call last):
File "./check", line 117, in 
  testfinder = TestFinder(test_dir=env.source_iotests)
File "/builds/.../qemu/tests/qemu-iotests/findtests.py", line 53, in 
__init__
  for line in f:
File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
  return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1591: 
ordinal not in range(128)


Can be solved by adding encoding='utf8' to the open().. But.. Adding it 
everywhere is not fun.

So, what about just define PYTHONUTF8=1 for running check in check-block.sh?

https://docs.python.org/3/using/cmdline.html#envvar-PYTHONUTF8


That looked nice, but we both missed the important line:

"New in version 3.7: See PEP 540 for more details."

So I'm back to explicitly requesting utf-8 encoding everywhere and that
seems to finally make it pass in the CI.

Kevin


diff --git a/tests/qemu-iotests/findtests.py b/tests/qemu-iotests/findtests.py
index d0c72efd6a..dd77b453b8 100644
--- a/tests/qemu-iotests/findtests.py
+++ b/tests/qemu-iotests/findtests.py
@@ -49,7 +49,7 @@ class TestFinder:
 os.path.isfile(f + '.out')]
  
  for t in self.all_tests:

-with open(t) as f:
+with open(t, encoding="utf-8") as f:
  for line in f:
  if line.startswith('# group: '):
  for g in line.split()[2:]:
@@ -57,7 +57,7 @@ class TestFinder:
  break
  
  def add_group_file(self, fname: str) -> None:

-with open(fname) as f:
+with open(fname, encoding="utf-8") as f:
  for line in f:
  line = line.strip()
  
diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py

index 046f9ce38f..a581be6a29 100644
--- a/tests/qemu-iotests/testrunner.py
+++ b/tests/qemu-iotests/testrunner.py
@@ -41,7 +41,8 @@ def silent_unlink(path: Path) -> None:
  
  
  def file_diff(file1: str, file2: str) -> List[str]:

-with open(file1) as f1, open(file2) as f2:
+with open(file1, encoding="utf-8") as f1, \
+ open(file2, encoding="utf-8") as f2:
  # We want to ignore spaces at line ends. There are a lot of mess about
  # it in iotests.
  # TODO: fix all tests to not produce extra spaces, fix all .out files
@@ -81,7 +82,7 @@ class LastElapsedTime(ContextManager['LastElapsedTime']):
  self.cache: Dict[str, Dict[str, Dict[str, float]]]
  
  try:

-with open(cache_file) as f:
+with open(cache_file, encoding="utf-8") as f:
  self.cache = json.load(f)
  except (OSError, ValueError):
  self.cache = {}
@@ -102,7 +103,7 @@ class LastElapsedTime(ContextManager['LastElapsedTime']):
  d.setdefault(self.env.imgproto, {})[self.env.imgfmt] = elapsed
  
  def save(self) -> None:

-with open(self.cache_file, 'w') as f:
+with open(self.cache_file, 'w', encoding="utf-8") as f:
  json.dump(self.cache, f)
  
  def __enter__(self) -> 'LastElapsedTime':

@@ -245,7 +246,7 @@ class TestRunner(ContextManager['TestRunner']):
  if self.env.debug:
  args.append('-d')
  
-with f_test.open() as f:

+with f_test.open(encoding="utf-8") as f:
  try:
  if f.readline() == '#!/usr/bin/env python3':
  args.insert(0, self.env.python)
@@ -256,7 +257,7 @@ class TestRunner(ContextManager['TestRunner']):
  env.update(self.test_run_env)
  
  t0 = time.time()

-with f_bad.open('w') as f:
+with f_bad.open('w', encoding="utf-8") as f:
  proc = subprocess.Popen(args, cwd=str(f_test.parent), env=env,
  stdout=f, stderr=subprocess.STDOUT)
  try:



OK, thanks for handling it!

When will we move to python 3.7?

--
Best regards,
Vladimir

[PULL 53/53] iotests/178: Pass value to invalid option

[Max Reitz]

ccd3b3b8112 has deprecated short-hand boolean options (i.e., options
with values).  All options without values are interpreted as boolean
options, so this includes the invalid option "snapshot.foo" used in
iotest 178.

So after ccd3b3b8112, 178 fails with:

  +qemu-img: warning: short-form boolean option 'snapshot.foo' deprecated
  +Please use snapshot.foo=on instead

Suppress that deprecation warning by passing some value to it (it does
not matter which, because the option is invalid anyway).

Fixes: ccd3b3b8112b670fdccf8a392b8419b173ffccb4
   ("qemu-option: warn for short-form boolean options")
Signed-off-by: Max Reitz 
Message-Id: <20210126123834.115915-1-mre...@redhat.com>
---
 tests/qemu-iotests/178   | 2 +-
 tests/qemu-iotests/178.out.qcow2 | 2 +-
 tests/qemu-iotests/178.out.raw   | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/tests/qemu-iotests/178 b/tests/qemu-iotests/178
index 3b1a7adce4..8df241ead8 100755
--- a/tests/qemu-iotests/178
+++ b/tests/qemu-iotests/178
@@ -57,7 +57,7 @@ $QEMU_IMG measure --image-opts # missing filename
 $QEMU_IMG measure -f qcow2 # missing filename
 $QEMU_IMG measure -l snap1 # missing filename
 $QEMU_IMG measure -o , # invalid option list
-$QEMU_IMG measure -l snapshot.foo # invalid snapshot option
+$QEMU_IMG measure -l snapshot.foo=bar # invalid snapshot option
 $QEMU_IMG measure --output foo # invalid output format
 $QEMU_IMG measure --size -1 # invalid image size
 $QEMU_IMG measure -O foo "$TEST_IMG" # unknown image file format
diff --git a/tests/qemu-iotests/178.out.qcow2 b/tests/qemu-iotests/178.out.qcow2
index c7997760fd..fe193fd5f4 100644
--- a/tests/qemu-iotests/178.out.qcow2
+++ b/tests/qemu-iotests/178.out.qcow2
@@ -11,7 +11,7 @@ qemu-img: --image-opts, -f, and -l require a filename 
argument.
 qemu-img: --image-opts, -f, and -l require a filename argument.
 qemu-img: Invalid option list: ,
 qemu-img: Invalid parameter 'snapshot.foo'
-qemu-img: Failed in parsing snapshot param 'snapshot.foo'
+qemu-img: Failed in parsing snapshot param 'snapshot.foo=bar'
 qemu-img: --output must be used with human or json as argument.
 qemu-img: Invalid image size specified. Must be between 0 and 
9223372036854775807.
 qemu-img: Unknown file format 'foo'
diff --git a/tests/qemu-iotests/178.out.raw b/tests/qemu-iotests/178.out.raw
index 20e17da115..445e460fad 100644
--- a/tests/qemu-iotests/178.out.raw
+++ b/tests/qemu-iotests/178.out.raw
@@ -11,7 +11,7 @@ qemu-img: --image-opts, -f, and -l require a filename 
argument.
 qemu-img: --image-opts, -f, and -l require a filename argument.
 qemu-img: Invalid option list: ,
 qemu-img: Invalid parameter 'snapshot.foo'
-qemu-img: Failed in parsing snapshot param 'snapshot.foo'
+qemu-img: Failed in parsing snapshot param 'snapshot.foo=bar'
 qemu-img: --output must be used with human or json as argument.
 qemu-img: Invalid image size specified. Must be between 0 and 
9223372036854775807.
 qemu-img: Unknown file format 'foo'
-- 
2.29.2

[PULL 50/53] block: report errno when flock fcntl fails

[Max Reitz]

From: David Edmondson 

When a call to fcntl(2) for the purpose of adding file locks fails
with an error other than EAGAIN or EACCES, report the error returned
by fcntl.

EAGAIN or EACCES are elided as they are considered to be common
failures, indicating that a conflicting lock is held by another
process.

No errors are elided when removing file locks.

Signed-off-by: David Edmondson 
Message-Id: <20210113164447.2545785-1-david.edmond...@oracle.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy 
Signed-off-by: Max Reitz 
---
 block/file-posix.c | 38 --
 1 file changed, 28 insertions(+), 10 deletions(-)

diff --git a/block/file-posix.c b/block/file-posix.c
index 00cdaaa2d4..11aafa9d82 100644
--- a/block/file-posix.c
+++ b/block/file-posix.c
@@ -216,6 +216,20 @@ typedef struct RawPosixAIOData {
 static int cdrom_reopen(BlockDriverState *bs);
 #endif
 
+/*
+ * Elide EAGAIN and EACCES details when failing to lock, as this
+ * indicates that the specified file region is already locked by
+ * another process, which is considered a common scenario.
+ */
+#define raw_lock_error_setg_errno(errp, err, fmt, ...)  \
+do {\
+if ((err) == EAGAIN || (err) == EACCES) {   \
+error_setg((errp), (fmt), ## __VA_ARGS__);  \
+} else {\
+error_setg_errno((errp), (err), (fmt), ## __VA_ARGS__); \
+}   \
+} while (0)
+
 #if defined(__NetBSD__)
 static int raw_normalize_devicepath(const char **filename, Error **errp)
 {
@@ -836,7 +850,8 @@ static int raw_apply_lock_bytes(BDRVRawState *s, int fd,
 if ((perm_lock_bits & bit) && !(locked_perm & bit)) {
 ret = qemu_lock_fd(fd, off, 1, false);
 if (ret) {
-error_setg(errp, "Failed to lock byte %d", off);
+raw_lock_error_setg_errno(errp, -ret, "Failed to lock byte %d",
+  off);
 return ret;
 } else if (s) {
 s->locked_perm |= bit;
@@ -844,7 +859,7 @@ static int raw_apply_lock_bytes(BDRVRawState *s, int fd,
 } else if (unlock && (locked_perm & bit) && !(perm_lock_bits & bit)) {
 ret = qemu_unlock_fd(fd, off, 1);
 if (ret) {
-error_setg(errp, "Failed to unlock byte %d", off);
+error_setg_errno(errp, -ret, "Failed to unlock byte %d", off);
 return ret;
 } else if (s) {
 s->locked_perm &= ~bit;
@@ -857,7 +872,8 @@ static int raw_apply_lock_bytes(BDRVRawState *s, int fd,
 if ((shared_perm_lock_bits & bit) && !(locked_shared_perm & bit)) {
 ret = qemu_lock_fd(fd, off, 1, false);
 if (ret) {
-error_setg(errp, "Failed to lock byte %d", off);
+raw_lock_error_setg_errno(errp, -ret, "Failed to lock byte %d",
+  off);
 return ret;
 } else if (s) {
 s->locked_shared_perm |= bit;
@@ -866,7 +882,7 @@ static int raw_apply_lock_bytes(BDRVRawState *s, int fd,
!(shared_perm_lock_bits & bit)) {
 ret = qemu_unlock_fd(fd, off, 1);
 if (ret) {
-error_setg(errp, "Failed to unlock byte %d", off);
+error_setg_errno(errp, -ret, "Failed to unlock byte %d", off);
 return ret;
 } else if (s) {
 s->locked_shared_perm &= ~bit;
@@ -890,9 +906,10 @@ static int raw_check_lock_bytes(int fd, uint64_t perm, 
uint64_t shared_perm,
 ret = qemu_lock_fd_test(fd, off, 1, true);
 if (ret) {
 char *perm_name = bdrv_perm_names(p);
-error_setg(errp,
-   "Failed to get \"%s\" lock",
-   perm_name);
+
+raw_lock_error_setg_errno(errp, -ret,
+  "Failed to get \"%s\" lock",
+  perm_name);
 g_free(perm_name);
 return ret;
 }
@@ -905,9 +922,10 @@ static int raw_check_lock_bytes(int fd, uint64_t perm, 
uint64_t shared_perm,
 ret = qemu_lock_fd_test(fd, off, 1, true);
 if (ret) {
 char *perm_name = bdrv_perm_names(p);
-error_setg(errp,
-   "Failed to get shared \"%s\" lock",
-   perm_name);
+
+raw_lock_error_setg_errno(errp, -ret,
+  "Failed to get shared \"%s\" lock",
+  perm_name);
 g_free(perm_name);
 return ret;
 }
-- 
2.29.2

Re: [PATCH v9 03/11] configure: check for sys/disk.h

[Warner Losh]

On Mon, Jan 25, 2021 at 6:33 PM Joelle van Dyne  wrote:

> Some BSD platforms do not have this header.
>
> Signed-off-by: Joelle van Dyne 
> ---
>  meson.build| 1 +
>  block.c| 2 +-
>  block/file-posix.c | 2 +-
>  3 files changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/meson.build b/meson.build
> index 27110075df..6818d97df5 100644
> --- a/meson.build
> +++ b/meson.build
> @@ -1117,6 +1117,7 @@ config_host_data.set('HAVE_PTY_H',
> cc.has_header('pty.h'))
>  config_host_data.set('HAVE_SYS_IOCCOM_H', cc.has_header('sys/ioccom.h'))
>  config_host_data.set('HAVE_SYS_KCOV_H', cc.has_header('sys/kcov.h'))
>  config_host_data.set('HAVE_HOST_BLOCK_DEVICE', have_host_block_device)
> +config_host_data.set('HAVE_SYS_DISK_H', cc.has_header('sys/disk.h'))
>
>  ignored = ['CONFIG_QEMU_INTERP_PREFIX'] # actually per-target
>  arrays = ['CONFIG_AUDIO_DRIVERS', 'CONFIG_BDRV_RW_WHITELIST',
> 'CONFIG_BDRV_RO_WHITELIST']
> diff --git a/block.c b/block.c
> index 8b9d457546..c4cf391dea 100644
> --- a/block.c
> +++ b/block.c
> @@ -54,7 +54,7 @@
>  #ifdef CONFIG_BSD
>  #include 
>  #include 
> -#ifndef __DragonFly__
> +#if defined(HAVE_SYS_DISK_H)
>  #include 
>  #endif
>  #endif
> diff --git a/block/file-posix.c b/block/file-posix.c
> index 11d2021346..666d3e7504 100644
> --- a/block/file-posix.c
> +++ b/block/file-posix.c
> @@ -2320,7 +2320,7 @@ again:
>  }
>  if (size == 0)
>  #endif
> -#if defined(__APPLE__) && defined(__MACH__)
> +#if defined(HAVE_SYS_DISK_H) && defined(__APPLE__) && defined(__MACH__)
>

Why is this needed? __DragonFly__ doesn't define either __APPLE__ or
__MACH__

Warner


>  {
>  uint64_t sectors = 0;
>  uint32_t sector_size = 0;
> --
> 2.28.0
>
>
>

Re: [PATCH v9 03/11] configure: check for sys/disk.h

[Warner Losh]

On Tue, Jan 26, 2021 at 12:08 AM Philippe Mathieu-Daudé 
wrote:

> On 1/26/21 6:55 AM, Joelle van Dyne wrote:
> > Previously, the only case where sys/disk.h does not exist is on
> > platforms that define __DragonFly__. However, iOS also does not have
> > this header. Previously, I had it as
> >
> > #if defined(__DragonFly__) || defined(CONFIG_IOS)
> >
> > But there was a code review comment that we should use feature flags
> > instead of platform defines. So I added the HAS_SYS_DISK_H flag.
>
> On technical lists, it's best to avoid top-posting, and to
> instead reply inline to make the conversation easier to follow.
>
> >
> > -j
> >
> > On Mon, Jan 25, 2021 at 8:35 PM Warner Losh  wrote:
> >>
> >>
> >>
> >> On Mon, Jan 25, 2021 at 6:33 PM Joelle van Dyne  wrote:
> >>>
> >>> Some BSD platforms do not have this header.
> >>>
> >>> Signed-off-by: Joelle van Dyne 
> >>> ---
> >>>  meson.build| 1 +
> >>>  block.c| 2 +-
> >>>  block/file-posix.c | 2 +-
> >>>  3 files changed, 3 insertions(+), 2 deletions(-)
> >>>
> >>> diff --git a/meson.build b/meson.build
> >>> index 27110075df..6818d97df5 100644
> >>> --- a/meson.build
> >>> +++ b/meson.build
> >>> @@ -1117,6 +1117,7 @@ config_host_data.set('HAVE_PTY_H',
> cc.has_header('pty.h'))
> >>>  config_host_data.set('HAVE_SYS_IOCCOM_H',
> cc.has_header('sys/ioccom.h'))
> >>>  config_host_data.set('HAVE_SYS_KCOV_H', cc.has_header('sys/kcov.h'))
> >>>  config_host_data.set('HAVE_HOST_BLOCK_DEVICE', have_host_block_device)
> >>> +config_host_data.set('HAVE_SYS_DISK_H', cc.has_header('sys/disk.h'))
> >>>
> >>>  ignored = ['CONFIG_QEMU_INTERP_PREFIX'] # actually per-target
> >>>  arrays = ['CONFIG_AUDIO_DRIVERS', 'CONFIG_BDRV_RW_WHITELIST',
> 'CONFIG_BDRV_RO_WHITELIST']
> >>> diff --git a/block.c b/block.c
> >>> index 8b9d457546..c4cf391dea 100644
> >>> --- a/block.c
> >>> +++ b/block.c
> >>> @@ -54,7 +54,7 @@
> >>>  #ifdef CONFIG_BSD
> >>>  #include 
> >>>  #include 
> >>> -#ifndef __DragonFly__
> >>> +#if defined(HAVE_SYS_DISK_H)
> >>>  #include 
> >>>  #endif
> >>>  #endif
> >>> diff --git a/block/file-posix.c b/block/file-posix.c
> >>> index 11d2021346..666d3e7504 100644
> >>> --- a/block/file-posix.c
> >>> +++ b/block/file-posix.c
> >>> @@ -2320,7 +2320,7 @@ again:
> >>>  }
> >>>  if (size == 0)
> >>>  #endif
> >>> -#if defined(__APPLE__) && defined(__MACH__)
> >>> +#if defined(HAVE_SYS_DISK_H) && defined(__APPLE__) &&
> defined(__MACH__)
> >>
> >>
> >> Why is this needed? __DragonFly__ doesn't define either __APPLE__ or
> __MACH__
>
> Hmm we could also add:
>
>   config_host_data.set('HAVE_DKIOCGETBLOCKCOUNT', cc.compiles(...))
>
> Then this block would be easier to read:
>
>   #if defined(HAVE_DKIOCGETBLOCKCOUNT)
>   ...
>
> (Maybe this is what Warner meant?)
>

Close. I'd test it more directly since DKIOCGETBLOCKCOUNT is already a
#define, and is unlikely to change...

When I saw Joelle's response, I realized I'd been needlessly cryptic in my
comments, so posted what I had in mind for cleanup. I'm not sure if the
norms of qemu code reviews would say my suggestion was too big to be in
scope, or not.

Warner

>>
> >> Warner
> >>
> >>>
> >>>  {
> >>>  uint64_t sectors = 0;
> >>>  uint32_t sector_size = 0;
> >>> --
> >>> 2.28.0
> >>>
> >>>
> >
>
>

[PULL 51/53] iotests: Add test for the regression fixed in c8bf9a9169

[Max Reitz]

From: Alberto Garcia 

Signed-off-by: Alberto Garcia 
Suggested-by: Maxim Levitsky 
Reviewed-by: Maxim Levitsky 
Message-Id: <20210112170540.2912-1-be...@igalia.com>
[mreitz: Add "# group:" line]
Signed-off-by: Max Reitz 
---
 tests/qemu-iotests/313 | 104 +
 tests/qemu-iotests/313.out |  29 +++
 tests/qemu-iotests/group   |   1 +
 3 files changed, 134 insertions(+)
 create mode 100755 tests/qemu-iotests/313
 create mode 100644 tests/qemu-iotests/313.out

diff --git a/tests/qemu-iotests/313 b/tests/qemu-iotests/313
new file mode 100755
index 00..a75655b7ef
--- /dev/null
+++ b/tests/qemu-iotests/313
@@ -0,0 +1,104 @@
+#!/usr/bin/env bash
+# group: rw auto quick
+#
+# Test for the regression fixed in commit c8bf9a9169
+#
+# Copyright (C) 2020 Igalia, S.L.
+# Author: Alberto Garcia 
+# Based on a test case by Maxim Levitsky 
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see .
+#
+
+# creator
+owner=be...@igalia.com
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+status=1# failure is the default!
+
+_cleanup()
+{
+_cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+_unsupported_imgopts cluster_size refcount_bits extended_l2 compat=0.10 
data_file
+
+# The cluster size must be at least the granularity of the mirror job (4KB)
+# Note that larger cluster sizes will produce very large images (several GBs)
+cluster_size=4096
+refcount_bits=64 # Make it equal to the L2 entry size for convenience
+options="cluster_size=${cluster_size},refcount_bits=${refcount_bits}"
+
+# Number of refcount entries per refcount blocks
+ref_entries=$(( ${cluster_size} * 8 / ${refcount_bits} ))
+
+# Number of data clusters needed to fill a refcount block
+# Equals ${ref_entries} minus two (one L2 table and one refcount block)
+data_clusters_per_refblock=$(( ${ref_entries} - 2 ))
+
+# Number of entries in the refcount cache
+ref_blocks=4
+
+# Write enough data clusters to fill the refcount cache and allocate
+# one more refcount block.
+# Subtract 3 clusters from the total: qcow2 header, refcount table, L1 table
+total_data_clusters=$(( ${data_clusters_per_refblock} * ${ref_blocks} + 1 - 3 
))
+
+# Total size to write in bytes
+total_size=$(( ${total_data_clusters} * ${cluster_size} ))
+
+echo
+echo '### Create the image'
+echo
+TEST_IMG_FILE=$TEST_IMG.base _make_test_img -o $options $total_size | 
_filter_img_create_size
+
+echo
+echo '### Write data to allocate more refcount blocks than the cache can hold'
+echo
+$QEMU_IO -c "write -P 1 0 $total_size" $TEST_IMG.base | _filter_qemu_io
+
+echo
+echo '### Create an overlay'
+echo
+_make_test_img -F $IMGFMT -b $TEST_IMG.base -o $options | 
_filter_img_create_size
+
+echo
+echo '### Fill the overlay with zeroes'
+echo
+$QEMU_IO -c "write -z 0 $total_size" $TEST_IMG | _filter_qemu_io
+
+echo
+echo '### Commit changes to the base image'
+echo
+$QEMU_IMG commit $TEST_IMG
+
+echo
+echo '### Check the base image'
+echo
+$QEMU_IMG check $TEST_IMG.base
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/313.out b/tests/qemu-iotests/313.out
new file mode 100644
index 00..adb9f7bd95
--- /dev/null
+++ b/tests/qemu-iotests/313.out
@@ -0,0 +1,29 @@
+QA output created by 313
+
+### Create the image
+
+Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=SIZE
+
+### Write data to allocate more refcount blocks than the cache can hold
+
+wrote 8347648/8347648 bytes at offset 0
+7.961 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+### Create an overlay
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=SIZE 
backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=IMGFMT
+
+### Fill the overlay with zeroes
+
+wrote 8347648/8347648 bytes at offset 0
+7.961 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+### Commit changes to the base image
+
+Image committed.
+
+### Check the base image
+
+No errors were found on the image.
+Image end offset: 8396800
+*** done
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index d4a3e36a9a..a0d0bf1688 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -320,3 +320,4 @@
 309 rw auto quick
 310 rw quick
 312 rw quick
+313 rw auto quick
-- 
2.29.2

[PULL 45/53] block/block-copy: drop unused block_copy_set_progress_callback()

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy 

Drop unused code.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20210116214705.822267-20-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 include/block/block-copy.h |  6 --
 block/block-copy.c | 15 ---
 2 files changed, 21 deletions(-)

diff --git a/include/block/block-copy.h b/include/block/block-copy.h
index 7821850f88..1cbea0b79b 100644
--- a/include/block/block-copy.h
+++ b/include/block/block-copy.h
@@ -18,7 +18,6 @@
 #include "block/block.h"
 #include "qemu/co-shared-resource.h"
 
-typedef void (*ProgressBytesCallbackFunc)(int64_t bytes, void *opaque);
 typedef void (*BlockCopyAsyncCallbackFunc)(void *opaque);
 typedef struct BlockCopyState BlockCopyState;
 typedef struct BlockCopyCallState BlockCopyCallState;
@@ -28,11 +27,6 @@ BlockCopyState *block_copy_state_new(BdrvChild *source, 
BdrvChild *target,
  BdrvRequestFlags write_flags,
  Error **errp);
 
-void block_copy_set_progress_callback(
-BlockCopyState *s,
-ProgressBytesCallbackFunc progress_bytes_callback,
-void *progress_opaque);
-
 void block_copy_set_progress_meter(BlockCopyState *s, ProgressMeter *pm);
 
 void block_copy_state_free(BlockCopyState *s);
diff --git a/block/block-copy.c b/block/block-copy.c
index 61d82d9a1c..2ea8b28684 100644
--- a/block/block-copy.c
+++ b/block/block-copy.c
@@ -110,9 +110,6 @@ typedef struct BlockCopyState {
 bool skip_unallocated;
 
 ProgressMeter *progress;
-/* progress_bytes_callback: called when some copying progress is done. */
-ProgressBytesCallbackFunc progress_bytes_callback;
-void *progress_opaque;
 
 SharedResource *mem;
 
@@ -298,15 +295,6 @@ BlockCopyState *block_copy_state_new(BdrvChild *source, 
BdrvChild *target,
 return s;
 }
 
-void block_copy_set_progress_callback(
-BlockCopyState *s,
-ProgressBytesCallbackFunc progress_bytes_callback,
-void *progress_opaque)
-{
-s->progress_bytes_callback = progress_bytes_callback;
-s->progress_opaque = progress_opaque;
-}
-
 void block_copy_set_progress_meter(BlockCopyState *s, ProgressMeter *pm)
 {
 s->progress = pm;
@@ -454,9 +442,6 @@ static coroutine_fn int block_copy_task_entry(AioTask *task)
 t->call_state->error_is_read = error_is_read;
 } else {
 progress_work_done(t->s->progress, t->bytes);
-if (t->s->progress_bytes_callback) {
-t->s->progress_bytes_callback(t->bytes, t->s->progress_opaque);
-}
 }
 co_put_to_shres(t->s->mem, t->bytes);
 block_copy_task_end(t, ret);
-- 
2.29.2

Re: [PATCH v9 03/11] configure: check for sys/disk.h

[Warner Losh]

On Mon, Jan 25, 2021 at 10:55 PM Joelle van Dyne  wrote:

> Previously, the only case where sys/disk.h does not exist is on
> platforms that define __DragonFly__. However, iOS also does not have
> this header. Previously, I had it as
>
> #if defined(__DragonFly__) || defined(CONFIG_IOS)
>
> But there was a code review comment that we should use feature flags
> instead of platform defines. So I added the HAS_SYS_DISK_H flag.
>

Right. I like that the #include is now protected like that. However,
sys/disk.h never was standardized and varies considerably among the systems
that it exists on.


> -j
>
> On Mon, Jan 25, 2021 at 8:35 PM Warner Losh  wrote:
> >
> >
> >
> > On Mon, Jan 25, 2021 at 6:33 PM Joelle van Dyne  wrote:
> >>
> >> Some BSD platforms do not have this header.
> >>
> >> Signed-off-by: Joelle van Dyne 
> >> ---
> >>  meson.build| 1 +
> >>  block.c| 2 +-
> >>  block/file-posix.c | 2 +-
> >>  3 files changed, 3 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/meson.build b/meson.build
> >> index 27110075df..6818d97df5 100644
> >> --- a/meson.build
> >> +++ b/meson.build
> >> @@ -1117,6 +1117,7 @@ config_host_data.set('HAVE_PTY_H',
> cc.has_header('pty.h'))
> >>  config_host_data.set('HAVE_SYS_IOCCOM_H',
> cc.has_header('sys/ioccom.h'))
> >>  config_host_data.set('HAVE_SYS_KCOV_H', cc.has_header('sys/kcov.h'))
> >>  config_host_data.set('HAVE_HOST_BLOCK_DEVICE', have_host_block_device)
> >> +config_host_data.set('HAVE_SYS_DISK_H', cc.has_header('sys/disk.h'))
> >>
> >>  ignored = ['CONFIG_QEMU_INTERP_PREFIX'] # actually per-target
> >>  arrays = ['CONFIG_AUDIO_DRIVERS', 'CONFIG_BDRV_RW_WHITELIST',
> 'CONFIG_BDRV_RO_WHITELIST']
> >> diff --git a/block.c b/block.c
> >> index 8b9d457546..c4cf391dea 100644
> >> --- a/block.c
> >> +++ b/block.c
> >> @@ -54,7 +54,7 @@
> >>  #ifdef CONFIG_BSD
> >>  #include 
> >>  #include 
> >> -#ifndef __DragonFly__
> >> +#if defined(HAVE_SYS_DISK_H)
> >>  #include 
> >>  #endif
> >>  #endif
> >> diff --git a/block/file-posix.c b/block/file-posix.c
> >> index 11d2021346..666d3e7504 100644
> >> --- a/block/file-posix.c
> >> +++ b/block/file-posix.c
> >> @@ -2320,7 +2320,7 @@ again:
> >>  }
> >>  if (size == 0)
> >>  #endif
> >> -#if defined(__APPLE__) && defined(__MACH__)
> >> +#if defined(HAVE_SYS_DISK_H) && defined(__APPLE__) && defined(__MACH__)
> >
> >
> > Why is this needed? __DragonFly__ doesn't define either __APPLE__ or
> __MACH_
>

Which is why I asked this question...

Let me ask it another way. Why not base this on the
ioctl  DKIOCGETBLOCKCOUNT like the rest of this function? It's simple and
on platforms that don't have that ioctl, it won't be used.  I don't even
know how to read the proposed change logically. If IOS doesn't have this
interface, then you'll need another #else  to work reliably
anyway, since the seek trick that's used there may or may not work. However
that starts to get kinda nested and twisty.  So maybe something more like
the following would make it clearer... though that might be beyond the
scope of what you're trying to do.

diff --git a/block/file-posix.c b/block/file-posix.c
index 00cdaaa2d4..704ded68b0 100644
--- a/block/file-posix.c
+++ b/block/file-posix.c
@@ -2295,8 +2295,10 @@ static int64_t raw_getlength(BlockDriverState *bs)
 again:
 #endif
 if (!fstat(fd, ) && (S_IFCHR & sb.st_mode)) {
+size = 0;
 #ifdef DIOCGMEDIASIZE
 if (ioctl(fd, DIOCGMEDIASIZE, (off_t *)))
+size = 0;
 #elif defined(DIOCGPART)
 {
 struct partinfo pi;
@@ -2305,9 +2307,7 @@ again:
 else
 size = 0;
 }
-if (size == 0)
-#endif
-#if defined(__APPLE__) && defined(__MACH__)
+#elif defined(DKIOCGETBLOCKCOUNT) && defined(DKIOCGETBLOCKSIZE)
 {
 uint64_t sectors = 0;
 uint32_t sector_size = 0;
@@ -2315,19 +2315,15 @@ again:
 if (ioctl(fd, DKIOCGETBLOCKCOUNT, ) == 0
&& ioctl(fd, DKIOCGETBLOCKSIZE, _size) == 0) {
 size = sectors * sector_size;
-} else {
-size = lseek(fd, 0LL, SEEK_END);
-if (size < 0) {
-return -errno;
-}
 }
 }
-#else
-size = lseek(fd, 0LL, SEEK_END);
+#endif
+if (size == 0) {
+size = lseek(fd, 0LL, SEEK_END);
+}
 if (size < 0) {
 return -errno;
 }
-#endif
 #if defined(__FreeBSD__) || defined(__FreeBSD_kernel__)
 switch(s->type) {
 case FTYPE_CD:

Warner

>
> >>
> >>  {
> >>  uint64_t sectors = 0;
> >>  uint32_t sector_size = 0;
> >> --
> >> 2.28.0
> >>
> >>
>

[PULL 52/53] iotests/118: Drop 'change' test

[Max Reitz]

Commit 0afec75734331 removed the 'change' QMP command, so we can no
longer test it in 118.

Fixes: 0afec75734331a0b52fa3aa4235220eda8c7846f
   ('qmp: remove deprecated "change" command')
Signed-off-by: Max Reitz 
Message-Id: <20210126104833.57026-1-mre...@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé 
---
 tests/qemu-iotests/118 | 20 +---
 tests/qemu-iotests/118.out |  4 ++--
 2 files changed, 3 insertions(+), 21 deletions(-)

diff --git a/tests/qemu-iotests/118 b/tests/qemu-iotests/118
index 1a2e219057..cae52ffa5e 100755
--- a/tests/qemu-iotests/118
+++ b/tests/qemu-iotests/118
@@ -1,8 +1,7 @@
 #!/usr/bin/env python3
 # group: rw
 #
-# Test case for the QMP 'change' command and all other associated
-# commands
+# Test case for media change monitor commands
 #
 # Copyright (C) 2015 Red Hat, Inc.
 #
@@ -74,23 +73,6 @@ class ChangeBaseClass(iotests.QMPTestCase):
 
 class GeneralChangeTestsBaseClass(ChangeBaseClass):
 
-def test_change(self):
-# 'change' requires a drive name, so skip the test for blockdev
-if not self.use_drive:
-return
-
-result = self.vm.qmp('change', device='drive0', target=new_img,
-   arg=iotests.imgfmt)
-self.assert_qmp(result, 'return', {})
-
-self.wait_for_open()
-self.wait_for_close()
-
-result = self.vm.qmp('query-block')
-if self.has_real_tray:
-self.assert_qmp(result, 'return[0]/tray_open', False)
-self.assert_qmp(result, 'return[0]/inserted/image/filename', new_img)
-
 def test_blockdev_change_medium(self):
 result = self.vm.qmp('blockdev-change-medium',
  id=self.device_name, filename=new_img,
diff --git a/tests/qemu-iotests/118.out b/tests/qemu-iotests/118.out
index bf5bfd5aca..0a70391105 100644
--- a/tests/qemu-iotests/118.out
+++ b/tests/qemu-iotests/118.out
@@ -1,5 +1,5 @@
-...
+...
 --
-Ran 167 tests
+Ran 155 tests
 
 OK
-- 
2.29.2

[PULL 49/53] simplebench: add bench-backup.py

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy 

Add script to benchmark new backup architecture.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Message-Id: <20210116214705.822267-24-vsement...@virtuozzo.com>
[mreitz: s/not unsupported/not supported/]
Signed-off-by: Max Reitz 
---
 scripts/simplebench/bench-backup.py | 167 
 1 file changed, 167 insertions(+)
 create mode 100755 scripts/simplebench/bench-backup.py

diff --git a/scripts/simplebench/bench-backup.py 
b/scripts/simplebench/bench-backup.py
new file mode 100755
index 00..33a1ecfefa
--- /dev/null
+++ b/scripts/simplebench/bench-backup.py
@@ -0,0 +1,167 @@
+#!/usr/bin/env python3
+#
+# Bench backup block-job
+#
+# Copyright (c) 2020 Virtuozzo International GmbH.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see .
+#
+
+import argparse
+import json
+
+import simplebench
+from results_to_text import results_to_text
+from bench_block_job import bench_block_copy, drv_file, drv_nbd
+
+
+def bench_func(env, case):
+""" Handle one "cell" of benchmarking table. """
+cmd_options = env['cmd-options'] if 'cmd-options' in env else {}
+return bench_block_copy(env['qemu-binary'], env['cmd'],
+cmd_options,
+case['source'], case['target'])
+
+
+def bench(args):
+test_cases = []
+
+sources = {}
+targets = {}
+for d in args.dir:
+label, path = d.split(':')  # paths with colon not supported
+sources[label] = drv_file(path + '/test-source')
+targets[label] = drv_file(path + '/test-target')
+
+if args.nbd:
+nbd = args.nbd.split(':')
+host = nbd[0]
+port = '10809' if len(nbd) == 1 else nbd[1]
+drv = drv_nbd(host, port)
+sources['nbd'] = drv
+targets['nbd'] = drv
+
+for t in args.test:
+src, dst = t.split(':')
+
+test_cases.append({
+'id': t,
+'source': sources[src],
+'target': targets[dst]
+})
+
+binaries = []  # list of (, , [])
+for i, q in enumerate(args.env):
+name_path = q.split(':')
+if len(name_path) == 1:
+label = f'q{i}'
+path_opts = name_path[0].split(',')
+else:
+assert len(name_path) == 2  # paths with colon not supported
+label = name_path[0]
+path_opts = name_path[1].split(',')
+
+binaries.append((label, path_opts[0], path_opts[1:]))
+
+test_envs = []
+
+bin_paths = {}
+for i, q in enumerate(args.env):
+opts = q.split(',')
+label_path = opts[0]
+opts = opts[1:]
+
+if ':' in label_path:
+# path with colon inside is not supported
+label, path = label_path.split(':')
+bin_paths[label] = path
+elif label_path in bin_paths:
+label = label_path
+path = bin_paths[label]
+else:
+path = label_path
+label = f'q{i}'
+bin_paths[label] = path
+
+x_perf = {}
+is_mirror = False
+for opt in opts:
+if opt == 'mirror':
+is_mirror = True
+elif opt == 'copy-range=on':
+x_perf['use-copy-range'] = True
+elif opt == 'copy-range=off':
+x_perf['use-copy-range'] = False
+elif opt.startswith('max-workers='):
+x_perf['max-workers'] = int(opt.split('=')[1])
+
+if is_mirror:
+assert not x_perf
+test_envs.append({
+'id': f'mirror({label})',
+'cmd': 'blockdev-mirror',
+'qemu-binary': path
+})
+else:
+test_envs.append({
+'id': f'backup({label})\n' + '\n'.join(opts),
+'cmd': 'blockdev-backup',
+'cmd-options': {'x-perf': x_perf} if x_perf else {},
+'qemu-binary': path
+})
+
+result = simplebench.bench(bench_func, test_envs, test_cases, count=3)
+with open('results.json', 'w') as f:
+json.dump(result, f, indent=4)
+print(results_to_text(result))
+
+
+class ExtendAction(argparse.Action):
+def __call__(self, parser, namespace, values, option_string=None):
+items = getattr(namespace, self.dest) or []
+items.extend(values)
+

[PULL 46/53] block/block-copy: drop unused argument of block_copy()

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy 

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20210116214705.822267-21-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 include/block/block-copy.h |  2 +-
 block/backup-top.c |  2 +-
 block/block-copy.c | 10 ++
 3 files changed, 4 insertions(+), 10 deletions(-)

diff --git a/include/block/block-copy.h b/include/block/block-copy.h
index 1cbea0b79b..338f2ea7fd 100644
--- a/include/block/block-copy.h
+++ b/include/block/block-copy.h
@@ -35,7 +35,7 @@ int64_t block_copy_reset_unallocated(BlockCopyState *s,
  int64_t offset, int64_t *count);
 
 int coroutine_fn block_copy(BlockCopyState *s, int64_t offset, int64_t bytes,
-bool ignore_ratelimit, bool *error_is_read);
+bool ignore_ratelimit);
 
 /*
  * Run block-copy in a coroutine, create corresponding BlockCopyCallState
diff --git a/block/backup-top.c b/block/backup-top.c
index 779956ddc2..6e7e7bf340 100644
--- a/block/backup-top.c
+++ b/block/backup-top.c
@@ -61,7 +61,7 @@ static coroutine_fn int backup_top_cbw(BlockDriverState *bs, 
uint64_t offset,
 off = QEMU_ALIGN_DOWN(offset, s->cluster_size);
 end = QEMU_ALIGN_UP(offset + bytes, s->cluster_size);
 
-return block_copy(s->bcs, off, end - off, true, NULL);
+return block_copy(s->bcs, off, end - off, true);
 }
 
 static int coroutine_fn backup_top_co_pdiscard(BlockDriverState *bs,
diff --git a/block/block-copy.c b/block/block-copy.c
index 2ea8b28684..39ae481c8b 100644
--- a/block/block-copy.c
+++ b/block/block-copy.c
@@ -723,7 +723,7 @@ static int coroutine_fn 
block_copy_common(BlockCopyCallState *call_state)
 }
 
 int coroutine_fn block_copy(BlockCopyState *s, int64_t start, int64_t bytes,
-bool ignore_ratelimit, bool *error_is_read)
+bool ignore_ratelimit)
 {
 BlockCopyCallState call_state = {
 .s = s,
@@ -733,13 +733,7 @@ int coroutine_fn block_copy(BlockCopyState *s, int64_t 
start, int64_t bytes,
 .max_workers = BLOCK_COPY_MAX_WORKERS,
 };
 
-int ret = block_copy_common(_state);
-
-if (error_is_read && ret < 0) {
-*error_is_read = call_state.error_is_read;
-}
-
-return ret;
+return block_copy_common(_state);
 }
 
 static void coroutine_fn block_copy_async_co_entry(void *opaque)
-- 
2.29.2

[PULL 37/53] iotests/129: Limit backup's max-chunk/max-workers

[Max Reitz]

Right now, this does not change anything, because backup ignores
max-chunk and max-workers.  However, as soon as backup is switched over
to block-copy for the background copying process, we will need it to
keep 129 passing.

Signed-off-by: Max Reitz 
Message-Id: <20210120102043.28346-1-mre...@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy 
---
 tests/qemu-iotests/129 | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/129 b/tests/qemu-iotests/129
index 2f7b28d4a0..5251e2669e 100755
--- a/tests/qemu-iotests/129
+++ b/tests/qemu-iotests/129
@@ -71,9 +71,14 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
   sync="full", buf_size=65536)
 
 def test_drive_backup(self):
+# Limit max-chunk and max-workers so that block-copy will not
+# launch so many workers working on so much data each that
+# stop's bdrv_drain_all() would finish the job
 self.do_test_stop("drive-backup", device="drive0",
   target=self.target_img, format=iotests.imgfmt,
-  sync="full")
+  sync="full",
+  x_perf={ 'max-chunk': 65536,
+   'max-workers': 8 })
 
 def test_block_commit(self):
 # Add overlay above the source node so that we actually use a
-- 
2.29.2

[PULL 48/53] simplebench: bench_block_job: add cmd_options argument

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy 

Add argument to allow additional block-job options.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20210116214705.822267-23-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 scripts/simplebench/bench-example.py   |  2 +-
 scripts/simplebench/bench_block_job.py | 11 +++
 2 files changed, 8 insertions(+), 5 deletions(-)

diff --git a/scripts/simplebench/bench-example.py 
b/scripts/simplebench/bench-example.py
index d9c7f7bc17..4864435f39 100644
--- a/scripts/simplebench/bench-example.py
+++ b/scripts/simplebench/bench-example.py
@@ -25,7 +25,7 @@ from bench_block_job import bench_block_copy, drv_file, 
drv_nbd
 
 def bench_func(env, case):
 """ Handle one "cell" of benchmarking table. """
-return bench_block_copy(env['qemu_binary'], env['cmd'],
+return bench_block_copy(env['qemu_binary'], env['cmd'], {}
 case['source'], case['target'])
 
 
diff --git a/scripts/simplebench/bench_block_job.py 
b/scripts/simplebench/bench_block_job.py
index a0dda1dc4e..7332845c1c 100755
--- a/scripts/simplebench/bench_block_job.py
+++ b/scripts/simplebench/bench_block_job.py
@@ -78,16 +78,19 @@ def bench_block_job(cmd, cmd_args, qemu_args):
 
 
 # Bench backup or mirror
-def bench_block_copy(qemu_binary, cmd, source, target):
+def bench_block_copy(qemu_binary, cmd, cmd_options, source, target):
 """Helper to run bench_block_job() for mirror or backup"""
 assert cmd in ('blockdev-backup', 'blockdev-mirror')
 
 source['node-name'] = 'source'
 target['node-name'] = 'target'
 
-return bench_block_job(cmd,
-   {'job-id': 'job0', 'device': 'source',
-'target': 'target', 'sync': 'full'},
+cmd_options['job-id'] = 'job0'
+cmd_options['device'] = 'source'
+cmd_options['target'] = 'target'
+cmd_options['sync'] = 'full'
+
+return bench_block_job(cmd, cmd_options,
[qemu_binary,
 '-blockdev', json.dumps(source),
 '-blockdev', json.dumps(target)])
-- 
2.29.2

[PULL 42/53] block/backup: drop extra gotos from backup_run()

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy 

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20210116214705.822267-17-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 block/backup.c | 12 +---
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/block/backup.c b/block/backup.c
index 5522c0f3fe..466608ee55 100644
--- a/block/backup.c
+++ b/block/backup.c
@@ -236,7 +236,7 @@ static void backup_init_bcs_bitmap(BackupBlockJob *job)
 static int coroutine_fn backup_run(Job *job, Error **errp)
 {
 BackupBlockJob *s = container_of(job, BackupBlockJob, common.job);
-int ret = 0;
+int ret;
 
 backup_init_bcs_bitmap(s);
 
@@ -246,13 +246,12 @@ static int coroutine_fn backup_run(Job *job, Error **errp)
 
 for (offset = 0; offset < s->len; ) {
 if (yield_and_check(s)) {
-ret = -ECANCELED;
-goto out;
+return -ECANCELED;
 }
 
 ret = block_copy_reset_unallocated(s->bcs, offset, );
 if (ret < 0) {
-goto out;
+return ret;
 }
 
 offset += count;
@@ -273,11 +272,10 @@ static int coroutine_fn backup_run(Job *job, Error **errp)
 job_yield(job);
 }
 } else {
-ret = backup_loop(s);
+return backup_loop(s);
 }
 
- out:
-return ret;
+return 0;
 }
 
 static const BlockJobDriver backup_job_driver = {
-- 
2.29.2

[PULL 35/53] qapi: backup: add max-chunk and max-workers to x-perf struct

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy 

Add new parameters to configure future backup features. The patch
doesn't introduce aio backup requests (so we actually have only one
worker) neither requests larger than one cluster. Still, formally we
satisfy these maximums anyway, so add the parameters now, to facilitate
further patch which will really change backup job behavior.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20210116214705.822267-11-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 qapi/block-core.json | 13 -
 block/backup.c   | 28 +++-
 block/replication.c  |  2 +-
 blockdev.c   |  8 +++-
 4 files changed, 43 insertions(+), 8 deletions(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index 83f661d7f6..abcd41ed63 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -1379,10 +1379,21 @@
 #
 # @use-copy-range: Use copy offloading. Default true.
 #
+# @max-workers: Maximum number of parallel requests for the sustained 
background
+#   copying process. Doesn't influence copy-before-write 
operations.
+#   Default 64.
+#
+# @max-chunk: Maximum request length for the sustained background copying
+# process. Doesn't influence copy-before-write operations.
+# 0 means unlimited. If max-chunk is non-zero then it should not be
+# less than job cluster size which is calculated as maximum of
+# target image cluster size and 64k. Default 0.
+#
 # Since: 6.0
 ##
 { 'struct': 'BackupPerf',
-  'data': { '*use-copy-range': 'bool' }}
+  'data': { '*use-copy-range': 'bool',
+'*max-workers': 'int', '*max-chunk': 'int64' } }
 
 ##
 # @BackupCommon:
diff --git a/block/backup.c b/block/backup.c
index 09ff5a92ef..5522c0f3fe 100644
--- a/block/backup.c
+++ b/block/backup.c
@@ -388,6 +388,29 @@ BlockJob *backup_job_create(const char *job_id, 
BlockDriverState *bs,
 return NULL;
 }
 
+cluster_size = backup_calculate_cluster_size(target, errp);
+if (cluster_size < 0) {
+goto error;
+}
+
+if (perf->max_workers < 1) {
+error_setg(errp, "max-workers must be greater than zero");
+return NULL;
+}
+
+if (perf->max_chunk < 0) {
+error_setg(errp, "max-chunk must be zero (which means no limit) or "
+   "positive");
+return NULL;
+}
+
+if (perf->max_chunk && perf->max_chunk < cluster_size) {
+error_setg(errp, "Required max-chunk (%" PRIi64 ") is less than backup 
"
+   "cluster size (%" PRIi64 ")", perf->max_chunk, 
cluster_size);
+return NULL;
+}
+
+
 if (sync_bitmap) {
 /* If we need to write to this bitmap, check that we can: */
 if (bitmap_mode != BITMAP_SYNC_MODE_NEVER &&
@@ -420,11 +443,6 @@ BlockJob *backup_job_create(const char *job_id, 
BlockDriverState *bs,
 goto error;
 }
 
-cluster_size = backup_calculate_cluster_size(target, errp);
-if (cluster_size < 0) {
-goto error;
-}
-
 /*
  * If source is in backing chain of target assume that target is going to 
be
  * used for "image fleecing", i.e. it should represent a kind of snapshot 
of
diff --git a/block/replication.c b/block/replication.c
index 22ffc811ee..97be7ef4de 100644
--- a/block/replication.c
+++ b/block/replication.c
@@ -454,7 +454,7 @@ static void replication_start(ReplicationState *rs, 
ReplicationMode mode,
 int64_t active_length, hidden_length, disk_length;
 AioContext *aio_context;
 Error *local_err = NULL;
-BackupPerf perf = { .use_copy_range = true };
+BackupPerf perf = { .use_copy_range = true, .max_workers = 1 };
 
 aio_context = bdrv_get_aio_context(bs);
 aio_context_acquire(aio_context);
diff --git a/blockdev.c b/blockdev.c
index fc88dc03e1..25aaacf253 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -2829,7 +2829,7 @@ static BlockJob *do_backup_common(BackupCommon *backup,
 {
 BlockJob *job = NULL;
 BdrvDirtyBitmap *bmap = NULL;
-BackupPerf perf = { .use_copy_range = true };
+BackupPerf perf = { .use_copy_range = true, .max_workers = 64 };
 int job_flags = JOB_DEFAULT;
 
 if (!backup->has_speed) {
@@ -2858,6 +2858,12 @@ static BlockJob *do_backup_common(BackupCommon *backup,
 if (backup->x_perf->has_use_copy_range) {
 perf.use_copy_range = backup->x_perf->use_copy_range;
 }
+if (backup->x_perf->has_max_workers) {
+perf.max_workers = backup->x_perf->max_workers;
+}
+if (backup->x_perf->has_max_chunk) {
+perf.max_chunk = backup->x_perf->max_chunk;
+}
 }
 
 if ((backup->sync == MIRROR_SYNC_MODE_BITMAP) ||
-- 
2.29.2

[PULL 47/53] simplebench/bench_block_job: use correct shebang line with python3

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy 

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20210116214705.822267-22-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 scripts/simplebench/bench_block_job.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/simplebench/bench_block_job.py 
b/scripts/simplebench/bench_block_job.py
index 9808d696cf..a0dda1dc4e 100755
--- a/scripts/simplebench/bench_block_job.py
+++ b/scripts/simplebench/bench_block_job.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 #
 # Benchmark block jobs
 #
-- 
2.29.2

[PULL 44/53] qapi: backup: disable copy_range by default

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy 

Further commit will add a benchmark
(scripts/simplebench/bench-backup.py), which will show that backup
works better with async parallel requests (previous commit) and
disabled copy_range. So, let's disable copy_range by default.

Note: the option was added several commits ago with default to true,
to follow old behavior (the feature was enabled unconditionally), and
only now we are going to change the default behavior.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20210116214705.822267-19-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 qapi/block-core.json | 2 +-
 blockdev.c   | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index abcd41ed63..9f555d5c1d 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -1377,7 +1377,7 @@
 # Optional parameters for backup. These parameters don't affect
 # functionality, but may significantly affect performance.
 #
-# @use-copy-range: Use copy offloading. Default true.
+# @use-copy-range: Use copy offloading. Default false.
 #
 # @max-workers: Maximum number of parallel requests for the sustained 
background
 #   copying process. Doesn't influence copy-before-write 
operations.
diff --git a/blockdev.c b/blockdev.c
index 25aaacf253..93417f6302 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -2829,7 +2829,7 @@ static BlockJob *do_backup_common(BackupCommon *backup,
 {
 BlockJob *job = NULL;
 BdrvDirtyBitmap *bmap = NULL;
-BackupPerf perf = { .use_copy_range = true, .max_workers = 64 };
+BackupPerf perf = { .max_workers = 64 };
 int job_flags = JOB_DEFAULT;
 
 if (!backup->has_speed) {
-- 
2.29.2

[PULL 39/53] iotests: 219: prepare for backup over block-copy

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy 

The further change of moving backup to be a one block-copy call will
make copying chunk-size and cluster-size two separate things. So, even
with 64k cluster sized qcow2 image, default chunk would be 1M.
Test 219 depends on specified chunk-size. Update it for explicit
chunk-size for backup as for mirror.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20210116214705.822267-14-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 tests/qemu-iotests/219 | 13 +++--
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/tests/qemu-iotests/219 b/tests/qemu-iotests/219
index 16c3ca7fff..d1757e9e6f 100755
--- a/tests/qemu-iotests/219
+++ b/tests/qemu-iotests/219
@@ -204,13 +204,13 @@ with iotests.FilePath('disk.img') as disk_path, \
 # but related to this also automatic state transitions like job
 # completion), but still get pause points often enough to avoid making this
 # test very slow, it's important to have the right ratio between speed and
-# buf_size.
+# copy-chunk-size.
 #
-# For backup, buf_size is hard-coded to the source image cluster size 
(64k),
-# so we'll pick the same for mirror. The slice time, i.e. the granularity
-# of the rate limiting is 100ms. With a speed of 256k per second, we can
-# get four pause points per second. This gives us 250ms per iteration,
-# which should be enough to stay deterministic.
+# Chose 64k copy-chunk-size both for mirror (by buf_size) and backup (by
+# x-max-chunk). The slice time, i.e. the granularity of the rate limiting
+# is 100ms. With a speed of 256k per second, we can get four pause points
+# per second. This gives us 250ms per iteration, which should be enough to
+# stay deterministic.
 
 test_job_lifecycle(vm, 'drive-mirror', has_ready=True, job_args={
 'device': 'drive0-node',
@@ -227,6 +227,7 @@ with iotests.FilePath('disk.img') as disk_path, \
 'target': copy_path,
 'sync': 'full',
 'speed': 262144,
+'x-perf': {'max-chunk': 65536},
 'auto-finalize': auto_finalize,
 'auto-dismiss': auto_dismiss,
 })
-- 
2.29.2

[PULL 36/53] iotests: 56: prepare for backup over block-copy

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy 

After introducing parallel async copy requests instead of plain
cluster-by-cluster copying loop, we'll have to wait for paused status,
as we need to wait for several parallel request. So, let's gently wait
instead of just asserting that job already paused.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20210116214705.822267-12-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 tests/qemu-iotests/056 | 9 +++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/tests/qemu-iotests/056 b/tests/qemu-iotests/056
index 0e6b8591e7..b459a3f1e8 100755
--- a/tests/qemu-iotests/056
+++ b/tests/qemu-iotests/056
@@ -308,8 +308,13 @@ class BackupTest(iotests.QMPTestCase):
 event = self.vm.event_wait(name="BLOCK_JOB_ERROR",
match={'data': {'device': 'drive0'}})
 self.assertNotEqual(event, None)
-# OK, job should be wedged
-res = self.vm.qmp('query-block-jobs')
+# OK, job should pause, but it can't do it immediately, as it can't
+# cancel other parallel requests (which didn't fail)
+with iotests.Timeout(60, "Timeout waiting for backup actually paused"):
+while True:
+res = self.vm.qmp('query-block-jobs')
+if res['return'][0]['status'] == 'paused':
+break
 self.assert_qmp(res, 'return[0]/status', 'paused')
 res = self.vm.qmp('block-job-dismiss', id='drive0')
 self.assert_qmp(res, 'error/desc',
-- 
2.29.2

[PULL 34/53] job: call job_enter from job_pause

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy 

If main job coroutine called job_yield (while some background process
is in progress), we should give it a chance to call job_pause_point().
It will be used in backup, when moved on async block-copy.

Note, that job_user_pause is not enough: we want to handle
child_job_drained_begin() as well, which call job_pause().

Still, if job is already in job_do_yield() in job_pause_point() we
should not enter it.

iotest 109 output is modified: on stop we do bdrv_drain_all() which now
triggers job pause immediately (and pause after ready is standby).

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Message-Id: <20210116214705.822267-10-vsement...@virtuozzo.com>
Reviewed-by: Max Reitz 
Signed-off-by: Max Reitz 
---
 job.c  |  3 +++
 tests/qemu-iotests/109.out | 24 
 2 files changed, 27 insertions(+)

diff --git a/job.c b/job.c
index 8fecf38960..3aaaebafe2 100644
--- a/job.c
+++ b/job.c
@@ -553,6 +553,9 @@ static bool job_timer_not_pending(Job *job)
 void job_pause(Job *job)
 {
 job->pause_count++;
+if (!job->paused) {
+job_enter(job);
+}
 }
 
 void job_resume(Job *job)
diff --git a/tests/qemu-iotests/109.out b/tests/qemu-iotests/109.out
index 6e73406cdb..8f839b4b7f 100644
--- a/tests/qemu-iotests/109.out
+++ b/tests/qemu-iotests/109.out
@@ -42,6 +42,8 @@ read 512/512 bytes at offset 0
 {"execute":"quit"}
 {"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "standby", "id": "src"}}
+{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "src"}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "src"}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "src"}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"BLOCK_JOB_COMPLETED", "data": {"device": "src", "len": 1024, "offset": 1024, 
"speed": 0, "type": "mirror"}}
@@ -91,6 +93,8 @@ read 512/512 bytes at offset 0
 {"execute":"quit"}
 {"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "standby", "id": "src"}}
+{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "src"}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "src"}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "src"}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"BLOCK_JOB_COMPLETED", "data": {"device": "src", "len": 197120, "offset": 
197120, "speed": 0, "type": "mirror"}}
@@ -140,6 +144,8 @@ read 512/512 bytes at offset 0
 {"execute":"quit"}
 {"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "standby", "id": "src"}}
+{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "src"}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "src"}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "src"}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"BLOCK_JOB_COMPLETED", "data": {"device": "src", "len": 327680, "offset": 
327680, "speed": 0, "type": "mirror"}}
@@ -189,6 +195,8 @@ read 512/512 bytes at offset 0
 {"execute":"quit"}
 {"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "standby", "id": "src"}}
+{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "src"}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "src"}}
 {"timestamp": {"seconds":  

[PULL 31/53] block/block-copy: add ratelimit to block-copy

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy 

We are going to directly use one async block-copy operation for backup
job, so we need rate limiter.

We want to maintain current backup behavior: only background copying is
limited and copy-before-write operations only participate in limit
calculation. Therefore we need one rate limiter for block-copy state
and boolean flag for block-copy call state for actual limitation.

Note, that we can't just calculate each chunk in limiter after
successful copying: it will not save us from starting a lot of async
sub-requests which will exceed limit too much. Instead let's use the
following scheme on sub-request creation:
1. If at the moment limit is not exceeded, create the request and
account it immediately.
2. If at the moment limit is already exceeded, drop create sub-request
and handle limit instead (by sleep).
With this approach we'll never exceed the limit more than by one
sub-request (which pretty much matches current backup behavior).

Note also, that if there is in-flight block-copy async call,
block_copy_kick() should be used after set-speed to apply new setup
faster. For that block_copy_kick() published in this patch.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20210116214705.822267-7-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 include/block/block-copy.h |  5 -
 block/backup-top.c |  2 +-
 block/backup.c |  2 +-
 block/block-copy.c | 46 +-
 4 files changed, 51 insertions(+), 4 deletions(-)

diff --git a/include/block/block-copy.h b/include/block/block-copy.h
index 22372aa375..b5a53ad59e 100644
--- a/include/block/block-copy.h
+++ b/include/block/block-copy.h
@@ -41,7 +41,7 @@ int64_t block_copy_reset_unallocated(BlockCopyState *s,
  int64_t offset, int64_t *count);
 
 int coroutine_fn block_copy(BlockCopyState *s, int64_t offset, int64_t bytes,
-bool *error_is_read);
+bool ignore_ratelimit, bool *error_is_read);
 
 /*
  * Run block-copy in a coroutine, create corresponding BlockCopyCallState
@@ -76,6 +76,9 @@ bool block_copy_call_succeeded(BlockCopyCallState 
*call_state);
 bool block_copy_call_failed(BlockCopyCallState *call_state);
 int block_copy_call_status(BlockCopyCallState *call_state, bool 
*error_is_read);
 
+void block_copy_set_speed(BlockCopyState *s, uint64_t speed);
+void block_copy_kick(BlockCopyCallState *call_state);
+
 BdrvDirtyBitmap *block_copy_dirty_bitmap(BlockCopyState *s);
 void block_copy_set_skip_unallocated(BlockCopyState *s, bool skip);
 
diff --git a/block/backup-top.c b/block/backup-top.c
index 789acf6965..779956ddc2 100644
--- a/block/backup-top.c
+++ b/block/backup-top.c
@@ -61,7 +61,7 @@ static coroutine_fn int backup_top_cbw(BlockDriverState *bs, 
uint64_t offset,
 off = QEMU_ALIGN_DOWN(offset, s->cluster_size);
 end = QEMU_ALIGN_UP(offset + bytes, s->cluster_size);
 
-return block_copy(s->bcs, off, end - off, NULL);
+return block_copy(s->bcs, off, end - off, true, NULL);
 }
 
 static int coroutine_fn backup_top_co_pdiscard(BlockDriverState *bs,
diff --git a/block/backup.c b/block/backup.c
index 4b07e9115d..09ff5a92ef 100644
--- a/block/backup.c
+++ b/block/backup.c
@@ -72,7 +72,7 @@ static int coroutine_fn backup_do_cow(BackupBlockJob *job,
 
 trace_backup_do_cow_enter(job, start, offset, bytes);
 
-ret = block_copy(job->bcs, start, end - start, error_is_read);
+ret = block_copy(job->bcs, start, end - start, true, error_is_read);
 
 trace_backup_do_cow_return(job, offset, bytes, ret);
 
diff --git a/block/block-copy.c b/block/block-copy.c
index 6bf1735b93..fa27450b14 100644
--- a/block/block-copy.c
+++ b/block/block-copy.c
@@ -26,6 +26,7 @@
 #define BLOCK_COPY_MAX_BUFFER (1 * MiB)
 #define BLOCK_COPY_MAX_MEM (128 * MiB)
 #define BLOCK_COPY_MAX_WORKERS 64
+#define BLOCK_COPY_SLICE_TIME 1ULL /* ns */
 
 static coroutine_fn int block_copy_task_entry(AioTask *task);
 
@@ -36,6 +37,7 @@ typedef struct BlockCopyCallState {
 int64_t bytes;
 int max_workers;
 int64_t max_chunk;
+bool ignore_ratelimit;
 BlockCopyAsyncCallbackFunc cb;
 void *cb_opaque;
 
@@ -48,6 +50,7 @@ typedef struct BlockCopyCallState {
 /* State */
 int ret;
 bool finished;
+QemuCoSleepState *sleep_state;
 
 /* OUT parameters */
 bool error_is_read;
@@ -111,6 +114,9 @@ typedef struct BlockCopyState {
 void *progress_opaque;
 
 SharedResource *mem;
+
+uint64_t speed;
+RateLimit rate_limit;
 } BlockCopyState;
 
 static BlockCopyTask *find_conflicting_task(BlockCopyState *s,
@@ -623,6 +629,21 @@ block_copy_dirty_clusters(BlockCopyCallState *call_state)
 }
 task->zeroes = ret & BDRV_BLOCK_ZERO;
 
+if (s->speed) {
+if (!call_state->ignore_ratelimit) {
+uint64_t ns = ratelimit_calculate_delay(>rate_limit, 0);
+   

[PULL 40/53] iotests: 257: prepare for backup over block-copy

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy 

Iotest 257 dumps a lot of in-progress information of backup job, such
as offset and bitmap dirtiness. Further commit will move backup to be
one block-copy call, which will introduce async parallel requests
instead of plain cluster-by-cluster copying. To keep things
deterministic, allow only one worker (only one copy request at a time)
for this test.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20210116214705.822267-15-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 tests/qemu-iotests/257 |   1 +
 tests/qemu-iotests/257.out | 306 ++---
 2 files changed, 154 insertions(+), 153 deletions(-)

diff --git a/tests/qemu-iotests/257 b/tests/qemu-iotests/257
index a2f4b5afe6..7cd2520829 100755
--- a/tests/qemu-iotests/257
+++ b/tests/qemu-iotests/257
@@ -192,6 +192,7 @@ def blockdev_backup(vm, device, target, sync, **kwargs):
 target=target,
 sync=sync,
 filter_node_name='backup-top',
+x_perf={'max-workers': 1},
 **kwargs)
 return result
 
diff --git a/tests/qemu-iotests/257.out b/tests/qemu-iotests/257.out
index 64dd460055..a7ba512f4c 100644
--- a/tests/qemu-iotests/257.out
+++ b/tests/qemu-iotests/257.out
@@ -30,7 +30,7 @@ write -P0x76 0x3ff 0x1
 {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
 {"return": {}}
 {}
-{"execute": "blockdev-backup", "arguments": {"device": "drive0", 
"filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", 
"target": "ref_target_0"}}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", 
"filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", 
"target": "ref_target_0", "x-perf": {"max-workers": 1}}}
 {"return": {}}
 {"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, 
"speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": 
{"microseconds": "USECS", "seconds": "SECS"}}
 
@@ -78,7 +78,7 @@ expecting 6 dirty sectors; have 6. OK!
 {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
 {"return": {}}
 {}
-{"execute": "blockdev-backup", "arguments": {"device": "drive0", 
"filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", 
"target": "ref_target_1"}}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", 
"filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", 
"target": "ref_target_1", "x-perf": {"max-workers": 1}}}
 {"return": {}}
 {"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, 
"speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": 
{"microseconds": "USECS", "seconds": "SECS"}}
 
@@ -92,7 +92,7 @@ expecting 6 dirty sectors; have 6. OK!
 {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
 {"return": {}}
 {}
-{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": 
"bitmap0", "bitmap-mode": "never", "device": "drive0", "filter-node-name": 
"backup-top", "job-id": "backup_1", "sync": "bitmap", "target": 
"backup_target_1"}}
+{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": 
"bitmap0", "bitmap-mode": "never", "device": "drive0", "filter-node-name": 
"backup-top", "job-id": "backup_1", "sync": "bitmap", "target": 
"backup_target_1", "x-perf": {"max-workers": 1}}}
 {"return": {}}
 
 --- Write #2 ---
@@ -205,7 +205,7 @@ expecting 15 dirty sectors; have 15. OK!
 {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
 {"return": {}}
 {}
-{"execute": "blockdev-backup", "arguments": {"device": "drive0", 
"filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", 
"target": "ref_target_2"}}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", 
"filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", 
"target": "ref_target_2", "x-perf": {"max-workers": 1}}}
 {"return": {}}
 {"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, 
"speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": 
{"microseconds": "USECS", "seconds": "SECS"}}
 
@@ -219,7 +219,7 @@ expecting 15 dirty sectors; have 15. OK!
 {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
 {"return": {}}
 {}
-{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": 
"bitmap0", "bitmap-mode": "never", "device": "drive0", "filter-node-name": 
"backup-top", "job-id": "backup_2", "sync": "bitmap", "target": 
"backup_target_2"}}
+{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": 
"bitmap0", "bitmap-mode": "never", "device": "drive0", "filter-node-name": 
"backup-top", "job-id": "backup_2", "sync": "bitmap", "target": 
"backup_target_2", "x-perf": {"max-workers": 1}}}
 {"return": {}}
 {"execute": "job-finalize", "arguments": {"id": "backup_2"}}
 {"return": {}}
@@ -290,7 

[PULL 43/53] backup: move to block-copy

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy 

This brings async request handling and block-status driven chunk sizes
to backup out of the box, which improves backup performance.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20210116214705.822267-18-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 block/backup.c | 187 +++--
 1 file changed, 120 insertions(+), 67 deletions(-)

diff --git a/block/backup.c b/block/backup.c
index 466608ee55..cc525d5544 100644
--- a/block/backup.c
+++ b/block/backup.c
@@ -22,7 +22,6 @@
 #include "block/block-copy.h"
 #include "qapi/error.h"
 #include "qapi/qmp/qerror.h"
-#include "qemu/ratelimit.h"
 #include "qemu/cutils.h"
 #include "sysemu/block-backend.h"
 #include "qemu/bitmap.h"
@@ -44,41 +43,17 @@ typedef struct BackupBlockJob {
 BlockdevOnError on_source_error;
 BlockdevOnError on_target_error;
 uint64_t len;
-uint64_t bytes_read;
 int64_t cluster_size;
 BackupPerf perf;
 
 BlockCopyState *bcs;
+
+bool wait;
+BlockCopyCallState *bg_bcs_call;
 } BackupBlockJob;
 
 static const BlockJobDriver backup_job_driver;
 
-static void backup_progress_bytes_callback(int64_t bytes, void *opaque)
-{
-BackupBlockJob *s = opaque;
-
-s->bytes_read += bytes;
-}
-
-static int coroutine_fn backup_do_cow(BackupBlockJob *job,
-  int64_t offset, uint64_t bytes,
-  bool *error_is_read)
-{
-int ret = 0;
-int64_t start, end; /* bytes */
-
-start = QEMU_ALIGN_DOWN(offset, job->cluster_size);
-end = QEMU_ALIGN_UP(bytes + offset, job->cluster_size);
-
-trace_backup_do_cow_enter(job, start, offset, bytes);
-
-ret = block_copy(job->bcs, start, end - start, true, error_is_read);
-
-trace_backup_do_cow_return(job, offset, bytes, ret);
-
-return ret;
-}
-
 static void backup_cleanup_sync_bitmap(BackupBlockJob *job, int ret)
 {
 BdrvDirtyBitmap *bm;
@@ -158,53 +133,96 @@ static BlockErrorAction 
backup_error_action(BackupBlockJob *job,
 }
 }
 
-static bool coroutine_fn yield_and_check(BackupBlockJob *job)
+static void coroutine_fn backup_block_copy_callback(void *opaque)
 {
-uint64_t delay_ns;
-
-if (job_is_cancelled(>common.job)) {
-return true;
-}
-
-/*
- * We need to yield even for delay_ns = 0 so that bdrv_drain_all() can
- * return. Without a yield, the VM would not reboot.
- */
-delay_ns = block_job_ratelimit_get_delay(>common, job->bytes_read);
-job->bytes_read = 0;
-job_sleep_ns(>common.job, delay_ns);
+BackupBlockJob *s = opaque;
 
-if (job_is_cancelled(>common.job)) {
-return true;
+if (s->wait) {
+s->wait = false;
+aio_co_wake(s->common.job.co);
+} else {
+job_enter(>common.job);
 }
-
-return false;
 }
 
 static int coroutine_fn backup_loop(BackupBlockJob *job)
 {
-bool error_is_read;
-int64_t offset;
-BdrvDirtyBitmapIter *bdbi;
+BlockCopyCallState *s = NULL;
 int ret = 0;
+bool error_is_read;
+BlockErrorAction act;
+
+while (true) { /* retry loop */
+job->bg_bcs_call = s = block_copy_async(job->bcs, 0,
+QEMU_ALIGN_UP(job->len, job->cluster_size),
+job->perf.max_workers, job->perf.max_chunk,
+backup_block_copy_callback, job);
+
+while (!block_copy_call_finished(s) &&
+   !job_is_cancelled(>common.job))
+{
+job_yield(>common.job);
+}
 
-bdbi = bdrv_dirty_iter_new(block_copy_dirty_bitmap(job->bcs));
-while ((offset = bdrv_dirty_iter_next(bdbi)) != -1) {
-do {
-if (yield_and_check(job)) {
-goto out;
-}
-ret = backup_do_cow(job, offset, job->cluster_size, 
_is_read);
-if (ret < 0 && backup_error_action(job, error_is_read, -ret) ==
-   BLOCK_ERROR_ACTION_REPORT)
-{
-goto out;
-}
-} while (ret < 0);
+if (!block_copy_call_finished(s)) {
+assert(job_is_cancelled(>common.job));
+/*
+ * Note that we can't use job_yield() here, as it doesn't work for
+ * cancelled job.
+ */
+block_copy_call_cancel(s);
+job->wait = true;
+qemu_coroutine_yield();
+assert(block_copy_call_finished(s));
+ret = 0;
+goto out;
+}
+
+if (job_is_cancelled(>common.job) ||
+block_copy_call_succeeded(s))
+{
+ret = 0;
+goto out;
+}
+
+if (block_copy_call_cancelled(s)) {
+/*
+ * Job is not cancelled but only block-copy call. This is possible
+ * after job pause. Now the pause is finished, start new block-copy
+ * iteration.
+ */
+

[PULL 41/53] block/block-copy: make progress_bytes_callback optional

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy 

We are going to stop use of this callback in the following commit.
Still the callback handling code will be dropped in a separate commit.
So, for now let's make it optional.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20210116214705.822267-16-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 block/block-copy.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/block/block-copy.c b/block/block-copy.c
index 82cf945693..61d82d9a1c 100644
--- a/block/block-copy.c
+++ b/block/block-copy.c
@@ -454,7 +454,9 @@ static coroutine_fn int block_copy_task_entry(AioTask *task)
 t->call_state->error_is_read = error_is_read;
 } else {
 progress_work_done(t->s->progress, t->bytes);
-t->s->progress_bytes_callback(t->bytes, t->s->progress_opaque);
+if (t->s->progress_bytes_callback) {
+t->s->progress_bytes_callback(t->bytes, t->s->progress_opaque);
+}
 }
 co_put_to_shres(t->s->mem, t->bytes);
 block_copy_task_end(t, ret);
-- 
2.29.2

[PULL 29/53] block/block-copy: add max_chunk and max_workers parameters

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy 

They will be used for backup.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20210116214705.822267-5-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 include/block/block-copy.h |  6 ++
 block/block-copy.c | 11 +--
 2 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/include/block/block-copy.h b/include/block/block-copy.h
index 8c225ebf81..22372aa375 100644
--- a/include/block/block-copy.h
+++ b/include/block/block-copy.h
@@ -49,9 +49,15 @@ int coroutine_fn block_copy(BlockCopyState *s, int64_t 
offset, int64_t bytes,
  *
  * Caller is responsible to call block_copy_call_free() to free
  * BlockCopyCallState object.
+ *
+ * @max_workers means maximum of parallel coroutines to execute sub-requests,
+ * must be > 0.
+ *
+ * @max_chunk means maximum length for one IO operation. Zero means unlimited.
  */
 BlockCopyCallState *block_copy_async(BlockCopyState *s,
  int64_t offset, int64_t bytes,
+ int max_workers, int64_t max_chunk,
  BlockCopyAsyncCallbackFunc cb,
  void *cb_opaque);
 
diff --git a/block/block-copy.c b/block/block-copy.c
index 74655b86f8..35213bd832 100644
--- a/block/block-copy.c
+++ b/block/block-copy.c
@@ -34,6 +34,8 @@ typedef struct BlockCopyCallState {
 BlockCopyState *s;
 int64_t offset;
 int64_t bytes;
+int max_workers;
+int64_t max_chunk;
 BlockCopyAsyncCallbackFunc cb;
 void *cb_opaque;
 
@@ -148,10 +150,11 @@ static BlockCopyTask 
*block_copy_task_create(BlockCopyState *s,
  int64_t offset, int64_t bytes)
 {
 BlockCopyTask *task;
+int64_t max_chunk = MIN_NON_ZERO(s->copy_size, call_state->max_chunk);
 
 if (!bdrv_dirty_bitmap_next_dirty_area(s->copy_bitmap,
offset, offset + bytes,
-   s->copy_size, , ))
+   max_chunk, , ))
 {
 return NULL;
 }
@@ -623,7 +626,7 @@ block_copy_dirty_clusters(BlockCopyCallState *call_state)
 bytes = end - offset;
 
 if (!aio && bytes) {
-aio = aio_task_pool_new(BLOCK_COPY_MAX_WORKERS);
+aio = aio_task_pool_new(call_state->max_workers);
 }
 
 ret = block_copy_task_run(aio, task);
@@ -701,6 +704,7 @@ int coroutine_fn block_copy(BlockCopyState *s, int64_t 
start, int64_t bytes,
 .s = s,
 .offset = start,
 .bytes = bytes,
+.max_workers = BLOCK_COPY_MAX_WORKERS,
 };
 
 int ret = block_copy_common(_state);
@@ -719,6 +723,7 @@ static void coroutine_fn block_copy_async_co_entry(void 
*opaque)
 
 BlockCopyCallState *block_copy_async(BlockCopyState *s,
  int64_t offset, int64_t bytes,
+ int max_workers, int64_t max_chunk,
  BlockCopyAsyncCallbackFunc cb,
  void *cb_opaque)
 {
@@ -728,6 +733,8 @@ BlockCopyCallState *block_copy_async(BlockCopyState *s,
 .s = s,
 .offset = offset,
 .bytes = bytes,
+.max_workers = max_workers,
+.max_chunk = max_chunk,
 .cb = cb,
 .cb_opaque = cb_opaque,
 
-- 
2.29.2

[PULL 26/53] qapi: backup: add perf.use-copy-range parameter

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy 

Experiments show, that copy_range is not always making things faster.
So, to make experimentation simpler, let's add a parameter. Some more
perf parameters will be added soon, so here is a new struct.

For now, add new backup qmp parameter with x- prefix for the following
reasons:

 - We are going to add more performance parameters, some will be
   related to the whole block-copy process, some only to background
   copying in backup (ignored for copy-before-write operations).
 - On the other hand, we are going to use block-copy interface in other
   block jobs, which will need performance options as well.. And it
   should be the same structure or at least somehow related.

So, there are too much unclean things about how the interface and now
we need the new options mostly for testing. Let's keep them
experimental for a while.

In do_backup_common() new x-perf parameter handled in a way to
make further options addition simpler.

We add use-copy-range with default=true, and we'll change the default
in further patch, after moving backup to use block-copy.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20210116214705.822267-2-vsement...@virtuozzo.com>
[mreitz: s/5\.2/6.0/]
Signed-off-by: Max Reitz 
---
 qapi/block-core.json   | 17 -
 block/backup-top.h |  1 +
 include/block/block-copy.h |  2 +-
 include/block/block_int.h  |  3 +++
 block/backup-top.c |  4 +++-
 block/backup.c |  6 +-
 block/block-copy.c |  4 ++--
 block/replication.c|  2 ++
 blockdev.c |  8 
 9 files changed, 41 insertions(+), 6 deletions(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index 1d9dcd7d30..83f661d7f6 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -1371,6 +1371,19 @@
 { 'struct': 'BlockdevSnapshot',
   'data': { 'node': 'str', 'overlay': 'str' } }
 
+##
+# @BackupPerf:
+#
+# Optional parameters for backup. These parameters don't affect
+# functionality, but may significantly affect performance.
+#
+# @use-copy-range: Use copy offloading. Default true.
+#
+# Since: 6.0
+##
+{ 'struct': 'BackupPerf',
+  'data': { '*use-copy-range': 'bool' }}
+
 ##
 # @BackupCommon:
 #
@@ -1426,6 +1439,8 @@
 #above node specified by @drive. If this option is not 
given,
 #a node name is autogenerated. (Since: 4.2)
 #
+# @x-perf: Performance options. (Since 6.0)
+#
 # Note: @on-source-error and @on-target-error only affect background
 #   I/O.  If an error occurs during a guest write request, the device's
 #   rerror/werror actions will be used.
@@ -1440,7 +1455,7 @@
 '*on-source-error': 'BlockdevOnError',
 '*on-target-error': 'BlockdevOnError',
 '*auto-finalize': 'bool', '*auto-dismiss': 'bool',
-'*filter-node-name': 'str' } }
+'*filter-node-name': 'str', '*x-perf': 'BackupPerf'  } }
 
 ##
 # @DriveBackup:
diff --git a/block/backup-top.h b/block/backup-top.h
index e5cabfa197..b28b0031c4 100644
--- a/block/backup-top.h
+++ b/block/backup-top.h
@@ -33,6 +33,7 @@ BlockDriverState *bdrv_backup_top_append(BlockDriverState 
*source,
  BlockDriverState *target,
  const char *filter_node_name,
  uint64_t cluster_size,
+ BackupPerf *perf,
  BdrvRequestFlags write_flags,
  BlockCopyState **bcs,
  Error **errp);
diff --git a/include/block/block-copy.h b/include/block/block-copy.h
index aac85e1488..6397505f30 100644
--- a/include/block/block-copy.h
+++ b/include/block/block-copy.h
@@ -22,7 +22,7 @@ typedef void (*ProgressBytesCallbackFunc)(int64_t bytes, void 
*opaque);
 typedef struct BlockCopyState BlockCopyState;
 
 BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target,
- int64_t cluster_size,
+ int64_t cluster_size, bool use_copy_range,
  BdrvRequestFlags write_flags,
  Error **errp);
 
diff --git a/include/block/block_int.h b/include/block/block_int.h
index f4b844f310..d01fc23720 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -1266,6 +1266,8 @@ void mirror_start(const char *job_id, BlockDriverState 
*bs,
  * @sync_mode: What parts of the disk image should be copied to the 
destination.
  * @sync_bitmap: The dirty bitmap if sync_mode is 'bitmap' or 'incremental'
  * @bitmap_mode: The bitmap synchronization policy to use.
+ * @perf: Performance options. All actual fields assumed to be present,
+ *all ".has_*" fields are ignored.
  * @on_source_error: The action to take upon error reading 

[PULL 27/53] block/block-copy: More explicit call_state

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy 

Refactor common path to use BlockCopyCallState pointer as parameter, to
prepare it for use in asynchronous block-copy (at least, we'll need to
run block-copy in a coroutine, passing the whole parameters as one
pointer).

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20210116214705.822267-3-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 block/block-copy.c | 51 ++
 1 file changed, 38 insertions(+), 13 deletions(-)

diff --git a/block/block-copy.c b/block/block-copy.c
index 63398a171c..6ea55f1f9a 100644
--- a/block/block-copy.c
+++ b/block/block-copy.c
@@ -30,7 +30,15 @@
 static coroutine_fn int block_copy_task_entry(AioTask *task);
 
 typedef struct BlockCopyCallState {
+/* IN parameters */
+BlockCopyState *s;
+int64_t offset;
+int64_t bytes;
+
+/* State */
 bool failed;
+
+/* OUT parameters */
 bool error_is_read;
 } BlockCopyCallState;
 
@@ -544,15 +552,17 @@ int64_t block_copy_reset_unallocated(BlockCopyState *s,
  * Returns 1 if dirty clusters found and successfully copied, 0 if no dirty
  * clusters found and -errno on failure.
  */
-static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s,
-  int64_t offset, int64_t 
bytes,
-  bool *error_is_read)
+static int coroutine_fn
+block_copy_dirty_clusters(BlockCopyCallState *call_state)
 {
+BlockCopyState *s = call_state->s;
+int64_t offset = call_state->offset;
+int64_t bytes = call_state->bytes;
+
 int ret = 0;
 bool found_dirty = false;
 int64_t end = offset + bytes;
 AioTaskPool *aio = NULL;
-BlockCopyCallState call_state = {false, false};
 
 /*
  * block_copy() user is responsible for keeping source and target in same
@@ -568,7 +578,7 @@ static int coroutine_fn 
block_copy_dirty_clusters(BlockCopyState *s,
 BlockCopyTask *task;
 int64_t status_bytes;
 
-task = block_copy_task_create(s, _state, offset, bytes);
+task = block_copy_task_create(s, call_state, offset, bytes);
 if (!task) {
 /* No more dirty bits in the bitmap */
 trace_block_copy_skip_range(s, offset, bytes);
@@ -633,15 +643,12 @@ out:
 
 aio_task_pool_free(aio);
 }
-if (error_is_read && ret < 0) {
-*error_is_read = call_state.error_is_read;
-}
 
 return ret < 0 ? ret : found_dirty;
 }
 
 /*
- * block_copy
+ * block_copy_common
  *
  * Copy requested region, accordingly to dirty bitmap.
  * Collaborate with parallel block_copy requests: if they succeed it will help
@@ -649,16 +656,16 @@ out:
  * it means that some I/O operation failed in context of _this_ block_copy 
call,
  * not some parallel operation.
  */
-int coroutine_fn block_copy(BlockCopyState *s, int64_t offset, int64_t bytes,
-bool *error_is_read)
+static int coroutine_fn block_copy_common(BlockCopyCallState *call_state)
 {
 int ret;
 
 do {
-ret = block_copy_dirty_clusters(s, offset, bytes, error_is_read);
+ret = block_copy_dirty_clusters(call_state);
 
 if (ret == 0) {
-ret = block_copy_wait_one(s, offset, bytes);
+ret = block_copy_wait_one(call_state->s, call_state->offset,
+  call_state->bytes);
 }
 
 /*
@@ -675,6 +682,24 @@ int coroutine_fn block_copy(BlockCopyState *s, int64_t 
offset, int64_t bytes,
 return ret;
 }
 
+int coroutine_fn block_copy(BlockCopyState *s, int64_t start, int64_t bytes,
+bool *error_is_read)
+{
+BlockCopyCallState call_state = {
+.s = s,
+.offset = start,
+.bytes = bytes,
+};
+
+int ret = block_copy_common(_state);
+
+if (error_is_read && ret < 0) {
+*error_is_read = call_state.error_is_read;
+}
+
+return ret;
+}
+
 BdrvDirtyBitmap *block_copy_dirty_bitmap(BlockCopyState *s)
 {
 return s->copy_bitmap;
-- 
2.29.2

[PULL 38/53] iotests: 185: prepare for backup over block-copy

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy 

The further change of moving backup to be a one block-copy call will
make copying chunk-size and cluster-size two separate things. So, even
with 64k cluster sized qcow2 image, default chunk would be 1M.
185 test however assumes, that with speed limited to 64K, one iteration
would result in offset=64K. It will change, as first iteration would
result in offset=1M independently of speed.

So, let's explicitly specify, what test wants: set max-chunk to 64K, so
that one iteration is 64K. Note, that we don't need to limit
max-workers, as block-copy rate limiter will handle the situation and
wouldn't start new workers when speed limit is obviously reached.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20210116214705.822267-13-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 tests/qemu-iotests/185 | 3 ++-
 tests/qemu-iotests/185.out | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/tests/qemu-iotests/185 b/tests/qemu-iotests/185
index 7bc8fe5767..f2ec5c5ceb 100755
--- a/tests/qemu-iotests/185
+++ b/tests/qemu-iotests/185
@@ -183,7 +183,8 @@ _send_qemu_cmd $h \
   'target': '$TEST_IMG.copy',
   'format': '$IMGFMT',
   'sync': 'full',
-  'speed': 65536 } }" \
+  'speed': 65536,
+  'x-perf': {'max-chunk': 65536} } }" \
 "return"
 
 # If we don't sleep here 'quit' command races with disk I/O
diff --git a/tests/qemu-iotests/185.out b/tests/qemu-iotests/185.out
index eab55d22bf..9dedc8eacb 100644
--- a/tests/qemu-iotests/185.out
+++ b/tests/qemu-iotests/185.out
@@ -88,7 +88,8 @@ Formatting 'TEST_DIR/t.qcow2.copy', fmt=qcow2 
cluster_size=65536 extended_l2=off
   'target': 'TEST_DIR/t.IMGFMT.copy',
   'format': 'IMGFMT',
   'sync': 'full',
-  'speed': 65536 } }
+  'speed': 65536,
+  'x-perf': { 'max-chunk': 65536 } } }
 Formatting 'TEST_DIR/t.qcow2.copy', fmt=qcow2 cluster_size=65536 
extended_l2=off compression_type=zlib size=67108864 lazy_refcounts=off 
refcount_bits=16
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "created", "id": "disk"}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "running", "id": "disk"}}
-- 
2.29.2

[PULL 19/53] iotests/129: Do not check @busy

[Max Reitz]

@busy is false when the job is paused, which happens all the time
because that is how jobs yield (e.g. for mirror at least since commit
565ac01f8d3).

Back when 129 was added (2015), perhaps there was no better way of
checking whether the job was still actually running.  Now we have the
@status field (as of 58b295ba52c, i.e. 2018), which can give us exactly
that information.

Signed-off-by: Max Reitz 
Reviewed-by: Eric Blake 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Willian Rampazzo 
Message-Id: <20210118105720.14824-6-mre...@redhat.com>
---
 tests/qemu-iotests/129 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/129 b/tests/qemu-iotests/129
index bd29c54af8..0f2e5418ef 100755
--- a/tests/qemu-iotests/129
+++ b/tests/qemu-iotests/129
@@ -70,7 +70,7 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
 result = self.vm.qmp("stop")
 self.assert_qmp(result, 'return', {})
 result = self.vm.qmp("query-block-jobs")
-self.assert_qmp(result, 'return[0]/busy', True)
+self.assert_qmp(result, 'return[0]/status', 'running')
 self.assert_qmp(result, 'return[0]/ready', False)
 
 def test_drive_mirror(self):
-- 
2.29.2

[PULL 32/53] block/block-copy: add block_copy_cancel

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy 

Add function to cancel running async block-copy call. It will be used
in backup.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20210116214705.822267-8-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 include/block/block-copy.h | 13 +
 block/block-copy.c | 24 +++-
 2 files changed, 32 insertions(+), 5 deletions(-)

diff --git a/include/block/block-copy.h b/include/block/block-copy.h
index b5a53ad59e..7821850f88 100644
--- a/include/block/block-copy.h
+++ b/include/block/block-copy.h
@@ -74,11 +74,24 @@ void block_copy_call_free(BlockCopyCallState *call_state);
 bool block_copy_call_finished(BlockCopyCallState *call_state);
 bool block_copy_call_succeeded(BlockCopyCallState *call_state);
 bool block_copy_call_failed(BlockCopyCallState *call_state);
+bool block_copy_call_cancelled(BlockCopyCallState *call_state);
 int block_copy_call_status(BlockCopyCallState *call_state, bool 
*error_is_read);
 
 void block_copy_set_speed(BlockCopyState *s, uint64_t speed);
 void block_copy_kick(BlockCopyCallState *call_state);
 
+/*
+ * Cancel running block-copy call.
+ *
+ * Cancel leaves block-copy state valid: dirty bits are correct and you may use
+ * cancel +  to emulate pause/resume.
+ *
+ * Note also, that the cancel is async: it only marks block-copy call to be
+ * cancelled. So, the call may be cancelled (block_copy_call_cancelled() 
reports
+ * true) but not yet finished (block_copy_call_finished() reports false).
+ */
+void block_copy_call_cancel(BlockCopyCallState *call_state);
+
 BdrvDirtyBitmap *block_copy_dirty_bitmap(BlockCopyState *s);
 void block_copy_set_skip_unallocated(BlockCopyState *s, bool skip);
 
diff --git a/block/block-copy.c b/block/block-copy.c
index fa27450b14..82cf945693 100644
--- a/block/block-copy.c
+++ b/block/block-copy.c
@@ -51,6 +51,7 @@ typedef struct BlockCopyCallState {
 int ret;
 bool finished;
 QemuCoSleepState *sleep_state;
+bool cancelled;
 
 /* OUT parameters */
 bool error_is_read;
@@ -594,7 +595,7 @@ block_copy_dirty_clusters(BlockCopyCallState *call_state)
 assert(QEMU_IS_ALIGNED(offset, s->cluster_size));
 assert(QEMU_IS_ALIGNED(bytes, s->cluster_size));
 
-while (bytes && aio_task_pool_status(aio) == 0) {
+while (bytes && aio_task_pool_status(aio) == 0 && !call_state->cancelled) {
 BlockCopyTask *task;
 int64_t status_bytes;
 
@@ -707,7 +708,7 @@ static int coroutine_fn 
block_copy_common(BlockCopyCallState *call_state)
 do {
 ret = block_copy_dirty_clusters(call_state);
 
-if (ret == 0) {
+if (ret == 0 && !call_state->cancelled) {
 ret = block_copy_wait_one(call_state->s, call_state->offset,
   call_state->bytes);
 }
@@ -721,7 +722,7 @@ static int coroutine_fn 
block_copy_common(BlockCopyCallState *call_state)
  * 2. We have waited for some intersecting block-copy request
  *It may have failed and produced new dirty bits.
  */
-} while (ret > 0);
+} while (ret > 0 && !call_state->cancelled);
 
 call_state->finished = true;
 
@@ -801,12 +802,19 @@ bool block_copy_call_finished(BlockCopyCallState 
*call_state)
 
 bool block_copy_call_succeeded(BlockCopyCallState *call_state)
 {
-return call_state->finished && call_state->ret == 0;
+return call_state->finished && !call_state->cancelled &&
+call_state->ret == 0;
 }
 
 bool block_copy_call_failed(BlockCopyCallState *call_state)
 {
-return call_state->finished && call_state->ret < 0;
+return call_state->finished && !call_state->cancelled &&
+call_state->ret < 0;
+}
+
+bool block_copy_call_cancelled(BlockCopyCallState *call_state)
+{
+return call_state->cancelled;
 }
 
 int block_copy_call_status(BlockCopyCallState *call_state, bool *error_is_read)
@@ -818,6 +826,12 @@ int block_copy_call_status(BlockCopyCallState *call_state, 
bool *error_is_read)
 return call_state->ret;
 }
 
+void block_copy_call_cancel(BlockCopyCallState *call_state)
+{
+call_state->cancelled = true;
+block_copy_kick(call_state);
+}
+
 BdrvDirtyBitmap *block_copy_dirty_bitmap(BlockCopyState *s)
 {
 return s->copy_bitmap;
-- 
2.29.2

[PULL 28/53] block/block-copy: implement block_copy_async

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy 

We'll need async block-copy invocation to use in backup directly.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20210116214705.822267-4-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 include/block/block-copy.h | 29 ++
 block/block-copy.c | 81 --
 2 files changed, 106 insertions(+), 4 deletions(-)

diff --git a/include/block/block-copy.h b/include/block/block-copy.h
index 6397505f30..8c225ebf81 100644
--- a/include/block/block-copy.h
+++ b/include/block/block-copy.h
@@ -19,7 +19,9 @@
 #include "qemu/co-shared-resource.h"
 
 typedef void (*ProgressBytesCallbackFunc)(int64_t bytes, void *opaque);
+typedef void (*BlockCopyAsyncCallbackFunc)(void *opaque);
 typedef struct BlockCopyState BlockCopyState;
+typedef struct BlockCopyCallState BlockCopyCallState;
 
 BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target,
  int64_t cluster_size, bool use_copy_range,
@@ -41,6 +43,33 @@ int64_t block_copy_reset_unallocated(BlockCopyState *s,
 int coroutine_fn block_copy(BlockCopyState *s, int64_t offset, int64_t bytes,
 bool *error_is_read);
 
+/*
+ * Run block-copy in a coroutine, create corresponding BlockCopyCallState
+ * object and return pointer to it. Never returns NULL.
+ *
+ * Caller is responsible to call block_copy_call_free() to free
+ * BlockCopyCallState object.
+ */
+BlockCopyCallState *block_copy_async(BlockCopyState *s,
+ int64_t offset, int64_t bytes,
+ BlockCopyAsyncCallbackFunc cb,
+ void *cb_opaque);
+
+/*
+ * Free finished BlockCopyCallState. Trying to free running
+ * block-copy will crash.
+ */
+void block_copy_call_free(BlockCopyCallState *call_state);
+
+/*
+ * Note, that block-copy call is marked finished prior to calling
+ * the callback.
+ */
+bool block_copy_call_finished(BlockCopyCallState *call_state);
+bool block_copy_call_succeeded(BlockCopyCallState *call_state);
+bool block_copy_call_failed(BlockCopyCallState *call_state);
+int block_copy_call_status(BlockCopyCallState *call_state, bool 
*error_is_read);
+
 BdrvDirtyBitmap *block_copy_dirty_bitmap(BlockCopyState *s);
 void block_copy_set_skip_unallocated(BlockCopyState *s, bool skip);
 
diff --git a/block/block-copy.c b/block/block-copy.c
index 6ea55f1f9a..74655b86f8 100644
--- a/block/block-copy.c
+++ b/block/block-copy.c
@@ -30,13 +30,19 @@
 static coroutine_fn int block_copy_task_entry(AioTask *task);
 
 typedef struct BlockCopyCallState {
-/* IN parameters */
+/* IN parameters. Initialized in block_copy_async() and never changed. */
 BlockCopyState *s;
 int64_t offset;
 int64_t bytes;
+BlockCopyAsyncCallbackFunc cb;
+void *cb_opaque;
+
+/* Coroutine where async block-copy is running */
+Coroutine *co;
 
 /* State */
-bool failed;
+int ret;
+bool finished;
 
 /* OUT parameters */
 bool error_is_read;
@@ -428,8 +434,8 @@ static coroutine_fn int block_copy_task_entry(AioTask *task)
 
 ret = block_copy_do_copy(t->s, t->offset, t->bytes, t->zeroes,
  _is_read);
-if (ret < 0 && !t->call_state->failed) {
-t->call_state->failed = true;
+if (ret < 0 && !t->call_state->ret) {
+t->call_state->ret = ret;
 t->call_state->error_is_read = error_is_read;
 } else {
 progress_work_done(t->s->progress, t->bytes);
@@ -679,6 +685,12 @@ static int coroutine_fn 
block_copy_common(BlockCopyCallState *call_state)
  */
 } while (ret > 0);
 
+call_state->finished = true;
+
+if (call_state->cb) {
+call_state->cb(call_state->cb_opaque);
+}
+
 return ret;
 }
 
@@ -700,6 +712,67 @@ int coroutine_fn block_copy(BlockCopyState *s, int64_t 
start, int64_t bytes,
 return ret;
 }
 
+static void coroutine_fn block_copy_async_co_entry(void *opaque)
+{
+block_copy_common(opaque);
+}
+
+BlockCopyCallState *block_copy_async(BlockCopyState *s,
+ int64_t offset, int64_t bytes,
+ BlockCopyAsyncCallbackFunc cb,
+ void *cb_opaque)
+{
+BlockCopyCallState *call_state = g_new(BlockCopyCallState, 1);
+
+*call_state = (BlockCopyCallState) {
+.s = s,
+.offset = offset,
+.bytes = bytes,
+.cb = cb,
+.cb_opaque = cb_opaque,
+
+.co = qemu_coroutine_create(block_copy_async_co_entry, call_state),
+};
+
+qemu_coroutine_enter(call_state->co);
+
+return call_state;
+}
+
+void block_copy_call_free(BlockCopyCallState *call_state)
+{
+if (!call_state) {
+return;
+}
+
+assert(call_state->finished);
+g_free(call_state);
+}
+
+bool block_copy_call_finished(BlockCopyCallState *call_state)
+{
+

[PULL 23/53] iotests/129: Clean up pylint and mypy complaints

[Max Reitz]

And consequentially drop it from 297's skip list.

Signed-off-by: Max Reitz 
Reviewed-by: Willian Rampazzo 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
Message-Id: <20210118105720.14824-10-mre...@redhat.com>
---
 tests/qemu-iotests/129 | 4 ++--
 tests/qemu-iotests/297 | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/tests/qemu-iotests/129 b/tests/qemu-iotests/129
index 03843fc255..2f7b28d4a0 100755
--- a/tests/qemu-iotests/129
+++ b/tests/qemu-iotests/129
@@ -21,7 +21,6 @@
 
 import os
 import iotests
-import time
 
 class TestStopWithBlockJob(iotests.QMPTestCase):
 test_img = os.path.join(iotests.test_dir, 'test.img')
@@ -33,7 +32,8 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
 iotests.qemu_img('create', '-f', iotests.imgfmt, self.base_img, "1G")
 iotests.qemu_img('create', '-f', iotests.imgfmt, self.test_img,
  "-b", self.base_img, '-F', iotests.imgfmt)
-iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x5d 1M 128M', 
self.test_img)
+iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x5d 1M 128M',
+self.test_img)
 self.vm = iotests.VM()
 self.vm.add_object('throttle-group,id=tg0,x-bps-total=1024')
 
diff --git a/tests/qemu-iotests/297 b/tests/qemu-iotests/297
index b138b0539c..79e63f8625 100755
--- a/tests/qemu-iotests/297
+++ b/tests/qemu-iotests/297
@@ -28,7 +28,7 @@ import iotests
 # TODO: Empty this list!
 SKIP_FILES = (
 '030', '040', '041', '044', '045', '055', '056', '057', '065', '093',
-'096', '118', '124', '129', '132', '136', '139', '147', '148', '149',
+'096', '118', '124', '132', '136', '139', '147', '148', '149',
 '151', '152', '155', '163', '165', '169', '194', '196', '199', '202',
 '203', '205', '206', '207', '208', '210', '211', '212', '213', '216',
 '218', '219', '222', '224', '228', '234', '235', '236', '237', '238',
-- 
2.29.2

[PULL 21/53] iotests/129: Actually test a commit job

[Max Reitz]

Before this patch, test_block_commit() performs an active commit, which
under the hood is a mirror job.  If we want to test various different
block jobs, we should perhaps run an actual commit job instead.

Doing so requires adding an overlay above the source node before the
commit is done (and then specifying the source node as the top node for
the commit job).

Signed-off-by: Max Reitz 
Reviewed-by: Eric Blake 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Willian Rampazzo 
Message-Id: <20210118105720.14824-8-mre...@redhat.com>
---
 tests/qemu-iotests/129 | 27 +--
 1 file changed, 25 insertions(+), 2 deletions(-)

diff --git a/tests/qemu-iotests/129 b/tests/qemu-iotests/129
index 14c057f8ad..f4b447b04a 100755
--- a/tests/qemu-iotests/129
+++ b/tests/qemu-iotests/129
@@ -27,6 +27,7 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
 test_img = os.path.join(iotests.test_dir, 'test.img')
 target_img = os.path.join(iotests.test_dir, 'target.img')
 base_img = os.path.join(iotests.test_dir, 'base.img')
+overlay_img = os.path.join(iotests.test_dir, 'overlay.img')
 
 def setUp(self):
 iotests.qemu_img('create', '-f', iotests.imgfmt, self.base_img, "1G")
@@ -37,6 +38,7 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
 self.vm.add_object('throttle-group,id=tg0,x-bps-total=1024')
 
 source_drive = 'driver=throttle,' \
+   'node-name=source,' \
'throttle-group=tg0,' \
f'file.driver={iotests.imgfmt},' \
f'file.file.filename={self.test_img}'
@@ -46,7 +48,8 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
 
 def tearDown(self):
 self.vm.shutdown()
-for img in (self.test_img, self.target_img, self.base_img):
+for img in (self.test_img, self.target_img, self.base_img,
+self.overlay_img):
 iotests.try_remove(img)
 
 def do_test_stop(self, cmd, **args):
@@ -73,7 +76,27 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
   sync="full")
 
 def test_block_commit(self):
-self.do_test_stop("block-commit", device="drive0")
+# Add overlay above the source node so that we actually use a
+# commit job instead of a mirror job
+
+iotests.qemu_img('create', '-f', iotests.imgfmt, self.overlay_img,
+ '1G')
+
+result = self.vm.qmp('blockdev-add', **{
+ 'node-name': 'overlay',
+ 'driver': iotests.imgfmt,
+ 'file': {
+ 'driver': 'file',
+ 'filename': self.overlay_img
+ }
+ })
+self.assert_qmp(result, 'return', {})
+
+result = self.vm.qmp('blockdev-snapshot',
+ node='source', overlay='overlay')
+self.assert_qmp(result, 'return', {})
+
+self.do_test_stop('block-commit', device='drive0', top_node='source')
 
 if __name__ == '__main__':
 iotests.main(supported_fmts=["qcow2"],
-- 
2.29.2

[PULL 25/53] coroutine-sigaltstack: Add SIGUSR2 mutex

[Max Reitz]

Disposition (action) for any given signal is global for the process.
When two threads run coroutine-sigaltstack's qemu_coroutine_new()
concurrently, they may interfere with each other: One of them may revert
the SIGUSR2 handler to SIG_DFL, between the other thread (a) setting up
coroutine_trampoline() as the handler and (b) raising SIGUSR2.  That
SIGUSR2 will then terminate the QEMU process abnormally.

We have to ensure that only one thread at a time can modify the
process-global SIGUSR2 handler.  To do so, wrap the whole section where
that is done in a mutex.

Alternatively, we could for example have the SIGUSR2 handler always be
coroutine_trampoline(), so there would be no need to invoke sigaction()
in qemu_coroutine_new().  Laszlo has posted a patch to do so here:

  https://lists.nongnu.org/archive/html/qemu-devel/2021-01/msg05962.html

However, given that coroutine-sigaltstack is more of a fallback
implementation for platforms that do not support ucontext, that change
may be a bit too invasive to be comfortable with it.  The mutex proposed
here may negatively impact performance, but the change is much simpler.

Signed-off-by: Max Reitz 
Message-Id: <20210125120305.19520-1-mre...@redhat.com>
Reviewed-by: Laszlo Ersek 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
---
 util/coroutine-sigaltstack.c | 9 +
 1 file changed, 9 insertions(+)

diff --git a/util/coroutine-sigaltstack.c b/util/coroutine-sigaltstack.c
index aade82afb8..e99b8a4f9c 100644
--- a/util/coroutine-sigaltstack.c
+++ b/util/coroutine-sigaltstack.c
@@ -157,6 +157,7 @@ Coroutine *qemu_coroutine_new(void)
 sigset_t sigs;
 sigset_t osigs;
 sigjmp_buf old_env;
+static pthread_mutex_t sigusr2_mutex = PTHREAD_MUTEX_INITIALIZER;
 
 /* The way to manipulate stack is with the sigaltstack function. We
  * prepare a stack, with it delivering a signal to ourselves and then
@@ -186,6 +187,12 @@ Coroutine *qemu_coroutine_new(void)
 sa.sa_handler = coroutine_trampoline;
 sigfillset(_mask);
 sa.sa_flags = SA_ONSTACK;
+
+/*
+ * sigaction() is a process-global operation.  We must not run
+ * this code in multiple threads at once.
+ */
+pthread_mutex_lock(_mutex);
 if (sigaction(SIGUSR2, , ) != 0) {
 abort();
 }
@@ -234,6 +241,8 @@ Coroutine *qemu_coroutine_new(void)
  * Restore the old SIGUSR2 signal handler and mask
  */
 sigaction(SIGUSR2, , NULL);
+pthread_mutex_unlock(_mutex);
+
 pthread_sigmask(SIG_SETMASK, , NULL);
 
 /*
-- 
2.29.2

[PULL 20/53] iotests/129: Use throttle node

[Max Reitz]

Throttling on the BB has not affected block jobs in a while, so it is
possible that one of the jobs in 129 finishes before the VM is stopped.
We can fix that by running the job from a throttle node.

Signed-off-by: Max Reitz 
Reviewed-by: Eric Blake 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Willian Rampazzo 
Message-Id: <20210118105720.14824-7-mre...@redhat.com>
---
 tests/qemu-iotests/129 | 37 +
 1 file changed, 13 insertions(+), 24 deletions(-)

diff --git a/tests/qemu-iotests/129 b/tests/qemu-iotests/129
index 0f2e5418ef..14c057f8ad 100755
--- a/tests/qemu-iotests/129
+++ b/tests/qemu-iotests/129
@@ -33,20 +33,18 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
 iotests.qemu_img('create', '-f', iotests.imgfmt, self.test_img,
  "-b", self.base_img, '-F', iotests.imgfmt)
 iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x5d 1M 128M', 
self.test_img)
-self.vm = iotests.VM().add_drive(self.test_img)
+self.vm = iotests.VM()
+self.vm.add_object('throttle-group,id=tg0,x-bps-total=1024')
+
+source_drive = 'driver=throttle,' \
+   'throttle-group=tg0,' \
+   f'file.driver={iotests.imgfmt},' \
+   f'file.file.filename={self.test_img}'
+
+self.vm.add_drive(None, source_drive)
 self.vm.launch()
 
 def tearDown(self):
-params = {"device": "drive0",
-  "bps": 0,
-  "bps_rd": 0,
-  "bps_wr": 0,
-  "iops": 0,
-  "iops_rd": 0,
-  "iops_wr": 0,
- }
-result = self.vm.qmp("block_set_io_throttle", conv_keys=False,
- **params)
 self.vm.shutdown()
 for img in (self.test_img, self.target_img, self.base_img):
 iotests.try_remove(img)
@@ -54,33 +52,24 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
 def do_test_stop(self, cmd, **args):
 """Test 'stop' while block job is running on a throttled drive.
 The 'stop' command shouldn't drain the job"""
-params = {"device": "drive0",
-  "bps": 1024,
-  "bps_rd": 0,
-  "bps_wr": 0,
-  "iops": 0,
-  "iops_rd": 0,
-  "iops_wr": 0,
- }
-result = self.vm.qmp("block_set_io_throttle", conv_keys=False,
- **params)
-self.assert_qmp(result, 'return', {})
 result = self.vm.qmp(cmd, **args)
 self.assert_qmp(result, 'return', {})
+
 result = self.vm.qmp("stop")
 self.assert_qmp(result, 'return', {})
 result = self.vm.qmp("query-block-jobs")
+
 self.assert_qmp(result, 'return[0]/status', 'running')
 self.assert_qmp(result, 'return[0]/ready', False)
 
 def test_drive_mirror(self):
 self.do_test_stop("drive-mirror", device="drive0",
-  target=self.target_img,
+  target=self.target_img, format=iotests.imgfmt,
   sync="full")
 
 def test_drive_backup(self):
 self.do_test_stop("drive-backup", device="drive0",
-  target=self.target_img,
+  target=self.target_img, format=iotests.imgfmt,
   sync="full")
 
 def test_block_commit(self):
-- 
2.29.2

[PULL 33/53] blockjob: add set_speed to BlockJobDriver

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy 

We are going to use async block-copy call in backup, so we'll need to
passthrough setting backup speed to block-copy call.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20210116214705.822267-9-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 include/block/blockjob_int.h | 2 ++
 blockjob.c   | 6 ++
 2 files changed, 8 insertions(+)

diff --git a/include/block/blockjob_int.h b/include/block/blockjob_int.h
index e2824a36a8..6633d83da2 100644
--- a/include/block/blockjob_int.h
+++ b/include/block/blockjob_int.h
@@ -52,6 +52,8 @@ struct BlockJobDriver {
  * besides job->blk to the new AioContext.
  */
 void (*attached_aio_context)(BlockJob *job, AioContext *new_context);
+
+void (*set_speed)(BlockJob *job, int64_t speed);
 };
 
 /**
diff --git a/blockjob.c b/blockjob.c
index 98ac8af982..db3a21699c 100644
--- a/blockjob.c
+++ b/blockjob.c
@@ -256,6 +256,7 @@ static bool job_timer_pending(Job *job)
 
 void block_job_set_speed(BlockJob *job, int64_t speed, Error **errp)
 {
+const BlockJobDriver *drv = block_job_driver(job);
 int64_t old_speed = job->speed;
 
 if (job_apply_verb(>job, JOB_VERB_SET_SPEED, errp)) {
@@ -270,6 +271,11 @@ void block_job_set_speed(BlockJob *job, int64_t speed, 
Error **errp)
 ratelimit_set_speed(>limit, speed, BLOCK_JOB_SLICE_TIME);
 
 job->speed = speed;
+
+if (drv->set_speed) {
+drv->set_speed(job, speed);
+}
+
 if (speed && speed <= old_speed) {
 return;
 }
-- 
2.29.2

[PULL 15/53] iotests.py: Assume a couple of variables as given

[Max Reitz]

There are a couple of environment variables that we fetch with
os.environ.get() without supplying a default.  Clearly they are required
and expected to be set by the ./check script (as evidenced by
execute_setup_common(), which checks for test_dir and
qemu_default_machine to be set, and aborts if they are not).

Using .get() this way has the disadvantage of returning an Optional[str]
type, which mypy will complain about when tests just assume these values
to be str.

Use [] instead, which raises a KeyError for environment variables that
are not set.  When this exception is raised, catch it and move the abort
code from execute_setup_common() there.

Drop the 'assert iotests.sock_dir is not None' from iotest 300, because
that sort of thing is precisely what this patch wants to prevent.

Signed-off-by: Max Reitz 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Willian Rampazzo 
Message-Id: <20210118105720.14824-2-mre...@redhat.com>
---
 tests/qemu-iotests/300|  1 -
 tests/qemu-iotests/iotests.py | 26 +-
 2 files changed, 13 insertions(+), 14 deletions(-)

diff --git a/tests/qemu-iotests/300 b/tests/qemu-iotests/300
index 23aca59d9c..38ef5945b7 100755
--- a/tests/qemu-iotests/300
+++ b/tests/qemu-iotests/300
@@ -28,7 +28,6 @@ import qemu
 
 BlockBitmapMapping = List[Dict[str, Union[str, List[Dict[str, str]
 
-assert iotests.sock_dir is not None
 mig_sock = os.path.join(iotests.sock_dir, 'mig_sock')
 
 
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 2e89c0ab1a..45cb9bd288 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -75,12 +75,20 @@ qemu_opts = os.environ.get('QEMU_OPTIONS', 
'').strip().split(' ')
 
 imgfmt = os.environ.get('IMGFMT', 'raw')
 imgproto = os.environ.get('IMGPROTO', 'file')
-test_dir = os.environ.get('TEST_DIR')
-sock_dir = os.environ.get('SOCK_DIR')
 output_dir = os.environ.get('OUTPUT_DIR', '.')
-cachemode = os.environ.get('CACHEMODE')
-aiomode = os.environ.get('AIOMODE')
-qemu_default_machine = os.environ.get('QEMU_DEFAULT_MACHINE')
+
+try:
+test_dir = os.environ['TEST_DIR']
+sock_dir = os.environ['SOCK_DIR']
+cachemode = os.environ['CACHEMODE']
+aiomode = os.environ['AIOMODE']
+qemu_default_machine = os.environ['QEMU_DEFAULT_MACHINE']
+except KeyError:
+# We are using these variables as proxies to indicate that we're
+# not being run via "check". There may be other things set up by
+# "check" that individual test cases rely on.
+sys.stderr.write('Please run this test via the "check" script\n')
+sys.exit(os.EX_USAGE)
 
 socket_scm_helper = os.environ.get('SOCKET_SCM_HELPER', 'socket_scm_helper')
 
@@ -1286,14 +1294,6 @@ def execute_setup_common(supported_fmts: Sequence[str] = 
(),
 """
 # Note: Python 3.6 and pylint do not like 'Collection' so use 'Sequence'.
 
-# We are using TEST_DIR and QEMU_DEFAULT_MACHINE as proxies to
-# indicate that we're not being run via "check". There may be
-# other things set up by "check" that individual test cases rely
-# on.
-if test_dir is None or qemu_default_machine is None:
-sys.stderr.write('Please run this test via the "check" script\n')
-sys.exit(os.EX_USAGE)
-
 debug = '-d' in sys.argv
 if debug:
 sys.argv.remove('-d')
-- 
2.29.2

[PULL 30/53] block/block-copy: add list of all call-states

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy 

It simplifies debugging.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20210116214705.822267-6-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 block/block-copy.c | 11 ++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/block/block-copy.c b/block/block-copy.c
index 35213bd832..6bf1735b93 100644
--- a/block/block-copy.c
+++ b/block/block-copy.c
@@ -42,6 +42,9 @@ typedef struct BlockCopyCallState {
 /* Coroutine where async block-copy is running */
 Coroutine *co;
 
+/* To reference all call states from BlockCopyState */
+QLIST_ENTRY(BlockCopyCallState) list;
+
 /* State */
 int ret;
 bool finished;
@@ -81,7 +84,8 @@ typedef struct BlockCopyState {
 bool use_copy_range;
 int64_t copy_size;
 uint64_t len;
-QLIST_HEAD(, BlockCopyTask) tasks;
+QLIST_HEAD(, BlockCopyTask) tasks; /* All tasks from all block-copy calls 
*/
+QLIST_HEAD(, BlockCopyCallState) calls;
 
 BdrvRequestFlags write_flags;
 
@@ -282,6 +286,7 @@ BlockCopyState *block_copy_state_new(BdrvChild *source, 
BdrvChild *target,
 }
 
 QLIST_INIT(>tasks);
+QLIST_INIT(>calls);
 
 return s;
 }
@@ -669,6 +674,8 @@ static int coroutine_fn 
block_copy_common(BlockCopyCallState *call_state)
 {
 int ret;
 
+QLIST_INSERT_HEAD(_state->s->calls, call_state, list);
+
 do {
 ret = block_copy_dirty_clusters(call_state);
 
@@ -694,6 +701,8 @@ static int coroutine_fn 
block_copy_common(BlockCopyCallState *call_state)
 call_state->cb(call_state->cb_opaque);
 }
 
+QLIST_REMOVE(call_state, list);
+
 return ret;
 }
 
-- 
2.29.2

[PULL 18/53] iotests/129: Remove test images in tearDown()

[Max Reitz]

Signed-off-by: Max Reitz 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Eric Blake 
Reviewed-by: Willian Rampazzo 
Message-Id: <20210118105720.14824-5-mre...@redhat.com>
---
 tests/qemu-iotests/129 | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tests/qemu-iotests/129 b/tests/qemu-iotests/129
index f57a2e19f6..bd29c54af8 100755
--- a/tests/qemu-iotests/129
+++ b/tests/qemu-iotests/129
@@ -48,6 +48,8 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
 result = self.vm.qmp("block_set_io_throttle", conv_keys=False,
  **params)
 self.vm.shutdown()
+for img in (self.test_img, self.target_img, self.base_img):
+iotests.try_remove(img)
 
 def do_test_stop(self, cmd, **args):
 """Test 'stop' while block job is running on a throttled drive.
-- 
2.29.2

[PULL 17/53] iotests: Move try_remove to iotests.py

[Max Reitz]

Signed-off-by: Max Reitz 
Reviewed-by: Eric Blake 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Willian Rampazzo 
Message-Id: <20210118105720.14824-4-mre...@redhat.com>
---
 tests/qemu-iotests/124|  8 +---
 tests/qemu-iotests/iotests.py | 11 +++
 2 files changed, 8 insertions(+), 11 deletions(-)

diff --git a/tests/qemu-iotests/124 b/tests/qemu-iotests/124
index 3b21bc497f..90cdbd8e24 100755
--- a/tests/qemu-iotests/124
+++ b/tests/qemu-iotests/124
@@ -23,6 +23,7 @@
 
 import os
 import iotests
+from iotests import try_remove
 
 
 def io_write_patterns(img, patterns):
@@ -30,13 +31,6 @@ def io_write_patterns(img, patterns):
 iotests.qemu_io('-c', 'write -P%s %s %s' % pattern, img)
 
 
-def try_remove(img):
-try:
-os.remove(img)
-except OSError:
-pass
-
-
 def transaction_action(action, **kwargs):
 return {
 'type': action,
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 45cb9bd288..335e6feb70 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -515,12 +515,15 @@ class FilePath:
 return False
 
 
+def try_remove(img):
+try:
+os.remove(img)
+except OSError:
+pass
+
 def file_path_remover():
 for path in reversed(file_path_remover.paths):
-try:
-os.remove(path)
-except OSError:
-pass
+try_remove(path)
 
 
 def file_path(*names, base_dir=test_dir):
-- 
2.29.2

[PULL 12/53] iotests: 30: prepare to COR filter insertion by stream job

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy 

test_stream_parallel run parallel stream jobs, intersecting so that top
of one is base of another. It's OK now, but it would be a problem if
insert the filter, as one job will want to use another job's filter as
above_base node.

Correct thing to do is move to new interface: "bottom" argument instead
of base. This guarantees that jobs don't intersect by their actions.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20201216061703.70908-12-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 tests/qemu-iotests/030 | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/030 b/tests/qemu-iotests/030
index 890784b116..f8a692432c 100755
--- a/tests/qemu-iotests/030
+++ b/tests/qemu-iotests/030
@@ -246,7 +246,9 @@ class TestParallelOps(iotests.QMPTestCase):
 node_name = 'node%d' % i
 job_id = 'stream-%s' % node_name
 pending_jobs.append(job_id)
-result = self.vm.qmp('block-stream', device=node_name, 
job_id=job_id, base=self.imgs[i-2], speed=1024)
+result = self.vm.qmp('block-stream', device=node_name,
+ job_id=job_id, bottom=f'node{i-1}',
+ speed=1024)
 self.assert_qmp(result, 'return', {})
 
 for job in pending_jobs:
-- 
2.29.2

[PULL 24/53] iotests/300: Clean up pylint and mypy complaints

[Max Reitz]

And consequentially drop it from 297's skip list.

Signed-off-by: Max Reitz 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Willian Rampazzo 
Message-Id: <20210118105720.14824-11-mre...@redhat.com>
---
 tests/qemu-iotests/297 |  2 +-
 tests/qemu-iotests/300 | 18 +++---
 2 files changed, 16 insertions(+), 4 deletions(-)

diff --git a/tests/qemu-iotests/297 b/tests/qemu-iotests/297
index 79e63f8625..a37910b42d 100755
--- a/tests/qemu-iotests/297
+++ b/tests/qemu-iotests/297
@@ -34,7 +34,7 @@ SKIP_FILES = (
 '218', '219', '222', '224', '228', '234', '235', '236', '237', '238',
 '240', '242', '245', '246', '248', '255', '256', '257', '258', '260',
 '262', '264', '266', '274', '277', '280', '281', '295', '296', '298',
-'299', '300', '302', '303', '304', '307',
+'299', '302', '303', '304', '307',
 'nbd-fault-injector.py', 'qcow2.py', 'qcow2_format.py', 'qed.py'
 )
 
diff --git a/tests/qemu-iotests/300 b/tests/qemu-iotests/300
index 38ef5945b7..43264d883d 100755
--- a/tests/qemu-iotests/300
+++ b/tests/qemu-iotests/300
@@ -23,7 +23,11 @@ import os
 import random
 import re
 from typing import Dict, List, Optional, Union
+
 import iotests
+
+# Import qemu after iotests.py has amended sys.path
+# pylint: disable=wrong-import-order
 import qemu
 
 BlockBitmapMapping = List[Dict[str, Union[str, List[Dict[str, str]
@@ -111,10 +115,14 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
 If @msg is None, check that there has not been any error.
 """
 self.vm_b.shutdown()
+
+log = self.vm_b.get_log()
+assert log is not None  # Loaded after shutdown
+
 if msg is None:
-self.assertNotIn('qemu-system-', self.vm_b.get_log())
+self.assertNotIn('qemu-system-', log)
 else:
-self.assertIn(msg, self.vm_b.get_log())
+self.assertIn(msg, log)
 
 @staticmethod
 def mapping(node_name: str, node_alias: str,
@@ -446,9 +454,13 @@ class 
TestBlockBitmapMappingErrors(TestDirtyBitmapMigration):
 
 # Check for the error in the source's log
 self.vm_a.shutdown()
+
+log = self.vm_a.get_log()
+assert log is not None  # Loaded after shutdown
+
 self.assertIn(f"Cannot migrate bitmap '{name}' on node "
   f"'{self.src_node_name}': Name is longer than 255 bytes",
-  self.vm_a.get_log())
+  log)
 
 # Expect abnormal shutdown of the destination VM because of
 # the failed migration
-- 
2.29.2

[PULL 13/53] block/stream: add s->target_bs

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy 

Add a direct link to target bs for convenience and to simplify
following commit which will insert COR filter above target bs.

This is a part of original commit written by Andrey.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20201216061703.70908-13-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 block/stream.c | 23 ++-
 1 file changed, 10 insertions(+), 13 deletions(-)

diff --git a/block/stream.c b/block/stream.c
index 045d6bc76b..626dfa2b22 100644
--- a/block/stream.c
+++ b/block/stream.c
@@ -33,6 +33,7 @@ typedef struct StreamBlockJob {
 BlockJob common;
 BlockDriverState *base_overlay; /* COW overlay (stream from this) */
 BlockDriverState *above_base;   /* Node directly above the base */
+BlockDriverState *target_bs;
 BlockdevOnError on_error;
 char *backing_file_str;
 bool bs_read_only;
@@ -53,23 +54,20 @@ static void stream_abort(Job *job)
 StreamBlockJob *s = container_of(job, StreamBlockJob, common.job);
 
 if (s->chain_frozen) {
-BlockJob *bjob = >common;
-bdrv_unfreeze_backing_chain(blk_bs(bjob->blk), s->above_base);
+bdrv_unfreeze_backing_chain(s->target_bs, s->above_base);
 }
 }
 
 static int stream_prepare(Job *job)
 {
 StreamBlockJob *s = container_of(job, StreamBlockJob, common.job);
-BlockJob *bjob = >common;
-BlockDriverState *bs = blk_bs(bjob->blk);
-BlockDriverState *unfiltered_bs = bdrv_skip_filters(bs);
+BlockDriverState *unfiltered_bs = bdrv_skip_filters(s->target_bs);
 BlockDriverState *base = bdrv_filter_or_cow_bs(s->above_base);
 BlockDriverState *unfiltered_base = bdrv_skip_filters(base);
 Error *local_err = NULL;
 int ret = 0;
 
-bdrv_unfreeze_backing_chain(bs, s->above_base);
+bdrv_unfreeze_backing_chain(s->target_bs, s->above_base);
 s->chain_frozen = false;
 
 if (bdrv_cow_child(unfiltered_bs)) {
@@ -95,13 +93,12 @@ static void stream_clean(Job *job)
 {
 StreamBlockJob *s = container_of(job, StreamBlockJob, common.job);
 BlockJob *bjob = >common;
-BlockDriverState *bs = blk_bs(bjob->blk);
 
 /* Reopen the image back in read-only mode if necessary */
 if (s->bs_read_only) {
 /* Give up write permissions before making it read-only */
 blk_set_perm(bjob->blk, 0, BLK_PERM_ALL, _abort);
-bdrv_reopen_set_read_only(bs, true, NULL);
+bdrv_reopen_set_read_only(s->target_bs, true, NULL);
 }
 
 g_free(s->backing_file_str);
@@ -111,8 +108,7 @@ static int coroutine_fn stream_run(Job *job, Error **errp)
 {
 StreamBlockJob *s = container_of(job, StreamBlockJob, common.job);
 BlockBackend *blk = s->common.blk;
-BlockDriverState *bs = blk_bs(blk);
-BlockDriverState *unfiltered_bs = bdrv_skip_filters(bs);
+BlockDriverState *unfiltered_bs = bdrv_skip_filters(s->target_bs);
 bool enable_cor = !bdrv_cow_child(s->base_overlay);
 int64_t len;
 int64_t offset = 0;
@@ -125,7 +121,7 @@ static int coroutine_fn stream_run(Job *job, Error **errp)
 return 0;
 }
 
-len = bdrv_getlength(bs);
+len = bdrv_getlength(s->target_bs);
 if (len < 0) {
 return len;
 }
@@ -137,7 +133,7 @@ static int coroutine_fn stream_run(Job *job, Error **errp)
  * account.
  */
 if (enable_cor) {
-bdrv_enable_copy_on_read(bs);
+bdrv_enable_copy_on_read(s->target_bs);
 }
 
 for ( ; offset < len; offset += n) {
@@ -199,7 +195,7 @@ static int coroutine_fn stream_run(Job *job, Error **errp)
 }
 
 if (enable_cor) {
-bdrv_disable_copy_on_read(bs);
+bdrv_disable_copy_on_read(s->target_bs);
 }
 
 /* Do not remove the backing file if an error was there but ignored. */
@@ -314,6 +310,7 @@ void stream_start(const char *job_id, BlockDriverState *bs,
 s->base_overlay = base_overlay;
 s->above_base = above_base;
 s->backing_file_str = g_strdup(backing_file_str);
+s->target_bs = bs;
 s->bs_read_only = bs_read_only;
 s->chain_frozen = true;
 
-- 
2.29.2

[PULL 14/53] block: apply COR-filter to block-stream jobs

[Max Reitz]

From: Andrey Shinkevich 

This patch completes the series with the COR-filter applied to
block-stream operations.

Adding the filter makes it possible in future implement discarding
copied regions in backing files during the block-stream job, to reduce
the disk overuse (we need control on permissions).

Also, the filter now is smart enough to do copy-on-read with specified
base, so we have benefit on guest reads even when doing block-stream of
the part of the backing chain.

Several iotests are slightly modified due to filter insertion.

Signed-off-by: Andrey Shinkevich 
Signed-off-by: Vladimir Sementsov-Ogievskiy 
Message-Id: <20201216061703.70908-14-vsement...@virtuozzo.com>
Reviewed-by: Max Reitz 
Signed-off-by: Max Reitz 
---
 block/stream.c | 105 ++---
 tests/qemu-iotests/030 |   8 +--
 tests/qemu-iotests/141.out |   2 +-
 tests/qemu-iotests/245 |  20 ---
 4 files changed, 80 insertions(+), 55 deletions(-)

diff --git a/block/stream.c b/block/stream.c
index 626dfa2b22..1fa742b0db 100644
--- a/block/stream.c
+++ b/block/stream.c
@@ -17,8 +17,10 @@
 #include "block/blockjob_int.h"
 #include "qapi/error.h"
 #include "qapi/qmp/qerror.h"
+#include "qapi/qmp/qdict.h"
 #include "qemu/ratelimit.h"
 #include "sysemu/block-backend.h"
+#include "block/copy-on-read.h"
 
 enum {
 /*
@@ -33,11 +35,11 @@ typedef struct StreamBlockJob {
 BlockJob common;
 BlockDriverState *base_overlay; /* COW overlay (stream from this) */
 BlockDriverState *above_base;   /* Node directly above the base */
+BlockDriverState *cor_filter_bs;
 BlockDriverState *target_bs;
 BlockdevOnError on_error;
 char *backing_file_str;
 bool bs_read_only;
-bool chain_frozen;
 } StreamBlockJob;
 
 static int coroutine_fn stream_populate(BlockBackend *blk,
@@ -45,17 +47,7 @@ static int coroutine_fn stream_populate(BlockBackend *blk,
 {
 assert(bytes < SIZE_MAX);
 
-return blk_co_preadv(blk, offset, bytes, NULL,
- BDRV_REQ_COPY_ON_READ | BDRV_REQ_PREFETCH);
-}
-
-static void stream_abort(Job *job)
-{
-StreamBlockJob *s = container_of(job, StreamBlockJob, common.job);
-
-if (s->chain_frozen) {
-bdrv_unfreeze_backing_chain(s->target_bs, s->above_base);
-}
+return blk_co_preadv(blk, offset, bytes, NULL, BDRV_REQ_PREFETCH);
 }
 
 static int stream_prepare(Job *job)
@@ -67,8 +59,9 @@ static int stream_prepare(Job *job)
 Error *local_err = NULL;
 int ret = 0;
 
-bdrv_unfreeze_backing_chain(s->target_bs, s->above_base);
-s->chain_frozen = false;
+/* We should drop filter at this point, as filter hold the backing chain */
+bdrv_cor_filter_drop(s->cor_filter_bs);
+s->cor_filter_bs = NULL;
 
 if (bdrv_cow_child(unfiltered_bs)) {
 const char *base_id = NULL, *base_fmt = NULL;
@@ -94,6 +87,11 @@ static void stream_clean(Job *job)
 StreamBlockJob *s = container_of(job, StreamBlockJob, common.job);
 BlockJob *bjob = >common;
 
+if (s->cor_filter_bs) {
+bdrv_cor_filter_drop(s->cor_filter_bs);
+s->cor_filter_bs = NULL;
+}
+
 /* Reopen the image back in read-only mode if necessary */
 if (s->bs_read_only) {
 /* Give up write permissions before making it read-only */
@@ -109,7 +107,6 @@ static int coroutine_fn stream_run(Job *job, Error **errp)
 StreamBlockJob *s = container_of(job, StreamBlockJob, common.job);
 BlockBackend *blk = s->common.blk;
 BlockDriverState *unfiltered_bs = bdrv_skip_filters(s->target_bs);
-bool enable_cor = !bdrv_cow_child(s->base_overlay);
 int64_t len;
 int64_t offset = 0;
 uint64_t delay_ns = 0;
@@ -127,15 +124,6 @@ static int coroutine_fn stream_run(Job *job, Error **errp)
 }
 job_progress_set_remaining(>common.job, len);
 
-/* Turn on copy-on-read for the whole block device so that guest read
- * requests help us make progress.  Only do this when copying the entire
- * backing chain since the copy-on-read operation does not take base into
- * account.
- */
-if (enable_cor) {
-bdrv_enable_copy_on_read(s->target_bs);
-}
-
 for ( ; offset < len; offset += n) {
 bool copy;
 int ret;
@@ -194,10 +182,6 @@ static int coroutine_fn stream_run(Job *job, Error **errp)
 }
 }
 
-if (enable_cor) {
-bdrv_disable_copy_on_read(s->target_bs);
-}
-
 /* Do not remove the backing file if an error was there but ignored. */
 return error;
 }
@@ -209,7 +193,6 @@ static const BlockJobDriver stream_job_driver = {
 .free  = block_job_free,
 .run   = stream_run,
 .prepare   = stream_prepare,
-.abort = stream_abort,
 .clean = stream_clean,
 .user_resume   = block_job_user_resume,
 },
@@ -228,7 +211,9 @@ void stream_start(const char *job_id, BlockDriverState *bs,
 bool bs_read_only;
 int basic_flags = 

[PULL 22/53] iotests/129: Limit mirror job's buffer size

[Max Reitz]

Issuing 'stop' on the VM drains all nodes.  If the mirror job has many
large requests in flight, this may lead to significant I/O that looks a
bit like 'stop' would make the job try to complete (which is what 129
should verify not to happen).

We can limit the I/O in flight by limiting the buffer size, so mirror
will make very little progress during the 'stop' drain.

(We do not need to do anything about commit, which has a buffer size of
512 kB by default; or backup, which goes cluster by cluster.  Once we
have asynchronous requests for backup, that will change, but then we can
fine-tune the backup job to only perform a single request on a very
small chunk, too.)

Signed-off-by: Max Reitz 
Reviewed-by: Eric Blake 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Willian Rampazzo 
Message-Id: <20210118105720.14824-9-mre...@redhat.com>
---
 tests/qemu-iotests/129 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/129 b/tests/qemu-iotests/129
index f4b447b04a..03843fc255 100755
--- a/tests/qemu-iotests/129
+++ b/tests/qemu-iotests/129
@@ -68,7 +68,7 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
 def test_drive_mirror(self):
 self.do_test_stop("drive-mirror", device="drive0",
   target=self.target_img, format=iotests.imgfmt,
-  sync="full")
+  sync="full", buf_size=65536)
 
 def test_drive_backup(self):
 self.do_test_stop("drive-backup", device="drive0",
-- 
2.29.2

[PULL 08/53] block: include supported_read_flags into BDS structure

[Max Reitz]

From: Andrey Shinkevich 

Add the new member supported_read_flags to the BlockDriverState
structure. It will control the flags set for copy-on-read operations.
Make the block generic layer evaluate supported read flags before they
go to a block driver.

Suggested-by: Vladimir Sementsov-Ogievskiy 
Signed-off-by: Andrey Shinkevich 
Signed-off-by: Vladimir Sementsov-Ogievskiy 
 [vsementsov: use assert instead of abort]
Reviewed-by: Max Reitz 
Message-Id: <20201216061703.70908-8-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 include/block/block_int.h |  4 
 block/io.c| 10 --
 2 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/include/block/block_int.h b/include/block/block_int.h
index f3d0a0e60a..92d3754ead 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -881,6 +881,10 @@ struct BlockDriverState {
 /* I/O Limits */
 BlockLimits bl;
 
+/*
+ * Flags honored during pread
+ */
+unsigned int supported_read_flags;
 /* Flags honored during pwrite (so far: BDRV_REQ_FUA,
  * BDRV_REQ_WRITE_UNCHANGED).
  * If a driver does not support BDRV_REQ_WRITE_UNCHANGED, those
diff --git a/block/io.c b/block/io.c
index 95b1c56c06..d203435a73 100644
--- a/block/io.c
+++ b/block/io.c
@@ -1453,6 +1453,9 @@ static int coroutine_fn bdrv_aligned_preadv(BdrvChild 
*child,
 if (flags & BDRV_REQ_COPY_ON_READ) {
 int64_t pnum;
 
+/* The flag BDRV_REQ_COPY_ON_READ has reached its addressee */
+flags &= ~BDRV_REQ_COPY_ON_READ;
+
 ret = bdrv_is_allocated(bs, offset, bytes, );
 if (ret < 0) {
 goto out;
@@ -1474,9 +1477,11 @@ static int coroutine_fn bdrv_aligned_preadv(BdrvChild 
*child,
 goto out;
 }
 
+assert(!(flags & ~bs->supported_read_flags));
+
 max_bytes = ROUND_UP(MAX(0, total_bytes - offset), align);
 if (bytes <= max_bytes && bytes <= max_transfer) {
-ret = bdrv_driver_preadv(bs, offset, bytes, qiov, qiov_offset, 0);
+ret = bdrv_driver_preadv(bs, offset, bytes, qiov, qiov_offset, flags);
 goto out;
 }
 
@@ -1489,7 +1494,8 @@ static int coroutine_fn bdrv_aligned_preadv(BdrvChild 
*child,
 
 ret = bdrv_driver_preadv(bs, offset + bytes - bytes_remaining,
  num, qiov,
- qiov_offset + bytes - bytes_remaining, 0);
+ qiov_offset + bytes - bytes_remaining,
+ flags);
 max_bytes -= num;
 } else {
 num = bytes_remaining;
-- 
2.29.2

[PULL 06/53] qapi: copy-on-read filter: add 'bottom' option

[Max Reitz]

From: Andrey Shinkevich 

Add an option to limit copy-on-read operations to specified sub-chain
of backing-chain, to make copy-on-read filter useful for block-stream
job.

Suggested-by: Max Reitz 
Suggested-by: Vladimir Sementsov-Ogievskiy 
Signed-off-by: Andrey Shinkevich 
Signed-off-by: Vladimir Sementsov-Ogievskiy 
  [vsementsov: change subject, modified to freeze the chain,
   do some fixes]
Message-Id: <20201216061703.70908-6-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 qapi/block-core.json | 20 -
 block/copy-on-read.c | 98 +++-
 2 files changed, 115 insertions(+), 3 deletions(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index b55732d802..65167ebf56 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -3959,6 +3959,24 @@
   'data': { 'throttle-group': 'str',
 'file' : 'BlockdevRef'
  } }
+
+##
+# @BlockdevOptionsCor:
+#
+# Driver specific block device options for the copy-on-read driver.
+#
+# @bottom: The name of a non-filter node (allocation-bearing layer) that
+#  limits the COR operations in the backing chain (inclusive), so
+#  that no data below this node will be copied by this filter.
+#  If option is absent, the limit is not applied, so that data
+#  from all backing layers may be copied.
+#
+# Since: 6.0
+##
+{ 'struct': 'BlockdevOptionsCor',
+  'base': 'BlockdevOptionsGenericFormat',
+  'data': { '*bottom': 'str' } }
+
 ##
 # @BlockdevOptions:
 #
@@ -4011,7 +4029,7 @@
   'bochs':  'BlockdevOptionsGenericFormat',
   'cloop':  'BlockdevOptionsGenericFormat',
   'compress':   'BlockdevOptionsGenericFormat',
-  'copy-on-read':'BlockdevOptionsGenericFormat',
+  'copy-on-read':'BlockdevOptionsCor',
   'dmg':'BlockdevOptionsGenericFormat',
   'file':   'BlockdevOptionsFile',
   'ftp':'BlockdevOptionsCurlFtp',
diff --git a/block/copy-on-read.c b/block/copy-on-read.c
index 618c4c4f43..71560984f6 100644
--- a/block/copy-on-read.c
+++ b/block/copy-on-read.c
@@ -24,18 +24,24 @@
 #include "block/block_int.h"
 #include "qemu/module.h"
 #include "qapi/error.h"
+#include "qapi/qmp/qdict.h"
 #include "block/copy-on-read.h"
 
 
 typedef struct BDRVStateCOR {
 bool active;
+BlockDriverState *bottom_bs;
+bool chain_frozen;
 } BDRVStateCOR;
 
 
 static int cor_open(BlockDriverState *bs, QDict *options, int flags,
 Error **errp)
 {
+BlockDriverState *bottom_bs = NULL;
 BDRVStateCOR *state = bs->opaque;
+/* Find a bottom node name, if any */
+const char *bottom_node = qdict_get_try_str(options, "bottom");
 
 bs->file = bdrv_open_child(NULL, options, "file", bs, _of_bds,
BDRV_CHILD_FILTERED | BDRV_CHILD_PRIMARY,
@@ -51,7 +57,38 @@ static int cor_open(BlockDriverState *bs, QDict *options, 
int flags,
 ((BDRV_REQ_FUA | BDRV_REQ_MAY_UNMAP | BDRV_REQ_NO_FALLBACK) &
 bs->file->bs->supported_zero_flags);
 
+if (bottom_node) {
+bottom_bs = bdrv_find_node(bottom_node);
+if (!bottom_bs) {
+error_setg(errp, "Bottom node '%s' not found", bottom_node);
+qdict_del(options, "bottom");
+return -EINVAL;
+}
+qdict_del(options, "bottom");
+
+if (!bottom_bs->drv) {
+error_setg(errp, "Bottom node '%s' not opened", bottom_node);
+return -EINVAL;
+}
+
+if (bottom_bs->drv->is_filter) {
+error_setg(errp, "Bottom node '%s' is a filter", bottom_node);
+return -EINVAL;
+}
+
+if (bdrv_freeze_backing_chain(bs, bottom_bs, errp) < 0) {
+return -EINVAL;
+}
+state->chain_frozen = true;
+
+/*
+ * We do freeze the chain, so it shouldn't be removed. Still, storing a
+ * pointer worth bdrv_ref().
+ */
+bdrv_ref(bottom_bs);
+}
 state->active = true;
+state->bottom_bs = bottom_bs;
 
 /*
  * We don't need to call bdrv_child_refresh_perms() now as the permissions
@@ -107,8 +144,46 @@ static int coroutine_fn 
cor_co_preadv_part(BlockDriverState *bs,
size_t qiov_offset,
int flags)
 {
-return bdrv_co_preadv_part(bs->file, offset, bytes, qiov, qiov_offset,
-   flags | BDRV_REQ_COPY_ON_READ);
+int64_t n;
+int local_flags;
+int ret;
+BDRVStateCOR *state = bs->opaque;
+
+if (!state->bottom_bs) {
+return bdrv_co_preadv_part(bs->file, offset, bytes, qiov, qiov_offset,
+   flags | BDRV_REQ_COPY_ON_READ);
+}
+
+while (bytes) {
+local_flags = flags;
+
+/* In case of failure, try to copy-on-read anyway */
+ret = bdrv_is_allocated(bs->file->bs, offset, bytes, );
+if (ret <= 0) {
+ret = 

[PULL 07/53] iotests: add #310 to test bottom node in COR driver

[Max Reitz]

From: Andrey Shinkevich 

The test case #310 is similar to #216 by Max Reitz. The difference is
that the test #310 involves a bottom node to the COR filter driver.

Signed-off-by: Andrey Shinkevich 
Signed-off-by: Vladimir Sementsov-Ogievskiy 
  [vsementsov: detach backing to test reads from top, limit to qcow2]
Message-Id: <20201216061703.70908-7-vsement...@virtuozzo.com>
[mreitz: Add "# group:" line]
Signed-off-by: Max Reitz 
---
 tests/qemu-iotests/310 | 117 +
 tests/qemu-iotests/310.out |  15 +
 tests/qemu-iotests/group   |   1 +
 3 files changed, 133 insertions(+)
 create mode 100755 tests/qemu-iotests/310
 create mode 100644 tests/qemu-iotests/310.out

diff --git a/tests/qemu-iotests/310 b/tests/qemu-iotests/310
new file mode 100755
index 00..9d9c928c4b
--- /dev/null
+++ b/tests/qemu-iotests/310
@@ -0,0 +1,117 @@
+#!/usr/bin/env python3
+# group: rw quick
+#
+# Copy-on-read tests using a COR filter with a bottom node
+#
+# Copyright (C) 2018 Red Hat, Inc.
+# Copyright (c) 2020 Virtuozzo International GmbH
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see .
+#
+
+import iotests
+from iotests import log, qemu_img, qemu_io_silent
+
+# Need backing file support
+iotests.script_initialize(supported_fmts=['qcow2'],
+  supported_platforms=['linux'])
+
+log('')
+log('=== Copy-on-read across nodes ===')
+log('')
+
+# This test is similar to the 216 one by Max Reitz 
+# The difference is that this test case involves a bottom node to the
+# COR filter driver.
+
+with iotests.FilePath('base.img') as base_img_path, \
+ iotests.FilePath('mid.img') as mid_img_path, \
+ iotests.FilePath('top.img') as top_img_path, \
+ iotests.VM() as vm:
+
+log('--- Setting up images ---')
+log('')
+
+assert qemu_img('create', '-f', iotests.imgfmt, base_img_path, '64M') == 0
+assert qemu_io_silent(base_img_path, '-c', 'write -P 1 0M 1M') == 0
+assert qemu_io_silent(base_img_path, '-c', 'write -P 1 3M 1M') == 0
+assert qemu_img('create', '-f', iotests.imgfmt, '-b', base_img_path,
+'-F', iotests.imgfmt, mid_img_path) == 0
+assert qemu_io_silent(mid_img_path,  '-c', 'write -P 3 2M 1M') == 0
+assert qemu_io_silent(mid_img_path,  '-c', 'write -P 3 4M 1M') == 0
+assert qemu_img('create', '-f', iotests.imgfmt, '-b', mid_img_path,
+'-F', iotests.imgfmt, top_img_path) == 0
+assert qemu_io_silent(top_img_path,  '-c', 'write -P 2 1M 1M') == 0
+
+#  0 1 2 3 4
+# top2
+# mid  3   3
+# base 1 1
+
+log('Done')
+
+log('')
+log('--- Doing COR ---')
+log('')
+
+vm.launch()
+
+log(vm.qmp('blockdev-add',
+   node_name='node0',
+   driver='copy-on-read',
+   bottom='node2',
+   file={
+   'driver': iotests.imgfmt,
+   'file': {
+   'driver': 'file',
+   'filename': top_img_path
+   },
+   'backing': {
+   'node-name': 'node2',
+   'driver': iotests.imgfmt,
+   'file': {
+   'driver': 'file',
+   'filename': mid_img_path
+   },
+   'backing': {
+   'driver': iotests.imgfmt,
+   'file': {
+   'driver': 'file',
+   'filename': base_img_path
+   }
+   },
+   }
+   }))
+
+# Trigger COR
+log(vm.qmp('human-monitor-command',
+   command_line='qemu-io node0 "read 0 5M"'))
+
+vm.shutdown()
+
+log('')
+log('--- Checking COR result ---')
+log('')
+
+# Detach backing to check that we can read the data from the top level now
+assert qemu_img('rebase', '-u', '-b', '', '-f', iotests.imgfmt,
+top_img_path) == 0
+
+assert qemu_io_silent(top_img_path,  '-c', 'read -P 0 0 1M') == 0
+assert qemu_io_silent(top_img_path,  '-c', 'read -P 2 1M 1M') == 0
+assert qemu_io_silent(top_img_path,  '-c', 'read -P 3 2M 1M') == 0
+assert qemu_io_silent(top_img_path,  '-c', 'read -P 0 3M 1M') == 0
+assert qemu_io_silent(top_img_path,  '-c', 'read -P 

[PULL 09/53] copy-on-read: skip non-guest reads if no copy needed

[Max Reitz]

From: Andrey Shinkevich 

If the flag BDRV_REQ_PREFETCH was set, skip idling read/write
operations in COR-driver. It can be taken into account for the
COR-algorithms optimization. That check is being made during the
block stream job by the moment.

Add the BDRV_REQ_PREFETCH flag to the supported_read_flags of the
COR-filter.

block: Modify the comment for the flag BDRV_REQ_PREFETCH as we are
going to use it alone and pass it to the COR-filter driver for further
processing.

Signed-off-by: Andrey Shinkevich 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20201216061703.70908-9-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 include/block/block.h |  8 +---
 block/copy-on-read.c  | 14 ++
 2 files changed, 15 insertions(+), 7 deletions(-)

diff --git a/include/block/block.h b/include/block/block.h
index 127bdf3392..81fcaad5ac 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -72,9 +72,11 @@ typedef enum {
 BDRV_REQ_NO_FALLBACK= 0x100,
 
 /*
- * BDRV_REQ_PREFETCH may be used only together with BDRV_REQ_COPY_ON_READ
- * on read request and means that caller doesn't really need data to be
- * written to qiov parameter which may be NULL.
+ * BDRV_REQ_PREFETCH makes sense only in the context of copy-on-read
+ * (i.e., together with the BDRV_REQ_COPY_ON_READ flag or when a COR
+ * filter is involved), in which case it signals that the COR operation
+ * need not read the data into memory (qiov) but only ensure they are
+ * copied to the top layer (i.e., that COR operation is done).
  */
 BDRV_REQ_PREFETCH  = 0x200,
 
diff --git a/block/copy-on-read.c b/block/copy-on-read.c
index 71560984f6..9cad9e1b8c 100644
--- a/block/copy-on-read.c
+++ b/block/copy-on-read.c
@@ -50,6 +50,8 @@ static int cor_open(BlockDriverState *bs, QDict *options, int 
flags,
 return -EINVAL;
 }
 
+bs->supported_read_flags = BDRV_REQ_PREFETCH;
+
 bs->supported_write_flags = BDRV_REQ_WRITE_UNCHANGED |
 (BDRV_REQ_FUA & bs->file->bs->supported_write_flags);
 
@@ -172,10 +174,14 @@ static int coroutine_fn 
cor_co_preadv_part(BlockDriverState *bs,
 }
 }
 
-ret = bdrv_co_preadv_part(bs->file, offset, n, qiov, qiov_offset,
-  local_flags);
-if (ret < 0) {
-return ret;
+/* Skip if neither read nor write are needed */
+if ((local_flags & (BDRV_REQ_PREFETCH | BDRV_REQ_COPY_ON_READ)) !=
+BDRV_REQ_PREFETCH) {
+ret = bdrv_co_preadv_part(bs->file, offset, n, qiov, qiov_offset,
+  local_flags);
+if (ret < 0) {
+return ret;
+}
 }
 
 offset += n;
-- 
2.29.2

[PULL 05/53] qapi: add filter-node-name to block-stream

[Max Reitz]

From: Andrey Shinkevich 

Provide the possibility to pass the 'filter-node-name' parameter to the
block-stream job as it is done for the commit block job.

Signed-off-by: Andrey Shinkevich 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
  [vsementsov: comment indentation, s/Since: 5.2/Since: 6.0/]
Reviewed-by: Max Reitz 
Message-Id: <20201216061703.70908-5-vsement...@virtuozzo.com>
[mreitz: s/commit/stream/]
Signed-off-by: Max Reitz 
---
 qapi/block-core.json   | 6 ++
 include/block/block_int.h  | 7 ++-
 block/monitor/block-hmp-cmds.c | 4 ++--
 block/stream.c | 4 +++-
 blockdev.c | 4 +++-
 5 files changed, 20 insertions(+), 5 deletions(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index 3484986d1c..b55732d802 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -2543,6 +2543,11 @@
 #'stop' and 'enospc' can only be used if the block device
 #supports io-status (see BlockInfo).  Since 1.3.
 #
+# @filter-node-name: the node name that should be assigned to the
+#filter driver that the stream job inserts into the graph
+#above @device. If this option is not given, a node name is
+#autogenerated. (Since: 6.0)
+#
 # @auto-finalize: When false, this job will wait in a PENDING state after it 
has
 # finished its work, waiting for @block-job-finalize before
 # making any block graph changes.
@@ -2573,6 +2578,7 @@
   'data': { '*job-id': 'str', 'device': 'str', '*base': 'str',
 '*base-node': 'str', '*backing-file': 'str', '*speed': 'int',
 '*on-error': 'BlockdevOnError',
+'*filter-node-name': 'str',
 '*auto-finalize': 'bool', '*auto-dismiss': 'bool' } }
 
 ##
diff --git a/include/block/block_int.h b/include/block/block_int.h
index b9ef61fe4d..f3d0a0e60a 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -1143,6 +1143,9 @@ int is_windows_drive(const char *filename);
  *  See @BlockJobCreateFlags
  * @speed: The maximum speed, in bytes per second, or 0 for unlimited.
  * @on_error: The action to take upon error.
+ * @filter_node_name: The node name that should be assigned to the filter
+ *driver that the stream job inserts into the graph above
+ *@bs. NULL means that a node name should be autogenerated.
  * @errp: Error object.
  *
  * Start a streaming operation on @bs.  Clusters that are unallocated
@@ -1155,7 +1158,9 @@ int is_windows_drive(const char *filename);
 void stream_start(const char *job_id, BlockDriverState *bs,
   BlockDriverState *base, const char *backing_file_str,
   int creation_flags, int64_t speed,
-  BlockdevOnError on_error, Error **errp);
+  BlockdevOnError on_error,
+  const char *filter_node_name,
+  Error **errp);
 
 /**
  * commit_start:
diff --git a/block/monitor/block-hmp-cmds.c b/block/monitor/block-hmp-cmds.c
index d15a2be827..e8a58f326e 100644
--- a/block/monitor/block-hmp-cmds.c
+++ b/block/monitor/block-hmp-cmds.c
@@ -508,8 +508,8 @@ void hmp_block_stream(Monitor *mon, const QDict *qdict)
 
 qmp_block_stream(true, device, device, base != NULL, base, false, NULL,
  false, NULL, qdict_haskey(qdict, "speed"), speed, true,
- BLOCKDEV_ON_ERROR_REPORT, false, false, false, false,
- );
+ BLOCKDEV_ON_ERROR_REPORT, false, NULL, false, false, 
false,
+ false, );
 
 hmp_handle_error(mon, error);
 }
diff --git a/block/stream.c b/block/stream.c
index 236384f2f7..6e281c71ac 100644
--- a/block/stream.c
+++ b/block/stream.c
@@ -221,7 +221,9 @@ static const BlockJobDriver stream_job_driver = {
 void stream_start(const char *job_id, BlockDriverState *bs,
   BlockDriverState *base, const char *backing_file_str,
   int creation_flags, int64_t speed,
-  BlockdevOnError on_error, Error **errp)
+  BlockdevOnError on_error,
+  const char *filter_node_name,
+  Error **errp)
 {
 StreamBlockJob *s;
 BlockDriverState *iter;
diff --git a/blockdev.c b/blockdev.c
index 2431448c5d..e496356e10 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -2502,6 +2502,7 @@ void qmp_block_stream(bool has_job_id, const char 
*job_id, const char *device,
   bool has_backing_file, const char *backing_file,
   bool has_speed, int64_t speed,
   bool has_on_error, BlockdevOnError on_error,
+  bool has_filter_node_name, const char *filter_node_name,
   bool has_auto_finalize, bool auto_finalize,
   bool has_auto_dismiss, bool auto_dismiss,
   Error **errp)
@@ -2584,7 +2585,8 @@ void 

[PULL 16/53] iotests/297: Rewrite in Python and extend reach

[Max Reitz]

Instead of checking iotests.py only, check all Python files in the
qemu-iotests/ directory.  Of course, most of them do not pass, so there
is an extensive skip list for now.  (The only files that do pass are
209, 254, 283, and iotests.py.)

(Alternatively, we could have the opposite, i.e. an explicit list of
files that we do want to check, but I think it is better to check files
by default.)

Unless started in debug mode (./check -d), the output has no information
on which files are tested, so we will not have a problem e.g. with
backports, where some files may be missing when compared to upstream.

Besides the technical rewrite, some more things are changed:

- For the pylint invocation, PYTHONPATH is adjusted.  This mirrors
  setting MYPYPATH for mypy.

- Also, MYPYPATH is now derived from PYTHONPATH, so that we include
  paths set by the environment.  Maybe at some point we want to let the
  check script add '../../python/' to PYTHONPATH so that iotests.py does
  not need to do that.

- Passing --notes=FIXME,XXX to pylint suppresses warnings for TODO
  comments.  TODO is fine, we do not need 297 to complain about such
  comments.

- The "Success" line from mypy's output is suppressed, because (A) it
  does not add useful information, and (B) it would leak information
  about the files having been tested to the reference output, which we
  decidedly do not want.

Suggested-by: Vladimir Sementsov-Ogievskiy 
Signed-off-by: Max Reitz 
Message-Id: <20210118105720.14824-3-mre...@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy 
---
 tests/qemu-iotests/297 | 112 +
 tests/qemu-iotests/297.out |   5 +-
 2 files changed, 92 insertions(+), 25 deletions(-)

diff --git a/tests/qemu-iotests/297 b/tests/qemu-iotests/297
index 85bc1c0c85..b138b0539c 100755
--- a/tests/qemu-iotests/297
+++ b/tests/qemu-iotests/297
@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/usr/bin/env python3
 # group: meta
 #
 # Copyright (C) 2020 Red Hat, Inc.
@@ -16,30 +16,98 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see .
 
-seq=$(basename $0)
-echo "QA output created by $seq"
+import os
+import re
+import shutil
+import subprocess
+import sys
 
-status=1   # failure is the default!
+import iotests
 
-# get standard environment
-. ./common.rc
 
-if ! type -p "pylint-3" > /dev/null; then
-_notrun "pylint-3 not found"
-fi
-if ! type -p "mypy" > /dev/null; then
-_notrun "mypy not found"
-fi
+# TODO: Empty this list!
+SKIP_FILES = (
+'030', '040', '041', '044', '045', '055', '056', '057', '065', '093',
+'096', '118', '124', '129', '132', '136', '139', '147', '148', '149',
+'151', '152', '155', '163', '165', '169', '194', '196', '199', '202',
+'203', '205', '206', '207', '208', '210', '211', '212', '213', '216',
+'218', '219', '222', '224', '228', '234', '235', '236', '237', '238',
+'240', '242', '245', '246', '248', '255', '256', '257', '258', '260',
+'262', '264', '266', '274', '277', '280', '281', '295', '296', '298',
+'299', '300', '302', '303', '304', '307',
+'nbd-fault-injector.py', 'qcow2.py', 'qcow2_format.py', 'qed.py'
+)
 
-pylint-3 --score=n iotests.py
 
-MYPYPATH=../../python/ mypy --warn-unused-configs --disallow-subclassing-any \
---disallow-any-generics --disallow-incomplete-defs \
---disallow-untyped-decorators --no-implicit-optional \
---warn-redundant-casts --warn-unused-ignores \
---no-implicit-reexport iotests.py
+def is_python_file(filename):
+if not os.path.isfile(filename):
+return False
 
-# success, all done
-echo "*** done"
-rm -f $seq.full
-status=0
+if filename.endswith('.py'):
+return True
+
+with open(filename) as f:
+try:
+first_line = f.readline()
+return re.match('^#!.*python', first_line) is not None
+except UnicodeDecodeError:  # Ignore binary files
+return False
+
+
+def run_linters():
+files = [filename for filename in (set(os.listdir('.')) - set(SKIP_FILES))
+ if is_python_file(filename)]
+
+iotests.logger.debug('Files to be checked:')
+iotests.logger.debug(', '.join(sorted(files)))
+
+print('=== pylint ===')
+sys.stdout.flush()
+
+# Todo notes are fine, but fixme's or xxx's should probably just be
+# fixed (in tests, at least)
+env = os.environ.copy()
+qemu_module_path = os.path.join(os.path.dirname(__file__),
+'..', '..', 'python')
+try:
+env['PYTHONPATH'] += os.pathsep + qemu_module_path
+except KeyError:
+env['PYTHONPATH'] = qemu_module_path
+subprocess.run(('pylint-3', '--score=n', '--notes=FIXME,XXX', *files),
+   env=env, check=False)
+
+print('=== mypy ===')
+sys.stdout.flush()
+
+# We have to call mypy separately for each file.  Otherwise, it
+# will interpret all given files as 

[PULL 10/53] stream: rework backing-file changing

[Max Reitz]

From: Andrey Shinkevich 

Stream in stream_prepare calls bdrv_change_backing_file() to change
backing-file in the metadata of bs.

It may use either backing-file parameter given by user or just take
filename of base on job start.

Backing file format is determined by base on job finish.

There are some problems with this design, we solve only two by this
patch:

1. Consider scenario with backing-file unset. Current concept of stream
supports changing of the base during the job (we don't freeze link to
the base). So, we should not save base filename at job start,

  - let's determine name of the base on job finish.

2. Using direct base to determine filename and format is not very good:
base node may be a filter, so its filename may be JSON, and format_name
is not good for storing into qcow2 metadata as backing file format.

  - let's use unfiltered_base

Signed-off-by: Andrey Shinkevich 
Signed-off-by: Vladimir Sementsov-Ogievskiy 
  [vsementsov: change commit subject, change logic in stream_prepare]
Message-Id: <20201216061703.70908-10-vsement...@virtuozzo.com>
Reviewed-by: Max Reitz 
Signed-off-by: Max Reitz 
---
 block/stream.c | 9 +
 blockdev.c | 8 +---
 2 files changed, 6 insertions(+), 11 deletions(-)

diff --git a/block/stream.c b/block/stream.c
index 6e281c71ac..6a525a5edf 100644
--- a/block/stream.c
+++ b/block/stream.c
@@ -65,6 +65,7 @@ static int stream_prepare(Job *job)
 BlockDriverState *bs = blk_bs(bjob->blk);
 BlockDriverState *unfiltered_bs = bdrv_skip_filters(bs);
 BlockDriverState *base = bdrv_filter_or_cow_bs(s->above_base);
+BlockDriverState *unfiltered_base = bdrv_skip_filters(base);
 Error *local_err = NULL;
 int ret = 0;
 
@@ -73,10 +74,10 @@ static int stream_prepare(Job *job)
 
 if (bdrv_cow_child(unfiltered_bs)) {
 const char *base_id = NULL, *base_fmt = NULL;
-if (base) {
-base_id = s->backing_file_str;
-if (base->drv) {
-base_fmt = base->drv->format_name;
+if (unfiltered_base) {
+base_id = s->backing_file_str ?: unfiltered_base->filename;
+if (unfiltered_base->drv) {
+base_fmt = unfiltered_base->drv->format_name;
 }
 }
 bdrv_set_backing_hd(unfiltered_bs, base, _err);
diff --git a/blockdev.c b/blockdev.c
index e496356e10..8c03de582c 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -2511,7 +2511,6 @@ void qmp_block_stream(bool has_job_id, const char 
*job_id, const char *device,
 BlockDriverState *base_bs = NULL;
 AioContext *aio_context;
 Error *local_err = NULL;
-const char *base_name = NULL;
 int job_flags = JOB_DEFAULT;
 
 if (!has_on_error) {
@@ -2539,7 +2538,6 @@ void qmp_block_stream(bool has_job_id, const char 
*job_id, const char *device,
 goto out;
 }
 assert(bdrv_get_aio_context(base_bs) == aio_context);
-base_name = base;
 }
 
 if (has_base_node) {
@@ -2554,7 +2552,6 @@ void qmp_block_stream(bool has_job_id, const char 
*job_id, const char *device,
 }
 assert(bdrv_get_aio_context(base_bs) == aio_context);
 bdrv_refresh_filename(base_bs);
-base_name = base_bs->filename;
 }
 
 /* Check for op blockers in the whole chain between bs and base */
@@ -2574,9 +2571,6 @@ void qmp_block_stream(bool has_job_id, const char 
*job_id, const char *device,
 goto out;
 }
 
-/* backing_file string overrides base bs filename */
-base_name = has_backing_file ? backing_file : base_name;
-
 if (has_auto_finalize && !auto_finalize) {
 job_flags |= JOB_MANUAL_FINALIZE;
 }
@@ -2584,7 +2578,7 @@ void qmp_block_stream(bool has_job_id, const char 
*job_id, const char *device,
 job_flags |= JOB_MANUAL_DISMISS;
 }
 
-stream_start(has_job_id ? job_id : NULL, bs, base_bs, base_name,
+stream_start(has_job_id ? job_id : NULL, bs, base_bs, backing_file,
  job_flags, has_speed ? speed : 0, on_error,
  filter_node_name, _err);
 if (local_err) {
-- 
2.29.2

[PULL 04/53] copy-on-read: add filter drop function

[Max Reitz]

From: Andrey Shinkevich 

Provide API for the COR-filter removal. Also, drop the filter child
permissions for an inactive state when the filter node is being
removed.
To insert the filter, the block generic layer function
bdrv_insert_node() can be used.
The new function bdrv_cor_filter_drop() may be considered as an
intermediate solution before the QEMU permission update system has
overhauled. Then we are able to implement the API function
bdrv_remove_node() on the block generic layer.

Signed-off-by: Andrey Shinkevich 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20201216061703.70908-4-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 block/copy-on-read.h | 32 +
 block/copy-on-read.c | 56 
 2 files changed, 88 insertions(+)
 create mode 100644 block/copy-on-read.h

diff --git a/block/copy-on-read.h b/block/copy-on-read.h
new file mode 100644
index 00..7bf405dccd
--- /dev/null
+++ b/block/copy-on-read.h
@@ -0,0 +1,32 @@
+/*
+ * Copy-on-read filter block driver
+ *
+ * The filter driver performs Copy-On-Read (COR) operations
+ *
+ * Copyright (c) 2018-2020 Virtuozzo International GmbH.
+ *
+ * Author:
+ *   Andrey Shinkevich 
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see .
+ */
+
+#ifndef BLOCK_COPY_ON_READ
+#define BLOCK_COPY_ON_READ
+
+#include "block/block_int.h"
+
+void bdrv_cor_filter_drop(BlockDriverState *cor_filter_bs);
+
+#endif /* BLOCK_COPY_ON_READ */
diff --git a/block/copy-on-read.c b/block/copy-on-read.c
index cb03e0f2d3..618c4c4f43 100644
--- a/block/copy-on-read.c
+++ b/block/copy-on-read.c
@@ -23,11 +23,20 @@
 #include "qemu/osdep.h"
 #include "block/block_int.h"
 #include "qemu/module.h"
+#include "qapi/error.h"
+#include "block/copy-on-read.h"
+
+
+typedef struct BDRVStateCOR {
+bool active;
+} BDRVStateCOR;
 
 
 static int cor_open(BlockDriverState *bs, QDict *options, int flags,
 Error **errp)
 {
+BDRVStateCOR *state = bs->opaque;
+
 bs->file = bdrv_open_child(NULL, options, "file", bs, _of_bds,
BDRV_CHILD_FILTERED | BDRV_CHILD_PRIMARY,
false, errp);
@@ -42,6 +51,13 @@ static int cor_open(BlockDriverState *bs, QDict *options, 
int flags,
 ((BDRV_REQ_FUA | BDRV_REQ_MAY_UNMAP | BDRV_REQ_NO_FALLBACK) &
 bs->file->bs->supported_zero_flags);
 
+state->active = true;
+
+/*
+ * We don't need to call bdrv_child_refresh_perms() now as the permissions
+ * will be updated later when the filter node gets its parent.
+ */
+
 return 0;
 }
 
@@ -57,6 +73,17 @@ static void cor_child_perm(BlockDriverState *bs, BdrvChild 
*c,
uint64_t perm, uint64_t shared,
uint64_t *nperm, uint64_t *nshared)
 {
+BDRVStateCOR *s = bs->opaque;
+
+if (!s->active) {
+/*
+ * While the filter is being removed
+ */
+*nperm = 0;
+*nshared = BLK_PERM_ALL;
+return;
+}
+
 *nperm = perm & PERM_PASSTHROUGH;
 *nshared = (shared & PERM_PASSTHROUGH) | PERM_UNCHANGED;
 
@@ -135,6 +162,7 @@ static void cor_lock_medium(BlockDriverState *bs, bool 
locked)
 
 static BlockDriver bdrv_copy_on_read = {
 .format_name= "copy-on-read",
+.instance_size  = sizeof(BDRVStateCOR),
 
 .bdrv_open  = cor_open,
 .bdrv_child_perm= cor_child_perm,
@@ -154,6 +182,34 @@ static BlockDriver bdrv_copy_on_read = {
 .is_filter  = true,
 };
 
+
+void bdrv_cor_filter_drop(BlockDriverState *cor_filter_bs)
+{
+BdrvChild *child;
+BlockDriverState *bs;
+BDRVStateCOR *s = cor_filter_bs->opaque;
+
+child = bdrv_filter_child(cor_filter_bs);
+if (!child) {
+return;
+}
+bs = child->bs;
+
+/* Retain the BDS until we complete the graph change. */
+bdrv_ref(bs);
+/* Hold a guest back from writing while permissions are being reset. */
+bdrv_drained_begin(bs);
+/* Drop permissions before the graph change. */
+s->active = false;
+bdrv_child_refresh_perms(cor_filter_bs, child, _abort);
+bdrv_replace_node(cor_filter_bs, bs, _abort);
+
+bdrv_drained_end(bs);
+bdrv_unref(bs);
+

[PULL 03/53] block: add API function to insert a node

[Max Reitz]

From: Andrey Shinkevich 

Provide API for insertion a node to backing chain.

Suggested-by: Max Reitz 
Signed-off-by: Andrey Shinkevich 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20201216061703.70908-3-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 include/block/block.h |  2 ++
 block.c   | 25 +
 2 files changed, 27 insertions(+)

diff --git a/include/block/block.h b/include/block/block.h
index a193545b6a..127bdf3392 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -358,6 +358,8 @@ void bdrv_append(BlockDriverState *bs_new, BlockDriverState 
*bs_top,
  Error **errp);
 void bdrv_replace_node(BlockDriverState *from, BlockDriverState *to,
Error **errp);
+BlockDriverState *bdrv_insert_node(BlockDriverState *bs, QDict *node_options,
+   int flags, Error **errp);
 
 int bdrv_parse_aio(const char *mode, int *flags);
 int bdrv_parse_cache_mode(const char *mode, int *flags, bool *writethrough);
diff --git a/block.c b/block.c
index 8b9d457546..91a66d4f3e 100644
--- a/block.c
+++ b/block.c
@@ -4660,6 +4660,31 @@ static void bdrv_delete(BlockDriverState *bs)
 g_free(bs);
 }
 
+BlockDriverState *bdrv_insert_node(BlockDriverState *bs, QDict *node_options,
+   int flags, Error **errp)
+{
+BlockDriverState *new_node_bs;
+Error *local_err = NULL;
+
+new_node_bs = bdrv_open(NULL, NULL, node_options, flags, errp);
+if (new_node_bs == NULL) {
+error_prepend(errp, "Could not create node: ");
+return NULL;
+}
+
+bdrv_drained_begin(bs);
+bdrv_replace_node(bs, new_node_bs, _err);
+bdrv_drained_end(bs);
+
+if (local_err) {
+bdrv_unref(new_node_bs);
+error_propagate(errp, local_err);
+return NULL;
+}
+
+return new_node_bs;
+}
+
 /*
  * Run consistency checks on an image
  *
-- 
2.29.2

[PULL 11/53] qapi: block-stream: add "bottom" argument

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy 

The code already don't freeze base node and we try to make it prepared
for the situation when base node is changed during the operation. In
other words, block-stream doesn't own base node.

Let's introduce a new interface which should replace the current one,
which will in better relations with the code. Specifying bottom node
instead of base, and requiring it to be non-filter gives us the
following benefits:

 - drop difference between above_base and base_overlay, which will be
   renamed to just bottom, when old interface dropped

 - clean way to work with parallel streams/commits on the same backing
   chain, which otherwise become a problem when we introduce a filter
   for stream job

 - cleaner interface. Nobody will surprised the fact that base node may
   disappear during block-stream, when there is no word about "base" in
   the interface.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Message-Id: <20201216061703.70908-11-vsement...@virtuozzo.com>
Reviewed-by: Max Reitz 
Signed-off-by: Max Reitz 
---
 qapi/block-core.json   | 12 ---
 include/block/block_int.h  |  1 +
 block/monitor/block-hmp-cmds.c |  3 +-
 block/stream.c | 50 +++-
 blockdev.c | 59 --
 5 files changed, 94 insertions(+), 31 deletions(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index 65167ebf56..1d9dcd7d30 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -2517,10 +2517,14 @@
 # @device: the device or node name of the top image
 #
 # @base: the common backing file name.
-#It cannot be set if @base-node is also set.
+#It cannot be set if @base-node or @bottom is also set.
 #
 # @base-node: the node name of the backing file.
-# It cannot be set if @base is also set. (Since 2.8)
+# It cannot be set if @base or @bottom is also set. (Since 2.8)
+#
+# @bottom: the last node in the chain that should be streamed into
+#  top. It cannot be set if @base or @base-node is also set.
+#  It cannot be filter node. (Since 6.0)
 #
 # @backing-file: The backing file string to write into the top
 #image. This filename is not validated.
@@ -2576,8 +2580,8 @@
 ##
 { 'command': 'block-stream',
   'data': { '*job-id': 'str', 'device': 'str', '*base': 'str',
-'*base-node': 'str', '*backing-file': 'str', '*speed': 'int',
-'*on-error': 'BlockdevOnError',
+'*base-node': 'str', '*backing-file': 'str', '*bottom': 'str',
+'*speed': 'int', '*on-error': 'BlockdevOnError',
 '*filter-node-name': 'str',
 '*auto-finalize': 'bool', '*auto-dismiss': 'bool' } }
 
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 92d3754ead..f4b844f310 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -1161,6 +1161,7 @@ int is_windows_drive(const char *filename);
  */
 void stream_start(const char *job_id, BlockDriverState *bs,
   BlockDriverState *base, const char *backing_file_str,
+  BlockDriverState *bottom,
   int creation_flags, int64_t speed,
   BlockdevOnError on_error,
   const char *filter_node_name,
diff --git a/block/monitor/block-hmp-cmds.c b/block/monitor/block-hmp-cmds.c
index e8a58f326e..afd75ab628 100644
--- a/block/monitor/block-hmp-cmds.c
+++ b/block/monitor/block-hmp-cmds.c
@@ -507,7 +507,8 @@ void hmp_block_stream(Monitor *mon, const QDict *qdict)
 int64_t speed = qdict_get_try_int(qdict, "speed", 0);
 
 qmp_block_stream(true, device, device, base != NULL, base, false, NULL,
- false, NULL, qdict_haskey(qdict, "speed"), speed, true,
+ false, NULL, false, NULL,
+ qdict_haskey(qdict, "speed"), speed, true,
  BLOCKDEV_ON_ERROR_REPORT, false, NULL, false, false, 
false,
  false, );
 
diff --git a/block/stream.c b/block/stream.c
index 6a525a5edf..045d6bc76b 100644
--- a/block/stream.c
+++ b/block/stream.c
@@ -221,6 +221,7 @@ static const BlockJobDriver stream_job_driver = {
 
 void stream_start(const char *job_id, BlockDriverState *bs,
   BlockDriverState *base, const char *backing_file_str,
+  BlockDriverState *bottom,
   int creation_flags, int64_t speed,
   BlockdevOnError on_error,
   const char *filter_node_name,
@@ -230,25 +231,42 @@ void stream_start(const char *job_id, BlockDriverState 
*bs,
 BlockDriverState *iter;
 bool bs_read_only;
 int basic_flags = BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE_UNCHANGED;
-BlockDriverState *base_overlay = bdrv_find_overlay(bs, base);
+BlockDriverState *base_overlay;
 BlockDriverState *above_base;
 
-if (!base_overlay) {
-error_setg(errp, "'%s' is not in the backing chain of 

[PULL 02/53] copy-on-read: support preadv/pwritev_part functions

[Max Reitz]

From: Andrey Shinkevich 

Add support for the recently introduced functions
bdrv_co_preadv_part()
and
bdrv_co_pwritev_part()
to the COR-filter driver.

Signed-off-by: Andrey Shinkevich 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
Message-Id: <20201216061703.70908-2-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 block/copy-on-read.c | 28 
 1 file changed, 16 insertions(+), 12 deletions(-)

diff --git a/block/copy-on-read.c b/block/copy-on-read.c
index 2816e61afe..cb03e0f2d3 100644
--- a/block/copy-on-read.c
+++ b/block/copy-on-read.c
@@ -74,21 +74,25 @@ static int64_t cor_getlength(BlockDriverState *bs)
 }
 
 
-static int coroutine_fn cor_co_preadv(BlockDriverState *bs,
-  uint64_t offset, uint64_t bytes,
-  QEMUIOVector *qiov, int flags)
+static int coroutine_fn cor_co_preadv_part(BlockDriverState *bs,
+   uint64_t offset, uint64_t bytes,
+   QEMUIOVector *qiov,
+   size_t qiov_offset,
+   int flags)
 {
-return bdrv_co_preadv(bs->file, offset, bytes, qiov,
-  flags | BDRV_REQ_COPY_ON_READ);
+return bdrv_co_preadv_part(bs->file, offset, bytes, qiov, qiov_offset,
+   flags | BDRV_REQ_COPY_ON_READ);
 }
 
 
-static int coroutine_fn cor_co_pwritev(BlockDriverState *bs,
-   uint64_t offset, uint64_t bytes,
-   QEMUIOVector *qiov, int flags)
+static int coroutine_fn cor_co_pwritev_part(BlockDriverState *bs,
+uint64_t offset,
+uint64_t bytes,
+QEMUIOVector *qiov,
+size_t qiov_offset, int flags)
 {
-
-return bdrv_co_pwritev(bs->file, offset, bytes, qiov, flags);
+return bdrv_co_pwritev_part(bs->file, offset, bytes, qiov, qiov_offset,
+flags);
 }
 
 
@@ -137,8 +141,8 @@ static BlockDriver bdrv_copy_on_read = {
 
 .bdrv_getlength = cor_getlength,
 
-.bdrv_co_preadv = cor_co_preadv,
-.bdrv_co_pwritev= cor_co_pwritev,
+.bdrv_co_preadv_part= cor_co_preadv_part,
+.bdrv_co_pwritev_part   = cor_co_pwritev_part,
 .bdrv_co_pwrite_zeroes  = cor_co_pwrite_zeroes,
 .bdrv_co_pdiscard   = cor_co_pdiscard,
 .bdrv_co_pwritev_compressed = cor_co_pwritev_compressed,
-- 
2.29.2

[PULL 01/53] iotests: fix _check_o_direct

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy 

Unfortunately commit "iotests: handle tmpfs" breaks running iotests
with -nbd -nocache, as _check_o_direct tries to create
$TEST_IMG.test_o_direct, but in case of nbd TEST_IMG is something like
nbd+unix:///... , and test fails with message

  qemu-img: nbd+unix:///?socket[...]test_o_direct: Protocol driver
'nbd' does not support image creation, and opening the image
failed: Failed to connect to '/tmp/tmp.[...]/nbd/test_o_direct': No
such file or directory

Use TEST_DIR instead.

Fixes: cfdca2b9f9d4ca26bb2b2dfe8de3149092e39170
Signed-off-by: Vladimir Sementsov-Ogievskiy 
Message-Id: <20201218182012.47607-1-vsement...@virtuozzo.com>
Signed-off-by: Max Reitz 
---
 tests/qemu-iotests/common.rc | 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
index 29354654cc..297acf9b6a 100644
--- a/tests/qemu-iotests/common.rc
+++ b/tests/qemu-iotests/common.rc
@@ -821,9 +821,10 @@ _supported_cache_modes()
 # Check whether the filesystem supports O_DIRECT
 _check_o_direct()
 {
-$QEMU_IMG create -f raw "$TEST_IMG".test_o_direct 1M > /dev/null
-out=$($QEMU_IO -f raw -t none -c quit "$TEST_IMG".test_o_direct 2>&1)
-rm -f "$TEST_IMG".test_o_direct
+testfile="$TEST_DIR"/_check_o_direct
+$QEMU_IMG create -f raw "$testfile" 1M > /dev/null
+out=$($QEMU_IO -f raw -t none -c quit "$testfile" 2>&1)
+rm -f "$testfile"
 
 [[ "$out" != *"O_DIRECT"* ]]
 }
-- 
2.29.2

[PULL 00/53] Block patches

[Max Reitz]

The following changes since commit 31ee895047bdcf7387e3570cbd2a473c6f744b08:

  Merge remote-tracking branch 'remotes/jasowang/tags/net-pull-request' into 
staging (2021-01-25 15:56:13 +)

are available in the Git repository at:

  https://github.com/XanClic/qemu.git tags/pull-block-2021-01-26

for you to fetch changes up to bb24cdc5efee580e81f71c5ff0fd980f2cc179d0:

  iotests/178: Pass value to invalid option (2021-01-26 14:36:37 +0100)


Block patches:
- Make backup block jobs use asynchronous requests with the block-copy
  module
- Use COR filter node for stream block jobs
- Make coroutine-sigaltstack’s qemu_coroutine_new() function thread-safe
- Report error string when file locking fails with an unexpected errno
- iotest fixes, additions, and some refactoring


Alberto Garcia (1):
  iotests: Add test for the regression fixed in c8bf9a9169

Andrey Shinkevich (10):
  copy-on-read: support preadv/pwritev_part functions
  block: add API function to insert a node
  copy-on-read: add filter drop function
  qapi: add filter-node-name to block-stream
  qapi: copy-on-read filter: add 'bottom' option
  iotests: add #310 to test bottom node in COR driver
  block: include supported_read_flags into BDS structure
  copy-on-read: skip non-guest reads if no copy needed
  stream: rework backing-file changing
  block: apply COR-filter to block-stream jobs

David Edmondson (1):
  block: report errno when flock fcntl fails

Max Reitz (14):
  iotests.py: Assume a couple of variables as given
  iotests/297: Rewrite in Python and extend reach
  iotests: Move try_remove to iotests.py
  iotests/129: Remove test images in tearDown()
  iotests/129: Do not check @busy
  iotests/129: Use throttle node
  iotests/129: Actually test a commit job
  iotests/129: Limit mirror job's buffer size
  iotests/129: Clean up pylint and mypy complaints
  iotests/300: Clean up pylint and mypy complaints
  coroutine-sigaltstack: Add SIGUSR2 mutex
  iotests/129: Limit backup's max-chunk/max-workers
  iotests/118: Drop 'change' test
  iotests/178: Pass value to invalid option

Vladimir Sementsov-Ogievskiy (27):
  iotests: fix _check_o_direct
  qapi: block-stream: add "bottom" argument
  iotests: 30: prepare to COR filter insertion by stream job
  block/stream: add s->target_bs
  qapi: backup: add perf.use-copy-range parameter
  block/block-copy: More explicit call_state
  block/block-copy: implement block_copy_async
  block/block-copy: add max_chunk and max_workers parameters
  block/block-copy: add list of all call-states
  block/block-copy: add ratelimit to block-copy
  block/block-copy: add block_copy_cancel
  blockjob: add set_speed to BlockJobDriver
  job: call job_enter from job_pause
  qapi: backup: add max-chunk and max-workers to x-perf struct
  iotests: 56: prepare for backup over block-copy
  iotests: 185: prepare for backup over block-copy
  iotests: 219: prepare for backup over block-copy
  iotests: 257: prepare for backup over block-copy
  block/block-copy: make progress_bytes_callback optional
  block/backup: drop extra gotos from backup_run()
  backup: move to block-copy
  qapi: backup: disable copy_range by default
  block/block-copy: drop unused block_copy_set_progress_callback()
  block/block-copy: drop unused argument of block_copy()
  simplebench/bench_block_job: use correct shebang line with python3
  simplebench: bench_block_job: add cmd_options argument
  simplebench: add bench-backup.py

 qapi/block-core.json   |  66 +-
 block/backup-top.h |   1 +
 block/copy-on-read.h   |  32 +++
 include/block/block-copy.h |  61 -
 include/block/block.h  |  10 +-
 include/block/block_int.h  |  15 +-
 include/block/blockjob_int.h   |   2 +
 block.c|  25 ++
 block/backup-top.c |   6 +-
 block/backup.c | 233 ---
 block/block-copy.c | 227 +++---
 block/copy-on-read.c   | 184 ++-
 block/file-posix.c |  38 ++-
 block/io.c |  10 +-
 block/monitor/block-hmp-cmds.c |   7 +-
 block/replication.c|   2 +
 block/stream.c | 185 +--
 blockdev.c |  83 +--
 blockjob.c |   6 +
 job.c  |   3 +
 util/coroutine-sigaltstack.c   |   9 +
 scripts/simplebench/bench-backup.py| 167 ++
 scripts/simplebench/bench-example.py   |   2 +-
 scripts/simplebench/bench_block_job.py |  13 +-
 tests/qemu-iotests/030 |  12 +-
 tests/qemu-iotests/056 |   9 +-
 tests/qemu-iotests/109.out |  24 ++
 tests/qemu-iotests/118 

Re: [PATCH] iotests/178: Pass value to invalid option

[Kevin Wolf]

Am 26.01.2021 um 13:38 hat Max Reitz geschrieben:
> ccd3b3b8112 has deprecated short-hand boolean options (i.e., options
> with values).  All options without values are interpreted as boolean
> options, so this includes the invalid option "snapshot.foo" used in
> iotest 178.
> 
> So after ccd3b3b8112, 178 fails with:
> 
>   +qemu-img: warning: short-form boolean option 'snapshot.foo' deprecated
>   +Please use snapshot.foo=on instead
> 
> Suppress that deprecation warning by passing some value to it (it does
> not matter which, because the option is invalid anyway).
> 
> Fixes: ccd3b3b8112b670fdccf8a392b8419b173ffccb4
>("qemu-option: warn for short-form boolean options")
> Signed-off-by: Max Reitz 

Thanks, applied to the block branch.

Kevin

Re: [PATCH v9 0/6] Rework iotests/check

[Kevin Wolf]

Am 26.01.2021 um 14:19 hat Vladimir Sementsov-Ogievskiy geschrieben:
> 26.01.2021 15:53, Kevin Wolf wrote:
> > Am 25.01.2021 um 19:50 hat Vladimir Sementsov-Ogievskiy geschrieben:
> > > v9:
> > > 01: new, one more whitespace-fixing patch
> > > testenv: allow case when we don't have system-arch emulator, but have 
> > > several for another architectures
> > >   change direct os.access(..., os.X_OK) calls to new helper 
> > > function which also check that path is a file
> > > testrunner: s/fail/not run/ for 'No qualified output'
> > >  drop elapsed time arg for one of 'not run' results (now no 
> > > elapsed time for any 'not run' result)
> > 
> > More CI fun:
> > 
> > Traceback (most recent call last):
> >File "./check", line 117, in 
> >  testfinder = TestFinder(test_dir=env.source_iotests)
> >File "/builds/.../qemu/tests/qemu-iotests/findtests.py", line 53, in 
> > __init__
> >  for line in f:
> >File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
> >  return codecs.ascii_decode(input, self.errors)[0]
> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1591: 
> > ordinal not in range(128)
> 
> Can be solved by adding encoding='utf8' to the open().. But.. Adding
> it everywhere is not fun.
> 
> So, what about just define PYTHONUTF8=1 for running check in
> check-block.sh?

Ah, I didn't know this one. Yes, that's probably nicer than my attempt
of adding an explicit encoding everywhere.

> > Traceback (most recent call last):
> >File "./check", line 112, in 
> >  env = TestEnv(imgfmt=args.imgfmt, imgproto=args.imgproto,
> >File "/builds/.../qemu/tests/qemu-iotests/testenv.py", line 216, in 
> > __init__
> >  self.qemu_default_machine = get_default_machine(self.qemu_prog)
> >File "/builds/.../qemu/tests/qemu-iotests/testenv.py", line 41, in 
> > get_default_machine
> >  default_machine = next(m for m in machines if m.endswith(' (default)'))
> 
> Looking at original check, default_machine should be empty string in this 
> case.
> 
> so we need
> 
> diff --git a/tests/qemu-iotests/testenv.py b/tests/qemu-iotests/testenv.py
> index 8b80425670..222395caef 100644
> --- a/tests/qemu-iotests/testenv.py
> +++ b/tests/qemu-iotests/testenv.py
> @@ -38,7 +38,11 @@ def get_default_machine(qemu_prog: str) -> str:
>stdout=subprocess.PIPE).stdout
>  machines = outp.split('\n')
> -default_machine = next(m for m in machines if m.endswith(' (default)'))
> +try:
> +default_machine = next(m for m in machines if m.endswith(' 
> (default)'))
> +except StopIteration:
> +return ''
> +
>  default_machine = default_machine.split(' ', 1)[0]
>  alias_suf = ' (alias of {})'.format(default_machine)

This one looks a bit nicer than mine (using None instead of an
exception), too, so I'll apply this one as well for the next test cycle.

Kevin

Re: [PATCH v9 0/6] Rework iotests/check

[Vladimir Sementsov-Ogievskiy]

26.01.2021 15:53, Kevin Wolf wrote:

Am 25.01.2021 um 19:50 hat Vladimir Sementsov-Ogievskiy geschrieben:

v9:
01: new, one more whitespace-fixing patch
testenv: allow case when we don't have system-arch emulator, but have several 
for another architectures
  change direct os.access(..., os.X_OK) calls to new helper function 
which also check that path is a file
testrunner: s/fail/not run/ for 'No qualified output'
 drop elapsed time arg for one of 'not run' results (now no elapsed 
time for any 'not run' result)


More CI fun:

Traceback (most recent call last):
   File "./check", line 117, in 
 testfinder = TestFinder(test_dir=env.source_iotests)
   File "/builds/.../qemu/tests/qemu-iotests/findtests.py", line 53, in __init__
 for line in f:
   File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
 return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1591: 
ordinal not in range(128)


Can be solved by adding encoding='utf8' to the open().. But.. Adding it 
everywhere is not fun.

So, what about just define PYTHONUTF8=1 for running check in check-block.sh?

https://docs.python.org/3/using/cmdline.html#envvar-PYTHONUTF8


diff --git a/tests/check-block.sh b/tests/check-block.sh
index ac32fd67dd..9dcadc435f 100755
--- a/tests/check-block.sh
+++ b/tests/check-block.sh
@@ -74,6 +74,8 @@ cd tests/qemu-iotests
 # QEMU_CHECK_BLOCK_AUTO is used to disable some unstable sub-tests
 export QEMU_CHECK_BLOCK_AUTO=1
 
+export PYTHONUTF8=1

+
 ret=0
 for fmt in $format_list ; do
 ${PYTHON} ./check -makecheck -$fmt $group || ret=1



Traceback (most recent call last):
   File "./check", line 112, in 
 env = TestEnv(imgfmt=args.imgfmt, imgproto=args.imgproto,
   File "/builds/.../qemu/tests/qemu-iotests/testenv.py", line 216, in __init__
 self.qemu_default_machine = get_default_machine(self.qemu_prog)
   File "/builds/.../qemu/tests/qemu-iotests/testenv.py", line 41, in 
get_default_machine
 default_machine = next(m for m in machines if m.endswith(' (default)'))


Looking at original check, default_machine should be empty string in this case.

so we need

diff --git a/tests/qemu-iotests/testenv.py b/tests/qemu-iotests/testenv.py
index 8b80425670..222395caef 100644
--- a/tests/qemu-iotests/testenv.py
+++ b/tests/qemu-iotests/testenv.py
@@ -38,7 +38,11 @@ def get_default_machine(qemu_prog: str) -> str:
   stdout=subprocess.PIPE).stdout
 
 machines = outp.split('\n')

-default_machine = next(m for m in machines if m.endswith(' (default)'))
+try:
+default_machine = next(m for m in machines if m.endswith(' (default)'))
+except StopIteration:
+return ''
+
 default_machine = default_machine.split(' ', 1)[0]
 
 alias_suf = ' (alias of {})'.format(default_machine)




--
Best regards,
Vladimir

Re: [PATCH v9 0/6] Rework iotests/check

[Kevin Wolf]

Am 25.01.2021 um 19:50 hat Vladimir Sementsov-Ogievskiy geschrieben:
> v9:
> 01: new, one more whitespace-fixing patch
> testenv: allow case when we don't have system-arch emulator, but have several 
> for another architectures
>  change direct os.access(..., os.X_OK) calls to new helper function 
> which also check that path is a file
> testrunner: s/fail/not run/ for 'No qualified output'
> drop elapsed time arg for one of 'not run' results (now no 
> elapsed time for any 'not run' result)

More CI fun:

Traceback (most recent call last):
  File "./check", line 117, in 
testfinder = TestFinder(test_dir=env.source_iotests)
  File "/builds/.../qemu/tests/qemu-iotests/findtests.py", line 53, in __init__
for line in f:
  File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1591: 
ordinal not in range(128)

Traceback (most recent call last):
  File "./check", line 112, in 
env = TestEnv(imgfmt=args.imgfmt, imgproto=args.imgproto,
  File "/builds/.../qemu/tests/qemu-iotests/testenv.py", line 216, in __init__
self.qemu_default_machine = get_default_machine(self.qemu_prog)
  File "/builds/.../qemu/tests/qemu-iotests/testenv.py", line 41, in 
get_default_machine
default_machine = next(m for m in machines if m.endswith(' (default)'))
StopIteration

Kevin

[PATCH] iotests/178: Pass value to invalid option

[Max Reitz]

ccd3b3b8112 has deprecated short-hand boolean options (i.e., options
with values).  All options without values are interpreted as boolean
options, so this includes the invalid option "snapshot.foo" used in
iotest 178.

So after ccd3b3b8112, 178 fails with:

  +qemu-img: warning: short-form boolean option 'snapshot.foo' deprecated
  +Please use snapshot.foo=on instead

Suppress that deprecation warning by passing some value to it (it does
not matter which, because the option is invalid anyway).

Fixes: ccd3b3b8112b670fdccf8a392b8419b173ffccb4
   ("qemu-option: warn for short-form boolean options")
Signed-off-by: Max Reitz 
---
 tests/qemu-iotests/178   | 2 +-
 tests/qemu-iotests/178.out.qcow2 | 2 +-
 tests/qemu-iotests/178.out.raw   | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/tests/qemu-iotests/178 b/tests/qemu-iotests/178
index 3b1a7adce4..8df241ead8 100755
--- a/tests/qemu-iotests/178
+++ b/tests/qemu-iotests/178
@@ -57,7 +57,7 @@ $QEMU_IMG measure --image-opts # missing filename
 $QEMU_IMG measure -f qcow2 # missing filename
 $QEMU_IMG measure -l snap1 # missing filename
 $QEMU_IMG measure -o , # invalid option list
-$QEMU_IMG measure -l snapshot.foo # invalid snapshot option
+$QEMU_IMG measure -l snapshot.foo=bar # invalid snapshot option
 $QEMU_IMG measure --output foo # invalid output format
 $QEMU_IMG measure --size -1 # invalid image size
 $QEMU_IMG measure -O foo "$TEST_IMG" # unknown image file format
diff --git a/tests/qemu-iotests/178.out.qcow2 b/tests/qemu-iotests/178.out.qcow2
index c7997760fd..fe193fd5f4 100644
--- a/tests/qemu-iotests/178.out.qcow2
+++ b/tests/qemu-iotests/178.out.qcow2
@@ -11,7 +11,7 @@ qemu-img: --image-opts, -f, and -l require a filename 
argument.
 qemu-img: --image-opts, -f, and -l require a filename argument.
 qemu-img: Invalid option list: ,
 qemu-img: Invalid parameter 'snapshot.foo'
-qemu-img: Failed in parsing snapshot param 'snapshot.foo'
+qemu-img: Failed in parsing snapshot param 'snapshot.foo=bar'
 qemu-img: --output must be used with human or json as argument.
 qemu-img: Invalid image size specified. Must be between 0 and 
9223372036854775807.
 qemu-img: Unknown file format 'foo'
diff --git a/tests/qemu-iotests/178.out.raw b/tests/qemu-iotests/178.out.raw
index 20e17da115..445e460fad 100644
--- a/tests/qemu-iotests/178.out.raw
+++ b/tests/qemu-iotests/178.out.raw
@@ -11,7 +11,7 @@ qemu-img: --image-opts, -f, and -l require a filename 
argument.
 qemu-img: --image-opts, -f, and -l require a filename argument.
 qemu-img: Invalid option list: ,
 qemu-img: Invalid parameter 'snapshot.foo'
-qemu-img: Failed in parsing snapshot param 'snapshot.foo'
+qemu-img: Failed in parsing snapshot param 'snapshot.foo=bar'
 qemu-img: --output must be used with human or json as argument.
 qemu-img: Invalid image size specified. Must be between 0 and 
9223372036854775807.
 qemu-img: Unknown file format 'foo'
-- 
2.29.2

Re: [PATCH] iotests/118: Drop 'change' test

[Kevin Wolf]

Am 26.01.2021 um 12:30 hat Paolo Bonzini geschrieben:
> On 26/01/21 11:48, Max Reitz wrote:
> > Commit 0afec75734331 removed the 'change' QMP command, so we can no
> > longer test it in 118.
> > 
> > Fixes: 0afec75734331a0b52fa3aa4235220eda8c7846f
> > ('qmp: remove deprecated "change" command')
> > Signed-off-by: Max Reitz 
> > ---
> >   tests/qemu-iotests/118 | 17 -
> >   tests/qemu-iotests/118.out |  4 ++--
> >   2 files changed, 2 insertions(+), 19 deletions(-)
> 
> How come this didn't break GitLab CI?...

The test case is not in the auto group.

Kevin

Re: [PATCH] iotests/118: Drop 'change' test

[Max Reitz]

On 26.01.21 12:18, Kevin Wolf wrote:

Am 26.01.2021 um 11:48 hat Max Reitz geschrieben:

Commit 0afec75734331 removed the 'change' QMP command, so we can no
longer test it in 118.

Fixes: 0afec75734331a0b52fa3aa4235220eda8c7846f
('qmp: remove deprecated "change" command')
Signed-off-by: Max Reitz 


Thanks, applied to the block branch.

(I was just about to make the same change myself...)

If you don't mind, I'd squash in something like the following.

Kevin

diff --git a/tests/qemu-iotests/118 b/tests/qemu-iotests/118
index 88e8354e21..cae52ffa5e 100755
--- a/tests/qemu-iotests/118
+++ b/tests/qemu-iotests/118
@@ -1,8 +1,7 @@
  #!/usr/bin/env python3
  # group: rw
  #
-# Test case for the QMP 'change' command and all other associated
-# commands
+# Test case for media change monitor commands
  #
  # Copyright (C) 2015 Red Hat, Inc.
  #


Sure, looks good.

Do you plan on sending a pull request today?  I was, and so I’d like to 
include this patch then.


(Well, I was planning to send it last week, then came the 
coroutine-sigaltstack stuff, then this, and right now I’m debugging 178 
breaking after ccd3b3b8112b670fdccf8a392b8419b173ffccb4...)


Max

[PATCH v2] hw/block/nvme: add missing mor/mar constraint checks

[Klaus Jensen]

From: Klaus Jensen 

Firstly, if zoned.max_active is non-zero, zoned.max_open must be less
than or equal to zoned.max_active.

Secondly, if only zones.max_active is set, we have to explicitly set
zones.max_open or we end up with an invalid MAR/MOR configuration. This
is an artifact of the parameters not being zeroes-based like in the
spec.

Cc: Dmitry Fomichev 
Reported-by: Gollu Appalanaidu 
Signed-off-by: Klaus Jensen 
---

v2:

  * Jumped the gun on removing the check on zoned.max_open. It should
still be done since the device might only have a constraint on open
zones, not active.
  * Instead, added an explicit set of zoned.max_open if only
zoned.max_active is specifed.

 hw/block/nvme-ns.c | 12 
 1 file changed, 12 insertions(+)

diff --git a/hw/block/nvme-ns.c b/hw/block/nvme-ns.c
index 62b25cf69bfa..df514287b58f 100644
--- a/hw/block/nvme-ns.c
+++ b/hw/block/nvme-ns.c
@@ -153,6 +153,18 @@ static int nvme_ns_zoned_check_calc_geometry(NvmeNamespace 
*ns, Error **errp)
 return -1;
 }
 
+if (ns->params.max_active_zones) {
+if (ns->params.max_open_zones > ns->params.max_active_zones) {
+error_setg(errp, "max_open_zones (%u) exceeds max_active_zones 
(%u)",
+   ns->params.max_open_zones, ns->params.max_active_zones);
+return -1;
+}
+
+if (!ns->params.max_open_zones) {
+ns->params.max_open_zones = ns->params.max_active_zones;
+}
+}
+
 if (ns->params.zd_extension_size) {
 if (ns->params.zd_extension_size & 0x3f) {
 error_setg(errp,
-- 
2.30.0

[PATCH] hw/block/nvme: add missing mor/mar constraint check

[Klaus Jensen]

From: Klaus Jensen 

The Maximum Open Resources field shall be less than or equal to the
Maximum Active Resources field.

Verify that constraint and remove the redundant check that the Maximum
Open Resources field does not exceed the number of zones.

Cc: Dmitry Fomichev 
Reported-by: Gollu Appalanaidu 
Signed-off-by: Klaus Jensen 
---
 hw/block/nvme-ns.c | 12 ++--
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/hw/block/nvme-ns.c b/hw/block/nvme-ns.c
index 62b25cf69bfa..9586f7f33b8c 100644
--- a/hw/block/nvme-ns.c
+++ b/hw/block/nvme-ns.c
@@ -140,12 +140,6 @@ static int nvme_ns_zoned_check_calc_geometry(NvmeNamespace 
*ns, Error **errp)
 return -1;
 }
 
-if (ns->params.max_open_zones > ns->num_zones) {
-error_setg(errp,
-   "max_open_zones value %u exceeds the number of zones %u",
-   ns->params.max_open_zones, ns->num_zones);
-return -1;
-}
 if (ns->params.max_active_zones > ns->num_zones) {
 error_setg(errp,
"max_active_zones value %u exceeds the number of zones %u",
@@ -153,6 +147,12 @@ static int nvme_ns_zoned_check_calc_geometry(NvmeNamespace 
*ns, Error **errp)
 return -1;
 }
 
+if (ns->params.max_open_zones > ns->params.max_active_zones) {
+error_setg(errp, "max_open_zones (%u) exceeds max_active_zones (%u)",
+   ns->params.max_open_zones, ns->params.max_active_zones);
+return -1;
+}
+
 if (ns->params.zd_extension_size) {
 if (ns->params.zd_extension_size & 0x3f) {
 error_setg(errp,
-- 
2.30.0

Re: [PATCH] iotests/118: Drop 'change' test

[Paolo Bonzini]

On 26/01/21 11:48, Max Reitz wrote:

Commit 0afec75734331 removed the 'change' QMP command, so we can no
longer test it in 118.

Fixes: 0afec75734331a0b52fa3aa4235220eda8c7846f
('qmp: remove deprecated "change" command')
Signed-off-by: Max Reitz 
---
  tests/qemu-iotests/118 | 17 -
  tests/qemu-iotests/118.out |  4 ++--
  2 files changed, 2 insertions(+), 19 deletions(-)


How come this didn't break GitLab CI?...

Paolo


diff --git a/tests/qemu-iotests/118 b/tests/qemu-iotests/118
index 1a2e219057..88e8354e21 100755
--- a/tests/qemu-iotests/118
+++ b/tests/qemu-iotests/118
@@ -74,23 +74,6 @@ class ChangeBaseClass(iotests.QMPTestCase):
  
  class GeneralChangeTestsBaseClass(ChangeBaseClass):
  
-def test_change(self):

-# 'change' requires a drive name, so skip the test for blockdev
-if not self.use_drive:
-return
-
-result = self.vm.qmp('change', device='drive0', target=new_img,
-   arg=iotests.imgfmt)
-self.assert_qmp(result, 'return', {})
-
-self.wait_for_open()
-self.wait_for_close()
-
-result = self.vm.qmp('query-block')
-if self.has_real_tray:
-self.assert_qmp(result, 'return[0]/tray_open', False)
-self.assert_qmp(result, 'return[0]/inserted/image/filename', new_img)
-
  def test_blockdev_change_medium(self):
  result = self.vm.qmp('blockdev-change-medium',
   id=self.device_name, filename=new_img,
diff --git a/tests/qemu-iotests/118.out b/tests/qemu-iotests/118.out
index bf5bfd5aca..0a70391105 100644
--- a/tests/qemu-iotests/118.out
+++ b/tests/qemu-iotests/118.out
@@ -1,5 +1,5 @@
-...
+...
  --
-Ran 167 tests
+Ran 155 tests
  
  OK

[PATCH V2 2/7] block/rbd: store object_size in BDRVRBDState

[Peter Lieven]

Signed-off-by: Peter Lieven 
---
 block/rbd.c | 18 +++---
 1 file changed, 7 insertions(+), 11 deletions(-)

diff --git a/block/rbd.c b/block/rbd.c
index a191c74619..1028596c68 100644
--- a/block/rbd.c
+++ b/block/rbd.c
@@ -90,6 +90,7 @@ typedef struct BDRVRBDState {
 char *snap;
 char *namespace;
 uint64_t image_size;
+uint64_t object_size;
 } BDRVRBDState;
 
 static int qemu_rbd_connect(rados_t *cluster, rados_ioctx_t *io_ctx,
@@ -663,6 +664,7 @@ static int qemu_rbd_open(BlockDriverState *bs, QDict 
*options, int flags,
 const QDictEntry *e;
 Error *local_err = NULL;
 char *keypairs, *secretid;
+rbd_image_info_t info;
 int r;
 
 keypairs = g_strdup(qdict_get_try_str(options, "=keyvalue-pairs"));
@@ -727,13 +729,15 @@ static int qemu_rbd_open(BlockDriverState *bs, QDict 
*options, int flags,
 goto failed_open;
 }
 
-r = rbd_get_size(s->image, >image_size);
+r = rbd_stat(s->image, , sizeof(info));
 if (r < 0) {
-error_setg_errno(errp, -r, "error getting image size from %s",
+error_setg_errno(errp, -r, "error getting image info from %s",
  s->image_name);
 rbd_close(s->image);
 goto failed_open;
 }
+s->image_size = info.size;
+s->object_size = info.obj_size;
 
 /* If we are using an rbd snapshot, we must be r/o, otherwise
  * leave as-is */
@@ -945,15 +949,7 @@ static BlockAIOCB *qemu_rbd_aio_flush(BlockDriverState *bs,
 static int qemu_rbd_getinfo(BlockDriverState *bs, BlockDriverInfo *bdi)
 {
 BDRVRBDState *s = bs->opaque;
-rbd_image_info_t info;
-int r;
-
-r = rbd_stat(s->image, , sizeof(info));
-if (r < 0) {
-return r;
-}
-
-bdi->cluster_size = info.obj_size;
+bdi->cluster_size = s->object_size;
 return 0;
 }
 
-- 
2.17.1

[PATCH V2 4/7] block/rbd: add bdrv_attach_aio_context

[Peter Lieven]

Signed-off-by: Peter Lieven 
---
 block/rbd.c | 15 +--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/block/rbd.c b/block/rbd.c
index f68ebcf240..7abd0252c9 100644
--- a/block/rbd.c
+++ b/block/rbd.c
@@ -91,6 +91,7 @@ typedef struct BDRVRBDState {
 char *namespace;
 uint64_t image_size;
 uint64_t object_size;
+AioContext *aio_context;
 } BDRVRBDState;
 
 static int qemu_rbd_connect(rados_t *cluster, rados_ioctx_t *io_ctx,
@@ -749,6 +750,8 @@ static int qemu_rbd_open(BlockDriverState *bs, QDict 
*options, int flags,
 }
 }
 
+s->aio_context = bdrv_get_aio_context(bs);
+
 /* When extending regular files, we get zeros from the OS */
 bs->supported_truncate_flags = BDRV_REQ_ZERO_WRITE;
 
@@ -839,8 +842,7 @@ static void rbd_finish_aiocb(rbd_completion_t c, RADOSCB 
*rcb)
 rcb->ret = rbd_aio_get_return_value(c);
 rbd_aio_release(c);
 
-replay_bh_schedule_oneshot_event(bdrv_get_aio_context(acb->common.bs),
- rbd_finish_bh, rcb);
+replay_bh_schedule_oneshot_event(acb->s->aio_context, rbd_finish_bh, rcb);
 }
 
 static BlockAIOCB *rbd_start_aio(BlockDriverState *bs,
@@ -1160,6 +1162,13 @@ static const char *const qemu_rbd_strong_runtime_opts[] 
= {
 NULL
 };
 
+static void qemu_rbd_attach_aio_context(BlockDriverState *bs,
+   AioContext *new_context)
+{
+BDRVRBDState *s = bs->opaque;
+s->aio_context = new_context;
+}
+
 static BlockDriver bdrv_rbd = {
 .format_name= "rbd",
 .instance_size  = sizeof(BDRVRBDState),
@@ -1189,6 +1198,8 @@ static BlockDriver bdrv_rbd = {
 .bdrv_snapshot_goto = qemu_rbd_snap_rollback,
 .bdrv_co_invalidate_cache = qemu_rbd_co_invalidate_cache,
 
+.bdrv_attach_aio_context  = qemu_rbd_attach_aio_context,
+
 .strong_runtime_opts= qemu_rbd_strong_runtime_opts,
 };
 
-- 
2.17.1

[PATCH V2 5/7] block/rbd: migrate from aio to coroutines

[Peter Lieven]

Signed-off-by: Peter Lieven 
---
 block/rbd.c | 253 ++--
 1 file changed, 86 insertions(+), 167 deletions(-)

diff --git a/block/rbd.c b/block/rbd.c
index 7abd0252c9..d11a3c6dd1 100644
--- a/block/rbd.c
+++ b/block/rbd.c
@@ -66,22 +66,6 @@ typedef enum {
 RBD_AIO_FLUSH
 } RBDAIOCmd;
 
-typedef struct RBDAIOCB {
-BlockAIOCB common;
-int64_t ret;
-QEMUIOVector *qiov;
-RBDAIOCmd cmd;
-int error;
-struct BDRVRBDState *s;
-} RBDAIOCB;
-
-typedef struct RADOSCB {
-RBDAIOCB *acb;
-struct BDRVRBDState *s;
-int64_t size;
-int64_t ret;
-} RADOSCB;
-
 typedef struct BDRVRBDState {
 rados_t cluster;
 rados_ioctx_t io_ctx;
@@ -94,6 +78,13 @@ typedef struct BDRVRBDState {
 AioContext *aio_context;
 } BDRVRBDState;
 
+typedef struct RBDTask {
+BDRVRBDState *s;
+Coroutine *co;
+bool complete;
+int64_t ret;
+} RBDTask;
+
 static int qemu_rbd_connect(rados_t *cluster, rados_ioctx_t *io_ctx,
 BlockdevOptionsRbd *opts, bool cache,
 const char *keypairs, const char *secretid,
@@ -316,13 +307,6 @@ static int qemu_rbd_set_keypairs(rados_t cluster, const 
char *keypairs_json,
 return ret;
 }
 
-static void qemu_rbd_memset(RADOSCB *rcb, int64_t offs)
-{
-RBDAIOCB *acb = rcb->acb;
-iov_memset(acb->qiov->iov, acb->qiov->niov, offs, 0,
-   acb->qiov->size - offs);
-}
-
 /* FIXME Deprecate and remove keypairs or make it available in QMP. */
 static int qemu_rbd_do_create(BlockdevCreateOptions *options,
   const char *keypairs, const char 
*password_secret,
@@ -440,46 +424,6 @@ exit:
 return ret;
 }
 
-/*
- * This aio completion is being called from rbd_finish_bh() and runs in qemu
- * BH context.
- */
-static void qemu_rbd_complete_aio(RADOSCB *rcb)
-{
-RBDAIOCB *acb = rcb->acb;
-int64_t r;
-
-r = rcb->ret;
-
-if (acb->cmd != RBD_AIO_READ) {
-if (r < 0) {
-acb->ret = r;
-acb->error = 1;
-} else if (!acb->error) {
-acb->ret = rcb->size;
-}
-} else {
-if (r < 0) {
-qemu_rbd_memset(rcb, 0);
-acb->ret = r;
-acb->error = 1;
-} else if (r < rcb->size) {
-qemu_rbd_memset(rcb, r);
-if (!acb->error) {
-acb->ret = rcb->size;
-}
-} else if (!acb->error) {
-acb->ret = r;
-}
-}
-
-g_free(rcb);
-
-acb->common.cb(acb->common.opaque, (acb->ret > 0 ? 0 : acb->ret));
-
-qemu_aio_unref(acb);
-}
-
 static char *qemu_rbd_mon_host(BlockdevOptionsRbd *opts, Error **errp)
 {
 const char **vals;
@@ -817,88 +761,49 @@ static int qemu_rbd_resize(BlockDriverState *bs, uint64_t 
size)
 return 0;
 }
 
-static const AIOCBInfo rbd_aiocb_info = {
-.aiocb_size = sizeof(RBDAIOCB),
-};
-
-static void rbd_finish_bh(void *opaque)
+static void qemu_rbd_finish_bh(void *opaque)
 {
-RADOSCB *rcb = opaque;
-qemu_rbd_complete_aio(rcb);
+RBDTask *task = opaque;
+task->complete = 1;
+aio_co_wake(task->co);
 }
 
-/*
- * This is the callback function for rbd_aio_read and _write
- *
- * Note: this function is being called from a non qemu thread so
- * we need to be careful about what we do here. Generally we only
- * schedule a BH, and do the rest of the io completion handling
- * from rbd_finish_bh() which runs in a qemu context.
- */
-static void rbd_finish_aiocb(rbd_completion_t c, RADOSCB *rcb)
+static void qemu_rbd_completion_cb(rbd_completion_t c, RBDTask *task)
 {
-RBDAIOCB *acb = rcb->acb;
-
-rcb->ret = rbd_aio_get_return_value(c);
+task->ret = rbd_aio_get_return_value(c);
 rbd_aio_release(c);
-
-replay_bh_schedule_oneshot_event(acb->s->aio_context, rbd_finish_bh, rcb);
+aio_bh_schedule_oneshot(task->s->aio_context, qemu_rbd_finish_bh, task);
 }
 
-static BlockAIOCB *rbd_start_aio(BlockDriverState *bs,
- int64_t off,
- QEMUIOVector *qiov,
- int64_t size,
- BlockCompletionFunc *cb,
- void *opaque,
- RBDAIOCmd cmd)
+static int coroutine_fn qemu_rbd_start_co(BlockDriverState *bs,
+  uint64_t offset,
+  uint64_t bytes,
+  QEMUIOVector *qiov,
+  int flags,
+  RBDAIOCmd cmd)
 {
-RBDAIOCB *acb;
-RADOSCB *rcb = NULL;
+BDRVRBDState *s = bs->opaque;
+RBDTask task = { .s = s, .co = qemu_coroutine_self() };
 rbd_completion_t c;
 int r;
 
-BDRVRBDState *s = bs->opaque;
-
-acb = qemu_aio_get(_aiocb_info, bs, cb, opaque);
-acb->cmd = cmd;
-acb->qiov = qiov;
- 

[PATCH V2 0/7] block/rbd: migrate to coroutines and add write zeroes support

[Peter Lieven]

this series migrates the qemu rbd driver from the old aio emulation
to native coroutines and adds write zeroes support which is important
for block operations.

To achive this we first bump the librbd requirement to the already
outdated luminous release of ceph to get rid of some wrappers and
ifdef'ry in the code.

V1->V2:
 - this patch is now rebased on top of current master with Paolos
   upcoming fixes for the meson.build script included:
- meson: accept either shared or static libraries if --disable-static
- meson: honor --enable-rbd if cc.links test fails
 - Patch 1: adjusted to meson.build script
 - Patch 2: unchanged
 - Patch 3: new patch
 - Patch 4: do not implement empty detach_aio_context callback [Jason]
 - Patch 5: - fix aio completion cleanup in error case [Jason]
- return error codes from librbd
 - Patch 6: - add support for thick provisioning [Jason]
- do not set write zeroes alignment
 - Patch 7: new patch

Peter Lieven (7):
  block/rbd: bump librbd requirement to luminous release
  block/rbd: store object_size in BDRVRBDState
  block/rbd: update s->image_size in qemu_rbd_getlength
  block/rbd: add bdrv_attach_aio_context
  block/rbd: migrate from aio to coroutines
  block/rbd: add write zeroes support
  block/rbd: drop qemu_rbd_refresh_limits

 block/rbd.c | 418 +---
 meson.build |  13 +-
 2 files changed, 142 insertions(+), 289 deletions(-)

-- 
2.17.1

[PATCH V2 6/7] block/rbd: add write zeroes support

[Peter Lieven]

Signed-off-by: Peter Lieven 
---
 block/rbd.c | 36 +++-
 1 file changed, 35 insertions(+), 1 deletion(-)

diff --git a/block/rbd.c b/block/rbd.c
index d11a3c6dd1..35dc1dc90e 100644
--- a/block/rbd.c
+++ b/block/rbd.c
@@ -63,7 +63,8 @@ typedef enum {
 RBD_AIO_READ,
 RBD_AIO_WRITE,
 RBD_AIO_DISCARD,
-RBD_AIO_FLUSH
+RBD_AIO_FLUSH,
+RBD_AIO_WRITE_ZEROES
 } RBDAIOCmd;
 
 typedef struct BDRVRBDState {
@@ -695,6 +696,9 @@ static int qemu_rbd_open(BlockDriverState *bs, QDict 
*options, int flags,
 }
 
 s->aio_context = bdrv_get_aio_context(bs);
+#ifdef LIBRBD_SUPPORTS_WRITE_ZEROES
+bs->supported_zero_flags = BDRV_REQ_MAY_UNMAP;
+#endif
 
 /* When extending regular files, we get zeros from the OS */
 bs->supported_truncate_flags = BDRV_REQ_ZERO_WRITE;
@@ -808,6 +812,18 @@ static int coroutine_fn qemu_rbd_start_co(BlockDriverState 
*bs,
 case RBD_AIO_FLUSH:
 r = rbd_aio_flush(s->image, c);
 break;
+#ifdef LIBRBD_SUPPORTS_WRITE_ZEROES
+case RBD_AIO_WRITE_ZEROES: {
+int zero_flags = 0;
+#ifdef RBD_WRITE_ZEROES_FLAG_THICK_PROVISION
+if (!(flags & BDRV_REQ_MAY_UNMAP)) {
+zero_flags = RBD_WRITE_ZEROES_FLAG_THICK_PROVISION;
+}
+#endif
+r = rbd_aio_write_zeroes(s->image, offset, bytes, c, zero_flags, 0);
+break;
+}
+#endif
 default:
 r = -EINVAL;
 }
@@ -878,6 +894,21 @@ static int coroutine_fn 
qemu_rbd_co_pdiscard(BlockDriverState *bs,
 return qemu_rbd_start_co(bs, offset, count, NULL, 0, RBD_AIO_DISCARD);
 }
 
+#ifdef LIBRBD_SUPPORTS_WRITE_ZEROES
+static int
+coroutine_fn qemu_rbd_co_pwrite_zeroes(BlockDriverState *bs, int64_t offset,
+  int count, BdrvRequestFlags flags)
+{
+#ifndef RBD_WRITE_ZEROES_FLAG_THICK_PROVISION
+if (!(flags & BDRV_REQ_MAY_UNMAP)) {
+return -ENOTSUP;
+}
+#endif
+return qemu_rbd_start_co(bs, offset, count, NULL, flags,
+ RBD_AIO_WRITE_ZEROES);
+}
+#endif
+
 static int qemu_rbd_getinfo(BlockDriverState *bs, BlockDriverInfo *bdi)
 {
 BDRVRBDState *s = bs->opaque;
@@ -1110,6 +1141,9 @@ static BlockDriver bdrv_rbd = {
 .bdrv_co_pwritev= qemu_rbd_co_pwritev,
 .bdrv_co_flush_to_disk  = qemu_rbd_co_flush,
 .bdrv_co_pdiscard   = qemu_rbd_co_pdiscard,
+#ifdef LIBRBD_SUPPORTS_WRITE_ZEROES
+.bdrv_co_pwrite_zeroes  = qemu_rbd_co_pwrite_zeroes,
+#endif
 
 .bdrv_snapshot_create   = qemu_rbd_snap_create,
 .bdrv_snapshot_delete   = qemu_rbd_snap_remove,
-- 
2.17.1

[PATCH V2 3/7] block/rbd: update s->image_size in qemu_rbd_getlength

[Peter Lieven]

in case the image size changed we should adjust our internally stored size as 
well.

Signed-off-by: Peter Lieven 
---
 block/rbd.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/block/rbd.c b/block/rbd.c
index 1028596c68..f68ebcf240 100644
--- a/block/rbd.c
+++ b/block/rbd.c
@@ -964,6 +964,7 @@ static int64_t qemu_rbd_getlength(BlockDriverState *bs)
 return r;
 }
 
+s->image_size = info.size;
 return info.size;
 }
 
-- 
2.17.1

[PATCH V2 1/7] block/rbd: bump librbd requirement to luminous release

[Peter Lieven]

even luminous (version 12.2) is unmaintained for over 3 years now.
Bump the requirement to get rid of the ifdef'ry in the code.

Signed-off-by: Peter Lieven 
---
 block/rbd.c | 120 
 meson.build |  13 --
 2 files changed, 17 insertions(+), 116 deletions(-)

diff --git a/block/rbd.c b/block/rbd.c
index 9071a00e3f..a191c74619 100644
--- a/block/rbd.c
+++ b/block/rbd.c
@@ -55,24 +55,10 @@
  * leading "\".
  */
 
-/* rbd_aio_discard added in 0.1.2 */
-#if LIBRBD_VERSION_CODE >= LIBRBD_VERSION(0, 1, 2)
-#define LIBRBD_SUPPORTS_DISCARD
-#else
-#undef LIBRBD_SUPPORTS_DISCARD
-#endif
-
 #define OBJ_MAX_SIZE (1UL << OBJ_DEFAULT_OBJ_ORDER)
 
 #define RBD_MAX_SNAPS 100
 
-/* The LIBRBD_SUPPORTS_IOVEC is defined in librbd.h */
-#ifdef LIBRBD_SUPPORTS_IOVEC
-#define LIBRBD_USE_IOVEC 1
-#else
-#define LIBRBD_USE_IOVEC 0
-#endif
-
 typedef enum {
 RBD_AIO_READ,
 RBD_AIO_WRITE,
@@ -84,7 +70,6 @@ typedef struct RBDAIOCB {
 BlockAIOCB common;
 int64_t ret;
 QEMUIOVector *qiov;
-char *bounce;
 RBDAIOCmd cmd;
 int error;
 struct BDRVRBDState *s;
@@ -94,7 +79,6 @@ typedef struct RADOSCB {
 RBDAIOCB *acb;
 struct BDRVRBDState *s;
 int64_t size;
-char *buf;
 int64_t ret;
 } RADOSCB;
 
@@ -332,13 +316,9 @@ static int qemu_rbd_set_keypairs(rados_t cluster, const 
char *keypairs_json,
 
 static void qemu_rbd_memset(RADOSCB *rcb, int64_t offs)
 {
-if (LIBRBD_USE_IOVEC) {
-RBDAIOCB *acb = rcb->acb;
-iov_memset(acb->qiov->iov, acb->qiov->niov, offs, 0,
-   acb->qiov->size - offs);
-} else {
-memset(rcb->buf + offs, 0, rcb->size - offs);
-}
+RBDAIOCB *acb = rcb->acb;
+iov_memset(acb->qiov->iov, acb->qiov->niov, offs, 0,
+   acb->qiov->size - offs);
 }
 
 /* FIXME Deprecate and remove keypairs or make it available in QMP. */
@@ -493,13 +473,6 @@ static void qemu_rbd_complete_aio(RADOSCB *rcb)
 
 g_free(rcb);
 
-if (!LIBRBD_USE_IOVEC) {
-if (acb->cmd == RBD_AIO_READ) {
-qemu_iovec_from_buf(acb->qiov, 0, acb->bounce, acb->qiov->size);
-}
-qemu_vfree(acb->bounce);
-}
-
 acb->common.cb(acb->common.opaque, (acb->ret > 0 ? 0 : acb->ret));
 
 qemu_aio_unref(acb);
@@ -866,28 +839,6 @@ static void rbd_finish_aiocb(rbd_completion_t c, RADOSCB 
*rcb)
  rbd_finish_bh, rcb);
 }
 
-static int rbd_aio_discard_wrapper(rbd_image_t image,
-   uint64_t off,
-   uint64_t len,
-   rbd_completion_t comp)
-{
-#ifdef LIBRBD_SUPPORTS_DISCARD
-return rbd_aio_discard(image, off, len, comp);
-#else
-return -ENOTSUP;
-#endif
-}
-
-static int rbd_aio_flush_wrapper(rbd_image_t image,
- rbd_completion_t comp)
-{
-#ifdef LIBRBD_SUPPORTS_AIO_FLUSH
-return rbd_aio_flush(image, comp);
-#else
-return -ENOTSUP;
-#endif
-}
-
 static BlockAIOCB *rbd_start_aio(BlockDriverState *bs,
  int64_t off,
  QEMUIOVector *qiov,
@@ -910,21 +861,6 @@ static BlockAIOCB *rbd_start_aio(BlockDriverState *bs,
 
 rcb = g_new(RADOSCB, 1);
 
-if (!LIBRBD_USE_IOVEC) {
-if (cmd == RBD_AIO_DISCARD || cmd == RBD_AIO_FLUSH) {
-acb->bounce = NULL;
-} else {
-acb->bounce = qemu_try_blockalign(bs, qiov->size);
-if (acb->bounce == NULL) {
-goto failed;
-}
-}
-if (cmd == RBD_AIO_WRITE) {
-qemu_iovec_to_buf(acb->qiov, 0, acb->bounce, qiov->size);
-}
-rcb->buf = acb->bounce;
-}
-
 acb->ret = 0;
 acb->error = 0;
 acb->s = s;
@@ -938,7 +874,7 @@ static BlockAIOCB *rbd_start_aio(BlockDriverState *bs,
 }
 
 switch (cmd) {
-case RBD_AIO_WRITE: {
+case RBD_AIO_WRITE:
 /*
  * RBD APIs don't allow us to write more than actual size, so in order
  * to support growing images, we resize the image before write
@@ -950,25 +886,16 @@ static BlockAIOCB *rbd_start_aio(BlockDriverState *bs,
 goto failed_completion;
 }
 }
-#ifdef LIBRBD_SUPPORTS_IOVEC
-r = rbd_aio_writev(s->image, qiov->iov, qiov->niov, off, c);
-#else
-r = rbd_aio_write(s->image, off, size, rcb->buf, c);
-#endif
+r = rbd_aio_writev(s->image, qiov->iov, qiov->niov, off, c);
 break;
-}
 case RBD_AIO_READ:
-#ifdef LIBRBD_SUPPORTS_IOVEC
-r = rbd_aio_readv(s->image, qiov->iov, qiov->niov, off, c);
-#else
-r = rbd_aio_read(s->image, off, size, rcb->buf, c);
-#endif
+r = rbd_aio_readv(s->image, qiov->iov, qiov->niov, off, c);
 break;
 case RBD_AIO_DISCARD:
-r = rbd_aio_discard_wrapper(s->image, off, size, c);
+r = rbd_aio_discard(s->image, off, size, c);
 

[PATCH V2 7/7] block/rbd: drop qemu_rbd_refresh_limits

[Peter Lieven]

librbd supports 1 byte alignment for all aio operations.

Currently, there is no API call to query limits from the ceph backend.
So drop the bdrv_refresh_limits completely until there is such an API call.

Signed-off-by: Peter Lieven 
---
 block/rbd.c | 9 -
 1 file changed, 9 deletions(-)

diff --git a/block/rbd.c b/block/rbd.c
index 35dc1dc90e..5f96fbf3d1 100644
--- a/block/rbd.c
+++ b/block/rbd.c
@@ -219,14 +219,6 @@ done:
 return;
 }
 
-
-static void qemu_rbd_refresh_limits(BlockDriverState *bs, Error **errp)
-{
-/* XXX Does RBD support AIO on less than 512-byte alignment? */
-bs->bl.request_alignment = 512;
-}
-
-
 static int qemu_rbd_set_auth(rados_t cluster, BlockdevOptionsRbd *opts,
  Error **errp)
 {
@@ -1124,7 +1116,6 @@ static BlockDriver bdrv_rbd = {
 .format_name= "rbd",
 .instance_size  = sizeof(BDRVRBDState),
 .bdrv_parse_filename= qemu_rbd_parse_filename,
-.bdrv_refresh_limits= qemu_rbd_refresh_limits,
 .bdrv_file_open = qemu_rbd_open,
 .bdrv_close = qemu_rbd_close,
 .bdrv_reopen_prepare= qemu_rbd_reopen_prepare,
-- 
2.17.1

Re: [PATCH v2 0/2] tests/qtest: Only run fuzz-tests when tested devices are available

[Philippe Mathieu-Daudé]

On 1/26/21 12:16 PM, Philippe Mathieu-Daudé wrote:
> Some tests/qtest/fuzz-test fail when the device tested is
> not available in the build. Fix this by only running the
> test when devices are available.

Forgot, since v1:

- Do not make the testing generic, keep it restricted to x86 (thuth)

> FWIW Alexander Bulekov suggested an improvement, putting each
> test in a directory named by the device tested. This series
> does not cover that.
> 
> Supersedes: <20210115150936.282-1-phi...@redhat.com>
> 
> Philippe Mathieu-Daudé (2):
>   tests/qtest: Only run fuzz-megasas-test if megasas device is available
>   tests/qtest: Only run fuzz-virtio-scsi when virtio-scsi is available

Re: [PATCH] iotests/118: Drop 'change' test

[Kevin Wolf]

Am 26.01.2021 um 11:48 hat Max Reitz geschrieben:
> Commit 0afec75734331 removed the 'change' QMP command, so we can no
> longer test it in 118.
> 
> Fixes: 0afec75734331a0b52fa3aa4235220eda8c7846f
>('qmp: remove deprecated "change" command')
> Signed-off-by: Max Reitz 

Thanks, applied to the block branch.

(I was just about to make the same change myself...)

If you don't mind, I'd squash in something like the following.

Kevin

diff --git a/tests/qemu-iotests/118 b/tests/qemu-iotests/118
index 88e8354e21..cae52ffa5e 100755
--- a/tests/qemu-iotests/118
+++ b/tests/qemu-iotests/118
@@ -1,8 +1,7 @@
 #!/usr/bin/env python3
 # group: rw
 #
-# Test case for the QMP 'change' command and all other associated
-# commands
+# Test case for media change monitor commands
 #
 # Copyright (C) 2015 Red Hat, Inc.
 #

[PATCH v2 2/2] tests/qtest: Only run fuzz-virtio-scsi when virtio-scsi is available

[Philippe Mathieu-Daudé]

This test fails when QEMU is built without the virtio-scsi device,
restrict it to its availability.

Reviewed-by: Michael S. Tsirkin 
Signed-off-by: Philippe Mathieu-Daudé 
---
 tests/qtest/fuzz-test.c | 51 
 tests/qtest/fuzz-virtio-scsi-test.c | 75 +
 MAINTAINERS |  1 +
 tests/qtest/meson.build |  1 +
 4 files changed, 77 insertions(+), 51 deletions(-)
 create mode 100644 tests/qtest/fuzz-virtio-scsi-test.c

diff --git a/tests/qtest/fuzz-test.c b/tests/qtest/fuzz-test.c
index 6188fbb8e96..d112798afe3 100644
--- a/tests/qtest/fuzz-test.c
+++ b/tests/qtest/fuzz-test.c
@@ -25,55 +25,6 @@ static void test_lp1878642_pci_bus_get_irq_level_assert(void)
 qtest_quit(s);
 }
 
-/*
- * Here a MemoryRegionCache pointed to an MMIO region but had a
- * larger size than the underlying region.
- */
-static void test_mmio_oob_from_memory_region_cache(void)
-{
-QTestState *s;
-
-s = qtest_init("-M pc-q35-5.2 -display none -m 512M "
-  "-device virtio-scsi,num_queues=8,addr=03.0 ");
-
-qtest_outl(s, 0xcf8, 0x80001811);
-qtest_outb(s, 0xcfc, 0x6e);
-qtest_outl(s, 0xcf8, 0x80001824);
-qtest_outl(s, 0xcf8, 0x80001813);
-qtest_outl(s, 0xcfc, 0xa08);
-qtest_outl(s, 0xcf8, 0x80001802);
-qtest_outl(s, 0xcfc, 0x5a175a63);
-qtest_outb(s, 0x6e08, 0x9e);
-qtest_writeb(s, 0x9f003, 0xff);
-qtest_writeb(s, 0x9f004, 0x01);
-qtest_writeb(s, 0x9e012, 0x0e);
-qtest_writeb(s, 0x9e01b, 0x0e);
-qtest_writeb(s, 0x9f006, 0x01);
-qtest_writeb(s, 0x9f008, 0x01);
-qtest_writeb(s, 0x9f00a, 0x01);
-qtest_writeb(s, 0x9f00c, 0x01);
-qtest_writeb(s, 0x9f00e, 0x01);
-qtest_writeb(s, 0x9f010, 0x01);
-qtest_writeb(s, 0x9f012, 0x01);
-qtest_writeb(s, 0x9f014, 0x01);
-qtest_writeb(s, 0x9f016, 0x01);
-qtest_writeb(s, 0x9f018, 0x01);
-qtest_writeb(s, 0x9f01a, 0x01);
-qtest_writeb(s, 0x9f01c, 0x01);
-qtest_writeb(s, 0x9f01e, 0x01);
-qtest_writeb(s, 0x9f020, 0x01);
-qtest_writeb(s, 0x9f022, 0x01);
-qtest_writeb(s, 0x9f024, 0x01);
-qtest_writeb(s, 0x9f026, 0x01);
-qtest_writeb(s, 0x9f028, 0x01);
-qtest_writeb(s, 0x9f02a, 0x01);
-qtest_writeb(s, 0x9f02c, 0x01);
-qtest_writeb(s, 0x9f02e, 0x01);
-qtest_writeb(s, 0x9f030, 0x01);
-qtest_outb(s, 0x6e10, 0x00);
-qtest_quit(s);
-}
-
 int main(int argc, char **argv)
 {
 const char *arch = qtest_get_arch();
@@ -83,8 +34,6 @@ int main(int argc, char **argv)
 if (strcmp(arch, "i386") == 0 || strcmp(arch, "x86_64") == 0) {
 qtest_add_func("fuzz/test_lp1878642_pci_bus_get_irq_level_assert",
test_lp1878642_pci_bus_get_irq_level_assert);
-qtest_add_func("fuzz/test_mmio_oob_from_memory_region_cache",
-   test_mmio_oob_from_memory_region_cache);
 }
 
 return g_test_run();
diff --git a/tests/qtest/fuzz-virtio-scsi-test.c 
b/tests/qtest/fuzz-virtio-scsi-test.c
new file mode 100644
index 000..aaf6d10e189
--- /dev/null
+++ b/tests/qtest/fuzz-virtio-scsi-test.c
@@ -0,0 +1,75 @@
+/*
+ * QTest fuzzer-generated testcase for virtio-scsi device
+ *
+ * Copyright (c) 2020 Li Qiang 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+
+#include "libqos/libqtest.h"
+
+/*
+ * Here a MemoryRegionCache pointed to an MMIO region but had a
+ * larger size than the underlying region.
+ */
+static void test_mmio_oob_from_memory_region_cache(void)
+{
+QTestState *s;
+
+s = qtest_init("-M pc-q35-5.2 -display none -m 512M "
+   "-device virtio-scsi,num_queues=8,addr=03.0 ");
+
+qtest_outl(s, 0xcf8, 0x80001811);
+qtest_outb(s, 0xcfc, 0x6e);
+qtest_outl(s, 0xcf8, 0x80001824);
+qtest_outl(s, 0xcf8, 0x80001813);
+qtest_outl(s, 0xcfc, 0xa08);
+qtest_outl(s, 0xcf8, 0x80001802);
+qtest_outl(s, 0xcfc, 0x5a175a63);
+qtest_outb(s, 0x6e08, 0x9e);
+qtest_writeb(s, 0x9f003, 0xff);
+qtest_writeb(s, 0x9f004, 0x01);
+qtest_writeb(s, 0x9e012, 0x0e);
+qtest_writeb(s, 0x9e01b, 0x0e);
+qtest_writeb(s, 0x9f006, 0x01);
+qtest_writeb(s, 0x9f008, 0x01);
+qtest_writeb(s, 0x9f00a, 0x01);
+qtest_writeb(s, 0x9f00c, 0x01);
+qtest_writeb(s, 0x9f00e, 0x01);
+qtest_writeb(s, 0x9f010, 0x01);
+qtest_writeb(s, 0x9f012, 0x01);
+qtest_writeb(s, 0x9f014, 0x01);
+qtest_writeb(s, 0x9f016, 0x01);
+qtest_writeb(s, 0x9f018, 0x01);
+qtest_writeb(s, 0x9f01a, 0x01);
+qtest_writeb(s, 0x9f01c, 0x01);
+qtest_writeb(s, 0x9f01e, 0x01);
+qtest_writeb(s, 0x9f020, 0x01);
+qtest_writeb(s, 0x9f022, 0x01);
+qtest_writeb(s, 0x9f024, 0x01);
+qtest_writeb(s, 0x9f026, 0x01);
+qtest_writeb(s, 0x9f028, 0x01);
+qtest_writeb(s, 0x9f02a, 0x01);
+qtest_writeb(s, 0x9f02c, 0x01);
+qtest_writeb(s, 

[PATCH v2 1/2] tests/qtest: Only run fuzz-megasas-test if megasas device is available

[Philippe Mathieu-Daudé]

This test fails when QEMU is built without the megasas device,
restrict it to its availability.

Signed-off-by: Philippe Mathieu-Daudé 
---
 tests/qtest/fuzz-megasas-test.c | 49 +
 tests/qtest/fuzz-test.c | 25 -
 MAINTAINERS |  1 +
 tests/qtest/meson.build |  4 ++-
 4 files changed, 53 insertions(+), 26 deletions(-)
 create mode 100644 tests/qtest/fuzz-megasas-test.c

diff --git a/tests/qtest/fuzz-megasas-test.c b/tests/qtest/fuzz-megasas-test.c
new file mode 100644
index 000..940a76bf25a
--- /dev/null
+++ b/tests/qtest/fuzz-megasas-test.c
@@ -0,0 +1,49 @@
+/*
+ * QTest fuzzer-generated testcase for megasas device
+ *
+ * Copyright (c) 2020 Li Qiang 
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+
+#include "libqos/libqtest.h"
+
+/*
+ * This used to trigger the assert in scsi_dma_complete
+ * https://bugs.launchpad.net/qemu/+bug/1878263
+ */
+static void test_lp1878263_megasas_zero_iov_cnt(void)
+{
+QTestState *s;
+
+s = qtest_init("-nographic -monitor none -serial none "
+   "-M q35 -device megasas -device scsi-cd,drive=null0 "
+   "-blockdev driver=null-co,read-zeroes=on,node-name=null0");
+qtest_outl(s, 0xcf8, 0x80001818);
+qtest_outl(s, 0xcfc, 0xc101);
+qtest_outl(s, 0xcf8, 0x8000181c);
+qtest_outl(s, 0xcf8, 0x80001804);
+qtest_outw(s, 0xcfc, 0x7);
+qtest_outl(s, 0xcf8, 0x8000186a);
+qtest_writeb(s, 0x14, 0xfe);
+qtest_writeb(s, 0x0, 0x02);
+qtest_outb(s, 0xc1c0, 0x17);
+qtest_quit(s);
+}
+
+int main(int argc, char **argv)
+{
+const char *arch = qtest_get_arch();
+
+g_test_init(, , NULL);
+
+if (strcmp(arch, "i386") == 0 || strcmp(arch, "x86_64") == 0) {
+qtest_add_func("fuzz/test_lp1878263_megasas_zero_iov_cnt",
+   test_lp1878263_megasas_zero_iov_cnt);
+}
+
+return g_test_run();
+}
diff --git a/tests/qtest/fuzz-test.c b/tests/qtest/fuzz-test.c
index cdb1100a0b8..6188fbb8e96 100644
--- a/tests/qtest/fuzz-test.c
+++ b/tests/qtest/fuzz-test.c
@@ -11,29 +11,6 @@
 
 #include "libqos/libqtest.h"
 
-/*
- * This used to trigger the assert in scsi_dma_complete
- * https://bugs.launchpad.net/qemu/+bug/1878263
- */
-static void test_lp1878263_megasas_zero_iov_cnt(void)
-{
-QTestState *s;
-
-s = qtest_init("-nographic -monitor none -serial none "
-   "-M q35 -device megasas -device scsi-cd,drive=null0 "
-   "-blockdev driver=null-co,read-zeroes=on,node-name=null0");
-qtest_outl(s, 0xcf8, 0x80001818);
-qtest_outl(s, 0xcfc, 0xc101);
-qtest_outl(s, 0xcf8, 0x8000181c);
-qtest_outl(s, 0xcf8, 0x80001804);
-qtest_outw(s, 0xcfc, 0x7);
-qtest_outl(s, 0xcf8, 0x8000186a);
-qtest_writeb(s, 0x14, 0xfe);
-qtest_writeb(s, 0x0, 0x02);
-qtest_outb(s, 0xc1c0, 0x17);
-qtest_quit(s);
-}
-
 static void test_lp1878642_pci_bus_get_irq_level_assert(void)
 {
 QTestState *s;
@@ -104,8 +81,6 @@ int main(int argc, char **argv)
 g_test_init(, , NULL);
 
 if (strcmp(arch, "i386") == 0 || strcmp(arch, "x86_64") == 0) {
-qtest_add_func("fuzz/test_lp1878263_megasas_zero_iov_cnt",
-   test_lp1878263_megasas_zero_iov_cnt);
 qtest_add_func("fuzz/test_lp1878642_pci_bus_get_irq_level_assert",
test_lp1878642_pci_bus_get_irq_level_assert);
 qtest_add_func("fuzz/test_mmio_oob_from_memory_region_cache",
diff --git a/MAINTAINERS b/MAINTAINERS
index 34359a99b8e..44cd74b03cd 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1925,6 +1925,7 @@ S: Supported
 F: hw/scsi/megasas.c
 F: hw/scsi/mfi.h
 F: tests/qtest/megasas-test.c
+F: tests/qtest/fuzz-megasas-test.c
 
 Network packet abstractions
 M: Dmitry Fleytman 
diff --git a/tests/qtest/meson.build b/tests/qtest/meson.build
index 16d04625b8b..85682d0dfce 100644
--- a/tests/qtest/meson.build
+++ b/tests/qtest/meson.build
@@ -4,7 +4,9 @@
   subdir_done()
 endif
 
-qtests_generic = [
+qtests_generic = \
+  (config_all_devices.has_key('CONFIG_MEGASAS_SCSI_PCI') ? 
['fuzz-megasas-test'] : []) + \
+  [
   'cdrom-test',
   'device-introspect-test',
   'machine-none-test',
-- 
2.26.2

[PATCH v2 0/2] tests/qtest: Only run fuzz-tests when tested devices are available

[Philippe Mathieu-Daudé]

Some tests/qtest/fuzz-test fail when the device tested is
not available in the build. Fix this by only running the
test when devices are available.

FWIW Alexander Bulekov suggested an improvement, putting each
test in a directory named by the device tested. This series
does not cover that.

Supersedes: <20210115150936.282-1-phi...@redhat.com>

Philippe Mathieu-Daudé (2):
  tests/qtest: Only run fuzz-megasas-test if megasas device is available
  tests/qtest: Only run fuzz-virtio-scsi when virtio-scsi is available

 tests/qtest/fuzz-megasas-test.c | 49 +++
 tests/qtest/fuzz-test.c | 76 -
 tests/qtest/fuzz-virtio-scsi-test.c | 75 
 MAINTAINERS |  2 +
 tests/qtest/meson.build |  5 +-
 5 files changed, 130 insertions(+), 77 deletions(-)
 create mode 100644 tests/qtest/fuzz-megasas-test.c
 create mode 100644 tests/qtest/fuzz-virtio-scsi-test.c

-- 
2.26.2

Re: [PATCH 3/4] tests/qtest: Only run fuzz-megasas-test if megasas device is available

[Philippe Mathieu-Daudé]

On 1/15/21 11:39 PM, Alexander Bulekov wrote:
> On 210115 1609, Philippe Mathieu-Daudé wrote:
>> This test fails when QEMU is built without the megasas device,
>> restrict it to its availability.
> 
> Should we just make a separate directory for fuzzer tests and have a
> separate source file for each reproducer (or for each device)? That way,
> we avoid confusion about what to do with new reproducers: they always go
> into e.g. tests/qtest/reproducers/device_name.c 

Yes probably. Do you mind sending a patch series?

Re: [PATCH 2/4] tests/qtest: Make fuzz-test generic to all targets

[Philippe Mathieu-Daudé]

On 1/15/21 11:21 PM, Thomas Huth wrote:
> On 15/01/2021 16.09, Philippe Mathieu-Daudé wrote:
>> Tests in fuzz-test's main() already check for the supported
>> architecture before adding tests, therefore this test is not
>> specific to the X86 target. Move it to the generic set.
> 
> As long as it does not run any test on non-x86, it does not make sense
> to move it to the generic set, does it? We'd only waste compile cycles
> that way?

OK, I'll resend this patch when the ARM reproducers are posted.