Re: [PATCH v2 04/15] python: add directory structure README.rst files

2020-10-14 Thread John Snow
On 10/14/20 2:05 PM, Philippe Mathieu-Daudé wrote: On 10/14/20 4:29 PM, John Snow wrote: Add short readmes to python/, python/qemu/, and python/qemu/core that explain the directory hierarchy. These readmes are visible when browsing Maybe readmes -> READMEs If you want it to match the

Re: [PATCH v11 09/13] copy-on-read: skip non-guest reads if no copy needed

2020-10-14 Thread Andrey Shinkevich
On 14.10.2020 15:51, Max Reitz wrote: On 12.10.20 19:43, Andrey Shinkevich wrote: If the flag BDRV_REQ_PREFETCH was set, pass it further to the COR-driver to skip unneeded reading. It can be taken into account for the COR-algorithms optimization. That check is being made during the block stream

Re: [PATCH v11 06/13] block: modify the comment for BDRV_REQ_PREFETCH flag

2020-10-14 Thread Andrey Shinkevich
On 14.10.2020 15:22, Max Reitz wrote: On 12.10.20 19:43, Andrey Shinkevich wrote: Modify the comment for the flag BDRV_REQ_PREFETCH as we are going to use it alone and pass it to the COR-filter driver for further processing. Signed-off-by: Andrey Shinkevich --- include/block/block.h | 7

Re: [PATCH v11 05/13] copy-on-read: limit COR operations to base in COR driver

2020-10-14 Thread Andrey Shinkevich
On 14.10.2020 15:01, Max Reitz wrote: On 12.10.20 19:43, Andrey Shinkevich wrote: Limit COR operations by the base node in the backing chain when the overlay base node name is given. It will be useful for a block stream job when the COR-filter is applied. The overlay base node is passed as the

Re: [PATCH v2 13/15] python: move .isort.cfg into setup.cfg

2020-10-14 Thread Philippe Mathieu-Daudé
On 10/14/20 4:29 PM, John Snow wrote: Signed-off-by: John Snow --- python/.isort.cfg | 7 --- python/setup.cfg | 8 2 files changed, 8 insertions(+), 7 deletions(-) Reviewed-by: Philippe Mathieu-Daudé

Re: [PATCH v2 04/15] python: add directory structure README.rst files

2020-10-14 Thread Philippe Mathieu-Daudé
On 10/14/20 4:29 PM, John Snow wrote: Add short readmes to python/, python/qemu/, and python/qemu/core that explain the directory hierarchy. These readmes are visible when browsing Maybe readmes -> READMEs Otherwise: Reviewed-by: Philippe Mathieu-Daudé the source on e.g. gitlab/github and

Re: [PATCH v2 01/15] python: create qemu.core package

2020-10-14 Thread Philippe Mathieu-Daudé
On 10/14/20 4:29 PM, John Snow wrote: move python/qemu/*.py to python/qemu/core/*.py and update import directives across the tree. This is done to create a PEP420 namespace package, in which we may create subpackages. To do this, the namespace directory ("qemu") should not have any modules in

Re: [PATCH v11 05/13] copy-on-read: limit COR operations to base in COR driver

2020-10-14 Thread Andrey Shinkevich
On 14.10.2020 14:59, Max Reitz wrote: On 12.10.20 19:43, Andrey Shinkevich wrote: Limit COR operations by the base node in the backing chain when the overlay base node name is given. It will be useful for a block stream job when the COR-filter is applied. The overlay base node is passed as the

Re: [PATCH] hw/block/nvme: add block utilization tracking

2020-10-14 Thread Keith Busch
On Wed, Oct 14, 2020 at 10:47:21AM +0200, Klaus Jensen wrote: > On Oct 13 14:06, Keith Busch wrote: > > > If we were going to support it here, wouldn't it make more sense to > > tie it to the filesystem's ability to support fallocate hole punch for > > the backing namespace, and check if the

Re: [PATCH v11 13/13] block: apply COR-filter to block-stream jobs

2020-10-14 Thread Max Reitz
On 12.10.20 19:43, Andrey Shinkevich wrote: > This patch completes the series with the COR-filter insertion for > block-stream operations. Adding the filter makes it possible for copied > regions to be discarded in backing files during the block-stream job, > what will reduce the disk overuse. >

Re: [PATCH v11 09/13] copy-on-read: skip non-guest reads if no copy needed

2020-10-14 Thread Vladimir Sementsov-Ogievskiy
14.10.2020 19:30, Max Reitz wrote: On 14.10.20 17:22, Vladimir Sementsov-Ogievskiy wrote: 14.10.2020 15:51, Max Reitz wrote: On 12.10.20 19:43, Andrey Shinkevich wrote: If the flag BDRV_REQ_PREFETCH was set, pass it further to the COR-driver to skip unneeded reading. It can be taken into

Re: [PATCH v11 04/13] copy-on-read: pass overlay base node name to COR driver

2020-10-14 Thread Max Reitz
On 14.10.20 18:08, Andrey Shinkevich wrote: > On 14.10.2020 14:09, Max Reitz wrote: >> On 12.10.20 19:43, Andrey Shinkevich wrote: >>> We are going to use the COR-filter for a block-stream job. >>> To limit COR operations by the base node in the backing chain during >>> stream job, pass the name

Re: [PATCH v11 09/13] copy-on-read: skip non-guest reads if no copy needed

2020-10-14 Thread Max Reitz
On 14.10.20 17:22, Vladimir Sementsov-Ogievskiy wrote: > 14.10.2020 15:51, Max Reitz wrote: >> On 12.10.20 19:43, Andrey Shinkevich wrote: >>> If the flag BDRV_REQ_PREFETCH was set, pass it further to the >>> COR-driver to skip unneeded reading. It can be taken into account for >>> the

Re: [PATCH v11 04/13] copy-on-read: pass overlay base node name to COR driver

2020-10-14 Thread Andrey Shinkevich
On 14.10.2020 14:09, Max Reitz wrote: On 12.10.20 19:43, Andrey Shinkevich wrote: We are going to use the COR-filter for a block-stream job. To limit COR operations by the base node in the backing chain during stream job, pass the name of overlay base node to the copy-on-read driver as base

Re: [PATCH v11 02/13] copy-on-read: add filter append/drop functions

2020-10-14 Thread Max Reitz
On 14.10.20 16:28, Andrey Shinkevich wrote: > On 14.10.2020 13:44, Max Reitz wrote: >> On 12.10.20 19:43, Andrey Shinkevich wrote: >>> Provide API for the COR-filter insertion/removal. >>> Also, drop the filter child permissions for an inactive state when the >>> filter node is being removed. >>>

Re: [PATCH v11 04/13] copy-on-read: pass overlay base node name to COR driver

2020-10-14 Thread Max Reitz
On 14.10.20 16:56, Vladimir Sementsov-Ogievskiy wrote: > 14.10.2020 14:57, Max Reitz wrote: >> On 14.10.20 13:09, Max Reitz wrote: >>> On 12.10.20 19:43, Andrey Shinkevich wrote: We are going to use the COR-filter for a block-stream job. To limit COR operations by the base node in the

Re: [PATCH v11 04/13] copy-on-read: pass overlay base node name to COR driver

2020-10-14 Thread Vladimir Sementsov-Ogievskiy
14.10.2020 19:08, Andrey Shinkevich wrote: On 14.10.2020 14:09, Max Reitz wrote: On 12.10.20 19:43, Andrey Shinkevich wrote: We are going to use the COR-filter for a block-stream job. To limit COR operations by the base node in the backing chain during stream job, pass the name of overlay base

Re: [PATCH v2 03/15] python: add VERSION file

2020-10-14 Thread John Snow
On 10/14/20 10:29 AM, John Snow wrote: Python infrastructure as it exists today is not capable reliably of single-sourcing a package version from a parent directory. The authors of pip are working to correct this, but as of today this is not possible to my knowledge. The problem is that when

[PATCH 15/15] block/nvme: Set request_alignment at initialization

2020-10-14 Thread Philippe Mathieu-Daudé
When introducing this driver in commit bdd6a90a9e5 ("block: Add VFIO based NVMe driver") we correctly set the request_alignment in nvme_refresh_limits() but forgot to set it at initialization. Do it now. Reported-by: Stefan Hajnoczi Signed-off-by: Philippe Mathieu-Daudé --- block/nvme.c | 1 +

[PATCH 14/15] block/nvme: Report warning with warn_report()

2020-10-14 Thread Philippe Mathieu-Daudé
Instead of displaying warning on stderr, use warn_report() which also displays it on the monitor. Signed-off-by: Philippe Mathieu-Daudé --- block/nvme.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/block/nvme.c b/block/nvme.c index ea5180d8a27..4d2b7fdf4f4 100644 ---

[PATCH 11/15] block/nvme: Trace controller capabilities

2020-10-14 Thread Philippe Mathieu-Daudé
Controllers have different capabilities and report them in the CAP register. We are particularly interested by the page size limits. Signed-off-by: Philippe Mathieu-Daudé --- block/nvme.c | 10 ++ block/trace-events | 1 + 2 files changed, 11 insertions(+) diff --git

[PATCH 13/15] block/nvme: Simplify ADMIN queue access

2020-10-14 Thread Philippe Mathieu-Daudé
We don't need to dereference from BDRVNVMeState each time. Use a NVMeQueuePair pointer to the admin queue and use it. The nvme_init() becomes easier to review, matching the style of nvme_add_io_queue(). Signed-off-by: Philippe Mathieu-Daudé --- block/nvme.c | 10 ++ 1 file changed, 6

[PATCH 12/15] block/nvme: Simplify device reset

2020-10-14 Thread Philippe Mathieu-Daudé
Avoid multiple endianess conversion by using device endianess. Signed-off-by: Philippe Mathieu-Daudé --- block/nvme.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/block/nvme.c b/block/nvme.c index fad727416ee..299fc82f40f 100644 --- a/block/nvme.c +++ b/block/nvme.c @@

[PATCH 08/15] block/nvme: Pass AioContext argument to nvme_add_io_queue()

2020-10-14 Thread Philippe Mathieu-Daudé
We want to get ride of the BlockDriverState pointer at some point, so pass aio_context along. Signed-off-by: Philippe Mathieu-Daudé --- block/nvme.c | 9 + 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/block/nvme.c b/block/nvme.c index 523814a1243..b841c5950c5 100644 ---

[PATCH 02/15] block/nvme: Trace nvme_poll_queue() per queue

2020-10-14 Thread Philippe Mathieu-Daudé
As we want to enable multiple queues, report the event in each nvme_poll_queue() call, rather than once in the callback calling nvme_poll_queues(). Signed-off-by: Philippe Mathieu-Daudé --- block/nvme.c | 2 +- block/trace-events | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)

[PATCH 06/15] block/nvme: Make nvme_identify() return boolean indicating error

2020-10-14 Thread Philippe Mathieu-Daudé
Just for consistency, following the example documented since commit e3fe3988d7 ("error: Document Error API usage rules"), return a boolean value indicating an error is set or not. Signed-off-by: Philippe Mathieu-Daudé --- block/nvme.c | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-)

[PATCH 09/15] block/nvme: Introduce Completion Queue definitions

2020-10-14 Thread Philippe Mathieu-Daudé
Rename Submission Queue flags with 'Sq' and introduce Completion Queue flag definitions. Signed-off-by: Philippe Mathieu-Daudé --- include/block/nvme.h | 17 +++-- 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/include/block/nvme.h b/include/block/nvme.h index

[PATCH 10/15] block/nvme: Use definitions instead of magic values in add_io_queue()

2020-10-14 Thread Philippe Mathieu-Daudé
Replace magic values by definitions, and simplifiy since the number of queues will never reach 64K. Signed-off-by: Philippe Mathieu-Daudé --- block/nvme.c | 9 + 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/block/nvme.c b/block/nvme.c index b841c5950c5..11fba2d754d

[PATCH 07/15] block/nvme: Make nvme_init_queue() return boolean indicating error

2020-10-14 Thread Philippe Mathieu-Daudé
Just for consistency, following the example documented since commit e3fe3988d7 ("error: Document Error API usage rules"), return a boolean value indicating an error is set or not. This simplifies a bit nvme_create_queue_pair(). Signed-off-by: Philippe Mathieu-Daudé --- block/nvme.c | 15

[PATCH 04/15] block/nvme: Improve nvme_free_req_queue_wait() trace information

2020-10-14 Thread Philippe Mathieu-Daudé
What we want to trace is the block driver state and the queue index. Suggested-by: Stefan Hajnoczi Signed-off-by: Philippe Mathieu-Daudé --- block/nvme.c | 2 +- block/trace-events | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/block/nvme.c b/block/nvme.c index

Re: [PATCH v3] migration/block-dirty-bitmap: fix uninitialized variable warning

2020-10-14 Thread Vladimir Sementsov-Ogievskiy
14.10.2020 14:44, Chen Qun wrote: A default value is provided for the variable 'bitmap_name' to avoid compiler warning. The compiler show warning: migration/block-dirty-bitmap.c:1090:13: warning: ‘bitmap_name’ may be used uninitialized in this function [-Wmaybe-uninitialized]

[PATCH 05/15] block/nvme: Trace queue pair creation/deletion

2020-10-14 Thread Philippe Mathieu-Daudé
Signed-off-by: Philippe Mathieu-Daudé --- block/nvme.c | 3 +++ block/trace-events | 2 ++ 2 files changed, 5 insertions(+) diff --git a/block/nvme.c b/block/nvme.c index e9410f2e0eb..95f19e12cd6 100644 --- a/block/nvme.c +++ b/block/nvme.c @@ -175,6 +175,7 @@ static void

[PATCH 03/15] block/nvme: Use unsigned integer for queue counter/size

2020-10-14 Thread Philippe Mathieu-Daudé
We can not have negative queue count/size/index, use unsigned type. Signed-off-by: Philippe Mathieu-Daudé --- block/nvme.c | 24 +++- block/trace-events | 10 +- 2 files changed, 16 insertions(+), 18 deletions(-) diff --git a/block/nvme.c b/block/nvme.c index

[PATCH 00/15] block/nvme: Improve debugging experience and minor fixes

2020-10-14 Thread Philippe Mathieu-Daudé
Another set of boring patches, in preparation of supporting multiple queues (the next series). Based-on: <20201014115253.25276-1-phi...@redhat.com> "util/vfio-helpers: Improve debugging experience" https://www.mail-archive.com/qemu-block@nongnu.org/msg75637.html Philippe Mathieu-Daudé (15):

[PATCH 01/15] block/nvme: Move nvme_poll_cb() earlier

2020-10-14 Thread Philippe Mathieu-Daudé
We are going to use this callback in nvme_add_io_queue() in few commits. To avoid forward-declaring it, move it before. No logical change. Reviewed-by: Stefan Hajnoczi Signed-off-by: Philippe Mathieu-Daudé --- block/nvme.c | 20 ++-- 1 file changed, 10 insertions(+), 10

Re: [PATCH v11 11/13] stream: mark backing-file argument as deprecated

2020-10-14 Thread Vladimir Sementsov-Ogievskiy
14.10.2020 18:03, Max Reitz wrote: On 12.10.20 19:43, Andrey Shinkevich wrote: Whereas the block-stream job starts using a backing file name of the base node overlay after the block-stream job completes, mark the QMP 'backing-file' argument as deprecated. Signed-off-by: Andrey Shinkevich ---

Re: [PATCH v11 10/13] stream: skip filters when writing backing file name to QCOW2 header

2020-10-14 Thread Vladimir Sementsov-Ogievskiy
14.10.2020 18:02, Max Reitz wrote: On 12.10.20 19:43, Andrey Shinkevich wrote: Avoid writing a filter JSON-name to QCOW2 image when the backing file is changed after the block stream job. Signed-off-by: Andrey Shinkevich --- block/stream.c | 9 + 1 file changed, 5 insertions(+), 4

Re: [PATCH v11 09/13] copy-on-read: skip non-guest reads if no copy needed

2020-10-14 Thread Vladimir Sementsov-Ogievskiy
14.10.2020 15:51, Max Reitz wrote: On 12.10.20 19:43, Andrey Shinkevich wrote: If the flag BDRV_REQ_PREFETCH was set, pass it further to the COR-driver to skip unneeded reading. It can be taken into account for the COR-algorithms optimization. That check is being made during the block stream

Re: [PATCH v11 12/13] stream: remove unused backing-file name parameter

2020-10-14 Thread Max Reitz
On 12.10.20 19:43, Andrey Shinkevich wrote: > The 'backing-file' argument is not used by the block-stream job. It > designates a backing file name to set in QCOW2 image header after the > block-stream job finished. A backing file name of the node above base > is used instead. > > Signed-off-by:

Re: [PATCH v11 06/13] block: modify the comment for BDRV_REQ_PREFETCH flag

2020-10-14 Thread Vladimir Sementsov-Ogievskiy
14.10.2020 15:22, Max Reitz wrote: On 12.10.20 19:43, Andrey Shinkevich wrote: Modify the comment for the flag BDRV_REQ_PREFETCH as we are going to use it alone and pass it to the COR-filter driver for further processing. Signed-off-by: Andrey Shinkevich --- include/block/block.h | 7

Re: [PATCH v11 11/13] stream: mark backing-file argument as deprecated

2020-10-14 Thread Max Reitz
On 12.10.20 19:43, Andrey Shinkevich wrote: > Whereas the block-stream job starts using a backing file name of the > base node overlay after the block-stream job completes, mark the QMP > 'backing-file' argument as deprecated. > > Signed-off-by: Andrey Shinkevich > --- >

Re: [PATCH v11 10/13] stream: skip filters when writing backing file name to QCOW2 header

2020-10-14 Thread Max Reitz
On 12.10.20 19:43, Andrey Shinkevich wrote: > Avoid writing a filter JSON-name to QCOW2 image when the backing file > is changed after the block stream job. > > Signed-off-by: Andrey Shinkevich > --- > block/stream.c | 9 + > 1 file changed, 5 insertions(+), 4 deletions(-) > > diff

Re: [PATCH v11 04/13] copy-on-read: pass overlay base node name to COR driver

2020-10-14 Thread Vladimir Sementsov-Ogievskiy
14.10.2020 14:57, Max Reitz wrote: On 14.10.20 13:09, Max Reitz wrote: On 12.10.20 19:43, Andrey Shinkevich wrote: We are going to use the COR-filter for a block-stream job. To limit COR operations by the base node in the backing chain during stream job, pass the name of overlay base node to

Re: [PATCH v4 7/7] nbd: Allow export of multiple bitmaps for one device

2020-10-14 Thread Vladimir Sementsov-Ogievskiy
10.10.2020 00:55, Eric Blake wrote: With this, 'qemu-nbd -B b0 -B b1 -f qcow2 img.qcow2' can let you sniff out multiple bitmaps from one server. qemu-img as client can still only read one bitmap per client connection, but other NBD clients (hello libnbd) can now read multiple bitmaps in a

[PATCH v2 15/15] python/qemu: add qemu package itself to pipenv

2020-10-14 Thread John Snow
This adds the python qemu packages themselves to the pipenv manifest. 'pipenv sync' will create a virtual environment sufficient to use the SDK. 'pipenv sync --dev' will create a virtual environment sufficient to use and test the SDK (with pylint, mypy, isort, flake8, etc.) The qemu packages are

[PATCH v2 11/15] python: move mypy.ini into setup.cfg

2020-10-14 Thread John Snow
mypy supports reading its configuration values from a central project configuration file; do so. Signed-off-by: John Snow --- python/mypy.ini | 4 python/setup.cfg | 5 + 2 files changed, 5 insertions(+), 4 deletions(-) delete mode 100644 python/mypy.ini diff --git a/python/mypy.ini

[PATCH v2 08/15] python: add pylint to pipenv

2020-10-14 Thread John Snow
We are specifying >= pylint 2.6.x for two reasons: 1. For setup.cfg support, added in pylint 2.5.x 2. To clarify that we are using a version that has incompatibly dropped bad-whitespace checks. Signed-off-by: John Snow --- python/Pipfile | 1 + python/Pipfile.lock | 127

[PATCH v2 13/15] python: move .isort.cfg into setup.cfg

2020-10-14 Thread John Snow
Signed-off-by: John Snow --- python/.isort.cfg | 7 --- python/setup.cfg | 8 2 files changed, 8 insertions(+), 7 deletions(-) delete mode 100644 python/.isort.cfg diff --git a/python/.isort.cfg b/python/.isort.cfg deleted file mode 100644 index 6d0fd6cc0d..00 ---

[PATCH v2 12/15] python: add mypy to pipenv

2020-10-14 Thread John Snow
0.730 appears to be about the oldest version that works with the features we want, including nice human readable output (to make sure iotest 297 passes), and type-parameterized Popen generics. 0.770, however, supports adding 'strict' to the config file, so require at least 0.770. mypy can be run

[PATCH v2 06/15] python: add pylint exceptions to __init__.py

2020-10-14 Thread John Snow
Pylint 2.5.x and 2.6.x have regressions that make import checking inconsistent, see: https: //github.com/PyCQA/pylint/issues/3609 https: //github.com/PyCQA/pylint/issues/3624 https: //github.com/PyCQA/pylint/issues/3651 Signed-off-by: John Snow --- python/qemu/core/__init__.py | 3 +++ 1 file

[PATCH v2 04/15] python: add directory structure README.rst files

2020-10-14 Thread John Snow
Add short readmes to python/, python/qemu/, and python/qemu/core that explain the directory hierarchy. These readmes are visible when browsing the source on e.g. gitlab/github and are designed to help new developers/users quickly make sense of the source tree. They are not designed for inclusion

[PATCH v2 01/15] python: create qemu.core package

2020-10-14 Thread John Snow
move python/qemu/*.py to python/qemu/core/*.py and update import directives across the tree. This is done to create a PEP420 namespace package, in which we may create subpackages. To do this, the namespace directory ("qemu") should not have any modules in it. Those files will go in a new 'core'

[PATCH v2 02/15] python: add qemu package installer

2020-10-14 Thread John Snow
Add setup.cfg and setup.py, necessary for installing a package via pip. Add a rst document explaining the basics of what this package is for and who to contact for more information. This document will be used as the landing page for the package on PyPI. I am not yet using a pyproject.toml style

[PATCH v2 07/15] python: move pylintrc into setup.cfg

2020-10-14 Thread John Snow
Delete the empty settings now that it's sharing a home with settings for other tools. pylint can now be run from this folder as "pylint qemu". Signed-off-by: John Snow --- python/qemu/core/pylintrc | 58 --- python/setup.cfg | 29

[PATCH v2 14/15] python/qemu: add isort to pipenv

2020-10-14 Thread John Snow
isort 5.0.0 through 5.0.4 has a bug that causes it to misinterpret certain "from ..." clauses that are not related to imports. Require 5.0.5 or greater. isort can be run with 'isort -c qemu' from the python root. Signed-off-by: John Snow --- python/Pipfile | 1 + python/Pipfile.lock | 4

[PATCH v2 05/15] python: Add pipenv support

2020-10-14 Thread John Snow
pipenv is a tool used for managing virtual environments with pinned, explicit dependencies. It is used for precisely recreating python virtual environments. pipenv uses two files to do this: (1) Pipfile, which is similar in purpose and scope to what setup.py lists. It specifies the requisite

[PATCH v2 00/15] python: create installable package

2020-10-14 Thread John Snow
Based-on: https://gitlab.com/jsnow/qemu/-/tree/python This series factors the python/qemu directory as an installable module. It does not yet actually change the mechanics of how any other python source in the tree actually consumes it (yet), beyond the import path. The point of this series is

[PATCH v2 03/15] python: add VERSION file

2020-10-14 Thread John Snow
Python infrastructure as it exists today is not capable reliably of single-sourcing a package version from a parent directory. The authors of pip are working to correct this, but as of today this is not possible to my knowledge. The problem is that when using pip to build and install a python

[PATCH v2 09/15] python: move flake8 config to setup.cfg

2020-10-14 Thread John Snow
Signed-off-by: John Snow --- python/qemu/core/.flake8 | 2 -- python/setup.cfg | 3 +++ 2 files changed, 3 insertions(+), 2 deletions(-) delete mode 100644 python/qemu/core/.flake8 diff --git a/python/qemu/core/.flake8 b/python/qemu/core/.flake8 deleted file mode 100644 index

[PATCH v2 10/15] python: Add flake8 to pipenv

2020-10-14 Thread John Snow
flake8 3.5.x does not support the --extend-ignore syntax used in the .flake8 file to gracefully extend default ignores, so 3.6.x is our minimum requirement. There is no known upper bound. flake8 can be run from the python/ directory with no arguments. Signed-off-by: John Snow ---

Re: [PATCH v11 02/13] copy-on-read: add filter append/drop functions

2020-10-14 Thread Andrey Shinkevich
On 14.10.2020 13:44, Max Reitz wrote: On 12.10.20 19:43, Andrey Shinkevich wrote: Provide API for the COR-filter insertion/removal. Also, drop the filter child permissions for an inactive state when the filter node is being removed. Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir

Re: [PATCH v6 03/11] hw/block/nvme: Add support for Namespace Types

2020-10-14 Thread Niklas Cassel
On Wed, Oct 14, 2020 at 06:42:04AM +0900, Dmitry Fomichev wrote: > From: Niklas Cassel > > Define the structures and constants required to implement > Namespace Types support. > > Namespace Types introduce a new command set, "I/O Command Sets", > that allows the host to retrieve the command

Re: [PATCH v11 09/13] copy-on-read: skip non-guest reads if no copy needed

2020-10-14 Thread Max Reitz
On 12.10.20 19:43, Andrey Shinkevich wrote: > If the flag BDRV_REQ_PREFETCH was set, pass it further to the > COR-driver to skip unneeded reading. It can be taken into account for > the COR-algorithms optimization. That check is being made during the > block stream job by the moment. > >

Re: [PATCH 0/9] util/vfio-helpers: Improve debugging experience

2020-10-14 Thread Fam Zheng
On Wed, 2020-10-14 at 14:42 +0200, Philippe Mathieu-Daudé wrote: > On 10/14/20 2:34 PM, Fam Zheng wrote: > > On Wed, 2020-10-14 at 13:52 +0200, Philippe Mathieu-Daudé wrote: > > > A bunch of boring patches that have been proven helpful > > > while debugging. > > > > > > Philippe Mathieu-Daudé

Re: [PATCH 0/9] util/vfio-helpers: Improve debugging experience

2020-10-14 Thread Philippe Mathieu-Daudé
On 10/14/20 2:34 PM, Fam Zheng wrote: On Wed, 2020-10-14 at 13:52 +0200, Philippe Mathieu-Daudé wrote: A bunch of boring patches that have been proven helpful while debugging. Philippe Mathieu-Daudé (9): util/vfio-helpers: Improve reporting unsupported IOMMU type util/vfio-helpers: Trace

Re: [PATCH 3/9] util/vfio-helpers: Trace PCI BAR region info

2020-10-14 Thread Philippe Mathieu-Daudé
On 10/14/20 2:23 PM, Fam Zheng wrote: On Wed, 2020-10-14 at 13:52 +0200, Philippe Mathieu-Daudé wrote: For debug purpose, trace BAR regions info. Signed-off-by: Philippe Mathieu-Daudé --- util/vfio-helpers.c | 8 util/trace-events | 1 + 2 files changed, 9 insertions(+) diff

Re: [PATCH v11 08/13] copy-on-read: add support for BDRV_REQ_PREFETCH to COR-filter

2020-10-14 Thread Max Reitz
On 12.10.20 19:43, Andrey Shinkevich wrote: > Add support for the BDRV_REQ_PREFETCH flag to the supported_write_flags s/write/read/ > of the COR-filter. > > Signed-off-by: Andrey Shinkevich > --- > block/copy-on-read.c | 1 + > 1 file changed, 1 insertion(+) > > diff --git

Re: [PATCH 0/9] util/vfio-helpers: Improve debugging experience

2020-10-14 Thread Fam Zheng
On Wed, 2020-10-14 at 13:52 +0200, Philippe Mathieu-Daudé wrote: > A bunch of boring patches that have been proven helpful > while debugging. > > Philippe Mathieu-Daudé (9): > util/vfio-helpers: Improve reporting unsupported IOMMU type > util/vfio-helpers: Trace PCI I/O config accesses >

Re: [PATCH v11 06/13] block: modify the comment for BDRV_REQ_PREFETCH flag

2020-10-14 Thread Max Reitz
On 12.10.20 19:43, Andrey Shinkevich wrote: > Modify the comment for the flag BDRV_REQ_PREFETCH as we are going to > use it alone and pass it to the COR-filter driver for further > processing. > > Signed-off-by: Andrey Shinkevich > --- > include/block/block.h | 7 --- > 1 file changed, 4

Re: [PATCH v11 07/13] block: include supported_read_flags into BDS structure

2020-10-14 Thread Max Reitz
On 12.10.20 19:43, Andrey Shinkevich wrote: > Add the new member supported_read_flags to BlockDriverState structure. > It will control the BDRV_REQ_PREFETCH flag set for copy-on-read > operations. > > Signed-off-by: Andrey Shinkevich > --- > include/block/block_int.h | 4 > 1 file changed,

Re: [PATCH 3/9] util/vfio-helpers: Trace PCI BAR region info

2020-10-14 Thread Fam Zheng
On Wed, 2020-10-14 at 13:52 +0200, Philippe Mathieu-Daudé wrote: > For debug purpose, trace BAR regions info. > > Signed-off-by: Philippe Mathieu-Daudé > --- > util/vfio-helpers.c | 8 > util/trace-events | 1 + > 2 files changed, 9 insertions(+) > > diff --git a/util/vfio-helpers.c

Re: [PATCH v4 6/7] nbd: Refactor counting of metadata contexts

2020-10-14 Thread Vladimir Sementsov-Ogievskiy
10.10.2020 00:55, Eric Blake wrote: Rather than open-code the count of negotiated contexts at several sites, embed it directly into the struct. Signed-off-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy -- Best regards, Vladimir

Re: [PATCH v4 5/7] nbd: Simplify qemu bitmap context name

2020-10-14 Thread Vladimir Sementsov-Ogievskiy
10.10.2020 00:55, Eric Blake wrote: Each dirty bitmap already knows its name; by reducing the scope of the places where we construct "qemu:dirty-bitmap:NAME" strings, tracking the name is more localized, and there are fewer per-export fields to worry about. This in turn will make it easier for

Re: [PATCH v4 4/7] nbd: Update qapi to support exporting multiple bitmaps

2020-10-14 Thread Vladimir Sementsov-Ogievskiy
10.10.2020 00:55, Eric Blake wrote: Since 'nbd-server-add' is deprecated, and 'block-export-add' is new to 5.2, we can still tweak the interface. Allowing 'bitmaps':['str'] is nicer than 'bitmap':'str'. This wires up the qapi and qemu-nbd changes to permit passing multiple bitmaps as distinct

Re: [PATCH v6 01/11] hw/block/nvme: Add Commands Supported and Effects log

2020-10-14 Thread Niklas Cassel
On Tue, Oct 13, 2020 at 05:50:34PM -0700, Keith Busch wrote: > On Wed, Oct 14, 2020 at 06:42:02AM +0900, Dmitry Fomichev wrote: > > +{ > > +NvmeEffectsLog log = {}; > > +uint32_t *dst_acs = log.acs, *dst_iocs = log.iocs; > > +uint32_t trans_len; > > +int i; > > + > > +

Re: [PATCH v11 05/13] copy-on-read: limit COR operations to base in COR driver

2020-10-14 Thread Max Reitz
On 12.10.20 19:43, Andrey Shinkevich wrote: > Limit COR operations by the base node in the backing chain when the > overlay base node name is given. It will be useful for a block stream > job when the COR-filter is applied. The overlay base node is passed as > the base itself may change due to

Re: [PATCH v11 05/13] copy-on-read: limit COR operations to base in COR driver

2020-10-14 Thread Max Reitz
On 12.10.20 19:43, Andrey Shinkevich wrote: > Limit COR operations by the base node in the backing chain when the > overlay base node name is given. It will be useful for a block stream > job when the COR-filter is applied. The overlay base node is passed as > the base itself may change due to

Re: [PATCH v6 05/11] hw/block/nvme: Support Zoned Namespace Command Set

2020-10-14 Thread Niklas Cassel
On Wed, Oct 14, 2020 at 06:42:06AM +0900, Dmitry Fomichev wrote: > The emulation code has been changed to advertise NVM Command Set when > "zoned" device property is not set (default) and Zoned Namespace > Command Set otherwise. > > Define values and structures that are needed to support Zoned >

[PATCH 5/9] util/vfio-helpers: Improve DMA trace events

2020-10-14 Thread Philippe Mathieu-Daudé
For debugging purpose, trace where DMA regions are mapped. Signed-off-by: Philippe Mathieu-Daudé --- util/vfio-helpers.c | 3 ++- util/trace-events | 5 +++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/util/vfio-helpers.c b/util/vfio-helpers.c index

[PATCH 9/9] util/vfio-helpers: Let qemu_vfio_verify_mappings() use error_report()

2020-10-14 Thread Philippe Mathieu-Daudé
Instead of displaying the error on stderr, use error_report() which also report to the monitor. Signed-off-by: Philippe Mathieu-Daudé --- util/vfio-helpers.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/util/vfio-helpers.c b/util/vfio-helpers.c index

Re: [PATCH v11 04/13] copy-on-read: pass overlay base node name to COR driver

2020-10-14 Thread Max Reitz
On 14.10.20 13:09, Max Reitz wrote: > On 12.10.20 19:43, Andrey Shinkevich wrote: >> We are going to use the COR-filter for a block-stream job. >> To limit COR operations by the base node in the backing chain during >> stream job, pass the name of overlay base node to the copy-on-read >> driver as

[PATCH 7/9] util/vfio-helpers: Let qemu_vfio_dma_map() propagate Error

2020-10-14 Thread Philippe Mathieu-Daudé
Currently qemu_vfio_dma_map() displays errors on stderr. When using management interface, this information is simply lost. Pass qemu_vfio_dma_map() an Error* argument so it can propagate the error to callers. Signed-off-by: Philippe Mathieu-Daudé --- include/qemu/vfio-helpers.h | 2 +-

[PATCH 0/9] util/vfio-helpers: Improve debugging experience

2020-10-14 Thread Philippe Mathieu-Daudé
A bunch of boring patches that have been proven helpful while debugging. Philippe Mathieu-Daudé (9): util/vfio-helpers: Improve reporting unsupported IOMMU type util/vfio-helpers: Trace PCI I/O config accesses util/vfio-helpers: Trace PCI BAR region info util/vfio-helpers: Trace where

Re: [PATCH v4 2/7] nbd: Add new qemu:allocation-depth metadata context

2020-10-14 Thread Vladimir Sementsov-Ogievskiy
10.10.2020 00:55, Eric Blake wrote: 'qemu-img map' provides a way to determine which extents of an image come from the top layer vs. inherited from a backing chain. This is useful information worth exposing over NBD. There is a proposal to add a QMP command block-dirty-bitmap-populate which

[PATCH 8/9] util/vfio-helpers: Let qemu_vfio_do_mapping() propagate Error

2020-10-14 Thread Philippe Mathieu-Daudé
Pass qemu_vfio_do_mapping() an Error* argument so it can propagate any error to callers. Replace error_report() which only report to the monitor by the more generic error_setg_errno(). Signed-off-by: Philippe Mathieu-Daudé --- util/vfio-helpers.c | 8 1 file changed, 4 insertions(+), 4

[PATCH 6/9] util/vfio-helpers: Convert vfio_dump_mapping to trace events

2020-10-14 Thread Philippe Mathieu-Daudé
The QEMU_VFIO_DEBUG definition is only modifiable at build-time. Trace events can be enabled at run-time. As we prefer the latter, convert qemu_vfio_dump_mappings() to use trace events instead of fprintf(). Signed-off-by: Philippe Mathieu-Daudé --- util/vfio-helpers.c | 19 ---

[PATCH 3/9] util/vfio-helpers: Trace PCI BAR region info

2020-10-14 Thread Philippe Mathieu-Daudé
For debug purpose, trace BAR regions info. Signed-off-by: Philippe Mathieu-Daudé --- util/vfio-helpers.c | 8 util/trace-events | 1 + 2 files changed, 9 insertions(+) diff --git a/util/vfio-helpers.c b/util/vfio-helpers.c index 1d4efafcaa4..cd6287c3a98 100644 ---

[PATCH 4/9] util/vfio-helpers: Trace where BARs are mapped

2020-10-14 Thread Philippe Mathieu-Daudé
For debugging purpose, trace where a BAR is mapped. Signed-off-by: Philippe Mathieu-Daudé --- util/vfio-helpers.c | 2 ++ util/trace-events | 1 + 2 files changed, 3 insertions(+) diff --git a/util/vfio-helpers.c b/util/vfio-helpers.c index cd6287c3a98..278c54902e7 100644 ---

[PATCH 2/9] util/vfio-helpers: Trace PCI I/O config accesses

2020-10-14 Thread Philippe Mathieu-Daudé
We sometime get kernel panic with some devices on Aarch64 hosts. Alex Williamson suggests it might be broken PCIe root complex. Add trace event to record the latest I/O access before crashing. In case, assert our accesses are aligned. Signed-off-by: Philippe Mathieu-Daudé --- Cc: Alex Williamson

[PATCH 1/9] util/vfio-helpers: Improve reporting unsupported IOMMU type

2020-10-14 Thread Philippe Mathieu-Daudé
Change the confuse "VFIO IOMMU check failed" error message by the explicit "VFIO IOMMU Type1 is not supported" once. Example on POWER: $ qemu-system-ppc64 -drive if=none,id=nvme0,file=nvme://0001:01:00.0/1,format=raw qemu-system-ppc64: -drive

[PATCH v3] migration/block-dirty-bitmap: fix uninitialized variable warning

2020-10-14 Thread Chen Qun
A default value is provided for the variable 'bitmap_name' to avoid compiler warning. The compiler show warning: migration/block-dirty-bitmap.c:1090:13: warning: ‘bitmap_name’ may be used uninitialized in this function [-Wmaybe-uninitialized] g_strlcpy(s->bitmap_name, bitmap_name,

[PATCH v3 9/9] hw/block/nvme: allow open to close zone transitions by controller

2020-10-14 Thread Klaus Jensen
From: Klaus Jensen Allow the controller to release open resources by transitioning implicitly and explicitly opened zones to closed. This is done using a naive "least recently opened" strategy. Signed-off-by: Klaus Jensen --- hw/block/nvme-ns.h| 5 hw/block/nvme-ns.c| 5

[PATCH v3 8/9] hw/block/nvme: track and enforce zone resources

2020-10-14 Thread Klaus Jensen
From: Klaus Jensen Track number of open/active resources. Signed-off-by: Klaus Jensen --- docs/specs/nvme.txt | 6 hw/block/nvme-ns.h | 7 + include/block/nvme.h | 2 ++ hw/block/nvme-ns.c | 25 +++-- hw/block/nvme.c | 66

[PATCH v3 5/9] hw/block/nvme: add the zone management receive command

2020-10-14 Thread Klaus Jensen
From: Klaus Jensen Add the Zone Management Receive command. Signed-off-by: Klaus Jensen --- hw/block/nvme-ns.h| 8 +++ hw/block/nvme.h | 1 + include/block/nvme.h | 46 + hw/block/nvme-ns.c| 12 +++- hw/block/nvme.c | 146

[PATCH v3 6/9] hw/block/nvme: add the zone management send command

2020-10-14 Thread Klaus Jensen
From: Klaus Jensen Add the Zone Management Send command. Signed-off-by: Klaus Jensen --- hw/block/nvme.h | 2 + include/block/nvme.h | 28 +++ hw/block/nvme.c | 384 +- hw/block/trace-events | 11 ++ 4 files changed, 420 insertions(+),

Re: [PATCH v6 02/11] hw/block/nvme: Generate namespace UUIDs

2020-10-14 Thread Klaus Jensen
On Oct 14 06:42, Dmitry Fomichev wrote: > In NVMe 1.4, a namespace must report an ID descriptor of UUID type > if it doesn't support EUI64 or NGUID. Add a new namespace property, > "uuid", that provides the user the option to either specify the UUID > explicitly or have a UUID generated

[PATCH v3 0/9] hw/block/nvme: zoned namespace command set

2020-10-14 Thread Klaus Jensen
From: Klaus Jensen Updated version of my proposal. Based-on: <20201014084324.333774-1-...@irrelevant.dk> Changes for v3 ~~ * Rebased on nvme-next with "[PATCH v2] hw/block/nvme: add dulbe support" applied. * "hw/block/nvme: add support for dulbe and block utilization

[PATCH v3 7/9] hw/block/nvme: add the zone append command

2020-10-14 Thread Klaus Jensen
From: Klaus Jensen Add the Zone Append command. Signed-off-by: Klaus Jensen --- hw/block/nvme.h | 6 ++ include/block/nvme.h | 7 +++ hw/block/nvme.c | 46 +++ hw/block/trace-events | 1 + 4 files changed, 60 insertions(+) diff

[PATCH v3 4/9] hw/block/nvme: add basic read/write for zoned namespaces

2020-10-14 Thread Klaus Jensen
From: Klaus Jensen This adds basic read and write for zoned namespaces. A zoned namespace is created by setting the iocs namespace parameter to 0x2 and specifying the zns.zcap parameter (zone capacity) in number of logical blocks per zone. If a zone size (zns.zsze) is not specified, the

[PATCH v3 3/9] hw/block/nvme: support namespace types

2020-10-14 Thread Klaus Jensen
From: Klaus Jensen Implement support for TP 4056 ("Namespace Types"). This adds the 'iocs' (I/O Command Set) device parameter to the nvme-ns device. Signed-off-by: Klaus Jensen --- docs/specs/nvme.txt | 3 + hw/block/nvme-ns.h| 11 ++- hw/block/nvme.h | 3 +

  1   2   >