Re: [PATCH 0/2] block: Skip COR for inactive nodes

2019-10-04 Thread Stefan Hajnoczi
On Tue, Oct 01, 2019 at 07:48:25PM +0200, Max Reitz wrote: > Hi, > > While working on $IMGOPTS support for Python iotests, I noticed a minor > bug. Let’s fix it, as you do with bugs. > > > Max Reitz (2): > block: Skip COR for inactive nodes > iotests/262: Switch source/dest VM launch order

Re: bitmap migration bug with -drive while block mirror runs

2019-10-04 Thread Vladimir Sementsov-Ogievskiy
04.10.2019 11:33, Peter Krempa wrote: > On Thu, Oct 03, 2019 at 19:34:56 -0400, John Snow wrote: >> On 10/3/19 6:14 AM, Vladimir Sementsov-Ogievskiy wrote: >>> 03.10.2019 0:35, John Snow wrote: On 10/2/19 6:46 AM, Peter Krempa wrote: >>> > > [...] > > (I'm sorry if I ignored something

[PULL v3 4/4] iotests: Remove Python 2 compatibility code

2019-10-04 Thread Kevin Wolf
Some scripts check the Python version number and have two code paths to accomodate both Python 2 and 3. Remove the code specific to Python 2 and assert the minimum version of 3.6 instead (check skips Python tests in this case, so the assertion would only ever trigger if a Python script is executed

[PULL v3 2/4] iotests: Test internal snapshots with -blockdev

2019-10-04 Thread Kevin Wolf
Signed-off-by: Kevin Wolf Reviewed-by: Peter Krempa Tested-by: Peter Krempa --- tests/qemu-iotests/267 | 168 tests/qemu-iotests/267.out | 182 +++ tests/qemu-iotests/common.filter | 11 +- tests/qemu-iotests/group

[PULL v3 3/4] iotests: Require Python 3.6 or later

2019-10-04 Thread Kevin Wolf
Running iotests is not required to build QEMU, so we can have stricter version requirements for Python here and can make use of new features and drop compatibility code earlier. This makes qemu-iotests skip all Python tests if a Python version before 3.6 is used for the build. Suggested-by:

[PULL v3 1/4] block/snapshot: Restrict set of snapshot nodes

2019-10-04 Thread Kevin Wolf
Nodes involved in internal snapshots were those that were returned by bdrv_next(), inserted and not read-only. bdrv_next() in turn returns all nodes that are either the root node of a BlockBackend or monitor-owned nodes. With the typical -drive use, this worked well enough. However, in the

[PULL v3 0/4] Block layer patches

2019-10-04 Thread Kevin Wolf
The following changes since commit 4f59102571fce49af180cfc6d4cdd2b5df7bdb14: Merge remote-tracking branch 'remotes/amarkovic/tags/mips-queue-oct-01-2019' into staging (2019-10-01 16:21:42 +0100) are available in the Git repository at: git://repo.or.cz/qemu/kevin.git tags/for-upstream for

Re: bitmap migration bug with -drive while block mirror runs

2019-10-04 Thread Peter Krempa
On Thu, Oct 03, 2019 at 19:34:56 -0400, John Snow wrote: > On 10/3/19 6:14 AM, Vladimir Sementsov-Ogievskiy wrote: > > 03.10.2019 0:35, John Snow wrote: > >> On 10/2/19 6:46 AM, Peter Krempa wrote: > > [...] (I'm sorry if I ignored something which might require input in the trimmed part but

Re: bitmap migration bug with -drive while block mirror runs

2019-10-04 Thread Vladimir Sementsov-Ogievskiy
04.10.2019 2:34, John Snow wrote: > > > On 10/3/19 6:14 AM, Vladimir Sementsov-Ogievskiy wrote: >> 03.10.2019 0:35, John Snow wrote: >>> On 10/2/19 6:46 AM, Peter Krempa wrote: >>> >>> [ * poof * ] >>> I'd like to re-iterate that the necessity to keep node names same on both sides

Re: [PATCH v15 5/5] block/backup: use backup-top instead of write notifiers

2019-10-04 Thread Max Reitz
On 01.10.19 15:14, Vladimir Sementsov-Ogievskiy wrote: > Drop write notifiers and use filter node instead. > > = Changes = > > 1. Add filter-node-name argument for backup qmp api. We have to do it > in this commit, as 257 needs to be fixed. > > 2. There are no more write notifiers here, so

Re: [Qemu-devel] [PATCH v9 07/17] blockdev: adds bdrv_parse_aio to use io_uring

2019-10-04 Thread Stefan Hajnoczi
On Wed, Aug 07, 2019 at 02:49:51PM +0200, Julia Suvorova via Qemu-devel wrote: > On Wed, Aug 7, 2019 at 2:06 PM Aarushi Mehta wrote: > > > > > > > > On Wed, 7 Aug, 2019, 17:15 Julia Suvorova, wrote: > >> > >> On Fri, Aug 2, 2019 at 1:41 AM Aarushi Mehta > >> wrote: > >> > +int

Re: [PATCH 3/4] block/mirror: support unaligned write in active mirror

2019-10-04 Thread Max Reitz
On 04.10.19 17:04, Vladimir Sementsov-Ogievskiy wrote: > 04.10.2019 17:48, Max Reitz wrote: >> On 04.10.19 15:22, Vladimir Sementsov-Ogievskiy wrote: >>> 04.10.2019 15:59, Max Reitz wrote: On 03.10.19 11:34, Vladimir Sementsov-Ogievskiy wrote: > 02.10.2019 18:52, Max Reitz wrote: >>

[PATCH 4/5] migration/block-dirty-bitmap: fix bitmaps migration during mirror job

2019-10-04 Thread Vladimir Sementsov-Ogievskiy
Important thing for bitmap migration is to select destination block node to obtain the migrated bitmap. Prepatch, on source we use bdrv_get_device_or_node_name() to identify the node, and on target we do bdrv_lookup_bs. bdrv_get_device_or_node_name() returns blk name only for direct children of

[PATCH 1/5] block: Mark commit and mirror as filter drivers

2019-10-04 Thread Vladimir Sementsov-Ogievskiy
From: Max Reitz The commit and mirror block nodes are filters, so they should be marked as such. Signed-off-by: Max Reitz Signed-off-by: Vladimir Sementsov-Ogievskiy [squash comment fix from another Max's patch and adjust commit msg] --- include/block/block_int.h | 8 +---

[PATCH 0/5] fix migration with bitmaps and mirror

2019-10-04 Thread Vladimir Sementsov-Ogievskiy
Hi all! It's a continuation for "bitmap migration bug with -drive while block mirror runs" <315cff78-dcdb-a3ce-2742-da3cc9f0c...@redhat.com> https://lists.gnu.org/archive/html/qemu-devel/2019-09/msg07241.html The problem is that bitmaps migrated to node with same node-name or blk-parent name.

Re: [PATCH v15 0/5] backup-top filter driver for backup

2019-10-04 Thread Max Reitz
On 01.10.19 15:14, Vladimir Sementsov-Ogievskiy wrote: > Hi all! > > These series introduce backup-top driver. It's a filter-node, which > do copy-before-write operation. Mirror uses filter-node for handling > guest writes, let's move to filter-node (from write-notifiers) for > backup too. > >

Re: [PATCH v15 0/5] backup-top filter driver for backup

2019-10-04 Thread Vladimir Sementsov-Ogievskiy
04.10.2019 17:21, Max Reitz wrote: > On 01.10.19 15:14, Vladimir Sementsov-Ogievskiy wrote: >> Hi all! >> >> These series introduce backup-top driver. It's a filter-node, which >> do copy-before-write operation. Mirror uses filter-node for handling >> guest writes, let's move to filter-node (from

Re: [PATCH 3/4] block/mirror: support unaligned write in active mirror

2019-10-04 Thread Vladimir Sementsov-Ogievskiy
04.10.2019 18:27, Max Reitz wrote: > On 04.10.19 17:04, Vladimir Sementsov-Ogievskiy wrote: >> 04.10.2019 17:48, Max Reitz wrote: >>> On 04.10.19 15:22, Vladimir Sementsov-Ogievskiy wrote: 04.10.2019 15:59, Max Reitz wrote: > On 03.10.19 11:34, Vladimir Sementsov-Ogievskiy wrote: >>

[PATCH 2/5] migretion/block-dirty-bitmap: refactor init_dirty_bitmap_migration

2019-10-04 Thread Vladimir Sementsov-Ogievskiy
Split out handling one bs, it is needed for the following commit, which will handle BlockBackends in separate. Signed-off-by: Vladimir Sementsov-Ogievskiy --- migration/block-dirty-bitmap.c | 93 +++--- 1 file changed, 51 insertions(+), 42 deletions(-) diff --git

[PATCH 5/5] iotests: 194: test also migration of dirty bitmap

2019-10-04 Thread Vladimir Sementsov-Ogievskiy
Test that dirty bitmap migration works when we deal with mirror. Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/194 | 14 ++ tests/qemu-iotests/194.out | 6 ++ 2 files changed, 16 insertions(+), 4 deletions(-) diff --git a/tests/qemu-iotests/194

Re: [PATCH v5 1/5] iotests: remove 'linux' from default supported platforms

2019-10-04 Thread Max Reitz
On 04.10.19 16:05, Peter Maydell wrote: > On Fri, 4 Oct 2019 at 14:50, Max Reitz wrote: >> On 04.10.19 15:16, Peter Maydell wrote: >>> 'make check' does have the restriction >>> that we don't want the tests to take too long to run, but in >>> general the block layer should be running some

Re: [PATCH 3/4] block/mirror: support unaligned write in active mirror

2019-10-04 Thread Max Reitz
On 04.10.19 15:22, Vladimir Sementsov-Ogievskiy wrote: > 04.10.2019 15:59, Max Reitz wrote: >> On 03.10.19 11:34, Vladimir Sementsov-Ogievskiy wrote: >>> 02.10.2019 18:52, Max Reitz wrote: On 02.10.19 17:06, Vladimir Sementsov-Ogievskiy wrote: > 02.10.2019 18:03, Vladimir

[PATCH 3/5] block/dirty-bitmap: add bdrv_has_named_bitmaps helper

2019-10-04 Thread Vladimir Sementsov-Ogievskiy
To be used for bitmap migration in further commit. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/block/dirty-bitmap.h | 1 + block/dirty-bitmap.c | 13 + 2 files changed, 14 insertions(+) diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h

Re: [PATCH 3/4] block/mirror: support unaligned write in active mirror

2019-10-04 Thread Vladimir Sementsov-Ogievskiy
04.10.2019 17:48, Max Reitz wrote: > On 04.10.19 15:22, Vladimir Sementsov-Ogievskiy wrote: >> 04.10.2019 15:59, Max Reitz wrote: >>> On 03.10.19 11:34, Vladimir Sementsov-Ogievskiy wrote: 02.10.2019 18:52, Max Reitz wrote: > On 02.10.19 17:06, Vladimir Sementsov-Ogievskiy wrote: >>

Re: [PATCH v5 0/5] iotests: use python logging

2019-10-04 Thread Max Reitz
On 18.09.19 01:45, John Snow wrote: > This series uses python logging to enable output conditionally on > iotests.log(). We unify an initialization call (which also enables > debugging output for those tests with -d) and then make the switch > inside of iotests. > > It will help alleviate the

Re: [PATCH v5 1/5] iotests: remove 'linux' from default supported platforms

2019-10-04 Thread Kevin Wolf
Am 02.10.2019 um 19:47 hat Max Reitz geschrieben: > On 02.10.19 18:44, Kevin Wolf wrote: > > Am 02.10.2019 um 13:57 hat Max Reitz geschrieben: > >> It usually worked fine for me because it’s rather rare that non-block > >> patches broke the iotests. > > > > I disagree. It happened all the time

Re: [PATCH v15 4/5] block: introduce backup-top filter driver

2019-10-04 Thread Max Reitz
On 01.10.19 15:14, Vladimir Sementsov-Ogievskiy wrote: > Backup-top filter caches write operations and does copy-before-write > operations. > > The driver will be used in backup instead of write-notifiers. > > Signed-off-by: Vladimir Sementsov-Ogievskiy > --- > block/backup-top.h | 41

Re: [PATCH v5 1/5] iotests: remove 'linux' from default supported platforms

2019-10-04 Thread Max Reitz
On 04.10.19 15:16, Peter Maydell wrote: > On Fri, 4 Oct 2019 at 13:45, Max Reitz wrote: >>> In the end, I don't care what code these patches touched. I do an >>> innocent git pull, and when I finally see that it's master that breaks >>> iotests and not my patches on top of it, I'm annoyed. >> >>

Re: [PATCH v5 1/5] iotests: remove 'linux' from default supported platforms

2019-10-04 Thread Peter Maydell
On Fri, 4 Oct 2019 at 14:50, Max Reitz wrote: > On 04.10.19 15:16, Peter Maydell wrote: > > 'make check' does have the restriction > > that we don't want the tests to take too long to run, but in > > general the block layer should be running some reasonable subset > > of tests in the project's

Re: [PATCH 02/67] iotests.py: Add @skip_for_imgopts()

2019-10-04 Thread Max Reitz
On 03.10.19 17:19, Vladimir Sementsov-Ogievskiy wrote: > 01.10.2019 22:46, Max Reitz wrote: >> Signed-off-by: Max Reitz >> --- >> tests/qemu-iotests/iotests.py | 13 + >> 1 file changed, 13 insertions(+) >> >> diff --git a/tests/qemu-iotests/iotests.py

Re: [PATCH 3/4] block/mirror: support unaligned write in active mirror

2019-10-04 Thread Max Reitz
On 03.10.19 11:34, Vladimir Sementsov-Ogievskiy wrote: > 02.10.2019 18:52, Max Reitz wrote: >> On 02.10.19 17:06, Vladimir Sementsov-Ogievskiy wrote: >>> 02.10.2019 18:03, Vladimir Sementsov-Ogievskiy wrote: 02.10.2019 17:57, Max Reitz wrote: > On 12.09.19 17:13, Vladimir

Re: [PATCH v5 1/5] iotests: remove 'linux' from default supported platforms

2019-10-04 Thread Peter Maydell
On Fri, 4 Oct 2019 at 13:45, Max Reitz wrote: > > In the end, I don't care what code these patches touched. I do an > > innocent git pull, and when I finally see that it's master that breaks > > iotests and not my patches on top of it, I'm annoyed. > > Hm. Part of my point was that this still

Re: [PATCH v2 3/6] qemu-nbd: add compression flag support

2019-10-04 Thread Roman Kagan
On Wed, Oct 02, 2019 at 05:22:43PM +0300, Andrey Shinkevich wrote: > Added possibility to write compressed data by using the > blk_write_compressed. This action has the limitations of the format > driver. For example we can't write compressed data over other. > > $ ./qemu-img create -f qcow2 -o

Re: [PATCH v5 1/5] iotests: remove 'linux' from default supported platforms

2019-10-04 Thread Max Reitz
On 04.10.19 12:19, Kevin Wolf wrote: > Am 02.10.2019 um 19:47 hat Max Reitz geschrieben: >> On 02.10.19 18:44, Kevin Wolf wrote: >>> Am 02.10.2019 um 13:57 hat Max Reitz geschrieben: It usually worked fine for me because it’s rather rare that non-block patches broke the iotests. >>> >>>

Re: bitmap migration bug with -drive while block mirror runs

2019-10-04 Thread Eric Blake
On 10/4/19 4:24 AM, Vladimir Sementsov-Ogievskiy wrote: The way I see it, we know an auto-generated node name will never be correct, but an explicitly specified one represents an explicit user configuration. It's wrong to use generated names for migration details, but it's never wrong to use

Re: [PATCH 3/4] block/mirror: support unaligned write in active mirror

2019-10-04 Thread Vladimir Sementsov-Ogievskiy
04.10.2019 15:59, Max Reitz wrote: > On 03.10.19 11:34, Vladimir Sementsov-Ogievskiy wrote: >> 02.10.2019 18:52, Max Reitz wrote: >>> On 02.10.19 17:06, Vladimir Sementsov-Ogievskiy wrote: 02.10.2019 18:03, Vladimir Sementsov-Ogievskiy wrote: > 02.10.2019 17:57, Max Reitz wrote: >> On

Re: [PATCH v2 02/11] qcrypto-luks: extend the create options for upcoming encryption key management

2019-10-04 Thread Max Reitz
On 13.09.19 00:30, Maxim Levitsky wrote: > Now you can specify which slot to put the encryption key to > Plus add 'active' option which will let user erase the key secret > instead of adding it. > Check that active=true it when creating. > > Signed-off-by: Maxim Levitsky > --- > block/crypto.c

Re: [PATCH v2 09/11] block/qcow2: implement blockdev-amend

2019-10-04 Thread Max Reitz
On 13.09.19 00:30, Maxim Levitsky wrote: > Currently only for changing crypto parameters Yep, that elegantly avoids most of the problems we’d have otherwise. :-) > Signed-off-by: Maxim Levitsky > --- > block/qcow2.c| 71 >

Re: [PATCH 4/4] Revert "mirror: Only mirror granularity-aligned chunks"

2019-10-04 Thread Max Reitz
On 12.09.19 17:13, Vladimir Sementsov-Ogievskiy wrote: > This reverts commit 9adc1cb49af8d4e54f57980b1eed5c0a4b2dafa6. > "mirror: Only mirror granularity-aligned chunks" > > Since previous commit unaligned chunks are supported by > do_sync_target_write. > > Signed-off-by: Vladimir

Re: [PATCH v9 2/3] block/nbd: nbd reconnect

2019-10-04 Thread Vladimir Sementsov-Ogievskiy
23 Sep 2019 22:23 Eric Blake wrote: On 9/17/19 12:13 PM, Vladimir Sementsov-Ogievskiy wrote: > Implement reconnect. To achieve this: > > 1. add new modes: >connecting-wait: means, that reconnecting is in progress, and there > were small number of reconnect attempts, so all requests are

Re: [PATCH v2 05/11] block/crypto: implement the encryption key management

2019-10-04 Thread Max Reitz
On 13.09.19 00:30, Maxim Levitsky wrote: > This implements the encryption key management > using the generic code in qcrypto layer > (currently only for qemu-img amend) > > This code adds another 'write_func' because the initialization > write_func works directly on the underlying file, > because

Re: [PATCH v2 07/11] block: add x-blockdev-amend qmp command

2019-10-04 Thread Max Reitz
On 13.09.19 00:30, Maxim Levitsky wrote: > Signed-off-by: Maxim Levitsky > --- > block/Makefile.objs | 2 +- > block/amend.c | 116 ++ > include/block/block_int.h | 23 ++-- > qapi/block-core.json | 26 + > qapi/job.json

Re: [PATCH v9 3/3] iotests: test nbd reconnect

2019-10-04 Thread Eric Blake
On 9/24/19 3:31 AM, Vladimir Sementsov-Ogievskiy wrote: +def qemu_nbd_popen(*args): +'''Run qemu-nbd in daemon mode and return the parent's exit code''' +return subprocess.Popen(qemu_nbd_args + ['--persistent'] + list(args)) + Should you also use a pid file here, and wait for the

Re: [PATCH 3/4] block/mirror: support unaligned write in active mirror

2019-10-04 Thread Max Reitz
On 12.09.19 17:13, Vladimir Sementsov-Ogievskiy wrote: > Prior 9adc1cb49af8d do_sync_target_write had a bug: it reset aligned-up > region in the dirty bitmap, which means that we may not copy some bytes > and assume them copied, which actually leads to producing corrupted > target. > > So

Re: [PATCH v2 00/11] RFC crypto/luks: encryption key managment using amend interface

2019-10-04 Thread Max Reitz
On 13.09.19 00:30, Maxim Levitsky wrote: > This patch series is continuation of my work to add encryption > key managment to luks/qcow2 with luks. > > This is second version of this patch set. > The changes are mostly addressing the review feedback, > plus I tested (and fixed sadly) the somewhat

Re: [PATCH v2 11/11] iotests : add tests for encryption key management

2019-10-04 Thread Max Reitz
On 13.09.19 00:30, Maxim Levitsky wrote: > Note that currently I add tests 300-302, which are > placeholders to ease the rebase. In final version > of these patches I will update these. > > Signed-off-by: Maxim Levitsky > --- > tests/qemu-iotests/300 | 202 + >