Re: [Qemu-block] [Qemu-devel] [PATCH v6 0/3] Fix qcow2+luks corruption introduced by commit 8ac0f15f335

2019-09-13 Thread no-reply
Patchew URL: https://patchew.org/QEMU/20190913172741.5662-1-mlevi...@redhat.com/ Hi, This series failed the docker-quick@centos7 build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally. === TEST SCRIPT BEGIN ===

Re: [Qemu-block] [Qemu-devel] [PATCH 3/4] iotests: Add @error to wait_until_completed

2019-09-13 Thread John Snow
On 9/12/19 9:56 AM, Max Reitz wrote: > Callers can use this new parameter to expect failure during the > completion process. > > Signed-off-by: Max Reitz > --- > tests/qemu-iotests/iotests.py | 18 -- > 1 file changed, 12 insertions(+), 6 deletions(-) > > diff --git

Re: [Qemu-block] [Qemu-devel] [PATCH 1/4] mirror: Do not dereference invalid pointers

2019-09-13 Thread John Snow
On 9/12/19 9:56 AM, Max Reitz wrote: > mirror_exit_common() may be called twice (if it is called from > mirror_prepare() and fails, it will be called from mirror_abort() > again). > > In such a case, many of the pointers in the MirrorBlockJob object will > already be freed. This can be seen

Re: [Qemu-block] [Qemu-devel] [PATCH v2 3/3] block/qcow2: proper locking on bitmap add/remove paths

2019-09-13 Thread John Snow
On 9/11/19 11:00 AM, Vladimir Sementsov-Ogievskiy wrote: > qmp_block_dirty_bitmap_add and do_block_dirty_bitmap_remove do acquire > aio context since 0a6c86d024c52b. But this is not enough: we also must > lock qcow2 mutex when access in-image metadata. Especially it concerns > freeing qcow2

Re: [Qemu-block] [Qemu-devel] [PATCH v2 2/3] block/dirty-bitmap: return int from bdrv_remove_persistent_dirty_bitmap

2019-09-13 Thread John Snow
On 9/11/19 11:00 AM, Vladimir Sementsov-Ogievskiy wrote: > It's more comfortable to not deal with local_err. > I agree. > Signed-off-by: Vladimir Sementsov-Ogievskiy > --- > block/qcow2.h| 5 ++--- > include/block/block_int.h| 6 +++--- > include/block/dirty-bitmap.h

Re: [Qemu-block] [Qemu-devel] [PATCH v2 1/3] block: move bdrv_can_store_new_dirty_bitmap to block/dirty-bitmap.c

2019-09-13 Thread John Snow
On 9/11/19 11:00 AM, Vladimir Sementsov-Ogievskiy wrote: > block/dirty-bitmap.c seems to be more appropriate for it and > bdrv_remove_persistent_dirty_bitmap already in it. > > Signed-off-by: Vladimir Sementsov-Ogievskiy No disagreements here. I think we put it in block.c initially because

Re: [Qemu-block] [PATCH 2/2] trace: Forbid event format ending with newline character

2019-09-13 Thread John Snow
On 9/13/19 5:00 PM, Philippe Mathieu-Daudé wrote: > On 9/13/19 10:01 PM, John Snow wrote: >> On 9/13/19 6:52 AM, Philippe Mathieu-Daudé wrote: >>> Event format ending with newlines confuse the trace reports. >>> Forbid them. >>> >>> Add a check to refuse new format added with trailing newline:

Re: [Qemu-block] [PATCH 2/2] trace: Forbid event format ending with newline character

2019-09-13 Thread Philippe Mathieu-Daudé
On 9/13/19 10:01 PM, John Snow wrote: > On 9/13/19 6:52 AM, Philippe Mathieu-Daudé wrote: >> Event format ending with newlines confuse the trace reports. >> Forbid them. >> >> Add a check to refuse new format added with trailing newline: >> >> $ make >> [...] >> GEN hw/misc/trace.h >>

Re: [Qemu-block] [PATCH v2] virtio-blk: schedule virtio_notify_config to run on main context

2019-09-13 Thread John Snow
On 9/13/19 6:56 AM, Sergio Lopez wrote: > virtio_notify_config() needs to acquire the global mutex, which isn't > allowed from an iothread, and may lead to a deadlock like this: > > - main thead > * Has acquired: qemu_global_mutex. > * Is trying the acquire: iothread AioContext lock via >

Re: [Qemu-block] [PATCH 0/2] trace: Forbid trailing newline in event format

2019-09-13 Thread John Snow
On 9/13/19 6:52 AM, Philippe Mathieu-Daudé wrote: > Hi Stefan, > > I'v been confused by trailing newline in trace reports, > so this series aims to fix this, by cleaning current > formats and add a check to catch new one introduced. > > Regards, > > Phil. > > Philippe Mathieu-Daudé (2): >

Re: [Qemu-block] [PATCH 2/2] trace: Forbid event format ending with newline character

2019-09-13 Thread John Snow
On 9/13/19 6:52 AM, Philippe Mathieu-Daudé wrote: > Event format ending with newlines confuse the trace reports. > Forbid them. > > Add a check to refuse new format added with trailing newline: > > $ make > [...] > GEN hw/misc/trace.h > Traceback (most recent call last): >

Re: [Qemu-block] [PATCH v2 1/2] blockdev: release the AioContext at drive_backup_prepare

2019-09-13 Thread John Snow
On 9/13/19 11:25 AM, Sergio Lopez wrote: > do_drive_backup() already acquires the AioContext, so release it > before the call. > > Signed-off-by: Sergio Lopez > --- > blockdev.c | 6 +- > 1 file changed, 1 insertion(+), 5 deletions(-) > > diff --git a/blockdev.c b/blockdev.c > index

Re: [Qemu-block] [Qemu-devel] [PATCH v2] util/hbitmap: strict hbitmap_reset

2019-09-13 Thread John Snow
On 9/12/19 4:20 AM, Vladimir Sementsov-Ogievskiy wrote: > 11.09.2019 20:59, John Snow wrote: >> >> >> On 9/11/19 11:13 AM, Vladimir Sementsov-Ogievskiy wrote: >>> 07.08.2019 19:27, John Snow wrote: On 8/6/19 12:19 PM, Vladimir Sementsov-Ogievskiy wrote: > 06.08.2019 19:09,

Re: [Qemu-block] [PATCH v6 2/3] block/qcow2: refactor threaded encryption code

2019-09-13 Thread Vladimir Sementsov-Ogievskiy
and a bit about empty lines 13.09.2019 20:27, Maxim Levitsky wrote: > Change the qcow2_co_encrypt to just receive full host and > guest offsets and in pariticular remove the > offset_in_cluster parameter of do_perform_cow_encrypt, > since it is misleading, because that offset can be larger than >

Re: [Qemu-block] [Qemu-devel] [PATCH v3 6/8] iotests: Test driver whitelisting in 093

2019-09-13 Thread John Snow
On 9/13/19 8:47 AM, Max Reitz wrote: > On 20.08.19 23:32, John Snow wrote: >> >> >> On 8/19/19 4:18 PM, Max Reitz wrote: >>> null-aio may not be whitelisted. Skip all test cases that require it. >>> >>> Signed-off-by: Max Reitz >>> --- >>> tests/qemu-iotests/093 | 12 +--- >>> 1 file

Re: [Qemu-block] [PATCH v11 04/14] block/backup: introduce BlockCopyState

2019-09-13 Thread Vladimir Sementsov-Ogievskiy
10.09.2019 13:23, Vladimir Sementsov-Ogievskiy wrote: > Split copying code part from backup to "block-copy", including separate > state structure and function renaming. This is needed to share it with > backup-top filter driver in further commits. > > Notes: > > 1. As BlockCopyState keeps own

Re: [Qemu-block] [Qemu-devel] [PATCH v6 2/3] block/qcow2: refactor threaded encryption code

2019-09-13 Thread Maxim Levitsky
On Fri, 2019-09-13 at 17:51 +, Vladimir Sementsov-Ogievskiy wrote: > 13.09.2019 20:27, Maxim Levitsky wrote: > > Change the qcow2_co_encrypt to just receive full host and > > guest offsets and in pariticular remove the > > offset_in_cluster parameter of do_perform_cow_encrypt, > > since it is

Re: [Qemu-block] [PATCH v6 2/3] block/qcow2: refactor threaded encryption code

2019-09-13 Thread Vladimir Sementsov-Ogievskiy
13.09.2019 20:27, Maxim Levitsky wrote: > Change the qcow2_co_encrypt to just receive full host and > guest offsets and in pariticular remove the > offset_in_cluster parameter of do_perform_cow_encrypt, > since it is misleading, because that offset can be larger than > cluster size currently. > >

[Qemu-block] [PATCH v6 2/3] block/qcow2: refactor threaded encryption code

2019-09-13 Thread Maxim Levitsky
Change the qcow2_co_encrypt to just receive full host and guest offsets and in pariticular remove the offset_in_cluster parameter of do_perform_cow_encrypt, since it is misleading, because that offset can be larger than cluster size currently. Remove the do_perform_cow_encrypt by merging it with

[Qemu-block] [PATCH v6 3/3] qemu-iotests: Add test for bz #1745922

2019-09-13 Thread Maxim Levitsky
Signed-off-by: Maxim Levitsky Tested-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/263 | 91 ++ tests/qemu-iotests/263.out | 40 + tests/qemu-iotests/group | 1 + 3 files changed, 132 insertions(+) create mode 100755

[Qemu-block] [PATCH v6 1/3] Fix qcow2+luks corruption introduced by commit 8ac0f15f335

2019-09-13 Thread Maxim Levitsky
This fixes subtle corruption introduced by luks threaded encryption in commit 8ac0f15f335 Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1745922 The corruption happens when we do a write that * writes to two or more unallocated clusters at once * doesn't fully cover the first sector

[Qemu-block] [PATCH v6 0/3] Fix qcow2+luks corruption introduced by commit 8ac0f15f335

2019-09-13 Thread Maxim Levitsky
Commit 8ac0f15f335 accidently broke the COW of non changed areas of newly allocated clusters, when the write spans multiple clusters, and needs COW both prior and after the write. This results in 'after' COW area being encrypted with wrong sector address, which render it corrupted. Bugzilla:

Re: [Qemu-block] [PATCH v5 3/3] qemu-iotests: Add test for bz #1745922

2019-09-13 Thread Maxim Levitsky
On Fri, 2019-09-13 at 16:57 +, Vladimir Sementsov-Ogievskiy wrote: > 13.09.2019 19:39, Maxim Levitsky wrote: > > On Fri, 2019-09-13 at 16:27 +, Vladimir Sementsov-Ogievskiy wrote: > > > 13.09.2019 18:28, Maxim Levitsky wrote: > > > > Signed-off-by: Maxim Levitsky > > > > --- > > > >

Re: [Qemu-block] [PATCH v5 3/3] qemu-iotests: Add test for bz #1745922

2019-09-13 Thread Vladimir Sementsov-Ogievskiy
13.09.2019 19:39, Maxim Levitsky wrote: > On Fri, 2019-09-13 at 16:27 +, Vladimir Sementsov-Ogievskiy wrote: >> 13.09.2019 18:28, Maxim Levitsky wrote: >>> Signed-off-by: Maxim Levitsky >>> --- >>>tests/qemu-iotests/263 | 91 ++ >>>

Re: [Qemu-block] [PATCH v5 3/3] qemu-iotests: Add test for bz #1745922

2019-09-13 Thread Maxim Levitsky
On Fri, 2019-09-13 at 16:27 +, Vladimir Sementsov-Ogievskiy wrote: > 13.09.2019 18:28, Maxim Levitsky wrote: > > Signed-off-by: Maxim Levitsky > > --- > > tests/qemu-iotests/263 | 91 ++ > > tests/qemu-iotests/263.out | 40 + > >

Re: [Qemu-block] [PATCH v5 2/3] block/qcow2: refactor threaded encryption code

2019-09-13 Thread Maxim Levitsky
On Fri, 2019-09-13 at 16:11 +, Vladimir Sementsov-Ogievskiy wrote: > 13.09.2019 18:28, Maxim Levitsky wrote: > > Change do_perform_cow_encrypt and its callee qcow2_co_encrypt > > to just receive full host and guest offsets and in pariticular > > remove the offset_in_cluster parameter of

Re: [Qemu-block] [PATCH v5 3/3] qemu-iotests: Add test for bz #1745922

2019-09-13 Thread Vladimir Sementsov-Ogievskiy
13.09.2019 18:28, Maxim Levitsky wrote: > Signed-off-by: Maxim Levitsky > --- > tests/qemu-iotests/263 | 91 ++ > tests/qemu-iotests/263.out | 40 + > tests/qemu-iotests/group | 2 + > 3 files changed, 133 insertions(+) > create

Re: [Qemu-block] [PATCH v5 2/3] block/qcow2: refactor threaded encryption code

2019-09-13 Thread Vladimir Sementsov-Ogievskiy
13.09.2019 18:28, Maxim Levitsky wrote: > Change do_perform_cow_encrypt and its callee qcow2_co_encrypt > to just receive full host and guest offsets and in pariticular > remove the offset_in_cluster parameter of do_perform_cow_encrypt, > since it is misleading, because that offset can be larger

Re: [Qemu-block] [PATCH v5 1/3] Fix qcow2+luks corruption introduced by commit 8ac0f15f335

2019-09-13 Thread Vladimir Sementsov-Ogievskiy
13.09.2019 18:28, Maxim Levitsky wrote: > This fixes subtle corruption introduced by luks threaded encryption > in commit 8ac0f15f335 > > Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1745922 > > The corruption happens when we do a write that > * writes to two or more unallocated

Re: [Qemu-block] [PATCH v11 00/14] backup-top filter driver for backup

2019-09-13 Thread Vladimir Sementsov-Ogievskiy
10.09.2019 13:23, Vladimir Sementsov-Ogievskiy wrote: > Hi all! > > These series introduce backup-top driver. It's a filter-node, which > do copy-before-write operation. Mirror uses filter-node for handling > guest writes, let's move to filter-node (from write-notifiers) for > backup too. > >

[Qemu-block] [PATCH v5 2/3] block/qcow2: refactor threaded encryption code

2019-09-13 Thread Maxim Levitsky
Change do_perform_cow_encrypt and its callee qcow2_co_encrypt to just receive full host and guest offsets and in pariticular remove the offset_in_cluster parameter of do_perform_cow_encrypt, since it is misleading, because that offset can be larger than cluster size currently. Also document the

[Qemu-block] [PATCH v5 3/3] qemu-iotests: Add test for bz #1745922

2019-09-13 Thread Maxim Levitsky
Signed-off-by: Maxim Levitsky --- tests/qemu-iotests/263 | 91 ++ tests/qemu-iotests/263.out | 40 + tests/qemu-iotests/group | 2 + 3 files changed, 133 insertions(+) create mode 100755 tests/qemu-iotests/263 create mode 100644

[Qemu-block] [PATCH v5 1/3] Fix qcow2+luks corruption introduced by commit 8ac0f15f335

2019-09-13 Thread Maxim Levitsky
This fixes subtle corruption introduced by luks threaded encryption in commit 8ac0f15f335 Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1745922 The corruption happens when we do a write that * writes to two or more unallocated clusters at once * doesn't fully cover the first sector

[Qemu-block] [PATCH v2 2/2] blockdev: honor bdrv_try_set_aio_context() context requirements

2019-09-13 Thread Sergio Lopez
bdrv_try_set_aio_context() requires that the old context is held, and the new context is not held. Fix all the ocurrences where it's not done this way. Suggested-by: Max Reitz Signed-off-by: Sergio Lopez --- blockdev.c | 121 + 1 file

[Qemu-block] [PATCH v5 0/3] Fix qcow2+luks corruption introduced by commit 8ac0f15f335

2019-09-13 Thread Maxim Levitsky
Commit 8ac0f15f335 accidently broke the COW of non changed areas of newly allocated clusters, when the write spans multiple clusters, and needs COW both prior and after the write. This results in 'after' COW area being encrypted with wrong sector address, which render it corrupted. Bugzilla:

[Qemu-block] [PATCH v2 1/2] blockdev: release the AioContext at drive_backup_prepare

2019-09-13 Thread Sergio Lopez
do_drive_backup() already acquires the AioContext, so release it before the call. Signed-off-by: Sergio Lopez --- blockdev.c | 6 +- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/blockdev.c b/blockdev.c index fbef6845c8..3927fdab80 100644 --- a/blockdev.c +++ b/blockdev.c @@

[Qemu-block] [PATCH v2 0/2] blockdev: avoid acquiring AioContext lock twice at do_drive_backup()

2019-09-13 Thread Sergio Lopez
do_drive_backup() acquires the AioContext lock of the corresponding BlockDriverState. This is not a problem when it's called from qmp_drive_backup(), but drive_backup_prepare() also acquires the lock before calling it. Additionally, Max Reitz pointed out that bdrv_try_set_aio_context() is called

Re: [Qemu-block] [Qemu-devel] [PATCH v4 1/3] block/qcow2: refactoring of threaded encryption code

2019-09-13 Thread Kevin Wolf
Am 13.09.2019 um 16:44 hat Vladimir Sementsov-Ogievskiy geschrieben: > 13.09.2019 17:37, Maxim Levitsky wrote: > > On Fri, 2019-09-13 at 16:24 +0200, Kevin Wolf wrote: > >> Am 13.09.2019 um 16:07 hat Maxim Levitsky geschrieben: > >>> On Fri, 2019-09-13 at 14:01 +, Vladimir Sementsov-Ogievskiy

Re: [Qemu-block] [Qemu-devel] [PATCH v4 1/3] block/qcow2: refactoring of threaded encryption code

2019-09-13 Thread Kevin Wolf
Am 13.09.2019 um 16:37 hat Maxim Levitsky geschrieben: > On Fri, 2019-09-13 at 16:24 +0200, Kevin Wolf wrote: > > Am 13.09.2019 um 16:07 hat Maxim Levitsky geschrieben: > > > On Fri, 2019-09-13 at 14:01 +, Vladimir Sementsov-Ogievskiy wrote: > > > > 13.09.2019 15:59, Maxim Levitsky wrote: > >

Re: [Qemu-block] [Qemu-devel] [PATCH v4 1/3] block/qcow2: refactoring of threaded encryption code

2019-09-13 Thread Vladimir Sementsov-Ogievskiy
13.09.2019 17:37, Maxim Levitsky wrote: > On Fri, 2019-09-13 at 16:24 +0200, Kevin Wolf wrote: >> Am 13.09.2019 um 16:07 hat Maxim Levitsky geschrieben: >>> On Fri, 2019-09-13 at 14:01 +, Vladimir Sementsov-Ogievskiy wrote: 13.09.2019 15:59, Maxim Levitsky wrote: > This commit tries

Re: [Qemu-block] [Qemu-devel] [PATCH v4 1/3] block/qcow2: refactoring of threaded encryption code

2019-09-13 Thread Maxim Levitsky
On Fri, 2019-09-13 at 16:24 +0200, Kevin Wolf wrote: > Am 13.09.2019 um 16:07 hat Maxim Levitsky geschrieben: > > On Fri, 2019-09-13 at 14:01 +, Vladimir Sementsov-Ogievskiy wrote: > > > 13.09.2019 15:59, Maxim Levitsky wrote: > > > > This commit tries to clarify few function arguments, > > >

Re: [Qemu-block] [Qemu-devel] [PATCH v4 1/3] block/qcow2: refactoring of threaded encryption code

2019-09-13 Thread Kevin Wolf
Am 13.09.2019 um 16:07 hat Maxim Levitsky geschrieben: > On Fri, 2019-09-13 at 14:01 +, Vladimir Sementsov-Ogievskiy wrote: > > 13.09.2019 15:59, Maxim Levitsky wrote: > > > This commit tries to clarify few function arguments, > > > and add comments describing the encrypt/decrypt interface > >

Re: [Qemu-block] [Qemu-devel] [PATCH v4 1/3] block/qcow2: refactoring of threaded encryption code

2019-09-13 Thread Vladimir Sementsov-Ogievskiy
13.09.2019 17:07, Maxim Levitsky wrote: > On Fri, 2019-09-13 at 14:01 +, Vladimir Sementsov-Ogievskiy wrote: >> 13.09.2019 15:59, Maxim Levitsky wrote: >>> This commit tries to clarify few function arguments, >>> and add comments describing the encrypt/decrypt interface >>> >>> Signed-off-by:

Re: [Qemu-block] [PATCH v6 28/42] stream: Deal with filters

2019-09-13 Thread Kevin Wolf
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben: > Because of the recent changes that make the stream job independent of > the base node and instead track the node above it, we have to split that > "bottom" node into two cases: The bottom COW node, and the node directly > above the base node

Re: [Qemu-block] [Qemu-devel] [PATCH v4 1/3] block/qcow2: refactoring of threaded encryption code

2019-09-13 Thread Maxim Levitsky
On Fri, 2019-09-13 at 14:01 +, Vladimir Sementsov-Ogievskiy wrote: > 13.09.2019 15:59, Maxim Levitsky wrote: > > This commit tries to clarify few function arguments, > > and add comments describing the encrypt/decrypt interface > > > > Signed-off-by: Maxim Levitsky > > --- > >

Re: [Qemu-block] [PATCH v4 1/3] block/qcow2: refactoring of threaded encryption code

2019-09-13 Thread Vladimir Sementsov-Ogievskiy
13.09.2019 15:59, Maxim Levitsky wrote: > This commit tries to clarify few function arguments, > and add comments describing the encrypt/decrypt interface > > Signed-off-by: Maxim Levitsky > --- > block/qcow2-cluster.c | 9 --- > block/qcow2-threads.c | 62

[Qemu-block] [PATCH v2 1/2] block/nvme: add support for write zeros

2019-09-13 Thread Maxim Levitsky
Signed-off-by: Maxim Levitsky --- block/nvme.c | 72 +++- block/trace-events | 1 + include/block/nvme.h | 19 +++- 3 files changed, 90 insertions(+), 2 deletions(-) diff --git a/block/nvme.c b/block/nvme.c index 5be3a39b63..d95265fae4

Re: [Qemu-block] [PULL 00/22] Block layer patches

2019-09-13 Thread Peter Maydell
On Thu, 12 Sep 2019 at 14:46, Kevin Wolf wrote: > > The following changes since commit 89ea03a7dc83ca36b670ba7f787802791fcb04b1: > > Merge remote-tracking branch > 'remotes/huth-gitlab/tags/m68k-pull-2019-09-07' into staging (2019-09-09 > 09:48:34 +0100) > > are available in the Git

[Qemu-block] [PATCH v2 0/2] block/nvme: add support for write zeros and discard

2019-09-13 Thread Maxim Levitsky
This is the second part of the patches I prepared for this driver back when I worked on mdev-nvme. V2: addressed review feedback, no major changes Best regards, Maxim Levitsky Maxim Levitsky (2): block/nvme: add support for write zeros block/nvme: add support for discard

[Qemu-block] [PATCH v2 2/2] block/nvme: add support for discard

2019-09-13 Thread Maxim Levitsky
Signed-off-by: Maxim Levitsky --- block/nvme.c | 83 ++ block/trace-events | 2 ++ 2 files changed, 85 insertions(+) diff --git a/block/nvme.c b/block/nvme.c index d95265fae4..c17edd6aae 100644 --- a/block/nvme.c +++ b/block/nvme.c @@ -112,6

Re: [Qemu-block] [Qemu-devel] [PATCH 1/2] block/nvme: add support for write zeros

2019-09-13 Thread Maxim Levitsky
On Tue, 2019-08-27 at 18:22 -0400, John Snow wrote: > Without a commit message, I have no real hope of reviewing this. I was > CC'd, though, so I'll give it a blind shot. > > We want to add write_zeroes support for block/nvme, but I can't really > verify any of that is correct or working without

Re: [Qemu-block] [PATCH v3 1/3] block/qcow2: refactoring of threaded encryption code

2019-09-13 Thread Maxim Levitsky
On Fri, 2019-09-13 at 12:37 +0200, Max Reitz wrote: > On 13.09.19 00:37, Maxim Levitsky wrote: > > This commit tries to clarify few function arguments, > > and add comments describing the encrypt/decrypt interface > > > > Signed-off-by: Maxim Levitsky > > --- > > block/qcow2-cluster.c | 8

[Qemu-block] [PATCH v4 2/3] block/qcow2: fix the corruption when rebasing luks encrypted files

2019-09-13 Thread Maxim Levitsky
This fixes subtle corruption introduced by luks threaded encryption in commit 8ac0f15f335 Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1745922 The corruption happens when we do a write that * writes to two or more unallocated clusters at once * doesn't fully cover the first sector

[Qemu-block] [PATCH v4 3/3] qemu-iotests: Add test for bz #1745922

2019-09-13 Thread Maxim Levitsky
Signed-off-by: Maxim Levitsky --- tests/qemu-iotests/263 | 91 ++ tests/qemu-iotests/263.out | 40 + tests/qemu-iotests/group | 1 + 3 files changed, 132 insertions(+) create mode 100755 tests/qemu-iotests/263 create mode 100644

[Qemu-block] [PATCH v4 1/3] block/qcow2: refactoring of threaded encryption code

2019-09-13 Thread Maxim Levitsky
This commit tries to clarify few function arguments, and add comments describing the encrypt/decrypt interface Signed-off-by: Maxim Levitsky --- block/qcow2-cluster.c | 9 --- block/qcow2-threads.c | 62 ++- block/qcow2.c | 5 ++--

[Qemu-block] [PATCH v4 0/3] Fix qcow2+luks corruption introduced by commit 8ac0f15f335

2019-09-13 Thread Maxim Levitsky
Commit 8ac0f15f335 accidently broke the COW of non changed areas of newly allocated clusters, when the write spans multiple clusters, and needs COW both prior and after the write. This results in 'after' COW area being encrypted with wrong sector address, which render it corrupted. Bugzilla:

Re: [Qemu-block] [PATCH v6 25/42] mirror: Deal with filters

2019-09-13 Thread Kevin Wolf
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben: > This includes some permission limiting (for example, we only need to > take the RESIZE permission for active commits where the base is smaller > than the top). > > Signed-off-by: Max Reitz > --- > block/mirror.c | 117

Re: [Qemu-block] [Qemu-devel] [PATCH v3 6/8] iotests: Test driver whitelisting in 093

2019-09-13 Thread Max Reitz
On 20.08.19 23:32, John Snow wrote: > > > On 8/19/19 4:18 PM, Max Reitz wrote: >> null-aio may not be whitelisted. Skip all test cases that require it. >> >> Signed-off-by: Max Reitz >> --- >> tests/qemu-iotests/093 | 12 +--- >> 1 file changed, 9 insertions(+), 3 deletions(-) >> >>

Re: [Qemu-block] [PULL 0/1] Block patches

2019-09-13 Thread Peter Maydell
On Wed, 11 Sep 2019 at 15:36, Stefan Hajnoczi wrote: > > The following changes since commit cc6613e244e86c66f83467eab5284825d7057cea: > > Merge remote-tracking branch 'remotes/stefanha/tags/tracing-pull-request' > into staging (2019-09-03 11:06:09 +0100) > > are available in the Git repository

Re: [Qemu-block] [PATCH v4 3/5] block/qcow2: refactor qcow2_co_preadv_part

2019-09-13 Thread Max Reitz
On 13.09.19 13:34, Kevin Wolf wrote: > Am 13.09.2019 um 13:06 hat Max Reitz geschrieben: >> On 13.09.19 12:53, Vladimir Sementsov-Ogievskiy wrote: >>> 13.09.2019 13:01, Kevin Wolf wrote: Am 16.08.2019 um 17:30 hat Vladimir Sementsov-Ogievskiy geschrieben: > Further patch will run partial

Re: [Qemu-block] [PATCH v3 3/3] qemu-iotests: Add test for bz #1745922

2019-09-13 Thread Maxim Levitsky
On Fri, 2019-09-13 at 13:01 +0200, Max Reitz wrote: > On 13.09.19 00:37, Maxim Levitsky wrote: > > Signed-off-by: Maxim Levitsky > > --- > > tests/qemu-iotests/263 | 91 ++ > > tests/qemu-iotests/263.out | 40 + > > tests/qemu-iotests/group

Re: [Qemu-block] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames

2019-09-13 Thread Max Reitz
Another gentle ping. Max On 24.06.19 19:39, Max Reitz wrote: > Hi, > > There are two explanations of this cover letter, a relative one (to v3) > and an absolute one. > > > *** Important note *** > > The final patch in this series is an example that converts most of > block-core.json to use

Re: [Qemu-block] [PATCH v2 6/7] curl: Handle success in multi_check_completion

2019-09-13 Thread Maxim Levitsky
On Fri, 2019-09-13 at 13:20 +0200, Max Reitz wrote: > On 10.09.19 18:13, Maxim Levitsky wrote: > > On Tue, 2019-09-10 at 14:41 +0200, Max Reitz wrote: > > > Background: As of cURL 7.59.0, it verifies that several functions are > > > not called from within a callback. Among these functions is > >

Re: [Qemu-block] [PATCH v2 0/7] block/curl: Fix hang and potential crash

2019-09-13 Thread Max Reitz
On 10.09.19 14:41, Max Reitz wrote: > Hi, > > As reported in https://bugzilla.redhat.com/show_bug.cgi?id=1740193, our > curl block driver can spontaneously hang. This becomes visible e.g. > when reading compressed qcow2 images: > > $ qemu-img convert -p -O raw -n \ >

Re: [Qemu-block] [PATCH v4 3/5] block/qcow2: refactor qcow2_co_preadv_part

2019-09-13 Thread Kevin Wolf
Am 13.09.2019 um 13:06 hat Max Reitz geschrieben: > On 13.09.19 12:53, Vladimir Sementsov-Ogievskiy wrote: > > 13.09.2019 13:01, Kevin Wolf wrote: > >> Am 16.08.2019 um 17:30 hat Vladimir Sementsov-Ogievskiy geschrieben: > >>> Further patch will run partial requests of iterations of > >>>

Re: [Qemu-block] [PATCH v3] blockjob: update nodes head while removing all bdrv

2019-09-13 Thread Max Reitz
On 11.09.19 12:03, Max Reitz wrote: > From: Sergio Lopez > > block_job_remove_all_bdrv() iterates through job->nodes, calling > bdrv_root_unref_child() for each entry. The call to the latter may > reach child_job_[can_]set_aio_ctx(), which will also attempt to > traverse job->nodes, potentially

Re: [Qemu-block] [PATCH v2 6/7] curl: Handle success in multi_check_completion

2019-09-13 Thread Max Reitz
On 10.09.19 18:13, Maxim Levitsky wrote: > On Tue, 2019-09-10 at 14:41 +0200, Max Reitz wrote: >> Background: As of cURL 7.59.0, it verifies that several functions are >> not called from within a callback. Among these functions is >> curl_multi_add_handle(). >> >> curl_read_cb() is a callback

Re: [Qemu-block] [PATCH V2 2/2] block/nfs: add support for nfs_umount

2019-09-13 Thread Max Reitz
On 13.09.19 12:09, ronnie sahlberg wrote: > On Wed, Sep 11, 2019 at 5:48 PM Max Reitz wrote: >> >> On 10.09.19 17:41, Peter Lieven wrote: >>> libnfs recently added support for unmounting. Add support >>> in Qemu too. >>> >>> Signed-off-by: Peter Lieven >>> --- >>> block/nfs.c | 3 +++ >>> 1

Re: [Qemu-block] [PATCH] blockdev: avoid acquiring AioContext lock twice at do_drive_backup()

2019-09-13 Thread Max Reitz
On 13.09.19 12:15, Sergio Lopez wrote: > > Max Reitz writes: > >> On 13.09.19 11:37, Sergio Lopez wrote: >>> >>> Max Reitz writes: >>> On 12.09.19 18:16, Sergio Lopez wrote: > do_drive_backup() acquires the AioContext lock of the corresponding > BlockDriverState. This is not a

Re: [Qemu-block] [PATCH v4 3/5] block/qcow2: refactor qcow2_co_preadv_part

2019-09-13 Thread Max Reitz
On 13.09.19 12:53, Vladimir Sementsov-Ogievskiy wrote: > 13.09.2019 13:01, Kevin Wolf wrote: >> Am 16.08.2019 um 17:30 hat Vladimir Sementsov-Ogievskiy geschrieben: >>> Further patch will run partial requests of iterations of >>> qcow2_co_preadv in parallel for performance reasons. To prepare for

Re: [Qemu-block] [PATCH v3 3/3] qemu-iotests: Add test for bz #1745922

2019-09-13 Thread Max Reitz
On 13.09.19 00:37, Maxim Levitsky wrote: > Signed-off-by: Maxim Levitsky > --- > tests/qemu-iotests/263 | 91 ++ > tests/qemu-iotests/263.out | 40 + > tests/qemu-iotests/group | 1 + > 3 files changed, 132 insertions(+) > create mode

[Qemu-block] [PATCH v2] virtio-blk: schedule virtio_notify_config to run on main context

2019-09-13 Thread Sergio Lopez
virtio_notify_config() needs to acquire the global mutex, which isn't allowed from an iothread, and may lead to a deadlock like this: - main thead * Has acquired: qemu_global_mutex. * Is trying the acquire: iothread AioContext lock via AIO_WAIT_WHILE (after aio_poll). - iothread *

Re: [Qemu-block] [PATCH v3 2/3] block/qcow2: fix the corruption when rebasing luks encrypted files

2019-09-13 Thread Max Reitz
On 13.09.19 00:37, Maxim Levitsky wrote: > This fixes subtle corruption introduced by luks threaded encryption > in commit 8ac0f15f335 > > Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1745922 > > The corruption happens when we do a write that >* writes to two or more unallocated

Re: [Qemu-block] [PATCH v4 3/5] block/qcow2: refactor qcow2_co_preadv_part

2019-09-13 Thread Vladimir Sementsov-Ogievskiy
13.09.2019 13:01, Kevin Wolf wrote: > Am 16.08.2019 um 17:30 hat Vladimir Sementsov-Ogievskiy geschrieben: >> Further patch will run partial requests of iterations of >> qcow2_co_preadv in parallel for performance reasons. To prepare for >> this, separate part which may be parallelized into

[Qemu-block] [PATCH 2/2] trace: Forbid event format ending with newline character

2019-09-13 Thread Philippe Mathieu-Daudé
Event format ending with newlines confuse the trace reports. Forbid them. Add a check to refuse new format added with trailing newline: $ make [...] GEN hw/misc/trace.h Traceback (most recent call last): File "scripts/tracetool.py", line 152, in main(sys.argv) File

[Qemu-block] [PATCH 1/2] trace: Remove trailing newline in events

2019-09-13 Thread Philippe Mathieu-Daudé
While the tracing frawework does not forbid trailing newline in events format string, using them lead to confuse output. It is the responsibility of the backend to properly end an event line. Some of our formats have trailing newlines, remove them. Signed-off-by: Philippe Mathieu-Daudé ---

[Qemu-block] [PATCH 0/2] trace: Forbid trailing newline in event format

2019-09-13 Thread Philippe Mathieu-Daudé
Hi Stefan, I'v been confused by trailing newline in trace reports, so this series aims to fix this, by cleaning current formats and add a check to catch new one introduced. Regards, Phil. Philippe Mathieu-Daudé (2): trace: Remove trailing newline in events trace: Forbid event format ending

Re: [Qemu-block] [PATCH v3 1/3] block/qcow2: refactoring of threaded encryption code

2019-09-13 Thread Max Reitz
On 13.09.19 00:37, Maxim Levitsky wrote: > This commit tries to clarify few function arguments, > and add comments describing the encrypt/decrypt interface > > Signed-off-by: Maxim Levitsky > --- > block/qcow2-cluster.c | 8 +++--- > block/qcow2-threads.c | 63

Re: [Qemu-block] [PATCH V2 1/2] block/nfs: tear down aio before nfs_close

2019-09-13 Thread Kevin Wolf
Am 13.09.2019 um 11:51 hat Max Reitz geschrieben: > On 10.09.19 17:41, Peter Lieven wrote: > > nfs_close is a sync call from libnfs and has its own event > > handler polling on the nfs FD. Avoid that both QEMU and libnfs > > are intefering here. > > > > CC: qemu-sta...@nongnu.org > >

Re: [Qemu-block] [PATCH] blockdev: avoid acquiring AioContext lock twice at do_drive_backup()

2019-09-13 Thread Sergio Lopez
Max Reitz writes: > On 13.09.19 11:37, Sergio Lopez wrote: >> >> Max Reitz writes: >> >>> On 12.09.19 18:16, Sergio Lopez wrote: do_drive_backup() acquires the AioContext lock of the corresponding BlockDriverState. This is not a problem when it's called from

Re: [Qemu-block] [PATCH V2 1/2] block/nfs: tear down aio before nfs_close

2019-09-13 Thread Peter Lieven
Am 13.09.19 um 11:51 schrieb Max Reitz: > On 10.09.19 17:41, Peter Lieven wrote: >> nfs_close is a sync call from libnfs and has its own event >> handler polling on the nfs FD. Avoid that both QEMU and libnfs >> are intefering here. >> >> CC: qemu-sta...@nongnu.org >> Signed-off-by: Peter Lieven

Re: [Qemu-block] [PATCH v4 3/5] block/qcow2: refactor qcow2_co_preadv_part

2019-09-13 Thread Kevin Wolf
Am 16.08.2019 um 17:30 hat Vladimir Sementsov-Ogievskiy geschrieben: > Further patch will run partial requests of iterations of > qcow2_co_preadv in parallel for performance reasons. To prepare for > this, separate part which may be parallelized into separate function > (qcow2_co_preadv_task). >

Re: [Qemu-block] [PATCH] blockdev: avoid acquiring AioContext lock twice at do_drive_backup()

2019-09-13 Thread Max Reitz
On 13.09.19 11:37, Sergio Lopez wrote: > > Max Reitz writes: > >> On 12.09.19 18:16, Sergio Lopez wrote: >>> do_drive_backup() acquires the AioContext lock of the corresponding >>> BlockDriverState. This is not a problem when it's called from >>> qmp_drive_backup(), but drive_backup_prepare()

Re: [Qemu-block] [PATCH V2 1/2] block/nfs: tear down aio before nfs_close

2019-09-13 Thread Max Reitz
On 10.09.19 17:41, Peter Lieven wrote: > nfs_close is a sync call from libnfs and has its own event > handler polling on the nfs FD. Avoid that both QEMU and libnfs > are intefering here. > > CC: qemu-sta...@nongnu.org > Signed-off-by: Peter Lieven > --- > block/nfs.c | 6 -- > 1 file

Re: [Qemu-block] [RFC PATCH] virtio-blk: schedule virtio_notify_config to run on main context

2019-09-13 Thread Kevin Wolf
Am 13.09.2019 um 11:28 hat Sergio Lopez geschrieben: > > Kevin Wolf writes: > > > Am 12.09.2019 um 21:51 hat Michael S. Tsirkin geschrieben: > >> On Thu, Sep 12, 2019 at 08:19:25PM +0200, Sergio Lopez wrote: > >> > Another AioContext-related issue, and this is a tricky one. > >> > > >> >

Re: [Qemu-block] [PATCH] blockdev: avoid acquiring AioContext lock twice at do_drive_backup()

2019-09-13 Thread Sergio Lopez
Max Reitz writes: > On 12.09.19 18:16, Sergio Lopez wrote: >> do_drive_backup() acquires the AioContext lock of the corresponding >> BlockDriverState. This is not a problem when it's called from >> qmp_drive_backup(), but drive_backup_prepare() also acquires the lock >> before calling it. >>

Re: [Qemu-block] [RFC PATCH] virtio-blk: schedule virtio_notify_config to run on main context

2019-09-13 Thread Sergio Lopez
Kevin Wolf writes: > Am 12.09.2019 um 21:51 hat Michael S. Tsirkin geschrieben: >> On Thu, Sep 12, 2019 at 08:19:25PM +0200, Sergio Lopez wrote: >> > Another AioContext-related issue, and this is a tricky one. >> > >> > Executing a QMP block_resize request for a virtio-blk device running >> >

Re: [Qemu-block] [PATCH v4 0/5] qcow2: async handling of fragmented io

2019-09-13 Thread Vladimir Sementsov-Ogievskiy
13.09.2019 11:58, Max Reitz wrote: > On 16.08.19 17:30, Vladimir Sementsov-Ogievskiy wrote: >> Hi all! >> >> Here is an asynchronous scheme for handling fragmented qcow2 >> reads and writes. Both qcow2 read and write functions loops through >> sequential portions of data. The series aim it to

Re: [Qemu-block] [RFC PATCH] virtio-blk: schedule virtio_notify_config to run on main context

2019-09-13 Thread Kevin Wolf
Am 12.09.2019 um 21:51 hat Michael S. Tsirkin geschrieben: > On Thu, Sep 12, 2019 at 08:19:25PM +0200, Sergio Lopez wrote: > > Another AioContext-related issue, and this is a tricky one. > > > > Executing a QMP block_resize request for a virtio-blk device running > > on an iothread may cause a

Re: [Qemu-block] [PATCH v4 0/5] qcow2: async handling of fragmented io

2019-09-13 Thread Max Reitz
On 16.08.19 17:30, Vladimir Sementsov-Ogievskiy wrote: > Hi all! > > Here is an asynchronous scheme for handling fragmented qcow2 > reads and writes. Both qcow2 read and write functions loops through > sequential portions of data. The series aim it to parallelize these > loops iterations. > It

Re: [Qemu-block] [PATCH] blockdev: avoid acquiring AioContext lock twice at do_drive_backup()

2019-09-13 Thread Max Reitz
On 12.09.19 18:16, Sergio Lopez wrote: > do_drive_backup() acquires the AioContext lock of the corresponding > BlockDriverState. This is not a problem when it's called from > qmp_drive_backup(), but drive_backup_prepare() also acquires the lock > before calling it. > > This change adds a

Re: [Qemu-block] [RFC PATCH] virtio-blk: schedule virtio_notify_config to run on main context

2019-09-13 Thread Sergio Lopez
Michael S. Tsirkin writes: > On Thu, Sep 12, 2019 at 08:19:25PM +0200, Sergio Lopez wrote: >> Another AioContext-related issue, and this is a tricky one. >> >> Executing a QMP block_resize request for a virtio-blk device running >> on an iothread may cause a deadlock involving the following