Re: [Qemu-block] [Qemu-devel] [PATCH v2 11/13] qcrypto-luks: refactoring: simplify the math used for keyslot locations

2019-09-12 Thread Maxim Levitsky
On Fri, 2019-09-06 at 14:17 +0100, Daniel P. Berrangé wrote: > On Mon, Aug 26, 2019 at 04:51:01PM +0300, Maxim Levitsky wrote: > > Signed-off-by: Maxim Levitsky > > --- > > crypto/block-luks.c | 64 + > > 1 file changed, 41 insertions(+), 23

Re: [Qemu-block] [Qemu-devel] [PATCH v2 09/13] qcrypto-block: extract check and parse header

2019-09-12 Thread Maxim Levitsky
On Fri, 2019-09-06 at 14:11 +0100, Daniel P. Berrangé wrote: > On Mon, Aug 26, 2019 at 04:50:59PM +0300, Maxim Levitsky wrote: > > This is just to make qcrypto_block_luks_open more > > reasonable in size. > > > > Signed-off-by: Maxim Levitsky > > --- > > crypto/block-luks.c | 254

Re: [Qemu-block] [PATCH] qcow2: Stop overwriting compressed clusters one by one

2019-09-12 Thread Alberto Garcia
On Thu 12 Sep 2019 01:33:05 AM CEST, John Snow wrote: >> This restriction comes from commit 095a9c58ce12afeeb90c2 from 2018. > > You accidentally typed a reasonably modern date. It's from *2008*! Oh my, and I reviewed the message 3 times ... if this one gets committed please correct the date.

Re: [Qemu-block] [PATCH] nbd/server: attach client channel to the export's AioContext

2019-09-12 Thread Sergio Lopez
Eric Blake writes: > On 9/11/19 12:21 PM, Eric Blake wrote: >> On 9/11/19 11:15 AM, Sergio Lopez wrote: >>> On creation, the export's AioContext is set to the same one as the >>> BlockBackend, while the AioContext in the client QIOChannel is left >>> untouched. >>> >>> As a result, when using

Re: [Qemu-block] [PATCH] block/create: Do not abort if a block driver is not available

2019-09-12 Thread Kevin Wolf
Am 12.09.2019 um 00:08 hat Philippe Mathieu-Daudé geschrieben: > The 'blockdev-create' QMP command was introduced as experimental > feature in commit b0292b851b8, using the assert() debug call. > It got promoted to 'stable' command in 3fb588a0f2c, but the > assert call was not removed. > > Some

[Qemu-block] [PATCH 02/12] qcrypto-luks: rename some fields in QCryptoBlockLUKSHeader

2019-09-12 Thread Maxim Levitsky
* key_bytes -> master_key_len * payload_offset = payload_offset_sector (to emphasise that this isn't byte offset) * key_offset -> key_offset_sector - same as above for luks slots Signed-off-by: Maxim Levitsky Reviewed-by: Daniel P. Berrangé --- crypto/block-luks.c | 91

[Qemu-block] [PATCH 05/12] qcrypto-luks: pass keyslot index rather that pointer to the keyslot

2019-09-12 Thread Maxim Levitsky
Another minor refactoring Signed-off-by: Maxim Levitsky Reviewed-by: Daniel P. Berrangé --- crypto/block-luks.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/crypto/block-luks.c b/crypto/block-luks.c index 9e59a791a6..b759cc8d19 100644 --- a/crypto/block-luks.c +++

[Qemu-block] [PATCH 12/12] qcrypto-luks: more rigorous header checking

2019-09-12 Thread Maxim Levitsky
Check that keyslots don't overlap with the data, and check that keyslots don't overlap with each other. (this is done using naive O(n^2) nested loops, but since there are just 8 keyslots, this doesn't really matter. Signed-off-by: Maxim Levitsky --- crypto/block-luks.c | 52

[Qemu-block] [PATCH 10/12] qcrypto-luks: extract store key function

2019-09-12 Thread Maxim Levitsky
This function will be used later to store new keys to the luks metadata Signed-off-by: Maxim Levitsky --- crypto/block-luks.c | 304 ++-- 1 file changed, 181 insertions(+), 123 deletions(-) diff --git a/crypto/block-luks.c b/crypto/block-luks.c index

Re: [Qemu-block] [Qemu-devel] [PATCH 05/10] block/crypto: implement the encryption key management

2019-09-12 Thread Maxim Levitsky
On Fri, 2019-09-06 at 15:04 +0100, Daniel P. Berrangé wrote: > On Fri, Aug 30, 2019 at 11:56:03PM +0300, Maxim Levitsky wrote: > > This implements the encryption key management > > using the generic code in qcrypto layer > > (currently only for qemu-img amend) > > > > This code adds another

Re: [Qemu-block] [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback

2019-09-12 Thread Kevin Wolf
Am 11.09.2019 um 13:00 hat Max Reitz geschrieben: > On 11.09.19 12:31, Kevin Wolf wrote: > > Am 11.09.2019 um 12:00 hat Max Reitz geschrieben: > >> So all in all I think it’s best to make the callback mandatory and add > >> two global helper functions. That’s simple enough and should prevent > >>

Re: [Qemu-block] [Qemu-devel] [PATCH v2 13/13] qcrypto-luks: implement more rigorous header checking

2019-09-12 Thread Maxim Levitsky
On Fri, 2019-09-06 at 14:34 +0100, Daniel P. Berrangé wrote: > On Mon, Aug 26, 2019 at 04:51:03PM +0300, Maxim Levitsky wrote: > > Check that keyslots don't overlap with the data, > > and check that keyslots don't overlap with each other. > > (this is done using naive O(n^2) nested loops, > > but

Re: [Qemu-block] [PATCH] nbd/server: attach client channel to the export's AioContext

2019-09-12 Thread Sergio Lopez
Kevin Wolf writes: > Am 11.09.2019 um 18:15 hat Sergio Lopez geschrieben: >> On creation, the export's AioContext is set to the same one as the >> BlockBackend, while the AioContext in the client QIOChannel is left >> untouched. >> >> As a result, when using data-plane,

Re: [Qemu-block] [Qemu-devel] [PATCH v2] util/hbitmap: strict hbitmap_reset

2019-09-12 Thread Vladimir Sementsov-Ogievskiy
11.09.2019 20:59, John Snow wrote: > > > On 9/11/19 11:13 AM, Vladimir Sementsov-Ogievskiy wrote: >> 07.08.2019 19:27, John Snow wrote: >>> >>> >>> On 8/6/19 12:19 PM, Vladimir Sementsov-Ogievskiy wrote: 06.08.2019 19:09, Max Reitz wrote: > On 06.08.19 17:26, Vladimir

[Qemu-block] [PATCH 06/12] qcrypto-luks: use the parsed encryption settings in QCryptoBlockLUKS

2019-09-12 Thread Maxim Levitsky
Prior to that patch, the parsed encryption settings were already stored into the QCryptoBlockLUKS but not used anywhere but in qcrypto_block_luks_get_info Using them simplifies the code Signed-off-by: Maxim Levitsky Reviewed-by: Daniel P. Berrangé --- crypto/block-luks.c | 169

[Qemu-block] [PATCH 09/12] qcrypto-luks: extract check and parse header

2019-09-12 Thread Maxim Levitsky
This is just to make qcrypto_block_luks_open more reasonable in size. Signed-off-by: Maxim Levitsky --- crypto/block-luks.c | 235 1 file changed, 127 insertions(+), 108 deletions(-) diff --git a/crypto/block-luks.c b/crypto/block-luks.c index

[Qemu-block] [PATCH 08/12] qcrypto-luks: extract store and load header

2019-09-12 Thread Maxim Levitsky
Signed-off-by: Maxim Levitsky --- crypto/block-luks.c | 158 ++-- 1 file changed, 94 insertions(+), 64 deletions(-) diff --git a/crypto/block-luks.c b/crypto/block-luks.c index ba63e9b442..c3f3488222 100644 --- a/crypto/block-luks.c +++

[Qemu-block] [PATCH 01/12] block-crypto: misc refactoring

2019-09-12 Thread Maxim Levitsky
* rename the write_func to create_write_func, and init_func to create_init_func this is preparation for other write_func that will be used to update the encryption keys. No functional changes Signed-off-by: Maxim Levitsky Reviewed-by: Daniel P. Berrangé --- block/crypto.c | 12

[Qemu-block] [PATCH 07/12] qcrypto-luks: purge unused error codes from open callback

2019-09-12 Thread Maxim Levitsky
These values are not used by generic crypto code anyway Signed-off-by: Maxim Levitsky --- crypto/block-luks.c | 26 +- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/crypto/block-luks.c b/crypto/block-luks.c index f3bfc921b2..ba63e9b442 100644 ---

Re: [Qemu-block] [PATCH] nbd/server: attach client channel to the export's AioContext

2019-09-12 Thread Sergio Lopez
Kevin Wolf writes: > Am 11.09.2019 um 23:33 hat Eric Blake geschrieben: >> On 9/11/19 12:21 PM, Eric Blake wrote: >> > On 9/11/19 11:15 AM, Sergio Lopez wrote: >> >> On creation, the export's AioContext is set to the same one as the >> >> BlockBackend, while the AioContext in the client

Re: [Qemu-block] [PATCH] nbd/server: attach client channel to the export's AioContext

2019-09-12 Thread Kevin Wolf
Am 12.09.2019 um 12:13 hat Sergio Lopez geschrieben: > > Kevin Wolf writes: > > > Am 11.09.2019 um 18:15 hat Sergio Lopez geschrieben: > >> On creation, the export's AioContext is set to the same one as the > >> BlockBackend, while the AioContext in the client QIOChannel is left > >> untouched.

Re: [Qemu-block] [PATCH] nbd/server: attach client channel to the export's AioContext

2019-09-12 Thread Kevin Wolf
Am 12.09.2019 um 12:30 hat Sergio Lopez geschrieben: > > Kevin Wolf writes: > > > Am 11.09.2019 um 23:33 hat Eric Blake geschrieben: > >> On 9/11/19 12:21 PM, Eric Blake wrote: > >> > On 9/11/19 11:15 AM, Sergio Lopez wrote: > >> >> On creation, the export's AioContext is set to the same one as

Re: [Qemu-block] [Qemu-devel] [PATCH v6 0/3] qcow2: add zstd cluster compression

2019-09-12 Thread Denis Plotnikov
ping! On 05.09.2019 12:31, Denis Plotnikov wrote: > v6: > * fixed zstd compressed length storing/loading [Eric] > * fixed wording, spec section placement [Eric] > > v5: > * type changed for compression_type at BDRVQcow2State [Kevin] > * fixed typos, grammar [Kevin] > * fixed default config zstd

[Qemu-block] [PATCH 11/12] qcrypto-luks: simplify the math used for keyslot locations

2019-09-12 Thread Maxim Levitsky
Signed-off-by: Maxim Levitsky Reviewed-by: Daniel P. Berrangé --- crypto/block-luks.c | 63 - 1 file changed, 40 insertions(+), 23 deletions(-) diff --git a/crypto/block-luks.c b/crypto/block-luks.c index c6045da33e..0d155c6614 100644 ---

Re: [Qemu-block] [Qemu-devel] [PATCH 03/10] qcrypto-luks: implement the encryption key management

2019-09-12 Thread Maxim Levitsky
On Fri, 2019-09-06 at 14:55 +0100, Daniel P. Berrangé wrote: > On Fri, Aug 30, 2019 at 11:56:01PM +0300, Maxim Levitsky wrote: > > Signed-off-by: Maxim Levitsky > > --- > > crypto/block-luks.c | 366 +++- > > 1 file changed, 364 insertions(+), 2

Re: [Qemu-block] [PATCH] qcow2: Stop overwriting compressed clusters one by one

2019-09-12 Thread Kevin Wolf
Am 12.09.2019 um 09:25 hat Alberto Garcia geschrieben: > On Thu 12 Sep 2019 01:33:05 AM CEST, John Snow wrote: > >> This restriction comes from commit 095a9c58ce12afeeb90c2 from 2018. > > > > You accidentally typed a reasonably modern date. It's from *2008*! > > Oh my, and I reviewed the message

Re: [Qemu-block] [Qemu-devel] [PATCH 04/10] block: amend: add 'force' option

2019-09-12 Thread Maxim Levitsky
On Fri, 2019-09-06 at 14:59 +0100, Daniel P. Berrangé wrote: > On Fri, Aug 30, 2019 at 11:56:02PM +0300, Maxim Levitsky wrote: > > This could do with some text to explain what this will be > used for. I actually added an explanation to the man page " +--force allows some unsafe operations.

Re: [Qemu-block] [PATCH] nbd/server: attach client channel to the export's AioContext

2019-09-12 Thread Kevin Wolf
Am 11.09.2019 um 23:33 hat Eric Blake geschrieben: > On 9/11/19 12:21 PM, Eric Blake wrote: > > On 9/11/19 11:15 AM, Sergio Lopez wrote: > >> On creation, the export's AioContext is set to the same one as the > >> BlockBackend, while the AioContext in the client QIOChannel is left > >> untouched.

Re: [Qemu-block] [PATCH] nbd/server: attach client channel to the export's AioContext

2019-09-12 Thread Kevin Wolf
Am 11.09.2019 um 18:15 hat Sergio Lopez geschrieben: > On creation, the export's AioContext is set to the same one as the > BlockBackend, while the AioContext in the client QIOChannel is left > untouched. > > As a result, when using data-plane, nbd_client_receive_next_request() > schedules

[Qemu-block] [PATCH 00/12] crypto/luks: preparation for encryption key managment

2019-09-12 Thread Maxim Levitsky
Hi! This patch series is the refactoring/preparation part of the former patch series I had sent which adds support for luks key management. This series includes all the feedback from the last review iteration and one new patch that removes errno values from .open callback of luks crypto driver

[Qemu-block] [PATCH 04/12] qcrypto-luks: simplify masterkey and masterkey length

2019-09-12 Thread Maxim Levitsky
Let the caller allocate masterkey Always use master key len from the header Signed-off-by: Maxim Levitsky Reviewed-by: Daniel P. Berrangé --- crypto/block-luks.c | 44 +--- 1 file changed, 21 insertions(+), 23 deletions(-) diff --git

[Qemu-block] [PATCH 03/12] qcrypto-luks: don't overwrite cipher_mode in header

2019-09-12 Thread Maxim Levitsky
This way we can store the header we loaded, which will be used in key management code Signed-off-by: Maxim Levitsky Reviewed-by: Daniel P. Berrangé --- crypto/block-luks.c | 9 ++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/crypto/block-luks.c b/crypto/block-luks.c

[Qemu-block] [PATCH v2] nbd/server: attach client channel to the export's AioContext

2019-09-12 Thread Sergio Lopez
On creation, the export's AioContext is set to the same one as the BlockBackend, while the AioContext in the client QIOChannel is left untouched. As a result, when using data-plane, nbd_client_receive_next_request() schedules coroutines in the IOThread AioContext, while the client's QIOChannel is

[Qemu-block] [PULL 12/22] iotests: skip 232 when run tests as root

2019-09-12 Thread Kevin Wolf
From: Vladimir Sementsov-Ogievskiy chmod a-w don't help under root, so skip the test in such case. Signed-off-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Kevin Wolf --- tests/qemu-iotests/232 | 6 ++ 1 file changed, 6 insertions(+) diff --git a/tests/qemu-iotests/232

[Qemu-block] [PULL 13/22] block/nfs: add support for nfs_umount

2019-09-12 Thread Kevin Wolf
From: Peter Lieven libnfs recently added support for unmounting. Add support in Qemu too. Signed-off-by: Peter Lieven Signed-off-by: Kevin Wolf --- block/nfs.c | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/block/nfs.c b/block/nfs.c index 0ec50953e4..9d30963fd8

[Qemu-block] [PULL 17/22] iotests: Valgrind fails with nonexistent directory

2019-09-12 Thread Kevin Wolf
From: Andrey Shinkevich The Valgrind uses the exported variable TMPDIR and fails if the directory does not exist. Let us exclude such a test case from being run under the Valgrind and notify the user of it. Suggested-by: Kevin Wolf Signed-off-by: Andrey Shinkevich Reviewed-by: John Snow

[Qemu-block] [PULL 21/22] block/create: Do not abort if a block driver is not available

2019-09-12 Thread Kevin Wolf
From: Philippe Mathieu-Daudé The 'blockdev-create' QMP command was introduced as experimental feature in commit b0292b851b8, using the assert() debug call. It got promoted to 'stable' command in 3fb588a0f2c, but the assert call was not removed. Some block drivers are optional, and

[Qemu-block] [PULL 19/22] iotests: extend sleeping time under Valgrind

2019-09-12 Thread Kevin Wolf
From: Andrey Shinkevich To synchronize the time when QEMU is running longer under the Valgrind, increase the sleeping time in the test 247. Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: John Snow Signed-off-by: Kevin Wolf --- tests/qemu-iotests/247

[Qemu-block] [PULL 08/22] iotests: Add supported protocols to execute_test()

2019-09-12 Thread Kevin Wolf
From: Max Reitz Signed-off-by: Max Reitz Signed-off-by: Kevin Wolf --- tests/qemu-iotests/iotests.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 84438e837c..b26271187c 100644 ---

[Qemu-block] [PULL 15/22] iotests: exclude killed processes from running under Valgrind

2019-09-12 Thread Kevin Wolf
From: Andrey Shinkevich The Valgrind tool fails to manage its termination in multi-threaded processes when they raise the signal SIGKILL. The bug has been reported to the Valgrind maintainers and was registered as the bug #409141: https://bugs.kde.org/show_bug.cgi?id=409141 Let's exclude

[Qemu-block] [PULL 16/22] iotests: Add casenotrun report to bash tests

2019-09-12 Thread Kevin Wolf
From: Andrey Shinkevich The new function _casenotrun() is to be invoked if a test case cannot be run for some reason. The user will be notified by a message passed to the function. It is the caller's responsibility to make skipped a particular test. Suggested-by: Kevin Wolf Signed-off-by:

Re: [Qemu-block] [Qemu-devel] [PATCH 0/3] proper locking on bitmap add/remove paths

2019-09-12 Thread Vladimir Sementsov-Ogievskiy
12.09.2019 0:46, John Snow wrote: > > > On 9/11/19 10:09 AM, Vladimir Sementsov-Ogievskiy wrote: >> 10.09.2019 23:37, no-re...@patchew.org wrote: >>> Patchew URL: >>> https://patchew.org/QEMU/20190910162724.79574-1-vsement...@virtuozzo.com/ >>> >>> >>> >>> Hi, >>> >>> This series failed the

[Qemu-block] [PULL 11/22] iotests: Test blockdev-create for vpc

2019-09-12 Thread Kevin Wolf
From: Max Reitz Signed-off-by: Max Reitz Signed-off-by: Kevin Wolf --- tests/qemu-iotests/266 | 153 + tests/qemu-iotests/266.out | 137 + tests/qemu-iotests/group | 1 + 3 files changed, 291 insertions(+) create

[Qemu-block] [PULL 10/22] iotests: Restrict nbd Python tests to nbd

2019-09-12 Thread Kevin Wolf
From: Max Reitz We have two Python unittest-style tests that test NBD. As such, they should specify supported_protocols=['nbd'] so they are skipped when the user wants to test some other protocol. Furthermore, we should restrict their choice of formats to 'raw'. The idea of a protocol/format

[Qemu-block] [PULL 09/22] iotests: Restrict file Python tests to file

2019-09-12 Thread Kevin Wolf
From: Max Reitz Most of our Python unittest-style tests only support the file protocol. You can run them with any other protocol, but the test will simply ignore your choice and use file anyway. We should let them signal that they require the file protocol so they are skipped when you want to

[Qemu-block] [PULL 01/22] qcow2: Fix the calculation of the maximum L2 cache size

2019-09-12 Thread Kevin Wolf
From: Alberto Garcia The size of the qcow2 L2 cache defaults to 32 MB, which can be easily larger than the maximum amount of L2 metadata that the image can have. For example: with 64 KB clusters the user would need a qcow2 image with a virtual size of 256 GB in order to have 32 MB of L2

[Qemu-block] [PULL 00/22] Block layer patches

2019-09-12 Thread Kevin Wolf
The following changes since commit 89ea03a7dc83ca36b670ba7f787802791fcb04b1: Merge remote-tracking branch 'remotes/huth-gitlab/tags/m68k-pull-2019-09-07' into staging (2019-09-09 09:48:34 +0100) are available in the Git repository at: git://repo.or.cz/qemu/kevin.git tags/for-upstream for

[Qemu-block] [PULL 06/22] file-posix: Fix has_write_zeroes after NO_FALLBACK

2019-09-12 Thread Kevin Wolf
If QEMU_AIO_NO_FALLBACK is given, we always return failure and don't even try to use the BLKZEROOUT ioctl. In this failure case, we shouldn't disable has_write_zeroes because we didn't learn anything about the ioctl. The next request might not set QEMU_AIO_NO_FALLBACK and we can still use the

[Qemu-block] [PATCH 4/4] iotests: Add test for failing mirror complete

2019-09-12 Thread Max Reitz
Signed-off-by: Max Reitz --- tests/qemu-iotests/041 | 44 ++ tests/qemu-iotests/041.out | 4 ++-- 2 files changed, 46 insertions(+), 2 deletions(-) diff --git a/tests/qemu-iotests/041 b/tests/qemu-iotests/041 index 8568426311..84bc6d6581 100755 ---

[Qemu-block] [PULL 14/22] iotests: allow Valgrind checking all QEMU processes

2019-09-12 Thread Kevin Wolf
From: Andrey Shinkevich With the '-valgrind' option, let all the QEMU processes be run under the Valgrind tool. The Valgrind own parameters may be set with its environment variable VALGRIND_OPTS, e.g. $ VALGRIND_OPTS="--leak-check=yes" ./check -valgrind or they may be listed in the Valgrind

[Qemu-block] [PATCH 0/4] mirror: Do not dereference invalid pointers

2019-09-12 Thread Max Reitz
Hi, The fix (patch 1) is pretty straightforward; patch 2 (which I need for the test) may not be. The biggest problem with patch 2 is that you can use it to uncover where our permission handling is broken. For example, devising the test case (patch 4) was very difficult because I kept running

[Qemu-block] [PATCH 2/4] blkdebug: Allow taking/unsharing permissions

2019-09-12 Thread Max Reitz
Sometimes it is useful to be able to add a node to the block graph that takes or unshare a certain set of permissions for debugging purposes. This patch adds this capability to blkdebug. (Note that you cannot make blkdebug release or share permissions that it needs to take or cannot share,

[Qemu-block] [PATCH 3/4] iotests: Add @error to wait_until_completed

2019-09-12 Thread Max Reitz
Callers can use this new parameter to expect failure during the completion process. Signed-off-by: Max Reitz --- tests/qemu-iotests/iotests.py | 18 -- 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py

[Qemu-block] [PATCH 1/4] mirror: Do not dereference invalid pointers

2019-09-12 Thread Max Reitz
mirror_exit_common() may be called twice (if it is called from mirror_prepare() and fails, it will be called from mirror_abort() again). In such a case, many of the pointers in the MirrorBlockJob object will already be freed. This can be seen most reliably for s->target, which is set to NULL

[Qemu-block] [PULL 22/22] qcow2: Stop overwriting compressed clusters one by one

2019-09-12 Thread Kevin Wolf
From: Alberto Garcia handle_alloc() tries to find as many contiguous clusters that need copy-on-write as possible in order to allocate all of them at the same time. However, compressed clusters are only overwritten one by one, so let's say that we have an image with 1024 consecutive compressed

[Qemu-block] [PULL 20/22] qemu-io: Don't leak pattern file in error path

2019-09-12 Thread Kevin Wolf
qemu_io_alloc_from_file() needs to close the pattern file even if some error occurred. Setting f = NULL in the success path and checking it for NULL in the error path isn't strictly necessary at this point, but let's do it anyway in case someone later adds a 'goto error' after closing the file.

[Qemu-block] [PULL 02/22] job: drop job_drain

2019-09-12 Thread Kevin Wolf
From: Vladimir Sementsov-Ogievskiy In job_finish_sync job_enter should be enough for a job to make some progress and draining is a wrong tool for it. So use job_enter directly here and drop job_drain with all related staff not used more. Suggested-by: Kevin Wolf Signed-off-by: Vladimir

[Qemu-block] [PULL 04/22] iotests: Test reverse sub-cluster qcow2 writes

2019-09-12 Thread Kevin Wolf
From: Max Reitz This exercises the regression introduced in commit 50ba5b2d994853b38fed10e0841b119da0f8b8e5. On my machine, it has close to a 50 % false-negative rate, but that should still be sufficient to test the fix. Signed-off-by: Max Reitz Reviewed-by: Stefano Garzarella Reviewed-by:

[Qemu-block] [PULL 03/22] block/file-posix: Reduce xfsctl() use

2019-09-12 Thread Kevin Wolf
From: Max Reitz This patch removes xfs_write_zeroes() and xfs_discard(). Both functions have been added just before the same feature was present through fallocate(): - fallocate() has supported PUNCH_HOLE for XFS since Linux 2.6.38 (March 2011); xfs_discard() was added in December 2010. -

[Qemu-block] [PULL 05/22] pr-manager: Fix invalid g_free() crash bug

2019-09-12 Thread Kevin Wolf
From: Markus Armbruster pr_manager_worker() passes its @opaque argument to g_free(). Wrong; it points to pr_manager_worker()'s automatic @data. Broken when commit 2f3a7ab39be converted @data from heap- to stack-allocated. Fix by deleting the g_free(). Fixes:

[Qemu-block] [PULL 07/22] vpc: Return 0 from vpc_co_create() on success

2019-09-12 Thread Kevin Wolf
From: Max Reitz blockdev_create_run() directly uses .bdrv_co_create()'s return value as the job's return value. Jobs must return 0 on success, not just any nonnegative value. Therefore, using blockdev-create for VPC images may currently fail as the vpc driver may return a positive integer.

[Qemu-block] [PULL 18/22] iotests: extended timeout under Valgrind

2019-09-12 Thread Kevin Wolf
From: Andrey Shinkevich As the iotests run longer under the Valgrind, the QEMU_COMM_TIMEOUT is to be increased in the test cases 028, 183 and 192 when running under the Valgrind. Suggested-by: Roman Kagan Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by:

[Qemu-block] [PATCH 4/4] Revert "mirror: Only mirror granularity-aligned chunks"

2019-09-12 Thread Vladimir Sementsov-Ogievskiy
This reverts commit 9adc1cb49af8d4e54f57980b1eed5c0a4b2dafa6. "mirror: Only mirror granularity-aligned chunks" Since previous commit unaligned chunks are supported by do_sync_target_write. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/mirror.c | 29 - 1

[Qemu-block] [PATCH 1/4] block/mirror: simplify do_sync_target_write

2019-09-12 Thread Vladimir Sementsov-Ogievskiy
do_sync_target_write is called from bdrv_mirror_top_do_write after write/discard operation, all inside active_write/active_write_settle protecting us from mirror iteration. So the whole area is dirty for sure, no reason to examine dirty bitmap. Signed-off-by: Vladimir Sementsov-Ogievskiy ---

[Qemu-block] [PATCH 0/4] active-mirror: support unaligned guest operations

2019-09-12 Thread Vladimir Sementsov-Ogievskiy
Commit 9adc1cb49af8d fixed a bug about unaligned (to dirty bitmap granularity) guest writes (and discards) by simply requesting corresponding alignment on mirror-top filter. However forcing large alignment obviously decreases performance of unaligned requests. So it's time for a new solution

[Qemu-block] [PATCH 3/4] block/mirror: support unaligned write in active mirror

2019-09-12 Thread Vladimir Sementsov-Ogievskiy
Prior 9adc1cb49af8d do_sync_target_write had a bug: it reset aligned-up region in the dirty bitmap, which means that we may not copy some bytes and assume them copied, which actually leads to producing corrupted target. So 9adc1cb49af8d forced dirty bitmap granularity to be request_alignment for

[Qemu-block] [PATCH 2/4] block/block-backend: add blk_co_pwritev_part

2019-09-12 Thread Vladimir Sementsov-Ogievskiy
Add blk write function with qiov_offset parameter. It's needed for the following commit. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/sysemu/block-backend.h | 4 block/block-backend.c | 17 + 2 files changed, 17 insertions(+), 4 deletions(-) diff --git

[Qemu-block] [PATCH] blockdev: avoid acquiring AioContext lock twice at do_drive_backup()

2019-09-12 Thread Sergio Lopez
do_drive_backup() acquires the AioContext lock of the corresponding BlockDriverState. This is not a problem when it's called from qmp_drive_backup(), but drive_backup_prepare() also acquires the lock before calling it. This change adds a BlockDriverState argument to do_drive_backup(), which is

[Qemu-block] [RFC PATCH] virtio-blk: schedule virtio_notify_config to run on main context

2019-09-12 Thread Sergio Lopez
Another AioContext-related issue, and this is a tricky one. Executing a QMP block_resize request for a virtio-blk device running on an iothread may cause a deadlock involving the following mutexes: - main thead * Has acquired: qemu_mutex_global. * Is trying the acquire: iothread AioContext

Re: [Qemu-block] [Qemu-devel] [PATCH 08/10] block/crypto: implement blockdev-amend

2019-09-12 Thread Maxim Levitsky
On Fri, 2019-09-06 at 15:10 +0100, Daniel P. Berrangé wrote: > On Fri, Aug 30, 2019 at 11:56:06PM +0300, Maxim Levitsky wrote: > > Signed-off-by: Maxim Levitsky > > --- > > block/crypto.c | 86 +--- > > qapi/block-core.json | 4 +-- > > 2 files

Re: [Qemu-block] [Qemu-devel] [PATCH 09/10] block/qcow2: implement blockdev-amend

2019-09-12 Thread Maxim Levitsky
On Fri, 2019-09-06 at 15:12 +0100, Daniel P. Berrangé wrote: > On Fri, Aug 30, 2019 at 11:56:07PM +0300, Maxim Levitsky wrote: > > Currently only for changing crypto parameters > > > > Signed-off-by: Maxim Levitsky > > --- > > block/qcow2.c| 71

Re: [Qemu-block] [RFC PATCH] virtio-blk: schedule virtio_notify_config to run on main context

2019-09-12 Thread Michael S. Tsirkin
On Thu, Sep 12, 2019 at 08:19:25PM +0200, Sergio Lopez wrote: > Another AioContext-related issue, and this is a tricky one. > > Executing a QMP block_resize request for a virtio-blk device running > on an iothread may cause a deadlock involving the following mutexes: > > - main thead > * Has

Re: [Qemu-block] [Qemu-devel] [PATCH 06/10] qcow2: implement crypto amend options

2019-09-12 Thread Maxim Levitsky
On Fri, 2019-09-06 at 15:06 +0100, Daniel P. Berrangé wrote: > On Fri, Aug 30, 2019 at 11:56:04PM +0300, Maxim Levitsky wrote: > > --- > > block/qcow2.c | 79 --- > > 1 file changed, 63 insertions(+), 16 deletions(-) > > > > @@ -4888,9 +4899,22 @@

[Qemu-block] [PATCH v2 02/11] qcrypto-luks: extend the create options for upcoming encryption key management

2019-09-12 Thread Maxim Levitsky
Now you can specify which slot to put the encryption key to Plus add 'active' option which will let user erase the key secret instead of adding it. Check that active=true it when creating. Signed-off-by: Maxim Levitsky --- block/crypto.c | 2 ++ block/crypto.h | 16

[Qemu-block] [PATCH v2 06/11] qcow2: implement crypto amend options

2019-09-12 Thread Maxim Levitsky
Signed-off-by: Maxim Levitsky --- block/qcow2.c | 77 +-- 1 file changed, 62 insertions(+), 15 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index 0618a63793..26f83aeb44 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -172,6 +172,25 @@

[Qemu-block] [PATCH v2 07/11] block: add x-blockdev-amend qmp command

2019-09-12 Thread Maxim Levitsky
Signed-off-by: Maxim Levitsky --- block/Makefile.objs | 2 +- block/amend.c | 116 ++ include/block/block_int.h | 23 ++-- qapi/block-core.json | 26 + qapi/job.json | 4 +- 5 files changed, 163

[Qemu-block] [PATCH v2 08/11] block/crypto: implement blockdev-amend

2019-09-12 Thread Maxim Levitsky
Signed-off-by: Maxim Levitsky Reviewed-by: Daniel P. Berrangé --- block/crypto.c | 85 ++-- qapi/block-core.json | 7 ++-- 2 files changed, 71 insertions(+), 21 deletions(-) diff --git a/block/crypto.c b/block/crypto.c index f42fa057e6..5905f7f520

[Qemu-block] [PATCH v3 3/3] qemu-iotests: Add test for bz #1745922

2019-09-12 Thread Maxim Levitsky
Signed-off-by: Maxim Levitsky --- tests/qemu-iotests/263 | 91 ++ tests/qemu-iotests/263.out | 40 + tests/qemu-iotests/group | 1 + 3 files changed, 132 insertions(+) create mode 100755 tests/qemu-iotests/263 create mode 100644

[Qemu-block] [PATCH v2 05/11] block/crypto: implement the encryption key management

2019-09-12 Thread Maxim Levitsky
This implements the encryption key management using the generic code in qcrypto layer (currently only for qemu-img amend) This code adds another 'write_func' because the initialization write_func works directly on the underlying file, because during the creation, there is no open instance of the

[Qemu-block] [PATCH v2 09/11] block/qcow2: implement blockdev-amend

2019-09-12 Thread Maxim Levitsky
Currently only for changing crypto parameters Signed-off-by: Maxim Levitsky --- block/qcow2.c| 71 qapi/block-core.json | 6 ++-- 2 files changed, 75 insertions(+), 2 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index

[Qemu-block] [PATCH v3 2/3] block/qcow2: fix the corruption when rebasing luks encrypted files

2019-09-12 Thread Maxim Levitsky
This fixes subtle corruption introduced by luks threaded encryption in commit 8ac0f15f335 Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1745922 The corruption happens when we do a write that * writes to two or more unallocated clusters at once * doesn't fully cover the first sector

[Qemu-block] [PATCH v2 00/11] RFC crypto/luks: encryption key managment using amend interface

2019-09-12 Thread Maxim Levitsky
This patch series is continuation of my work to add encryption key managment to luks/qcow2 with luks. This is second version of this patch set. The changes are mostly addressing the review feedback, plus I tested (and fixed sadly) the somewhat ugly code that allows to still write share a raw luks

[Qemu-block] [PATCH v2 04/11] block: amend: add 'force' option

2019-09-12 Thread Maxim Levitsky
'force' optinion will be used for some unsafe option amend operations. This includes things like erasing last keyslot in luks (which pretty much guarantees destroying the data, unless the master key is backed up by extrnal means, but that _might_ be desired result) Signed-off-by: Maxim

[Qemu-block] [PATCH v2 03/11] qcrypto-luks: implement the encryption key management

2019-09-12 Thread Maxim Levitsky
Signed-off-by: Maxim Levitsky --- crypto/block-luks.c | 356 +++- 1 file changed, 354 insertions(+), 2 deletions(-) diff --git a/crypto/block-luks.c b/crypto/block-luks.c index fed80e6646..26ce50b111 100644 --- a/crypto/block-luks.c +++

[Qemu-block] [PATCH v2 01/11] qcrypto: add suport for amend options

2019-09-12 Thread Maxim Levitsky
This adds the qcrypto_amend_options and corresponding crypto driver callbacks for the for encrypted key managedment Signed-off-by: Maxim Levitsky Reviewed-by: Daniel P. Berrangé --- crypto/block.c | 31 +++ crypto/blockpriv.h | 8

[Qemu-block] [PATCH v2 10/11] iotests: filter few more luks specific create options

2019-09-12 Thread Maxim Levitsky
Those options are test input anyway, and this allows more tests to be able to have same output on both qcow2 luks encrypted images and raw luks images Signed-off-by: Maxim Levitsky --- tests/qemu-iotests/087.out | 6 +++--- tests/qemu-iotests/134.out | 2 +-

[Qemu-block] [PATCH v2 11/11] iotests : add tests for encryption key management

2019-09-12 Thread Maxim Levitsky
Note that currently I add tests 300-302, which are placeholders to ease the rebase. In final version of these patches I will update these. Signed-off-by: Maxim Levitsky --- tests/qemu-iotests/300 | 202 + tests/qemu-iotests/300.out | 98 +++

[Qemu-block] [PATCH v3 0/3] Fix qcow2+luks corruption introduced by commit 8ac0f15f335

2019-09-12 Thread Maxim Levitsky
Commit 8ac0f15f335 accidently broke the COW of non changed areas of newly allocated clusters, when the write spans multiple clusters, and needs COW both prior and after the write. This results in 'after' COW area being encrypted with wrong sector address, which render it corrupted. Bugzilla:

Re: [Qemu-block] [Qemu-devel] [PATCH 1/2] LUKS: better error message when creating too large files

2019-09-12 Thread Maxim Levitsky
On Mon, 2019-07-22 at 10:05 +0100, Daniel P. Berrangé wrote: > On Sun, Jul 21, 2019 at 09:15:07PM +0300, Maxim Levitsky wrote: > > Currently if you attampt to create too large file with luks you > > get the following error message: > > > > Formatting 'test.luks', fmt=luks size=17592186044416

[Qemu-block] [PATCH v3 1/3] block/qcow2: refactoring of threaded encryption code

2019-09-12 Thread Maxim Levitsky
This commit tries to clarify few function arguments, and add comments describing the encrypt/decrypt interface Signed-off-by: Maxim Levitsky --- block/qcow2-cluster.c | 8 +++--- block/qcow2-threads.c | 63 ++- 2 files changed, 54 insertions(+), 17