Re: [PATCH v8 02/10] scripts: add coccinelle script to use auto propagated errp

2020-03-11 Thread Vladimir Sementsov-Ogievskiy
10.03.2020 18:47, Markus Armbruster wrote: Vladimir Sementsov-Ogievskiy writes: 09.03.2020 12:56, Markus Armbruster wrote: Suggest scripts: Coccinelle script to use auto-propagated errp or scripts: Coccinelle script to use ERRP_AUTO_PROPAGATE() Vladimir Sementsov-Ogievskiy

Re: [PATCH v5 0/5] qcow2: Implement zstd cluster compression method

2020-03-11 Thread Denis Plotnikov
ping! Is there any other comments/concerns/objections/suggestions according to the series except the minor ones from Alberto and Vladimir? If not, please, let me know, so I can resend the series with the minor changes for applying to the corresponding branch. Thanks! Denis On 04.03.2020

Re: [PATCH v4 00/10] Further bitmaps improvements

2020-03-11 Thread Vladimir Sementsov-Ogievskiy
10.03.2020 20:17, Max Reitz wrote: On 06.03.20 08:45, Vladimir Sementsov-Ogievskiy wrote: 26.02.2020 16:13, Max Reitz wrote: On 05.02.20 12:20, Vladimir Sementsov-Ogievskiy wrote: Hi! The main feature here is improvement of _next_dirty_area API, which I'm going to use then for backup /

RE: [PATCH] hw/scsi/megasas:Clean up some redundant code fix Clang warnings

2020-03-11 Thread Chenqun (kuhn)
>-Original Message- >From: Laurent Vivier [mailto:laur...@vivier.eu] >Sent: Tuesday, March 10, 2020 11:01 PM >To: Chenqun (kuhn) ; qemu- >de...@nongnu.org; qemu-triv...@nongnu.org >Cc: Fam Zheng ; Hannes Reinecke ; >Zhanghailiang ; qemu-block@nongnu.org; >Euler Robot ; Paolo Bonzini >

Re: [PATCH v4 00/10] Further bitmaps improvements

2020-03-11 Thread Vladimir Sementsov-Ogievskiy
11.03.2020 20:03, John Snow wrote: On 3/11/20 9:58 AM, Vladimir Sementsov-Ogievskiy wrote: 11.03.2020 12:55, Max Reitz wrote: On 11.03.20 07:17, Vladimir Sementsov-Ogievskiy wrote: 10.03.2020 20:17, Max Reitz wrote: On 06.03.20 08:45, Vladimir Sementsov-Ogievskiy wrote: 26.02.2020 16:13,

[PULL 4/9] aio-posix: move RCU_READ_LOCK() into run_poll_handlers()

2020-03-11 Thread Stefan Hajnoczi
Now that run_poll_handlers_once() is only called by run_poll_handlers() we can improve the CPU time profile by moving the expensive RCU_READ_LOCK() out of the polling loop. This reduces the run_poll_handlers() from 40% CPU to 10% CPU in perf's sampling profiler output. Signed-off-by: Stefan

[PULL 7/9] aio-posix: add io_uring fd monitoring implementation

2020-03-11 Thread Stefan Hajnoczi
The recent Linux io_uring API has several advantages over ppoll(2) and epoll(2). Details are given in the source code. Add an io_uring implementation and make it the default on Linux. Performance is the same as with epoll(7) but later patches add optimizations that take advantage of io_uring.

Re: [PULL 0/9] Block patches

2020-03-11 Thread no-reply
Patchew URL: https://patchew.org/QEMU/20200311124045.277969-1-stefa...@redhat.com/ Hi, This series seems to have some coding style problems. See output below for more information: Subject: [PULL 0/9] Block patches Message-id: 20200311124045.277969-1-stefa...@redhat.com Type: series === TEST

[PULL 00/19] Block patches

2020-03-11 Thread Max Reitz
The following changes since commit ba29883206d92a29ad5a466e679ccfc2ee6132ef: Merge remote-tracking branch 'remotes/borntraeger/tags/s390x-20200310' into staging (2020-03-10 16:50:28 +) are available in the Git repository at: https://github.com/XanClic/qemu.git

Re: [PATCH v4 00/10] Further bitmaps improvements

2020-03-11 Thread Vladimir Sementsov-Ogievskiy
11.03.2020 12:55, Max Reitz wrote: On 11.03.20 07:17, Vladimir Sementsov-Ogievskiy wrote: 10.03.2020 20:17, Max Reitz wrote: On 06.03.20 08:45, Vladimir Sementsov-Ogievskiy wrote: 26.02.2020 16:13, Max Reitz wrote: On 05.02.20 12:20, Vladimir Sementsov-Ogievskiy wrote: Hi! The main feature

Re: [PATCH v8 02/10] scripts: add coccinelle script to use auto propagated errp

2020-03-11 Thread Vladimir Sementsov-Ogievskiy
11.03.2020 17:41, Markus Armbruster wrote: Vladimir Sementsov-Ogievskiy writes: 11.03.2020 12:38, Markus Armbruster wrote: Vladimir Sementsov-Ogievskiy writes: 09.03.2020 12:56, Markus Armbruster wrote: Suggest scripts: Coccinelle script to use auto-propagated errp or

[PULL 6/9] aio-posix: simplify FDMonOps->update() prototype

2020-03-11 Thread Stefan Hajnoczi
The AioHandler *node, bool is_new arguments are more complicated to think about than simply being given AioHandler *old_node, AioHandler *new_node. Furthermore, the new Linux io_uring file descriptor monitoring mechanism added by the new patch requires access to both the old and the new nodes.

[PULL 07/19] iotests: Fix nonportable use of od --endian

2020-03-11 Thread Max Reitz
From: Eric Blake Tests 261 and 272 fail on RHEL 7 with coreutils 8.22, since od --endian was not added until coreutils 8.23. Fix this by manually constructing the final value one byte at a time. Fixes: fc8ba423 Reported-by: Andrey Shinkevich Signed-off-by: Eric Blake Reviewed-by: Max Reitz

[PULL 03/19] qemu-img: allow qemu-img measure --object without a filename

2020-03-11 Thread Max Reitz
From: Stefan Hajnoczi In most qemu-img sub-commands the --object option only makes sense when there is a filename. qemu-img measure is an exception because objects may be referenced from the image creation options instead of an existing image file. Allow --object without a filename.

[PULL 04/19] iotests: add 288 luks qemu-img measure test

2020-03-11 Thread Max Reitz
From: Stefan Hajnoczi This test exercises the block/crypto.c "luks" block driver .bdrv_measure() code. Signed-off-by: Stefan Hajnoczi Reviewed-by: Max Reitz Message-Id: <20200221112522.1497712-5-stefa...@redhat.com> [mreitz: Renamed test from 282 to 288] Signed-off-by: Max Reitz ---

[PULL 08/19] block/qcow2: do free crypto_opts in qcow2_close()

2020-03-11 Thread Max Reitz
From: Pan Nengyuan 'crypto_opts' forgot to free in qcow2_close(), this patch fix the bellow leak stack: Direct leak of 24 byte(s) in 1 object(s) allocated from: #0 0x7f0edd81f970 in __interceptor_calloc (/lib64/libasan.so.5+0xef970) #1 0x7f0edc6d149d in g_malloc0

[PULL 05/19] block/curl: HTTP header fields allow whitespace around values

2020-03-11 Thread Max Reitz
From: David Edmondson RFC 7230 section 3.2 indicates that whitespace is permitted between the field name and field value and after the field value. Signed-off-by: David Edmondson Message-Id: <20200224101310.101169-2-david.edmond...@oracle.com> Reviewed-by: Max Reitz Signed-off-by: Max Reitz

[PULL 11/19] job: refactor progress to separate object

2020-03-11 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy We need it in separate to pass to the block-copy object in the next commit. Cc: qemu-sta...@nongnu.org Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Andrey Shinkevich Reviewed-by: Max Reitz Message-Id:

Re: [PATCH v8 02/10] scripts: add coccinelle script to use auto propagated errp

2020-03-11 Thread Vladimir Sementsov-Ogievskiy
11.03.2020 12:38, Markus Armbruster wrote: Vladimir Sementsov-Ogievskiy writes: 09.03.2020 12:56, Markus Armbruster wrote: Suggest scripts: Coccinelle script to use auto-propagated errp or scripts: Coccinelle script to use ERRP_AUTO_PROPAGATE() Vladimir Sementsov-Ogievskiy

Re: [PATCH v8 02/10] scripts: add coccinelle script to use auto propagated errp

2020-03-11 Thread Vladimir Sementsov-Ogievskiy
09.03.2020 12:56, Markus Armbruster wrote: Suggest scripts: Coccinelle script to use auto-propagated errp or scripts: Coccinelle script to use ERRP_AUTO_PROPAGATE() Vladimir Sementsov-Ogievskiy writes: Script adds ERRP_AUTO_PROPAGATE macro invocation where appropriate and does

[PATCH v4 2/9] block/block-copy: fix progress calculation

2020-03-11 Thread Vladimir Sementsov-Ogievskiy
Assume we have two regions, A and B, and region B is in-flight now, region A is not yet touched, but it is unallocated and should be skipped. Correspondingly, as progress we have total = A + B current = 0 If we reset unallocated region A and call progress_reset_callback, it will calculate 0

[PULL 5/9] aio-posix: extract ppoll(2) and epoll(7) fd monitoring

2020-03-11 Thread Stefan Hajnoczi
The ppoll(2) and epoll(7) file descriptor monitoring implementations are mixed with the core util/aio-posix.c code. Before adding another implementation for Linux io_uring, extract out the existing ones so there is a clear interface and the core code is simpler. The new interface is

[PULL 8/9] aio-posix: support userspace polling of fd monitoring

2020-03-11 Thread Stefan Hajnoczi
Unlike ppoll(2) and epoll(7), Linux io_uring completions can be polled from userspace. Previously userspace polling was only allowed when all AioHandler's had an ->io_poll() callback. This prevented starvation of fds by userspace pollable handlers. Add the FDMonOps->need_wait() callback that

[PULL 02/19] luks: implement .bdrv_measure()

2020-03-11 Thread Max Reitz
From: Stefan Hajnoczi Add qemu-img measure support in the "luks" block driver. Signed-off-by: Stefan Hajnoczi Reviewed-by: Max Reitz Message-Id: <20200221112522.1497712-3-stefa...@redhat.com> Signed-off-by: Max Reitz --- block/crypto.c | 62 ++

[PULL 18/19] block/block-copy: reduce intersecting request lock

2020-03-11 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy Currently, block_copy operation lock the whole requested region. But there is no reason to lock clusters, which are already copied, it will disturb other parallel block_copy requests for no reason. Let's instead do the following: Lock only sub-region, which

[PULL 09/19] qemu-img: free memory before re-assign

2020-03-11 Thread Max Reitz
From: Pan Nengyuan collect_image_check() is called twice in img_check(), the filename/format will be alloced without free the original memory. It is not a big deal since the process will exit anyway, but seems like a clean code and it will remove the warning spotted by asan. Reported-by:

Re: [PATCH v8 02/10] scripts: add coccinelle script to use auto propagated errp

2020-03-11 Thread Markus Armbruster
Vladimir Sementsov-Ogievskiy writes: > 11.03.2020 12:38, Markus Armbruster wrote: >> Vladimir Sementsov-Ogievskiy writes: >> >>> 09.03.2020 12:56, Markus Armbruster wrote: Suggest scripts: Coccinelle script to use auto-propagated errp or scripts:

[PULL 16/19] block/block-copy: refactor interfaces to use bytes instead of end

2020-03-11 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy We have a lot of "chunk_end - start" invocations, let's switch to bytes/cur_bytes scheme instead. While being here, improve check on block_copy_do_copy parameters to not overflow when calculating nbytes and use int64_t for bytes in block_copy for consistency.

[PULL 19/19] block/block-copy: hide structure definitions

2020-03-11 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy Hide structure definitions and add explicit API instead, to keep an eye on the scope of the shared fields. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Andrey Shinkevich Reviewed-by: Max Reitz Message-Id:

[PULL 10/19] block/qcow2-threads: fix qcow2_decompress

2020-03-11 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy On success path we return what inflate() returns instead of 0. And it most probably works for Z_STREAM_END as it is positive, but is definitely broken for Z_BUF_ERROR. While being here, switch to errno return code, to be closer to qcow2_compress API (and usual

[PULL 9/9] aio-posix: remove idle poll handlers to improve scalability

2020-03-11 Thread Stefan Hajnoczi
When there are many poll handlers it's likely that some of them are idle most of the time. Remove handlers that haven't had activity recently so that the polling loop scales better for guests with a large number of devices. This feature only takes effect for the Linux io_uring fd monitoring

[PULL 3/9] aio-posix: completely stop polling when disabled

2020-03-11 Thread Stefan Hajnoczi
One iteration of polling is always performed even when polling is disabled. This is done because: 1. Userspace polling is cheaper than making a syscall. We might get lucky. 2. We must poll once more after polling has stopped in case an event occurred while stopping polling. However, there

Re: [PATCH v2 02/14] qcrypto/luks: implement encryption key management

2020-03-11 Thread Maxim Levitsky
On Tue, 2020-03-10 at 14:02 +0200, Maxim Levitsky wrote: > On Tue, 2020-03-10 at 12:59 +0100, Kevin Wolf wrote: > > Am 10.03.2020 um 12:05 hat Maxim Levitsky geschrieben: > > > On Tue, 2020-03-10 at 11:58 +0100, Max Reitz wrote: > > > > On 08.03.20 16:18, Maxim Levitsky wrote: > > > > > Next few

[PULL 01/19] luks: extract qcrypto_block_calculate_payload_offset()

2020-03-11 Thread Max Reitz
From: Stefan Hajnoczi The qcow2 .bdrv_measure() code calculates the crypto payload offset. This logic really belongs in crypto/block.c where it can be reused by other image formats. The "luks" block driver will need this same logic in order to implement .bdrv_measure(), so extract the

Re: [PULL 0/9] Block patches

2020-03-11 Thread no-reply
Patchew URL: https://patchew.org/QEMU/20200311124045.277969-1-stefa...@redhat.com/ Hi, This series failed the asan build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally. === TEST SCRIPT BEGIN === #!/bin/bash

[PATCH] iotests/026: Move v3-exclusive test to new file

2020-03-11 Thread Max Reitz
data_file does not work with v2, and we probably want 026 to keep working for v2 images. Thus, open a new file for v3-exclusive error path test cases. Fixes: 81311255f217859413c94f2cd9cebf2684bbda94 (“iotests/026: Test EIO on allocation in a data-file”) Signed-off-by: Max Reitz ---

[PULL 06/19] block/curl: HTTP header field names are case insensitive

2020-03-11 Thread Max Reitz
From: David Edmondson RFC 7230 section 3.2 indicates that HTTP header field names are case insensitive. Signed-off-by: David Edmondson Message-Id: <20200224101310.101169-3-david.edmond...@oracle.com> Reviewed-by: Max Reitz Signed-off-by: Max Reitz --- block/curl.c | 5 +++-- 1 file changed,

[PULL 15/19] block/block-copy: factor out find_conflicting_inflight_req

2020-03-11 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy Split find_conflicting_inflight_req to be used separately. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Andrey Shinkevich Reviewed-by: Max Reitz Message-Id: <20200311103004.7649-6-vsement...@virtuozzo.com> Signed-off-by: Max Reitz ---

[PULL 14/19] block/block-copy: use block_status

2020-03-11 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy Use bdrv_block_status_above to chose effective chunk size and to handle zeroes effectively. This substitutes checking for just being allocated or not, and drops old code path for it. Assistance by backup job is dropped too, as caching block-status information

[PULL 17/19] block/block-copy: rename start to offset in interfaces

2020-03-11 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy offset/bytes pair is more usual naming in block layer, let's use it. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Andrey Shinkevich Reviewed-by: Max Reitz Message-Id: <20200311103004.7649-8-vsement...@virtuozzo.com> Signed-off-by: Max Reitz ---

[PULL 12/19] block/block-copy: fix progress calculation

2020-03-11 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy Assume we have two regions, A and B, and region B is in-flight now, region A is not yet touched, but it is unallocated and should be skipped. Correspondingly, as progress we have total = A + B current = 0 If we reset unallocated region A and call

[PULL 13/19] block/block-copy: specialcase first copy_range request

2020-03-11 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy In block_copy_do_copy we fallback to read+write if copy_range failed. In this case copy_size is larger than defined for buffered IO, and there is corresponding commit. Still, backup copies data cluster by cluster, and most of requests are limited to one cluster

Re: [PATCH v8 02/10] scripts: add coccinelle script to use auto propagated errp

2020-03-11 Thread Vladimir Sementsov-Ogievskiy
11.03.2020 12:53, Markus Armbruster wrote: I think a v9 makes sense now. If any of the improvement ideas should turn into time sinks for you, let's talk. We don't need perfection, we only need to get to the point where we trust the script to do what we believe it does, understand its

Re: [PATCH 1/3] block/file-posix: add raw_getlength_fd

2020-03-11 Thread Max Reitz
On 30.01.20 16:22, Vladimir Sementsov-Ogievskiy wrote: > Add function which can handle separate fd, to be called from > raw_probe_alignment in the following commit. > > Signed-off-by: Vladimir Sementsov-Ogievskiy > --- > block/file-posix.c | 44 +++- > 1

Re: [PATCH 3/3] block: fail on open when file size is unaligned to request_alignment

2020-03-11 Thread Eric Blake
On 3/11/20 6:06 AM, Max Reitz wrote: On 30.01.20 16:22, Vladimir Sementsov-Ogievskiy wrote: Prior to the commit the following command lead to crash: ./qemu-io --image-opts -c 'write 0 512' \ driver=blkdebug,align=4096,image.driver=null-co,image.size=512 It failes on assertion in

Re: [PATCH v8 02/10] scripts: add coccinelle script to use auto propagated errp

2020-03-11 Thread Markus Armbruster
I think a v9 makes sense now. If any of the improvement ideas should turn into time sinks for you, let's talk. We don't need perfection, we only need to get to the point where we trust the script to do what we believe it does, understand its limitations, and know how to compensate for them.

Re: [PATCH v4 0/9] block-copy improvements: part I

2020-03-11 Thread Vladimir Sementsov-Ogievskiy
11.03.2020 14:37, Max Reitz wrote: On 11.03.20 11:29, Vladimir Sementsov-Ogievskiy wrote: v4: 01: add Max's r-b 02: rm ProgressResetCallbackFunc typedef 03-06: add Max's r-b 07: fix indentation 08: update in_flight_bytes in block_copy_inflight_req_shrink improve comment above block_copy()

[PULL 1/9] qemu/queue.h: clear linked list pointers on remove

2020-03-11 Thread Stefan Hajnoczi
Do not leave stale linked list pointers around after removal. It's safer to set them to NULL so that use-after-removal results in an immediate segfault. The RCU queue removal macros are unchanged since nodes may still be traversed after removal. Suggested-by: Paolo Bonzini Signed-off-by:

[PULL 0/9] Block patches

2020-03-11 Thread Stefan Hajnoczi
The following changes since commit 67f17e23baca5dd545fe98b01169cc351a70fe35: Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging (2020-03-06 17:15:36 +) are available in the Git repository at: https://github.com/stefanha/qemu.git tags/block-pull-request for you

Re: [PATCH v8 02/10] scripts: add coccinelle script to use auto propagated errp

2020-03-11 Thread Markus Armbruster
Vladimir Sementsov-Ogievskiy writes: > 10.03.2020 18:47, Markus Armbruster wrote: >> Vladimir Sementsov-Ogievskiy writes: >> >>> 09.03.2020 12:56, Markus Armbruster wrote: Suggest scripts: Coccinelle script to use auto-propagated errp or scripts:

Re: [PATCH v3 8/9] block/block-copy: reduce intersecting request lock

2020-03-11 Thread Vladimir Sementsov-Ogievskiy
10.03.2020 18:32, Max Reitz wrote: On 06.03.20 08:38, Vladimir Sementsov-Ogievskiy wrote: Currently, block_copy operation lock the whole requested region. But there is no reason to lock clusters, which are already copied, it will disturb other parallel block_copy requests for no reason. Let's

Re: [PATCH v4 00/10] Further bitmaps improvements

2020-03-11 Thread Max Reitz
On 11.03.20 07:17, Vladimir Sementsov-Ogievskiy wrote: > 10.03.2020 20:17, Max Reitz wrote: >> On 06.03.20 08:45, Vladimir Sementsov-Ogievskiy wrote: >>> 26.02.2020 16:13, Max Reitz wrote: On 05.02.20 12:20, Vladimir Sementsov-Ogievskiy wrote: > Hi! > > The main feature here is

Re: [PATCH v8 02/10] scripts: add coccinelle script to use auto propagated errp

2020-03-11 Thread Vladimir Sementsov-Ogievskiy
11.03.2020 12:04, Markus Armbruster wrote: Vladimir Sementsov-Ogievskiy writes: 10.03.2020 18:47, Markus Armbruster wrote: Vladimir Sementsov-Ogievskiy writes: 09.03.2020 12:56, Markus Armbruster wrote: Suggest scripts: Coccinelle script to use auto-propagated errp or

[PATCH v4 9/9] block/block-copy: hide structure definitions

2020-03-11 Thread Vladimir Sementsov-Ogievskiy
Hide structure definitions and add explicit API instead, to keep an eye on the scope of the shared fields. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Andrey Shinkevich Reviewed-by: Max Reitz --- include/block/block-copy.h | 52 +++--

[PATCH v4 3/9] block/block-copy: specialcase first copy_range request

2020-03-11 Thread Vladimir Sementsov-Ogievskiy
In block_copy_do_copy we fallback to read+write if copy_range failed. In this case copy_size is larger than defined for buffered IO, and there is corresponding commit. Still, backup copies data cluster by cluster, and most of requests are limited to one cluster anyway, so the only source of this

[PATCH v4 8/9] block/block-copy: reduce intersecting request lock

2020-03-11 Thread Vladimir Sementsov-Ogievskiy
Currently, block_copy operation lock the whole requested region. But there is no reason to lock clusters, which are already copied, it will disturb other parallel block_copy requests for no reason. Let's instead do the following: Lock only sub-region, which we are going to operate on. Then,

[PATCH v4 1/9] job: refactor progress to separate object

2020-03-11 Thread Vladimir Sementsov-Ogievskiy
We need it in separate to pass to the block-copy object in the next commit. Cc: qemu-sta...@nongnu.org Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Andrey Shinkevich Reviewed-by: Max Reitz --- include/qemu/job.h| 11 ++- include/qemu/progress_meter.h | 58

[PATCH v4 0/9] block-copy improvements: part I

2020-03-11 Thread Vladimir Sementsov-Ogievskiy
v4: 01: add Max's r-b 02: rm ProgressResetCallbackFunc typedef 03-06: add Max's r-b 07: fix indentation 08: update in_flight_bytes in block_copy_inflight_req_shrink improve comment above block_copy() definition 09: add Andrey's and Max's r-b drop hunk fixing indentation (merged to 07)

[PATCH v4 7/9] block/block-copy: rename start to offset in interfaces

2020-03-11 Thread Vladimir Sementsov-Ogievskiy
offset/bytes pair is more usual naming in block layer, let's use it. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Andrey Shinkevich Reviewed-by: Max Reitz --- include/block/block-copy.h | 4 +- block/block-copy.c | 82 +++--- 2 files

[PATCH v4 6/9] block/block-copy: refactor interfaces to use bytes instead of end

2020-03-11 Thread Vladimir Sementsov-Ogievskiy
We have a lot of "chunk_end - start" invocations, let's switch to bytes/cur_bytes scheme instead. While being here, improve check on block_copy_do_copy parameters to not overflow when calculating nbytes and use int64_t for bytes in block_copy for consistency. Signed-off-by: Vladimir

[PATCH v4 5/9] block/block-copy: factor out find_conflicting_inflight_req

2020-03-11 Thread Vladimir Sementsov-Ogievskiy
Split find_conflicting_inflight_req to be used separately. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Andrey Shinkevich Reviewed-by: Max Reitz --- block/block-copy.c | 31 +++ 1 file changed, 19 insertions(+), 12 deletions(-) diff --git

[PATCH v4 4/9] block/block-copy: use block_status

2020-03-11 Thread Vladimir Sementsov-Ogievskiy
Use bdrv_block_status_above to chose effective chunk size and to handle zeroes effectively. This substitutes checking for just being allocated or not, and drops old code path for it. Assistance by backup job is dropped too, as caching block-status information is more difficult than just caching

Re: [PATCH 2/3] block/file-posix: consider file size when fallback to max_align

2020-03-11 Thread Max Reitz
On 30.01.20 16:22, Vladimir Sementsov-Ogievskiy wrote: > If we failed to probe request_align, we fallback to max_align. But > this is wrong, if file size is not aligned to our max_align. Let's > instead chose alignment so that file size is a multiple of it. It’s entirely possible that the file

Re: [PATCH v4 0/9] block-copy improvements: part I

2020-03-11 Thread Max Reitz
On 11.03.20 11:29, Vladimir Sementsov-Ogievskiy wrote: > v4: > > 01: add Max's r-b > 02: rm ProgressResetCallbackFunc typedef > 03-06: add Max's r-b > 07: fix indentation > 08: update in_flight_bytes in block_copy_inflight_req_shrink > improve comment above block_copy() definition > 09: add

Re: [PATCH v8 02/10] scripts: add coccinelle script to use auto propagated errp

2020-03-11 Thread Vladimir Sementsov-Ogievskiy
09.03.2020 12:56, Markus Armbruster wrote: + +// Convert error clearing functions Suggest: Ensure @local_err is cleared on free But there is no local_err after conversion +( +-error_free(local_err); ++error_free_errp(errp); +| +-error_report_err(local_err); ++

Re: [PATCH v8 02/10] scripts: add coccinelle script to use auto propagated errp

2020-03-11 Thread Markus Armbruster
Vladimir Sementsov-Ogievskiy writes: > 09.03.2020 12:56, Markus Armbruster wrote: >>> + >>> +// Convert error clearing functions >> Suggest: Ensure @local_err is cleared on free > > But there is no local_err after conversion True. Hmm. What about this: // Convert calls to error_free(),

Re: [PATCH v8 02/10] scripts: add coccinelle script to use auto propagated errp

2020-03-11 Thread Vladimir Sementsov-Ogievskiy
11.03.2020 12:33, Markus Armbruster wrote: Vladimir Sementsov-Ogievskiy writes: 09.03.2020 12:56, Markus Armbruster wrote: + +// Convert error clearing functions Suggest: Ensure @local_err is cleared on free But there is no local_err after conversion True. Hmm. What about this:

Re: [PATCH v8 02/10] scripts: add coccinelle script to use auto propagated errp

2020-03-11 Thread Vladimir Sementsov-Ogievskiy
11.03.2020 9:55, Vladimir Sementsov-Ogievskiy wrote: 10.03.2020 18:47, Markus Armbruster wrote: Vladimir Sementsov-Ogievskiy writes: 09.03.2020 12:56, Markus Armbruster wrote: Suggest   scripts: Coccinelle script to use auto-propagated errp or   scripts: Coccinelle script to use

Re: [PATCH RESEND v2] block/nvme: introduce PMR support from NVMe 1.4 spec

2020-03-11 Thread Stefan Hajnoczi
On Tue, Mar 10, 2020 at 8:09 PM Andrzej Jakowski wrote: > On 3/10/20 2:51 AM, Stefan Hajnoczi wrote: > > On Fri, Mar 06, 2020 at 03:38:53PM -0700, Andrzej Jakowski wrote: > >> diff --git a/hw/block/nvme.c b/hw/block/nvme.c > >> index d28335cbf3..ff7e74d765 100644 > >> --- a/hw/block/nvme.c > >>

Re: [PATCH v8 02/10] scripts: add coccinelle script to use auto propagated errp

2020-03-11 Thread Markus Armbruster
Vladimir Sementsov-Ogievskiy writes: > 09.03.2020 12:56, Markus Armbruster wrote: >> Suggest >> >> scripts: Coccinelle script to use auto-propagated errp >> >> or >> >> scripts: Coccinelle script to use ERRP_AUTO_PROPAGATE() >> >> Vladimir Sementsov-Ogievskiy writes: [...] >>> +//

Re: [PATCH 3/3] block: fail on open when file size is unaligned to request_alignment

2020-03-11 Thread Max Reitz
On 30.01.20 16:22, Vladimir Sementsov-Ogievskiy wrote: > Prior to the commit the following command lead to crash: > > ./qemu-io --image-opts -c 'write 0 512' \ > driver=blkdebug,align=4096,image.driver=null-co,image.size=512 > > It failes on assertion in bdrv_aligned_pwritev: > "end_sector

[PULL 2/9] aio-posix: remove confusing QLIST_SAFE_REMOVE()

2020-03-11 Thread Stefan Hajnoczi
QLIST_SAFE_REMOVE() is confusing here because the node must be on the list. We actually just wanted to clear the linked list pointers when removing it from the list. QLIST_REMOVE() now does this, so switch to it. Suggested-by: Paolo Bonzini Signed-off-by: Stefan Hajnoczi Link:

[PULL 07/13] iotests: Add iothread cases to 155

2020-03-11 Thread Kevin Wolf
This patch adds test cases for attaching the backing chain to a mirror job target right before finalising the job, where the image is in a non-mainloop AioContext (i.e. the backing chain needs to be moved to the AioContext of the mirror target). This requires switching the test case from

[PULL 04/13] iotests: Fix run_job() with use_log=False

2020-03-11 Thread Kevin Wolf
The 'job-complete' QMP command should be run with qmp() rather than qmp_log() if use_log=False is passed. Signed-off-by: Kevin Wolf Message-Id: <20200310113831.27293-4-kw...@redhat.com> Reviewed-by: Peter Krempa Signed-off-by: Kevin Wolf --- tests/qemu-iotests/iotests.py | 5 - 1 file

[PULL 03/13] block: Relax restrictions for blockdev-snapshot

2020-03-11 Thread Kevin Wolf
blockdev-snapshot returned an error if the overlay was already in use, which it defined as having any BlockBackend parent. This is in fact both too strict (some parents can tolerate the change of visible data caused by attaching a backing file) and too loose (some non-BlockBackend parents may not

[PULL 01/13] qom-qmp-cmds: fix two memleaks in qmp_object_add

2020-03-11 Thread Kevin Wolf
From: Pan Nengyuan 'type/id' forgot to free in qmp_object_add, this patch fix that. The leak stack: Direct leak of 84 byte(s) in 6 object(s) allocated from: #0 0x7fe2a5ebf768 in __interceptor_malloc (/lib64/libasan.so.5+0xef768) #1 0x7fe2a505 in g_malloc

[PULL 06/13] block: Fix cross-AioContext blockdev-snapshot

2020-03-11 Thread Kevin Wolf
external_snapshot_prepare() tries to move the overlay to the AioContext of the backing file (the snapshotted node). However, it's possible that this doesn't work, but the backing file can instead be moved to the overlay's AioContext (e.g. opening the backing chain for a mirror target).

[PULL 02/13] block: Make bdrv_get_cumulative_perm() public

2020-03-11 Thread Kevin Wolf
Signed-off-by: Kevin Wolf Message-Id: <20200310113831.27293-2-kw...@redhat.com> Reviewed-by: Peter Krempa Signed-off-by: Kevin Wolf --- include/block/block_int.h | 3 +++ block.c | 6 ++ 2 files changed, 5 insertions(+), 4 deletions(-) diff --git

[PULL 10/13] block: introducing 'bdrv_co_delete_file' interface

2020-03-11 Thread Kevin Wolf
From: Daniel Henrique Barboza Adding to Block Drivers the capability of being able to clean up its created files can be useful in certain situations. For the LUKS driver, for instance, a failure in one of its authentication steps can leave files in the host that weren't there before. This patch

[PULL 12/13] crypto.c: cleanup created file when block_crypto_co_create_opts_luks fails

2020-03-11 Thread Kevin Wolf
From: Daniel Henrique Barboza When using a non-UTF8 secret to create a volume using qemu-img, the following error happens: $ qemu-img create -f luks --object secret,id=vol_1_encrypt0,file=vol_resize_pool.vol_1.secret.qzVQrI -o key-secret=vol_1_encrypt0 /var/tmp/pool_target/vol_1 10240K

Re: [PATCH v5 2/5] qcow2: introduce compression type feature

2020-03-11 Thread Max Reitz
On 04.03.20 14:35, Denis Plotnikov wrote: > The patch adds some preparation parts for incompatible compression type > feature to qcow2 allowing the use different compression methods for > image clusters (de)compressing. > > It is implied that the compression type is set on the image creation and

Re: [PULL 0/9] Block patches

2020-03-11 Thread Stefan Hajnoczi
On Wed, Mar 11, 2020 at 06:50:37AM -0700, no-re...@patchew.org wrote: > Patchew URL: > https://patchew.org/QEMU/20200311124045.277969-1-stefa...@redhat.com/ ... > dbus-daemon[12353]: Could not get password database information for UID of > current process: User "???" unknown or no memory to

[PULL 05/13] iotests: Test mirror with temporarily disabled target backing file

2020-03-11 Thread Kevin Wolf
The newly tested scenario is a common live storage migration scenario: The target node is opened without a backing file so that the active layer is mirrored while its backing chain can be copied in the background. The backing chain should be attached to the mirror target node when finalising the

Re: [PATCH v5 0/5] qcow2: Implement zstd cluster compression method

2020-03-11 Thread Max Reitz
On 11.03.20 08:31, Denis Plotnikov wrote: > ping! > > Is there any other comments/concerns/objections/suggestions according to > the series except the minor ones from Alberto and Vladimir? > If not, please, let me know, so I can resend the series with the minor > changes for applying to the

Re: [PATCH v4 00/10] Further bitmaps improvements

2020-03-11 Thread John Snow
On 3/11/20 9:58 AM, Vladimir Sementsov-Ogievskiy wrote: > 11.03.2020 12:55, Max Reitz wrote: >> On 11.03.20 07:17, Vladimir Sementsov-Ogievskiy wrote: >>> 10.03.2020 20:17, Max Reitz wrote: On 06.03.20 08:45, Vladimir Sementsov-Ogievskiy wrote: > 26.02.2020 16:13, Max Reitz wrote:

Re: [PATCH v5 0/5] qcow2: Implement zstd cluster compression method

2020-03-11 Thread Eric Blake
On 3/11/20 11:28 AM, Max Reitz wrote: On 11.03.20 08:31, Denis Plotnikov wrote: ping! Is there any other comments/concerns/objections/suggestions according to the series except the minor ones from Alberto and Vladimir? If not, please, let me know, so I can resend the series with the minor

[PULL 11/13] block.c: adding bdrv_co_delete_file

2020-03-11 Thread Kevin Wolf
From: Daniel Henrique Barboza Using the new 'bdrv_co_delete_file' interface, a pure co_routine function 'bdrv_co_delete_file' inside block.c can can be used in a way similar of the existing bdrv_create_file to to clean up a created file. We're creating a pure co_routine because the only caller

Re: [PULL 0/9] Block patches

2020-03-11 Thread Stefan Hajnoczi
On Wed, Mar 11, 2020 at 06:51:46AM -0700, no-re...@patchew.org wrote: > Patchew URL: > https://patchew.org/QEMU/20200311124045.277969-1-stefa...@redhat.com/ ... > === OUTPUT BEGIN === > 1/9 Checking commit 86ac8a79649d (qemu/queue.h: clear linked list pointers on > remove) > ERROR: do not use

Re: [PULL 0/9] Block patches

2020-03-11 Thread Peter Maydell
On Wed, 11 Mar 2020 at 12:40, Stefan Hajnoczi wrote: > > The following changes since commit 67f17e23baca5dd545fe98b01169cc351a70fe35: > > Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging > (2020-03-06 17:15:36 +) > > are available in the Git repository at: > >

[PULL 08/13] qapi: Add '@allow-write-only-overlay' feature for 'blockdev-snapshot'

2020-03-11 Thread Kevin Wolf
From: Peter Krempa Anounce that 'blockdev-snapshot' command's permissions allow changing of the backing file if the 'consistent_read' permission is not required. This is useful for libvirt to allow late opening of the backing chain during a blockdev-mirror. Signed-off-by: Peter Krempa

Re: [PATCH v4 1/4] ide/via: Get rid of via_init_ide()

2020-03-11 Thread Mark Cave-Ayland
On 10/03/2020 19:06, BALATON Zoltan wrote: > Follow example of CMD646 and remove via_init_ide function and do it > directly in board code instead. > > Signed-off-by: BALATON Zoltan > --- > hw/ide/via.c| 8 > hw/mips/mips_fulong2e.c | 5 - > include/hw/ide.h| 1

Re: [PATCH v4 3/4] pci: Honour wmask when resetting PCI_INTERRUPT_LINE

2020-03-11 Thread Mark Cave-Ayland
On 10/03/2020 19:06, BALATON Zoltan wrote: > The pci_do_device_reset() function (called from pci_device_reset) > clears the PCI_INTERRUPT_LINE config reg of devices on the bus but did > this without taking wmask into account. We'll have a device model now > that needs to set a constant value for

Re: [PATCH v4 4/4] via-ide: Also emulate non 100% native mode

2020-03-11 Thread Mark Cave-Ayland
On 10/03/2020 19:06, BALATON Zoltan wrote: > Some machines operate in "non 100% native mode" where interrupts are > fixed at legacy IDE interrupts and some guests expect this behaviour > without checking based on knowledge about hardware. Even Linux has > arch specific workarounds for this that

Re: [PATCH RESEND v2] block/nvme: introduce PMR support from NVMe 1.4 spec

2020-03-11 Thread Andrzej Jakowski
On 3/11/20 2:20 AM, Stefan Hajnoczi wrote: > Please try: > > $ git grep pmem > > backends/hostmem-file.c is the backend that can be used and the > pmem_persist() API can be used to flush writes. I've reworked this patch into hostmem-file type of backend. >From simple tests in virtual machine:

[PULL 00/13] Block layer patches

2020-03-11 Thread Kevin Wolf
The following changes since commit ba29883206d92a29ad5a466e679ccfc2ee6132ef: Merge remote-tracking branch 'remotes/borntraeger/tags/s390x-20200310' into staging (2020-03-10 16:50:28 +) are available in the Git repository at: git://repo.or.cz/qemu/kevin.git tags/for-upstream for you to

[PULL 13/13] qemu-iotests: adding LUKS cleanup for non-UTF8 secret error

2020-03-11 Thread Kevin Wolf
From: Daniel Henrique Barboza This patch adds a new test file to exercise the case where qemu-img fails to complete for the LUKS format when a non-UTF8 secret is used. Signed-off-by: Daniel Henrique Barboza Message-Id: <20200130213907.2830642-5-danielhb...@gmail.com> Signed-off-by: Kevin Wolf

[PULL 09/13] tests/qemu-iotests: Fix socket_scm_helper build path

2020-03-11 Thread Kevin Wolf
From: Philippe Mathieu-Daudé The socket_scm_helper path got corrupted during the mechanical refactor moving the qtests files into their own sub-directory. Fixes: 1e8a1fae7 ("test: Move qtests to a separate directory") Signed-off-by: Philippe Mathieu-Daudé Message-Id:

Re: [PATCH RESEND v2] block/nvme: introduce PMR support from NVMe 1.4 spec

2020-03-11 Thread Andrzej Jakowski
On 3/11/20 2:20 AM, Stefan Hajnoczi wrote: > Oh, I think I see what you mean. That is not how the term > "preallocated" is usually used in POSIX file systems. File systems > have sparse files by default and the term preallocation is used in the > context of fadvise(2) for reserving space. > >

Re: [PATCH v4 4/4] via-ide: Also emulate non 100% native mode

2020-03-11 Thread BALATON Zoltan
On Wed, 11 Mar 2020, Mark Cave-Ayland wrote: On 10/03/2020 19:06, BALATON Zoltan wrote: Some machines operate in "non 100% native mode" where interrupts are fixed at legacy IDE interrupts and some guests expect this behaviour without checking based on knowledge about hardware. Even Linux has