[Qemu-block] [PATCH v6 24/24] qemu-io: Relax 'alloc' now that block-status doesn't assert

Previously, the alloc command required that input parameters be sector-aligned and clamped to 32 bits, because the underlying bdrv_is_allocated used a 32-bit parameter and asserted aligned inputs. But now that we have fixed block status to report a 64-bit bytes value, and to properly round

[Qemu-block] [PATCH v6 23/24] qcow2: Relax is_zero() assertion

Now that bdrv_is_allocated accepts non-aligned inputs, we can remove the TODO added in earlier refactoring. Signed-off-by: Eric Blake --- v6: new patch [Kevin] --- block/qcow2.c | 12 +++- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/block/qcow2.c

[Qemu-block] [PATCH v6 22/24] block: Relax bdrv_aligned_preadv() assertion

Now that bdrv_is_allocated accepts non-aligned inputs, we can remove the TODO added in commit d6a644bb. Signed-off-by: Eric Blake Reviewed-by: John Snow --- v4-v5: no change v3: new patch [Kevin] --- block/io.c | 8 ++-- 1 file changed, 2

[Qemu-block] [PATCH v6 21/24] block: Align block status requests

Any device that has request_alignment greater than 512 should be unable to report status at a finer granularity; it may also be simpler for such devices to be guaranteed that the block layer has rounded things out to the granularity boundary (the way the block layer already rounds all other I/O

[Qemu-block] [PATCH v6 20/24] qemu-img: Change img_compare() to be byte-based

In the continuing quest to make more things byte-based, change the internal iteration of img_compare(). We can finally drop the TODO assertions added earlier, now that the entire algorithm is byte-based and no longer has to shift from bytes to sectors. Most of the change is mechanical

[Qemu-block] [PATCH v6 19/24] qemu-img: Change img_rebase() to be byte-based

In the continuing quest to make more things byte-based, change the internal iteration of img_rebase(). We can finally drop the TODO assertion added earlier, now that the entire algorithm is byte-based and no longer has to shift from bytes to sectors. Most of the change is mechanical

[Qemu-block] [PATCH v6 18/24] qemu-img: Change compare_sectors() to be byte-based

In the continuing quest to make more things byte-based, change compare_sectors(), renaming it to compare_buffers() in the process. Note that one caller (qemu-img compare) only cares about the first difference, while the other (qemu-img rebase) cares about how many consecutive sectors have the

[Qemu-block] [PATCH v6 15/24] qemu-img: Add find_nonzero()

During 'qemu-img compare', when we are checking that an allocated portion of one file is all zeros, we don't need to waste time computing how many additional sectors after the first non-zero byte are also non-zero. Create a new helper find_nonzero() to do the check for a first non-zero sector,

[Qemu-block] [PATCH v6 08/24] block: Switch bdrv_co_get_block_status() to byte-based

We are gradually converting to byte-based interfaces, as they are easier to reason about than sector-based. Convert another internal function (no semantic change); and as with its public counterpart, rename to bdrv_co_block_status() and split the offset return, to make the compiler enforce that

[Qemu-block] [PATCH v6 17/24] qemu-img: Change check_empty_sectors() to byte-based

Continue on the quest to make more things byte-based instead of sector-based. Signed-off-by: Eric Blake Reviewed-by: John Snow --- v4-v5: no change v3: new patch --- qemu-img.c | 27 +++ 1 file changed, 15 insertions(+), 12

[Qemu-block] [PATCH v6 14/24] qemu-img: Speed up compare on pre-allocated larger file

Compare the following images with all-zero contents: $ truncate --size 1M A $ qemu-img create -f qcow2 -o preallocation=off B 1G $ qemu-img create -f qcow2 -o preallocation=metadata C 1G On my machine, the difference is noticeable for pre-patch speeds, with more than an order of magnitude in

[Qemu-block] [PATCH v6 16/24] qemu-img: Drop redundant error message in compare

If a read error is encountered during 'qemu-img compare', we were printing the "Error while reading offset ..." message twice; this was because our helper function was awkward, printing output on some but not all paths. Fix it to consistently report errors on all paths, so that the callers do not

[Qemu-block] [PATCH v6 11/24] block: Switch bdrv_co_get_block_status_above() to byte-based

We are gradually converting to byte-based interfaces, as they are easier to reason about than sector-based. Convert another internal type (no semantic change), and rename it to match the corresponding public function rename. Signed-off-by: Eric Blake --- v6: rebase to split

[Qemu-block] [PATCH v6 12/24] block: Convert bdrv_get_block_status_above() to bytes

We are gradually moving away from sector-based interfaces, towards byte-based. In the common case, allocation is unlikely to ever use values that are not naturally sector-aligned, but it is possible that byte-based values will let us be more precise about allocation at the end of an unaligned

[Qemu-block] [PATCH v6 13/24] qemu-img: Simplify logic in img_compare()

As long as we are querying the status for a chunk smaller than the known image size, we are guaranteed that a successful return will have set pnum to a non-zero size (pnum is zero only for queries beyond the end of the file). Use that to slightly simplify the calculation of the current chunk size

[Qemu-block] [PATCH v6 10/24] block: Switch bdrv_common_block_status_above() to byte-based

We are gradually converting to byte-based interfaces, as they are easier to reason about than sector-based. Convert another internal function (no semantic change). Signed-off-by: Eric Blake --- v6: rebase to split return interface change [Kevin], R-b dropped v4-v5: no change

[Qemu-block] [PATCH v6 07/24] block: Convert bdrv_get_block_status() to bytes

We are gradually moving away from sector-based interfaces, towards byte-based. In the common case, allocation is unlikely to ever use values that are not naturally sector-aligned, but it is possible that byte-based values will let us be more precise about allocation at the end of an unaligned

[Qemu-block] [PATCH v6 06/24] qemu-img: Switch get_block_status() to byte-based

We are gradually converting to byte-based interfaces, as they are easier to reason about than sector-based. Continue by converting an internal function (no semantic change), and simplifying its caller accordingly. Signed-off-by: Eric Blake Reviewed-by: Fam Zheng

[Qemu-block] [PATCH v6 04/24] qcow2: Switch is_zero_sectors() to byte-based

We are gradually converting to byte-based interfaces, as they are easier to reason about than sector-based. Convert another internal function (no semantic change), and rename it to is_zero() in the process. Signed-off-by: Eric Blake Reviewed-by: Fam Zheng

[Qemu-block] [PATCH v6 05/24] block: Switch bdrv_make_zero() to byte-based

We are gradually converting to byte-based interfaces, as they are easier to reason about than sector-based. Change the internal loop iteration of zeroing a device to track by bytes instead of sectors (although we are still guaranteed that we iterate by steps that are sector-aligned).

[Qemu-block] [PATCH v6 09/24] block: Switch BdrvCoGetBlockStatusData to byte-based

We are gradually converting to byte-based interfaces, as they are easier to reason about than sector-based. Convert another internal type (no semantic change), and rename it to match the corresponding public function rename. Signed-off-by: Eric Blake --- v6: fix -O2

[Qemu-block] [PATCH v6 03/24] block: Make bdrv_round_to_clusters() signature more useful

In the process of converting sector-based interfaces to bytes, I'm finding it easier to represent a byte count as a 64-bit integer at the block layer (even if we are internally capped by SIZE_MAX or even INT_MAX for individual transactions, it's still nicer to not have to worry about

[Qemu-block] [PATCH v6 02/24] block: Add flag to avoid wasted work in bdrv_is_allocated()

Not all callers care about which BDS owns the mapping for a given range of the file, or where the zeroes lie within that mapping. In particular, bdrv_is_allocated() cares more about finding the largest run of allocated data from the guest perspective, whether or not that data is consecutive from

[Qemu-block] [PATCH v6 00/24] make bdrv_get_block_status byte-based

There are patches floating around to add NBD_CMD_BLOCK_STATUS, but NBD wants to report status on byte granularity (even if the reporting will probably be naturally aligned to sectors or even much higher levels). I've therefore started the task of converting our block status code to report at a

[Qemu-block] [PATCH v6 01/24] block: Allow NULL file for bdrv_get_block_status()

Not all callers care about which BDS owns the mapping for a given range of the file. This patch merely simplifies the callers by consolidating the logic in the common call point, while guaranteeing a non-NULL file to all the driver callbacks, for no semantic change. The only caller that does not

Re: [Qemu-block] [Qemu-devel] Patch to add helpful tracing output for driver authors in NVMe emulation

On 11/10/2017 15:08, Peter Maydell wrote: >> Tracing is the right tool to detect bad guest code, and I think it makes >> sense to mark conditions that shouldn't happen with a correctly >> operating guest driver. I'm not sure if an exclamation mark is the best >> syntax for this, because I wouldn't

[Qemu-block] [PULL 1/5] iotests: Set up Python logging

Set up Python logging module instead of relying on QEMUMachine._debug to enable debugging messages. Cc: Kevin Wolf Cc: Max Reitz Cc: qemu-block@nongnu.org Signed-off-by: Eduardo Habkost Message-Id:

Re: [Qemu-block] [Qemu-devel] [PATCH v5 14/23] qemu-img: Speed up compare on pre-allocated larger file

On 10/03/2017 09:00 PM, Eric Blake wrote: > Compare the following images with all-zero contents: > $ truncate --size 1M A > $ qemu-img create -f qcow2 -o preallocation=off B 1G > $ qemu-img create -f qcow2 -o preallocation=metadata C 1G > > Signed-off-by: Eric Blake >

Re: [Qemu-block] [Qemu-devel] [PATCH v5 01/23] block: Allow NULL file for bdrv_get_block_status()

On 10/11/2017 04:42 AM, Kevin Wolf wrote: > Am 10.10.2017 um 21:24 hat John Snow geschrieben: >> >> >> On 10/10/2017 03:00 PM, Eric Blake wrote: >>> On 10/10/2017 09:43 AM, Eric Blake wrote: >>> >> --- >> v5: use second label for cleaner exit logic [John], use local_pnum >> @@

Re: [Qemu-block] [PATCH v2 04/10] nbd-server: refactor simple reply sending

09.10.2017 22:18, Eric Blake wrote: [adding Dan for a qio question - search for your name below] On 10/09/2017 12:27 PM, Vladimir Sementsov-Ogievskiy wrote: Get rid of calculating structure fields offsets by hand and set_cork, rename nbd_co_send_reply to nbd_co_send_simple_reply. Do not use

Re: [Qemu-block] [PATCH v2 10/10] nbd: Minimal structured read for client

10.10.2017 11:02, Paolo Bonzini wrote: On 09/10/2017 19:27, Vladimir Sementsov-Ogievskiy wrote: +int ret = nbd_co_do_receive_one_chunk(s, handle, only_structured, + _ret, qiov, payload); + +if (ret < 0) { +s->quit = true; +} else { +

Re: [Qemu-block] [PATCH 0/3] qcow2: Fix preallocated truncation

On Mon, Oct 09, 2017 at 11:55:30PM +0200, Max Reitz wrote: > There are (at least...) two bugs in preallocated truncation, which this > series fixes. > > Both would have been apparent if the the related iotest had used the > default cluster size in addition to 512 byte clusters, but alas, it did >

Re: [Qemu-block] [PATCH 0/3] qcow2: Fix preallocated truncation

On 2017-10-09 23:55, Max Reitz wrote: > There are (at least...) two bugs in preallocated truncation, which this > series fixes. > > Both would have been apparent if the the related iotest had used the > default cluster size in addition to 512 byte clusters, but alas, it did > not. This is

Re: [Qemu-block] [Qemu-devel] Patch to add helpful tracing output for driver authors in NVMe emulation

On 10 October 2017 at 09:02, Kevin Wolf wrote: > Assertions are for checking that assumptions in qemu code hold true. > Here it's about bad guest code, and you can't let qemu abort for that. > > Tracing is the right tool to detect bad guest code, and I think it makes > sense to

Re: [Qemu-block] [Qemu-devel] Patch to add helpful tracing output for driver authors in NVMe emulation

On Tue, Oct 10, 2017 at 4:02 AM, Kevin Wolf wrote: > Am 10.10.2017 um 08:58 hat Markus Armbruster geschrieben: >> Doug Gale writes: >> >> > I used exclamations as a concise way of indicating that the driver did >> > something nonsensical, or horribly invalid,

Re: [Qemu-block] [PATCH 15/18] block/mirror: Add active mirroring

On 2017-10-10 12:16, Kevin Wolf wrote: > Am 18.09.2017 um 18:26 hat Max Reitz geschrieben: >> On 2017-09-18 12:06, Stefan Hajnoczi wrote: >>> On Sat, Sep 16, 2017 at 03:58:01PM +0200, Max Reitz wrote: On 2017-09-14 17:57, Stefan Hajnoczi wrote: > On Wed, Sep 13, 2017 at 08:19:07PM +0200,

Re: [Qemu-block] [PATCH 13/18] block/mirror: Keep write perm for pending writes

On 2017-10-10 11:58, Kevin Wolf wrote: > Am 13.09.2017 um 20:19 hat Max Reitz geschrieben: >> The owner of the mirror BDS might retire its write permission; but there >> may still be pending mirror operations so the mirror BDS cannot >> necessarily retire its write permission for its child then.

Re: [Qemu-block] [Qemu-devel] Co-routine re-entered recursively

[ Cc: qemu-block ] Am 09.10.2017 um 21:11 hat Gugnani, Shashank geschrieben: > Hi, > > I'm trying to modify the NVMe emulator in QEMU to use multiple threads. > > After making necessary changes, I see this error: 'Co-routine re-entered > recursively' > > Can anybody tell me the reason for

Re: [Qemu-block] [PATCH 10/18] block/mirror: Make source the file child

On 2017-10-10 11:47, Kevin Wolf wrote: > Am 13.09.2017 um 20:19 hat Max Reitz geschrieben: >> Regarding the source BDS, the mirror BDS is arguably a filter node. >> Therefore, the source BDS should be its "file" child. >> >> Signed-off-by: Max Reitz > > TODO: Justification why

Re: [Qemu-block] [PATCH 08/18] block/mirror: Use source as a BdrvChild

On 2017-10-10 11:27, Kevin Wolf wrote: > Am 13.09.2017 um 20:19 hat Max Reitz geschrieben: >> With this, the mirror_top_bs is no longer just a technically required >> node in the BDS graph but actually represents the block job operation. >> >> Signed-off-by: Max Reitz >> --- >>

Re: [Qemu-block] [PATCH 05/18] block/mirror: Convert to coroutines

On 2017-10-10 11:14, Kevin Wolf wrote: > Am 13.09.2017 um 20:18 hat Max Reitz geschrieben: >> In order to talk to the source BDS (and maybe in the future to the >> target BDS as well) directly, we need to convert our existing AIO >> requests into coroutine I/O requests. >> >> Signed-off-by: Max

Re: [Qemu-block] [PATCH 02/18] block: BDS deletion during bdrv_drain_recurse

On 2017-10-10 10:36, Kevin Wolf wrote: > Am 13.09.2017 um 20:18 hat Max Reitz geschrieben: >> Drainined a BDS child may lead to both the original BDS and/or its other >> children being deleted (e.g. if the original BDS represents a block >> job). We should prepare for this in both

Re: [Qemu-block] [Qemu-devel] [PATCH 2/3] qcow2: Always execute preallocate() in a coroutine

On 2017-10-10 00:28, Eric Blake wrote: > On 10/09/2017 04:55 PM, Max Reitz wrote: >> Some qcow2 functions (at least perform_cow()) expect s->lock to be >> taken. Therefore, if we want to make use of them, we should execute >> preallocate() (as "preallocate_co") in a coroutine so that we can use

Re: [Qemu-block] [Qemu-devel] [PATCH 1/3] qcow2: Fix unaligned preallocated truncation

On 2017-10-10 00:25, Eric Blake wrote: > On 10/09/2017 04:55 PM, Max Reitz wrote: >> A qcow2 image file's length is not required to have a length that is a >> multiple of the cluster size. However, qcow2_refcount_area() expects an >> aligned value for its @start_offset parameter, so we need to

Re: [Qemu-block] [PATCH RFC] file-posix: make lock_fd read-only

Am 11.10.2017 um 11:38 hat Vladimir Sementsov-Ogievskiy geschrieben: > 11.10.2017 12:22, Kevin Wolf wrote: > > [ Cc: Fam ] > > > > Am 10.10.2017 um 15:42 hat Vladimir Sementsov-Ogievskiy geschrieben: > > > We do not reopen lock_fd on bdrv_reopen which leads to problems on > > > reopen image RO.

Re: [Qemu-block] [PATCH RFC] file-posix: make lock_fd read-only

11.10.2017 12:22, Kevin Wolf wrote: [ Cc: Fam ] Am 10.10.2017 um 15:42 hat Vladimir Sementsov-Ogievskiy geschrieben: We do not reopen lock_fd on bdrv_reopen which leads to problems on reopen image RO. So, lets make lock_fd be always RO. This is correct, because qemu_lock_fd always called with

Re: [Qemu-block] [PATCH RFC] file-posix: make lock_fd read-only

[ Cc: Fam ] Am 10.10.2017 um 15:42 hat Vladimir Sementsov-Ogievskiy geschrieben: > We do not reopen lock_fd on bdrv_reopen which leads to problems on > reopen image RO. So, lets make lock_fd be always RO. > This is correct, because qemu_lock_fd always called with exclusive=false > on lock_fd. >

Re: [Qemu-block] [PATCH v2 10/10] nbd: Minimal structured read for client

10.10.2017 18:00, Paolo Bonzini wrote: On 10/10/2017 16:55, Vladimir Sementsov-Ogievskiy wrote: Hmm, would it be simpler just pass a function pointer, which should be called on each loop iteration? So, we will return to one common func nbd_co_receive_reply, but with two additional parameters:

Re: [Qemu-block] [Qemu-devel] [PATCH v5 01/23] block: Allow NULL file for bdrv_get_block_status()

Am 10.10.2017 um 21:24 hat John Snow geschrieben: > > > On 10/10/2017 03:00 PM, Eric Blake wrote: > > On 10/10/2017 09:43 AM, Eric Blake wrote: > > > --- > v5: use second label for cleaner exit logic [John], use local_pnum > >> > @@ -1811,16 +1811,19 @@ static int64_t