[RFC PATCH 07/23] qcow2: Add subcluster-related fields to BDRVQcow2State

2019-10-15 Thread Alberto Garcia
reated as if they had exactly one, with subcluster_size = cluster_size. Signed-off-by: Alberto Garcia --- block/qcow2.c | 5 + block/qcow2.h | 5 + 2 files changed, 10 insertions(+) diff --git a/block/qcow2.c b/block/qcow2.c index 4d16393e61..be9854c5ea 100644 --- a/block/qcow2.c +++ b/block/q

[RFC PATCH 14/23] qcow2: Add subcluster support to qcow2_get_cluster_offset()

2019-10-15 Thread Alberto Garcia
The logic of this function remains pretty much the same, except that it uses count_contiguous_subclusters(), which combines the logic of count_contiguous_clusters() / count_contiguous_clusters_unallocated() and checks individual subclusters. Signed-off-by: Alberto Garcia --- block/qcow2

[RFC PATCH 21/23] qcow2: Add subcluster support to handle_alloc_space()

2019-10-15 Thread Alberto Garcia
with zeroes the other subclusters if we can guarantee that we are not overwriting existing data. However this would waste more disk space, so we should first evaluate if it's really worth doing. Signed-off-by: Alberto Garcia --- block/qcow2.c | 9 + 1 file changed, 5 insertions(+), 4 deletions

[RFC PATCH 18/23] qcow2: Add subcluster support to expand_zero_clusters_in_l1()

2019-10-15 Thread Alberto Garcia
Two changes are needed in order to add subcluster support to this function: deallocated clusters must have their bitmaps cleared, and expanded clusters must have all the "subcluster allocated" bits set. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 2 ++ 1 file changed, 2

[RFC PATCH 02/23] qcow2: Split cluster_needs_cow() out of count_cow_clusters()

2019-10-15 Thread Alberto Garcia
We are going to need it in other places. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 34 +++--- 1 file changed, 19 insertions(+), 15 deletions(-) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index fe2523ed66..f462e169c0 100644 --- a/block

[RFC PATCH 08/23] qcow2: Add offset_to_sc_index()

2019-10-15 Thread Alberto Garcia
For a given offset, return the subcluster number within its cluster (i.e. with 32 subclusters per cluster it returns a number between 0 and 31). Signed-off-by: Alberto Garcia --- block/qcow2.h | 5 + 1 file changed, 5 insertions(+) diff --git a/block/qcow2.h b/block/qcow2.h index

[RFC PATCH 19/23] qcow2: Fix offset calculation in handle_dependencies()

2019-10-15 Thread Alberto Garcia
l2meta_cow_start() and l2meta_cow_end() are not necessarily cluster-aligned if the image has subclusters, so update the calculation of old_start and old_end to guarantee that no two requests try to write on the same cluster. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 4 ++-- 1

[RFC PATCH 22/23] qcow2: Restrict qcow2_co_pwrite_zeroes() to full clusters only

2019-10-15 Thread Alberto Garcia
Ideally it should be possible to zero individual subclusters using this function, but this is currently not implemented. Signed-off-by: Alberto Garcia --- block/qcow2.c | 6 ++ 1 file changed, 6 insertions(+) diff --git a/block/qcow2.c b/block/qcow2.c index c54278ab0b..2eb032aed7 100644

[RFC PATCH 15/23] qcow2: Add subcluster support to zero_in_l2_slice()

2019-10-15 Thread Alberto Garcia
Setting the QCOW_OFLAG_ZERO bit of the L2 entry is forbidden if an image has subclusters. Instead, the individual 'all zeroes' bits must be used. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 14 ++ 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/block

[RFC PATCH 23/23] qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit

2019-10-15 Thread Alberto Garcia
Now that the implementation of subclusters is complete we can finally add the necessary options to create and read images with this feature, which we call "extended L2 entries". Signed-off-by: Alberto Garcia --- block/qcow2.c| 47 ++ blo

[RFC PATCH 20/23] qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()

2019-10-15 Thread Alberto Garcia
The L2 bitmap needs to be updated after each write to indicate what new subclusters are now allocated. This needs to happen even if the cluster was already allocated and the L2 entry was otherwise valid. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 16 1 file

[RFC PATCH 00/23] Add subcluster allocation to qcow2

2019-10-15 Thread Alberto Garcia
s above). Theoretically we could offer a setting to configure this, but I'm not convinced that this is very useful. ======= As usual, feedback is welcome, Berto Alberto Garcia (23): qcow2: Add calculate_l2_meta() qcow2: Split cluster_needs_cow() out of count_cow_clusters() qc

[RFC PATCH 03/23] qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied()

2019-10-15 Thread Alberto Garcia
for subclusters. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 177 +++--- 1 file changed, 96 insertions(+), 81 deletions(-) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index f462e169c0..70b2e32f7e 100644 --- a/block/qcow2-cluster.c

[RFC PATCH 01/23] qcow2: Add calculate_l2_meta()

2019-10-15 Thread Alberto Garcia
handle_alloc() creates a QCowL2Meta structure in order to update the image metadata and perform the necessary copy-on-write operations. This patch moves that code to a separate function so it can be used from other places. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 76

[RFC PATCH 04/23] qcow2: Add get_l2_entry() and set_l2_entry()

2019-10-15 Thread Alberto Garcia
to get_l2_entry() and set_l2_entry(). Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 65 ++ block/qcow2-refcount.c | 17 +-- block/qcow2.h | 12 3 files changed, 55 insertions(+), 39 deletions(-) diff --git a/block/qcow2-cluster.c

[RFC PATCH 09/23] qcow2: Add l2_entry_size()

2019-10-15 Thread Alberto Garcia
. This function returns the proper value for the image. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 12 ++-- block/qcow2-refcount.c | 14 -- block/qcow2.c | 6 +++--- block/qcow2.h | 5 + 4 files changed, 22 insertions(+), 15 deletions(-) diff --git

[RFC PATCH 16/23] qcow2: Add subcluster support to discard_in_l2_slice()

2019-10-15 Thread Alberto Garcia
Setting the QCOW_OFLAG_ZERO bit of the L2 entry is forbidden if an image has subclusters. Instead, the individual 'all zeroes' bits must be used. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 6 +- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/block/qcow2

[RFC PATCH 12/23] qcow2: Handle QCOW2_CLUSTER_UNALLOCATED_SUBCLUSTER

2019-10-15 Thread Alberto Garcia
In the previous patch we added a new QCow2ClusterType named QCOW2_CLUSTER_UNALLOCATED_SUBCLUSTER. There is a couple of places where this new value needs to be handled, and that is what this patch does. Signed-off-by: Alberto Garcia --- block/qcow2.c | 10 +++--- 1 file changed, 7 insertions

[RFC PATCH 06/23] qcow2: Add dummy has_subclusters() function

2019-10-15 Thread Alberto Garcia
value. Signed-off-by: Alberto Garcia --- block/qcow2.h | 6 ++ 1 file changed, 6 insertions(+) diff --git a/block/qcow2.h b/block/qcow2.h index 0b68c55c01..6d6fc57f41 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -485,6 +485,12 @@ typedef enum QCow2MetadataOverlap { #define INV_OFFSET

[RFC PATCH 05/23] qcow2: Document the Extended L2 Entries feature

2019-10-15 Thread Alberto Garcia
-off-by: Alberto Garcia --- docs/interop/qcow2.txt | 68 -- docs/qcow2-cache.txt | 19 +++- 2 files changed, 83 insertions(+), 4 deletions(-) diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt index af5711e533..d34261f955 100644 --- a/docs

[RFC PATCH 10/23] qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap()

2019-10-15 Thread Alberto Garcia
the get/set_l2_bitmap() functions that are used to access the bitmaps. For convenience, these functions are no-ops when used in traditional qcow2 images. Signed-off-by: Alberto Garcia --- block/qcow2.h | 22 ++ 1 file changed, 22 insertions(+) diff --git a/block/qcow2.h b

Re: [PATCH v2] block: Reject misaligned write requests with BDRV_REQ_NO_FALLBACK

2019-10-14 Thread Alberto Garcia
On Mon 14 Oct 2019 11:26:01 AM CEST, Kevin Wolf wrote: >> Signed-off-by: Alberto Garcia > > Thanks, applied to the block branch. I'm a bit late now, but a possible trivial optimization is to flip the conditions, because checking the flag should be faster than checking the alignm

Re: [PATCH] block: Reject misaligned write requests with BDRV_REQ_NO_FALLBACK

2019-10-14 Thread Alberto Garcia
On Mon 14 Oct 2019 12:11:52 PM CEST, Vladimir Sementsov-Ogievskiy wrote: > 13.10.2019 23:48, Alberto Garcia wrote: >> The BDRV_REQ_NO_FALLBACK flag means that an operation should only be >> performed if it can be offloaded or otherwise performed efficiently. > > As I know,

[PATCH v2] block: Reject misaligned write requests with BDRV_REQ_NO_FALLBACK

2019-10-14 Thread Alberto Garcia
h +# +# Test write request with required alignment larger than the cluster size +# +# Copyright (C) 2019 Igalia, S.L. +# Author: Alberto Garcia +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# th

[PATCH] block: Reject misaligned write requests with BDRV_REQ_NO_FALLBACK

2019-10-13 Thread Alberto Garcia
n `!(flags & BDRV_REQ_NO_FALLBACK)' failed. Aborted Signed-off-by: Alberto Garcia --- block/io.c | 6 + tests/qemu-iotests/268 | 55 ++ tests/qemu-iotests/268.out | 7 + tests/qemu-iotests/group | 1 + 4 files changed, 69 insertions(+

Re: [Qemu-block] [PATCH] qcow2: Stop overwriting compressed clusters one by one

2019-09-12 Thread Alberto Garcia
On Thu 12 Sep 2019 01:33:05 AM CEST, John Snow wrote: >> This restriction comes from commit 095a9c58ce12afeeb90c2 from 2018. > > You accidentally typed a reasonably modern date. It's from *2008*! Oh my, and I reviewed the message 3 times ... if this one gets committed please correct the date.

[Qemu-block] [PATCH] qcow2: Stop overwriting compressed clusters one by one

2019-09-11 Thread Alberto Garcia
in one go: qemu-img create -f qcow2 hd.qcow2 64M qemu-io -c "write -z 0 64k" hd.qcow2 for f in `seq 64 64 65472`; do qemu-io -c "write -c ${f}k 64k" hd.qcow2 done Compared to the previous one, overwriting this image on my computer goes from 8.35s down

Re: [Qemu-block] [PATCH v9 3/9] block: add empty account cookie type

2019-09-09 Thread Alberto Garcia
On Fri 06 Sep 2019 06:01:14 PM CEST, Anton Nefedov wrote: > This adds some protection from accounting uninitialized cookie. > That is, block_acct_failed/done without previous block_acct_start; > in that case, cookie probably holds values from previous operation. > > (Note: it might also be

Re: [Qemu-block] [PATCH] qcow2: Fix the calculation of the maximum L2 cache size

2019-08-16 Thread Alberto Garcia
On Fri 16 Aug 2019 04:08:19 PM CEST, Kevin Wolf wrote: >> And yes, the odd value on the 512KB row on that we discussed last month >> is due to this same bug: >> >> https://lists.gnu.org/archive/html/qemu-block/2019-07/msg00496.html > > Hm... And suddently it makes sense. :-) > > So I assume all

Re: [Qemu-block] [PATCH] qcow2: Fix the calculation of the maximum L2 cache size

2019-08-16 Thread Alberto Garcia
On Fri 16 Aug 2019 02:59:21 PM CEST, Kevin Wolf wrote: > The requirement so that this bug doesn't affect the user seems to be > that the image size is a multiple of 64k * 8k = 512 MB. Which means > that users are probably often lucky enough in practice. Or rather: cluster_size^2 / 8, which, if my

Re: [Qemu-block] [PATCH] qcow2: Fix the calculation of the maximum L2 cache size

2019-08-16 Thread Alberto Garcia
Cc qemu-stable This bug means that under certain conditions it's impossible to create a cache large enough for the image, resulting in reduced I/O performance. On Fri, Aug 16, 2019 at 03:17:42PM +0300, Alberto Garcia wrote: > The size of the qcow2 L2 cache defaults to 32 MB, which can be eas

[Qemu-block] [PATCH] qcow2: Fix the calculation of the maximum L2 cache size

2019-08-16 Thread Alberto Garcia
ver QEMU rounds the numbers down and only creates 2 cache tables (128 KB), which is not enough for the image. A quick test doing 4KB random writes on a 1280 MB image gives me around 500 IOPS, while with the correct cache size I get 16K IOPS. Signed-off-by: Alberto Garcia --- block/qcow2.c

Re: [Qemu-block] [PATCH 1/3] test-throttle: Fix uninitialized use of burst_length

2019-08-13 Thread Alberto Garcia
> Signed-off-by: Andrey Shinkevich Reviewed-by: Alberto Garcia Berto

Re: [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-07-11 Thread Alberto Garcia
On Thu 11 Jul 2019 04:32:34 PM CEST, Kevin Wolf wrote: >> - It is possible to configure very easily the number of subclusters per >> cluster. It is now hardcoded to 32 in qcow2_do_open() but any power of >> 2 would work (just change the number there if you want to test >> it). Would an

Re: [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-07-11 Thread Alberto Garcia
On Thu 27 Jun 2019 07:08:29 PM CEST, Denis Lunev wrote: > But can we get a link to the repo with actual version of patches. Hi, I updated my code to increase the L2 entry size from 64 bits to 128 bits and thanks to this we now have 32 subclusters per cluster (32 bits for "subcluster allocated"

Re: [Qemu-block] [PATCH 1/5] block: Add BDS.never_freeze

2019-07-02 Thread Alberto Garcia
oming frozen. > > Signed-off-by: Max Reitz Reviewed-by: Alberto Garcia Berto

Re: [Qemu-block] [PATCH 2/5] iotests: Fix throttling in 030

2019-07-02 Thread Alberto Garcia
es the exact error messages > and just checks the error class is something that should be fixed in a > follow-up patch. > > Signed-off-by: Max Reitz Reviewed-by: Alberto Garcia Berto

Re: [Qemu-block] [PATCH 5/5] iotests: Add new case to 030

2019-07-02 Thread Alberto Garcia
On Fri 28 Jun 2019 12:32:55 AM CEST, Max Reitz wrote: > We recently removed the dependency of the stream job on its base node. > That makes it OK to use a commit filter node there. Test that. > > Signed-off-by: Max Reitz Reviewed-by: Alberto Garcia Berto

Re: [Qemu-block] [PATCH 1/5] block: Add BDS.never_freeze

2019-07-02 Thread Alberto Garcia
On Fri 28 Jun 2019 12:32:51 AM CEST, Max Reitz wrote: > @@ -4416,6 +4416,14 @@ int bdrv_freeze_backing_chain(BlockDriverState *bs, > BlockDriverState *base, > return -EPERM; > } > > +for (i = bs; i != base; i = backing_bs(i)) { > +if (i->backing &&

Re: [Qemu-block] [PATCH 3/5] iotests: Compare error messages in 030

2019-07-02 Thread Alberto Garcia
as the > commit and stream job were not allowed to overlap. > > Prevent such problems in the future by comparing the error description > instead. > > Signed-off-by: Max Reitz Reviewed-by: Alberto Garcia Berto

Re: [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Alberto Garcia
On Thu 27 Jun 2019 07:08:29 PM CEST, Denis Lunev wrote: > But can we get a link to the repo with actual version of patches. It's not in a state that can be published at the moment, but I'll try to have something available soon. Berto

Re: [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Alberto Garcia
On Fri 28 Jun 2019 04:16:28 PM CEST, Kevin Wolf wrote: >> >> >> >> I would consider 64k cluster/8k subcluster as too extreme >> >> >> >> for me. >> >> >> >> >> >> I forgot to add: this 64k/8k ratio is only with my current >> >> >> prototype. >> >> >> >> >> >> In practice if we go with the

Re: [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Alberto Garcia
On Fri 28 Jun 2019 05:09:11 PM CEST, Kevin Wolf wrote: > Am 28.06.2019 um 17:02 hat Alberto Garcia geschrieben: >> On Fri 28 Jun 2019 04:57:08 PM CEST, Kevin Wolf wrote: >> > Am 28.06.2019 um 16:43 hat Alberto Garcia geschrieben: >> >> On Thu 27 Jun 2019 06:05

Re: [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Alberto Garcia
On Fri 28 Jun 2019 04:57:08 PM CEST, Kevin Wolf wrote: > Am 28.06.2019 um 16:43 hat Alberto Garcia geschrieben: >> On Thu 27 Jun 2019 06:05:55 PM CEST, Denis Lunev wrote: >> > Please note, I am not talking now about your case with COW. Here the >> > allocation is perform

Re: [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Alberto Garcia
On Fri 28 Jun 2019 05:03:13 PM CEST, Denis Lunev wrote: > On 6/28/19 6:02 PM, Alberto Garcia wrote: >>>>> Please note, I am not talking now about your case with COW. Here the >>>>> allocation is performed on the sub-cluster basis, i.e. the abscence of >>>

Re: [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Alberto Garcia
On Thu 27 Jun 2019 06:05:55 PM CEST, Denis Lunev wrote: >>> Thus in respect to this patterns subclusters could give us benefits >>> of fast random IO and good reclaim rate. >> Exactly, but that fast random I/O would only happen when allocating >> new clusters. Once the clusters are allocated it

Re: [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Alberto Garcia
I pressed "send" too early, here's the last part of my reply: On Fri 28 Jun 2019 02:57:56 PM CEST, Alberto Garcia wrote: >> I also ran some tests on a rotating HDD drive. Here having >> subclusters doesn't make a big difference regardless of whether there >> is a bac

Re: [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Alberto Garcia
On Fri 28 Jun 2019 12:04:22 PM CEST, Kevin Wolf wrote: > Am 28.06.2019 um 11:53 hat Alberto Garcia geschrieben: >> On Fri 28 Jun 2019 11:20:57 AM CEST, Kevin Wolf wrote: >> >> >> I would consider 64k cluster/8k subcluster as too extreme for me. >> >> &

Re: [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Alberto Garcia
On Thu 27 Jun 2019 06:54:34 PM CEST, Kevin Wolf wrote: >> |-++-| >> | Cluster size | subclusters=on | subclusters=off | >> |-++-| >> | 2 MB (256 KB) | 571 IOPS | 124 IOPS | >> | 1 MB

Re: [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Alberto Garcia
On Fri 28 Jun 2019 11:20:57 AM CEST, Kevin Wolf wrote: >> >> I would consider 64k cluster/8k subcluster as too extreme for me. >> >> I forgot to add: this 64k/8k ratio is only with my current prototype. >> >> In practice if we go with the 128-bit L2 entries we would have 64 >> subclusters per

Re: [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-27 Thread Alberto Garcia
On Thu 27 Jun 2019 05:38:56 PM CEST, Alberto Garcia wrote: >> I would consider 64k cluster/8k subcluster as too extreme for me. I forgot to add: this 64k/8k ratio is only with my current prototype. In practice if we go with the 128-bit L2 entries we would have 64 subclusters per cluster,

Re: [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-27 Thread Alberto Garcia
On Thu 27 Jun 2019 04:19:25 PM CEST, Denis Lunev wrote: > Right now QCOW2 is not very efficient with default cluster size (64k) > for fast performance with big disks. Nowadays ppl uses really BIG > images and 1-2-3-8 Tb disks are really common. Unfortunately ppl want > to get random IO fast too.

[Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-27 Thread Alberto Garcia
Hi all, a couple of years ago I came to the mailing list with a proposal to extend the qcow2 format to add subcluster allocation. You can read the original message (and the discussion thread that came afterwards) here: https://lists.gnu.org/archive/html/qemu-block/2017-04/msg00178.html The

Re: [Qemu-block] [PATCH v2 0/2] blockdev: Overlays are not snapshots

2019-06-04 Thread Alberto Garcia
that > operation; if “Creating a snapshot” creates a file, that file must be > the snapshot, right? Well, no, it isn’t.) > > Let’s fix this as best as we can. Better Nate than lever. Reviewed-by: Alberto Garcia Berto

Re: [Qemu-block] [RFC 1/3] block: Add ImageRotationalInfo

2019-05-27 Thread Alberto Garcia
On Mon 27 May 2019 03:44:59 PM CEST, Anton Nefedov wrote: >> I think in general optimizing for SSDs should probably be the >> default. HDDs are slow anyway, so whoever uses them probably doesn’t >> care about performance too much anyway...? Whereas people using SSDs >> probably do. But as I

Re: [Qemu-block] [RFC 1/3] block: Add ImageRotationalInfo

2019-05-27 Thread Alberto Garcia
On Mon 27 May 2019 02:16:53 PM CEST, Max Reitz wrote: > On 26.05.19 17:08, Alberto Garcia wrote: >> On Fri 24 May 2019 07:28:10 PM CEST, Max Reitz wrote: >>> +## >>> +# @ImageRotationalInfo: >>> +# >>> +# Indicates whether an image is stored on a rot

Re: [Qemu-block] [RFC 1/3] block: Add ImageRotationalInfo

2019-05-26 Thread Alberto Garcia
On Fri 24 May 2019 07:28:10 PM CEST, Max Reitz wrote: > +## > +# @ImageRotationalInfo: > +# > +# Indicates whether an image is stored on a rotating disk or not. > +# > +# @solid-state: Image is stored on a solid-state drive > +# > +# @rotating:Image is stored on a rotating disk What happens

Re: [Qemu-block] [PATCH v14 0/1] qcow2: cluster space preallocation

2019-05-26 Thread Alberto Garcia
On Fri 24 May 2019 03:56:21 PM CEST, Max Reitz wrote: >> +---+---+--+---+--+--+ >> | file|before| after| gain | >> +---+---+--+---+--+--+ >> |ssd| 61.153 | 36.313 | 41% | >> |

Re: [Qemu-block] [PATCH v3 1/2] block: Use bdrv_unref_child() for all children in bdrv_close()

2019-05-23 Thread Alberto Garcia
On Mon 13 May 2019 03:46:17 PM CEST, Alberto Garcia wrote: > Now bdrv_close() unrefs all children (before this patch it was only > bs->file and bs->backing). As a result, none of the callers of > brvd_attach_child() should remove their reference to child_bs (because > this fu

Re: [Qemu-block] [PATCH v14 1/1] qcow2: skip writing zero buffers to empty COW areas

2019-05-22 Thread Alberto Garcia
perform_cow(). > > iotest 060: > write to the discarded cluster does not trigger COW anymore. > Use a backing image instead. > > Signed-off-by: Anton Nefedov Reviewed-by: Alberto Garcia Berto

Re: [Qemu-block] [PATCH v2] [RFC] qcow2: add compression type feature

2019-05-22 Thread Alberto Garcia
On Mon 20 May 2019 08:20:23 AM CEST, Denis Plotnikov wrote: > +/* Compression type extension */ > +if (s->compression_type != 0) { > +Qcow2CompressionTypeExt comp_header = { > +.compression_type = cpu_to_be32(s->compression_type), > +}; Shouldn't we have an

Re: [Qemu-block] [PATCH 4/4] iotests: Make 245 faster and more reliable

2019-05-16 Thread Alberto Garcia
rate limiting) for the > active commit job, because It never completes without an explicit > block-job-complete anyway. > > Signed-off-by: Max Reitz Reviewed-by: Alberto Garcia Berto

Re: [Qemu-block] [PATCH 3/4] iotests.py: Fix VM.run_job

2019-05-16 Thread Alberto Garcia
On Wed 15 May 2019 10:15:02 PM CEST, Max Reitz wrote: > log() is in the current module, there is no need to prefix it. In fact, > doing so may make VM.run_job() unusable in tests that never use > iotests.log() themselves. > > Signed-off-by: Max Reitz Reviewed-by: Alberto Garcia Berto

Re: [Qemu-block] [PATCH 2/4] iotests.py: Let assert_qmp() accept an array

2019-05-16 Thread Alberto Garcia
f-by: Max Reitz Reviewed-by: Alberto Garcia Berto

Re: [Qemu-block] [PATCH 1/4] block: Improve "Block node is read-only" message

2019-05-16 Thread Alberto Garcia
On Wed 15 May 2019 10:15:00 PM CEST, Max Reitz wrote: > This message does not make any sense when it appears as the response to > making an R/W node read-only. We should detect that case and emit a > different message, then. > > Signed-off-by: Max Reitz Reviewed-by: Alberto Garcia Berto

[Qemu-block] [PATCH] block: Use BDRV_REQUEST_MAX_BYTES instead of BDRV_REQUEST_MAX_SECTORS

2019-05-14 Thread Alberto Garcia
There are a few places in which we turn a number of bytes into sectors in order to compare the result against BDRV_REQUEST_MAX_SECTORS instead of using BDRV_REQUEST_MAX_BYTES directly. Signed-off-by: Alberto Garcia --- block/io.c | 6 +++--- qemu-io-cmds.c | 7 +++ 2 files changed, 6

Re: [Qemu-block] [PATCH v2 13/13] qemu-iotests: Test the x-blockdev-reopen QMP command

2019-05-14 Thread Alberto Garcia
On Tue 14 May 2019 03:02:33 PM CEST, Max Reitz wrote: >> Why would both be fine? backing_file refers to the backing file >> currently attached, and auto_backing_file refers to the one written >> on the image metadata, or am I wrong? > > After my series, backing_file refers to the image metadata.

Re: [Qemu-block] [PATCH v2 13/13] qemu-iotests: Test the x-blockdev-reopen QMP command

2019-05-14 Thread Alberto Garcia
On Sat 13 Apr 2019 02:53:42 AM CEST, Max Reitz wrote: >> Calling x-blockdev-reopen without 'backing' should only fail if >> >> a) the image has a backing file attached to it. >> In this case it doesn't: we just detached it in the previous line. >> >> b) there's a default backing file

Re: [Qemu-block] [PATCH v3 2/2] block: Make bdrv_root_attach_child() unref child_bs on failure

2019-05-13 Thread Alberto Garcia
On Mon 13 May 2019 04:31:16 PM CEST, Max Reitz wrote: >> @@ -2569,10 +2582,6 @@ BdrvChild *bdrv_open_child(const char *filename, >> } >> >> c = bdrv_attach_child(parent, bs, bdref_key, child_role, errp); >> -if (!c) { >> -bdrv_unref(bs); >> -return NULL; >> -}

[Qemu-block] [PATCH v3 2/2] block: Make bdrv_root_attach_child() unref child_bs on failure

2019-05-13 Thread Alberto Garcia
-by: Alberto Garcia --- block.c | 25 + block/block-backend.c | 3 +-- block/quorum.c| 1 - blockjob.c| 2 +- 4 files changed, 19 insertions(+), 12 deletions(-) diff --git a/block.c b/block.c index 3c3bd0f8d2..df727314ff 100644

[Qemu-block] [PATCH v3 0/2] block: Use bdrv_unref_child() for all children in bdrv_close()

2019-05-13 Thread Alberto Garcia
/archive/html/qemu-block/2019-03/msg01040.html Alberto Garcia (2): block: Use bdrv_unref_child() for all children in bdrv_close() block: Make bdrv_root_attach_child() unref child_bs on failure block.c | 41 - block/block-backend.c

[Qemu-block] [PATCH v3 1/2] block: Use bdrv_unref_child() for all children in bdrv_close()

2019-05-13 Thread Alberto Garcia
g). As a result, none of the callers of brvd_attach_child() should remove their reference to child_bs (because this function effectively steals that reference). This patch updates a couple of tests that where doing their own bdrv_unref(). Signed-off-by: Alberto Garcia --- block.c

Re: [Qemu-block] [Qemu-devel] [PATCH] qcow2: Define and use QCOW2_COMPRESSED_SECTOR_SIZE

2019-05-13 Thread Alberto Garcia
On Mon 13 May 2019 03:14:40 PM CEST, Stefano Garzarella wrote: >> Since the maximum allowed cluster size is 2MB, the value of the >> 'size' variable can never be larger than 4MB, which fits comfortably >> on a 32-bit integer. We would need to support 512MB clusters in order >> to have problems

Re: [Qemu-block] [Qemu-devel] [PATCH] qcow2: Define and use QCOW2_COMPRESSED_SECTOR_SIZE

2019-05-13 Thread Alberto Garcia
On Mon 13 May 2019 01:28:46 PM CEST, Stefano Garzarella wrote: >> +int size = QCOW2_COMPRESSED_SECTOR_SIZE * >> +(((l2_entry >> s->csize_shift) & s->csize_mask) + 1); > > What about using int64_t type for the 'size' variable? > (because the qcow2_free_clusters() 'size'

[Qemu-block] [PATCH] qcow2: Define and use QCOW2_COMPRESSED_SECTOR_SIZE

2019-05-10 Thread Alberto Garcia
-off-by: Alberto Garcia --- block/qcow2-cluster.c | 5 +++-- block/qcow2-refcount.c | 25 ++--- block/qcow2.c | 3 ++- block/qcow2.h | 4 4 files changed, 23 insertions(+), 14 deletions(-) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c

Re: [Qemu-block] [PATCH v6 6/8] qcow2: qcow2_co_preadv: improve locking

2019-05-09 Thread Alberto Garcia
king, so reduce locking to it. > > Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Alberto Garcia Berto

[Qemu-block] [PATCH v2] block: Use bdrv_unref_child() for all children in bdrv_close()

2019-05-08 Thread Alberto Garcia
ot to use bdrv_unref_child() in bdrv_close() anymore. After this there's also no need to remove bs->backing and bs->file separately from the rest of the children, so bdrv_close() can be simplified. This patch also updates a couple of tests that were doing their own bdrv_unref(). Signed-off-by:

[Qemu-block] [PATCH] qcow2: Assert that host cluster offsets fit in L2 table entries

2019-05-03 Thread Alberto Garcia
and size of the compressed data) and the situation with them is similar. In this case the masks are not constant but are stored in the csize_mask and cluster_offset_mask fields of BDRVQcow2State. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 14 -- 1 file changed, 12 insertions

[Qemu-block] [PATCH v2 5/5] qcow2: Remove BDRVQcow2State.cluster_sectors

2019-05-01 Thread Alberto Garcia
The last user of this field disappeared when we replace the sector-based bdrv_write() with the byte-based bdrv_pwrite(). Signed-off-by: Alberto Garcia --- block/qcow2.c | 1 - block/qcow2.h | 1 - 2 files changed, 2 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index a520d116ef

[Qemu-block] [PATCH v2 4/5] block: Remove bdrv_read() and bdrv_write()

2019-05-01 Thread Alberto Garcia
No one is using these functions anymore, all callers have switched to the byte-based bdrv_pread() and bdrv_pwrite() Signed-off-by: Alberto Garcia --- block/io.c| 43 +++ include/block/block.h | 4 2 files changed, 7 insertions(+), 40

[Qemu-block] [PATCH v2 1/5] qcow2: Replace bdrv_write() with bdrv_pwrite()

2019-05-01 Thread Alberto Garcia
There's only one bdrv_write() call left in the qcow2 code, and it can be trivially replaced with the byte-based bdrv_pwrite(). Signed-off-by: Alberto Garcia --- block/qcow2-refcount.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/block/qcow2-refcount.c b/block/qcow2

[Qemu-block] [PATCH v2 0/5] Remove bdrv_read() and bdrv_write()

2019-05-01 Thread Alberto Garcia
ite()' 002/5:[0006] [FC] 'vdi: Replace bdrv_{read,write}() with bdrv_{pread,pwrite}()' 003/5:[0004] [FC] 'vvfat: Replace bdrv_{read,write}() with bdrv_{pread,pwrite}()' 004/5:[0007] [FC] 'block: Remove bdrv_read() and bdrv_write()' 005/5:[] [--] 'qcow2: Remove BDRVQcow2State.cluster_sectors' Albe

[Qemu-block] [PATCH v2 2/5] vdi: Replace bdrv_{read, write}() with bdrv_{pread, pwrite}()

2019-05-01 Thread Alberto Garcia
There's only a couple of bdrv_read() and bdrv_write() calls left in the vdi code, and they can be trivially replaced with the byte-based bdrv_pread() and bdrv_pwrite(). Signed-off-by: Alberto Garcia --- block/vdi.c | 15 +-- 1 file changed, 9 insertions(+), 6 deletions(-) diff

[Qemu-block] [PATCH v2 3/5] vvfat: Replace bdrv_{read, write}() with bdrv_{pread, pwrite}()

2019-05-01 Thread Alberto Garcia
There's only a couple of bdrv_read() and bdrv_write() calls left in the vvfat code, and they can be trivially replaced with the byte-based bdrv_pread() and bdrv_pwrite(). Signed-off-by: Alberto Garcia --- block/vvfat.c | 12 +++- 1 file changed, 7 insertions(+), 5 deletions(-) diff

[Qemu-block] [PATCH] qcow2: Fix error handling in the compression code

2019-04-30 Thread Alberto Garcia
from errno.h. c) Both qcow2_compress() and qcow2_co_do_compress() return a negative value on failure, but qcow2_co_pwritev_compressed() stores the value in an unsigned data type. Signed-off-by: Alberto Garcia --- block/qcow2.c | 18 +- 1 file changed, 9 insertions(+), 9

[Qemu-block] [PATCH 2/5] vdi: Replace bdrv_{read, write}() with bdrv_{pread, pwrite}()

2019-04-29 Thread Alberto Garcia
There's only a couple of bdrv_read() and bdrv_write() calls left in the vdi code, and they can be trivially replaced with the byte-based bdrv_pread() and bdrv_pwrite(). Signed-off-by: Alberto Garcia --- block/vdi.c | 11 ++- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git

[Qemu-block] [PATCH 5/5] qcow2: Remove BDRVQcow2State.cluster_sectors

2019-04-29 Thread Alberto Garcia
The last user of this field disappeared when we replace the sector-based bdrv_write() with the byte-based bdrv_pwrite(). Signed-off-by: Alberto Garcia --- block/qcow2.c | 1 - block/qcow2.h | 1 - 2 files changed, 2 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index 3ace3b2209

[Qemu-block] [PATCH 0/5] Remove bdrv_read() and bdrv_write()

2019-04-29 Thread Alberto Garcia
Hi, this API only had a few users left so it can be easily removed. Regards, Berto Alberto Garcia (5): qcow2: Replace bdrv_write() with bdrv_pwrite() vdi: Replace bdrv_{read,write}() with bdrv_{pread,pwrite}() vvfat: Replace bdrv_{read,write}() with bdrv_{pread,pwrite}() block: Remove

[Qemu-block] [PATCH 1/5] qcow2: Replace bdrv_write() with bdrv_pwrite()

2019-04-29 Thread Alberto Garcia
There's only one bdrv_write() call left in the qcow2 code, and it can be trivially replaced with the byte-based bdrv_pwrite(). Signed-off-by: Alberto Garcia --- block/qcow2-refcount.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/block/qcow2-refcount.c b/block/qcow2

[Qemu-block] [PATCH 4/5] block: Remove bdrv_read() and bdrv_write()

2019-04-29 Thread Alberto Garcia
No one is using these functions anymore, all callers have switched to the byte-based bdrv_pread() and bdrv_pwrite() Signed-off-by: Alberto Garcia --- block/io.c| 36 include/block/block.h | 4 2 files changed, 40 deletions(-) diff --git

[Qemu-block] [PATCH 3/5] vvfat: Replace bdrv_{read, write}() with bdrv_{pread, pwrite}()

2019-04-29 Thread Alberto Garcia
There's only a couple of bdrv_read() and bdrv_write() calls left in the vvfat code, and they can be trivially replaced with the byte-based bdrv_pread() and bdrv_pwrite(). Signed-off-by: Alberto Garcia --- block/vvfat.c | 10 ++ 1 file changed, 6 insertions(+), 4 deletions(-) diff --git

[Qemu-block] [PATCH v3 2/2] iotests: Check that images are in read-only mode after block-commit

2019-04-29 Thread Alberto Garcia
This tests the fix from the previous patch. Signed-off-by: Alberto Garcia --- tests/qemu-iotests/249 | 115 + tests/qemu-iotests/249.out | 35 ++ tests/qemu-iotests/group | 1 + 3 files changed, 151 insertions(+) create mode

[Qemu-block] [PATCH v3 0/2] commit: Make base read-only if there is an early failure

2019-04-29 Thread Alberto Garcia
is downstream-only The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively Alberto Garcia (2): commit: Make base read-only if there is an early failure iotests: Check that images are in read-only mode after block-commit block/commit.c | 3 ++ tests/qemu

[Qemu-block] [PATCH v3 1/2] commit: Make base read-only if there is an early failure

2019-04-29 Thread Alberto Garcia
You can reproduce this by passing an invalid filter-node-name (like "1234") to block-commit. In this case the base image is put in read-write mode but is never reset back to read-only. Signed-off-by: Alberto Garcia Reviewed-by: Max Reitz --- block/commit.c | 3 +++ 1 file changed, 3

Re: [Qemu-block] [PATCH for-4.1 v2 0/2] commit: Make base read-only if there is an early failure

2019-04-26 Thread Alberto Garcia
ping On Thu 11 Apr 2019 02:32:26 PM CEST, Alberto Garcia wrote: > Hi, > > this is the same patch I posted yesterday, but with a test case. > > Berto > > Alberto Garcia (2): > commit: Make base read-only if there is an early failure > iotests: Check that images

Re: [Qemu-block] [PATCH for-4.1] block: Use bdrv_unref_child() for all children in bdrv_close()

2019-04-26 Thread Alberto Garcia
ping On Sun 31 Mar 2019 01:17:47 PM CEST, Alberto Garcia wrote: > bdrv_unref_child() does the following things: > > - Updates the child->bs->inherits_from pointer. > - Calls bdrv_detach_child() to remove the BdrvChild from bs->children. > - Calls bdrv_un

Re: [Qemu-block] [PATCH for-4.1] commit: Use bdrv_append() in commit_start()

2019-04-26 Thread Alberto Garcia
ping On Wed 03 Apr 2019 04:37:48 PM CEST, Alberto Garcia wrote: > This function combines bdrv_set_backing_hd() and bdrv_replace_node() > so we can use it to simplify the code a bit in commit_start(). > > Signed-off-by: Alberto Garcia > --- > block/commit.c | 11 +-- &g

Re: [Qemu-block] [PATCH for-4.1] block: Assert that drv->bdrv_child_perm is set in bdrv_child_perm()

2019-04-26 Thread Alberto Garcia
ping On Thu 04 Apr 2019 01:29:53 PM CEST, Alberto Garcia wrote: > There is no need to check for this because all block drivers that have > children implement bdrv_child_perm and all callers already ensure that > bs->drv is set. > > Furthermore, if this check would fail then the

Re: [Qemu-block] [PATCH v5 1/3] block: include base when checking image chain for block allocation

2019-04-25 Thread Alberto Garcia
On Mon 22 Apr 2019 12:18:58 PM CEST, Vladimir Sementsov-Ogievskiy wrote: > * Given an image chain: ... -> [BASE] -> [INTER1] -> [INTER2] -> [TOP] > * > * Return true if (a prefix of) the given range is allocated in any image > - * between BASE and TOP (inclusive). BASE can be NULL to check

Re: [Qemu-block] [PATCH] iotests: Make 182 do without device_add

2019-04-25 Thread Alberto Garcia
fda0082b00ae963 > Reported-by: Danilo C. L. de Paula > Signed-off-by: Max Reitz Acked-by: Alberto Garcia Berto

<    4   5   6   7   8   9   10   11   12   13   >