[PATCH 2/3] qcow2: Don't round the L1 table allocation up to the sector size

The L1 table is read from disk using the byte-based bdrv_pread() and is never accessed beyond its last element, so there's no need to allocate more memory than that. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 5 ++--- block/qcow2-refcount.c | 2 +- block/qcow2-snapshot.c | 3

[PATCH 1/3] qcow2: Require that the virtual size is a multiple of the sector size

no good use case that would require us to try to handle them so let's just treat them as unsupported. Signed-off-by: Alberto Garcia --- block/qcow2.c | 7 +++ docs/interop/qcow2.txt | 3 ++- tests/qemu-iotests/080 | 7 +++ tests/qemu-iotests/080.out | 4 4 files

[PATCH 0/3] qcow2: Misc BDRV_SECTOR_SIZE updates

This small series gets rid of all the remaining instances of hardcoded sector sizes in the qcow2 code and adds a check for images whose virtual size is not a multiple of the sector size. See the individual patches for details. Berto Alberto Garcia (3): qcow2: Require that the virtual size

[PATCH 3/3] qcow2: Use BDRV_SECTOR_SIZE instead of the hardcoded value

This replaces all remaining instances in the qcow2 code. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 2 +- block/qcow2.c | 16 +--- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index 932fc48919

[RFC PATCH v3 12/27] qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*

of QCow2ClusterType with their QCow2SubclusterType equivalents (as returned by qcow2_cluster_to_subcluster_type()). This patch only changes the data types, there are no semantic changes. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 19 +- block/qcow2.c | 82

[RFC PATCH v3 26/27] qcow2: Add subcluster support to qcow2_measure()

Extended L2 entries are bigger than normal L2 entries so this has an impact on the amount of metadata needed for a qcow2 file. Signed-off-by: Alberto Garcia --- block/qcow2.c | 19 --- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c

[RFC PATCH v3 25/27] qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit

Now that the implementation of subclusters is complete we can finally add the necessary options to create and read images with this feature, which we call "extended L2 entries". Signed-off-by: Alberto Garcia --- block/qcow2.c| 65 ++-- blo

[RFC PATCH v3 23/27] qcow2: Add subcluster support to handle_alloc_space()

with zeroes the other subclusters if we can guarantee that we are not overwriting existing data. However this would waste more disk space, so we should first evaluate if it's really worth doing. Signed-off-by: Alberto Garcia --- block/qcow2.c | 9 + 1 file changed, 5 insertions(+), 4 deletions

[RFC PATCH v3 21/27] qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()

The L2 bitmap needs to be updated after each write to indicate what new subclusters are now allocated. This needs to happen even if the cluster was already allocated and the L2 entry was otherwise valid. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 17 + 1 file

[RFC PATCH v3 06/27] qcow2: Add dummy has_subclusters() function

value. Signed-off-by: Alberto Garcia --- block/qcow2.h | 6 ++ 1 file changed, 6 insertions(+) diff --git a/block/qcow2.h b/block/qcow2.h index 6823d3f68f..1db3fc5dbc 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -495,6 +495,12 @@ typedef enum QCow2MetadataOverlap { #define INV_OFFSET

[RFC PATCH v3 16/27] qcow2: Add subcluster support to zero_in_l2_slice()

Setting the QCOW_OFLAG_ZERO bit of the L2 entry is forbidden if an image has subclusters. Instead, the individual 'all zeroes' bits must be used. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 14 ++ 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/block

[RFC PATCH v3 10/27] qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap()

the get/set_l2_bitmap() functions that are used to access the bitmaps. For convenience we allow calling get_l2_bitmap() on images without subclusters, although the caller does not need and should ignore the returned value. Signed-off-by: Alberto Garcia --- block/qcow2.h | 22

[RFC PATCH v3 18/27] qcow2: Add subcluster support to check_refcounts_l2()

Setting the QCOW_OFLAG_ZERO bit of the L2 entry is forbidden if an image has subclusters. Instead, the individual 'all zeroes' bits must be used. Signed-off-by: Alberto Garcia --- block/qcow2-refcount.c | 9 +++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/block/qcow2

[RFC PATCH v3 09/27] qcow2: Add l2_entry_size()

. This function returns the proper value for the image. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 12 ++-- block/qcow2-refcount.c | 14 -- block/qcow2.c | 8 block/qcow2.h | 9 + 4 files changed, 27 insertions(+), 16 deletions(-) diff

[RFC PATCH v3 14/27] qcow2: Add subcluster support to calculate_l2_meta()

-zeroes bit set then we need copy-on-write on subcluster #3. - If we are overwriting an old cluster and subcluster #3 was allocated then there is no need to copy-on-write. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 140 +- 1 file

[RFC PATCH v3 05/27] qcow2: Document the Extended L2 Entries feature

-off-by: Alberto Garcia --- docs/interop/qcow2.txt | 68 -- docs/qcow2-cache.txt | 19 +++- 2 files changed, 83 insertions(+), 4 deletions(-) diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt index af5711e533..d34261f955 100644 --- a/docs

[RFC PATCH v3 20/27] qcow2: Fix offset calculation in handle_dependencies()

l2meta_cow_start() and l2meta_cow_end() are not necessarily cluster-aligned if the image has subclusters, so update the calculation of old_start and old_end to guarantee that no two requests try to write on the same cluster. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 4 ++-- 1

[RFC PATCH v3 15/27] qcow2: Add subcluster support to qcow2_get_cluster_offset()

The logic of this function remains pretty much the same, except that it uses count_contiguous_subclusters(), which combines the logic of count_contiguous_clusters() / count_contiguous_clusters_unallocated() and checks individual subclusters. Signed-off-by: Alberto Garcia --- block/qcow2

[RFC PATCH v3 03/27] qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied()

for subclusters. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 226 +++--- 1 file changed, 126 insertions(+), 100 deletions(-) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index e078bddcc2..9387f15866 100644 --- a/block/qcow2-cluster.c

[RFC PATCH v3 27/27] iotests: Add tests for qcow2 images with extended L2 entries

Signed-off-by: Alberto Garcia --- tests/qemu-iotests/271 | 256 + tests/qemu-iotests/271.out | 208 ++ tests/qemu-iotests/group | 1 + 3 files changed, 465 insertions(+) create mode 100755 tests/qemu-iotests/271 create

[RFC PATCH v3 04/27] qcow2: Add get_l2_entry() and set_l2_entry()

to get_l2_entry() and set_l2_entry(). Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 65 ++ block/qcow2-refcount.c | 17 +-- block/qcow2.h | 12 3 files changed, 55 insertions(+), 39 deletions(-) diff --git a/block/qcow2-cluster.c

[RFC PATCH v3 00/27] Add subcluster allocation to qcow2

/27:[down] 'qcow2: Add subcluster support to qcow2_measure()' 027/27:[0046] [FC] 'iotests: Add tests for qcow2 images with extended L2 entries' Alberto Garcia (27): qcow2: Add calculate_l2_meta() qcow2: Split cluster_needs_cow() out of count_cow_clusters() qcow2: Process

[RFC PATCH v3 19/27] qcow2: Add subcluster support to expand_zero_clusters_in_l1()

Two changes are needed in order to add subcluster support to this function: deallocated clusters must have their bitmaps cleared, and expanded clusters must have all the "subcluster allocated" bits set. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 6 ++ 1 file

[RFC PATCH v3 02/27] qcow2: Split cluster_needs_cow() out of count_cow_clusters()

We are going to need it in other places. Signed-off-by: Alberto Garcia Reviewed-by: Vladimir Sementsov-Ogievskiy --- block/qcow2-cluster.c | 34 +++--- 1 file changed, 19 insertions(+), 15 deletions(-) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c

[RFC PATCH v3 08/27] qcow2: Add offset_to_sc_index()

For a given offset, return the subcluster number within its cluster (i.e. with 32 subclusters per cluster it returns a number between 0 and 31). Signed-off-by: Alberto Garcia --- block/qcow2.h | 5 + 1 file changed, 5 insertions(+) diff --git a/block/qcow2.h b/block/qcow2.h index

[RFC PATCH v3 22/27] qcow2: Clear the L2 bitmap when allocating a compressed cluster

Compressed clusters always have the bitmap part of the extended L2 entry set to 0. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index ed291a4042..af0f01621c 100644 --- a/block/qcow2

[RFC PATCH v3 13/27] qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC

When dealing with subcluster types there is a new value called QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC that has no equivalent in QCow2ClusterType. This patch handles that value in all places where subcluster types are processed. Signed-off-by: Alberto Garcia --- block/qcow2.c | 12 +--- 1

[RFC PATCH v3 11/27] qcow2: Add QCow2SubclusterType and qcow2_get_subcluster_type()

patch. Signed-off-by: Alberto Garcia --- block/qcow2.h | 92 +++ 1 file changed, 92 insertions(+) diff --git a/block/qcow2.h b/block/qcow2.h index 64b0a814f4..321ba9550f 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -80,6 +80,15 @@ #define

[RFC PATCH v3 07/27] qcow2: Add subcluster-related fields to BDRVQcow2State

reated as if they had exactly one, with subcluster_size = cluster_size. Signed-off-by: Alberto Garcia --- block/qcow2.c | 5 + block/qcow2.h | 5 + 2 files changed, 10 insertions(+) diff --git a/block/qcow2.c b/block/qcow2.c index 3866b47946..cbd857e9c7 100644 --- a/block/qcow2.c +++ b/block/q

[RFC PATCH v3 01/27] qcow2: Add calculate_l2_meta()

handle_alloc() creates a QCowL2Meta structure in order to update the image metadata and perform the necessary copy-on-write operations. This patch moves that code to a separate function so it can be used from other places. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 77

[RFC PATCH v3 24/27] qcow2: Restrict qcow2_co_pwrite_zeroes() to full clusters only

Ideally it should be possible to zero individual subclusters using this function, but this is currently not implemented. Signed-off-by: Alberto Garcia --- block/qcow2.c | 6 ++ 1 file changed, 6 insertions(+) diff --git a/block/qcow2.c b/block/qcow2.c index 242001afa2..0267722065 100644

[RFC PATCH v3 17/27] qcow2: Add subcluster support to discard_in_l2_slice()

Setting the QCOW_OFLAG_ZERO bit of the L2 entry is forbidden if an image has subclusters. Instead, the individual 'all zeroes' bits must be used. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 6 +- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/block/qcow2

[PATCH] qcow2: Use offset_into_cluster()

There's a couple of places left in the qcow2 code that still do the calculation manually, so let's replace them. Signed-off-by: Alberto Garcia --- block/qcow2.c | 8 +++- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index 7c18721741..3866b47946

Re: [PATCH] qemu-img: fix info --backing-chain --image-opts

FMT.mid > > Signed-off-by: Stefan Hajnoczi Reviewed-by: Alberto Garcia Berto

Re: [PATCH for-5.0 v2 02/23] blockdev: Allow resizing everywhere

On Mon 11 Nov 2019 05:01:55 PM CET, Max Reitz wrote: > @@ -3177,11 +3177,6 @@ void qmp_block_resize(bool has_device, const char > *device, > aio_context = bdrv_get_aio_context(bs); > aio_context_acquire(aio_context); > > -if (!bdrv_is_first_non_filter(bs)) { > -

Re: [PATCH] blockjob: Fix error message for negative speed

quot;: "Invalid parameter 'speed'"}} > > Make it use QERR_INVALID_PARAMETER_VALUE instead: > > {"error": {"class": "GenericError", "desc": "Parameter 'speed' expects a > non-negative value"}} > > Signed-off-by: Kevin Wolf Reviewed-by: Alberto Garcia Berto

Re: [PATCH] throttle-groups: fix memory leak in throttle_group_set_limits

On Tue 26 Nov 2019 09:17:02 AM CET, pannengy...@huawei.com wrote: > --- a/block/throttle-groups.c > +++ b/block/throttle-groups.c > @@ -912,6 +912,7 @@ static void throttle_group_set_limits(Object *obj, > Visitor *v, > unlock: > qemu_mutex_unlock(>lock); > ret: > +

Re: [PATCH v3 2/8] block: Add no_fallback parameter to bdrv_co_truncate()

On Fri 22 Nov 2019 05:05:05 PM CET, Kevin Wolf wrote: > @@ -3405,6 +3412,7 @@ typedef struct TruncateCo { > int64_t offset; > bool exact; > PreallocMode prealloc; > +bool no_fallback; > Error **errp; > int ret; > } TruncateCo; You add the 'no_fallback' field here...

Re: [PATCH v3 3/8] qcow2: Declare BDRV_REQ_NO_FALLBACK supported

is passed down to the block driver of the external data file. We > are forwarding the BDRV_REQ_NO_FALLBACK flag there, though, so this is > fine, too. > > Declare the flag supported therefore. > > Signed-off-by: Kevin Wolf Reviewed-by: Alberto Garcia Berto

Re: [PATCH v2 2/6] block: truncate: Don't make backing file data visible

t; > After commit top.qcow2 to mid.qcow2, the following happens: > > mid.qcow2: CB-C00C0 (correct result) > mid.qcow2: CB-C--C- (before this fix) > > Without the fix, blocks that previously read as zeros on top.qcow2 > suddenly turn into A. > > Signed-off-by: Kevin Wolf Reviewed-by: Alberto Garcia Berto

Re: [PATCH v2 2/6] block: truncate: Don't make backing file data visible

On Thu 21 Nov 2019 03:33:31 PM CET, Kevin Wolf wrote: > For commit it's an image corruptor. For resize, I'll admit that it's > just wrong behaviour that is probably harmless in most cases, but it's > still wrong behaviour. It would be a corruptor in the same way as > commit if it allowed resizing

Re: [PATCH v2 4/6] iotests: Fix timeout in run_job()

Blake > Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Alberto Garcia Berto

Re: [PATCH v2 3/6] iotests: Add qemu_io_log()

On Wed 20 Nov 2019 07:44:58 PM CET, Kevin Wolf wrote: > Add a function that runs qemu-io and logs the output with the > appropriate filters applied. > > Signed-off-by: Kevin Wolf > Reviewed-by: Eric Blake > Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Alberto Garcia Berto

Re: [PATCH v2 5/6] iotests: Support job-complete in run_job()

y: Vladimir Sementsov-Ogievskiy Reviewed-by: Alberto Garcia Berto

Re: qcow2 preallocation and backing files

On Wed 20 Nov 2019 04:58:38 PM CET, Vladimir Sementsov-Ogievskiy wrote: >> But then PREALLOC_MODE_OFF implies that the L2 metadata should be >> preallocated (all clusters should be QCOW2_CLUSTER_ZERO_PLAIN), at >> least when there is a backing file. > > Kevin proposed a fix that alters

Re: [PATCH 1/6] block: bdrv_co_do_pwrite_zeroes: 64 bit 'bytes' parameter

> Signed-off-by: Kevin Wolf Reviewed-by: Alberto Garcia Berto

Re: qcow2 preallocation and backing files

On Wed 20 Nov 2019 01:27:53 PM CET, Vladimir Semeeausntsov-Ogievskiy wrote: > 3. Also, the latter way is inconsistent with discard. Discarded > regions returns zeroes, not clusters from backing. I think discard and > truncate should behave in the same safe zero way. But then PREALLOC_MODE_OFF

qcow2 preallocation and backing files

Hi, as we discussed yesterday on IRC there's an inconsistency in the way qcow2 preallocation works. Let's create an image and fill it with data: $ qemu-img create -f raw base.img 1M $ qemu-io -f raw -c 'write -P 0xFF 0 1M' base.img Now QEMU won't let us create a new image backed by

Re: [RFC PATCH v2 24/26] qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit

On Tue 05 Nov 2019 01:47:58 PM CET, Max Reitz wrote: >> @@ -1347,6 +1347,12 @@ static int coroutine_fn >> qcow2_do_open(BlockDriverState *bs, QDict *options, >> s->subcluster_size = s->cluster_size / s->subclusters_per_cluster; >> s->subcluster_bits = ctz32(s->subcluster_size); >> >>

Re: [RFC PATCH v2 20/26] qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()

On Tue 05 Nov 2019 12:43:16 PM CET, Max Reitz wrote: > Speaking of handle_copied(); both elements of Qcow2COWRegion are of > type unsigned. handle_copied() doesn’t look like it takes any > precautions to limit the range to even UINT_MAX (and it should > probably limit it to INT_MAX). Or rather,

Re: [RFC PATCH v2 18/26] qcow2: Add subcluster support to expand_zero_clusters_in_l1()

On Tue 05 Nov 2019 12:05:02 PM CET, Max Reitz wrote: >> @@ -2102,6 +2103,7 @@ static int expand_zero_clusters_in_l1(BlockDriverState >> *bs, uint64_t *l1_table, >> } else { >> set_l2_entry(s, l2_slice, j, offset); >> } >> +

Re: [RFC PATCH v2 16/26] qcow2: Add subcluster support to discard_in_l2_slice()

On Mon 04 Nov 2019 04:07:35 PM CET, Max Reitz wrote: >> /* First remove L2 entries */ >> qcow2_cache_entry_mark_dirty(s->l2_table_cache, l2_slice); >> -if (!full_discard && s->qcow_version >= 3) { >> +if (has_subclusters(s)) { >> +set_l2_entry(s,

Re: [RFC PATCH v2 15/26] qcow2: Add subcluster support to zero_in_l2_slice()

On Mon 04 Nov 2019 04:10:58 PM CET, Max Reitz wrote: >>> qcow2_cache_entry_mark_dirty(s->l2_table_cache, l2_slice); >>> if (cluster_type == QCOW2_CLUSTER_COMPRESSED || unmap) { >>> -set_l2_entry(s, l2_slice, l2_index + i, QCOW_OFLAG_ZERO); >>>

Re: [RFC PATCH v2 10/26] qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap()

On Wed 30 Oct 2019 05:55:04 PM CET, Max Reitz wrote: >> This patch also adds the get/set_l2_bitmap() functions that are used >> to access the bitmaps. For convenience, these functions are no-ops >> when used in traditional qcow2 images. > > Granted, I haven’t seen the following patches yet, but if

Re: [RFC PATCH v2 03/26] qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied()

On Wed 30 Oct 2019 03:24:08 PM CET, Max Reitz wrote: >> static void calculate_l2_meta(BlockDriverState *bs, uint64_t host_offset, >>uint64_t guest_offset, uint64_t bytes, >> - QCowL2Meta **m, bool keep_old) >> +

Re: [RFC PATCH v2 14/26] qcow2: Add subcluster support to qcow2_get_cluster_offset()

On Mon 04 Nov 2019 03:58:57 PM CET, Max Reitz wrote: > OTOH, what I don’t like so far about this series is that the “cluster > logic” is still everywhere when I think it should just be about > subclusters now. (Except in few places where it must be about > clusters as in something that can have a

Re: [RFC PATCH v2 13/26] qcow2: Add subcluster support to calculate_l2_meta()

On Mon 04 Nov 2019 03:21:41 PM CET, Max Reitz wrote: >> If an image has subclusters then there are more copy-on-write >> scenarios that we need to consider. Let's say we have a write request >> from the middle of subcluster #3 until the end of the cluster: >> >>- If the cluster is new, then

Re: [PATCH 1/2] block: Remove 'backing': null from bs->{explicit_, }options

d() to remove the 'backing' option from > bs->options and bs->explicit_options even for the case where it > specifies that no backing file is wanted. > > Reported-by: Peter Krempa > Signed-off-by: Kevin Wolf Reviewed-by: Alberto Garcia Berto

Re: [PATCH 2/2] iotests: Test multiple blockdev-snapshot calls

On Fri 08 Nov 2019 09:53:12 AM CET, Kevin Wolf wrote: > +# Test large write to a qcow2 image This doesn't belong here I guess :) I wonder if this test could go in 245 instead. Berto

Re: [RFC PATCH v2 12/26] qcow2: Handle QCOW2_CLUSTER_UNALLOCATED_SUBCLUSTER

On Mon 04 Nov 2019 02:10:37 PM CET, Max Reitz wrote: [QCOW2_CLUSTER_UNALLOCATED_SUBCLUSTER] > I still don’t know what you’re doing in the later patches, but to me > it looks a bit like you don’t dare breaking up the existing structure > that just deals with clusters. Yeah, I decided to extend

Re: [RFC 0/3] block/file-posix: Work around XFS bug

On Mon 04 Nov 2019 04:14:56 PM CET, Max Reitz wrote: >>> No, what I meant was that the original problem that led to >>> c8bb23cbdbe would go away. >> >> Ah, right. Not quite, according to my numbers: >> >> |--++-+-| >> | Cluster size |

Re: [RFC 0/3] block/file-posix: Work around XFS bug

On Mon 04 Nov 2019 03:25:12 PM CET, Max Reitz wrote: > So, it's obvious that c8bb23cbdbe32f5c326 is significant for 1M > cluster-size, even on rotational disk, which means that previous > assumption about calling handle_alloc_space() only for ssd is wrong, > we need smarter

Re: [RFC 0/3] block/file-posix: Work around XFS bug

On Fri 25 Oct 2019 04:19:30 PM CEST, Max Reitz wrote: >>> So, it's obvious that c8bb23cbdbe32f5c326 is significant for 1M >>> cluster-size, even on rotational disk, which means that previous >>> assumption about calling handle_alloc_space() only for ssd is wrong, >>> we need smarter heuristics..

Re: [RFC PATCH v2 12/26] qcow2: Handle QCOW2_CLUSTER_UNALLOCATED_SUBCLUSTER

On Mon 04 Nov 2019 01:57:42 PM CET, Max Reitz wrote: > On 26.10.19 23:25, Alberto Garcia wrote: >> In the previous patch we added a new QCow2ClusterType named >> QCOW2_CLUSTER_UNALLOCATED_SUBCLUSTER. There is a couple of places >> where this new value needs to be hand

Re: [RFC PATCH v2 05/26] qcow2: Document the Extended L2 Entries feature

On Wed 30 Oct 2019 05:23:30 PM CET, Max Reitz wrote: >> +Subcluster Allocation Bitmap (for standard clusters): >> + >> +Bit 0 - 31: Allocation status (one bit per subcluster) >> + >> +1: the subcluster is allocated. In this case the >> + host

Re: [RFC PATCH v2 01/26] qcow2: Add calculate_l2_meta()

On Wed 30 Oct 2019 01:04:02 PM CET, Max Reitz wrote: > (I intended not to comment on such things on an RFC, but here I am...) No problem with that :-) > I’d call it host_cluster_offset to make clear that it points to a > cluster and isn’t the host offset for @guest_offset. Sure, why not. We can

Re: [RFC PATCH v2 01/26] qcow2: Add calculate_l2_meta()

On Mon 28 Oct 2019 01:50:54 PM CET, Vladimir Sementsov-Ogievskiy wrote: >> -.cow_end = { >> -.offset = nb_bytes, > > hmm this logic is changed.. after the patch, it would be not nb_bytes, but > > offset_into_cluster(s, guest_offset) + MIN(*bytes, nb_bytes - >

Re: [PATCH for-4.2 0/2] qcow2: Fix QCOW2_COMPRESSED_SECTOR_MASK

On Mon 28 Oct 2019 05:18:39 PM CET, Max Reitz wrote: > This fixes a bug reported on > https://bugs.launchpad.net/qemu/+bug/185. The problem is that > QCOW2_COMPRESSED_SECTOR_MASK is a 32-bit mask when it really needs to be > a 64-bit mask. Ouch! Reviewed-by: Alberto Garcia Berto

[RFC PATCH v2 20/26] qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()

The L2 bitmap needs to be updated after each write to indicate what new subclusters are now allocated. This needs to happen even if the cluster was already allocated and the L2 entry was otherwise valid. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 16 1 file

[RFC PATCH v2 04/26] qcow2: Add get_l2_entry() and set_l2_entry()

to get_l2_entry() and set_l2_entry(). Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 65 ++ block/qcow2-refcount.c | 17 +-- block/qcow2.h | 12 3 files changed, 55 insertions(+), 39 deletions(-) diff --git a/block/qcow2-cluster.c

[RFC PATCH v2 23/26] qcow2: Restrict qcow2_co_pwrite_zeroes() to full clusters only

Ideally it should be possible to zero individual subclusters using this function, but this is currently not implemented. Signed-off-by: Alberto Garcia --- block/qcow2.c | 6 ++ 1 file changed, 6 insertions(+) diff --git a/block/qcow2.c b/block/qcow2.c index 01322ca449..537569ce88 100644

[RFC PATCH v2 01/26] qcow2: Add calculate_l2_meta()

handle_alloc() creates a QCowL2Meta structure in order to update the image metadata and perform the necessary copy-on-write operations. This patch moves that code to a separate function so it can be used from other places. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 76

[RFC PATCH v2 10/26] qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap()

the get/set_l2_bitmap() functions that are used to access the bitmaps. For convenience, these functions are no-ops when used in traditional qcow2 images. Signed-off-by: Alberto Garcia --- block/qcow2.h | 22 ++ 1 file changed, 22 insertions(+) diff --git a/block/qcow2.h b

[RFC PATCH v2 16/26] qcow2: Add subcluster support to discard_in_l2_slice()

Setting the QCOW_OFLAG_ZERO bit of the L2 entry is forbidden if an image has subclusters. Instead, the individual 'all zeroes' bits must be used. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 6 +- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/block/qcow2

[RFC PATCH v2 17/26] qcow2: Add subcluster support to check_refcounts_l2()

Setting the QCOW_OFLAG_ZERO bit of the L2 entry is forbidden if an image has subclusters. Instead, the individual 'all zeroes' bits must be used. Signed-off-by: Alberto Garcia --- block/qcow2-refcount.c | 9 +++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/block/qcow2

[RFC PATCH v2 19/26] qcow2: Fix offset calculation in handle_dependencies()

l2meta_cow_start() and l2meta_cow_end() are not necessarily cluster-aligned if the image has subclusters, so update the calculation of old_start and old_end to guarantee that no two requests try to write on the same cluster. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 4 ++-- 1

[RFC PATCH v2 09/26] qcow2: Add l2_entry_size()

. This function returns the proper value for the image. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 12 ++-- block/qcow2-refcount.c | 14 -- block/qcow2.c | 6 +++--- block/qcow2.h | 5 + 4 files changed, 22 insertions(+), 15 deletions(-) diff --git

[RFC PATCH v2 18/26] qcow2: Add subcluster support to expand_zero_clusters_in_l1()

Two changes are needed in order to add subcluster support to this function: deallocated clusters must have their bitmaps cleared, and expanded clusters must have all the "subcluster allocated" bits set. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 2 ++ 1 file changed, 2

[RFC PATCH v2 14/26] qcow2: Add subcluster support to qcow2_get_cluster_offset()

The logic of this function remains pretty much the same, except that it uses count_contiguous_subclusters(), which combines the logic of count_contiguous_clusters() / count_contiguous_clusters_unallocated() and checks individual subclusters. Signed-off-by: Alberto Garcia --- block/qcow2

[RFC PATCH v2 22/26] qcow2: Add subcluster support to handle_alloc_space()

with zeroes the other subclusters if we can guarantee that we are not overwriting existing data. However this would waste more disk space, so we should first evaluate if it's really worth doing. Signed-off-by: Alberto Garcia --- block/qcow2.c | 9 + 1 file changed, 5 insertions(+), 4 deletions

[RFC PATCH v2 06/26] qcow2: Add dummy has_subclusters() function

value. Signed-off-by: Alberto Garcia --- block/qcow2.h | 6 ++ 1 file changed, 6 insertions(+) diff --git a/block/qcow2.h b/block/qcow2.h index 940cd4c236..b3826b37c1 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -488,6 +488,12 @@ typedef enum QCow2MetadataOverlap { #define INV_OFFSET

[RFC PATCH v2 21/26] qcow2: Clear the L2 bitmap when allocating a compressed cluster

Compressed clusters always have the bitmap part of the extended L2 entry set to 0. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 1 + 1 file changed, 1 insertion(+) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index acb7226e03..3ba8a98073 100644 --- a/block/qcow2

[RFC PATCH v2 13/26] qcow2: Add subcluster support to calculate_l2_meta()

-zeroes bit set then we need copy-on-write on subcluster #3. - If we are overwriting an old cluster and subcluster #3 was allocated then there is no need to copy-on-write. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 136 +- 1 file

[RFC PATCH v2 24/26] qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit

Now that the implementation of subclusters is complete we can finally add the necessary options to create and read images with this feature, which we call "extended L2 entries". Signed-off-by: Alberto Garcia --- block/qcow2.c| 46 ++ blo

[RFC PATCH v2 26/26] iotests: Add tests for qcow2 images with extended L2 entries

Signed-off-by: Alberto Garcia --- tests/qemu-iotests/271 | 235 + tests/qemu-iotests/271.out | 183 + tests/qemu-iotests/group | 1 + 3 files changed, 419 insertions(+) create mode 100755 tests/qemu-iotests/271 create mode

Re: [RFC PATCH 00/23] Add subcluster allocation to qcow2

On Wed 23 Oct 2019 12:39:14 PM CEST, Vladimir Sementsov-Ogievskiy wrote: > Hi! > > This is very interesting! Could you please export a branch to look at, > as patches can't be applied on master now :( I just sent a new version with some updates and rebased on top of the current master. Berto

[RFC PATCH v2 03/26] qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied()

for subclusters. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 177 +++--- 1 file changed, 96 insertions(+), 81 deletions(-) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index aa1010a515..ee6b46f917 100644 --- a/block/qcow2-cluster.c

[RFC PATCH v2 02/26] qcow2: Split cluster_needs_cow() out of count_cow_clusters()

We are going to need it in other places. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 34 +++--- 1 file changed, 19 insertions(+), 15 deletions(-) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index 6c1dcdc781..aa1010a515 100644 --- a/block

[RFC PATCH v2 11/26] qcow2: Add qcow2_get_subcluster_type()

returned for images with extended L2 entries. Signed-off-by: Alberto Garcia --- block/qcow2.h | 62 +++ 1 file changed, 62 insertions(+) diff --git a/block/qcow2.h b/block/qcow2.h index 741c41c80f..23a2532ff2 100644 --- a/block/qcow2.h +++ b/block

[RFC PATCH v2 00/26] Add subcluster allocation to qcow2

L2 entries' Alberto Garcia (26): qcow2: Add calculate_l2_meta() qcow2: Split cluster_needs_cow() out of count_cow_clusters() qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied() qcow2: Add get_l2_entry() and set_l2_entry() qcow2: Document the Extended L2 Entries feature

[RFC PATCH v2 15/26] qcow2: Add subcluster support to zero_in_l2_slice()

Setting the QCOW_OFLAG_ZERO bit of the L2 entry is forbidden if an image has subclusters. Instead, the individual 'all zeroes' bits must be used. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 14 ++ 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/block

[RFC PATCH v2 07/26] qcow2: Add subcluster-related fields to BDRVQcow2State

reated as if they had exactly one, with subcluster_size = cluster_size. Signed-off-by: Alberto Garcia --- block/qcow2.c | 5 + block/qcow2.h | 5 + 2 files changed, 10 insertions(+) diff --git a/block/qcow2.c b/block/qcow2.c index 0bc69e6996..ed8b81c7b7 100644 --- a/block/qcow2.c +++ b/block/q

[RFC PATCH v2 05/26] qcow2: Document the Extended L2 Entries feature

-off-by: Alberto Garcia --- docs/interop/qcow2.txt | 68 -- docs/qcow2-cache.txt | 19 +++- 2 files changed, 83 insertions(+), 4 deletions(-) diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt index af5711e533..d34261f955 100644 --- a/docs

[RFC PATCH v2 12/26] qcow2: Handle QCOW2_CLUSTER_UNALLOCATED_SUBCLUSTER

In the previous patch we added a new QCow2ClusterType named QCOW2_CLUSTER_UNALLOCATED_SUBCLUSTER. There is a couple of places where this new value needs to be handled, and that is what this patch does. Signed-off-by: Alberto Garcia --- block/qcow2.c | 13 + 1 file changed, 9

[RFC PATCH v2 25/26] qcow2: Allow preallocation and backing files if extended_l2 is set

it is perfectly possible to have an allocated cluster with all its subclusters unallocated. Signed-off-by: Alberto Garcia --- block/qcow2.c | 7 --- tests/qemu-iotests/206.out | 2 +- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index b1fa7ab5da

[RFC PATCH v2 08/26] qcow2: Add offset_to_sc_index()

For a given offset, return the subcluster number within its cluster (i.e. with 32 subclusters per cluster it returns a number between 0 and 31). Signed-off-by: Alberto Garcia --- block/qcow2.h | 5 + 1 file changed, 5 insertions(+) diff --git a/block/qcow2.h b/block/qcow2.h index

Re: [RFC PATCH 00/23] Add subcluster allocation to qcow2

On Tue 15 Oct 2019 06:05:23 PM CEST, Eric Blake wrote: >> 6356 5548 4740 3932 3124 2316 15 8 7 0 >> >> <-> <-> >>

[RFC PATCH 13/23] qcow2: Add subcluster support to calculate_l2_meta()

-zeroes bit set then we need copy-on-write on subcluster #3. - If we are overwriting an old cluster and subcluster #3 was allocated then there is no need to copy-on-write. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 136 +- 1 file

[RFC PATCH 17/23] qcow2: Add subcluster support to check_refcounts_l2()

Setting the QCOW_OFLAG_ZERO bit of the L2 entry is forbidden if an image has subclusters. Instead, the individual 'all zeroes' bits must be used. Signed-off-by: Alberto Garcia --- block/qcow2-refcount.c | 9 +++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/block/qcow2

[RFC PATCH 11/23] qcow2: Add qcow2_get_subcluster_type()

returned for images with extended L2 entries. Signed-off-by: Alberto Garcia --- block/qcow2.h | 62 +++ 1 file changed, 62 insertions(+) diff --git a/block/qcow2.h b/block/qcow2.h index d9fe883fe0..60e4bf963e 100644 --- a/block/qcow2.h +++ b/block

<    3   4   5   6   7   8   9   10   11   12   >