Re: [Qemu-block] [Qemu-devel] [PATCH v7 0/4] Add shrink image for qcow2

[Pavel Butsykin]

On 16.09.2017 17:56, Max Reitz wrote:

On 2017-08-22 01:31, John Snow wrote:



On 08/17/2017 05:15 AM, Pavel Butsykin wrote:

This patch add shrinking of the image file for qcow2. As a result, this allows
us to reduce the virtual image size and free up space on the disk without
copying the image. Image can be fragmented and shrink is done by punching holes
in the image file.

# ./qemu-img create -f qcow2 image.qcow2 4G
Formatting 'image.qcow2', fmt=qcow2 size=4294967296 encryption=off 
cluster_size=65536 lazy_refcounts=off refcount_bits=16

# ./qemu-io -c "write -P 0x22 0 1G" image.qcow2
wrote 1073741824/1073741824 bytes at offset 0
1 GiB, 1 ops; 0:00:04.59 (222.886 MiB/sec and 0.2177 ops/sec)

# ./qemu-img resize image.qcow2 512M
warning: qemu-img: Shrinking an image will delete all data beyond the shrunken 
image's end. Before performing such an operation, make sure there is no 
important data there.
error: qemu-img: Use the --shrink option to perform a shrink operation.

# ./qemu-img resize --shrink image.qcow2 128M
Image resized.

# ./qemu-img info image.qcow2
image: image.qcow2
file format: qcow2
virtual size: 128M (134217728 bytes)
disk size: 128M
cluster_size: 65536
Format specific information:
 compat: 1.1
 lazy refcounts: false
 refcount bits: 16
 corrupt: false

# du -h image.qcow2
129Mimage.qcow2



It looks sane to me, and already has a full set of R-Bs from Max. Are we
waiting for Kevin?


Now I found that I was waiting for my own R-b for patch 3.  I hadn't yet
reviewed that patch in its current state (I had reviewed it for v5, but
it has changed quite a bit in v6)


I decided that small changes are not considered :) I'm sorry for the
confusion with the R-b.



Max

Re: [Qemu-block] [Qemu-devel] [PATCH v7 0/4] Add shrink image for qcow2

[Max Reitz]

On 2017-08-22 01:31, John Snow wrote:
> 
> 
> On 08/17/2017 05:15 AM, Pavel Butsykin wrote:
>> This patch add shrinking of the image file for qcow2. As a result, this 
>> allows
>> us to reduce the virtual image size and free up space on the disk without
>> copying the image. Image can be fragmented and shrink is done by punching 
>> holes
>> in the image file.
>>
>> # ./qemu-img create -f qcow2 image.qcow2 4G
>> Formatting 'image.qcow2', fmt=qcow2 size=4294967296 encryption=off 
>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>
>> # ./qemu-io -c "write -P 0x22 0 1G" image.qcow2
>> wrote 1073741824/1073741824 bytes at offset 0
>> 1 GiB, 1 ops; 0:00:04.59 (222.886 MiB/sec and 0.2177 ops/sec)
>>
>> # ./qemu-img resize image.qcow2 512M
>> warning: qemu-img: Shrinking an image will delete all data beyond the 
>> shrunken image's end. Before performing such an operation, make sure there 
>> is no important data there.
>> error: qemu-img: Use the --shrink option to perform a shrink operation.
>>
>> # ./qemu-img resize --shrink image.qcow2 128M
>> Image resized.
>>
>> # ./qemu-img info image.qcow2
>> image: image.qcow2
>> file format: qcow2
>> virtual size: 128M (134217728 bytes)
>> disk size: 128M
>> cluster_size: 65536
>> Format specific information:
>> compat: 1.1
>> lazy refcounts: false
>> refcount bits: 16
>> corrupt: false
>>
>> # du -h image.qcow2
>> 129Mimage.qcow2
>>
> 
> It looks sane to me, and already has a full set of R-Bs from Max. Are we
> waiting for Kevin?

Now I found that I was waiting for my own R-b for patch 3.  I hadn't yet
reviewed that patch in its current state (I had reviewed it for v5, but
it has changed quite a bit in v6).

Max



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-block] [Qemu-devel] [PATCH v7 0/4] Add shrink image for qcow2

[Max Reitz]

On 2017-08-22 01:31, John Snow wrote:
> 
> 
> On 08/17/2017 05:15 AM, Pavel Butsykin wrote:
>> This patch add shrinking of the image file for qcow2. As a result, this 
>> allows
>> us to reduce the virtual image size and free up space on the disk without
>> copying the image. Image can be fragmented and shrink is done by punching 
>> holes
>> in the image file.
>>
>> # ./qemu-img create -f qcow2 image.qcow2 4G
>> Formatting 'image.qcow2', fmt=qcow2 size=4294967296 encryption=off 
>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>
>> # ./qemu-io -c "write -P 0x22 0 1G" image.qcow2
>> wrote 1073741824/1073741824 bytes at offset 0
>> 1 GiB, 1 ops; 0:00:04.59 (222.886 MiB/sec and 0.2177 ops/sec)
>>
>> # ./qemu-img resize image.qcow2 512M
>> warning: qemu-img: Shrinking an image will delete all data beyond the 
>> shrunken image's end. Before performing such an operation, make sure there 
>> is no important data there.
>> error: qemu-img: Use the --shrink option to perform a shrink operation.
>>
>> # ./qemu-img resize --shrink image.qcow2 128M
>> Image resized.
>>
>> # ./qemu-img info image.qcow2
>> image: image.qcow2
>> file format: qcow2
>> virtual size: 128M (134217728 bytes)
>> disk size: 128M
>> cluster_size: 65536
>> Format specific information:
>> compat: 1.1
>> lazy refcounts: false
>> refcount bits: 16
>> corrupt: false
>>
>> # du -h image.qcow2
>> 129Mimage.qcow2
>>
> 
> It looks sane to me, and already has a full set of R-Bs from Max. Are we
> waiting for Kevin?

We were waiting for Kevin to handle 2.10 patches so he could go into PTO
and for me to come back from PTO. ;-)

I'm still sifting through my inbox... When I'm done, I'll take a look,
but I have no idea how long that might last.
(Since 2.10 isn't out yet, there might be more pressing issues still...?
I don't quite now yet, to be honest...)

> It looks like in Vladimir's series we sidestepped the problem of what
> happens if we resize with persistent bitmaps: we deny the operation,
> regardless of if we are shrinking, growing, or have bitmaps that are
> read-only, frozen, or what-have-you.
> 
> It shouldn't be too hard to add soon, but this is fine for now.
> (I think for adding it we just need to make sure the bitmaps aren't
> frozen and are not read-only, and the implementation bitmap structure
> can already handle truncation in either direction just fine.)
> 
> Anyway;
> 
> Reviewed-by: John Snow 

Thanks, in any case. :-)

Max



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-block] [Qemu-devel] [PATCH v7 0/4] Add shrink image for qcow2

[John Snow]

On 08/17/2017 05:15 AM, Pavel Butsykin wrote:
> This patch add shrinking of the image file for qcow2. As a result, this allows
> us to reduce the virtual image size and free up space on the disk without
> copying the image. Image can be fragmented and shrink is done by punching 
> holes
> in the image file.
> 
> # ./qemu-img create -f qcow2 image.qcow2 4G
> Formatting 'image.qcow2', fmt=qcow2 size=4294967296 encryption=off 
> cluster_size=65536 lazy_refcounts=off refcount_bits=16
> 
> # ./qemu-io -c "write -P 0x22 0 1G" image.qcow2
> wrote 1073741824/1073741824 bytes at offset 0
> 1 GiB, 1 ops; 0:00:04.59 (222.886 MiB/sec and 0.2177 ops/sec)
> 
> # ./qemu-img resize image.qcow2 512M
> warning: qemu-img: Shrinking an image will delete all data beyond the 
> shrunken image's end. Before performing such an operation, make sure there is 
> no important data there.
> error: qemu-img: Use the --shrink option to perform a shrink operation.
> 
> # ./qemu-img resize --shrink image.qcow2 128M
> Image resized.
> 
> # ./qemu-img info image.qcow2
> image: image.qcow2
> file format: qcow2
> virtual size: 128M (134217728 bytes)
> disk size: 128M
> cluster_size: 65536
> Format specific information:
> compat: 1.1
> lazy refcounts: false
> refcount bits: 16
> corrupt: false
> 
> # du -h image.qcow2
> 129Mimage.qcow2
> 

It looks sane to me, and already has a full set of R-Bs from Max. Are we
waiting for Kevin?

It looks like in Vladimir's series we sidestepped the problem of what
happens if we resize with persistent bitmaps: we deny the operation,
regardless of if we are shrinking, growing, or have bitmaps that are
read-only, frozen, or what-have-you.

It shouldn't be too hard to add soon, but this is fine for now.
(I think for adding it we just need to make sure the bitmaps aren't
frozen and are not read-only, and the implementation bitmap structure
can already handle truncation in either direction just fine.)

Anyway;

Reviewed-by: John Snow 


> Changes from v1:
> - add --shrink flag for qemu-img resize
> - add qcow2_cache_discard
> - simplify qcow2_shrink_l1_table() to reduce the likelihood of image 
> corruption
> - add new qemu-iotests for shrinking images
> 
> Changes from v2:
> - replace qprintf() on error_report() (1)
> - rewrite warning messages (1)
> - enforce --shrink flag for all formats except raw (1)
> - split qcow2_cache_discard() (2)
> - minor fixes according to comments (3)
> - rewrite the last part of qcow2_shrink_reftable() to avoid
>   qcow2_free_clusters() calls inside (3)
> - improve test for shrinking image (4)
> 
> Changes from v3:
> - rebase on "Implement a warning_report function" Alistair's patch-set (1)
> - spelling fixes (1)
> - the man page fix according to the discussion (1)
> - add call qcow2_signal_corruption() in case of image corruption (3)
> 
> Changes from v4:
> - rebase on https://github.com/XanClic/qemu/commits/block Max's block branch
> 
> Changes from v5:
> - the condition refcount == 0 should be enough to evict the l2/refcount 
> cluster
>   from the cache (2)
> - overwrite the l1/refcount table in memory with zeros, even if overwriting 
> the
>   l1/refcount table on disk has failed (3)
> - replace g_try_malloc() on g_malloc() for allocation reftable_tmp (3)
> 
> Changes from v6:
> - rebase on master 1f29673387
> 
> Pavel Butsykin (4):
>   qemu-img: add --shrink flag for resize
>   qcow2: add qcow2_cache_discard
>   qcow2: add shrink image support
>   qemu-iotests: add shrinking image test
> 
>  block/qcow2-cache.c|  26 +++
>  block/qcow2-cluster.c  |  50 +
>  block/qcow2-refcount.c | 140 -
>  block/qcow2.c  |  43 +---
>  block/qcow2.h  |  17 +
>  qapi/block-core.json   |   3 +-
>  qemu-img-cmds.hx   |   4 +-
>  qemu-img.c |  23 ++
>  qemu-img.texi  |   6 +-
>  tests/qemu-iotests/102 |   4 +-
>  tests/qemu-iotests/163 | 170 
> +
>  tests/qemu-iotests/163.out |   5 ++
>  tests/qemu-iotests/group   |   1 +
>  13 files changed, 475 insertions(+), 17 deletions(-)
>  create mode 100644 tests/qemu-iotests/163
>  create mode 100644 tests/qemu-iotests/163.out
>