[PATCH v9 1/9] copy-on-read: Support preadv/pwritev_part functions

2020-09-28 Thread Andrey Shinkevich via
Add support for the recently introduced functions bdrv_co_preadv_part() and bdrv_co_pwritev_part() to the COR-filter driver. Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- block/copy-on-read.c | 28 1 file changed, 16 insertions

[PATCH v9 0/9] Apply COR-filter to the block-stream permanently

2020-09-28 Thread Andrey Shinkevich via
file name. 05: New implementation based on Max' review. 06: New. 07: New. The patch "freeze link to base node..." was deleted. 08: New. 09: The filter node options are initialized. The v8 Message-Id: <1598633579-221780-1-git-send-email-andrey.shinkev...@virtuozzo.com> A

[PATCH v9 7/9] stream: skip filters when writing backing file name to QCOW2 header

2020-09-28 Thread Andrey Shinkevich via
Avoid writing a filter JSON-name to QCOW2 image when the backing file is changed after the block stream job. Signed-off-by: Andrey Shinkevich --- block/stream.c | 9 + 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/block/stream.c b/block/stream.c index e0540ee..b0719e9

[PATCH v9 4/9] copy-on-read: pass base node name to COR driver

2020-09-28 Thread Andrey Shinkevich via
To limit the guest's COR operations by the base node in the backing chain during stream job, pass the base node name to the copy-on-read driver. The rest of the functionality will be implemented in the patch that follows. Signed-off-by: Andrey Shinkevich --- block/copy-on-read.c

[PATCH v9 5/9] copy-on-read: limit guest COR activity to base in COR driver

2020-09-28 Thread Andrey Shinkevich via
Limit the guest's COR operations by the base node in the backing chain when the base node name is given. It will be useful for a block stream job when the COR-filter is applied. Signed-off-by: Andrey Shinkevich --- block/copy-on-read.c | 38 -- 1 file ch

[PATCH v9 3/9] qapi: add filter-node-name to block-stream

2020-09-28 Thread Andrey Shinkevich via
Provide the possibility to pass the 'filter-node-name' parameter to the block-stream job as it is done for the commit block job. Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- block/monitor/block-hmp-cmds.c | 4 ++-- block/stream.c

[PATCH v9 6/9] copy-on-read: skip non-guest reads if no copy needed

2020-09-28 Thread Andrey Shinkevich via
If the flag BDRV_REQ_PREFETCH was set, pass it further to the COR-driver to skip unneeded reading. It can be taken into account for the COR-algorithms optimization. That check is being made during the block stream job by the moment. Signed-off-by: Andrey Shinkevich --- block/copy-on-read.c | 14

Re: [PATCH v8 2/7] copy-on-read: add filter append/drop functions

2020-09-24 Thread Andrey Shinkevich
On 24.09.2020 20:29, Andrey Shinkevich wrote: On 24.09.2020 18:00, Max Reitz wrote: On 24.09.20 16:51, Vladimir Sementsov-Ogievskiy wrote: 24.09.2020 16:25, Max Reitz wrote: On 23.09.20 16:38, Vladimir Sementsov-Ogievskiy wrote: 17.09.2020 19:09, Andrey Shinkevich wrote: On 04.09.2020 14:22

Re: [PATCH v8 2/7] copy-on-read: add filter append/drop functions

2020-09-24 Thread Andrey Shinkevich
On 24.09.2020 18:00, Max Reitz wrote: On 24.09.20 16:51, Vladimir Sementsov-Ogievskiy wrote: 24.09.2020 16:25, Max Reitz wrote: On 23.09.20 16:38, Vladimir Sementsov-Ogievskiy wrote: 17.09.2020 19:09, Andrey Shinkevich wrote: On 04.09.2020 14:22, Max Reitz wrote: On 28.08.20 18:52, Andrey

Re: [PATCH v8 5/7] copy-on-read: limit guest writes to base in COR driver

2020-09-22 Thread Andrey Shinkevich
On 04.09.2020 16:59, Vladimir Sementsov-Ogievskiy wrote: 04.09.2020 15:50, Max Reitz wrote: On 28.08.20 18:52, Andrey Shinkevich wrote: Limit the guest's COR operations by the base node in the backing chain during a stream job. I don’t understand.   Shouldn’t we limit the areas where w

Re: [PATCH v8 2/7] copy-on-read: add filter append/drop functions

2020-09-17 Thread Andrey Shinkevich
On 04.09.2020 14:22, Max Reitz wrote: On 28.08.20 18:52, Andrey Shinkevich wrote: Provide API for the COR-filter insertion/removal. ... Also, drop the filter child permissions for an inactive state when the filter node is being removed. Do we need .active for that? Shouldn’t it be

[PATCH v8 7/7] block: apply COR-filter to block-stream jobs

2020-08-28 Thread Andrey Shinkevich via
concept of the parallel jobs with common nodes is considered vital no more. Signed-off-by: Andrey Shinkevich --- block/stream.c | 78 +++--- tests/qemu-iotests/030 | 50 +++-- tests/qemu-iotests/030.out | 4 +-- tests

[PATCH v8 5/7] copy-on-read: limit guest writes to base in COR driver

2020-08-28 Thread Andrey Shinkevich via
Limit the guest's COR operations by the base node in the backing chain during a stream job. Signed-off-by: Andrey Shinkevich --- block/copy-on-read.c | 49 + 1 file changed, 49 insertions(+) diff --git a/block/copy-on-read.c b/block/co

[PATCH v8 6/7] block-stream: freeze link to base node during stream job

2020-08-28 Thread Andrey Shinkevich via
the common backing chain. Signed-off-by: Andrey Shinkevich --- block/stream.c | 29 ++-- tests/qemu-iotests/030 | 10 +-- tests/qemu-iotests/245 | 2 +- tests/qemu-iotests/258 | 161 - tests/qemu-iotests/258.out | 33

[PATCH v8 3/7] qapi: add filter-node-name to block-stream

2020-08-28 Thread Andrey Shinkevich via
Provide the possibility to pass the 'filter-node-name' parameter to the block-stream job as it is done for the commit block job. Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- block/monitor/block-hmp-cmds.c | 4 ++-- block/stream.c

[PATCH v8 4/7] copy-on-read: pass base file name to COR driver

2020-08-28 Thread Andrey Shinkevich via
To limit the guest's COR operations by the base node in the backing chain during stream job, pass the base file name to the copy-on-read driver. The rest of the functionality will be implemented in the patch that follows. Signed-off-by: Andrey Shinkevich --- block/copy-on-read.c

[PATCH v8 1/7] copy-on-read: Support preadv/pwritev_part functions

2020-08-28 Thread Andrey Shinkevich via
Add support for the recently introduced functions bdrv_co_preadv_part() and bdrv_co_pwritev_part() to the COR-filter driver. Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- block/copy-on-read.c | 28 1 file changed, 16 insertions

[PATCH v8 2/7] copy-on-read: add filter append/drop functions

2020-08-28 Thread Andrey Shinkevich via
Provide API for the COR-filter insertion/removal. Also, drop the filter child permissions for an inactive state when the filter node is being removed. Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- block/copy-on-read.c | 104

[PATCH v8 0/7] Apply COR-filter to the block-stream permanently

2020-08-28 Thread Andrey Shinkevich via
changed to 'Since: 5.2'. 04: New. 05: New. 06: New. 07: The extra bs variable and the enable_cor were dropped. The v7 Message-Id: <1598257914-887267-1-git-send-email-andrey.shinkev...@virtuozzo.com> Andrey Shinkevich (7): copy-on-read: Support preadv/pwritev_part fun

Re: [PATCH v7 4/4] block: apply COR-filter to block-stream jobs

2020-08-28 Thread Andrey Shinkevich
On 24.08.2020 14:30, Vladimir Sementsov-Ogievskiy wrote: 24.08.2020 11:31, Andrey Shinkevich wrote: This patch completes the series with the COR-filter insertion for block-stream operations. Adding the filter makes it possible for copied regions to be discarded in backing files during the block

Re: [PATCH v7 3/4] qapi: add filter-node-name to block-stream

2020-08-25 Thread Andrey Shinkevich
On 25.08.2020 09:37, Markus Armbruster wrote: Andrey Shinkevich writes: Provide the possibility to pass the 'filter-node-name' parameter to the block-stream job as it is done for the commit block job. Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy [

Re: [PATCH v6 4/4] block: apply COR-filter to block-stream jobs

2020-08-24 Thread Andrey Shinkevich
On 24.08.2020 11:20, Vladimir Sementsov-Ogievskiy wrote: 23.08.2020 22:28, Andrey Shinkevich wrote: On 19.08.2020 13:46, Vladimir Sementsov-Ogievskiy wrote: 19.08.2020 00:24, Andrey Shinkevich wrote: The patch completes the series with the COR-filter insertion to any block-stream operation

[PATCH v7 2/4] copy-on-read: add filter append/drop functions

2020-08-24 Thread Andrey Shinkevich
Provide API for the COR-filter insertion/removal. Also, drop the filter child permissions for an inactive state when the filter node is being removed. Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- block/copy-on-read.c | 104

[PATCH v7 4/4] block: apply COR-filter to block-stream jobs

2020-08-24 Thread Andrey Shinkevich
concept of the parallel jobs with common nodes is considered vital no more. Signed-off-by: Andrey Shinkevich --- block/stream.c | 58 +++--- tests/qemu-iotests/030 | 50 --- tests/qemu-iotests/030.out | 4

[PATCH v7 3/4] qapi: add filter-node-name to block-stream

2020-08-24 Thread Andrey Shinkevich
Provide the possibility to pass the 'filter-node-name' parameter to the block-stream job as it is done for the commit block job. Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- block/monitor/block-hmp-cmds.c | 4 ++-- block/stream.c

[PATCH v7 0/4] Apply COR-filter to the block-stream permanently

2020-08-24 Thread Andrey Shinkevich
file change no more. I urge to keep it. The v6 Message-Id: <1597785880-431103-1-git-send-email-andrey.shinkev...@virtuozzo.com> Andrey Shinkevich (4): copy-on-read: Support preadv/pwritev_part functions copy-on-read: add filter append/drop functi

[PATCH v7 1/4] copy-on-read: Support preadv/pwritev_part functions

2020-08-24 Thread Andrey Shinkevich
Add support for the recently introduced functions bdrv_co_preadv_part() and bdrv_co_pwritev_part() to the COR-filter driver. Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- block/copy-on-read.c | 28 1 file changed, 16 insertions

Re: [PATCH v6 2/4] copy-on-read: add filter append/drop functions

2020-08-23 Thread Andrey Shinkevich
On 23.08.2020 22:35, Andrey Shinkevich wrote: On 19.08.2020 13:21, Vladimir Sementsov-Ogievskiy wrote: 19.08.2020 00:24, Andrey Shinkevich wrote: Provide API for the COR-filter insertion/removal. Also, drop the filter child permissions for an inactive state when the filter node is being

Re: [PATCH v6 4/4] block: apply COR-filter to block-stream jobs

2020-08-23 Thread Andrey Shinkevich
On 19.08.2020 13:46, Vladimir Sementsov-Ogievskiy wrote: 19.08.2020 00:24, Andrey Shinkevich wrote: The patch completes the series with the COR-filter insertion to any block-stream operation. It also makes changes to the iotests 030. The test case 'test_stream_parallel' was dele

Re: [PATCH v6 2/4] copy-on-read: add filter append/drop functions

2020-08-23 Thread Andrey Shinkevich
On 19.08.2020 13:21, Vladimir Sementsov-Ogievskiy wrote: 19.08.2020 00:24, Andrey Shinkevich wrote: Provide API for the COR-filter insertion/removal. Also, drop the filter child permissions for an inactive state when the filter node is being removed. Signed-off-by: Andrey Shinkevich

Re: [PATCH v6 3/4] qapi: add filter-node-name to block-stream

2020-08-23 Thread Andrey Shinkevich
On 19.08.2020 13:29, Vladimir Sementsov-Ogievskiy wrote: 19.08.2020 00:24, Andrey Shinkevich wrote: Provide the possibility to pass the 'filter-node-name' parameter to the block-stream job as it is done for the commit block job. That will be needed for further iotests implementations.

Re: [PATCH v13 00/11] iotests: Dump QCOW2 dirty bitmaps metadata

2020-08-20 Thread Andrey Shinkevich
On 20.08.2020 03:49, Eric Blake wrote: On 8/14/20 6:56 AM, Andrey Shinkevich wrote: Dear Eric! Vladimir has compeated reviewing this series. I have not received any other responses to it so far. So, is it good for pull request now? Would you please consider taking this series as you did it

[PATCH v6 0/4] Apply COR-filter to the block-stream permanently

2020-08-18 Thread Andrey Shinkevich
ies "block: Deal with filters". The minimum number of patches were kept. Not all the iotests were checked for pass. 04: The test case iotests:030:test_stream_parallel was removed due to multiple errors. Andrey Shinkevich (4): copy-on-read: Support preadv/pwritev_part func

[PATCH v6 4/4] block: apply COR-filter to block-stream jobs

2020-08-18 Thread Andrey Shinkevich
The patch completes the series with the COR-filter insertion to any block-stream operation. It also makes changes to the iotests 030. The test case 'test_stream_parallel' was deleted due to multiple errors. Signed-off-by: Andrey Shinkevich --- block/stream.c

[PATCH v6 2/4] copy-on-read: add filter append/drop functions

2020-08-18 Thread Andrey Shinkevich
Provide API for the COR-filter insertion/removal. Also, drop the filter child permissions for an inactive state when the filter node is being removed. Signed-off-by: Andrey Shinkevich --- block/copy-on-read.c | 103 +++ block/copy-on-read.h | 36

[PATCH v6 3/4] qapi: add filter-node-name to block-stream

2020-08-18 Thread Andrey Shinkevich
Provide the possibility to pass the 'filter-node-name' parameter to the block-stream job as it is done for the commit block job. That will be needed for further iotests implementations. Signed-off-by: Andrey Shinkevich --- block/monitor/block-hmp-cmds.c | 4 ++-- bloc

[PATCH v6 1/4] copy-on-read: Support preadv/pwritev_part functions

2020-08-18 Thread Andrey Shinkevich
Add support for the recently introduced functions bdrv_co_preadv_part() and bdrv_co_pwritev_part() to the COR-filter driver. Signed-off-by: Andrey Shinkevich --- block/copy-on-read.c | 28 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/block/copy-on

Re: [PATCH v7 14/47] stream: Deal with filters

2020-08-18 Thread Andrey Shinkevich
Reviewed-by: Andrey Shinkevich On 10.08.2020 14:04, Vladimir Sementsov-Ogievskiy wrote: 10.08.2020 11:12, Max Reitz wrote: On 07.08.20 12:29, Vladimir Sementsov-Ogievskiy wrote: 16.07.2020 17:59, Max Reitz wrote: On 10.07.20 19:41, Andrey Shinkevich wrote: On 10.07.2020 18:24, Max Reitz

Re: [PATCH v7 14/47] stream: Deal with filters

2020-08-14 Thread Andrey Shinkevich
On 10.08.2020 14:04, Vladimir Sementsov-Ogievskiy wrote: 10.08.2020 11:12, Max Reitz wrote: On 07.08.20 12:29, Vladimir Sementsov-Ogievskiy wrote: 16.07.2020 17:59, Max Reitz wrote: On 10.07.20 19:41, Andrey Shinkevich wrote: On 10.07.2020 18:24, Max Reitz wrote: On 09.07.20 16:52, Andrey

Re: [PATCH v13 00/11] iotests: Dump QCOW2 dirty bitmaps metadata

2020-08-14 Thread Andrey Shinkevich
extra dict variables removed. to_dict() renamed to to_json(). The to_json() added to the class Qcow2BitmapTable. (By Vladimir). Andrey Shinkevich (11): iotests: add test for QCOW2 header dump qcow2_format.py: make printable data an extension class member qcow2_format.py: c

Re: [PATCH v13 09/11] qcow2_format.py: collect fields to dump in JSON format

2020-08-07 Thread Andrey Shinkevich
On 07.08.2020 09:30, Vladimir Sementsov-Ogievskiy wrote: 06.08.2020 22:35, Andrey Shinkevich wrote: As __dict__ is being extended with class members we do not want to print, add the to_dict() method to classes that returns a dictionary to_json() ... that returns a json-dumpable object New

[PATCH v13 04/11] qcow2_format.py: dump bitmap flags in human readable way.

2020-08-06 Thread Andrey Shinkevich
Introduce the class BitmapFlags that parses a bitmap flags mask. Suggested-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/qcow2_format.py | 16 1 file changed, 16 insertions(+) diff --git a

[PATCH v13 05/11] qcow2_format.py: Dump bitmap directory information

2020-08-06 Thread Andrey Shinkevich
27;]) type 1 granularity_bits 16 name_size 8 extra_data_size 0 Suggested-by: Kevin Wolf Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/303.out | 18 +++ tests/qemu-iotests/qcow2_form

[PATCH v13 08/11] qcow2.py: Introduce '-j' key to dump in JSON format

2020-08-06 Thread Andrey Shinkevich
Add the command key to the qcow2.py arguments list to dump QCOW2 metadata in JSON format. Here is the suggested way to do that. The implementation of the dump in JSON format is in the patch that follows. Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu

[PATCH v13 06/11] qcow2_format.py: pass cluster size to substructures

2020-08-06 Thread Andrey Shinkevich
The cluster size of an image is the QcowHeader class member and may be obtained by dependent extension structures such as Qcow2BitmapExt for further bitmap table details print. Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/qcow2_format.py

[PATCH v13 00/11] iotests: Dump QCOW2 dirty bitmaps metadata

2020-08-06 Thread Andrey Shinkevich
removed. to_dict() renamed to to_json(). The to_json() added to the class Qcow2BitmapTable. (By Vladimir). Andrey Shinkevich (11): iotests: add test for QCOW2 header dump qcow2_format.py: make printable data an extension class member qcow2_format.py: change Qcow2BitmapExt

[PATCH v13 09/11] qcow2_format.py: collect fields to dump in JSON format

2020-08-06 Thread Andrey Shinkevich
Sementsov-Ogievskiy Signed-off-by: Andrey Shinkevich --- tests/qemu-iotests/qcow2_format.py | 36 1 file changed, 36 insertions(+) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py index de0adcb..5a298b2 100644 --- a/tests

[PATCH v13 10/11] qcow2_format.py: support dumping metadata in JSON format

2020-08-06 Thread Andrey Shinkevich
"bitmap_table": [ { "type": "serialized", "offset": 655360 }, ... Suggested-by: Vlad

[PATCH v13 02/11] qcow2_format.py: make printable data an extension class member

2020-08-06 Thread Andrey Shinkevich
Let us differ binary data type from string one for the extension data variable and keep the string as the QcowHeaderExtension class member. Signed-off-by: Andrey Shinkevich Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/qcow2_format.py | 14

[PATCH v13 03/11] qcow2_format.py: change Qcow2BitmapExt initialization method

2020-08-06 Thread Andrey Shinkevich
n the initialization chain. The implementation comes with the patch that follows. Suggested-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/qcow2_format.py | 36 ++-- 1 file change

[PATCH v13 11/11] iotests: dump QCOW2 header in JSON in #303

2020-08-06 Thread Andrey Shinkevich
Extend the test case #303 by dumping QCOW2 image metadata in JSON format. Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/303 | 3 ++ tests/qemu-iotests/303.out | 76 ++ 2 files changed, 79

[PATCH v13 07/11] qcow2_format.py: Dump bitmap table serialized entries

2020-08-06 Thread Andrey Shinkevich
Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/303.out | 4 +++ tests/qemu-iotests/qcow2_format.py | 50 ++ 2 files changed, 54 insertions(+) diff --git a/tests/qemu-iotests/303.out b/tests/qemu-iotests

[PATCH v13 01/11] iotests: add test for QCOW2 header dump

2020-08-06 Thread Andrey Shinkevich
test later. Suggested-by: Eric Blake Signed-off-by: Andrey Shinkevich Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/303 | 60 ++ tests/qemu-iotests/303.out | 60

Re: [PATCH v12 09/11] qcow2_format.py: collect fields to dump in JSON format

2020-08-06 Thread Andrey Shinkevich
On 05.08.2020 19:58, Vladimir Sementsov-Ogievskiy wrote: 30.07.2020 17:15, Andrey Shinkevich wrote: As __dict__ is being extended with class members we do not want to print, add the to_dict() method to classes that returns a dictionary with desired fields and their values. Extend it in subclass

Re: [PATCH v12 01/11] iotests: add test for QCOW2 header dump

2020-08-05 Thread Andrey Shinkevich
On 05.08.2020 14:23, Vladimir Sementsov-Ogievskiy wrote: 30.07.2020 17:15, Andrey Shinkevich wrote: The simple script creates a QCOW2 image and fills it with some data. Two bitmaps are created as well. Then the script reads the image header with extensions from the disk by running the script

Re: [PATCH v7 46/47] iotests: Add test for commit in sub directory

2020-08-02 Thread Andrey Shinkevich
t 0 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) qemu-img: Block job failed: No space left on device + +Testing commit in sub-directory with relative filenames + +Formatting 'subdir/t.IMGFMT.base', fmt=IMGFMT size=1048576 +Formatting 'subdir/t.IMGFMT.mid', fmt=IMGFMT size=1048576 backing_file=t.IMGFMT.base +Formatting 'subdir/t.IMGFMT', fmt=IMGFMT size=1048576 backing_file=t.IMGFMT.mid +Image committed. +qemu-img: Did not find 'subdir/t.IMGFMT.mid' in the backing chain of 'subdir/t.IMGFMT' +Image committed. +Image committed. *** done Reviewed-by: Andrey Shinkevich

Re: [PATCH v7 47/47] iotests: Test committing to overridden backing

2020-08-02 Thread Andrey Shinkevich
out +++ b/tests/qemu-iotests/040.out @@ -1,5 +1,5 @@ -... +. -- -Ran 63 tests +Ran 65 tests OK

Re: [PATCH v7 45/47] iotests: Add filter mirror test cases

2020-08-02 Thread Andrey Shinkevich
Reviewed-by: Andrey Shinkevich

Re: [PATCH v12 01/11] iotests: add test for QCOW2 header dump

2020-07-31 Thread Andrey Shinkevich
On 30.07.2020 22:05, Eric Blake wrote: On 7/30/20 9:15 AM, Andrey Shinkevich wrote: The simple script creates a QCOW2 image and fills it with some data. Two bitmaps are created as well. Then the script reads the image header with extensions from the disk by running the script qcow2.py and dumps

[PATCH v12 05/11] qcow2_format.py: Dump bitmap directory information

2020-07-30 Thread Andrey Shinkevich
27;]) type 1 granularity_bits 16 name_size 8 extra_data_size 0 Suggested-by: Kevin Wolf Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/303.out | 18 +++ tests/qemu-iotests/qcow2_form

[PATCH v12 02/11] qcow2_format.py: make printable data an extension class member

2020-07-30 Thread Andrey Shinkevich
Let us differ binary data type from string one for the extension data variable and keep the string as the QcowHeaderExtension class member. Signed-off-by: Andrey Shinkevich Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/qcow2_format.py | 14

[PATCH v12 01/11] iotests: add test for QCOW2 header dump

2020-07-30 Thread Andrey Shinkevich
test later. Suggested-by: Eric Blake Signed-off-by: Andrey Shinkevich --- tests/qemu-iotests/303 | 59 ++ tests/qemu-iotests/303.out | 64 ++ tests/qemu-iotests/group | 1 + 3 files changed, 124 insertions

[PATCH v12 00/11] iotests: Dump QCOW2 dirty bitmaps metadata

2020-07-30 Thread Andrey Shinkevich
removed from bitmap table dump. 10: The class Qcow2HeaderExtensionsDoc was removed. 11: New. The #303 was extended by dumping QCOW2 metadata in JSON format. Andrey Shinkevich (11): iotests: add test for QCOW2 header dump qcow2_format.py: make printable data an extension class member qco

Re: [PATCH v11 10/11] qcow2_format.py: introduce Qcow2HeaderExtensionsDoc class

2020-07-30 Thread Andrey Shinkevich
On 28.07.2020 14:36, Vladimir Sementsov-Ogievskiy wrote: 17.07.2020 11:14, Andrey Shinkevich wrote: Per original script design, QcowHeader class may dump the QCOW2 header info separately from the QCOW2 extensions info. To implement the to_dict() method for dumping extensions, let us introduce

[PATCH v12 09/11] qcow2_format.py: collect fields to dump in JSON format

2020-07-30 Thread Andrey Shinkevich
Sementsov-Ogievskiy Signed-off-by: Andrey Shinkevich --- tests/qemu-iotests/qcow2_format.py | 34 ++ 1 file changed, 34 insertions(+) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py index 2000de3..a4114cb 100644 --- a/tests

[PATCH v12 07/11] qcow2_format.py: Dump bitmap table serialized entries

2020-07-30 Thread Andrey Shinkevich
Signed-off-by: Andrey Shinkevich --- tests/qemu-iotests/303.out | 4 tests/qemu-iotests/qcow2_format.py | 47 ++ 2 files changed, 51 insertions(+) diff --git a/tests/qemu-iotests/303.out b/tests/qemu-iotests/303.out index dc3739b..d581fb4 100644 --- a

[PATCH v12 04/11] qcow2_format.py: dump bitmap flags in human readable way.

2020-07-30 Thread Andrey Shinkevich
Introduce the class BitmapFlags that parses a bitmap flags mask. Suggested-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/qcow2_format.py | 16 1 file changed, 16 insertions(+) diff --git a

[PATCH v12 08/11] qcow2.py: Introduce '-j' key to dump in JSON format

2020-07-30 Thread Andrey Shinkevich
Add the command key to the qcow2.py arguments list to dump QCOW2 metadata in JSON format. Here is the suggested way to do that. The implementation of the dump in JSON format is in the patch that follows. Signed-off-by: Andrey Shinkevich --- tests/qemu-iotests/qcow2.py| 18

[PATCH v12 10/11] qcow2_format.py: support dumping metadata in JSON format

2020-07-30 Thread Andrey Shinkevich
"bitmap_table": [ { "type": "serialized", "offset": 655360 }, ... Suggested-by: Vl

[PATCH v12 03/11] qcow2_format.py: change Qcow2BitmapExt initialization method

2020-07-30 Thread Andrey Shinkevich
n the initialization chain. The implementation comes with the patch that follows. Suggested-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/qcow2_format.py | 36 ++-- 1 file change

[PATCH v12 11/11] iotests: dump QCOW2 header in JSON in #303

2020-07-30 Thread Andrey Shinkevich
Extend the test case #303 by dumping QCOW2 image metadata in JSON format. Signed-off-by: Andrey Shinkevich --- tests/qemu-iotests/303 | 3 ++ tests/qemu-iotests/303.out | 76 ++ 2 files changed, 79 insertions(+) diff --git a/tests/qemu-iotests

[PATCH v12 06/11] qcow2_format.py: pass cluster size to substructures

2020-07-30 Thread Andrey Shinkevich
The cluster size of an image is the QcowHeader class member and may be obtained by dependent extension structures such as Qcow2BitmapExt for further bitmap table details print. Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/qcow2_format.py

Re: [PATCH v11 10/11] qcow2_format.py: introduce Qcow2HeaderExtensionsDoc class

2020-07-30 Thread Andrey Shinkevich
On 28.07.2020 14:36, Vladimir Sementsov-Ogievskiy wrote: 17.07.2020 11:14, Andrey Shinkevich wrote: Per original script design, QcowHeader class may dump the QCOW2 header info separately from the QCOW2 extensions info. To implement the to_dict() method for dumping extensions, let us introduce

Re: [PATCH v11 09/11] qcow2_format.py: collect fields to dump in JSON format

2020-07-28 Thread Andrey Shinkevich
On 28.07.2020 14:09, Vladimir Sementsov-Ogievskiy wrote: 17.07.2020 11:14, Andrey Shinkevich wrote: As __dict__ is being extended with class members we do not want to print, add the to_dict() method to classes that returns a dictionary with desired fields and their values. Extend it in subclass

Re: [PATCH v11 10/11] qcow2_format.py: introduce Qcow2HeaderExtensionsDoc class

2020-07-28 Thread Andrey Shinkevich
On 28.07.2020 14:36, Vladimir Sementsov-Ogievskiy wrote: 17.07.2020 11:14, Andrey Shinkevich wrote: Per original script design, QcowHeader class may dump the QCOW2 header info separately from the QCOW2 extensions info. To implement the to_dict() method for dumping extensions, let us introduce

Re: [PATCH v7 44/47] iotests: Add filter commit test cases

2020-07-27 Thread Andrey Shinkevich
on3 format string f'{rad_or_write} ..' might be used instead of the .format one. Andrey + pattern_file) +self.assertFalse('Pattern verification failed' in result) + +def setUp(self): ... Reviewed-by: Andrey Shinkevich

Re: [PATCH v7 43/47] iotests: Let complete_and_wait() work with commit

2020-07-27 Thread Andrey Shinkevich
or=completion_error) -self.assert_qmp(event, 'data/type', 'mirror') +self.assertTrue(event['data']['type'] in ['mirror', 'commit']) def pause_wait(self, job_id='job0'): with Timeout(3, "Timeout waiting for job to pause"): Reviewed-by: Andrey Shinkevich

Re: [PATCH v7 42/47] iotests: Test that qcow2's data-file is flushed

2020-07-27 Thread Andrey Shinkevich
et Length Mapped to File Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data Images are identical. Images are identical. + +=== Flushing should flush the data file === + +wrote 512/512 bytes at offset 0 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +Success: qemu-io failed, so the data file was flushed *** done Reviewed-by: Andrey Shinkevich

Re: [PATCH v7 41/47] block: Leave BDS.backing_file constant

2020-07-27 Thread Andrey Shinkevich
)) { +backing_format = bs->backing->bs->drv->format_name; +} In case bdrv_backing_overridden() returns true , should we invoke bdrv_refresh_filename() and assign the format_name then? Andrey +if (backing_format) { +info->backing_filename_format = g_strdup(backing_format); info->has_backing_filename_format = true; } g_free(backing_filename2); ... Reviewed-by: Andrey Shinkevich

Re: [PATCH v7 40/47] block: Inline bdrv_co_block_status_from_*()

2020-07-24 Thread Andrey Shinkevich
.bdrv_co_drain_begin = throttle_co_drain_begin, .bdrv_co_drain_end = throttle_co_drain_end, Reviewed-by: Andrey Shinkevich

Re: [PATCH v7 38/47] block: Drop backing_bs()

2020-07-24 Thread Andrey Shinkevich
ithout using bdrv_find_format() */ Reviewed-by: Andrey Shinkevich

Re: [PATCH v7 37/47] qemu-img: Use child access functions

2020-07-24 Thread Andrey Shinkevich
0x6a0 0x40TEST_DIR/t.IMGFMT +0x6e0 0x120 TEST_DIR/t.IMGFMT.base No errors were found on the image. *** done Reviewed-by: Andrey Shinkevich

Re: [PATCH v7 29/47] blockdev: Use CAF in external_snapshot_prepare()

2020-07-24 Thread Andrey Shinkevich
On 24.07.2020 12:23, Max Reitz wrote: On 20.07.20 18:08, Andrey Shinkevich wrote: On 25.06.2020 18:21, Max Reitz wrote: This allows us to differentiate between filters and nodes with COW backing files: Filters cannot be used as overlays at all (for this function). Signed-off-by: Max Reitz

Re: [PATCH v7 35/47] commit: Deal with filters

2020-07-24 Thread Andrey Shinkevich
On 23.07.2020 20:15, Andrey Shinkevich wrote: On 25.06.2020 18:22, Max Reitz wrote: This includes some permission limiting (for example, we only need to take the RESIZE permission if the base is smaller than the top). Signed-off-by: Max Reitz ---   block/block-backend.c  |  9

Re: [PATCH v7 33/47] mirror: Deal with filters

2020-07-24 Thread Andrey Shinkevich
On 24.07.2020 12:49, Max Reitz wrote: On 22.07.20 20:31, Andrey Shinkevich wrote: On 25.06.2020 18:22, Max Reitz wrote: This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top). Use this

Re: [PATCH v7 28/47] block/null: Implement bdrv_get_allocated_file_size

2020-07-24 Thread Andrey Shinkevich
On 24.07.2020 11:58, Max Reitz wrote: On 20.07.20 17:10, Andrey Shinkevich wrote: On 25.06.2020 18:21, Max Reitz wrote: It is trivial, so we might as well do it. Signed-off-by: Max Reitz ---   block/null.c   | 7 +++   tests/qemu-iotests/153.out | 2 +-   tests/qemu-iotests

Re: [PATCH v7 36/47] nbd: Use CAF when looking for dirty bitmap

2020-07-23 Thread Andrey Shinkevich
cow_bs(bs); } if (bm == NULL) { Reviewed-by: Andrey Shinkevich

Re: [PATCH v7 35/47] commit: Deal with filters

2020-07-23 Thread Andrey Shinkevich
quot;intermediate node", iter, 0, - BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE, - errp); + iter_shared_perms, errp); if (ret < 0) { goto fail; } ... Reviewed-by: Andrey Shinkevich

Re: [PATCH v7 34/47] backup: Deal with filters

2020-07-23 Thread Andrey Shinkevich
to avoid COW if possible. */ ret = bdrv_get_info(target, &bdi); -if (ret == -ENOTSUP && !target->backing) { +if (ret == -ENOTSUP && !target_does_cow) { /* Cluster size is not defined */ ... Reviewed-by: Andrey Shinkevich

Re: [PATCH v7 33/47] mirror: Deal with filters

2020-07-22 Thread Andrey Shinkevich
On 25.06.2020 18:22, Max Reitz wrote: This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top). Use this opportunity to rename qmp_drive_mirror()'s "source" BDS to "target_backing_bs", because that

Re: [PATCH v7 32/47] block-copy: Use CAF to find sync=top base

2020-07-21 Thread Andrey Shinkevich
t;skip_unallocated) { +base = bdrv_backing_chain_next(s->source->bs); } else { base = NULL; } Reviewed-by: Andrey Shinkevich

Re: [PATCH v7 31/47] block: Use child access functions for QAPI queries

2020-07-21 Thread Andrey Shinkevich
otest 184, as the throttled node now appears as a backing child. Signed-off-by: Max Reitz --- block/qapi.c | 33 - tests/qemu-iotests/184.out | 8 +++- 2 files changed, 27 insertions(+), 14 deletions(-) ... Reviewed-by: Andrey Shinkevich

Re: [PATCH v7 30/47] block: Report data child for query-blockstats

2020-07-21 Thread Andrey Shinkevich
f (blk_level && bs->backing) { Reviewed-by: Andrey Shinkevich

Re: [PATCH v7 29/47] blockdev: Use CAF in external_snapshot_prepare()

2020-07-20 Thread Andrey Shinkevich
if (bdrv_cow_child(state->new_bs)) { error_setg(errp, "The overlay already has a backing image"); goto out; } Reviewed-by: Andrey Shinkevich

Re: [PATCH v7 26/47] block: Improve get_allocated_file_size's default

2020-07-20 Thread Andrey Shinkevich
to the size of their primary child */ +return bdrv_primary_allocated_file_size(bs); +} else { +/* Other drivers default to summing their children's sizes */ +return bdrv_sum_allocated_file_size(bs); } -return -ENOTSUP; } /** Reviewed-by: Andrey Shinkevich

Re: [PATCH v7 28/47] block/null: Implement bdrv_get_allocated_file_size

2020-07-20 Thread Andrey Shinkevich
"filename": "null-co://", -"format": "null-co" +"format": "null-co", +"actual-size": SIZE }, "iops_wr": 0, "ro": false, Reviewed-by: Andrey Shinkevich

Re: [PATCH v7 27/47] blkverify: Use bdrv_sum_allocated_file_size()

2020-07-20 Thread Andrey Shinkevich
readv, .bdrv_co_pwritev = blkverify_co_pwritev, Reviewed-by: Andrey Shinkevich

[PATCH v11 04/11] qcow2_format.py: dump bitmap flags in human readable way.

2020-07-17 Thread Andrey Shinkevich
Introduce the class BitmapFlags that parses a bitmap flags mask. Suggested-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/qcow2_format.py | 16 1 file changed, 16 insertions(+) diff --git a

[PATCH v11 06/11] qcow2_format.py: pass cluster size to substructures

2020-07-17 Thread Andrey Shinkevich
The cluster size of an image is the QcowHeader class member and may be obtained by dependent extension structures such as Qcow2BitmapExt for further bitmap table details print. Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/qcow2_format.py

[PATCH v11 03/11] qcow2_format.py: change Qcow2BitmapExt initialization method

2020-07-17 Thread Andrey Shinkevich
n the initialization chain. The implementation comes with the patch that follows. Suggested-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Andrey Shinkevich --- tests/qemu-iotests/qcow2_format.py | 36 ++-- 1 file changed, 22 insertions(+), 14 deletions(-) diff --

<    1   2   3   4   5   6   7   8   9   >