[PATCH v7 1/4] qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 144, 182, 242, 255
Signed-off-by: Denis Plotnikov
Reviewed-by: Vladimir Sementsov-Ogievskiy
---
qapi/block-core.json | 22 +-
block/qcow2.h| 20 +-
include/block/block_int.h| 1 +
block/qcow2.c| 113 +++
tests/qemu-iotests/031.out | 14 ++--
tests/qemu-iotests/036.out | 4 +-
tests/qemu-iotests/049.out | 102 ++--
tests/qemu-iotests/060.out | 1 +
tests/qemu-iotests/061.out | 34 ++
tests/qemu-iotests/065 | 28 +---
tests/qemu-iotests/080 | 2 +-
tests/qemu-iotests/144.out | 4 +-
tests/qemu-iotests/182.out | 2 +-
tests/qemu-iotests/242.out | 5 ++
tests/qemu-iotests/255.out | 8 +--
tests/qemu-iotests/common.filter | 3 +-
16 files changed, 267 insertions(+), 96 deletions(-)
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 85e27bb61f..a306484973 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -78,6 +78,8 @@
#
# @bitmaps: A list of qcow2 bitmap details (since 4.0)
#
+# @compression-type: the image cluster compression method (since 5.0)
+#
# Since: 1.7
##
{ 'struct': 'ImageInfoSpecificQCow2',
@@ -89,7 +91,8 @@
'*corrupt': 'bool',
'refcount-bits': 'int',
'*encrypt': 'ImageInfoSpecificQCow2Encryption',
- '*bitmaps': ['Qcow2BitmapInfo']
+ '*bitmaps': ['Qcow2BitmapInfo'],
+ 'compression-type': 'Qcow2CompressionType'
} }
##
@@ -4392,6 +4395,18 @@
'data': [ 'v2', 'v3' ] }
+##
+# @Qcow2CompressionType:
+#
+# Compression type used in qcow2 image file
+#
+# @zlib: zlib compression, see <http://zlib.net/>
+#
+# Since: 5.0
+##
+{ 'enum': 'Qcow2CompressionType',
+ 'data': [ 'zlib' ] }
+
##
# @BlockdevCreateOptionsQcow2:
#
@@ -4415,6 +4430,8 @@
# allowed values: off, falloc, full, metadata)
# @lazy-refcounts: True if refcounts may be updated lazily (default: off)
# @refcount-bits: Width of reference counts in bits (default: 16)
+# @compression-type: The image cluster compression method
+#(default: zlib, since 5.0)
#
# Since: 2.12
##
@@ -4430,7 +4447,8 @@
'*cluster-size':'size',
'*preallocation': 'PreallocMode',
'*lazy-refcounts': 'bool',
-'*refcount-bits': 'int' } }
+'*refcount-bits': 'int',
+'*compression-type':'Qcow2CompressionType' } }
##
# @BlockdevCreateOptionsQed:
diff --git a/block/qcow2.h b/block/qcow2.h
index 0942126232..cb6bf2ab83 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -146,8 +146,16 @@ typedef struct QCowHeader {
uint32_t refcount_order;
uint32_t header_length;
+
+/* Additional fields */
+uint8_t compression_type;
+
+/* header must be a multiple of 8 */
+uint8_t padding[7];
} QEMU_PACKED QCowHeader;
+QEMU_BUILD_BUG_ON(sizeof(QCowHeader) % 8 != 0);
+
typedef struct QEMU_PACKED QCowSnapshotHeader {
/* header is 8 byte aligned */
uint64_t l1_table_offset;
@@ -216,13 +224,16 @@ enum {
QCOW2_INCOMPAT_DIRTY_BITNR = 0,
QCOW2_INCOMPAT_CORRUPT_BITNR= 1,
QCOW2_INCOMPAT_DATA_FILE_BITNR = 2,
+QCOW2_INCOMPAT_COMPRESSION_BITNR = 3,
QCOW2_INCOMPAT_DIRTY= 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
QCOW2_INCOMPAT_CORRUPT = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
QCOW2_INCOMPAT_DATA_FILE= 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
+
[PATCH v7 0/4] qcow2: Implement zstd cluster compression method
v7: * use qapi_enum_parse instead of the open-coding [Eric] * fix wording, typos and spelling [Eric] v6: * "block/qcow2-threads: fix qcow2_decompress" is removed from the series since it has been accepted by Max already * add compile time checking for Qcow2Header to be a multiple of 8 [Max, Alberto] * report error on qcow2 amending when the compression type is actually chnged [Max] * remove the extra space and the extra new line [Max] * re-arrange acks and signed-off-s [Vladimir] v5: * replace -ENOTSUP with abort in qcow2_co_decompress [Vladimir] * set cluster size for all test cases in the beginning of the 287 test v4: * the series is rebased on top of 01 "block/qcow2-threads: fix qcow2_decompress" * 01 is just a no-change resend to avoid extra dependencies. Still, it may be merged in separate v3: * remove redundant max compression type value check [Vladimir, Eric] (the switch below checks everything) * prevent compression type changing on "qemu-img amend" [Vladimir] * remove zstd config setting, since it has been added already by "migration" patches [Vladimir] * change the compression type error message [Vladimir] * fix alignment and 80-chars exceeding [Vladimir] v2: * rework compression type setting [Vladimir] * squash iotest changes to the compression type introduction patch [Vladimir, Eric] * fix zstd availability checking in zstd iotest [Vladimir] * remove unnecessry casting [Eric] * remove rudundant checks [Eric] * fix compressed cluster layout in qcow2 spec [Vladimir] * fix wording [Eric, Vladimir] * fix compression type filtering in iotests [Eric] v1: the initial series Denis Plotnikov (4): qcow2: introduce compression type feature qcow2: rework the cluster compression routine qcow2: add zstd cluster compression iotests: 287: add qcow2 compression type test docs/interop/qcow2.txt | 20 configure| 2 +- qapi/block-core.json | 23 +++- block/qcow2.h| 20 +++- include/block/block_int.h| 1 + block/qcow2-threads.c| 195 +-- block/qcow2.c| 120 +++ tests/qemu-iotests/031.out | 14 +-- tests/qemu-iotests/036.out | 4 +- tests/qemu-iotests/049.out | 102 tests/qemu-iotests/060.out | 1 + tests/qemu-iotests/061.out | 34 +++--- tests/qemu-iotests/065 | 28 +++-- tests/qemu-iotests/080 | 2 +- tests/qemu-iotests/144.out | 4 +- tests/qemu-iotests/182.out | 2 +- tests/qemu-iotests/242.out | 5 + tests/qemu-iotests/255.out | 8 +- tests/qemu-iotests/287 | 128 tests/qemu-iotests/287.out | 43 +++ tests/qemu-iotests/common.filter | 3 +- tests/qemu-iotests/group | 1 + 22 files changed, 652 insertions(+), 108 deletions(-) create mode 100755 tests/qemu-iotests/287 create mode 100644 tests/qemu-iotests/287.out -- 2.17.0
Re: [PATCH v6 1/4] qcow2: introduce compression type feature
Thanks for the comments.
I'll make the fixes accordingly and re-sent the series shortly.
Denis
On 14.03.2020 00:40, Eric Blake wrote:
On 3/12/20 4:22 AM, Denis Plotnikov wrote:
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression
than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression
type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some
changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for all the tests
Presumably this filter is optional, and we will not use it on the
specific new tests that prove zstd compression works - but that should
be later in the series, so for this patch it is okay.
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feture compression type
feature
backing_file_offset += 56 (8 + 48 -> header_change +
fature_table_change)
feature
(interesting that you have two different changed spellings ;)
* add "compression type" for test output matching when it isn't
filtered
affected tests: 049, 060, 061, 065, 144, 182, 242, 255
Or maybe the comment above should be changed to "many tests" rather
than "all the tests".
Signed-off-by: Denis Plotnikov
Reviewed-by: Vladimir Sementsov-Ogievskiy
---
qapi/block-core.json | 22 +-
block/qcow2.h | 20 -
include/block/block_int.h | 1 +
block/qcow2.c | 121 +++
tests/qemu-iotests/031.out | 14 ++--
tests/qemu-iotests/036.out | 4 +-
tests/qemu-iotests/049.out | 102 +-
tests/qemu-iotests/060.out | 1 +
tests/qemu-iotests/061.out | 34 +
tests/qemu-iotests/065 | 28 ---
tests/qemu-iotests/080 | 2 +-
tests/qemu-iotests/144.out | 4 +-
tests/qemu-iotests/182.out | 2 +-
tests/qemu-iotests/242.out | 5 ++
tests/qemu-iotests/255.out | 8 +-
tests/qemu-iotests/common.filter | 3 +-
16 files changed, 275 insertions(+), 96 deletions(-)
+++ b/block/qcow2.h
@@ -146,8 +146,16 @@ typedef struct QCowHeader {
uint32_t refcount_order;
uint32_t header_length;
+
+ /* Additional fields */
+ uint8_t compression_type;
+
+ /* header must be a multiple of 8 */
+ uint8_t padding[7];
Why two spaces after uint8_t (twice)?
@@ -369,6 +380,13 @@ typedef struct BDRVQcow2State {
bool metadata_preallocation_checked;
bool metadata_preallocation;
+ /*
+ * Compression type used for the image. Default: 0 - ZLIB
+ * The image compression type is set on image creation.
+ * The only way to change the compression type is to convert the
image
+ * with the desired compression type set
Missing trailing '.'. Maybe someday we can get 'qemu-img amend' to
also adjust the compression type in-place; if that's something we
think we might do, then this could be better worded as "For now, the
only way to change...".
+++ b/block/qcow2.c
@@ -1242,6 +1242,48 @@ static int
qcow2_update_options(BlockDriverState *bs, QDict *options,
return ret;
}
+static int validate_compression_type(BDRVQcow2State *s, Error **errp)
+
+static int qcow2_compression_type_from_format(const char *ct)
+{
+ if (g_str_equal(ct, "zlib")) {
+ return QCOW2_COMPRESSION_TYPE_ZLIB;
+ } else {
+ return -EINVAL;
+ }
Why are you open-coding this?
qapi_enum_parse(_lookup, ct, -1, errp)
should do what you use this for, and automatically updates itself when
you add zstd to the qapi enum later.
@@ -3401,6 +3493,8 @@ qcow2_co_create(BlockdevCreateOptions
*create_options, Error **errp)
.refcount_table_offset = cpu_to_be64(cluster_size),
.refcount_table_clusters = cpu_to_be32(1),
.refcount_order = cpu_to_be32(refcount_order),
+ /* don't deal with endians since compression_type is 1 byte
long */
endianness
+ .compression_type = compression_type,
.header_length = cpu_to_
[PATCH v7 3/4] qcow2: add zstd cluster compression
zstd significantly reduces cluster compression time.
It provides better compression performance maintaining
the same level of the compression ratio in comparison with
zlib, which, at the moment, is the only compression
method available.
The performance test results:
Test compresses and decompresses qemu qcow2 image with just
installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G
The test was conducted with brd disk to reduce the influence
of disk subsystem to the test results.
The results is given in seconds.
compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2
[zlib|zstd]_compressed.img uncompressed.img
compression decompression
zlib zstd zlib zstd
real 65.5 16.3 (-75 %)1.9 1.6 (-16 %)
user 65.0 15.85.3 2.5
sys 3.30.22.0 2.0
Both ZLIB and ZSTD gave the same compression ratio: 1.57
compressed image size in both cases: 1.4G
Signed-off-by: Denis Plotnikov
Reviewed-by: Vladimir Sementsov-Ogievskiy
Reviewed-by: Alberto Garcia
QAPI part:
Acked-by: Markus Armbruster
---
docs/interop/qcow2.txt | 20 +++
configure | 2 +-
qapi/block-core.json | 3 +-
block/qcow2-threads.c | 124 +
block/qcow2.c | 7 +++
5 files changed, 154 insertions(+), 2 deletions(-)
diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt
index 5597e24474..9048114445 100644
--- a/docs/interop/qcow2.txt
+++ b/docs/interop/qcow2.txt
@@ -208,6 +208,7 @@ version 2.
Available compression type values:
0: zlib <https://www.zlib.net/>
+1: zstd <http://github.com/facebook/zstd>
=== Header padding ===
@@ -575,11 +576,30 @@ Compressed Clusters Descriptor (x = 62 - (cluster_bits -
8)):
Another compressed cluster may map to the tail of the final
sector used by this compressed cluster.
+The layout of the compressed data depends on the
compression
+type used for the image (see compressed cluster layout).
+
If a cluster is unallocated, read requests shall read the data from the backing
file (except if bit 0 in the Standard Cluster Descriptor is set). If there is
no backing file or the backing file is smaller than the image, they shall read
zeros for all parts that are not covered by the backing file.
+=== Compressed Cluster Layout ===
+
+The compressed cluster data has a layout depending on the compression
+type used for the image, as follows:
+
+Compressed data layout for the available compression types:
+data_space_lenght - data chunk length available to store a compressed cluster.
+(for more details see "Compressed Clusters Descriptor")
+x = data_space_length - 1
+
+0: (default) zlib <http://zlib.net/>:
+Byte 0 - x: the compressed data content
+ all the space provided used for compressed data
+1: zstd <http://github.com/facebook/zstd>:
+Byte 0 - 3: the length of compressed data in bytes
+ 4 - x: the compressed data content
== Snapshots ==
diff --git a/configure b/configure
index caa65f5883..b2a0aa241a 100755
--- a/configure
+++ b/configure
@@ -1835,7 +1835,7 @@ disabled with --disable-FEATURE, default is enabled if
available:
lzfse support of lzfse compression library
(for reading lzfse-compressed dmg images)
zstdsupport for zstd compression library
- (for migration compression)
+ (for migration compression and qcow2 cluster compression)
seccomp seccomp support
coroutine-pool coroutine freelist (better performance)
glusterfs GlusterFS backend
diff --git a/qapi/block-core.json b/qapi/block-core.json
index a306484973..8953451818 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -4401,11 +4401,12 @@
# Compression type used in qcow2 image file
#
# @zlib: zlib compression, see <http://zlib.net/>
+# @zstd: zstd compression, see <http://github.com/facebook/zstd>
#
# Since: 5.0
##
{ 'enum': 'Qcow2CompressionType',
- 'data': [ 'zlib' ] }
+ 'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
##
# @BlockdevCreateOptionsQcow2:
diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
index 7dbaf53489..b2d1c6d395 100644
--- a/block/qcow2-threads.c
+++ b/block/qcow2-threads.c
@@ -28,6 +28,11 @@
#define ZLIB_CONST
#include
+#ifdef CONFIG_ZSTD
+#include
+#include
+#endif
+
#include "qcow2.h"
#include "block/thread-pool
[PATCH v7 4/4] iotests: 287: add qcow2 compression type test
The test checks fulfilling qcow2 requiriements for the compression
type feature and zstd compression type operability.
Signed-off-by: Denis Plotnikov
Reviewed-by: Vladimir Sementsov-Ogievskiy
---
tests/qemu-iotests/287 | 128 +
tests/qemu-iotests/287.out | 43 +
tests/qemu-iotests/group | 1 +
3 files changed, 172 insertions(+)
create mode 100755 tests/qemu-iotests/287
create mode 100644 tests/qemu-iotests/287.out
diff --git a/tests/qemu-iotests/287 b/tests/qemu-iotests/287
new file mode 100755
index 00..49d15b3d43
--- /dev/null
+++ b/tests/qemu-iotests/287
@@ -0,0 +1,128 @@
+#!/usr/bin/env bash
+#
+# Test case for an image using zstd compression
+#
+# Copyright (c) 2020 Virtuozzo International GmbH
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
+owner=dplotni...@virtuozzo.com
+
+seq="$(basename $0)"
+echo "QA output created by $seq"
+
+status=1 # failure is the default!
+
+_cleanup()
+{
+ _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# standard environment
+. ./common.rc
+. ./common.filter
+
+# This tests qocw2-specific low-level functionality
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+
+# for all the cases
+CLUSTER_SIZE=65536
+
+# Check if we can run this test.
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M | grep "Invalid parameter
'zstd'" 2>&1 1>/dev/null
+
+ZSTD_SUPPORTED=$?
+
+if (($ZSTD_SUPPORTED==0)); then
+_notrun "ZSTD is disabled"
+fi
+
+# Test: when compression is zlib the incompatible bit is unset
+echo
+echo "=== Testing compression type incompatible bit setting for zlib ==="
+echo
+
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+# Test: when compression differs from zlib the incompatible bit is set
+echo
+echo "=== Testing compression type incompatible bit setting for zstd ==="
+echo
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+# Test: an image can't be openned if compression type is zlib and
+# incompatible feature compression type is set
+echo
+echo "=== Testing zlib with incompatible bit set ==="
+echo
+
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" set-feature-bit incompatible 3
+# to make sure the bit was actually set
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
+if (($?==0)); then
+echo "Error: The image openned successfully. The image must not be openned"
+fi
+
+# Test: an image can't be openned if compression type is NOT zlib and
+# incompatible feature compression type is UNSET
+echo
+echo "=== Testing zstd with incompatible bit unset ==="
+echo
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" set-header incompatible_features 0
+# to make sure the bit was actually unset
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
+if (($?==0)); then
+echo "Error: The image openned successfully. The image must not be openned"
+fi
+# Test: check compression type values
+echo
+echo "=== Testing compression type values ==="
+echo
+# zlib=0
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+od -j104 -N1 -An -vtu1 "$TEST_IMG"
+
+# zstd=1
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+od -j104 -N1 -An -vtu1 "$TEST_IMG"
+
+# Test: using zstd compression, write to and read from an image
+echo
+echo "=== Testing reading and writing with zstd ==="
+echo
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$QEMU_IO -c "write -c -P 0xAC 65536 64k " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -P 0xAC 65536 65536 " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -v 131070 8 " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -v 65534 8" "$TEST_IMG" | _filter_qemu_io
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
[PATCH v7 2/4] qcow2: rework the cluster compression routine
The patch enables processing the image compression type defined
for the image and chooses an appropriate method for image clusters
(de)compression.
Signed-off-by: Denis Plotnikov
Reviewed-by: Vladimir Sementsov-Ogievskiy
Reviewed-by: Alberto Garcia
---
block/qcow2-threads.c | 71 ---
1 file changed, 60 insertions(+), 11 deletions(-)
diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
index a68126f291..7dbaf53489 100644
--- a/block/qcow2-threads.c
+++ b/block/qcow2-threads.c
@@ -74,7 +74,9 @@ typedef struct Qcow2CompressData {
} Qcow2CompressData;
/*
- * qcow2_compress()
+ * qcow2_zlib_compress()
+ *
+ * Compress @src_size bytes of data using zlib compression method
*
* @dest - destination buffer, @dest_size bytes
* @src - source buffer, @src_size bytes
@@ -83,8 +85,8 @@ typedef struct Qcow2CompressData {
* -ENOMEM destination buffer is not enough to store compressed data
* -EIOon any other error
*/
-static ssize_t qcow2_compress(void *dest, size_t dest_size,
- const void *src, size_t src_size)
+static ssize_t qcow2_zlib_compress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
{
ssize_t ret;
z_stream strm;
@@ -119,10 +121,10 @@ static ssize_t qcow2_compress(void *dest, size_t
dest_size,
}
/*
- * qcow2_decompress()
+ * qcow2_zlib_decompress()
*
* Decompress some data (not more than @src_size bytes) to produce exactly
- * @dest_size bytes.
+ * @dest_size bytes using zlib compression method
*
* @dest - destination buffer, @dest_size bytes
* @src - source buffer, @src_size bytes
@@ -130,8 +132,8 @@ static ssize_t qcow2_compress(void *dest, size_t dest_size,
* Returns: 0 on success
* -EIO on fail
*/
-static ssize_t qcow2_decompress(void *dest, size_t dest_size,
-const void *src, size_t src_size)
+static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
{
int ret;
z_stream strm;
@@ -191,20 +193,67 @@ qcow2_co_do_compress(BlockDriverState *bs, void *dest,
size_t dest_size,
return arg.ret;
}
+/*
+ * qcow2_co_compress()
+ *
+ * Compress @src_size bytes of data using the compression
+ * method defined by the image compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: compressed size on success
+ * a negative error code on failure
+ */
ssize_t coroutine_fn
qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
const void *src, size_t src_size)
{
-return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
-qcow2_compress);
+BDRVQcow2State *s = bs->opaque;
+Qcow2CompressFunc fn;
+
+switch (s->compression_type) {
+case QCOW2_COMPRESSION_TYPE_ZLIB:
+fn = qcow2_zlib_compress;
+break;
+
+default:
+abort();
+}
+
+return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
}
+/*
+ * qcow2_co_decompress()
+ *
+ * Decompress some data (not more than @src_size bytes) to produce exactly
+ * @dest_size bytes using the compression method defined by the image
+ * compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ * a negative error code on failure
+ */
ssize_t coroutine_fn
qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size,
const void *src, size_t src_size)
{
-return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
-qcow2_decompress);
+BDRVQcow2State *s = bs->opaque;
+Qcow2CompressFunc fn;
+
+switch (s->compression_type) {
+case QCOW2_COMPRESSION_TYPE_ZLIB:
+fn = qcow2_zlib_decompress;
+break;
+
+default:
+abort();
+}
+
+return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
}
--
2.17.0
[PATCH v6 3/4] qcow2: add zstd cluster compression
zstd significantly reduces cluster compression time.
It provides better compression performance maintaining
the same level of the compression ratio in comparison with
zlib, which, at the moment, is the only compression
method available.
The performance test results:
Test compresses and decompresses qemu qcow2 image with just
installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G
The test was conducted with brd disk to reduce the influence
of disk subsystem to the test results.
The results is given in seconds.
compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2
[zlib|zstd]_compressed.img uncompressed.img
compression decompression
zlib zstd zlib zstd
real 65.5 16.3 (-75 %)1.9 1.6 (-16 %)
user 65.0 15.85.3 2.5
sys 3.30.22.0 2.0
Both ZLIB and ZSTD gave the same compression ratio: 1.57
compressed image size in both cases: 1.4G
Signed-off-by: Denis Plotnikov
Reviewed-by: Vladimir Sementsov-Ogievskiy
Reviewed-by: Alberto Garcia
QAPI part:
Acked-by: Markus Armbruster
---
docs/interop/qcow2.txt | 20 +++
configure | 2 +-
qapi/block-core.json | 3 +-
block/qcow2-threads.c | 124 +
block/qcow2.c | 11
5 files changed, 158 insertions(+), 2 deletions(-)
diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt
index 5597e24474..9048114445 100644
--- a/docs/interop/qcow2.txt
+++ b/docs/interop/qcow2.txt
@@ -208,6 +208,7 @@ version 2.
Available compression type values:
0: zlib <https://www.zlib.net/>
+1: zstd <http://github.com/facebook/zstd>
=== Header padding ===
@@ -575,11 +576,30 @@ Compressed Clusters Descriptor (x = 62 - (cluster_bits -
8)):
Another compressed cluster may map to the tail of the final
sector used by this compressed cluster.
+The layout of the compressed data depends on the
compression
+type used for the image (see compressed cluster layout).
+
If a cluster is unallocated, read requests shall read the data from the backing
file (except if bit 0 in the Standard Cluster Descriptor is set). If there is
no backing file or the backing file is smaller than the image, they shall read
zeros for all parts that are not covered by the backing file.
+=== Compressed Cluster Layout ===
+
+The compressed cluster data has a layout depending on the compression
+type used for the image, as follows:
+
+Compressed data layout for the available compression types:
+data_space_lenght - data chunk length available to store a compressed cluster.
+(for more details see "Compressed Clusters Descriptor")
+x = data_space_length - 1
+
+0: (default) zlib <http://zlib.net/>:
+Byte 0 - x: the compressed data content
+ all the space provided used for compressed data
+1: zstd <http://github.com/facebook/zstd>:
+Byte 0 - 3: the length of compressed data in bytes
+ 4 - x: the compressed data content
== Snapshots ==
diff --git a/configure b/configure
index caa65f5883..b2a0aa241a 100755
--- a/configure
+++ b/configure
@@ -1835,7 +1835,7 @@ disabled with --disable-FEATURE, default is enabled if
available:
lzfse support of lzfse compression library
(for reading lzfse-compressed dmg images)
zstdsupport for zstd compression library
- (for migration compression)
+ (for migration compression and qcow2 cluster compression)
seccomp seccomp support
coroutine-pool coroutine freelist (better performance)
glusterfs GlusterFS backend
diff --git a/qapi/block-core.json b/qapi/block-core.json
index a306484973..8953451818 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -4401,11 +4401,12 @@
# Compression type used in qcow2 image file
#
# @zlib: zlib compression, see <http://zlib.net/>
+# @zstd: zstd compression, see <http://github.com/facebook/zstd>
#
# Since: 5.0
##
{ 'enum': 'Qcow2CompressionType',
- 'data': [ 'zlib' ] }
+ 'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
##
# @BlockdevCreateOptionsQcow2:
diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
index 7dbaf53489..b2d1c6d395 100644
--- a/block/qcow2-threads.c
+++ b/block/qcow2-threads.c
@@ -28,6 +28,11 @@
#define ZLIB_CONST
#include
+#ifdef CONFIG_ZSTD
+#include
+#include
+#endif
+
#include "qcow2.h"
#include "block/thread-pool
[PATCH v6 1/4] qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for all the tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feture compression type
backing_file_offset += 56 (8 + 48 -> header_change + fature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 144, 182, 242, 255
Signed-off-by: Denis Plotnikov
Reviewed-by: Vladimir Sementsov-Ogievskiy
---
qapi/block-core.json | 22 +-
block/qcow2.h| 20 -
include/block/block_int.h| 1 +
block/qcow2.c| 121 +++
tests/qemu-iotests/031.out | 14 ++--
tests/qemu-iotests/036.out | 4 +-
tests/qemu-iotests/049.out | 102 +-
tests/qemu-iotests/060.out | 1 +
tests/qemu-iotests/061.out | 34 +
tests/qemu-iotests/065 | 28 ---
tests/qemu-iotests/080 | 2 +-
tests/qemu-iotests/144.out | 4 +-
tests/qemu-iotests/182.out | 2 +-
tests/qemu-iotests/242.out | 5 ++
tests/qemu-iotests/255.out | 8 +-
tests/qemu-iotests/common.filter | 3 +-
16 files changed, 275 insertions(+), 96 deletions(-)
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 85e27bb61f..a306484973 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -78,6 +78,8 @@
#
# @bitmaps: A list of qcow2 bitmap details (since 4.0)
#
+# @compression-type: the image cluster compression method (since 5.0)
+#
# Since: 1.7
##
{ 'struct': 'ImageInfoSpecificQCow2',
@@ -89,7 +91,8 @@
'*corrupt': 'bool',
'refcount-bits': 'int',
'*encrypt': 'ImageInfoSpecificQCow2Encryption',
- '*bitmaps': ['Qcow2BitmapInfo']
+ '*bitmaps': ['Qcow2BitmapInfo'],
+ 'compression-type': 'Qcow2CompressionType'
} }
##
@@ -4392,6 +4395,18 @@
'data': [ 'v2', 'v3' ] }
+##
+# @Qcow2CompressionType:
+#
+# Compression type used in qcow2 image file
+#
+# @zlib: zlib compression, see <http://zlib.net/>
+#
+# Since: 5.0
+##
+{ 'enum': 'Qcow2CompressionType',
+ 'data': [ 'zlib' ] }
+
##
# @BlockdevCreateOptionsQcow2:
#
@@ -4415,6 +4430,8 @@
# allowed values: off, falloc, full, metadata)
# @lazy-refcounts: True if refcounts may be updated lazily (default: off)
# @refcount-bits: Width of reference counts in bits (default: 16)
+# @compression-type: The image cluster compression method
+#(default: zlib, since 5.0)
#
# Since: 2.12
##
@@ -4430,7 +4447,8 @@
'*cluster-size':'size',
'*preallocation': 'PreallocMode',
'*lazy-refcounts': 'bool',
-'*refcount-bits': 'int' } }
+'*refcount-bits': 'int',
+'*compression-type':'Qcow2CompressionType' } }
##
# @BlockdevCreateOptionsQed:
diff --git a/block/qcow2.h b/block/qcow2.h
index 0942126232..e434cff9ff 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -146,8 +146,16 @@ typedef struct QCowHeader {
uint32_t refcount_order;
uint32_t header_length;
+
+/* Additional fields */
+uint8_t compression_type;
+
+/* header must be a multiple of 8 */
+uint8_t padding[7];
} QEMU_PACKED QCowHeader;
+QEMU_BUILD_BUG_ON(sizeof(QCowHeader) % 8 != 0);
+
typedef struct QEMU_PACKED QCowSnapshotHeader {
/* header is 8 byte aligned */
uint64_t l1_table_offset;
@@ -216,13 +224,16 @@ enum {
QCOW2_INCOMPAT_DIRTY_BITNR = 0,
QCOW2_INCOMPAT_CORRUPT_BITNR= 1,
QCOW2_INCOMPAT_DATA_FILE_BITNR = 2,
+QCOW2_INCOMPAT_COMPRESSION_BITNR = 3,
QCOW2_INCOMPAT_DIRTY= 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
QCOW2_INCOMPAT_CORRUPT = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
QCOW2_INCOMPAT_DATA_FILE= 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
+
[PATCH v6 2/4] qcow2: rework the cluster compression routine
The patch enables processing the image compression type defined
for the image and chooses an appropriate method for image clusters
(de)compression.
Signed-off-by: Denis Plotnikov
Reviewed-by: Vladimir Sementsov-Ogievskiy
Reviewed-by: Alberto Garcia
---
block/qcow2-threads.c | 71 ---
1 file changed, 60 insertions(+), 11 deletions(-)
diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
index a68126f291..7dbaf53489 100644
--- a/block/qcow2-threads.c
+++ b/block/qcow2-threads.c
@@ -74,7 +74,9 @@ typedef struct Qcow2CompressData {
} Qcow2CompressData;
/*
- * qcow2_compress()
+ * qcow2_zlib_compress()
+ *
+ * Compress @src_size bytes of data using zlib compression method
*
* @dest - destination buffer, @dest_size bytes
* @src - source buffer, @src_size bytes
@@ -83,8 +85,8 @@ typedef struct Qcow2CompressData {
* -ENOMEM destination buffer is not enough to store compressed data
* -EIOon any other error
*/
-static ssize_t qcow2_compress(void *dest, size_t dest_size,
- const void *src, size_t src_size)
+static ssize_t qcow2_zlib_compress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
{
ssize_t ret;
z_stream strm;
@@ -119,10 +121,10 @@ static ssize_t qcow2_compress(void *dest, size_t
dest_size,
}
/*
- * qcow2_decompress()
+ * qcow2_zlib_decompress()
*
* Decompress some data (not more than @src_size bytes) to produce exactly
- * @dest_size bytes.
+ * @dest_size bytes using zlib compression method
*
* @dest - destination buffer, @dest_size bytes
* @src - source buffer, @src_size bytes
@@ -130,8 +132,8 @@ static ssize_t qcow2_compress(void *dest, size_t dest_size,
* Returns: 0 on success
* -EIO on fail
*/
-static ssize_t qcow2_decompress(void *dest, size_t dest_size,
-const void *src, size_t src_size)
+static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
{
int ret;
z_stream strm;
@@ -191,20 +193,67 @@ qcow2_co_do_compress(BlockDriverState *bs, void *dest,
size_t dest_size,
return arg.ret;
}
+/*
+ * qcow2_co_compress()
+ *
+ * Compress @src_size bytes of data using the compression
+ * method defined by the image compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: compressed size on success
+ * a negative error code on failure
+ */
ssize_t coroutine_fn
qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
const void *src, size_t src_size)
{
-return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
-qcow2_compress);
+BDRVQcow2State *s = bs->opaque;
+Qcow2CompressFunc fn;
+
+switch (s->compression_type) {
+case QCOW2_COMPRESSION_TYPE_ZLIB:
+fn = qcow2_zlib_compress;
+break;
+
+default:
+abort();
+}
+
+return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
}
+/*
+ * qcow2_co_decompress()
+ *
+ * Decompress some data (not more than @src_size bytes) to produce exactly
+ * @dest_size bytes using the compression method defined by the image
+ * compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ * a negative error code on failure
+ */
ssize_t coroutine_fn
qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size,
const void *src, size_t src_size)
{
-return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
-qcow2_decompress);
+BDRVQcow2State *s = bs->opaque;
+Qcow2CompressFunc fn;
+
+switch (s->compression_type) {
+case QCOW2_COMPRESSION_TYPE_ZLIB:
+fn = qcow2_zlib_decompress;
+break;
+
+default:
+abort();
+}
+
+return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
}
--
2.17.0
[PATCH v6 0/4] qcow2: Implement zstd cluster compression method
v6: * "block/qcow2-threads: fix qcow2_decompress" is removed from the series since it has been accepted by Max already * add compile time checking for Qcow2Header to be a multiple of 8 [Max, Alberto] * report error on qcow2 amending when the compression type is actually chnged [Max] * remove the extra space and the extra new line [Max] * re-arrange acks and signed-off-s [Vladimir] v5: * replace -ENOTSUP with abort in qcow2_co_decompress [Vladimir] * set cluster size for all test cases in the beginning of the 287 test v4: * the series is rebased on top of 01 "block/qcow2-threads: fix qcow2_decompress" * 01 is just a no-change resend to avoid extra dependencies. Still, it may be merged in separate v3: * remove redundant max compression type value check [Vladimir, Eric] (the switch below checks everything) * prevent compression type changing on "qemu-img amend" [Vladimir] * remove zstd config setting, since it has been added already by "migration" patches [Vladimir] * change the compression type error message [Vladimir] * fix alignment and 80-chars exceeding [Vladimir] v2: * rework compression type setting [Vladimir] * squash iotest changes to the compression type introduction patch [Vladimir, Eric] * fix zstd availability checking in zstd iotest [Vladimir] * remove unnecessry casting [Eric] * remove rudundant checks [Eric] * fix compressed cluster layout in qcow2 spec [Vladimir] * fix wording [Eric, Vladimir] * fix compression type filtering in iotests [Eric] v1: the initial series Denis Plotnikov (4): qcow2: introduce compression type feature qcow2: rework the cluster compression routine qcow2: add zstd cluster compression iotests: 287: add qcow2 compression type test docs/interop/qcow2.txt | 20 configure| 2 +- qapi/block-core.json | 23 +++- block/qcow2.h| 20 +++- include/block/block_int.h| 1 + block/qcow2-threads.c| 195 +-- block/qcow2.c| 132 + tests/qemu-iotests/031.out | 14 +-- tests/qemu-iotests/036.out | 4 +- tests/qemu-iotests/049.out | 102 tests/qemu-iotests/060.out | 1 + tests/qemu-iotests/061.out | 34 +++--- tests/qemu-iotests/065 | 28 +++-- tests/qemu-iotests/080 | 2 +- tests/qemu-iotests/144.out | 4 +- tests/qemu-iotests/182.out | 2 +- tests/qemu-iotests/242.out | 5 + tests/qemu-iotests/255.out | 8 +- tests/qemu-iotests/287 | 128 tests/qemu-iotests/287.out | 43 +++ tests/qemu-iotests/common.filter | 3 +- tests/qemu-iotests/group | 1 + 22 files changed, 664 insertions(+), 108 deletions(-) create mode 100755 tests/qemu-iotests/287 create mode 100644 tests/qemu-iotests/287.out -- 2.17.0
[PATCH v6 4/4] iotests: 287: add qcow2 compression type test
The test checks fulfilling qcow2 requiriements for the compression
type feature and zstd compression type operability.
Signed-off-by: Denis Plotnikov
Reviewed-by: Vladimir Sementsov-Ogievskiy
---
tests/qemu-iotests/287 | 128 +
tests/qemu-iotests/287.out | 43 +
tests/qemu-iotests/group | 1 +
3 files changed, 172 insertions(+)
create mode 100755 tests/qemu-iotests/287
create mode 100644 tests/qemu-iotests/287.out
diff --git a/tests/qemu-iotests/287 b/tests/qemu-iotests/287
new file mode 100755
index 00..49d15b3d43
--- /dev/null
+++ b/tests/qemu-iotests/287
@@ -0,0 +1,128 @@
+#!/usr/bin/env bash
+#
+# Test case for an image using zstd compression
+#
+# Copyright (c) 2020 Virtuozzo International GmbH
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
+owner=dplotni...@virtuozzo.com
+
+seq="$(basename $0)"
+echo "QA output created by $seq"
+
+status=1 # failure is the default!
+
+_cleanup()
+{
+ _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# standard environment
+. ./common.rc
+. ./common.filter
+
+# This tests qocw2-specific low-level functionality
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+
+# for all the cases
+CLUSTER_SIZE=65536
+
+# Check if we can run this test.
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M | grep "Invalid parameter
'zstd'" 2>&1 1>/dev/null
+
+ZSTD_SUPPORTED=$?
+
+if (($ZSTD_SUPPORTED==0)); then
+_notrun "ZSTD is disabled"
+fi
+
+# Test: when compression is zlib the incompatible bit is unset
+echo
+echo "=== Testing compression type incompatible bit setting for zlib ==="
+echo
+
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+# Test: when compression differs from zlib the incompatible bit is set
+echo
+echo "=== Testing compression type incompatible bit setting for zstd ==="
+echo
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+# Test: an image can't be openned if compression type is zlib and
+# incompatible feature compression type is set
+echo
+echo "=== Testing zlib with incompatible bit set ==="
+echo
+
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" set-feature-bit incompatible 3
+# to make sure the bit was actually set
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
+if (($?==0)); then
+echo "Error: The image openned successfully. The image must not be openned"
+fi
+
+# Test: an image can't be openned if compression type is NOT zlib and
+# incompatible feature compression type is UNSET
+echo
+echo "=== Testing zstd with incompatible bit unset ==="
+echo
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" set-header incompatible_features 0
+# to make sure the bit was actually unset
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
+if (($?==0)); then
+echo "Error: The image openned successfully. The image must not be openned"
+fi
+# Test: check compression type values
+echo
+echo "=== Testing compression type values ==="
+echo
+# zlib=0
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+od -j104 -N1 -An -vtu1 "$TEST_IMG"
+
+# zstd=1
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+od -j104 -N1 -An -vtu1 "$TEST_IMG"
+
+# Test: using zstd compression, write to and read from an image
+echo
+echo "=== Testing reading and writing with zstd ==="
+echo
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$QEMU_IO -c "write -c -P 0xAC 65536 64k " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -P 0xAC 65536 65536 " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -v 131070 8 " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -v 65534 8" "$TEST_IMG" | _filter_qemu_io
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
Re: [PATCH v5 0/5] qcow2: Implement zstd cluster compression method
ping! Is there any other comments/concerns/objections/suggestions according to the series except the minor ones from Alberto and Vladimir? If not, please, let me know, so I can resend the series with the minor changes for applying to the corresponding branch. Thanks! Denis On 04.03.2020 16:35, Denis Plotnikov wrote: v5: * replace -ENOTSUP with abort in qcow2_co_decompress [Vladimir] * set cluster size for all test cases in the beginning of the 287 test v4: * the series is rebased on top of 01 "block/qcow2-threads: fix qcow2_decompress" * 01 is just a no-change resend to avoid extra dependencies. Still, it may be merged in separate v3: * remove redundant max compression type value check [Vladimir, Eric] (the switch below checks everything) * prevent compression type changing on "qemu-img amend" [Vladimir] * remove zstd config setting, since it has been added already by "migration" patches [Vladimir] * change the compression type error message [Vladimir] * fix alignment and 80-chars exceeding [Vladimir] v2: * rework compression type setting [Vladimir] * squash iotest changes to the compression type introduction patch [Vladimir, Eric] * fix zstd availability checking in zstd iotest [Vladimir] * remove unnecessry casting [Eric] * remove rudundant checks [Eric] * fix compressed cluster layout in qcow2 spec [Vladimir] * fix wording [Eric, Vladimir] * fix compression type filtering in iotests [Eric] v1: the initial series Denis Plotnikov (4): qcow2: introduce compression type feature qcow2: rework the cluster compression routine qcow2: add zstd cluster compression iotests: 287: add qcow2 compression type test Vladimir Sementsov-Ogievskiy (1): block/qcow2-threads: fix qcow2_decompress docs/interop/qcow2.txt | 20 +++ configure| 2 +- qapi/block-core.json | 23 +++- block/qcow2.h| 18 ++- include/block/block_int.h| 1 + block/qcow2-threads.c| 206 --- block/qcow2.c| 108 tests/qemu-iotests/031.out | 14 +-- tests/qemu-iotests/036.out | 4 +- tests/qemu-iotests/049.out | 102 +++ tests/qemu-iotests/060.out | 1 + tests/qemu-iotests/061.out | 34 ++--- tests/qemu-iotests/065 | 28 +++-- tests/qemu-iotests/080 | 2 +- tests/qemu-iotests/144.out | 4 +- tests/qemu-iotests/182.out | 2 +- tests/qemu-iotests/242.out | 5 + tests/qemu-iotests/255.out | 8 +- tests/qemu-iotests/287 | 128 +++ tests/qemu-iotests/287.out | 43 +++ tests/qemu-iotests/common.filter | 3 +- tests/qemu-iotests/group | 1 + 22 files changed, 644 insertions(+), 113 deletions(-) create mode 100755 tests/qemu-iotests/287 create mode 100644 tests/qemu-iotests/287.out
[PATCH v5 2/5] qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for all the tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feture compression type
backing_file_offset += 56 (8 + 48 -> header_change + fature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 144, 182, 242, 255
Signed-off-by: Denis Plotnikov
Reviewed-by: Vladimir Sementsov-Ogievskiy
---
qapi/block-core.json | 22 ++-
block/qcow2.h| 18 +-
include/block/block_int.h| 1 +
block/qcow2.c| 101 ++
tests/qemu-iotests/031.out | 14 ++---
tests/qemu-iotests/036.out | 4 +-
tests/qemu-iotests/049.out | 102 +++
tests/qemu-iotests/060.out | 1 +
tests/qemu-iotests/061.out | 34 ++-
tests/qemu-iotests/065 | 28 ++---
tests/qemu-iotests/080 | 2 +-
tests/qemu-iotests/144.out | 4 +-
tests/qemu-iotests/182.out | 2 +-
tests/qemu-iotests/242.out | 5 ++
tests/qemu-iotests/255.out | 8 +--
tests/qemu-iotests/common.filter | 3 +-
16 files changed, 253 insertions(+), 96 deletions(-)
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 85e27bb61f..a67eb8bff4 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -78,6 +78,8 @@
#
# @bitmaps: A list of qcow2 bitmap details (since 4.0)
#
+# @compression-type: the image cluster compression method (since 5.0)
+#
# Since: 1.7
##
{ 'struct': 'ImageInfoSpecificQCow2',
@@ -89,7 +91,8 @@
'*corrupt': 'bool',
'refcount-bits': 'int',
'*encrypt': 'ImageInfoSpecificQCow2Encryption',
- '*bitmaps': ['Qcow2BitmapInfo']
+ '*bitmaps': ['Qcow2BitmapInfo'],
+ 'compression-type': 'Qcow2CompressionType'
} }
##
@@ -4392,6 +4395,18 @@
'data': [ 'v2', 'v3' ] }
+##
+# @Qcow2CompressionType:
+#
+# Compression type used in qcow2 image file
+#
+# @zlib: zlib compression, see <http://zlib.net/>
+#
+# Since: 5.0
+##
+{ 'enum': 'Qcow2CompressionType',
+ 'data': [ 'zlib' ] }
+
##
# @BlockdevCreateOptionsQcow2:
#
@@ -4415,6 +4430,8 @@
# allowed values: off, falloc, full, metadata)
# @lazy-refcounts: True if refcounts may be updated lazily (default: off)
# @refcount-bits: Width of reference counts in bits (default: 16)
+# @compression-type: The image cluster compression method
+#(default: zlib, since 5.0)
#
# Since: 2.12
##
@@ -4430,7 +4447,8 @@
'*cluster-size':'size',
'*preallocation': 'PreallocMode',
'*lazy-refcounts': 'bool',
-'*refcount-bits': 'int' } }
+'*refcount-bits': 'int',
+'*compression-type':'Qcow2CompressionType' } }
##
# @BlockdevCreateOptionsQed:
diff --git a/block/qcow2.h b/block/qcow2.h
index 0942126232..485effcb70 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -146,6 +146,12 @@ typedef struct QCowHeader {
uint32_t refcount_order;
uint32_t header_length;
+
+/* Additional fields */
+uint8_t compression_type;
+
+/* header must be a multiple of 8 */
+uint8_t padding[7];
} QEMU_PACKED QCowHeader;
typedef struct QEMU_PACKED QCowSnapshotHeader {
@@ -216,13 +222,16 @@ enum {
QCOW2_INCOMPAT_DIRTY_BITNR = 0,
QCOW2_INCOMPAT_CORRUPT_BITNR= 1,
QCOW2_INCOMPAT_DATA_FILE_BITNR = 2,
+QCOW2_INCOMPAT_COMPRESSION_BITNR = 3,
QCOW2_INCOMPAT_DIRTY= 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
QCOW2_INCOMPAT_CORRUPT = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
QCOW2_INCOMPAT_DATA_FILE= 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
+QCOW2_INCOMPAT_COMPRESSION = 1 << QCOW2_INCOMPAT_COMPRESSION_BITNR,
QCOW2_INCOM
[PATCH v5 5/5] iotests: 287: add qcow2 compression type test
The test checks fulfilling qcow2 requiriements for the compression
type feature and zstd compression type operability.
Signed-off-by: Denis Plotnikov
Reviewed-by: Vladimir Sementsov-Ogievskiy
---
tests/qemu-iotests/287 | 128 +
tests/qemu-iotests/287.out | 43 +
tests/qemu-iotests/group | 1 +
3 files changed, 172 insertions(+)
create mode 100755 tests/qemu-iotests/287
create mode 100644 tests/qemu-iotests/287.out
diff --git a/tests/qemu-iotests/287 b/tests/qemu-iotests/287
new file mode 100755
index 00..49d15b3d43
--- /dev/null
+++ b/tests/qemu-iotests/287
@@ -0,0 +1,128 @@
+#!/usr/bin/env bash
+#
+# Test case for an image using zstd compression
+#
+# Copyright (c) 2020 Virtuozzo International GmbH
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
+owner=dplotni...@virtuozzo.com
+
+seq="$(basename $0)"
+echo "QA output created by $seq"
+
+status=1 # failure is the default!
+
+_cleanup()
+{
+ _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# standard environment
+. ./common.rc
+. ./common.filter
+
+# This tests qocw2-specific low-level functionality
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+
+# for all the cases
+CLUSTER_SIZE=65536
+
+# Check if we can run this test.
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M | grep "Invalid parameter
'zstd'" 2>&1 1>/dev/null
+
+ZSTD_SUPPORTED=$?
+
+if (($ZSTD_SUPPORTED==0)); then
+_notrun "ZSTD is disabled"
+fi
+
+# Test: when compression is zlib the incompatible bit is unset
+echo
+echo "=== Testing compression type incompatible bit setting for zlib ==="
+echo
+
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+# Test: when compression differs from zlib the incompatible bit is set
+echo
+echo "=== Testing compression type incompatible bit setting for zstd ==="
+echo
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+# Test: an image can't be openned if compression type is zlib and
+# incompatible feature compression type is set
+echo
+echo "=== Testing zlib with incompatible bit set ==="
+echo
+
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" set-feature-bit incompatible 3
+# to make sure the bit was actually set
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
+if (($?==0)); then
+echo "Error: The image openned successfully. The image must not be openned"
+fi
+
+# Test: an image can't be openned if compression type is NOT zlib and
+# incompatible feature compression type is UNSET
+echo
+echo "=== Testing zstd with incompatible bit unset ==="
+echo
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" set-header incompatible_features 0
+# to make sure the bit was actually unset
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
+if (($?==0)); then
+echo "Error: The image openned successfully. The image must not be openned"
+fi
+# Test: check compression type values
+echo
+echo "=== Testing compression type values ==="
+echo
+# zlib=0
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+od -j104 -N1 -An -vtu1 "$TEST_IMG"
+
+# zstd=1
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+od -j104 -N1 -An -vtu1 "$TEST_IMG"
+
+# Test: using zstd compression, write to and read from an image
+echo
+echo "=== Testing reading and writing with zstd ==="
+echo
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$QEMU_IO -c "write -c -P 0xAC 65536 64k " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -P 0xAC 65536 65536 " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -v 131070 8 " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -v 65534 8" "$TEST_IMG" | _filter_qemu_io
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
[PATCH v5 4/5] qcow2: add zstd cluster compression
zstd significantly reduces cluster compression time.
It provides better compression performance maintaining
the same level of the compression ratio in comparison with
zlib, which, at the moment, is the only compression
method available.
The performance test results:
Test compresses and decompresses qemu qcow2 image with just
installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G
The test was conducted with brd disk to reduce the influence
of disk subsystem to the test results.
The results is given in seconds.
compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2
[zlib|zstd]_compressed.img uncompressed.img
compression decompression
zlib zstd zlib zstd
real 65.5 16.3 (-75 %)1.9 1.6 (-16 %)
user 65.0 15.85.3 2.5
sys 3.30.22.0 2.0
Both ZLIB and ZSTD gave the same compression ratio: 1.57
compressed image size in both cases: 1.4G
Signed-off-by: Denis Plotnikov
QAPI part:
Acked-by: Markus Armbruster
Reviewed-by: Vladimir Sementsov-Ogievskiy
---
docs/interop/qcow2.txt | 20 +++
configure | 2 +-
qapi/block-core.json | 3 +-
block/qcow2-threads.c | 123 +
block/qcow2.c | 7 +++
5 files changed, 153 insertions(+), 2 deletions(-)
diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt
index 5597e24474..9048114445 100644
--- a/docs/interop/qcow2.txt
+++ b/docs/interop/qcow2.txt
@@ -208,6 +208,7 @@ version 2.
Available compression type values:
0: zlib <https://www.zlib.net/>
+1: zstd <http://github.com/facebook/zstd>
=== Header padding ===
@@ -575,11 +576,30 @@ Compressed Clusters Descriptor (x = 62 - (cluster_bits -
8)):
Another compressed cluster may map to the tail of the final
sector used by this compressed cluster.
+The layout of the compressed data depends on the
compression
+type used for the image (see compressed cluster layout).
+
If a cluster is unallocated, read requests shall read the data from the backing
file (except if bit 0 in the Standard Cluster Descriptor is set). If there is
no backing file or the backing file is smaller than the image, they shall read
zeros for all parts that are not covered by the backing file.
+=== Compressed Cluster Layout ===
+
+The compressed cluster data has a layout depending on the compression
+type used for the image, as follows:
+
+Compressed data layout for the available compression types:
+data_space_lenght - data chunk length available to store a compressed cluster.
+(for more details see "Compressed Clusters Descriptor")
+x = data_space_length - 1
+
+0: (default) zlib <http://zlib.net/>:
+Byte 0 - x: the compressed data content
+ all the space provided used for compressed data
+1: zstd <http://github.com/facebook/zstd>:
+Byte 0 - 3: the length of compressed data in bytes
+ 4 - x: the compressed data content
== Snapshots ==
diff --git a/configure b/configure
index caa65f5883..b2a0aa241a 100755
--- a/configure
+++ b/configure
@@ -1835,7 +1835,7 @@ disabled with --disable-FEATURE, default is enabled if
available:
lzfse support of lzfse compression library
(for reading lzfse-compressed dmg images)
zstdsupport for zstd compression library
- (for migration compression)
+ (for migration compression and qcow2 cluster compression)
seccomp seccomp support
coroutine-pool coroutine freelist (better performance)
glusterfs GlusterFS backend
diff --git a/qapi/block-core.json b/qapi/block-core.json
index a67eb8bff4..84889fb741 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -4401,11 +4401,12 @@
# Compression type used in qcow2 image file
#
# @zlib: zlib compression, see <http://zlib.net/>
+# @zstd: zstd compression, see <http://github.com/facebook/zstd>
#
# Since: 5.0
##
{ 'enum': 'Qcow2CompressionType',
- 'data': [ 'zlib' ] }
+ 'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
##
# @BlockdevCreateOptionsQcow2:
diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
index 7dbaf53489..eeae68e88e 100644
--- a/block/qcow2-threads.c
+++ b/block/qcow2-threads.c
@@ -28,6 +28,11 @@
#define ZLIB_CONST
#include
+#ifdef CONFIG_ZSTD
+#include
+#include
+#endif
+
#include "qcow2.h"
#include "block/thread-pool.h"
#include "cr
[PATCH v5 3/5] qcow2: rework the cluster compression routine
The patch enables processing the image compression type defined
for the image and chooses an appropriate method for image clusters
(de)compression.
Signed-off-by: Denis Plotnikov
Reviewed-by: Vladimir Sementsov-Ogievskiy
---
block/qcow2-threads.c | 71 ---
1 file changed, 60 insertions(+), 11 deletions(-)
diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
index a68126f291..7dbaf53489 100644
--- a/block/qcow2-threads.c
+++ b/block/qcow2-threads.c
@@ -74,7 +74,9 @@ typedef struct Qcow2CompressData {
} Qcow2CompressData;
/*
- * qcow2_compress()
+ * qcow2_zlib_compress()
+ *
+ * Compress @src_size bytes of data using zlib compression method
*
* @dest - destination buffer, @dest_size bytes
* @src - source buffer, @src_size bytes
@@ -83,8 +85,8 @@ typedef struct Qcow2CompressData {
* -ENOMEM destination buffer is not enough to store compressed data
* -EIOon any other error
*/
-static ssize_t qcow2_compress(void *dest, size_t dest_size,
- const void *src, size_t src_size)
+static ssize_t qcow2_zlib_compress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
{
ssize_t ret;
z_stream strm;
@@ -119,10 +121,10 @@ static ssize_t qcow2_compress(void *dest, size_t
dest_size,
}
/*
- * qcow2_decompress()
+ * qcow2_zlib_decompress()
*
* Decompress some data (not more than @src_size bytes) to produce exactly
- * @dest_size bytes.
+ * @dest_size bytes using zlib compression method
*
* @dest - destination buffer, @dest_size bytes
* @src - source buffer, @src_size bytes
@@ -130,8 +132,8 @@ static ssize_t qcow2_compress(void *dest, size_t dest_size,
* Returns: 0 on success
* -EIO on fail
*/
-static ssize_t qcow2_decompress(void *dest, size_t dest_size,
-const void *src, size_t src_size)
+static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
{
int ret;
z_stream strm;
@@ -191,20 +193,67 @@ qcow2_co_do_compress(BlockDriverState *bs, void *dest,
size_t dest_size,
return arg.ret;
}
+/*
+ * qcow2_co_compress()
+ *
+ * Compress @src_size bytes of data using the compression
+ * method defined by the image compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: compressed size on success
+ * a negative error code on failure
+ */
ssize_t coroutine_fn
qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
const void *src, size_t src_size)
{
-return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
-qcow2_compress);
+BDRVQcow2State *s = bs->opaque;
+Qcow2CompressFunc fn;
+
+switch (s->compression_type) {
+case QCOW2_COMPRESSION_TYPE_ZLIB:
+fn = qcow2_zlib_compress;
+break;
+
+default:
+abort();
+}
+
+return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
}
+/*
+ * qcow2_co_decompress()
+ *
+ * Decompress some data (not more than @src_size bytes) to produce exactly
+ * @dest_size bytes using the compression method defined by the image
+ * compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ * a negative error code on failure
+ */
ssize_t coroutine_fn
qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size,
const void *src, size_t src_size)
{
-return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
-qcow2_decompress);
+BDRVQcow2State *s = bs->opaque;
+Qcow2CompressFunc fn;
+
+switch (s->compression_type) {
+case QCOW2_COMPRESSION_TYPE_ZLIB:
+fn = qcow2_zlib_decompress;
+break;
+
+default:
+abort();
+}
+
+return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
}
--
2.17.0
[PATCH v5 0/5] qcow2: Implement zstd cluster compression method
v5: * replace -ENOTSUP with abort in qcow2_co_decompress [Vladimir] * set cluster size for all test cases in the beginning of the 287 test v4: * the series is rebased on top of 01 "block/qcow2-threads: fix qcow2_decompress" * 01 is just a no-change resend to avoid extra dependencies. Still, it may be merged in separate v3: * remove redundant max compression type value check [Vladimir, Eric] (the switch below checks everything) * prevent compression type changing on "qemu-img amend" [Vladimir] * remove zstd config setting, since it has been added already by "migration" patches [Vladimir] * change the compression type error message [Vladimir] * fix alignment and 80-chars exceeding [Vladimir] v2: * rework compression type setting [Vladimir] * squash iotest changes to the compression type introduction patch [Vladimir, Eric] * fix zstd availability checking in zstd iotest [Vladimir] * remove unnecessry casting [Eric] * remove rudundant checks [Eric] * fix compressed cluster layout in qcow2 spec [Vladimir] * fix wording [Eric, Vladimir] * fix compression type filtering in iotests [Eric] v1: the initial series Denis Plotnikov (4): qcow2: introduce compression type feature qcow2: rework the cluster compression routine qcow2: add zstd cluster compression iotests: 287: add qcow2 compression type test Vladimir Sementsov-Ogievskiy (1): block/qcow2-threads: fix qcow2_decompress docs/interop/qcow2.txt | 20 +++ configure| 2 +- qapi/block-core.json | 23 +++- block/qcow2.h| 18 ++- include/block/block_int.h| 1 + block/qcow2-threads.c| 206 --- block/qcow2.c| 108 tests/qemu-iotests/031.out | 14 +-- tests/qemu-iotests/036.out | 4 +- tests/qemu-iotests/049.out | 102 +++ tests/qemu-iotests/060.out | 1 + tests/qemu-iotests/061.out | 34 ++--- tests/qemu-iotests/065 | 28 +++-- tests/qemu-iotests/080 | 2 +- tests/qemu-iotests/144.out | 4 +- tests/qemu-iotests/182.out | 2 +- tests/qemu-iotests/242.out | 5 + tests/qemu-iotests/255.out | 8 +- tests/qemu-iotests/287 | 128 +++ tests/qemu-iotests/287.out | 43 +++ tests/qemu-iotests/common.filter | 3 +- tests/qemu-iotests/group | 1 + 22 files changed, 644 insertions(+), 113 deletions(-) create mode 100755 tests/qemu-iotests/287 create mode 100644 tests/qemu-iotests/287.out -- 2.17.0
[PATCH v5 1/5] block/qcow2-threads: fix qcow2_decompress
From: Vladimir Sementsov-Ogievskiy
On success path we return what inflate() returns instead of 0. And it
most probably works for Z_STREAM_END as it is positive, but is
definitely broken for Z_BUF_ERROR.
While being here, switch to errno return code, to be closer to
qcow2_compress API (and usual expectations).
Revert condition in if to be more positive. Drop dead initialization of
ret.
Cc: qemu-sta...@nongnu.org # v4.0
Fixes: 341926ab83e2b
Signed-off-by: Vladimir Sementsov-Ogievskiy
Reviewed-by: Alberto Garcia
---
block/qcow2-threads.c | 12 +++-
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
index 77bb578cdf..a68126f291 100644
--- a/block/qcow2-threads.c
+++ b/block/qcow2-threads.c
@@ -128,12 +128,12 @@ static ssize_t qcow2_compress(void *dest, size_t
dest_size,
* @src - source buffer, @src_size bytes
*
* Returns: 0 on success
- * -1 on fail
+ * -EIO on fail
*/
static ssize_t qcow2_decompress(void *dest, size_t dest_size,
const void *src, size_t src_size)
{
-int ret = 0;
+int ret;
z_stream strm;
memset(, 0, sizeof(strm));
@@ -144,17 +144,19 @@ static ssize_t qcow2_decompress(void *dest, size_t
dest_size,
ret = inflateInit2(, -12);
if (ret != Z_OK) {
-return -1;
+return -EIO;
}
ret = inflate(, Z_FINISH);
-if ((ret != Z_STREAM_END && ret != Z_BUF_ERROR) || strm.avail_out != 0) {
+if ((ret == Z_STREAM_END || ret == Z_BUF_ERROR) && strm.avail_out == 0) {
/*
* We approve Z_BUF_ERROR because we need @dest buffer to be filled,
but
* @src buffer may be processed partly (because in qcow2 we know size
of
* compressed data with precision of one sector)
*/
-ret = -1;
+ret = 0;
+} else {
+ret = -EIO;
}
inflateEnd();
--
2.17.0
Re: [PATCH v4 5/5] iotests: 287: add qcow2 compression type test
On 04.03.2020 14:27, Vladimir Sementsov-Ogievskiy wrote: 03.03.2020 16:34, Denis Plotnikov wrote: The test checks fulfilling qcow2 requiriements for the compression type feature and zstd compression type operability. Signed-off-by: Denis Plotnikov --- tests/qemu-iotests/287 | 127 + tests/qemu-iotests/287.out | 43 + tests/qemu-iotests/group | 1 + 3 files changed, 171 insertions(+) create mode 100755 tests/qemu-iotests/287 create mode 100644 tests/qemu-iotests/287.out diff --git a/tests/qemu-iotests/287 b/tests/qemu-iotests/287 new file mode 100755 index 00..39cb665c85 --- /dev/null +++ b/tests/qemu-iotests/287 [..] +# Test: using zstd compression, write to and read from an image +echo +echo "=== Testing reading and writing with zstd ===" +echo + +CLUSTER_SIZE=65536 +IMGOPTS='compression_type=zstd' _make_test_img 64M As I understand, you should define env variable assignments on the same line with _make_test_img so that they be passed to it, like CLUSTER_SIZE=65536 IMGOPTS='compression_type=zstd' _make_test_img 64M It works like a regular env variable and can be defined on another line above. Anyway, I'll move "CLUSTER_SIZE=65536" to the beginning of the test to avoid any confusions. with this: Reviewed-by: Vladimir Sementsov-Ogievskiy Thanks for reviewing the series! l'll send v5 with all modifications shortly. Denis +$QEMU_IO -c "write -c -P 0xAC 65536 64k " "$TEST_IMG" | _filter_qemu_io you may s/65536/64k/ +$QEMU_IO -c "read -P 0xAC 65536 65536 " "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c "read -v 131070 8 " "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c "read -v 65534 8" "$TEST_IMG" | _filter_qemu_io + [..]
Re: [PATCH v4 4/5] qcow2: add zstd cluster compression
On 04.03.2020 10:49, Vladimir Sementsov-Ogievskiy wrote:
03.03.2020 16:34, Denis Plotnikov wrote:
zstd significantly reduces cluster compression time.
It provides better compression performance maintaining
the same level of the compression ratio in comparison with
zlib, which, at the moment, is the only compression
method available.
The performance test results:
Test compresses and decompresses qemu qcow2 image with just
installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G
The test was conducted with brd disk to reduce the influence
of disk subsystem to the test results.
The results is given in seconds.
compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2
[zlib|zstd]_compressed.img uncompressed.img
compression decompression
zlib zstd zlib zstd
real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %)
user 65.0 15.8 5.3 2.5
sys 3.3 0.2 2.0 2.0
Both ZLIB and ZSTD gave the same compression ratio: 1.57
compressed image size in both cases: 1.4G
Signed-off-by: Denis Plotnikov
---
[..]
+static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
+{
+ size_t ret;
+
+ /*
+ * steal ZSTD_LEN_BUF bytes in the very beginning of the buffer
+ * to store compressed chunk size
+ */
+ char *d_buf = ((char *) dest) + ZSTD_LEN_BUF;
+
+ /*
+ * sanity check that we can store the compressed data length,
+ * and there is some space left for the compressor buffer
+ */
+ if (dest_size <= ZSTD_LEN_BUF) {
+ return -ENOMEM;
+ }
+
+ dest_size -= ZSTD_LEN_BUF;
+
+ ret = ZSTD_compress(d_buf, dest_size, src, src_size, 5);
You may want to define ZSTD_COMPRESSION_LEVEL constant instead of raw
number.
I didn't introduce it intentionally. zlib compression has the
compression level hardcoded as well.
I think it's better to introduce the compression level for both of them
in the future but not in the scope of this series.
anyway,
Reviewed-by: Vladimir Sementsov-Ogievskiy
[PATCH v4 4/5] qcow2: add zstd cluster compression
zstd significantly reduces cluster compression time.
It provides better compression performance maintaining
the same level of the compression ratio in comparison with
zlib, which, at the moment, is the only compression
method available.
The performance test results:
Test compresses and decompresses qemu qcow2 image with just
installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G
The test was conducted with brd disk to reduce the influence
of disk subsystem to the test results.
The results is given in seconds.
compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2
[zlib|zstd]_compressed.img uncompressed.img
compression decompression
zlib zstd zlib zstd
real 65.5 16.3 (-75 %)1.9 1.6 (-16 %)
user 65.0 15.85.3 2.5
sys 3.30.22.0 2.0
Both ZLIB and ZSTD gave the same compression ratio: 1.57
compressed image size in both cases: 1.4G
Signed-off-by: Denis Plotnikov
---
docs/interop/qcow2.txt | 20 +++
configure | 2 +-
qapi/block-core.json | 3 +-
block/qcow2-threads.c | 123 +
block/qcow2.c | 7 +++
5 files changed, 153 insertions(+), 2 deletions(-)
diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt
index 5597e24474..9048114445 100644
--- a/docs/interop/qcow2.txt
+++ b/docs/interop/qcow2.txt
@@ -208,6 +208,7 @@ version 2.
Available compression type values:
0: zlib <https://www.zlib.net/>
+1: zstd <http://github.com/facebook/zstd>
=== Header padding ===
@@ -575,11 +576,30 @@ Compressed Clusters Descriptor (x = 62 - (cluster_bits -
8)):
Another compressed cluster may map to the tail of the final
sector used by this compressed cluster.
+The layout of the compressed data depends on the
compression
+type used for the image (see compressed cluster layout).
+
If a cluster is unallocated, read requests shall read the data from the backing
file (except if bit 0 in the Standard Cluster Descriptor is set). If there is
no backing file or the backing file is smaller than the image, they shall read
zeros for all parts that are not covered by the backing file.
+=== Compressed Cluster Layout ===
+
+The compressed cluster data has a layout depending on the compression
+type used for the image, as follows:
+
+Compressed data layout for the available compression types:
+data_space_lenght - data chunk length available to store a compressed cluster.
+(for more details see "Compressed Clusters Descriptor")
+x = data_space_length - 1
+
+0: (default) zlib <http://zlib.net/>:
+Byte 0 - x: the compressed data content
+ all the space provided used for compressed data
+1: zstd <http://github.com/facebook/zstd>:
+Byte 0 - 3: the length of compressed data in bytes
+ 4 - x: the compressed data content
== Snapshots ==
diff --git a/configure b/configure
index caa65f5883..b2a0aa241a 100755
--- a/configure
+++ b/configure
@@ -1835,7 +1835,7 @@ disabled with --disable-FEATURE, default is enabled if
available:
lzfse support of lzfse compression library
(for reading lzfse-compressed dmg images)
zstdsupport for zstd compression library
- (for migration compression)
+ (for migration compression and qcow2 cluster compression)
seccomp seccomp support
coroutine-pool coroutine freelist (better performance)
glusterfs GlusterFS backend
diff --git a/qapi/block-core.json b/qapi/block-core.json
index a67eb8bff4..84889fb741 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -4401,11 +4401,12 @@
# Compression type used in qcow2 image file
#
# @zlib: zlib compression, see <http://zlib.net/>
+# @zstd: zstd compression, see <http://github.com/facebook/zstd>
#
# Since: 5.0
##
{ 'enum': 'Qcow2CompressionType',
- 'data': [ 'zlib' ] }
+ 'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
##
# @BlockdevCreateOptionsQcow2:
diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
index 9bfcda6918..0d09208d27 100644
--- a/block/qcow2-threads.c
+++ b/block/qcow2-threads.c
@@ -28,6 +28,11 @@
#define ZLIB_CONST
#include
+#ifdef CONFIG_ZSTD
+#include
+#include
+#endif
+
#include "qcow2.h"
#include "block/thread-pool.h"
#include "crypto.h"
@@ -166,6 +171,114 @@ static ssize_t qcow2_zlib_decompress(void *d
[PATCH v4 5/5] iotests: 287: add qcow2 compression type test
The test checks fulfilling qcow2 requiriements for the compression
type feature and zstd compression type operability.
Signed-off-by: Denis Plotnikov
---
tests/qemu-iotests/287 | 127 +
tests/qemu-iotests/287.out | 43 +
tests/qemu-iotests/group | 1 +
3 files changed, 171 insertions(+)
create mode 100755 tests/qemu-iotests/287
create mode 100644 tests/qemu-iotests/287.out
diff --git a/tests/qemu-iotests/287 b/tests/qemu-iotests/287
new file mode 100755
index 00..39cb665c85
--- /dev/null
+++ b/tests/qemu-iotests/287
@@ -0,0 +1,127 @@
+#!/usr/bin/env bash
+#
+# Test case for an image using zstd compression
+#
+# Copyright (c) 2020 Virtuozzo International GmbH
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
+owner=dplotni...@virtuozzo.com
+
+seq="$(basename $0)"
+echo "QA output created by $seq"
+
+status=1 # failure is the default!
+
+_cleanup()
+{
+ _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# standard environment
+. ./common.rc
+. ./common.filter
+
+# This tests qocw2-specific low-level functionality
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+
+# Check if we can run this test.
+IMGOPTS='compression_type=zstd'
+
+_make_test_img 64M | grep "Invalid parameter 'zstd'" 2>&1 1>/dev/null
+
+ZSTD_SUPPORTED=$?
+
+if (($ZSTD_SUPPORTED==0)); then
+_notrun "ZSTD is disabled"
+fi
+
+# Test: when compression is zlib the incompatible bit is unset
+echo
+echo "=== Testing compression type incompatible bit setting for zlib ==="
+echo
+
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+# Test: when compression differs from zlib the incompatible bit is set
+echo
+echo "=== Testing compression type incompatible bit setting for zstd ==="
+echo
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+# Test: an image can't be openned if compression type is zlib and
+# incompatible feature compression type is set
+echo
+echo "=== Testing zlib with incompatible bit set ==="
+echo
+
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" set-feature-bit incompatible 3
+# to make sure the bit was actually set
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
+if (($?==0)); then
+echo "Error: The image openned successfully. The image must not be openned"
+fi
+
+# Test: an image can't be openned if compression type is NOT zlib and
+# incompatible feature compression type is UNSET
+echo
+echo "=== Testing zstd with incompatible bit unset ==="
+echo
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" set-header incompatible_features 0
+# to make sure the bit was actually unset
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
+if (($?==0)); then
+echo "Error: The image openned successfully. The image must not be openned"
+fi
+# Test: check compression type values
+echo
+echo "=== Testing compression type values ==="
+echo
+# zlib=0
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+od -j104 -N1 -An -vtu1 "$TEST_IMG"
+
+# zstd=1
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+od -j104 -N1 -An -vtu1 "$TEST_IMG"
+
+# Test: using zstd compression, write to and read from an image
+echo
+echo "=== Testing reading and writing with zstd ==="
+echo
+
+CLUSTER_SIZE=65536
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$QEMU_IO -c "write -c -P 0xAC 65536 64k " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -P 0xAC 65536 65536 " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -v 131070 8 " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -v 65534 8" "$TEST_IMG" | _filter_qemu_io
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/287.out b/tests/qemu
[PATCH v4 3/5] qcow2: rework the cluster compression routine
The patch enables processing the image compression type defined
for the image and chooses an appropriate method for image clusters
(de)compression.
Signed-off-by: Denis Plotnikov
---
block/qcow2-threads.c | 71 ---
1 file changed, 60 insertions(+), 11 deletions(-)
diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
index a68126f291..9bfcda6918 100644
--- a/block/qcow2-threads.c
+++ b/block/qcow2-threads.c
@@ -74,7 +74,9 @@ typedef struct Qcow2CompressData {
} Qcow2CompressData;
/*
- * qcow2_compress()
+ * qcow2_zlib_compress()
+ *
+ * Compress @src_size bytes of data using zlib compression method
*
* @dest - destination buffer, @dest_size bytes
* @src - source buffer, @src_size bytes
@@ -83,8 +85,8 @@ typedef struct Qcow2CompressData {
* -ENOMEM destination buffer is not enough to store compressed data
* -EIOon any other error
*/
-static ssize_t qcow2_compress(void *dest, size_t dest_size,
- const void *src, size_t src_size)
+static ssize_t qcow2_zlib_compress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
{
ssize_t ret;
z_stream strm;
@@ -119,10 +121,10 @@ static ssize_t qcow2_compress(void *dest, size_t
dest_size,
}
/*
- * qcow2_decompress()
+ * qcow2_zlib_decompress()
*
* Decompress some data (not more than @src_size bytes) to produce exactly
- * @dest_size bytes.
+ * @dest_size bytes using zlib compression method
*
* @dest - destination buffer, @dest_size bytes
* @src - source buffer, @src_size bytes
@@ -130,8 +132,8 @@ static ssize_t qcow2_compress(void *dest, size_t dest_size,
* Returns: 0 on success
* -EIO on fail
*/
-static ssize_t qcow2_decompress(void *dest, size_t dest_size,
-const void *src, size_t src_size)
+static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
{
int ret;
z_stream strm;
@@ -191,20 +193,67 @@ qcow2_co_do_compress(BlockDriverState *bs, void *dest,
size_t dest_size,
return arg.ret;
}
+/*
+ * qcow2_co_compress()
+ *
+ * Compress @src_size bytes of data using the compression
+ * method defined by the image compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: compressed size on success
+ * a negative error code on failure
+ */
ssize_t coroutine_fn
qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
const void *src, size_t src_size)
{
-return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
-qcow2_compress);
+BDRVQcow2State *s = bs->opaque;
+Qcow2CompressFunc fn;
+
+switch (s->compression_type) {
+case QCOW2_COMPRESSION_TYPE_ZLIB:
+fn = qcow2_zlib_compress;
+break;
+
+default:
+abort();
+}
+
+return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
}
+/*
+ * qcow2_co_decompress()
+ *
+ * Decompress some data (not more than @src_size bytes) to produce exactly
+ * @dest_size bytes using the compression method defined by the image
+ * compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ * a negative error code on failure
+ */
ssize_t coroutine_fn
qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size,
const void *src, size_t src_size)
{
-return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
-qcow2_decompress);
+BDRVQcow2State *s = bs->opaque;
+Qcow2CompressFunc fn;
+
+switch (s->compression_type) {
+case QCOW2_COMPRESSION_TYPE_ZLIB:
+fn = qcow2_zlib_decompress;
+break;
+
+default:
+return -ENOTSUP;
+}
+
+return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
}
--
2.17.0
[PATCH v4 1/5] block/qcow2-threads: fix qcow2_decompress
From: Vladimir Sementsov-Ogievskiy
On success path we return what inflate() returns instead of 0. And it
most probably works for Z_STREAM_END as it is positive, but is
definitely broken for Z_BUF_ERROR.
While being here, switch to errno return code, to be closer to
qcow2_compress API (and usual expectations).
Revert condition in if to be more positive. Drop dead initialization of
ret.
Cc: qemu-sta...@nongnu.org # v4.0
Fixes: 341926ab83e2b
Signed-off-by: Vladimir Sementsov-Ogievskiy
Reviewed-by: Alberto Garcia
---
block/qcow2-threads.c | 12 +++-
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
index 77bb578cdf..a68126f291 100644
--- a/block/qcow2-threads.c
+++ b/block/qcow2-threads.c
@@ -128,12 +128,12 @@ static ssize_t qcow2_compress(void *dest, size_t
dest_size,
* @src - source buffer, @src_size bytes
*
* Returns: 0 on success
- * -1 on fail
+ * -EIO on fail
*/
static ssize_t qcow2_decompress(void *dest, size_t dest_size,
const void *src, size_t src_size)
{
-int ret = 0;
+int ret;
z_stream strm;
memset(, 0, sizeof(strm));
@@ -144,17 +144,19 @@ static ssize_t qcow2_decompress(void *dest, size_t
dest_size,
ret = inflateInit2(, -12);
if (ret != Z_OK) {
-return -1;
+return -EIO;
}
ret = inflate(, Z_FINISH);
-if ((ret != Z_STREAM_END && ret != Z_BUF_ERROR) || strm.avail_out != 0) {
+if ((ret == Z_STREAM_END || ret == Z_BUF_ERROR) && strm.avail_out == 0) {
/*
* We approve Z_BUF_ERROR because we need @dest buffer to be filled,
but
* @src buffer may be processed partly (because in qcow2 we know size
of
* compressed data with precision of one sector)
*/
-ret = -1;
+ret = 0;
+} else {
+ret = -EIO;
}
inflateEnd();
--
2.17.0
[PATCH v4 2/5] qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for all the tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feture compression type
backing_file_offset += 56 (8 + 48 -> header_change + fature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 144, 182, 242, 255
Signed-off-by: Denis Plotnikov
---
qapi/block-core.json | 22 ++-
block/qcow2.h| 18 +-
include/block/block_int.h| 1 +
block/qcow2.c| 101 ++
tests/qemu-iotests/031.out | 14 ++---
tests/qemu-iotests/036.out | 4 +-
tests/qemu-iotests/049.out | 102 +++
tests/qemu-iotests/060.out | 1 +
tests/qemu-iotests/061.out | 34 ++-
tests/qemu-iotests/065 | 28 ++---
tests/qemu-iotests/080 | 2 +-
tests/qemu-iotests/144.out | 4 +-
tests/qemu-iotests/182.out | 2 +-
tests/qemu-iotests/242.out | 5 ++
tests/qemu-iotests/255.out | 8 +--
tests/qemu-iotests/common.filter | 3 +-
16 files changed, 253 insertions(+), 96 deletions(-)
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 85e27bb61f..a67eb8bff4 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -78,6 +78,8 @@
#
# @bitmaps: A list of qcow2 bitmap details (since 4.0)
#
+# @compression-type: the image cluster compression method (since 5.0)
+#
# Since: 1.7
##
{ 'struct': 'ImageInfoSpecificQCow2',
@@ -89,7 +91,8 @@
'*corrupt': 'bool',
'refcount-bits': 'int',
'*encrypt': 'ImageInfoSpecificQCow2Encryption',
- '*bitmaps': ['Qcow2BitmapInfo']
+ '*bitmaps': ['Qcow2BitmapInfo'],
+ 'compression-type': 'Qcow2CompressionType'
} }
##
@@ -4392,6 +4395,18 @@
'data': [ 'v2', 'v3' ] }
+##
+# @Qcow2CompressionType:
+#
+# Compression type used in qcow2 image file
+#
+# @zlib: zlib compression, see <http://zlib.net/>
+#
+# Since: 5.0
+##
+{ 'enum': 'Qcow2CompressionType',
+ 'data': [ 'zlib' ] }
+
##
# @BlockdevCreateOptionsQcow2:
#
@@ -4415,6 +4430,8 @@
# allowed values: off, falloc, full, metadata)
# @lazy-refcounts: True if refcounts may be updated lazily (default: off)
# @refcount-bits: Width of reference counts in bits (default: 16)
+# @compression-type: The image cluster compression method
+#(default: zlib, since 5.0)
#
# Since: 2.12
##
@@ -4430,7 +4447,8 @@
'*cluster-size':'size',
'*preallocation': 'PreallocMode',
'*lazy-refcounts': 'bool',
-'*refcount-bits': 'int' } }
+'*refcount-bits': 'int',
+'*compression-type':'Qcow2CompressionType' } }
##
# @BlockdevCreateOptionsQed:
diff --git a/block/qcow2.h b/block/qcow2.h
index 0942126232..485effcb70 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -146,6 +146,12 @@ typedef struct QCowHeader {
uint32_t refcount_order;
uint32_t header_length;
+
+/* Additional fields */
+uint8_t compression_type;
+
+/* header must be a multiple of 8 */
+uint8_t padding[7];
} QEMU_PACKED QCowHeader;
typedef struct QEMU_PACKED QCowSnapshotHeader {
@@ -216,13 +222,16 @@ enum {
QCOW2_INCOMPAT_DIRTY_BITNR = 0,
QCOW2_INCOMPAT_CORRUPT_BITNR= 1,
QCOW2_INCOMPAT_DATA_FILE_BITNR = 2,
+QCOW2_INCOMPAT_COMPRESSION_BITNR = 3,
QCOW2_INCOMPAT_DIRTY= 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
QCOW2_INCOMPAT_CORRUPT = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
QCOW2_INCOMPAT_DATA_FILE= 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
+QCOW2_INCOMPAT_COMPRESSION = 1 << QCOW2_INCOMPAT_COMPRESSION_BITNR,
QCOW2_INCOMPAT_MASK = QCOW2_INCOMPAT_DIRTY
[PATCH v4 0/5] qcow2: Implement zstd cluster compression method
v4: * the series is rebased on top of 01 "block/qcow2-threads: fix qcow2_decompress" * 01 is just a no-change resend to avoid extra dependencies. Still, it may be merged in separate v3: * remove redundant max compression type value check [Vladimir, Eric] (the switch below checks everything) * prevent compression type changing on "qemu-img amend" [Vladimir] * remove zstd config setting, since it has been added already by "migration" patches [Vladimir] * change the compression type error message [Vladimir] * fix alignment and 80-chars exceeding [Vladimir] v2: * rework compression type setting [Vladimir] * squash iotest changes to the compression type introduction patch [Vladimir, Eric] * fix zstd availability checking in zstd iotest [Vladimir] * remove unnecessry casting [Eric] * remove rudundant checks [Eric] * fix compressed cluster layout in qcow2 spec [Vladimir] * fix wording [Eric, Vladimir] * fix compression type filtering in iotests [Eric] v1: the initial series --- zstd comression method is faster than the only available zlib. The series adds zstd to the methods available for clusters compression. The implementation is done with respect to the recently added compression type additional header to the qcow2 specification. Denis Plotnikov (4): qcow2: introduce compression type feature qcow2: rework the cluster compression routine qcow2: add zstd cluster compression iotests: 287: add qcow2 compression type test Vladimir Sementsov-Ogievskiy (1): block/qcow2-threads: fix qcow2_decompress docs/interop/qcow2.txt | 20 +++ configure| 2 +- qapi/block-core.json | 23 +++- block/qcow2.h| 18 ++- include/block/block_int.h| 1 + block/qcow2-threads.c| 206 --- block/qcow2.c| 108 tests/qemu-iotests/031.out | 14 +-- tests/qemu-iotests/036.out | 4 +- tests/qemu-iotests/049.out | 102 +++ tests/qemu-iotests/060.out | 1 + tests/qemu-iotests/061.out | 34 ++--- tests/qemu-iotests/065 | 28 +++-- tests/qemu-iotests/080 | 2 +- tests/qemu-iotests/144.out | 4 +- tests/qemu-iotests/182.out | 2 +- tests/qemu-iotests/242.out | 5 + tests/qemu-iotests/255.out | 8 +- tests/qemu-iotests/287 | 127 +++ tests/qemu-iotests/287.out | 43 +++ tests/qemu-iotests/common.filter | 3 +- tests/qemu-iotests/group | 1 + 22 files changed, 643 insertions(+), 113 deletions(-) create mode 100755 tests/qemu-iotests/287 create mode 100644 tests/qemu-iotests/287.out -- 2.17.0
[PATCH] configure: change a typo in zstd config
Package manager --exist flag is used instead of --exists. Fix it. Signed-off-by: Denis Plotnikov --- configure | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/configure b/configure index 7b373bc0bb..caa65f5883 100755 --- a/configure +++ b/configure @@ -2464,7 +2464,7 @@ fi # zstd check if test "$zstd" != "no" ; then -if $pkg_config --exist libzstd ; then +if $pkg_config --exists libzstd ; then zstd_cflags="$($pkg_config --cflags libzstd)" zstd_libs="$($pkg_config --libs libzstd)" LIBS="$zstd_libs $LIBS" -- 2.17.0
[PATCH v3 1/4] qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for all the tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feture compression type
backing_file_offset += 56 (8 + 48 -> header_change + fature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 144, 182, 242, 255
Signed-off-by: Denis Plotnikov
---
qapi/block-core.json | 22 ++-
block/qcow2.h| 18 +-
include/block/block_int.h| 1 +
block/qcow2.c| 101 ++
tests/qemu-iotests/031.out | 14 ++---
tests/qemu-iotests/036.out | 4 +-
tests/qemu-iotests/049.out | 102 +++
tests/qemu-iotests/060.out | 1 +
tests/qemu-iotests/061.out | 34 ++-
tests/qemu-iotests/065 | 28 ++---
tests/qemu-iotests/080 | 2 +-
tests/qemu-iotests/144.out | 4 +-
tests/qemu-iotests/182.out | 2 +-
tests/qemu-iotests/242.out | 5 ++
tests/qemu-iotests/255.out | 8 +--
tests/qemu-iotests/common.filter | 3 +-
16 files changed, 253 insertions(+), 96 deletions(-)
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 85e27bb61f..a67eb8bff4 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -78,6 +78,8 @@
#
# @bitmaps: A list of qcow2 bitmap details (since 4.0)
#
+# @compression-type: the image cluster compression method (since 5.0)
+#
# Since: 1.7
##
{ 'struct': 'ImageInfoSpecificQCow2',
@@ -89,7 +91,8 @@
'*corrupt': 'bool',
'refcount-bits': 'int',
'*encrypt': 'ImageInfoSpecificQCow2Encryption',
- '*bitmaps': ['Qcow2BitmapInfo']
+ '*bitmaps': ['Qcow2BitmapInfo'],
+ 'compression-type': 'Qcow2CompressionType'
} }
##
@@ -4392,6 +4395,18 @@
'data': [ 'v2', 'v3' ] }
+##
+# @Qcow2CompressionType:
+#
+# Compression type used in qcow2 image file
+#
+# @zlib: zlib compression, see <http://zlib.net/>
+#
+# Since: 5.0
+##
+{ 'enum': 'Qcow2CompressionType',
+ 'data': [ 'zlib' ] }
+
##
# @BlockdevCreateOptionsQcow2:
#
@@ -4415,6 +4430,8 @@
# allowed values: off, falloc, full, metadata)
# @lazy-refcounts: True if refcounts may be updated lazily (default: off)
# @refcount-bits: Width of reference counts in bits (default: 16)
+# @compression-type: The image cluster compression method
+#(default: zlib, since 5.0)
#
# Since: 2.12
##
@@ -4430,7 +4447,8 @@
'*cluster-size':'size',
'*preallocation': 'PreallocMode',
'*lazy-refcounts': 'bool',
-'*refcount-bits': 'int' } }
+'*refcount-bits': 'int',
+'*compression-type':'Qcow2CompressionType' } }
##
# @BlockdevCreateOptionsQed:
diff --git a/block/qcow2.h b/block/qcow2.h
index 0942126232..485effcb70 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -146,6 +146,12 @@ typedef struct QCowHeader {
uint32_t refcount_order;
uint32_t header_length;
+
+/* Additional fields */
+uint8_t compression_type;
+
+/* header must be a multiple of 8 */
+uint8_t padding[7];
} QEMU_PACKED QCowHeader;
typedef struct QEMU_PACKED QCowSnapshotHeader {
@@ -216,13 +222,16 @@ enum {
QCOW2_INCOMPAT_DIRTY_BITNR = 0,
QCOW2_INCOMPAT_CORRUPT_BITNR= 1,
QCOW2_INCOMPAT_DATA_FILE_BITNR = 2,
+QCOW2_INCOMPAT_COMPRESSION_BITNR = 3,
QCOW2_INCOMPAT_DIRTY= 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
QCOW2_INCOMPAT_CORRUPT = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
QCOW2_INCOMPAT_DATA_FILE= 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
+QCOW2_INCOMPAT_COMPRESSION = 1 << QCOW2_INCOMPAT_COMPRESSION_BITNR,
QCOW2_INCOMPAT_MASK = QCOW2_INCOMPAT_DIRTY
[PATCH v3 0/4] qcow2: Implement zstd cluster compression method
v3: * remove redundant max compression type value check [Vladimir, Eric] (the switch below checks everything) * prevent compression type changing on "qemu-img amend" [Vladimir] * remove zstd config setting, since it has been added already by "migration" patches [Vladimir] * change the compression type error message [Vladimir] * fix alignment and 80-chars exceeding [Vladimir] v2: * rework compression type setting [Vladimir] * squash iotest changes to the compression type introduction patch [Vladimir, Eric] * fix zstd availability checking in zstd iotest [Vladimir] * remove unnecessry casting [Eric] * remove rudundant checks [Eric] * fix compressed cluster layout in qcow2 spec [Vladimir] * fix wording [Eric, Vladimir] * fix compression type filtering in iotests [Eric] v1: the initial series --- zstd comression method is faster than the only available zlib. The series adds zstd to the methods available for clusters compression. The implementation is done with respect to the recently added compression type additional header to the qcow2 specification. Denis Plotnikov (4): qcow2: introduce compression type feature qcow2: rework the cluster compression routine qcow2: add zstd cluster compression iotests: 287: add qcow2 compression type test docs/interop/qcow2.txt | 20 configure| 2 +- qapi/block-core.json | 23 +++- block/qcow2.h| 18 ++- include/block/block_int.h| 1 + block/qcow2-threads.c| 200 --- block/qcow2.c| 108 + tests/qemu-iotests/031.out | 14 +-- tests/qemu-iotests/036.out | 4 +- tests/qemu-iotests/049.out | 102 tests/qemu-iotests/060.out | 1 + tests/qemu-iotests/061.out | 34 +++--- tests/qemu-iotests/065 | 28 +++-- tests/qemu-iotests/080 | 2 +- tests/qemu-iotests/144.out | 4 +- tests/qemu-iotests/182.out | 2 +- tests/qemu-iotests/242.out | 5 + tests/qemu-iotests/255.out | 8 +- tests/qemu-iotests/287 | 127 tests/qemu-iotests/287.out | 43 +++ tests/qemu-iotests/common.filter | 3 +- tests/qemu-iotests/group | 1 + 22 files changed, 639 insertions(+), 111 deletions(-) create mode 100755 tests/qemu-iotests/287 create mode 100644 tests/qemu-iotests/287.out -- 2.17.0
[PATCH v3 4/4] iotests: 287: add qcow2 compression type test
The test checks fulfilling qcow2 requiriements for the compression
type feature and zstd compression type operability.
Signed-off-by: Denis Plotnikov
---
tests/qemu-iotests/287 | 127 +
tests/qemu-iotests/287.out | 43 +
tests/qemu-iotests/group | 1 +
3 files changed, 171 insertions(+)
create mode 100755 tests/qemu-iotests/287
create mode 100644 tests/qemu-iotests/287.out
diff --git a/tests/qemu-iotests/287 b/tests/qemu-iotests/287
new file mode 100755
index 00..39cb665c85
--- /dev/null
+++ b/tests/qemu-iotests/287
@@ -0,0 +1,127 @@
+#!/usr/bin/env bash
+#
+# Test case for an image using zstd compression
+#
+# Copyright (c) 2020 Virtuozzo International GmbH
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
+owner=dplotni...@virtuozzo.com
+
+seq="$(basename $0)"
+echo "QA output created by $seq"
+
+status=1 # failure is the default!
+
+_cleanup()
+{
+ _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# standard environment
+. ./common.rc
+. ./common.filter
+
+# This tests qocw2-specific low-level functionality
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+
+# Check if we can run this test.
+IMGOPTS='compression_type=zstd'
+
+_make_test_img 64M | grep "Invalid parameter 'zstd'" 2>&1 1>/dev/null
+
+ZSTD_SUPPORTED=$?
+
+if (($ZSTD_SUPPORTED==0)); then
+_notrun "ZSTD is disabled"
+fi
+
+# Test: when compression is zlib the incompatible bit is unset
+echo
+echo "=== Testing compression type incompatible bit setting for zlib ==="
+echo
+
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+# Test: when compression differs from zlib the incompatible bit is set
+echo
+echo "=== Testing compression type incompatible bit setting for zstd ==="
+echo
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+# Test: an image can't be openned if compression type is zlib and
+# incompatible feature compression type is set
+echo
+echo "=== Testing zlib with incompatible bit set ==="
+echo
+
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" set-feature-bit incompatible 3
+# to make sure the bit was actually set
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
+if (($?==0)); then
+echo "Error: The image openned successfully. The image must not be openned"
+fi
+
+# Test: an image can't be openned if compression type is NOT zlib and
+# incompatible feature compression type is UNSET
+echo
+echo "=== Testing zstd with incompatible bit unset ==="
+echo
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" set-header incompatible_features 0
+# to make sure the bit was actually unset
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
+if (($?==0)); then
+echo "Error: The image openned successfully. The image must not be openned"
+fi
+# Test: check compression type values
+echo
+echo "=== Testing compression type values ==="
+echo
+# zlib=0
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+od -j104 -N1 -An -vtu1 "$TEST_IMG"
+
+# zstd=1
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+od -j104 -N1 -An -vtu1 "$TEST_IMG"
+
+# Test: using zstd compression, write to and read from an image
+echo
+echo "=== Testing reading and writing with zstd ==="
+echo
+
+CLUSTER_SIZE=65536
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$QEMU_IO -c "write -c -P 0xAC 65536 64k " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -P 0xAC 65536 65536 " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -v 131070 8 " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -v 65534 8" "$TEST_IMG" | _filter_qemu_io
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/287.out b/tests/qemu
[PATCH v3 3/4] qcow2: add zstd cluster compression
zstd significantly reduces cluster compression time.
It provides better compression performance maintaining
the same level of the compression ratio in comparison with
zlib, which, at the moment, is the only compression
method available.
The performance test results:
Test compresses and decompresses qemu qcow2 image with just
installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G
The test was conducted with brd disk to reduce the influence
of disk subsystem to the test results.
The results is given in seconds.
compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2
[zlib|zstd]_compressed.img uncompressed.img
compression decompression
zlib zstd zlib zstd
real 65.5 16.3 (-75 %)1.9 1.6 (-16 %)
user 65.0 15.85.3 2.5
sys 3.30.22.0 2.0
Both ZLIB and ZSTD gave the same compression ratio: 1.57
compressed image size in both cases: 1.4G
Signed-off-by: Denis Plotnikov
---
docs/interop/qcow2.txt | 20 +++
configure | 2 +-
qapi/block-core.json | 3 +-
block/qcow2-threads.c | 123 +
block/qcow2.c | 7 +++
5 files changed, 153 insertions(+), 2 deletions(-)
diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt
index 5597e24474..9048114445 100644
--- a/docs/interop/qcow2.txt
+++ b/docs/interop/qcow2.txt
@@ -208,6 +208,7 @@ version 2.
Available compression type values:
0: zlib <https://www.zlib.net/>
+1: zstd <http://github.com/facebook/zstd>
=== Header padding ===
@@ -575,11 +576,30 @@ Compressed Clusters Descriptor (x = 62 - (cluster_bits -
8)):
Another compressed cluster may map to the tail of the final
sector used by this compressed cluster.
+The layout of the compressed data depends on the
compression
+type used for the image (see compressed cluster layout).
+
If a cluster is unallocated, read requests shall read the data from the backing
file (except if bit 0 in the Standard Cluster Descriptor is set). If there is
no backing file or the backing file is smaller than the image, they shall read
zeros for all parts that are not covered by the backing file.
+=== Compressed Cluster Layout ===
+
+The compressed cluster data has a layout depending on the compression
+type used for the image, as follows:
+
+Compressed data layout for the available compression types:
+data_space_lenght - data chunk length available to store a compressed cluster.
+(for more details see "Compressed Clusters Descriptor")
+x = data_space_length - 1
+
+0: (default) zlib <http://zlib.net/>:
+Byte 0 - x: the compressed data content
+ all the space provided used for compressed data
+1: zstd <http://github.com/facebook/zstd>:
+Byte 0 - 3: the length of compressed data in bytes
+ 4 - x: the compressed data content
== Snapshots ==
diff --git a/configure b/configure
index caa65f5883..b2a0aa241a 100755
--- a/configure
+++ b/configure
@@ -1835,7 +1835,7 @@ disabled with --disable-FEATURE, default is enabled if
available:
lzfse support of lzfse compression library
(for reading lzfse-compressed dmg images)
zstdsupport for zstd compression library
- (for migration compression)
+ (for migration compression and qcow2 cluster compression)
seccomp seccomp support
coroutine-pool coroutine freelist (better performance)
glusterfs GlusterFS backend
diff --git a/qapi/block-core.json b/qapi/block-core.json
index a67eb8bff4..84889fb741 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -4401,11 +4401,12 @@
# Compression type used in qcow2 image file
#
# @zlib: zlib compression, see <http://zlib.net/>
+# @zstd: zstd compression, see <http://github.com/facebook/zstd>
#
# Since: 5.0
##
{ 'enum': 'Qcow2CompressionType',
- 'data': [ 'zlib' ] }
+ 'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
##
# @BlockdevCreateOptionsQcow2:
diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
index 9288a4f852..1020de48dd 100644
--- a/block/qcow2-threads.c
+++ b/block/qcow2-threads.c
@@ -28,6 +28,11 @@
#define ZLIB_CONST
#include
+#ifdef CONFIG_ZSTD
+#include
+#include
+#endif
+
#include "qcow2.h"
#include "block/thread-pool.h"
#include "crypto.h"
@@ -164,6 +169,114 @@ static ssize_t qcow2_zlib_decompress(void *d
[PATCH v3 2/4] qcow2: rework the cluster compression routine
The patch enables processing the image compression type defined
for the image and chooses an appropriate method for image clusters
(de)compression.
Signed-off-by: Denis Plotnikov
---
block/qcow2-threads.c | 77 +++
1 file changed, 63 insertions(+), 14 deletions(-)
diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
index 77bb578cdf..9288a4f852 100644
--- a/block/qcow2-threads.c
+++ b/block/qcow2-threads.c
@@ -74,7 +74,9 @@ typedef struct Qcow2CompressData {
} Qcow2CompressData;
/*
- * qcow2_compress()
+ * qcow2_zlib_compress()
+ *
+ * Compress @src_size bytes of data using zlib compression method
*
* @dest - destination buffer, @dest_size bytes
* @src - source buffer, @src_size bytes
@@ -83,8 +85,8 @@ typedef struct Qcow2CompressData {
* -ENOMEM destination buffer is not enough to store compressed data
* -EIOon any other error
*/
-static ssize_t qcow2_compress(void *dest, size_t dest_size,
- const void *src, size_t src_size)
+static ssize_t qcow2_zlib_compress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
{
ssize_t ret;
z_stream strm;
@@ -119,19 +121,19 @@ static ssize_t qcow2_compress(void *dest, size_t
dest_size,
}
/*
- * qcow2_decompress()
+ * qcow2_zlib_decompress()
*
* Decompress some data (not more than @src_size bytes) to produce exactly
- * @dest_size bytes.
+ * @dest_size bytes using zlib compression method
*
* @dest - destination buffer, @dest_size bytes
* @src - source buffer, @src_size bytes
*
* Returns: 0 on success
- * -1 on fail
+ * -EIO on failure
*/
-static ssize_t qcow2_decompress(void *dest, size_t dest_size,
-const void *src, size_t src_size)
+static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
{
int ret = 0;
z_stream strm;
@@ -144,7 +146,7 @@ static ssize_t qcow2_decompress(void *dest, size_t
dest_size,
ret = inflateInit2(, -12);
if (ret != Z_OK) {
-return -1;
+return -EIO;
}
ret = inflate(, Z_FINISH);
@@ -154,7 +156,7 @@ static ssize_t qcow2_decompress(void *dest, size_t
dest_size,
* @src buffer may be processed partly (because in qcow2 we know size
of
* compressed data with precision of one sector)
*/
-ret = -1;
+ret = -EIO;
}
inflateEnd();
@@ -189,20 +191,67 @@ qcow2_co_do_compress(BlockDriverState *bs, void *dest,
size_t dest_size,
return arg.ret;
}
+/*
+ * qcow2_co_compress()
+ *
+ * Compress @src_size bytes of data using the compression
+ * method defined by the image compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: compressed size on success
+ * a negative error code on failure
+ */
ssize_t coroutine_fn
qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
const void *src, size_t src_size)
{
-return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
-qcow2_compress);
+BDRVQcow2State *s = bs->opaque;
+Qcow2CompressFunc fn;
+
+switch (s->compression_type) {
+case QCOW2_COMPRESSION_TYPE_ZLIB:
+fn = qcow2_zlib_compress;
+break;
+
+default:
+abort();
+}
+
+return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
}
+/*
+ * qcow2_co_decompress()
+ *
+ * Decompress some data (not more than @src_size bytes) to produce exactly
+ * @dest_size bytes using the compression method defined by the image
+ * compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ * a negative error code on failure
+ */
ssize_t coroutine_fn
qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size,
const void *src, size_t src_size)
{
-return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
-qcow2_decompress);
+BDRVQcow2State *s = bs->opaque;
+Qcow2CompressFunc fn;
+
+switch (s->compression_type) {
+case QCOW2_COMPRESSION_TYPE_ZLIB:
+fn = qcow2_zlib_decompress;
+break;
+
+default:
+return -ENOTSUP;
+}
+
+return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
}
--
2.17.0
Re: [PATCH v0 0/2] allow to set 'drive' property on a realized block device
On 02.03.2020 16:38, Kevin Wolf wrote: Am 10.11.2019 um 20:03 hat Denis Plotnikov geschrieben: This allows to replace the file on a block device and is useful to workaround the cases (migration) when the VM image is placed on some shared storage with exclusive file opening model but the image should be open form more than one app. The previous version of approaching the workaround was based on the "blockdev-change-medium" command modification but had some flaws: * semantics: blockdev-change-medium is aimed to be used with removable devices only * interface: it can't accept all possible combination of parameters for the "drive" replacement (creation). More details here: http://patchwork.ozlabs.org/patch/1179329/ The current series suggests another approach: 1. blockdev-add 2. qom-set disk.drive = the blockdev added (this is what the series adds) Are you still planning to send another version? Kevin Not in the near future :) There is an unresolved problem with bitmap-migration is case of block dev replacement. Still don't know how to do it in the proper way. Denis
Re: [PATCH v2 1/4] qcow2: introduce compression type feature
On 02.03.2020 14:24, Vladimir Sementsov-Ogievskiy wrote:
02.03.2020 11:21, Denis Plotnikov wrote:
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression
than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression
type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some
changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for all the tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feture compression type
backing_file_offset += 56 (8 + 48 -> header_change +
fature_table_change)
* add "compression type" for test output matching when it isn't
filtered
affected tests: 049, 060, 061, 065, 144, 182, 242, 255
Signed-off-by: Denis Plotnikov
I'm almost OK with this patch. Some notes below and:
Seems, new option should be handled in qcow2_amend_options among other
unsupported ones (otherwise qcow2_amend_options aborts).
---
qapi/block-core.json | 22 ++-
block/qcow2.h | 18 -
include/block/block_int.h | 1 +
block/qcow2.c | 109 +++
tests/qemu-iotests/031.out | 14 ++--
tests/qemu-iotests/036.out | 4 +-
tests/qemu-iotests/049.out | 102 ++---
tests/qemu-iotests/060.out | 1 +
tests/qemu-iotests/061.out | 34 ++
tests/qemu-iotests/065 | 20 +++---
tests/qemu-iotests/080 | 2 +-
tests/qemu-iotests/144.out | 4 +-
tests/qemu-iotests/182.out | 2 +-
tests/qemu-iotests/242.out | 5 ++
tests/qemu-iotests/255.out | 8 +--
tests/qemu-iotests/common.filter | 3 +-
16 files changed, 255 insertions(+), 94 deletions(-)
[..]
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -146,6 +146,12 @@ typedef struct QCowHeader {
uint32_t refcount_order;
uint32_t header_length;
+
+ /* Additional fields */
+ uint8_t compression_type;
+
+ /* header must be a multiple of 8 */
+ uint8_t padding[7];
} QEMU_PACKED QCowHeader;
typedef struct QEMU_PACKED QCowSnapshotHeader {
@@ -216,13 +222,16 @@ enum {
QCOW2_INCOMPAT_DIRTY_BITNR = 0,
QCOW2_INCOMPAT_CORRUPT_BITNR = 1,
QCOW2_INCOMPAT_DATA_FILE_BITNR = 2,
+ QCOW2_INCOMPAT_COMPRESSION_BITNR= 3,
checkpatch complains. I think, you can just use one space before '='
and don't
care about alignment.
ok
QCOW2_INCOMPAT_DIRTY = 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
QCOW2_INCOMPAT_CORRUPT = 1 <<
QCOW2_INCOMPAT_CORRUPT_BITNR,
QCOW2_INCOMPAT_DATA_FILE = 1 <<
QCOW2_INCOMPAT_DATA_FILE_BITNR,
+ QCOW2_INCOMPAT_COMPRESSION = 1 <<
QCOW2_INCOMPAT_COMPRESSION_BITNR,
QCOW2_INCOMPAT_MASK = QCOW2_INCOMPAT_DIRTY
| QCOW2_INCOMPAT_CORRUPT
- | QCOW2_INCOMPAT_DATA_FILE,
+ | QCOW2_INCOMPAT_DATA_FILE
+ | QCOW2_INCOMPAT_COMPRESSION,
};
/* Compatible feature bits */
@@ -369,6 +378,13 @@ typedef struct BDRVQcow2State {
bool metadata_preallocation_checked;
bool metadata_preallocation;
+ /*
+ * Compression type used for the image. Default: 0 - ZLIB
+ * The image compression type is set on image creation.
+ * The only way to change the compression type is to convert the
image
+ * with the desired compression type set
+ */
+ Qcow2CompressionType compression_type;
} BDRVQcow2State;
typedef struct Qcow2COWRegion {
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 6f9fd5e20e..2c6bb9dc99 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -57,6 +57,7 @@
#define BLOCK_OPT_REFCOUNT_BITS "refcount_bits"
#define BLOCK_OPT_DATA_FILE "data_file"
#define BLOCK_OPT_DATA_FILE_RAW "data_file_raw"
+#define BLOCK_OPT_COMPRESSION_TYPE "compression_type"
#define BLOCK_PROBE_BUF_SIZE
Re: [PATCH v2 0/4] qcow2: Implement zstd cluster compression method
On 02.03.2020 11:51, Vladimir Sementsov-Ogievskiy wrote: Doesn't apply to master, as zstd already exists in ./configure :) (for migration) M, will rebase it. Any other comments? 02.03.2020 11:21, Denis Plotnikov wrote: v2: * rework compression type setting [Vladimir] * squash iotest changes to the compression type introduction patch [Vladimir, Eric] * fix zstd availability checking in zstd iotest [Vladimir] * remove unnecessry casting [Eric] * remove rudundant checks [Eric] * fix compressed cluster layout in qcow2 spec [Vladimir] * fix wording [Eric, Vladimir] * fix compression type filtering in iotests [Eric] v1: the initial series --- zstd comression method is faster than the only available zlib. The series adds zstd to the methods available for clusters compression. The implementation is done with respect to the recently added compression type additional header to the qcow2 specification. Denis Plotnikov (4): qcow2: introduce compression type feature qcow2: rework the cluster compression routine qcow2: add zstd cluster compression iotests: 287: add qcow2 compression type test docs/interop/qcow2.txt | 20 configure | 29 + qapi/block-core.json | 23 +++- block/qcow2.h | 18 ++- include/block/block_int.h | 1 + block/qcow2-threads.c | 197 --- block/qcow2.c | 116 ++ tests/qemu-iotests/031.out | 14 +-- tests/qemu-iotests/036.out | 4 +- tests/qemu-iotests/049.out | 102 tests/qemu-iotests/060.out | 1 + tests/qemu-iotests/061.out | 34 +++--- tests/qemu-iotests/065 | 20 ++-- tests/qemu-iotests/080 | 2 +- tests/qemu-iotests/144.out | 4 +- tests/qemu-iotests/182.out | 2 +- tests/qemu-iotests/242.out | 5 + tests/qemu-iotests/255.out | 8 +- tests/qemu-iotests/287 | 127 tests/qemu-iotests/287.out | 43 +++ tests/qemu-iotests/common.filter | 3 +- tests/qemu-iotests/group | 1 + 22 files changed, 666 insertions(+), 108 deletions(-) create mode 100755 tests/qemu-iotests/287 create mode 100644 tests/qemu-iotests/287.out
[PATCH v2 1/4] qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for all the tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feture compression type
backing_file_offset += 56 (8 + 48 -> header_change + fature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 144, 182, 242, 255
Signed-off-by: Denis Plotnikov
---
qapi/block-core.json | 22 ++-
block/qcow2.h| 18 -
include/block/block_int.h| 1 +
block/qcow2.c| 109 +++
tests/qemu-iotests/031.out | 14 ++--
tests/qemu-iotests/036.out | 4 +-
tests/qemu-iotests/049.out | 102 ++---
tests/qemu-iotests/060.out | 1 +
tests/qemu-iotests/061.out | 34 ++
tests/qemu-iotests/065 | 20 +++---
tests/qemu-iotests/080 | 2 +-
tests/qemu-iotests/144.out | 4 +-
tests/qemu-iotests/182.out | 2 +-
tests/qemu-iotests/242.out | 5 ++
tests/qemu-iotests/255.out | 8 +--
tests/qemu-iotests/common.filter | 3 +-
16 files changed, 255 insertions(+), 94 deletions(-)
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 85e27bb61f..a67eb8bff4 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -78,6 +78,8 @@
#
# @bitmaps: A list of qcow2 bitmap details (since 4.0)
#
+# @compression-type: the image cluster compression method (since 5.0)
+#
# Since: 1.7
##
{ 'struct': 'ImageInfoSpecificQCow2',
@@ -89,7 +91,8 @@
'*corrupt': 'bool',
'refcount-bits': 'int',
'*encrypt': 'ImageInfoSpecificQCow2Encryption',
- '*bitmaps': ['Qcow2BitmapInfo']
+ '*bitmaps': ['Qcow2BitmapInfo'],
+ 'compression-type': 'Qcow2CompressionType'
} }
##
@@ -4392,6 +4395,18 @@
'data': [ 'v2', 'v3' ] }
+##
+# @Qcow2CompressionType:
+#
+# Compression type used in qcow2 image file
+#
+# @zlib: zlib compression, see <http://zlib.net/>
+#
+# Since: 5.0
+##
+{ 'enum': 'Qcow2CompressionType',
+ 'data': [ 'zlib' ] }
+
##
# @BlockdevCreateOptionsQcow2:
#
@@ -4415,6 +4430,8 @@
# allowed values: off, falloc, full, metadata)
# @lazy-refcounts: True if refcounts may be updated lazily (default: off)
# @refcount-bits: Width of reference counts in bits (default: 16)
+# @compression-type: The image cluster compression method
+#(default: zlib, since 5.0)
#
# Since: 2.12
##
@@ -4430,7 +4447,8 @@
'*cluster-size':'size',
'*preallocation': 'PreallocMode',
'*lazy-refcounts': 'bool',
-'*refcount-bits': 'int' } }
+'*refcount-bits': 'int',
+'*compression-type':'Qcow2CompressionType' } }
##
# @BlockdevCreateOptionsQed:
diff --git a/block/qcow2.h b/block/qcow2.h
index 0942126232..6a43495c27 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -146,6 +146,12 @@ typedef struct QCowHeader {
uint32_t refcount_order;
uint32_t header_length;
+
+/* Additional fields */
+uint8_t compression_type;
+
+/* header must be a multiple of 8 */
+uint8_t padding[7];
} QEMU_PACKED QCowHeader;
typedef struct QEMU_PACKED QCowSnapshotHeader {
@@ -216,13 +222,16 @@ enum {
QCOW2_INCOMPAT_DIRTY_BITNR = 0,
QCOW2_INCOMPAT_CORRUPT_BITNR= 1,
QCOW2_INCOMPAT_DATA_FILE_BITNR = 2,
+QCOW2_INCOMPAT_COMPRESSION_BITNR= 3,
QCOW2_INCOMPAT_DIRTY= 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
QCOW2_INCOMPAT_CORRUPT = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
QCOW2_INCOMPAT_DATA_FILE= 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
+QCOW2_INCOMPAT_COMPRESSION = 1 << QCOW2_INCOMPAT_COMPRESSION_BITNR,
QCOW2_INCOMPAT_MASK = QCOW2_INCOMPAT_DIRTY
[PATCH v2 0/4] qcow2: Implement zstd cluster compression method
v2: * rework compression type setting [Vladimir] * squash iotest changes to the compression type introduction patch [Vladimir, Eric] * fix zstd availability checking in zstd iotest [Vladimir] * remove unnecessry casting [Eric] * remove rudundant checks [Eric] * fix compressed cluster layout in qcow2 spec [Vladimir] * fix wording [Eric, Vladimir] * fix compression type filtering in iotests [Eric] v1: the initial series --- zstd comression method is faster than the only available zlib. The series adds zstd to the methods available for clusters compression. The implementation is done with respect to the recently added compression type additional header to the qcow2 specification. Denis Plotnikov (4): qcow2: introduce compression type feature qcow2: rework the cluster compression routine qcow2: add zstd cluster compression iotests: 287: add qcow2 compression type test docs/interop/qcow2.txt | 20 configure| 29 + qapi/block-core.json | 23 +++- block/qcow2.h| 18 ++- include/block/block_int.h| 1 + block/qcow2-threads.c| 197 --- block/qcow2.c| 116 ++ tests/qemu-iotests/031.out | 14 +-- tests/qemu-iotests/036.out | 4 +- tests/qemu-iotests/049.out | 102 tests/qemu-iotests/060.out | 1 + tests/qemu-iotests/061.out | 34 +++--- tests/qemu-iotests/065 | 20 ++-- tests/qemu-iotests/080 | 2 +- tests/qemu-iotests/144.out | 4 +- tests/qemu-iotests/182.out | 2 +- tests/qemu-iotests/242.out | 5 + tests/qemu-iotests/255.out | 8 +- tests/qemu-iotests/287 | 127 tests/qemu-iotests/287.out | 43 +++ tests/qemu-iotests/common.filter | 3 +- tests/qemu-iotests/group | 1 + 22 files changed, 666 insertions(+), 108 deletions(-) create mode 100755 tests/qemu-iotests/287 create mode 100644 tests/qemu-iotests/287.out -- 2.17.0
[PATCH v2 2/4] qcow2: rework the cluster compression routine
The patch enables processing the image compression type defined
for the image and chooses an appropriate method for image clusters
(de)compression.
Signed-off-by: Denis Plotnikov
---
block/qcow2-threads.c | 77 +++
1 file changed, 63 insertions(+), 14 deletions(-)
diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
index 77bb578cdf..9288a4f852 100644
--- a/block/qcow2-threads.c
+++ b/block/qcow2-threads.c
@@ -74,7 +74,9 @@ typedef struct Qcow2CompressData {
} Qcow2CompressData;
/*
- * qcow2_compress()
+ * qcow2_zlib_compress()
+ *
+ * Compress @src_size bytes of data using zlib compression method
*
* @dest - destination buffer, @dest_size bytes
* @src - source buffer, @src_size bytes
@@ -83,8 +85,8 @@ typedef struct Qcow2CompressData {
* -ENOMEM destination buffer is not enough to store compressed data
* -EIOon any other error
*/
-static ssize_t qcow2_compress(void *dest, size_t dest_size,
- const void *src, size_t src_size)
+static ssize_t qcow2_zlib_compress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
{
ssize_t ret;
z_stream strm;
@@ -119,19 +121,19 @@ static ssize_t qcow2_compress(void *dest, size_t
dest_size,
}
/*
- * qcow2_decompress()
+ * qcow2_zlib_decompress()
*
* Decompress some data (not more than @src_size bytes) to produce exactly
- * @dest_size bytes.
+ * @dest_size bytes using zlib compression method
*
* @dest - destination buffer, @dest_size bytes
* @src - source buffer, @src_size bytes
*
* Returns: 0 on success
- * -1 on fail
+ * -EIO on failure
*/
-static ssize_t qcow2_decompress(void *dest, size_t dest_size,
-const void *src, size_t src_size)
+static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
{
int ret = 0;
z_stream strm;
@@ -144,7 +146,7 @@ static ssize_t qcow2_decompress(void *dest, size_t
dest_size,
ret = inflateInit2(, -12);
if (ret != Z_OK) {
-return -1;
+return -EIO;
}
ret = inflate(, Z_FINISH);
@@ -154,7 +156,7 @@ static ssize_t qcow2_decompress(void *dest, size_t
dest_size,
* @src buffer may be processed partly (because in qcow2 we know size
of
* compressed data with precision of one sector)
*/
-ret = -1;
+ret = -EIO;
}
inflateEnd();
@@ -189,20 +191,67 @@ qcow2_co_do_compress(BlockDriverState *bs, void *dest,
size_t dest_size,
return arg.ret;
}
+/*
+ * qcow2_co_compress()
+ *
+ * Compress @src_size bytes of data using the compression
+ * method defined by the image compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: compressed size on success
+ * a negative error code on failure
+ */
ssize_t coroutine_fn
qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
const void *src, size_t src_size)
{
-return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
-qcow2_compress);
+BDRVQcow2State *s = bs->opaque;
+Qcow2CompressFunc fn;
+
+switch (s->compression_type) {
+case QCOW2_COMPRESSION_TYPE_ZLIB:
+fn = qcow2_zlib_compress;
+break;
+
+default:
+abort();
+}
+
+return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
}
+/*
+ * qcow2_co_decompress()
+ *
+ * Decompress some data (not more than @src_size bytes) to produce exactly
+ * @dest_size bytes using the compression method defined by the image
+ * compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ * a negative error code on failure
+ */
ssize_t coroutine_fn
qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size,
const void *src, size_t src_size)
{
-return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
-qcow2_decompress);
+BDRVQcow2State *s = bs->opaque;
+Qcow2CompressFunc fn;
+
+switch (s->compression_type) {
+case QCOW2_COMPRESSION_TYPE_ZLIB:
+fn = qcow2_zlib_decompress;
+break;
+
+default:
+return -ENOTSUP;
+}
+
+return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
}
--
2.17.0
[PATCH v2 4/4] iotests: 287: add qcow2 compression type test
The test checks fulfilling qcow2 requiriements for the compression
type feature and zstd compression type operability.
Signed-off-by: Denis Plotnikov
---
tests/qemu-iotests/287 | 127 +
tests/qemu-iotests/287.out | 43 +
tests/qemu-iotests/group | 1 +
3 files changed, 171 insertions(+)
create mode 100755 tests/qemu-iotests/287
create mode 100644 tests/qemu-iotests/287.out
diff --git a/tests/qemu-iotests/287 b/tests/qemu-iotests/287
new file mode 100755
index 00..39cb665c85
--- /dev/null
+++ b/tests/qemu-iotests/287
@@ -0,0 +1,127 @@
+#!/usr/bin/env bash
+#
+# Test case for an image using zstd compression
+#
+# Copyright (c) 2020 Virtuozzo International GmbH
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
+owner=dplotni...@virtuozzo.com
+
+seq="$(basename $0)"
+echo "QA output created by $seq"
+
+status=1 # failure is the default!
+
+_cleanup()
+{
+ _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# standard environment
+. ./common.rc
+. ./common.filter
+
+# This tests qocw2-specific low-level functionality
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+
+# Check if we can run this test.
+IMGOPTS='compression_type=zstd'
+
+_make_test_img 64M | grep "Invalid parameter 'zstd'" 2>&1 1>/dev/null
+
+ZSTD_SUPPORTED=$?
+
+if (($ZSTD_SUPPORTED==0)); then
+_notrun "ZSTD is disabled"
+fi
+
+# Test: when compression is zlib the incompatible bit is unset
+echo
+echo "=== Testing compression type incompatible bit setting for zlib ==="
+echo
+
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+# Test: when compression differs from zlib the incompatible bit is set
+echo
+echo "=== Testing compression type incompatible bit setting for zstd ==="
+echo
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+# Test: an image can't be openned if compression type is zlib and
+# incompatible feature compression type is set
+echo
+echo "=== Testing zlib with incompatible bit set ==="
+echo
+
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" set-feature-bit incompatible 3
+# to make sure the bit was actually set
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
+if (($?==0)); then
+echo "Error: The image openned successfully. The image must not be openned"
+fi
+
+# Test: an image can't be openned if compression type is NOT zlib and
+# incompatible feature compression type is UNSET
+echo
+echo "=== Testing zstd with incompatible bit unset ==="
+echo
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" set-header incompatible_features 0
+# to make sure the bit was actually unset
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
+if (($?==0)); then
+echo "Error: The image openned successfully. The image must not be openned"
+fi
+# Test: check compression type values
+echo
+echo "=== Testing compression type values ==="
+echo
+# zlib=0
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+od -j104 -N1 -An -vtu1 "$TEST_IMG"
+
+# zstd=1
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+od -j104 -N1 -An -vtu1 "$TEST_IMG"
+
+# Test: using zstd compression, write to and read from an image
+echo
+echo "=== Testing reading and writing with zstd ==="
+echo
+
+CLUSTER_SIZE=65536
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$QEMU_IO -c "write -c -P 0xAC 65536 64k " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -P 0xAC 65536 65536 " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -v 131070 8 " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -v 65534 8" "$TEST_IMG" | _filter_qemu_io
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/287.out b/tests/qemu
[PATCH v2 3/4] qcow2: add zstd cluster compression
zstd significantly reduces cluster compression time.
It provides better compression performance maintaining
the same level of the compression ratio in comparison with
zlib, which, at the moment, is the only compression
method available.
The performance test results:
Test compresses and decompresses qemu qcow2 image with just
installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G
The test was conducted with brd disk to reduce the influence
of disk subsystem to the test results.
The results is given in seconds.
compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2
[zlib|zstd]_compressed.img uncompressed.img
compression decompression
zlib zstd zlib zstd
real 65.5 16.3 (-75 %)1.9 1.6 (-16 %)
user 65.0 15.85.3 2.5
sys 3.30.22.0 2.0
Both ZLIB and ZSTD gave the same compression ratio: 1.57
compressed image size in both cases: 1.4G
Signed-off-by: Denis Plotnikov
---
docs/interop/qcow2.txt | 20 +++
configure | 29 ++
qapi/block-core.json | 3 +-
block/qcow2-threads.c | 120 +
block/qcow2.c | 7 +++
5 files changed, 178 insertions(+), 1 deletion(-)
diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt
index 5597e24474..9048114445 100644
--- a/docs/interop/qcow2.txt
+++ b/docs/interop/qcow2.txt
@@ -208,6 +208,7 @@ version 2.
Available compression type values:
0: zlib <https://www.zlib.net/>
+1: zstd <http://github.com/facebook/zstd>
=== Header padding ===
@@ -575,11 +576,30 @@ Compressed Clusters Descriptor (x = 62 - (cluster_bits -
8)):
Another compressed cluster may map to the tail of the final
sector used by this compressed cluster.
+The layout of the compressed data depends on the
compression
+type used for the image (see compressed cluster layout).
+
If a cluster is unallocated, read requests shall read the data from the backing
file (except if bit 0 in the Standard Cluster Descriptor is set). If there is
no backing file or the backing file is smaller than the image, they shall read
zeros for all parts that are not covered by the backing file.
+=== Compressed Cluster Layout ===
+
+The compressed cluster data has a layout depending on the compression
+type used for the image, as follows:
+
+Compressed data layout for the available compression types:
+data_space_lenght - data chunk length available to store a compressed cluster.
+(for more details see "Compressed Clusters Descriptor")
+x = data_space_length - 1
+
+0: (default) zlib <http://zlib.net/>:
+Byte 0 - x: the compressed data content
+ all the space provided used for compressed data
+1: zstd <http://github.com/facebook/zstd>:
+Byte 0 - 3: the length of compressed data in bytes
+ 4 - x: the compressed data content
== Snapshots ==
diff --git a/configure b/configure
index 48d6f89d57..4690a7ea9f 100755
--- a/configure
+++ b/configure
@@ -444,6 +444,7 @@ opengl_dmabuf="no"
cpuid_h="no"
avx2_opt=""
zlib="yes"
+zstd=""
capstone=""
lzo=""
snappy=""
@@ -1371,6 +1372,10 @@ for opt do
;;
--disable-lzfse) lzfse="no"
;;
+ --enable-zstd) zstd="yes"
+ ;;
+ --disable-zstd) zstd="no"
+ ;;
--enable-guest-agent) guest_agent="yes"
;;
--disable-guest-agent) guest_agent="no"
@@ -1829,6 +1834,7 @@ disabled with --disable-FEATURE, default is enabled if
available:
(for reading bzip2-compressed dmg images)
lzfse support of lzfse compression library
(for reading lzfse-compressed dmg images)
+ zstdsupport of zstd compression library
seccomp seccomp support
coroutine-pool coroutine freelist (better performance)
glusterfs GlusterFS backend
@@ -2453,6 +2459,25 @@ EOF
fi
fi
+#
+# zstd check
+
+if test "$zstd" != "no" ; then
+cat > $TMPC << EOF
+#include
+int main(void) { ZSTD_versionNumber(); return 0; }
+EOF
+if compile_prog "" "-lzstd" ; then
+LIBS="$LIBS -lzstd"
+zstd="yes"
+else
+if test "$zstd" = "yes"; then
+feature_not_found "zstd" "Install libzstd-devel"
+
Re: [PATCH v1 3/8] qcow2: add zstd cluster compression
On 27.02.2020 17:18, Vladimir Sementsov-Ogievskiy wrote:
27.02.2020 17:11, Denis Plotnikov wrote:
On 27.02.2020 12:55, Vladimir Sementsov-Ogievskiy wrote:
27.02.2020 10:29, Denis Plotnikov wrote:
zstd significantly reduces cluster compression time.
It provides better compression performance maintaining
the same level of the compression ratio in comparison with
zlib, which, at the moment, is the only compression
method available.
The performance test results:
Test compresses and decompresses qemu qcow2 image with just
installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G
The test was conducted with brd disk to reduce the influence
of disk subsystem to the test results.
The results is given in seconds.
compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2
[zlib|zstd]_compressed.img uncompressed.img
compression decompression
zlib zstd zlib zstd
real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %)
user 65.0 15.8 5.3 2.5
sys 3.3 0.2 2.0 2.0
Both ZLIB and ZSTD gave the same compression ratio: 1.57
compressed image size in both cases: 1.4G
Signed-off-by: Denis Plotnikov
---
block/qcow2-threads.c | 122
+
block/qcow2.c | 7 +++
configure | 29 ++
docs/interop/qcow2.txt | 18 ++
qapi/block-core.json | 3 +-
5 files changed, 178 insertions(+), 1 deletion(-)
diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
index 1c128e9840..e942c4d7e5 100644
--- a/block/qcow2-threads.c
+++ b/block/qcow2-threads.c
@@ -28,6 +28,11 @@
#define ZLIB_CONST
#include
+#ifdef CONFIG_ZSTD
+#include
+#include
+#endif
+
#include "qcow2.h"
#include "block/thread-pool.h"
#include "crypto.h"
@@ -164,6 +169,113 @@ static ssize_t qcow2_zlib_decompress(void
*dest, size_t dest_size,
return ret;
}
+#ifdef CONFIG_ZSTD
+
+#define ZSTD_LEN_BUF 4
+
+/*
+ * qcow2_zstd_compress()
+ *
+ * Compress @src_size bytes of data using zstd compression method
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: compressed size on success
This doesn't match qcow2_co_compress definition. You should return 0
on success.
does it? I'd rather say it doesn't match to qcow2_co_compress
description in the function comment, which we can change actually,
because qcow2_co_compress is used like:
Oh, yes, you are right. Then we should change the comment.
block/qcow2.c:
static coroutine_fn int
qcow2_co_pwritev_compressed_task(BlockDriverState *bs,
uint64_t offset, uint64_t bytes,
QEMUIOVector *qiov, size_t
qiov_offset)
{
...
out_buf = g_malloc(s->cluster_size);
out_len = qcow2_co_compress(bs, out_buf, s->cluster_size - 1,
buf, s->cluster_size);
if (out_len == -ENOMEM) {
/* could not compress: write normal cluster */
ret = qcow2_co_pwritev_part(bs, offset, bytes, qiov,
qiov_offset, 0);
if (ret < 0) {
goto fail;
}
goto success;
} else if (out_len < 0) {
ret = -EINVAL;
goto fail;
}
qemu_co_mutex_lock(>lock);
ret = qcow2_alloc_compressed_cluster_offset(bs, offset, out_len,
<<<<<<<<<<<<
_offset);
...
}
+ * -ENOMEM destination buffer is not enough to store
compressed data
+ * -EIO on any other error
+ */
+
+static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
+{
+ size_t ret;
+
+ /*
+ * steal ZSTD_LEN_BUF bytes in the very beginng of the buffer
beginning
+ * to store compressed chunk size
+ */
+ char *d_buf = ((char *) dest) + ZSTD_LEN_BUF;
+
+ /*
+ * sanity check that we can store the compressed data length,
+ * and there is some space left for the compressor buffer
+ */
+ if (dest_size <= ZSTD_LEN_BUF) {
+ return -ENOMEM;
+ }
+
+ dest_size -= ZSTD_LEN_BUF;
+
+ ret = ZSTD_compress(d_buf, dest_size, src, src_size, 5);
+
+ if (ZSTD_isError(ret)) {
+ if (ZSTD_getErrorCode(ret) == ZSTD_error_dstSize_tooSmall) {
+ return -ENOMEM;
+ } else {
+ return -EIO;
+ }
+ }
+
+ /* paraniod sanity check that we can store the commpressed
size */
+ if (ret > UINT_MAX) {
+ return -ENOMEM;
+ }
I'd use UINT32_MAX, possibly even more paranoid)
ok
+
+ /* store the compressed chunk
Re: [PATCH v1 4/8] iotests: filter out compression_type
On 27.02.2020 17:03, Eric Blake wrote: On 2/27/20 1:29 AM, Denis Plotnikov wrote: After adding compression type feature to qcow2 format, qemu framework commands reporting the image settingd, e.g. "qemu-img create", started settings reporting the compression type for the image which breaks the iotests output matching. To fix it, add compression_type=zlib to the list of filtered image parameters. Signed-off-by: Denis Plotnikov --- tests/qemu-iotests/common.filter | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) This should be squashed in to the patch that caused the breakage (3/8, if I'm right). diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter index 3f8ee3e5f7..c6962d199c 100644 --- a/tests/qemu-iotests/common.filter +++ b/tests/qemu-iotests/common.filter @@ -152,7 +152,8 @@ _filter_img_create() -e "s# refcount_bits=[0-9]\\+##g" \ -e "s# key-secret=[a-zA-Z0-9]\\+##g" \ -e "s# iter-time=[0-9]\\+##g" \ - -e "s# force_size=\\(on\\|off\\)##g" + -e "s# force_size=\\(on\\|off\\)##g" \ + -e "s# compression_type=zlib##g" Do you really want to hard-code just zlib, or should this be more generic as compression_type=[a-zA-Z0-9]\\+ as done on other lines like key-secret? When I did this I meant additional implicit check that the default compression type is zlib. But non of the other items in the filter don't do it. So I'll change it to be consistent. Thanks! Denis
Re: [PATCH v1 3/8] qcow2: add zstd cluster compression
On 27.02.2020 17:01, Eric Blake wrote:
On 2/27/20 1:29 AM, Denis Plotnikov wrote:
zstd significantly reduces cluster compression time.
It provides better compression performance maintaining
the same level of the compression ratio in comparison with
zlib, which, at the moment, is the only compression
method available.
+static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
+{
+ size_t ret;
+
+ /*
+ * steal ZSTD_LEN_BUF bytes in the very beginng of the buffer
beginning
+ * to store compressed chunk size
+ */
+ char *d_buf = ((char *) dest) + ZSTD_LEN_BUF;
+
+ /*
+ * sanity check that we can store the compressed data length,
+ * and there is some space left for the compressor buffer
+ */
+ if (dest_size <= ZSTD_LEN_BUF) {
+ return -ENOMEM;
+ }
+
+ dest_size -= ZSTD_LEN_BUF;
+
+ ret = ZSTD_compress(d_buf, dest_size, src, src_size, 5);
+
+ if (ZSTD_isError(ret)) {
+ if (ZSTD_getErrorCode(ret) == ZSTD_error_dstSize_tooSmall) {
+ return -ENOMEM;
+ } else {
+ return -EIO;
+ }
+ }
+
+ /* paraniod sanity check that we can store the commpressed size */
paranoid, compressed
+ if (ret > UINT_MAX) {
+ return -ENOMEM;
+ }
This is pointless. Better is to ensure that we actually compressed
data (the pigeonhole principle states that there are some inputs that
MUST result in inflation, in order for most other inputs to result in
compression). But that check was satisfied by checking for
ZSTD_error_dstSize_tooSmall, which is what happens for one of those
uncompressible inputs. Namely, zstd will never return a result larger
than dest_size, and since dest_size is smaller than UINT_MAX on entry,
this check is pointless. But if you want something, I'd be okay with:
assert(ret <= dest_size).
Taking into account that this is "just in case" and I'm trying to
protect the first 4 bytes of the buffer from the overflow and
I can't imagine the situation when we deal with cluster sizes greater
than UINT32_MAX but the input size is size_t which can be > UINT32_MAX
on 64bit archs.
I'd rather stick with
if (ret > UINT32_MAX) {
return -ENOMEM;
}
as Vladimir suggested.
I'm not sure that the assert is good here, since it stops the system
operating and this isn't potential error but a limitation
Does it work for you?
Denis
+++ b/docs/interop/qcow2.txt
@@ -208,6 +208,7 @@ version 2.
Available compression type values:
0: zlib <https://www.zlib.net/>
+ 1: zstd <http://github.com/facebook/zstd>
=== Header padding ===
@@ -575,11 +576,28 @@ Compressed Clusters Descriptor (x = 62 -
(cluster_bits - 8)):
Another compressed cluster may map to the tail
of the final
sector used by this compressed cluster.
+ The layout of the compressed data depends on
the compression
+ type used for the image (see compressed cluster
layout).
+
If a cluster is unallocated, read requests shall read the data from
the backing
file (except if bit 0 in the Standard Cluster Descriptor is set).
If there is
no backing file or the backing file is smaller than the image, they
shall read
zeros for all parts that are not covered by the backing file.
+=== Compressed Cluster Layout ===
+
+The compressed cluster data has a layout depending on the compression
+type used for the image, as follows:
+
+Compressed data layout for the available compression types:
+(x = data_space_length - 1)
+
+ 0: (default) zlib <http://zlib.net/>:
+ Byte 0 - x: the compressed data content
+ all the space provided used for
compressed data
+ 1: zstd <http://github.com/facebook/zstd>:
+ Byte 0 - 3: the length of compressed data in bytes
+ 4 - x: the compressed data content
== Snapshots ==
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 873fbef3b5..4b6e576c44 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -4401,11 +4401,12 @@
# Compression type used in qcow2 image file
#
# @zlib: zlib compression, see <http://zlib.net/>
+# @zstd: zstd compression, see <http://github.com/facebook/zstd>
#
# Since: 5.0
##
{ 'enum': 'Qcow2CompressionType',
- 'data': [ 'zlib' ] }
+ 'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' }
] }
The spec and UI changes are okay.
Re: [PATCH v1 1/8] qcow2: introduce compression type feature
On 27.02.2020 16:48, Eric Blake wrote:
On 2/27/20 1:29 AM, Denis Plotnikov wrote:
The patch adds some preparation parts for incompatible compression type
feature to Qcow2 that indicates which allow to use different compression
to qcow2, allowing the use of different
methods for image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression
than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression
type
are backward compatible with older qemu versions.
Signed-off-by: Denis Plotnikov
---
block/qcow2.c | 105 ++
block/qcow2.h | 31 ---
include/block/block_int.h | 1 +
qapi/block-core.json | 22 +++-
4 files changed, 150 insertions(+), 9 deletions(-)
diff --git a/block/qcow2.c b/block/qcow2.c
index 3c754f616b..2ccb2cabd1 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1242,6 +1242,50 @@ static int
qcow2_update_options(BlockDriverState *bs, QDict *options,
return ret;
}
+static int validate_compression_type(BDRVQcow2State *s, Error **errp)
+{
+ /*
+ * Sanity check
+ * according to qcow2 spec, the compression type is 1-byte field
+ * but in BDRVQcow2State the compression_type is enum sizeof(int)
+ * so, the max compression_type value is 255.
+ */
+ if (s->compression_type > 0xff) {
+ error_setg(errp, "qcow2: compression type value is too big");
+ return -EINVAL;
+ }
Hmm - I think it may be worth a tweak to qcow2.txt to call out:
104: compression_type
105 - 111: padding, must be 0
or else call out:
104-111: compression type
and just blindly use all 8 bytes for the value even though really only
1 or two values will ever be defined. Of course, that moves the byte
in question from 104 to 111, thanks to our big endian encoding, but as
this series is the first one installing a non-zero value in those 8
bytes, and as we just finished documenting that the header length must
be a multiple of 8, there is no real impact - we can make such tweaks
up until the 5.0 release.
+
+ switch (s->compression_type) {
+ case QCOW2_COMPRESSION_TYPE_ZLIB:
+ break;
+
+ default:
+ error_setg(errp, "qcow2: unknown compression type: %u",
+ s->compression_type);
+ return -ENOTSUP;
+ }
Having two checks feels redundant, compared to just letting the
default catch all unrecognized values in that field.
Looks like it is.
+
+ /*
+ * if the compression type differs from QCOW2_COMPRESSION_TYPE_ZLIB
+ * the incompatible feature flag must be set
+ */
+ if (s->compression_type == QCOW2_COMPRESSION_TYPE_ZLIB) {
+ if (s->incompatible_features &
QCOW2_INCOMPAT_COMPRESSION_TYPE) {
+ error_setg(errp, "qcow2: Compression type incompatible
feature "
+ "bit must not be set");
+ return -EINVAL;
+ }
+ } else {
+ if (!(s->incompatible_features &
QCOW2_INCOMPAT_COMPRESSION_TYPE)) {
+ error_setg(errp, "qcow2: Compression type incompatible
feature "
+ "bit must be set");
+ return -EINVAL;
+ }
+ }
Matches what we documented in the spec.
+
+ return 0;
+}
+
/* Called with s->lock held. */
static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict
*options,
int flags, Error **errp)
@@ -1357,6 +1401,26 @@ static int coroutine_fn
qcow2_do_open(BlockDriverState *bs, QDict *options,
s->compatible_features = header.compatible_features;
s->autoclear_features = header.autoclear_features;
+ /*
+ * Handle compression type
+ * Older qcow2 images don't contain the compression type header.
+ * Distinguish them by the header length and use
+ * the only valid (default) compression type in that case
+ */
+ if (header.header_length > offsetof(QCowHeader,
compression_type)) {
+ /*
+ * don't deal with endians since compression_type is 1 byte
long
+ */
+ s->compression_type = header.compression_type;
Changes if you go with my suggestion of just making the
compression_type field occupy 8 bytes in the qcow2 header. (And if
you want to keep it 1 byte, I still think the spec should call out
explicit padding bytes).
+ } else {
+ s->compression_type = QCOW2_COMPRESSION_TYPE_ZLIB;
+ }
+
+ ret = validate_compression_type(s, errp);
+ if (ret) {
+ goto fail;
Re: [PATCH v1 8/8] iotests: 287: add qcow2 compression type test
On 27.02.2020 13:29, Vladimir Sementsov-Ogievskiy wrote:
27.02.2020 10:29, Denis Plotnikov wrote:
The test checks fulfilling qcow2 requiriements for the compression
type feature and zstd compression type operability.
Signed-off-by: Denis Plotnikov
---
tests/qemu-iotests/287 | 123 +
tests/qemu-iotests/287.out | 41 +
tests/qemu-iotests/group | 1 +
3 files changed, 165 insertions(+)
create mode 100755 tests/qemu-iotests/287
create mode 100644 tests/qemu-iotests/287.out
diff --git a/tests/qemu-iotests/287 b/tests/qemu-iotests/287
new file mode 100755
index 00..41b916f690
--- /dev/null
+++ b/tests/qemu-iotests/287
@@ -0,0 +1,123 @@
+#!/usr/bin/env bash
+#
+# Test case for an image using zstd compression
+#
+# Copyright (c) 2020 Virtuozzo International GmbH
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
+owner=dplotni...@virtuozzo.com
+
+seq="$(basename $0)"
+echo "QA output created by $seq"
+
+status=1 # failure is the default!
+
+_cleanup()
+{
+ _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# standard environment
+. ./common.rc
+. ./common.filter
+
+# This tests qocw2-specific low-level functionality
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+
+P=`echo "$QEMU_PROG" | sed "s/qemu-system-x86_64//"`
+
+grep "CONFIG_ZSTD=y" "$P"../config-host.mak >/dev/null
+RES=$?
Hmm. This will not work for other architectures and for
out of tree builds. Also, it checks config but not current
binary (they may be out of sync, or even unrelated).
Probably better try to create image with zstd compression type
and handle expected error.
What if the error is "unable to create an image with zstd", although it
has to be?
I think the best way is to ask qemu binary whether it supports zstd, but
it doesn't available by now (should be?)
Is there any other way to make sure that the std compression test has to
be executed?
+if (($RES)); then
+ _notrun "ZSTD is disabled in the current configuration"
+fi
+
+# Test: when compression is zlib the incompatible is unset
+echo
+echo "=== Testing compression type incompatible bit setting for zlib
==="
+echo
+
+_make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+# Test: when compression differs from zlib the incompatible bit is set
+echo
+echo "=== Testing compression type incompatible bit setting for zstd
==="
+echo
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+# Test: an image can't be openned if compression type is zlib and
opened
+# incompatible feature compression type is set
+echo
+echo "=== Testing zlib with incompatible bit set ==="
+echo
+
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" set-feature-bit incompatible 3
+# to make sure the bit was actually set
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
+if (($?==0)); then
+ echo "Error: The image openned successfully. The image must not
be openned"
+fi
may be better to instead keep error output and just check it..
I add the explicit message to reduce the investigating time of what
happened and what should it be.
If it isn't that important I'd rather leave it as is.
+
+# Test: an image can't be openned if compression type is NOT zlib and
+# incompatible feature compression type is UNSET
+echo
+echo "=== Testing zstd with incompatible bit unset ==="
+echo
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" set-header incompatible_features 0
+# to make sure the bit was actually unset
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
+if (($?==0)); then
+ echo "Error: The image openned successfully. The image must not
be openned"
+fi
+# Test: check compression type values
+echo
+echo "=== Testing compression type values ==="
+echo
+# zlib=0
+IMGOPTS='compression_type=zlib' _make
Re: [PATCH v1 6/8] iotests: add "compression type" for test output matching
On 27.02.2020 13:09, Vladimir Sementsov-Ogievskiy wrote: 27.02.2020 13:04, Vladimir Sementsov-Ogievskiy wrote: 27.02.2020 10:29, Denis Plotnikov wrote: Affected tests: 049, 060, 061, 065, 144, 182, 242, 255 After adding the compression type feature for qcow2, the compression type is reported on image quering. Add the corresponding values of the "compression type" for the tests' output matching. And this and the following patch. Ideally, patch should not break any iotests. This means that all iotest updates should be merged to the patch which changes their output. Probably, you can split behavior-changing patch, to reduce iotest-updates per patch, but anyway, big patch with a lot of iotests updates is better than patch which breaks iotests. Or we can try to reduce behavior changes in case of zlib: - keep header small in case of zlib, not adding zero field - don't add feature table entry, if compress type is zlib - don't report compression type on quering, if it is zlib - then, all iotests output will be saved. And, then, if we need, we can change these theree points one-by-one, updating iotests outputs. But I doubt that we really need it, compatible behavior seems good enough. I think I would put some efforts in arranging the iotest patches so they don't break any tests after applying with the cost of creating a gigantic patch with the compression type implementation and iotest fixes Denis Signed-off-by: Denis Plotnikov --- tests/qemu-iotests/049.out | 102 ++--- tests/qemu-iotests/060.out | 1 + tests/qemu-iotests/061.out | 6 +++ tests/qemu-iotests/065 | 20 +--- tests/qemu-iotests/144.out | 4 +- tests/qemu-iotests/182.out | 2 +- tests/qemu-iotests/242.out | 5 ++ tests/qemu-iotests/255.out | 8 +-- 8 files changed, 82 insertions(+), 66 deletions(-) diff --git a/tests/qemu-iotests/049.out b/tests/qemu-iotests/049.out index affa55b341..a5cfba1756 100644 --- a/tests/qemu-iotests/049.out +++ b/tests/qemu-iotests/049.out @@ -4,90 +4,90 @@ QA output created by 049 == 1. Traditional size parameter == qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024b -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1k -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1K -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1G -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1T -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024.0 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024.0b -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5k -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cl
Re: [PATCH v1 3/8] qcow2: add zstd cluster compression
On 27.02.2020 12:55, Vladimir Sementsov-Ogievskiy wrote:
27.02.2020 10:29, Denis Plotnikov wrote:
zstd significantly reduces cluster compression time.
It provides better compression performance maintaining
the same level of the compression ratio in comparison with
zlib, which, at the moment, is the only compression
method available.
The performance test results:
Test compresses and decompresses qemu qcow2 image with just
installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G
The test was conducted with brd disk to reduce the influence
of disk subsystem to the test results.
The results is given in seconds.
compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2
[zlib|zstd]_compressed.img uncompressed.img
compression decompression
zlib zstd zlib zstd
real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %)
user 65.0 15.8 5.3 2.5
sys 3.3 0.2 2.0 2.0
Both ZLIB and ZSTD gave the same compression ratio: 1.57
compressed image size in both cases: 1.4G
Signed-off-by: Denis Plotnikov
---
block/qcow2-threads.c | 122 +
block/qcow2.c | 7 +++
configure | 29 ++
docs/interop/qcow2.txt | 18 ++
qapi/block-core.json | 3 +-
5 files changed, 178 insertions(+), 1 deletion(-)
diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
index 1c128e9840..e942c4d7e5 100644
--- a/block/qcow2-threads.c
+++ b/block/qcow2-threads.c
@@ -28,6 +28,11 @@
#define ZLIB_CONST
#include
+#ifdef CONFIG_ZSTD
+#include
+#include
+#endif
+
#include "qcow2.h"
#include "block/thread-pool.h"
#include "crypto.h"
@@ -164,6 +169,113 @@ static ssize_t qcow2_zlib_decompress(void
*dest, size_t dest_size,
return ret;
}
+#ifdef CONFIG_ZSTD
+
+#define ZSTD_LEN_BUF 4
+
+/*
+ * qcow2_zstd_compress()
+ *
+ * Compress @src_size bytes of data using zstd compression method
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: compressed size on success
This doesn't match qcow2_co_compress definition. You should return 0
on success.
does it? I'd rather say it doesn't match to qcow2_co_compress
description in the function comment, which we can change actually,
because qcow2_co_compress is used like:
block/qcow2.c:
static coroutine_fn int
qcow2_co_pwritev_compressed_task(BlockDriverState *bs,
uint64_t offset, uint64_t bytes,
QEMUIOVector *qiov, size_t qiov_offset)
{
...
out_buf = g_malloc(s->cluster_size);
out_len = qcow2_co_compress(bs, out_buf, s->cluster_size - 1,
buf, s->cluster_size);
if (out_len == -ENOMEM) {
/* could not compress: write normal cluster */
ret = qcow2_co_pwritev_part(bs, offset, bytes, qiov,
qiov_offset, 0);
if (ret < 0) {
goto fail;
}
goto success;
} else if (out_len < 0) {
ret = -EINVAL;
goto fail;
}
qemu_co_mutex_lock(>lock);
ret = qcow2_alloc_compressed_cluster_offset(bs, offset, out_len,
<<<<<<<<<<<<
_offset);
...
}
+ * -ENOMEM destination buffer is not enough to store
compressed data
+ * -EIO on any other error
+ */
+
+static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
+{
+ size_t ret;
+
+ /*
+ * steal ZSTD_LEN_BUF bytes in the very beginng of the buffer
beginning
+ * to store compressed chunk size
+ */
+ char *d_buf = ((char *) dest) + ZSTD_LEN_BUF;
+
+ /*
+ * sanity check that we can store the compressed data length,
+ * and there is some space left for the compressor buffer
+ */
+ if (dest_size <= ZSTD_LEN_BUF) {
+ return -ENOMEM;
+ }
+
+ dest_size -= ZSTD_LEN_BUF;
+
+ ret = ZSTD_compress(d_buf, dest_size, src, src_size, 5);
+
+ if (ZSTD_isError(ret)) {
+ if (ZSTD_getErrorCode(ret) == ZSTD_error_dstSize_tooSmall) {
+ return -ENOMEM;
+ } else {
+ return -EIO;
+ }
+ }
+
+ /* paraniod sanity check that we can store the commpressed size */
+ if (ret > UINT_MAX) {
+ return -ENOMEM;
+ }
I'd use UINT32_MAX, possibly even more paranoid)
ok
+
+ /* store the compressed chunk size in the very beginning of the
buffer */
+ stl_be_p(dest, ret);
+
+ return ret + ZSTD_LEN_BUF;
return 0;
+}
+
+/*
+ * qcow2_zstd_decompress()
+ *
+ * Decompress some data (not m
[PATCH v1 6/8] iotests: add "compression type" for test output matching
Affected tests: 049, 060, 061, 065, 144, 182, 242, 255 After adding the compression type feature for qcow2, the compression type is reported on image quering. Add the corresponding values of the "compression type" for the tests' output matching. Signed-off-by: Denis Plotnikov --- tests/qemu-iotests/049.out | 102 ++--- tests/qemu-iotests/060.out | 1 + tests/qemu-iotests/061.out | 6 +++ tests/qemu-iotests/065 | 20 +--- tests/qemu-iotests/144.out | 4 +- tests/qemu-iotests/182.out | 2 +- tests/qemu-iotests/242.out | 5 ++ tests/qemu-iotests/255.out | 8 +-- 8 files changed, 82 insertions(+), 66 deletions(-) diff --git a/tests/qemu-iotests/049.out b/tests/qemu-iotests/049.out index affa55b341..a5cfba1756 100644 --- a/tests/qemu-iotests/049.out +++ b/tests/qemu-iotests/049.out @@ -4,90 +4,90 @@ QA output created by 049 == 1. Traditional size parameter == qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024b -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1k -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1K -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1G -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1T -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024.0 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024.0b -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5k -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5K -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5G -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5T -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_ref
[PATCH v1 5/8] iotests: fix header size, feature table size and backing file offset
Affected tests: 031, 036, 061 Because of adding the compression type feature, some size values in the qcow2 v3 header are changed: header_size +=8: 1 byte compression type 7 bytes padding feature_table += 48: incompatible feture compression type backing_file_offset += 56 (8 + 48 -> header_change + fature_table_change) Change the values for the test output comparison accordingly. Signed-off-by: Denis Plotnikov --- tests/qemu-iotests/031.out | 14 +++--- tests/qemu-iotests/036.out | 4 ++-- tests/qemu-iotests/061.out | 28 ++-- 3 files changed, 23 insertions(+), 23 deletions(-) diff --git a/tests/qemu-iotests/031.out b/tests/qemu-iotests/031.out index d535e407bc..ed51afe9ce 100644 --- a/tests/qemu-iotests/031.out +++ b/tests/qemu-iotests/031.out @@ -113,11 +113,11 @@ incompatible_features [] compatible_features [] autoclear_features[] refcount_order4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length192 +length240 data Header extension: @@ -146,11 +146,11 @@ incompatible_features [] compatible_features [] autoclear_features[] refcount_order4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length192 +length240 data Header extension: @@ -164,7 +164,7 @@ No errors were found on the image. magic 0x514649fb version 3 -backing_file_offset 0x178 +backing_file_offset 0x1b0 backing_file_size 0x17 cluster_bits 16 size 67108864 @@ -179,7 +179,7 @@ incompatible_features [] compatible_features [] autoclear_features[] refcount_order4 -header_length 104 +header_length 112 Header extension: magic 0xe2792aca @@ -188,7 +188,7 @@ data 'host_device' Header extension: magic 0x6803f857 -length192 +length240 data Header extension: diff --git a/tests/qemu-iotests/036.out b/tests/qemu-iotests/036.out index 0b52b934e1..fb509f6357 100644 --- a/tests/qemu-iotests/036.out +++ b/tests/qemu-iotests/036.out @@ -26,7 +26,7 @@ compatible_features [] autoclear_features[63] Header extension: magic 0x6803f857 -length192 +length240 data @@ -38,7 +38,7 @@ compatible_features [] autoclear_features[] Header extension: magic 0x6803f857 -length192 +length240 data *** done diff --git a/tests/qemu-iotests/061.out b/tests/qemu-iotests/061.out index 8b3091a412..cea7fedfdc 100644 --- a/tests/qemu-iotests/061.out +++ b/tests/qemu-iotests/061.out @@ -22,11 +22,11 @@ incompatible_features [] compatible_features [0] autoclear_features[] refcount_order4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length192 +length240 data magic 0x514649fb @@ -80,11 +80,11 @@ incompatible_features [] compatible_features [0] autoclear_features[] refcount_order4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length192 +length240 data magic 0x514649fb @@ -136,11 +136,11 @@ incompatible_features [0] compatible_features [0] autoclear_features[] refcount_order4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length192 +length240 data ERROR cluster 5 refcount=0 reference=1 @@ -191,11 +191,11 @@ incompatible_features [] compatible_features [42] autoclear_features[42] refcount_order4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length192 +length240 data magic 0x514649fb @@ -260,11 +260,11 @@ incompatible_features [] compatible_features [0] autoclear_features[] refcount_order4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -len
[PATCH v1 2/8] qcow2: rework the cluster compression routine
The patch enables processing the image compression type defined
for the image and chooses an appropriate method for image clusters
(de)compression.
Signed-off-by: Denis Plotnikov
---
block/qcow2-threads.c | 77 +++
1 file changed, 63 insertions(+), 14 deletions(-)
diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
index 77bb578cdf..1c128e9840 100644
--- a/block/qcow2-threads.c
+++ b/block/qcow2-threads.c
@@ -74,7 +74,9 @@ typedef struct Qcow2CompressData {
} Qcow2CompressData;
/*
- * qcow2_compress()
+ * qcow2_zlib_compress()
+ *
+ * Compress @src_size bytes of data using zlib compression method
*
* @dest - destination buffer, @dest_size bytes
* @src - source buffer, @src_size bytes
@@ -83,8 +85,8 @@ typedef struct Qcow2CompressData {
* -ENOMEM destination buffer is not enough to store compressed data
* -EIOon any other error
*/
-static ssize_t qcow2_compress(void *dest, size_t dest_size,
- const void *src, size_t src_size)
+static ssize_t qcow2_zlib_compress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
{
ssize_t ret;
z_stream strm;
@@ -119,19 +121,19 @@ static ssize_t qcow2_compress(void *dest, size_t
dest_size,
}
/*
- * qcow2_decompress()
+ * qcow2_zlib_decompress()
*
* Decompress some data (not more than @src_size bytes) to produce exactly
- * @dest_size bytes.
+ * @dest_size bytes using zlib compression method
*
* @dest - destination buffer, @dest_size bytes
* @src - source buffer, @src_size bytes
*
* Returns: 0 on success
- * -1 on fail
+ * -EIO on failure
*/
-static ssize_t qcow2_decompress(void *dest, size_t dest_size,
-const void *src, size_t src_size)
+static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
{
int ret = 0;
z_stream strm;
@@ -144,7 +146,7 @@ static ssize_t qcow2_decompress(void *dest, size_t
dest_size,
ret = inflateInit2(, -12);
if (ret != Z_OK) {
-return -1;
+return -EIO;
}
ret = inflate(, Z_FINISH);
@@ -154,7 +156,7 @@ static ssize_t qcow2_decompress(void *dest, size_t
dest_size,
* @src buffer may be processed partly (because in qcow2 we know size
of
* compressed data with precision of one sector)
*/
-ret = -1;
+ret = -EIO;
}
inflateEnd();
@@ -189,20 +191,67 @@ qcow2_co_do_compress(BlockDriverState *bs, void *dest,
size_t dest_size,
return arg.ret;
}
+/*
+ * qcow2_co_compress()
+ *
+ * Compress @src_size bytes of data using the compression
+ * method defined by the image compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ * a negative error code on failure
+ */
ssize_t coroutine_fn
qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
const void *src, size_t src_size)
{
-return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
-qcow2_compress);
+BDRVQcow2State *s = bs->opaque;
+Qcow2CompressFunc fn;
+
+switch (s->compression_type) {
+case QCOW2_COMPRESSION_TYPE_ZLIB:
+fn = qcow2_zlib_compress;
+break;
+
+default:
+return -ENOTSUP;
+}
+
+return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
}
+/*
+ * qcow2_co_decompress()
+ *
+ * Decompress some data (not more than @src_size bytes) to produce exactly
+ * @dest_size bytes using the compression method defined by the image
+ * compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ * a negative error code on failure
+ */
ssize_t coroutine_fn
qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size,
const void *src, size_t src_size)
{
-return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
-qcow2_decompress);
+BDRVQcow2State *s = bs->opaque;
+Qcow2CompressFunc fn;
+
+switch (s->compression_type) {
+case QCOW2_COMPRESSION_TYPE_ZLIB:
+fn = qcow2_zlib_decompress;
+break;
+
+default:
+return -ENOTSUP;
+}
+
+return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
}
--
2.17.0
[PATCH v1 4/8] iotests: filter out compression_type
After adding compression type feature to qcow2 format, qemu framework commands reporting the image settingd, e.g. "qemu-img create", started reporting the compression type for the image which breaks the iotests output matching. To fix it, add compression_type=zlib to the list of filtered image parameters. Signed-off-by: Denis Plotnikov --- tests/qemu-iotests/common.filter | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter index 3f8ee3e5f7..c6962d199c 100644 --- a/tests/qemu-iotests/common.filter +++ b/tests/qemu-iotests/common.filter @@ -152,7 +152,8 @@ _filter_img_create() -e "s# refcount_bits=[0-9]\\+##g" \ -e "s# key-secret=[a-zA-Z0-9]\\+##g" \ -e "s# iter-time=[0-9]\\+##g" \ --e "s# force_size=\\(on\\|off\\)##g" +-e "s# force_size=\\(on\\|off\\)##g" \ +-e "s# compression_type=zlib##g" } _filter_img_info() -- 2.17.0
[PATCH v1 0/8] qcow2: Implement zstd cluster compression method
zstd comression method is faster than the only available zlib. The series adds zstd to the methods available for clusters compression. The implementation is done with respect to the recently added compression type additional header to the qcow2 specification. Denis Plotnikov (8): qcow2: introduce compression type feature qcow2: rework the cluster compression routine qcow2: add zstd cluster compression iotests: filter out compression_type iotests: fix header size, feature table size and backing file offset iotests: add "compression type" for test output matching iotests: 080: update header size value because of adding compression type iotests: 287: add qcow2 compression type test block/qcow2-threads.c| 199 --- block/qcow2.c| 112 + block/qcow2.h| 31 +++-- configure| 29 + docs/interop/qcow2.txt | 18 +++ include/block/block_int.h| 1 + qapi/block-core.json | 23 +++- tests/qemu-iotests/031.out | 14 +-- tests/qemu-iotests/036.out | 4 +- tests/qemu-iotests/049.out | 102 tests/qemu-iotests/060.out | 1 + tests/qemu-iotests/061.out | 34 +++--- tests/qemu-iotests/065 | 20 ++-- tests/qemu-iotests/080 | 2 +- tests/qemu-iotests/144.out | 4 +- tests/qemu-iotests/182.out | 2 +- tests/qemu-iotests/242.out | 5 + tests/qemu-iotests/255.out | 8 +- tests/qemu-iotests/287 | 123 +++ tests/qemu-iotests/287.out | 41 +++ tests/qemu-iotests/common.filter | 3 +- tests/qemu-iotests/group | 1 + 22 files changed, 663 insertions(+), 114 deletions(-) create mode 100755 tests/qemu-iotests/287 create mode 100644 tests/qemu-iotests/287.out -- 2.17.0
[PATCH v1 3/8] qcow2: add zstd cluster compression
zstd significantly reduces cluster compression time.
It provides better compression performance maintaining
the same level of the compression ratio in comparison with
zlib, which, at the moment, is the only compression
method available.
The performance test results:
Test compresses and decompresses qemu qcow2 image with just
installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G
The test was conducted with brd disk to reduce the influence
of disk subsystem to the test results.
The results is given in seconds.
compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2
[zlib|zstd]_compressed.img uncompressed.img
compression decompression
zlib zstd zlib zstd
real 65.5 16.3 (-75 %)1.9 1.6 (-16 %)
user 65.0 15.85.3 2.5
sys 3.30.22.0 2.0
Both ZLIB and ZSTD gave the same compression ratio: 1.57
compressed image size in both cases: 1.4G
Signed-off-by: Denis Plotnikov
---
block/qcow2-threads.c | 122 +
block/qcow2.c | 7 +++
configure | 29 ++
docs/interop/qcow2.txt | 18 ++
qapi/block-core.json | 3 +-
5 files changed, 178 insertions(+), 1 deletion(-)
diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
index 1c128e9840..e942c4d7e5 100644
--- a/block/qcow2-threads.c
+++ b/block/qcow2-threads.c
@@ -28,6 +28,11 @@
#define ZLIB_CONST
#include
+#ifdef CONFIG_ZSTD
+#include
+#include
+#endif
+
#include "qcow2.h"
#include "block/thread-pool.h"
#include "crypto.h"
@@ -164,6 +169,113 @@ static ssize_t qcow2_zlib_decompress(void *dest, size_t
dest_size,
return ret;
}
+#ifdef CONFIG_ZSTD
+
+#define ZSTD_LEN_BUF 4
+
+/*
+ * qcow2_zstd_compress()
+ *
+ * Compress @src_size bytes of data using zstd compression method
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: compressed size on success
+ * -ENOMEM destination buffer is not enough to store compressed data
+ * -EIOon any other error
+ */
+
+static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
+{
+size_t ret;
+
+/*
+ * steal ZSTD_LEN_BUF bytes in the very beginng of the buffer
+ * to store compressed chunk size
+ */
+char *d_buf = ((char *) dest) + ZSTD_LEN_BUF;
+
+/*
+ * sanity check that we can store the compressed data length,
+ * and there is some space left for the compressor buffer
+ */
+if (dest_size <= ZSTD_LEN_BUF) {
+return -ENOMEM;
+}
+
+dest_size -= ZSTD_LEN_BUF;
+
+ret = ZSTD_compress(d_buf, dest_size, src, src_size, 5);
+
+if (ZSTD_isError(ret)) {
+if (ZSTD_getErrorCode(ret) == ZSTD_error_dstSize_tooSmall) {
+return -ENOMEM;
+} else {
+return -EIO;
+}
+}
+
+/* paraniod sanity check that we can store the commpressed size */
+if (ret > UINT_MAX) {
+return -ENOMEM;
+}
+
+/* store the compressed chunk size in the very beginning of the buffer */
+stl_be_p(dest, ret);
+
+return ret + ZSTD_LEN_BUF;
+}
+
+/*
+ * qcow2_zstd_decompress()
+ *
+ * Decompress some data (not more than @src_size bytes) to produce exactly
+ * @dest_size bytes using zstd compression method
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ * -EIO on any error
+ */
+
+static ssize_t qcow2_zstd_decompress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
+{
+/*
+ * zstd decompress wants to know the exact length of the data.
+ * For that purpose, on compression, the length is stored in
+ * the very beginning of the compressed buffer
+ */
+size_t s_size;
+const char *s_buf = ((const char *) src) + ZSTD_LEN_BUF;
+
+/*
+ * sanity check that we can read 4 byte the content length and
+ * and there is some content to decompress
+ */
+if (src_size <= ZSTD_LEN_BUF) {
+return -EIO;
+}
+
+s_size = ldl_be_p(src);
+
+/* sanity check that the buffer is big enough to read the content from */
+if (src_size - ZSTD_LEN_BUF < s_size) {
+return -EIO;
+}
+
+if (ZSTD_isError(
+ZSTD_decompress(dest, dest_size, s_buf, s_size))) {
+return -EIO;
+}
+
+return 0;
+}
+#endif
+
static int qcow2_compress_pool_func(void *opaque)
{
Qcow2CompressData *data = opaque;
@@ -215,6 +327,11 @@ qcow2_co_compress(BlockDriverState *bs,
[PATCH v1 1/8] qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to Qcow2 that indicates which allow to use different compression
methods for image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Signed-off-by: Denis Plotnikov
---
block/qcow2.c | 105 ++
block/qcow2.h | 31 ---
include/block/block_int.h | 1 +
qapi/block-core.json | 22 +++-
4 files changed, 150 insertions(+), 9 deletions(-)
diff --git a/block/qcow2.c b/block/qcow2.c
index 3c754f616b..2ccb2cabd1 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1242,6 +1242,50 @@ static int qcow2_update_options(BlockDriverState *bs,
QDict *options,
return ret;
}
+static int validate_compression_type(BDRVQcow2State *s, Error **errp)
+{
+/*
+ * Sanity check
+ * according to qcow2 spec, the compression type is 1-byte field
+ * but in BDRVQcow2State the compression_type is enum sizeof(int)
+ * so, the max compression_type value is 255.
+ */
+if (s->compression_type > 0xff) {
+error_setg(errp, "qcow2: compression type value is too big");
+return -EINVAL;
+}
+
+switch (s->compression_type) {
+case QCOW2_COMPRESSION_TYPE_ZLIB:
+break;
+
+default:
+error_setg(errp, "qcow2: unknown compression type: %u",
+ s->compression_type);
+return -ENOTSUP;
+}
+
+/*
+ * if the compression type differs from QCOW2_COMPRESSION_TYPE_ZLIB
+ * the incompatible feature flag must be set
+ */
+if (s->compression_type == QCOW2_COMPRESSION_TYPE_ZLIB) {
+if (s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION_TYPE) {
+error_setg(errp, "qcow2: Compression type incompatible feature "
+ "bit must not be set");
+return -EINVAL;
+}
+} else {
+if (!(s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION_TYPE)) {
+error_setg(errp, "qcow2: Compression type incompatible feature "
+ "bit must be set");
+return -EINVAL;
+}
+}
+
+return 0;
+}
+
/* Called with s->lock held. */
static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
int flags, Error **errp)
@@ -1357,6 +1401,26 @@ static int coroutine_fn qcow2_do_open(BlockDriverState
*bs, QDict *options,
s->compatible_features = header.compatible_features;
s->autoclear_features = header.autoclear_features;
+/*
+ * Handle compression type
+ * Older qcow2 images don't contain the compression type header.
+ * Distinguish them by the header length and use
+ * the only valid (default) compression type in that case
+ */
+if (header.header_length > offsetof(QCowHeader, compression_type)) {
+/*
+ * don't deal with endians since compression_type is 1 byte long
+ */
+s->compression_type = header.compression_type;
+} else {
+s->compression_type = QCOW2_COMPRESSION_TYPE_ZLIB;
+}
+
+ret = validate_compression_type(s, errp);
+if (ret) {
+goto fail;
+}
+
if (s->incompatible_features & ~QCOW2_INCOMPAT_MASK) {
void *feature_table = NULL;
qcow2_read_extensions(bs, header.header_length, ext_end,
@@ -2720,6 +2784,12 @@ int qcow2_update_header(BlockDriverState *bs)
total_size = bs->total_sectors * BDRV_SECTOR_SIZE;
refcount_table_clusters = s->refcount_table_size >> (s->cluster_bits - 3);
+ret = validate_compression_type(s, NULL);
+
+if (ret) {
+goto fail;
+}
+
*header = (QCowHeader) {
/* Version 2 fields */
.magic = cpu_to_be32(QCOW_MAGIC),
@@ -2742,6 +2812,7 @@ int qcow2_update_header(BlockDriverState *bs)
.autoclear_features = cpu_to_be64(s->autoclear_features),
.refcount_order = cpu_to_be32(s->refcount_order),
.header_length = cpu_to_be32(header_length),
+.compression_type = (uint8_t) s->compression_type,
};
/* For older versions, write a shorter header */
@@ -2839,6 +2910,11 @@ int qcow2_update_header(BlockDriverState *bs)
.bit = QCOW2_COMPAT_LAZY_REFCOUNTS_BITNR,
[PATCH v1 7/8] iotests: 080: update header size value because of adding compression type
Signed-off-by: Denis Plotnikov --- tests/qemu-iotests/080 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/qemu-iotests/080 b/tests/qemu-iotests/080 index a3d13c414e..7588c63b6c 100755 --- a/tests/qemu-iotests/080 +++ b/tests/qemu-iotests/080 @@ -45,7 +45,7 @@ _supported_os Linux # - This is generally a test for compat=1.1 images _unsupported_imgopts 'refcount_bits=1[^0-9]' data_file 'compat=0.10' -header_size=104 +header_size=112 offset_backing_file_offset=8 offset_backing_file_size=16 -- 2.17.0
[PATCH v1 8/8] iotests: 287: add qcow2 compression type test
The test checks fulfilling qcow2 requiriements for the compression
type feature and zstd compression type operability.
Signed-off-by: Denis Plotnikov
---
tests/qemu-iotests/287 | 123 +
tests/qemu-iotests/287.out | 41 +
tests/qemu-iotests/group | 1 +
3 files changed, 165 insertions(+)
create mode 100755 tests/qemu-iotests/287
create mode 100644 tests/qemu-iotests/287.out
diff --git a/tests/qemu-iotests/287 b/tests/qemu-iotests/287
new file mode 100755
index 00..41b916f690
--- /dev/null
+++ b/tests/qemu-iotests/287
@@ -0,0 +1,123 @@
+#!/usr/bin/env bash
+#
+# Test case for an image using zstd compression
+#
+# Copyright (c) 2020 Virtuozzo International GmbH
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
+owner=dplotni...@virtuozzo.com
+
+seq="$(basename $0)"
+echo "QA output created by $seq"
+
+status=1 # failure is the default!
+
+_cleanup()
+{
+ _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# standard environment
+. ./common.rc
+. ./common.filter
+
+# This tests qocw2-specific low-level functionality
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+
+P=`echo "$QEMU_PROG" | sed "s/qemu-system-x86_64//"`
+
+grep "CONFIG_ZSTD=y" "$P"../config-host.mak >/dev/null
+RES=$?
+if (($RES)); then
+_notrun "ZSTD is disabled in the current configuration"
+fi
+
+# Test: when compression is zlib the incompatible is unset
+echo
+echo "=== Testing compression type incompatible bit setting for zlib ==="
+echo
+
+_make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+# Test: when compression differs from zlib the incompatible bit is set
+echo
+echo "=== Testing compression type incompatible bit setting for zstd ==="
+echo
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+# Test: an image can't be openned if compression type is zlib and
+# incompatible feature compression type is set
+echo
+echo "=== Testing zlib with incompatible bit set ==="
+echo
+
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" set-feature-bit incompatible 3
+# to make sure the bit was actually set
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
+if (($?==0)); then
+echo "Error: The image openned successfully. The image must not be openned"
+fi
+
+# Test: an image can't be openned if compression type is NOT zlib and
+# incompatible feature compression type is UNSET
+echo
+echo "=== Testing zstd with incompatible bit unset ==="
+echo
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" set-header incompatible_features 0
+# to make sure the bit was actually unset
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
+if (($?==0)); then
+echo "Error: The image openned successfully. The image must not be openned"
+fi
+# Test: check compression type values
+echo
+echo "=== Testing compression type values ==="
+echo
+# zlib=0
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+od -j104 -N1 -An -vtu1 "$TEST_IMG"
+
+# zstd=1
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+od -j104 -N1 -An -vtu1 "$TEST_IMG"
+
+# Test: using zstd compression, write to and read from an image
+echo
+echo "=== Testing reading and writing with zstd ==="
+echo
+
+CLUSTER_SIZE=65536
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$QEMU_IO -c "write -c 0 64k " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -v 0 10 " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -v 65530 8" "$TEST_IMG" | _filter_qemu_io
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/287.out b/tests/qemu-iotests/287.out
new file mode 100644
index 00..4218254ce0
--- /dev/null
+++ b/tests/qemu-iotests/287.out
@@ -0,0 +1,41 @@
+QA output creat
Re: [PATCH v2] virtio: increase virtuqueue size for virtio-scsi and virtio-blk
On 18.02.2020 16:59, Denis Plotnikov wrote: On 18.02.2020 16:53, Stefan Hajnoczi wrote: On Thu, Feb 13, 2020 at 05:59:27PM +0300, Denis Plotnikov wrote: v1: * seg_max default value changing removed --- The goal is to reduce the amount of requests issued by a guest on 1M reads/writes. This rises the performance up to 4% on that kind of disk access pattern. The maximum chunk size to be used for the guest disk accessing is limited with seg_max parameter, which represents the max amount of pices in the scatter-geather list in one guest disk request. Since seg_max is virqueue_size dependent, increasing the virtqueue size increases seg_max, which, in turn, increases the maximum size of data to be read/write from a guest disk. More details in the original problem statment: https://lists.gnu.org/archive/html/qemu-devel/2017-12/msg03721.html Suggested-by: Denis V. Lunev Signed-off-by: Denis Plotnikov --- hw/block/virtio-blk.c | 2 +- hw/core/machine.c | 2 ++ hw/scsi/virtio-scsi.c | 2 +- 3 files changed, 4 insertions(+), 2 deletions(-) I fixed up the "virtuqueue" typo in the commit message and the mis-formatted commit description (git-am(1) stops including lines after the first "---"). Actually, I sent the corrected version v3 of the patch last week. But it seems it got lost among that gigantic patch flow in the mailing list :) Thanks for applying! Denis Thanks, applied to my block tree: https://github.com/stefanha/qemu/commits/block Stefan I'm going to send the test checking the virtqueue-sizes for machine types a little bit later. Denis
Re: [PATCH v2] virtio: increase virtuqueue size for virtio-scsi and virtio-blk
On 18.02.2020 16:53, Stefan Hajnoczi wrote: On Thu, Feb 13, 2020 at 05:59:27PM +0300, Denis Plotnikov wrote: v1: * seg_max default value changing removed --- The goal is to reduce the amount of requests issued by a guest on 1M reads/writes. This rises the performance up to 4% on that kind of disk access pattern. The maximum chunk size to be used for the guest disk accessing is limited with seg_max parameter, which represents the max amount of pices in the scatter-geather list in one guest disk request. Since seg_max is virqueue_size dependent, increasing the virtqueue size increases seg_max, which, in turn, increases the maximum size of data to be read/write from a guest disk. More details in the original problem statment: https://lists.gnu.org/archive/html/qemu-devel/2017-12/msg03721.html Suggested-by: Denis V. Lunev Signed-off-by: Denis Plotnikov --- hw/block/virtio-blk.c | 2 +- hw/core/machine.c | 2 ++ hw/scsi/virtio-scsi.c | 2 +- 3 files changed, 4 insertions(+), 2 deletions(-) I fixed up the "virtuqueue" typo in the commit message and the mis-formatted commit description (git-am(1) stops including lines after the first "---"). Actually, I sent the corrected version v3 of the patch last week. But it seems it got lost among that gigantic patch flow in the mailing list :) Thanks for applying! Denis Thanks, applied to my block tree: https://github.com/stefanha/qemu/commits/block Stefan
[PATCH v3] virtio: increase virtqueue size for virtio-scsi and virtio-blk
The goal is to reduce the amount of requests issued by a guest on
1M reads/writes. This rises the performance up to 4% on that kind of
disk access pattern.
The maximum chunk size to be used for the guest disk accessing is
limited with seg_max parameter, which represents the max amount of
pices in the scatter-geather list in one guest disk request.
Since seg_max is virqueue_size dependent, increasing the virtqueue
size increases seg_max, which, in turn, increases the maximum size
of data to be read/write from a guest disk.
More details in the original problem statment:
https://lists.gnu.org/archive/html/qemu-devel/2017-12/msg03721.html
Suggested-by: Denis V. Lunev
Signed-off-by: Denis Plotnikov
---
v3:
* typos fixed
v2:
* seg_max default value changing removed
---
hw/block/virtio-blk.c | 2 +-
hw/core/machine.c | 2 ++
hw/scsi/virtio-scsi.c | 2 +-
3 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
index 09f46ed85f..142863a3b2 100644
--- a/hw/block/virtio-blk.c
+++ b/hw/block/virtio-blk.c
@@ -1272,7 +1272,7 @@ static Property virtio_blk_properties[] = {
DEFINE_PROP_BIT("request-merging", VirtIOBlock, conf.request_merging, 0,
true),
DEFINE_PROP_UINT16("num-queues", VirtIOBlock, conf.num_queues, 1),
-DEFINE_PROP_UINT16("queue-size", VirtIOBlock, conf.queue_size, 128),
+DEFINE_PROP_UINT16("queue-size", VirtIOBlock, conf.queue_size, 256),
DEFINE_PROP_BOOL("seg-max-adjust", VirtIOBlock, conf.seg_max_adjust, true),
DEFINE_PROP_LINK("iothread", VirtIOBlock, conf.iothread, TYPE_IOTHREAD,
IOThread *),
diff --git a/hw/core/machine.c b/hw/core/machine.c
index 2501b540ec..3427d6cf4c 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -28,6 +28,8 @@
#include "hw/mem/nvdimm.h"
GlobalProperty hw_compat_4_2[] = {
+{ "virtio-blk-device", "queue-size", "128"},
+{ "virtio-scsi-device", "virtqueue_size", "128"},
{ "virtio-blk-device", "x-enable-wce-if-config-wce", "off" },
{ "virtio-blk-device", "seg-max-adjust", "off"},
{ "virtio-scsi-device", "seg_max_adjust", "off"},
diff --git a/hw/scsi/virtio-scsi.c b/hw/scsi/virtio-scsi.c
index 3b61563609..472bbd233b 100644
--- a/hw/scsi/virtio-scsi.c
+++ b/hw/scsi/virtio-scsi.c
@@ -965,7 +965,7 @@ static void virtio_scsi_device_unrealize(DeviceState *dev,
Error **errp)
static Property virtio_scsi_properties[] = {
DEFINE_PROP_UINT32("num_queues", VirtIOSCSI, parent_obj.conf.num_queues,
1),
DEFINE_PROP_UINT32("virtqueue_size", VirtIOSCSI,
- parent_obj.conf.virtqueue_size, 128),
+ parent_obj.conf.virtqueue_size, 256),
DEFINE_PROP_BOOL("seg_max_adjust", VirtIOSCSI,
parent_obj.conf.seg_max_adjust, true),
DEFINE_PROP_UINT32("max_sectors", VirtIOSCSI, parent_obj.conf.max_sectors,
--
2.17.0
[PATCH v2] virtio: increase virtuqueue size for virtio-scsi and virtio-blk
v1:
* seg_max default value changing removed
---
The goal is to reduce the amount of requests issued by a guest on
1M reads/writes. This rises the performance up to 4% on that kind of
disk access pattern.
The maximum chunk size to be used for the guest disk accessing is
limited with seg_max parameter, which represents the max amount of
pices in the scatter-geather list in one guest disk request.
Since seg_max is virqueue_size dependent, increasing the virtqueue
size increases seg_max, which, in turn, increases the maximum size
of data to be read/write from a guest disk.
More details in the original problem statment:
https://lists.gnu.org/archive/html/qemu-devel/2017-12/msg03721.html
Suggested-by: Denis V. Lunev
Signed-off-by: Denis Plotnikov
---
hw/block/virtio-blk.c | 2 +-
hw/core/machine.c | 2 ++
hw/scsi/virtio-scsi.c | 2 +-
3 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
index 09f46ed85f..142863a3b2 100644
--- a/hw/block/virtio-blk.c
+++ b/hw/block/virtio-blk.c
@@ -1272,7 +1272,7 @@ static Property virtio_blk_properties[] = {
DEFINE_PROP_BIT("request-merging", VirtIOBlock, conf.request_merging, 0,
true),
DEFINE_PROP_UINT16("num-queues", VirtIOBlock, conf.num_queues, 1),
-DEFINE_PROP_UINT16("queue-size", VirtIOBlock, conf.queue_size, 128),
+DEFINE_PROP_UINT16("queue-size", VirtIOBlock, conf.queue_size, 256),
DEFINE_PROP_BOOL("seg-max-adjust", VirtIOBlock, conf.seg_max_adjust, true),
DEFINE_PROP_LINK("iothread", VirtIOBlock, conf.iothread, TYPE_IOTHREAD,
IOThread *),
diff --git a/hw/core/machine.c b/hw/core/machine.c
index 2501b540ec..3427d6cf4c 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -28,6 +28,8 @@
#include "hw/mem/nvdimm.h"
GlobalProperty hw_compat_4_2[] = {
+{ "virtio-blk-device", "queue-size", "128"},
+{ "virtio-scsi-device", "virtqueue_size", "128"},
{ "virtio-blk-device", "x-enable-wce-if-config-wce", "off" },
{ "virtio-blk-device", "seg-max-adjust", "off"},
{ "virtio-scsi-device", "seg_max_adjust", "off"},
diff --git a/hw/scsi/virtio-scsi.c b/hw/scsi/virtio-scsi.c
index 3b61563609..472bbd233b 100644
--- a/hw/scsi/virtio-scsi.c
+++ b/hw/scsi/virtio-scsi.c
@@ -965,7 +965,7 @@ static void virtio_scsi_device_unrealize(DeviceState *dev,
Error **errp)
static Property virtio_scsi_properties[] = {
DEFINE_PROP_UINT32("num_queues", VirtIOSCSI, parent_obj.conf.num_queues,
1),
DEFINE_PROP_UINT32("virtqueue_size", VirtIOSCSI,
- parent_obj.conf.virtqueue_size, 128),
+ parent_obj.conf.virtqueue_size, 256),
DEFINE_PROP_BOOL("seg_max_adjust", VirtIOSCSI,
parent_obj.conf.seg_max_adjust, true),
DEFINE_PROP_UINT32("max_sectors", VirtIOSCSI, parent_obj.conf.max_sectors,
--
2.17.0
Re: [PATCH v2] virtio: increase virtuqueue size for virtio-scsi and virtio-blk
On 13.02.2020 14:45, Stefan Hajnoczi wrote: On Thu, Feb 13, 2020 at 12:28:25PM +0300, Denis Plotnikov wrote: On 13.02.2020 12:08, Stefan Hajnoczi wrote: On Thu, Feb 13, 2020 at 11:08:35AM +0300, Denis Plotnikov wrote: On 12.02.2020 18:43, Stefan Hajnoczi wrote: On Tue, Feb 11, 2020 at 05:14:14PM +0300, Denis Plotnikov wrote: The goal is to reduce the amount of requests issued by a guest on 1M reads/writes. This rises the performance up to 4% on that kind of disk access pattern. The maximum chunk size to be used for the guest disk accessing is limited with seg_max parameter, which represents the max amount of pices in the scatter-geather list in one guest disk request. Since seg_max is virqueue_size dependent, increasing the virtqueue size increases seg_max, which, in turn, increases the maximum size of data to be read/write from a guest disk. More details in the original problem statment: https://lists.gnu.org/archive/html/qemu-devel/2017-12/msg03721.html Suggested-by: Denis V. Lunev Signed-off-by: Denis Plotnikov --- hw/block/virtio-blk.c | 4 ++-- hw/core/machine.c | 2 ++ hw/scsi/virtio-scsi.c | 4 ++-- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c index 09f46ed85f..6df3a7a6df 100644 --- a/hw/block/virtio-blk.c +++ b/hw/block/virtio-blk.c @@ -914,7 +914,7 @@ static void virtio_blk_update_config(VirtIODevice *vdev, uint8_t *config) memset(, 0, sizeof(blkcfg)); virtio_stq_p(vdev, , capacity); virtio_stl_p(vdev, _max, - s->conf.seg_max_adjust ? s->conf.queue_size - 2 : 128 - 2); + s->conf.seg_max_adjust ? s->conf.queue_size - 2 : 256 - 2); This value must not change on older machine types. Yes, that's true, but .. So does this patch need to turn seg-max-adjust *on* in hw_compat_4_2 so that old machine types get 126 instead of 254? If we set seg-max-adjust "on" in older machine types, the setups using them and having queue_sizes set , for example, 1024 will also set seg_max to 1024 - 2 which isn't the expected behavior: older mt didn't change seg_max in that case and stuck with 128 - 2. So, should we, instead, leave the default 128 - 2, for seg_max? Argh! Good point :-). How about a seg_max_default property that is initialized to 254 for modern machines and 126 to old machines? Hmm, but we'll achieve the same but with more code changes, don't we? 254 is because the queue-size is 256. We gonna leave 128-2 for older machine types just for not breaking anything. All other seg_max adjustment is provided by seg_max_adjust which is "on" by default in modern machine types. to summarize: modern mt defaults: seg_max_adjust = on queue_size = 256 => default seg_max = 254 => changing queue-size will change seg_max = queue_size - 2 old mt defaults: seg_max_adjust = off queue_size = 128 => default seg_max = 126 => changing queue-size won't change seg_max, it's always = 126 like it was before You're right! The only strange case is a modern machine type with seg_max_adjust=off, where queue_size will be 256 but seg_max will be 126. But no user would want to disable seg_max_adjust, so it's okay. I agree with you that the line of code can remain unchanged: /* * Only old machine types use seg_max_adjust=off and there the default * value of queue_size is 128. */ virtio_stl_p(vdev, _max, s->conf.seg_max_adjust ? s->conf.queue_size - 2 : 128 - 2); Stefan Ok, I'll resend the patch sortly Thanks! Denis
Re: [PATCH v2] virtio: increase virtuqueue size for virtio-scsi and virtio-blk
On 13.02.2020 12:08, Stefan Hajnoczi wrote: On Thu, Feb 13, 2020 at 11:08:35AM +0300, Denis Plotnikov wrote: On 12.02.2020 18:43, Stefan Hajnoczi wrote: On Tue, Feb 11, 2020 at 05:14:14PM +0300, Denis Plotnikov wrote: The goal is to reduce the amount of requests issued by a guest on 1M reads/writes. This rises the performance up to 4% on that kind of disk access pattern. The maximum chunk size to be used for the guest disk accessing is limited with seg_max parameter, which represents the max amount of pices in the scatter-geather list in one guest disk request. Since seg_max is virqueue_size dependent, increasing the virtqueue size increases seg_max, which, in turn, increases the maximum size of data to be read/write from a guest disk. More details in the original problem statment: https://lists.gnu.org/archive/html/qemu-devel/2017-12/msg03721.html Suggested-by: Denis V. Lunev Signed-off-by: Denis Plotnikov --- hw/block/virtio-blk.c | 4 ++-- hw/core/machine.c | 2 ++ hw/scsi/virtio-scsi.c | 4 ++-- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c index 09f46ed85f..6df3a7a6df 100644 --- a/hw/block/virtio-blk.c +++ b/hw/block/virtio-blk.c @@ -914,7 +914,7 @@ static void virtio_blk_update_config(VirtIODevice *vdev, uint8_t *config) memset(, 0, sizeof(blkcfg)); virtio_stq_p(vdev, , capacity); virtio_stl_p(vdev, _max, - s->conf.seg_max_adjust ? s->conf.queue_size - 2 : 128 - 2); + s->conf.seg_max_adjust ? s->conf.queue_size - 2 : 256 - 2); This value must not change on older machine types. Yes, that's true, but .. So does this patch need to turn seg-max-adjust *on* in hw_compat_4_2 so that old machine types get 126 instead of 254? If we set seg-max-adjust "on" in older machine types, the setups using them and having queue_sizes set , for example, 1024 will also set seg_max to 1024 - 2 which isn't the expected behavior: older mt didn't change seg_max in that case and stuck with 128 - 2. So, should we, instead, leave the default 128 - 2, for seg_max? Argh! Good point :-). How about a seg_max_default property that is initialized to 254 for modern machines and 126 to old machines? Hmm, but we'll achieve the same but with more code changes, don't we? 254 is because the queue-size is 256. We gonna leave 128-2 for older machine types just for not breaking anything. All other seg_max adjustment is provided by seg_max_adjust which is "on" by default in modern machine types. to summarize: modern mt defaults: seg_max_adjust = on queue_size = 256 => default seg_max = 254 => changing queue-size will change seg_max = queue_size - 2 old mt defaults: seg_max_adjust = off queue_size = 128 => default seg_max = 126 => changing queue-size won't change seg_max, it's always = 126 like it was before Denis Stefan
Re: [PATCH v2] virtio: increase virtuqueue size for virtio-scsi and virtio-blk
On 12.02.2020 18:43, Stefan Hajnoczi wrote:
On Tue, Feb 11, 2020 at 05:14:14PM +0300, Denis Plotnikov wrote:
The goal is to reduce the amount of requests issued by a guest on
1M reads/writes. This rises the performance up to 4% on that kind of
disk access pattern.
The maximum chunk size to be used for the guest disk accessing is
limited with seg_max parameter, which represents the max amount of
pices in the scatter-geather list in one guest disk request.
Since seg_max is virqueue_size dependent, increasing the virtqueue
size increases seg_max, which, in turn, increases the maximum size
of data to be read/write from a guest disk.
More details in the original problem statment:
https://lists.gnu.org/archive/html/qemu-devel/2017-12/msg03721.html
Suggested-by: Denis V. Lunev
Signed-off-by: Denis Plotnikov
---
hw/block/virtio-blk.c | 4 ++--
hw/core/machine.c | 2 ++
hw/scsi/virtio-scsi.c | 4 ++--
3 files changed, 6 insertions(+), 4 deletions(-)
diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
index 09f46ed85f..6df3a7a6df 100644
--- a/hw/block/virtio-blk.c
+++ b/hw/block/virtio-blk.c
@@ -914,7 +914,7 @@ static void virtio_blk_update_config(VirtIODevice *vdev,
uint8_t *config)
memset(, 0, sizeof(blkcfg));
virtio_stq_p(vdev, , capacity);
virtio_stl_p(vdev, _max,
- s->conf.seg_max_adjust ? s->conf.queue_size - 2 : 128 - 2);
+ s->conf.seg_max_adjust ? s->conf.queue_size - 2 : 256 - 2);
This value must not change on older machine types.
Yes, that's true, but ..
So does this patch
need to turn seg-max-adjust *on* in hw_compat_4_2 so that old machine
types get 126 instead of 254?
If we set seg-max-adjust "on" in older machine types, the setups using
them and having queue_sizes set , for example, 1024 will also set
seg_max to 1024 - 2 which isn't the expected behavior: older mt didn't
change seg_max in that case and stuck with 128 - 2.
So, should we, instead, leave the default 128 - 2, for seg_max?
Denis
virtio_stw_p(vdev, , conf->cyls);
virtio_stl_p(vdev, _size, blk_size);
virtio_stw_p(vdev, _io_size, conf->min_io_size / blk_size);
@@ -1272,7 +1272,7 @@ static Property virtio_blk_properties[] = {
DEFINE_PROP_BIT("request-merging", VirtIOBlock, conf.request_merging, 0,
true),
DEFINE_PROP_UINT16("num-queues", VirtIOBlock, conf.num_queues, 1),
-DEFINE_PROP_UINT16("queue-size", VirtIOBlock, conf.queue_size, 128),
+DEFINE_PROP_UINT16("queue-size", VirtIOBlock, conf.queue_size, 256),
DEFINE_PROP_BOOL("seg-max-adjust", VirtIOBlock, conf.seg_max_adjust,
true),
DEFINE_PROP_LINK("iothread", VirtIOBlock, conf.iothread, TYPE_IOTHREAD,
IOThread *),
diff --git a/hw/core/machine.c b/hw/core/machine.c
index 2501b540ec..3427d6cf4c 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -28,6 +28,8 @@
#include "hw/mem/nvdimm.h"
GlobalProperty hw_compat_4_2[] = {
+{ "virtio-blk-device", "queue-size", "128"},
+{ "virtio-scsi-device", "virtqueue_size", "128"},
{ "virtio-blk-device", "x-enable-wce-if-config-wce", "off" },
{ "virtio-blk-device", "seg-max-adjust", "off"},
{ "virtio-scsi-device", "seg_max_adjust", "off"},
diff --git a/hw/scsi/virtio-scsi.c b/hw/scsi/virtio-scsi.c
index 3b61563609..b38f50a429 100644
--- a/hw/scsi/virtio-scsi.c
+++ b/hw/scsi/virtio-scsi.c
@@ -660,7 +660,7 @@ static void virtio_scsi_get_config(VirtIODevice *vdev,
virtio_stl_p(vdev, >num_queues, s->conf.num_queues);
virtio_stl_p(vdev, >seg_max,
- s->conf.seg_max_adjust ? s->conf.virtqueue_size - 2 : 128 -
2);
+ s->conf.seg_max_adjust ? s->conf.virtqueue_size - 2 : 256 -
2);
virtio_stl_p(vdev, >max_sectors, s->conf.max_sectors);
virtio_stl_p(vdev, >cmd_per_lun, s->conf.cmd_per_lun);
virtio_stl_p(vdev, >event_info_size, sizeof(VirtIOSCSIEvent));
@@ -965,7 +965,7 @@ static void virtio_scsi_device_unrealize(DeviceState *dev,
Error **errp)
static Property virtio_scsi_properties[] = {
DEFINE_PROP_UINT32("num_queues", VirtIOSCSI, parent_obj.conf.num_queues,
1),
DEFINE_PROP_UINT32("virtqueue_size", VirtIOSCSI,
- parent_obj.conf.virtqueue_size, 128),
+ parent_obj.conf.virtqueue_size, 256),
DEFINE_PROP_BOOL("seg_max_adjust", VirtIOSCSI,
parent_obj.conf.seg_max_adjust, true),
DEFINE_PROP_UINT32("max_sectors", VirtIOSCSI, parent_obj.conf.max_sectors,
--
2.17.0
Re: [PATCH v1 0/2] Improve virtio_check_params test
On 11.02.2020 17:37, Philippe Mathieu-Daudé wrote: Hi Denis, On 2/11/20 3:25 PM, Denis Plotnikov wrote: * fixed failing on non-existed machine type removal * the test refactored to add more parameters to check Gereral questions left: How to restric test for using: 1. on a set of target OS-es 2. on a set target architectures Denis Plotnikov (2): tests/acceptance/virtio_check_params: remove excluded machine types carefully tests/acceptance/virtio_check_params: prepare to check different params tests/acceptance/virtio_check_params.py | 52 - 1 file changed, 33 insertions(+), 19 deletions(-) Have you noticed my other series suggested by Cornelia? It runs your test on S390X and PPC: https://www.mail-archive.com/qemu-devel@nongnu.org/msg675092.html https://www.mail-archive.com/qemu-devel@nongnu.org/msg675095.html Hi, Philippe Seems that I've missed them. I just made patches upon the fresh master. Can I get a git tree which has those patches applied? Or should I wait while the patches landed to qemu master and the rebase on them? Denis
[PATCH v1 0/2] Improve virtio_check_params test
* fixed failing on non-existed machine type removal * the test refactored to add more parameters to check Gereral questions left: How to restric test for using: 1. on a set of target OS-es 2. on a set target architectures Denis Plotnikov (2): tests/acceptance/virtio_check_params: remove excluded machine types carefully tests/acceptance/virtio_check_params: prepare to check different params tests/acceptance/virtio_check_params.py | 52 - 1 file changed, 33 insertions(+), 19 deletions(-) -- 2.17.0
[PATCH v1 2/2] tests/acceptance/virtio_check_params: prepare to check different params
Signed-off-by: Denis Plotnikov
---
tests/acceptance/virtio_check_params.py | 38 ++---
1 file changed, 22 insertions(+), 16 deletions(-)
diff --git a/tests/acceptance/virtio_check_params.py
b/tests/acceptance/virtio_check_params.py
index deec89bf86..e578952a97 100644
--- a/tests/acceptance/virtio_check_params.py
+++ b/tests/acceptance/virtio_check_params.py
@@ -43,7 +43,7 @@ VM_DEV_PARAMS = {'virtio-scsi-pci': ['-device',
'virtio-scsi-pci,id=scsi0'],
EXCLUDED_MACHINES = ['none', 'isapc', 'microvm']
-class VirtioMaxSegSettingsCheck(Test):
+class VirtioParamsCheck(Test):
@staticmethod
def make_pattern(props):
pattern_items = ['{0} = \w+'.format(prop) for prop in props]
@@ -75,12 +75,12 @@ class VirtioMaxSegSettingsCheck(Test):
props[p[0]] = p[1]
return query_ok, props, error
-def check_mt(self, mt, dev_type_name):
-mt['device'] = dev_type_name # Only for the debug() call.
+def check_mt(self, mt, expected_vals, dev_type_name):
+msg = "mt: %s dev: %s" % (mt, dev_type_name) # For debug() call only.
logger = logging.getLogger('machine')
-logger.debug(mt)
+logger.debug(msg)
with QEMUMachine(self.qemu_bin) as vm:
-vm.set_machine(mt["name"])
+vm.set_machine(mt)
vm.add_args('-nodefaults')
for s in VM_DEV_PARAMS[dev_type_name]:
vm.add_args(s)
@@ -92,11 +92,15 @@ class VirtioMaxSegSettingsCheck(Test):
error = sys.exc_info()[0]
if not query_ok:
-self.fail('machine type {0}: {1}'.format(mt['name'], error))
+self.fail('machine type {0}: {1}'.format(mt, error))
for prop_name, prop_val in props.items():
-expected_val = mt[prop_name]
-self.assertEqual(expected_val, prop_val)
+expected_val = expected_vals[prop_name]
+msg = 'Property value mismatch for (MT: {0}, '\
+ 'property name: {1}): expected value: "{2}" '\
+ 'actual value: "{3}"'\
+ .format(mt, prop_name, expected_val, prop_val)
+self.assertEqual(expected_val, prop_val, msg)
@staticmethod
def seg_max_adjust_enabled(mt):
@@ -128,25 +132,27 @@ class VirtioMaxSegSettingsCheck(Test):
@skip("break multi-arch CI")
def test_machine_types(self):
-# collect all machine types except 'none', 'isapc', 'microvm'
+# collect all machine types
with QEMUMachine(self.qemu_bin) as vm:
vm.launch()
machines = [m['name'] for m in vm.command('query-machines')]
vm.shutdown()
+# ..and exclude non-relevant ones
machines = self.filter_machines(machines)
for dev_type in DEV_TYPES:
-# create the list of machine types and their parameters.
-mtypes = list()
+# define expected parameters for each machine type
+mt_expected_vals = dict()
for m in machines:
if self.seg_max_adjust_enabled(m):
enabled = 'true'
else:
enabled = 'false'
-mtypes.append({'name': m,
- DEV_TYPES[dev_type]['seg_max_adjust']: enabled})
-# test each machine type for a device type
-for mt in mtypes:
-self.check_mt(mt, dev_type)
+mt_expected_vals[m] = {
+DEV_TYPES[dev_type]['seg_max_adjust']: enabled }
+
+# test each machine type
+for mt in mt_expected_vals:
+self.check_mt(mt, mt_expected_vals[mt], dev_type)
--
2.17.0
[PATCH v1 1/2] tests/acceptance/virtio_check_params: remove excluded machine types carefully
Before, the test failed if an excluded machine type was absent in the machine
types lists.
Signed-off-by: Denis Plotnikov
---
tests/acceptance/virtio_check_params.py | 14 +++---
1 file changed, 11 insertions(+), 3 deletions(-)
diff --git a/tests/acceptance/virtio_check_params.py
b/tests/acceptance/virtio_check_params.py
index 87e6c839d1..deec89bf86 100644
--- a/tests/acceptance/virtio_check_params.py
+++ b/tests/acceptance/virtio_check_params.py
@@ -40,6 +40,8 @@ VM_DEV_PARAMS = {'virtio-scsi-pci': ['-device',
'virtio-scsi-pci,id=scsi0'],
'-drive',
'driver=null-co,id=drive0,if=none']}
+EXCLUDED_MACHINES = ['none', 'isapc', 'microvm']
+
class VirtioMaxSegSettingsCheck(Test):
@staticmethod
@@ -117,6 +119,13 @@ class VirtioMaxSegSettingsCheck(Test):
return True
return False
+@staticmethod
+def filter_machines(machines):
+for mt in EXCLUDED_MACHINES:
+if mt in machines:
+machines.remove(mt)
+return machines
+
@skip("break multi-arch CI")
def test_machine_types(self):
# collect all machine types except 'none', 'isapc', 'microvm'
@@ -124,9 +133,8 @@ class VirtioMaxSegSettingsCheck(Test):
vm.launch()
machines = [m['name'] for m in vm.command('query-machines')]
vm.shutdown()
-machines.remove('none')
-machines.remove('isapc')
-machines.remove('microvm')
+
+machines = self.filter_machines(machines)
for dev_type in DEV_TYPES:
# create the list of machine types and their parameters.
--
2.17.0
[PATCH v2] virtio: increase virtuqueue size for virtio-scsi and virtio-blk
The goal is to reduce the amount of requests issued by a guest on
1M reads/writes. This rises the performance up to 4% on that kind of
disk access pattern.
The maximum chunk size to be used for the guest disk accessing is
limited with seg_max parameter, which represents the max amount of
pices in the scatter-geather list in one guest disk request.
Since seg_max is virqueue_size dependent, increasing the virtqueue
size increases seg_max, which, in turn, increases the maximum size
of data to be read/write from a guest disk.
More details in the original problem statment:
https://lists.gnu.org/archive/html/qemu-devel/2017-12/msg03721.html
Suggested-by: Denis V. Lunev
Signed-off-by: Denis Plotnikov
---
hw/block/virtio-blk.c | 4 ++--
hw/core/machine.c | 2 ++
hw/scsi/virtio-scsi.c | 4 ++--
3 files changed, 6 insertions(+), 4 deletions(-)
diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
index 09f46ed85f..6df3a7a6df 100644
--- a/hw/block/virtio-blk.c
+++ b/hw/block/virtio-blk.c
@@ -914,7 +914,7 @@ static void virtio_blk_update_config(VirtIODevice *vdev,
uint8_t *config)
memset(, 0, sizeof(blkcfg));
virtio_stq_p(vdev, , capacity);
virtio_stl_p(vdev, _max,
- s->conf.seg_max_adjust ? s->conf.queue_size - 2 : 128 - 2);
+ s->conf.seg_max_adjust ? s->conf.queue_size - 2 : 256 - 2);
virtio_stw_p(vdev, , conf->cyls);
virtio_stl_p(vdev, _size, blk_size);
virtio_stw_p(vdev, _io_size, conf->min_io_size / blk_size);
@@ -1272,7 +1272,7 @@ static Property virtio_blk_properties[] = {
DEFINE_PROP_BIT("request-merging", VirtIOBlock, conf.request_merging, 0,
true),
DEFINE_PROP_UINT16("num-queues", VirtIOBlock, conf.num_queues, 1),
-DEFINE_PROP_UINT16("queue-size", VirtIOBlock, conf.queue_size, 128),
+DEFINE_PROP_UINT16("queue-size", VirtIOBlock, conf.queue_size, 256),
DEFINE_PROP_BOOL("seg-max-adjust", VirtIOBlock, conf.seg_max_adjust, true),
DEFINE_PROP_LINK("iothread", VirtIOBlock, conf.iothread, TYPE_IOTHREAD,
IOThread *),
diff --git a/hw/core/machine.c b/hw/core/machine.c
index 2501b540ec..3427d6cf4c 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -28,6 +28,8 @@
#include "hw/mem/nvdimm.h"
GlobalProperty hw_compat_4_2[] = {
+{ "virtio-blk-device", "queue-size", "128"},
+{ "virtio-scsi-device", "virtqueue_size", "128"},
{ "virtio-blk-device", "x-enable-wce-if-config-wce", "off" },
{ "virtio-blk-device", "seg-max-adjust", "off"},
{ "virtio-scsi-device", "seg_max_adjust", "off"},
diff --git a/hw/scsi/virtio-scsi.c b/hw/scsi/virtio-scsi.c
index 3b61563609..b38f50a429 100644
--- a/hw/scsi/virtio-scsi.c
+++ b/hw/scsi/virtio-scsi.c
@@ -660,7 +660,7 @@ static void virtio_scsi_get_config(VirtIODevice *vdev,
virtio_stl_p(vdev, >num_queues, s->conf.num_queues);
virtio_stl_p(vdev, >seg_max,
- s->conf.seg_max_adjust ? s->conf.virtqueue_size - 2 : 128 -
2);
+ s->conf.seg_max_adjust ? s->conf.virtqueue_size - 2 : 256 -
2);
virtio_stl_p(vdev, >max_sectors, s->conf.max_sectors);
virtio_stl_p(vdev, >cmd_per_lun, s->conf.cmd_per_lun);
virtio_stl_p(vdev, >event_info_size, sizeof(VirtIOSCSIEvent));
@@ -965,7 +965,7 @@ static void virtio_scsi_device_unrealize(DeviceState *dev,
Error **errp)
static Property virtio_scsi_properties[] = {
DEFINE_PROP_UINT32("num_queues", VirtIOSCSI, parent_obj.conf.num_queues,
1),
DEFINE_PROP_UINT32("virtqueue_size", VirtIOSCSI,
- parent_obj.conf.virtqueue_size, 128),
+ parent_obj.conf.virtqueue_size, 256),
DEFINE_PROP_BOOL("seg_max_adjust", VirtIOSCSI,
parent_obj.conf.seg_max_adjust, true),
DEFINE_PROP_UINT32("max_sectors", VirtIOSCSI, parent_obj.conf.max_sectors,
--
2.17.0
[PATCH] pc: remove erroneous seg_max_adjust setting for vhost-blk-device
vhost-blk-device isn't a part of qemu.git
Signed-off-by: Denis Plotnikov
---
hw/core/machine.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/hw/core/machine.c b/hw/core/machine.c
index d8e30e4895..2501b540ec 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -31,7 +31,6 @@ GlobalProperty hw_compat_4_2[] = {
{ "virtio-blk-device", "x-enable-wce-if-config-wce", "off" },
{ "virtio-blk-device", "seg-max-adjust", "off"},
{ "virtio-scsi-device", "seg_max_adjust", "off"},
-{ "vhost-blk-device", "seg_max_adjust", "off"},
{ "usb-host", "suppress-remote-wake", "off" },
{ "usb-redir", "suppress-remote-wake", "off" },
};
--
2.17.0
Re: [PATCH v1 2/4] virtio: increase virtuqueue size for virtio-scsi and virtio-blk
On 09.02.2020 10:49, Michael S. Tsirkin wrote:
On Fri, Feb 07, 2020 at 11:48:05AM +0300, Denis Plotnikov wrote:
On 05.02.2020 14:19, Stefan Hajnoczi wrote:
On Tue, Feb 04, 2020 at 12:59:04PM +0300, Denis Plotnikov wrote:
On 30.01.2020 17:58, Stefan Hajnoczi wrote:
On Wed, Jan 29, 2020 at 05:07:00PM +0300, Denis Plotnikov wrote:
The goal is to reduce the amount of requests issued by a guest on
1M reads/writes. This rises the performance up to 4% on that kind of
disk access pattern.
The maximum chunk size to be used for the guest disk accessing is
limited with seg_max parameter, which represents the max amount of
pices in the scatter-geather list in one guest disk request.
Since seg_max is virqueue_size dependent, increasing the virtqueue
size increases seg_max, which, in turn, increases the maximum size
of data to be read/write from guest disk.
More details in the original problem statment:
https://lists.gnu.org/archive/html/qemu-devel/2017-12/msg03721.html
Suggested-by: Denis V. Lunev
Signed-off-by: Denis Plotnikov
---
hw/core/machine.c | 3 +++
include/hw/virtio/virtio.h | 2 +-
2 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/hw/core/machine.c b/hw/core/machine.c
index 3e288bfceb..8bc401d8b7 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -28,6 +28,9 @@
#include "hw/mem/nvdimm.h"
GlobalProperty hw_compat_4_2[] = {
+{ "virtio-blk-device", "queue-size", "128"},
+{ "virtio-scsi-device", "virtqueue_size", "128"},
+{ "vhost-blk-device", "virtqueue_size", "128"},
vhost-blk-device?! Who has this? It's not in qemu.git so please omit
this line. ;-)
So in this case the line:
{ "vhost-blk-device", "seg_max_adjust", "off"},
introduced by my patch:
commit 1bf8a989a566b2ba41c197004ec2a02562a766a4
Author: Denis Plotnikov
Date: Fri Dec 20 17:09:04 2019 +0300
virtio: make seg_max virtqueue size dependent
is also wrong. It should be:
{ "vhost-scsi-device", "seg_max_adjust", "off"},
Am I right?
It's just called "vhost-scsi":
include/hw/virtio/vhost-scsi.h:#define TYPE_VHOST_SCSI "vhost-scsi"
On the other hand, do you want to do this for the vhost-user-blk,
vhost-user-scsi, and vhost-scsi devices that exist in qemu.git? Those
devices would benefit from better performance too.
After thinking about that for a while, I think we shouldn't extend queue
sizes for vhost-user-blk, vhost-user-scsi and vhost-scsi.
This is because increasing the queue sizes seems to be just useless for
them: the all thing is about increasing the queue sizes for increasing
seg_max (it limits the max block query size from the guest). For
virtio-blk-device and virtio-scsi-device it makes sense, since they have
seg-max-adjust property which, if true, sets seg_max to virtqueue_size-2.
vhost-scsi also have this property but it seems the property just doesn't
affect anything (remove it?).
Also vhost-user-blk, vhost-user-scsi and vhost-scsi don't do any seg_max
settings. If I understand correctly, their backends are ment to be
responsible for doing that.
The queue size is set by qemu IIRC.
So, what about changing the queue sizes just for virtio-blk-device and
virtio-scsi-device?
Hmm that would break ability to migrate between userspace and vhost
backends, would it not?
I'm not sure I've understood what you meant.
Just for the record, I was going to change virtqueue-size for
virtio-blk-device and virtio-scsi-device since they can adjust seg_max
to the specified queue size and I don't want to touch vhost-s and
vhost-user-s since they don't have adjustable seg_max for now.
Denis
Denis
It seems to be so. We also have the test checking those settings:
tests/acceptance/virtio_seg_max_adjust.py
For now it checks virtio-scsi-pci and virtio-blk-pci.
I'm going to extend it for the virtqueue size checking.
If I change vhost-user-blk, vhost-user-scsi and vhost-scsi it's worth
to check those devices too. But I don't know how to form a command line
for that 3 devices since they should involve some third party components as
backends (kernel modules, DPDK, etc.) and they seems to be not available in
the
qemu git.
Is there any way to do it with some qit.qemu available stubs or something
else?
If so, could you please point out the proper way to do it?
qemu.git has contrib/vhost-user-blk/ and contrib/vhost-user-scsi/ if
you need to test those vhost-user devices without external dependencies.
Stefan
Re: [PATCH v1 2/4] virtio: increase virtuqueue size for virtio-scsi and virtio-blk
On 05.02.2020 14:19, Stefan Hajnoczi wrote:
On Tue, Feb 04, 2020 at 12:59:04PM +0300, Denis Plotnikov wrote:
On 30.01.2020 17:58, Stefan Hajnoczi wrote:
On Wed, Jan 29, 2020 at 05:07:00PM +0300, Denis Plotnikov wrote:
The goal is to reduce the amount of requests issued by a guest on
1M reads/writes. This rises the performance up to 4% on that kind of
disk access pattern.
The maximum chunk size to be used for the guest disk accessing is
limited with seg_max parameter, which represents the max amount of
pices in the scatter-geather list in one guest disk request.
Since seg_max is virqueue_size dependent, increasing the virtqueue
size increases seg_max, which, in turn, increases the maximum size
of data to be read/write from guest disk.
More details in the original problem statment:
https://lists.gnu.org/archive/html/qemu-devel/2017-12/msg03721.html
Suggested-by: Denis V. Lunev
Signed-off-by: Denis Plotnikov
---
hw/core/machine.c | 3 +++
include/hw/virtio/virtio.h | 2 +-
2 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/hw/core/machine.c b/hw/core/machine.c
index 3e288bfceb..8bc401d8b7 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -28,6 +28,9 @@
#include "hw/mem/nvdimm.h"
GlobalProperty hw_compat_4_2[] = {
+{ "virtio-blk-device", "queue-size", "128"},
+{ "virtio-scsi-device", "virtqueue_size", "128"},
+{ "vhost-blk-device", "virtqueue_size", "128"},
vhost-blk-device?! Who has this? It's not in qemu.git so please omit
this line. ;-)
So in this case the line:
{ "vhost-blk-device", "seg_max_adjust", "off"},
introduced by my patch:
commit 1bf8a989a566b2ba41c197004ec2a02562a766a4
Author: Denis Plotnikov
Date: Fri Dec 20 17:09:04 2019 +0300
virtio: make seg_max virtqueue size dependent
is also wrong. It should be:
{ "vhost-scsi-device", "seg_max_adjust", "off"},
Am I right?
It's just called "vhost-scsi":
include/hw/virtio/vhost-scsi.h:#define TYPE_VHOST_SCSI "vhost-scsi"
On the other hand, do you want to do this for the vhost-user-blk,
vhost-user-scsi, and vhost-scsi devices that exist in qemu.git? Those
devices would benefit from better performance too.
After thinking about that for a while, I think we shouldn't extend queue
sizes for vhost-user-blk, vhost-user-scsi and vhost-scsi.
This is because increasing the queue sizes seems to be just useless for
them: the all thing is about increasing the queue sizes for increasing
seg_max (it limits the max block query size from the guest). For
virtio-blk-device and virtio-scsi-device it makes sense, since they have
seg-max-adjust property which, if true, sets seg_max to
virtqueue_size-2. vhost-scsi also have this property but it seems the
property just doesn't affect anything (remove it?).
Also vhost-user-blk, vhost-user-scsi and vhost-scsi don't do any seg_max
settings. If I understand correctly, their backends are ment to be
responsible for doing that.
So, what about changing the queue sizes just for virtio-blk-device and
virtio-scsi-device?
Denis
It seems to be so. We also have the test checking those settings:
tests/acceptance/virtio_seg_max_adjust.py
For now it checks virtio-scsi-pci and virtio-blk-pci.
I'm going to extend it for the virtqueue size checking.
If I change vhost-user-blk, vhost-user-scsi and vhost-scsi it's worth
to check those devices too. But I don't know how to form a command line
for that 3 devices since they should involve some third party components as
backends (kernel modules, DPDK, etc.) and they seems to be not available in
the
qemu git.
Is there any way to do it with some qit.qemu available stubs or something
else?
If so, could you please point out the proper way to do it?
qemu.git has contrib/vhost-user-blk/ and contrib/vhost-user-scsi/ if
you need to test those vhost-user devices without external dependencies.
Stefan
Re: [PATCH v1 2/4] virtio: increase virtuqueue size for virtio-scsi and virtio-blk
On 30.01.2020 17:58, Stefan Hajnoczi wrote:
On Wed, Jan 29, 2020 at 05:07:00PM +0300, Denis Plotnikov wrote:
The goal is to reduce the amount of requests issued by a guest on
1M reads/writes. This rises the performance up to 4% on that kind of
disk access pattern.
The maximum chunk size to be used for the guest disk accessing is
limited with seg_max parameter, which represents the max amount of
pices in the scatter-geather list in one guest disk request.
Since seg_max is virqueue_size dependent, increasing the virtqueue
size increases seg_max, which, in turn, increases the maximum size
of data to be read/write from guest disk.
More details in the original problem statment:
https://lists.gnu.org/archive/html/qemu-devel/2017-12/msg03721.html
Suggested-by: Denis V. Lunev
Signed-off-by: Denis Plotnikov
---
hw/core/machine.c | 3 +++
include/hw/virtio/virtio.h | 2 +-
2 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/hw/core/machine.c b/hw/core/machine.c
index 3e288bfceb..8bc401d8b7 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -28,6 +28,9 @@
#include "hw/mem/nvdimm.h"
GlobalProperty hw_compat_4_2[] = {
+{ "virtio-blk-device", "queue-size", "128"},
+{ "virtio-scsi-device", "virtqueue_size", "128"},
+{ "vhost-blk-device", "virtqueue_size", "128"},
vhost-blk-device?! Who has this? It's not in qemu.git so please omit
this line. ;-)
So in this case the line:
{ "vhost-blk-device", "seg_max_adjust", "off"},
introduced by my patch:
commit 1bf8a989a566b2ba41c197004ec2a02562a766a4
Author: Denis Plotnikov
Date: Fri Dec 20 17:09:04 2019 +0300
virtio: make seg_max virtqueue size dependent
is also wrong. It should be:
{ "vhost-scsi-device", "seg_max_adjust", "off"},
Am I right?
On the other hand, do you want to do this for the vhost-user-blk,
vhost-user-scsi, and vhost-scsi devices that exist in qemu.git? Those
devices would benefit from better performance too.
It seems to be so. We also have the test checking those settings:
tests/acceptance/virtio_seg_max_adjust.py
For now it checks virtio-scsi-pci and virtio-blk-pci.
I'm going to extend it for the virtqueue size checking.
If I change vhost-user-blk, vhost-user-scsi and vhost-scsi it's worth
to check those devices too. But I don't know how to form a command line
for that 3 devices since they should involve some third party components as
backends (kernel modules, DPDK, etc.) and they seems to be not available
in the
qemu git.
Is there any way to do it with some qit.qemu available stubs or
something else?
If so, could you please point out the proper way to do it?
Thanks!
Denis
Re: [PATCH v1 1/4] virtio: introduce VIRTQUEUE_DEFUALT_SIZE instead of hardcoded constants
On 03.02.2020 15:51, Michael S. Tsirkin wrote:
On Mon, Feb 03, 2020 at 03:17:07PM +0300, Denis Plotnikov wrote:
On 30.01.2020 16:38, Michael S. Tsirkin wrote:
On Wed, Jan 29, 2020 at 05:06:59PM +0300, Denis Plotnikov wrote:
Signed-off-by: Denis Plotnikov
I'm not sure what the point is. It's more or less an accident that
these two devices share the queue size, this constance
makes no sense to me.
Ok, then let's just make a separate queue length constant for each type.
it's just a number, I don't think we need a constant here.
If you feel it needs documentation, add a comment!
I just thought that the meaningful name for the number would be better
for the code understanding.
Anyway, If doesn't improve anything I'll just change the number and add
a comment what it means.
Denis
(Will redo and send in the next series)
Thanks!
Denis
---
hw/block/virtio-blk.c | 6 --
hw/scsi/virtio-scsi.c | 5 +++--
include/hw/virtio/virtio.h | 1 +
3 files changed, 8 insertions(+), 4 deletions(-)
diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
index 09f46ed85f..72f935033f 100644
--- a/hw/block/virtio-blk.c
+++ b/hw/block/virtio-blk.c
@@ -914,7 +914,8 @@ static void virtio_blk_update_config(VirtIODevice *vdev,
uint8_t *config)
memset(, 0, sizeof(blkcfg));
virtio_stq_p(vdev, , capacity);
virtio_stl_p(vdev, _max,
- s->conf.seg_max_adjust ? s->conf.queue_size - 2 : 128 - 2);
+ s->conf.seg_max_adjust ? s->conf.queue_size - 2 :
+ VIRTQUEUE_DEFAULT_SIZE - 2);
virtio_stw_p(vdev, , conf->cyls);
virtio_stl_p(vdev, _size, blk_size);
virtio_stw_p(vdev, _io_size, conf->min_io_size / blk_size);
@@ -1272,7 +1273,8 @@ static Property virtio_blk_properties[] = {
DEFINE_PROP_BIT("request-merging", VirtIOBlock, conf.request_merging, 0,
true),
DEFINE_PROP_UINT16("num-queues", VirtIOBlock, conf.num_queues, 1),
-DEFINE_PROP_UINT16("queue-size", VirtIOBlock, conf.queue_size, 128),
+DEFINE_PROP_UINT16("queue-size", VirtIOBlock, conf.queue_size,
+ VIRTQUEUE_DEFAULT_SIZE),
DEFINE_PROP_BOOL("seg-max-adjust", VirtIOBlock, conf.seg_max_adjust,
true),
DEFINE_PROP_LINK("iothread", VirtIOBlock, conf.iothread, TYPE_IOTHREAD,
IOThread *),
diff --git a/hw/scsi/virtio-scsi.c b/hw/scsi/virtio-scsi.c
index 3b61563609..36f66046ae 100644
--- a/hw/scsi/virtio-scsi.c
+++ b/hw/scsi/virtio-scsi.c
@@ -660,7 +660,8 @@ static void virtio_scsi_get_config(VirtIODevice *vdev,
virtio_stl_p(vdev, >num_queues, s->conf.num_queues);
virtio_stl_p(vdev, >seg_max,
- s->conf.seg_max_adjust ? s->conf.virtqueue_size - 2 : 128 -
2);
+ s->conf.seg_max_adjust ? s->conf.virtqueue_size - 2 :
+ VIRTQUEUE_DEFAULT_SIZE - 2);
virtio_stl_p(vdev, >max_sectors, s->conf.max_sectors);
virtio_stl_p(vdev, >cmd_per_lun, s->conf.cmd_per_lun);
virtio_stl_p(vdev, >event_info_size, sizeof(VirtIOSCSIEvent));
@@ -965,7 +966,7 @@ static void virtio_scsi_device_unrealize(DeviceState *dev,
Error **errp)
static Property virtio_scsi_properties[] = {
DEFINE_PROP_UINT32("num_queues", VirtIOSCSI, parent_obj.conf.num_queues,
1),
DEFINE_PROP_UINT32("virtqueue_size", VirtIOSCSI,
- parent_obj.conf.virtqueue_size, 128),
+ parent_obj.conf.virtqueue_size, VIRTQUEUE_DEFAULT_SIZE),
DEFINE_PROP_BOOL("seg_max_adjust", VirtIOSCSI,
parent_obj.conf.seg_max_adjust, true),
DEFINE_PROP_UINT32("max_sectors", VirtIOSCSI,
parent_obj.conf.max_sectors,
diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
index b69d517496..a66ea2368b 100644
--- a/include/hw/virtio/virtio.h
+++ b/include/hw/virtio/virtio.h
@@ -48,6 +48,7 @@ size_t virtio_feature_get_config_size(VirtIOFeature *features,
typedef struct VirtQueue VirtQueue;
#define VIRTQUEUE_MAX_SIZE 1024
+#define VIRTQUEUE_DEFAULT_SIZE 128
typedef struct VirtQueueElement
{
--
2.17.0
Re: [PATCH v1 3/4] tests: add virtuqueue size checking to virtio_seg_max_adjust test
On 30.01.2020 16:42, Michael S. Tsirkin wrote:
On Wed, Jan 29, 2020 at 05:07:01PM +0300, Denis Plotnikov wrote:
This is due to the change in the default virtqueue_size in the
latest machine type to improve guest disks performance.
Sorry what is due to the change?
Signed-off-by: Denis Plotnikov
---
tests/acceptance/virtio_seg_max_adjust.py | 33 ++-
1 file changed, 21 insertions(+), 12 deletions(-)
diff --git a/tests/acceptance/virtio_seg_max_adjust.py
b/tests/acceptance/virtio_seg_max_adjust.py
index 5458573138..645d69b313 100755
--- a/tests/acceptance/virtio_seg_max_adjust.py
+++ b/tests/acceptance/virtio_seg_max_adjust.py
@@ -27,8 +27,10 @@ from qemu.machine import QEMUMachine
from avocado_qemu import Test
#list of machine types and virtqueue properties to test
-VIRTIO_SCSI_PROPS = {'seg_max_adjust': 'seg_max_adjust'}
-VIRTIO_BLK_PROPS = {'seg_max_adjust': 'seg-max-adjust'}
+VIRTIO_SCSI_PROPS = {'seg_max_adjust': 'seg_max_adjust',
+ 'queue_size': 'virtqueue_size'}
+VIRTIO_BLK_PROPS = {'seg_max_adjust': 'seg-max-adjust',
+'queue_size': 'queue-size'}
DEV_TYPES = {'virtio-scsi-pci': VIRTIO_SCSI_PROPS,
'virtio-blk-pci': VIRTIO_BLK_PROPS}
@@ -40,7 +42,7 @@ VM_DEV_PARAMS = {'virtio-scsi-pci': ['-device',
'virtio-scsi-pci,id=scsi0'],
'driver=null-co,id=drive0,if=none']}
-class VirtioMaxSegSettingsCheck(Test):
+class VirtioPramsCheck(Test):
@staticmethod
def make_pattern(props):
pattern_items = ['{0} = \w+'.format(prop) for prop in props]
@@ -72,20 +74,24 @@ class VirtioMaxSegSettingsCheck(Test):
props[p[0]] = p[1]
return query_ok, props, error
-def check_mt(self, mt, dev_type_name):
+def check_mt(self, mt, expected_props, dev_type_name):
with QEMUMachine(self.qemu_bin) as vm:
-vm.set_machine(mt["name"])
+vm.set_machine(mt)
for s in VM_DEV_PARAMS[dev_type_name]:
vm.add_args(s)
vm.launch()
query_ok, props, error = self.query_virtqueue(vm, dev_type_name)
if not query_ok:
-self.fail('machine type {0}: {1}'.format(mt['name'], error))
+self.fail('machine type {0}: {1}'.format(mt, error))
for prop_name, prop_val in props.items():
-expected_val = mt[prop_name]
-self.assertEqual(expected_val, prop_val)
+expected_val = expected_props[prop_name]
+msg = 'Property value mismatch for (MT: {0}, '\
+ 'property name: {1}): expected value: "{2}" '\
+ 'actual value: "{3}"'\
+ .format(mt, prop_name, expected_val, prop_val)
+self.assertEqual(expected_val, prop_val, msg)
Looks like an unrelated change, no?
Yep, I'd better split the patches and add the test improvements in the
separate one.
Denis
@staticmethod
def seg_max_adjust_enabled(mt):
@@ -120,15 +126,18 @@ class VirtioMaxSegSettingsCheck(Test):
for dev_type in DEV_TYPES:
# create the list of machine types and their parameters.
-mtypes = list()
+mtypes = dict()
for m in machines:
if self.seg_max_adjust_enabled(m):
enabled = 'true'
+queue_size = '256'
else:
enabled = 'false'
-mtypes.append({'name': m,
- DEV_TYPES[dev_type]['seg_max_adjust']: enabled})
+queue_size = '128'
+mtypes[m] = {
+DEV_TYPES[dev_type]['seg_max_adjust']: enabled,
+DEV_TYPES[dev_type]['queue_size']: queue_size }
# test each machine type for a device type
for mt in mtypes:
-self.check_mt(mt, dev_type)
+self.check_mt(mt, mtypes[mt], dev_type)
--
2.17.0
Re: [PATCH v1 2/4] virtio: increase virtuqueue size for virtio-scsi and virtio-blk
On 30.01.2020 16:40, Michael S. Tsirkin wrote:
On Wed, Jan 29, 2020 at 05:07:00PM +0300, Denis Plotnikov wrote:
The goal is to reduce the amount of requests issued by a guest on
1M reads/writes. This rises the performance up to 4% on that kind of
disk access pattern.
The maximum chunk size to be used for the guest disk accessing is
limited with seg_max parameter, which represents the max amount of
pices in the scatter-geather list in one guest disk request.
Since seg_max is virqueue_size dependent, increasing the virtqueue
size increases seg_max, which, in turn, increases the maximum size
of data to be read/write from guest disk.
More details in the original problem statment:
https://lists.gnu.org/archive/html/qemu-devel/2017-12/msg03721.html
Suggested-by: Denis V. Lunev
Signed-off-by: Denis Plotnikov
looks good but let's just patch each device separately.
Ok
Denis
---
hw/core/machine.c | 3 +++
include/hw/virtio/virtio.h | 2 +-
2 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/hw/core/machine.c b/hw/core/machine.c
index 3e288bfceb..8bc401d8b7 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -28,6 +28,9 @@
#include "hw/mem/nvdimm.h"
GlobalProperty hw_compat_4_2[] = {
+{ "virtio-blk-device", "queue-size", "128"},
+{ "virtio-scsi-device", "virtqueue_size", "128"},
+{ "vhost-blk-device", "virtqueue_size", "128"},
{ "virtio-blk-device", "x-enable-wce-if-config-wce", "off" },
{ "virtio-blk-device", "seg-max-adjust", "off"},
{ "virtio-scsi-device", "seg_max_adjust", "off"},
diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
index a66ea2368b..16d540e390 100644
--- a/include/hw/virtio/virtio.h
+++ b/include/hw/virtio/virtio.h
@@ -48,7 +48,7 @@ size_t virtio_feature_get_config_size(VirtIOFeature *features,
typedef struct VirtQueue VirtQueue;
#define VIRTQUEUE_MAX_SIZE 1024
-#define VIRTQUEUE_DEFAULT_SIZE 128
+#define VIRTQUEUE_DEFAULT_SIZE 256
typedef struct VirtQueueElement
{
--
2.17.0
Re: [PATCH v1 1/4] virtio: introduce VIRTQUEUE_DEFUALT_SIZE instead of hardcoded constants
On 30.01.2020 16:38, Michael S. Tsirkin wrote:
On Wed, Jan 29, 2020 at 05:06:59PM +0300, Denis Plotnikov wrote:
Signed-off-by: Denis Plotnikov
I'm not sure what the point is. It's more or less an accident that
these two devices share the queue size, this constance
makes no sense to me.
Ok, then let's just make a separate queue length constant for each type.
(Will redo and send in the next series)
Thanks!
Denis
---
hw/block/virtio-blk.c | 6 --
hw/scsi/virtio-scsi.c | 5 +++--
include/hw/virtio/virtio.h | 1 +
3 files changed, 8 insertions(+), 4 deletions(-)
diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
index 09f46ed85f..72f935033f 100644
--- a/hw/block/virtio-blk.c
+++ b/hw/block/virtio-blk.c
@@ -914,7 +914,8 @@ static void virtio_blk_update_config(VirtIODevice *vdev,
uint8_t *config)
memset(, 0, sizeof(blkcfg));
virtio_stq_p(vdev, , capacity);
virtio_stl_p(vdev, _max,
- s->conf.seg_max_adjust ? s->conf.queue_size - 2 : 128 - 2);
+ s->conf.seg_max_adjust ? s->conf.queue_size - 2 :
+ VIRTQUEUE_DEFAULT_SIZE - 2);
virtio_stw_p(vdev, , conf->cyls);
virtio_stl_p(vdev, _size, blk_size);
virtio_stw_p(vdev, _io_size, conf->min_io_size / blk_size);
@@ -1272,7 +1273,8 @@ static Property virtio_blk_properties[] = {
DEFINE_PROP_BIT("request-merging", VirtIOBlock, conf.request_merging, 0,
true),
DEFINE_PROP_UINT16("num-queues", VirtIOBlock, conf.num_queues, 1),
-DEFINE_PROP_UINT16("queue-size", VirtIOBlock, conf.queue_size, 128),
+DEFINE_PROP_UINT16("queue-size", VirtIOBlock, conf.queue_size,
+ VIRTQUEUE_DEFAULT_SIZE),
DEFINE_PROP_BOOL("seg-max-adjust", VirtIOBlock, conf.seg_max_adjust,
true),
DEFINE_PROP_LINK("iothread", VirtIOBlock, conf.iothread, TYPE_IOTHREAD,
IOThread *),
diff --git a/hw/scsi/virtio-scsi.c b/hw/scsi/virtio-scsi.c
index 3b61563609..36f66046ae 100644
--- a/hw/scsi/virtio-scsi.c
+++ b/hw/scsi/virtio-scsi.c
@@ -660,7 +660,8 @@ static void virtio_scsi_get_config(VirtIODevice *vdev,
virtio_stl_p(vdev, >num_queues, s->conf.num_queues);
virtio_stl_p(vdev, >seg_max,
- s->conf.seg_max_adjust ? s->conf.virtqueue_size - 2 : 128 -
2);
+ s->conf.seg_max_adjust ? s->conf.virtqueue_size - 2 :
+ VIRTQUEUE_DEFAULT_SIZE - 2);
virtio_stl_p(vdev, >max_sectors, s->conf.max_sectors);
virtio_stl_p(vdev, >cmd_per_lun, s->conf.cmd_per_lun);
virtio_stl_p(vdev, >event_info_size, sizeof(VirtIOSCSIEvent));
@@ -965,7 +966,7 @@ static void virtio_scsi_device_unrealize(DeviceState *dev,
Error **errp)
static Property virtio_scsi_properties[] = {
DEFINE_PROP_UINT32("num_queues", VirtIOSCSI, parent_obj.conf.num_queues,
1),
DEFINE_PROP_UINT32("virtqueue_size", VirtIOSCSI,
- parent_obj.conf.virtqueue_size, 128),
+ parent_obj.conf.virtqueue_size, VIRTQUEUE_DEFAULT_SIZE),
DEFINE_PROP_BOOL("seg_max_adjust", VirtIOSCSI,
parent_obj.conf.seg_max_adjust, true),
DEFINE_PROP_UINT32("max_sectors", VirtIOSCSI, parent_obj.conf.max_sectors,
diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
index b69d517496..a66ea2368b 100644
--- a/include/hw/virtio/virtio.h
+++ b/include/hw/virtio/virtio.h
@@ -48,6 +48,7 @@ size_t virtio_feature_get_config_size(VirtIOFeature *features,
typedef struct VirtQueue VirtQueue;
#define VIRTQUEUE_MAX_SIZE 1024
+#define VIRTQUEUE_DEFAULT_SIZE 128
typedef struct VirtQueueElement
{
--
2.17.0
Re: [PATCH v1 1/4] virtio: introduce VIRTQUEUE_DEFUALT_SIZE instead of hardcoded constants
On 30.01.2020 17:56, Stefan Hajnoczi wrote:
On Wed, Jan 29, 2020 at 06:55:18PM +0100, Cornelia Huck wrote:
On Wed, 29 Jan 2020 17:06:59 +0300
Denis Plotnikov wrote:
Signed-off-by: Denis Plotnikov
---
hw/block/virtio-blk.c | 6 --
hw/scsi/virtio-scsi.c | 5 +++--
include/hw/virtio/virtio.h | 1 +
3 files changed, 8 insertions(+), 4 deletions(-)
diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
index 09f46ed85f..72f935033f 100644
--- a/hw/block/virtio-blk.c
+++ b/hw/block/virtio-blk.c
@@ -914,7 +914,8 @@ static void virtio_blk_update_config(VirtIODevice *vdev,
uint8_t *config)
memset(, 0, sizeof(blkcfg));
virtio_stq_p(vdev, , capacity);
virtio_stl_p(vdev, _max,
- s->conf.seg_max_adjust ? s->conf.queue_size - 2 : 128 - 2);
+ s->conf.seg_max_adjust ? s->conf.queue_size - 2 :
+ VIRTQUEUE_DEFAULT_SIZE - 2);
virtio_stw_p(vdev, , conf->cyls);
virtio_stl_p(vdev, _size, blk_size);
virtio_stw_p(vdev, _io_size, conf->min_io_size / blk_size);
@@ -1272,7 +1273,8 @@ static Property virtio_blk_properties[] = {
DEFINE_PROP_BIT("request-merging", VirtIOBlock, conf.request_merging, 0,
true),
DEFINE_PROP_UINT16("num-queues", VirtIOBlock, conf.num_queues, 1),
-DEFINE_PROP_UINT16("queue-size", VirtIOBlock, conf.queue_size, 128),
+DEFINE_PROP_UINT16("queue-size", VirtIOBlock, conf.queue_size,
+ VIRTQUEUE_DEFAULT_SIZE),
DEFINE_PROP_BOOL("seg-max-adjust", VirtIOBlock, conf.seg_max_adjust,
true),
DEFINE_PROP_LINK("iothread", VirtIOBlock, conf.iothread, TYPE_IOTHREAD,
IOThread *),
diff --git a/hw/scsi/virtio-scsi.c b/hw/scsi/virtio-scsi.c
index 3b61563609..36f66046ae 100644
--- a/hw/scsi/virtio-scsi.c
+++ b/hw/scsi/virtio-scsi.c
@@ -660,7 +660,8 @@ static void virtio_scsi_get_config(VirtIODevice *vdev,
virtio_stl_p(vdev, >num_queues, s->conf.num_queues);
virtio_stl_p(vdev, >seg_max,
- s->conf.seg_max_adjust ? s->conf.virtqueue_size - 2 : 128 -
2);
+ s->conf.seg_max_adjust ? s->conf.virtqueue_size - 2 :
+ VIRTQUEUE_DEFAULT_SIZE - 2);
virtio_stl_p(vdev, >max_sectors, s->conf.max_sectors);
virtio_stl_p(vdev, >cmd_per_lun, s->conf.cmd_per_lun);
virtio_stl_p(vdev, >event_info_size, sizeof(VirtIOSCSIEvent));
@@ -965,7 +966,7 @@ static void virtio_scsi_device_unrealize(DeviceState *dev,
Error **errp)
static Property virtio_scsi_properties[] = {
DEFINE_PROP_UINT32("num_queues", VirtIOSCSI, parent_obj.conf.num_queues,
1),
DEFINE_PROP_UINT32("virtqueue_size", VirtIOSCSI,
- parent_obj.conf.virtqueue_size, 128),
+ parent_obj.conf.virtqueue_size, VIRTQUEUE_DEFAULT_SIZE),
DEFINE_PROP_BOOL("seg_max_adjust", VirtIOSCSI,
parent_obj.conf.seg_max_adjust, true),
DEFINE_PROP_UINT32("max_sectors", VirtIOSCSI, parent_obj.conf.max_sectors,
diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
index b69d517496..a66ea2368b 100644
--- a/include/hw/virtio/virtio.h
+++ b/include/hw/virtio/virtio.h
@@ -48,6 +48,7 @@ size_t virtio_feature_get_config_size(VirtIOFeature *features,
typedef struct VirtQueue VirtQueue;
#define VIRTQUEUE_MAX_SIZE 1024
+#define VIRTQUEUE_DEFAULT_SIZE 128
Going from the header only, this looks like a value that is supposed to
be used for every virtqueue... but from the users, this is only for blk
and scsi.
I don't think adding a default for everything makes sense, even if the
same value makes sense for blk and scsi.
Agreed, this value is too general. VIRTIO_BLK_VQ_DEFAULT_SIZE and
VIRTIO_SCSI_VQ_DEFAULT_SIZE would make sense to me.
Stefan
agree, that would be better.
Will redo and resend the series.
Denis
[PATCH v1 2/4] virtio: increase virtuqueue size for virtio-scsi and virtio-blk
The goal is to reduce the amount of requests issued by a guest on
1M reads/writes. This rises the performance up to 4% on that kind of
disk access pattern.
The maximum chunk size to be used for the guest disk accessing is
limited with seg_max parameter, which represents the max amount of
pices in the scatter-geather list in one guest disk request.
Since seg_max is virqueue_size dependent, increasing the virtqueue
size increases seg_max, which, in turn, increases the maximum size
of data to be read/write from guest disk.
More details in the original problem statment:
https://lists.gnu.org/archive/html/qemu-devel/2017-12/msg03721.html
Suggested-by: Denis V. Lunev
Signed-off-by: Denis Plotnikov
---
hw/core/machine.c | 3 +++
include/hw/virtio/virtio.h | 2 +-
2 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/hw/core/machine.c b/hw/core/machine.c
index 3e288bfceb..8bc401d8b7 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -28,6 +28,9 @@
#include "hw/mem/nvdimm.h"
GlobalProperty hw_compat_4_2[] = {
+{ "virtio-blk-device", "queue-size", "128"},
+{ "virtio-scsi-device", "virtqueue_size", "128"},
+{ "vhost-blk-device", "virtqueue_size", "128"},
{ "virtio-blk-device", "x-enable-wce-if-config-wce", "off" },
{ "virtio-blk-device", "seg-max-adjust", "off"},
{ "virtio-scsi-device", "seg_max_adjust", "off"},
diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
index a66ea2368b..16d540e390 100644
--- a/include/hw/virtio/virtio.h
+++ b/include/hw/virtio/virtio.h
@@ -48,7 +48,7 @@ size_t virtio_feature_get_config_size(VirtIOFeature *features,
typedef struct VirtQueue VirtQueue;
#define VIRTQUEUE_MAX_SIZE 1024
-#define VIRTQUEUE_DEFAULT_SIZE 128
+#define VIRTQUEUE_DEFAULT_SIZE 256
typedef struct VirtQueueElement
{
--
2.17.0
[PATCH v1 0/4] Increase default virtqueue size to improve performance
The goal is to increase the performance of the block layer on
1M reads/writes up to 4% by reducing the amount of requests issued by a guest
using virtio-scsi or virtio-blk devices.
Original problem description:
https://lists.gnu.org/archive/html/qemu-devel/2017-12/msg03721.html
Denis Plotnikov (4):
virtio: introduce VIRTQUEUE_DEFUALT_SIZE instead of hardcoded
constants
virtio: increase virtuqueue size for virtio-scsi and virtio-blk
tests: add virtuqueue size checking to virtio_seg_max_adjust test
tests: rename virtio_seg_max_adjust to virtio_check_params
hw/block/virtio-blk.c | 6 ++--
hw/core/machine.c | 3 ++
hw/scsi/virtio-scsi.c | 5 +--
include/hw/virtio/virtio.h| 1 +
...g_max_adjust.py => virtio_check_params.py} | 33 ---
5 files changed, 32 insertions(+), 16 deletions(-)
rename tests/acceptance/{virtio_seg_max_adjust.py => virtio_check_params.py}
(79%)
--
2.17.0
[PATCH v1 3/4] tests: add virtuqueue size checking to virtio_seg_max_adjust test
This is due to the change in the default virtqueue_size in the
latest machine type to improve guest disks performance.
Signed-off-by: Denis Plotnikov
---
tests/acceptance/virtio_seg_max_adjust.py | 33 ++-
1 file changed, 21 insertions(+), 12 deletions(-)
diff --git a/tests/acceptance/virtio_seg_max_adjust.py
b/tests/acceptance/virtio_seg_max_adjust.py
index 5458573138..645d69b313 100755
--- a/tests/acceptance/virtio_seg_max_adjust.py
+++ b/tests/acceptance/virtio_seg_max_adjust.py
@@ -27,8 +27,10 @@ from qemu.machine import QEMUMachine
from avocado_qemu import Test
#list of machine types and virtqueue properties to test
-VIRTIO_SCSI_PROPS = {'seg_max_adjust': 'seg_max_adjust'}
-VIRTIO_BLK_PROPS = {'seg_max_adjust': 'seg-max-adjust'}
+VIRTIO_SCSI_PROPS = {'seg_max_adjust': 'seg_max_adjust',
+ 'queue_size': 'virtqueue_size'}
+VIRTIO_BLK_PROPS = {'seg_max_adjust': 'seg-max-adjust',
+'queue_size': 'queue-size'}
DEV_TYPES = {'virtio-scsi-pci': VIRTIO_SCSI_PROPS,
'virtio-blk-pci': VIRTIO_BLK_PROPS}
@@ -40,7 +42,7 @@ VM_DEV_PARAMS = {'virtio-scsi-pci': ['-device',
'virtio-scsi-pci,id=scsi0'],
'driver=null-co,id=drive0,if=none']}
-class VirtioMaxSegSettingsCheck(Test):
+class VirtioPramsCheck(Test):
@staticmethod
def make_pattern(props):
pattern_items = ['{0} = \w+'.format(prop) for prop in props]
@@ -72,20 +74,24 @@ class VirtioMaxSegSettingsCheck(Test):
props[p[0]] = p[1]
return query_ok, props, error
-def check_mt(self, mt, dev_type_name):
+def check_mt(self, mt, expected_props, dev_type_name):
with QEMUMachine(self.qemu_bin) as vm:
-vm.set_machine(mt["name"])
+vm.set_machine(mt)
for s in VM_DEV_PARAMS[dev_type_name]:
vm.add_args(s)
vm.launch()
query_ok, props, error = self.query_virtqueue(vm, dev_type_name)
if not query_ok:
-self.fail('machine type {0}: {1}'.format(mt['name'], error))
+self.fail('machine type {0}: {1}'.format(mt, error))
for prop_name, prop_val in props.items():
-expected_val = mt[prop_name]
-self.assertEqual(expected_val, prop_val)
+expected_val = expected_props[prop_name]
+msg = 'Property value mismatch for (MT: {0}, '\
+ 'property name: {1}): expected value: "{2}" '\
+ 'actual value: "{3}"'\
+ .format(mt, prop_name, expected_val, prop_val)
+self.assertEqual(expected_val, prop_val, msg)
@staticmethod
def seg_max_adjust_enabled(mt):
@@ -120,15 +126,18 @@ class VirtioMaxSegSettingsCheck(Test):
for dev_type in DEV_TYPES:
# create the list of machine types and their parameters.
-mtypes = list()
+mtypes = dict()
for m in machines:
if self.seg_max_adjust_enabled(m):
enabled = 'true'
+queue_size = '256'
else:
enabled = 'false'
-mtypes.append({'name': m,
- DEV_TYPES[dev_type]['seg_max_adjust']: enabled})
+queue_size = '128'
+mtypes[m] = {
+DEV_TYPES[dev_type]['seg_max_adjust']: enabled,
+DEV_TYPES[dev_type]['queue_size']: queue_size }
# test each machine type for a device type
for mt in mtypes:
-self.check_mt(mt, dev_type)
+self.check_mt(mt, mtypes[mt], dev_type)
--
2.17.0
[PATCH v1 1/4] virtio: introduce VIRTQUEUE_DEFUALT_SIZE instead of hardcoded constants
Signed-off-by: Denis Plotnikov
---
hw/block/virtio-blk.c | 6 --
hw/scsi/virtio-scsi.c | 5 +++--
include/hw/virtio/virtio.h | 1 +
3 files changed, 8 insertions(+), 4 deletions(-)
diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
index 09f46ed85f..72f935033f 100644
--- a/hw/block/virtio-blk.c
+++ b/hw/block/virtio-blk.c
@@ -914,7 +914,8 @@ static void virtio_blk_update_config(VirtIODevice *vdev,
uint8_t *config)
memset(, 0, sizeof(blkcfg));
virtio_stq_p(vdev, , capacity);
virtio_stl_p(vdev, _max,
- s->conf.seg_max_adjust ? s->conf.queue_size - 2 : 128 - 2);
+ s->conf.seg_max_adjust ? s->conf.queue_size - 2 :
+ VIRTQUEUE_DEFAULT_SIZE - 2);
virtio_stw_p(vdev, , conf->cyls);
virtio_stl_p(vdev, _size, blk_size);
virtio_stw_p(vdev, _io_size, conf->min_io_size / blk_size);
@@ -1272,7 +1273,8 @@ static Property virtio_blk_properties[] = {
DEFINE_PROP_BIT("request-merging", VirtIOBlock, conf.request_merging, 0,
true),
DEFINE_PROP_UINT16("num-queues", VirtIOBlock, conf.num_queues, 1),
-DEFINE_PROP_UINT16("queue-size", VirtIOBlock, conf.queue_size, 128),
+DEFINE_PROP_UINT16("queue-size", VirtIOBlock, conf.queue_size,
+ VIRTQUEUE_DEFAULT_SIZE),
DEFINE_PROP_BOOL("seg-max-adjust", VirtIOBlock, conf.seg_max_adjust, true),
DEFINE_PROP_LINK("iothread", VirtIOBlock, conf.iothread, TYPE_IOTHREAD,
IOThread *),
diff --git a/hw/scsi/virtio-scsi.c b/hw/scsi/virtio-scsi.c
index 3b61563609..36f66046ae 100644
--- a/hw/scsi/virtio-scsi.c
+++ b/hw/scsi/virtio-scsi.c
@@ -660,7 +660,8 @@ static void virtio_scsi_get_config(VirtIODevice *vdev,
virtio_stl_p(vdev, >num_queues, s->conf.num_queues);
virtio_stl_p(vdev, >seg_max,
- s->conf.seg_max_adjust ? s->conf.virtqueue_size - 2 : 128 -
2);
+ s->conf.seg_max_adjust ? s->conf.virtqueue_size - 2 :
+ VIRTQUEUE_DEFAULT_SIZE - 2);
virtio_stl_p(vdev, >max_sectors, s->conf.max_sectors);
virtio_stl_p(vdev, >cmd_per_lun, s->conf.cmd_per_lun);
virtio_stl_p(vdev, >event_info_size, sizeof(VirtIOSCSIEvent));
@@ -965,7 +966,7 @@ static void virtio_scsi_device_unrealize(DeviceState *dev,
Error **errp)
static Property virtio_scsi_properties[] = {
DEFINE_PROP_UINT32("num_queues", VirtIOSCSI, parent_obj.conf.num_queues,
1),
DEFINE_PROP_UINT32("virtqueue_size", VirtIOSCSI,
- parent_obj.conf.virtqueue_size, 128),
+ parent_obj.conf.virtqueue_size, VIRTQUEUE_DEFAULT_SIZE),
DEFINE_PROP_BOOL("seg_max_adjust", VirtIOSCSI,
parent_obj.conf.seg_max_adjust, true),
DEFINE_PROP_UINT32("max_sectors", VirtIOSCSI, parent_obj.conf.max_sectors,
diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
index b69d517496..a66ea2368b 100644
--- a/include/hw/virtio/virtio.h
+++ b/include/hw/virtio/virtio.h
@@ -48,6 +48,7 @@ size_t virtio_feature_get_config_size(VirtIOFeature *features,
typedef struct VirtQueue VirtQueue;
#define VIRTQUEUE_MAX_SIZE 1024
+#define VIRTQUEUE_DEFAULT_SIZE 128
typedef struct VirtQueueElement
{
--
2.17.0
[PATCH v1 4/4] tests: rename virtio_seg_max_adjust to virtio_check_params
Since, virtio_seg_max_adjust checks not only seg_max, but also
virtqueue_size parameter, let's make the test more general and
add new parameters to be checked there in the future.
Signed-off-by: Denis Plotnikov
---
.../{virtio_seg_max_adjust.py => virtio_check_params.py} | 0
1 file changed, 0 insertions(+), 0 deletions(-)
rename tests/acceptance/{virtio_seg_max_adjust.py => virtio_check_params.py}
(100%)
diff --git a/tests/acceptance/virtio_seg_max_adjust.py
b/tests/acceptance/virtio_check_params.py
similarity index 100%
rename from tests/acceptance/virtio_seg_max_adjust.py
rename to tests/acceptance/virtio_check_params.py
--
2.17.0
Re: [PATCH 0/3] migration: add sztd compression
Hi, Juan I'll read the series soon. Thanks for sending that to me! Denis On 24.01.2020 15:43, Juan Quintela wrote: > Denis Plotnikov wrote: >> zstd date compression algorithm shows better performance on data compression. >> It might be useful to employ the algorithm in VM migration to reduce CPU >> usage. >> A user will be able to choose between those algorithms, therefor >> compress-type >> migration parameter is added. >> >> Here are some results of performance comparison zstd vs gzip: > Please, could you comment on the series: > > [PATCH v3 00/21] Multifd Migration Compression > > That series integrated zstd/zlib compression on top of multifd, > advantages over "old" compression code are: > - We don't have to copy data back and forth > - The unit of compression is 512KB instead of 4kb > - We "conserve" the compression state between packets (this is specially >interesting for zstd, that uses dictionaries) > >> host: i7-4790 8xCPU @ 3.60GHz, 16G RAM >> migration to the same host >> VM: 2xVCPU, 8G RAM total >> 5G RAM used, memory populated with postgreqsl data >> produced by pgbench performance benchmark >> >> >> Threads: 1 compress – 1 decompress >> >> zstd provides slightly less compression ratio with almost the same >> CPU usage but copes with RAM compression roghly 2 times faster >> >> compression type zlib | zstd >> - >> compression level 1 5 | 1 5 >> compression ratio 6.927.05 | 6.696.89 >> cpu idle, %82 83| 86 80 >> time, sec 49 71| 26 31 >> time diff to zlib, sec -25 -41 >> >> >> Threads: 8 compress – 2 decompress >> >> zstd provides the same migration time with less cpu consumption >> >> compression type none |gzip(zlib)| zstd >> -- >> compression level- | 1 5 9| 1 5 15 >> compression ratio- | 6.94 6.997.14 | 6.646.89 >> 6.93 >> time, sec154 | 22 23 27 | 23 23 25 >> cpu idle, % 99| 45 30 12 | 70 52 23 >> cpu idle diff to zlib | | -25%-22%-11% > I don't have handy results, but it looked for me like you: > - zstd has a way better latency than zlib (i.e. the packet cames sooner) > - And it compress much better > > On the migration test (best possible case for a compressor, as we are > writting just one byte of each page, and we write the same value in all > pages): > > - zlib: compress 512KB -> 2500 bytes > - zstd: compess 512KB -> 52 bytes (yeap, I tested several times, it >looked too small) > > See that I posted another patch to "delete" the old compression code. > Why? > - I have been unable to modify migration-test to test it and work >reliablely (only way was to allow a really huge downtime) > - Even with slow networking (1Gigabit) I got really mixed results, >because as it is so slow, the guest continue dirtying memory, and in >my tests it was never a winner > > Thanks, Juan. >
[PATCH v1] virtio-mmio: update queue size on guest write
Some guests read back queue size after writing it.
Always update the on size write otherwise they might be confused.
Signed-off-by: Denis Plotnikov
---
hw/virtio/virtio-mmio.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/hw/virtio/virtio-mmio.c b/hw/virtio/virtio-mmio.c
index 94d934c44b..1e40a74869 100644
--- a/hw/virtio/virtio-mmio.c
+++ b/hw/virtio/virtio-mmio.c
@@ -295,8 +295,9 @@ static void virtio_mmio_write(void *opaque, hwaddr offset,
uint64_t value,
break;
case VIRTIO_MMIO_QUEUE_NUM:
trace_virtio_mmio_queue_write(value, VIRTQUEUE_MAX_SIZE);
+virtio_queue_set_num(vdev, vdev->queue_sel, value);
+
if (proxy->legacy) {
-virtio_queue_set_num(vdev, vdev->queue_sel, value);
virtio_queue_update_rings(vdev, vdev->queue_sel);
} else {
proxy->vqs[vdev->queue_sel].num = value;
--
2.17.0
Re: [PATCH v1] virtio-pci: store virtqueue size directly to a device
On 23.12.2019 17:31, Michael S. Tsirkin wrote:
> On Mon, Dec 23, 2019 at 02:37:58PM +0300, Denis Plotnikov wrote:
>> Currenly, the virtqueue size is saved to the proxy on pci writing and
>> is read from the device pci reading.
>> The virtqueue size is propagated later on form the proxy to the device
>> on virqueue enabling stage.
>>
>> This could be a problem, if a guest, on the virtqueue configuration, sets
>> the size and then re-read it immediatly before the queue enabling
>> in order to check if the desiged size has been set.
>>
>> This happens in seabios: (sebios snippet)
>>
>> vp_find_vq()
>> {
>> ...
>> /* check if the queue is available */
>> if (vp->use_modern) {
>> num = vp_read(>common, virtio_pci_common_cfg, queue_size);
>> if (num > MAX_QUEUE_NUM) {
>> vp_write(>common, virtio_pci_common_cfg, queue_size,
>> MAX_QUEUE_NUM);
>> num = vp_read(>common, virtio_pci_common_cfg, queue_size);
>> }
>> } else {
>> num = vp_read(>legacy, virtio_pci_legacy, queue_num);
>> }
>> if (!num) {
>> dprintf(1, "ERROR: queue size is 0\n");
>> goto fail;
>> }
>> if (num > MAX_QUEUE_NUM) {
>> dprintf(1, "ERROR: queue size %d > %d\n", num, MAX_QUEUE_NUM);
>> goto fail;
>> }
>> ...
>> }
>>
>> If the device queue num is greater then the max queue size supported by
>> seabios,
>> seabios tries to reduce the queue size, then re-read it again, I suppose to
>> check if the setting actually happens, and then checks the virtqueue size
>> again,
>> to deside whether it is satisfied with the vaule.
>> In this case, if device's virtqueue size is 512 and seabios max supported
>> queue
>> size is 256, seabios tries to set 256 but than read 512 again and can't
>> proceed
>> with that vaule, preventing the guest from successful booting.
>> The root case was investigated by Roman Kagan
>>
>> The patch fixes the problem, by propagating the queue size to the device
>> right
>> away, so the written value could be read on the next step, if the value was
>> ok for the device.
>>
>> Suggested-by: Roman Kagan
>> Suggested-by: Michael S. Tsirkin
>> Signed-off-by: Denis Plotnikov
> Thanks, I already have this queued as:
>
> commit 8aabbbd9d04f95d5581d2275362996ecb5516dd9
> Author: Michael S. Tsirkin
> Date: Fri Dec 13 09:22:48 2019 -0500
>
> virtio: update queue size on guest write
>
> Some guests read back queue size after writing it.
> Update the size immediatly upon write otherwise
> they get confused.
>
> Signed-off-by: Michael S. Tsirkin
>
> I would appreciate checking other transports, they likely
> need the same fix.
ok, I'll send the patch shortly
>
>
>> ---
>> hw/virtio/virtio-pci.c | 2 ++
>> 1 file changed, 2 insertions(+)
>>
>> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
>> index c6b47a9c73..e5c759e19e 100644
>> --- a/hw/virtio/virtio-pci.c
>> +++ b/hw/virtio/virtio-pci.c
>> @@ -1256,6 +1256,8 @@ static void virtio_pci_common_write(void *opaque,
>> hwaddr addr,
>> break;
>> case VIRTIO_PCI_COMMON_Q_SIZE:
>> proxy->vqs[vdev->queue_sel].num = val;
>> +virtio_queue_set_num(vdev, vdev->queue_sel,
>> + proxy->vqs[vdev->queue_sel].num);
>> break;
>> case VIRTIO_PCI_COMMON_Q_MSIX:
>> msix_vector_unuse(>pci_dev,
>> --
>> 2.17.0
[PATCH v1] virtio: stregthen virtqueue size invariants
1. virtqueue_size is a power of 2
2. virtqueue_size > 2, since seg_max is virtqueue_size - 2
Signed-off-by: Denis Plotnikov
---
hw/virtio/virtio.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
index 04716b5f6c..e3ab69061e 100644
--- a/hw/virtio/virtio.c
+++ b/hw/virtio/virtio.c
@@ -2166,7 +2166,8 @@ void virtio_queue_set_num(VirtIODevice *vdev, int n, int
num)
*/
if (!!num != !!vdev->vq[n].vring.num ||
num > VIRTQUEUE_MAX_SIZE ||
-num < 0) {
+num < 2 ||
+!is_power_of_2(num)) {
return;
}
vdev->vq[n].vring.num = num;
--
2.17.0
[PATCH v1] virtio-pci: store virtqueue size directly to a device
Currenly, the virtqueue size is saved to the proxy on pci writing and
is read from the device pci reading.
The virtqueue size is propagated later on form the proxy to the device
on virqueue enabling stage.
This could be a problem, if a guest, on the virtqueue configuration, sets
the size and then re-read it immediatly before the queue enabling
in order to check if the desiged size has been set.
This happens in seabios: (sebios snippet)
vp_find_vq()
{
...
/* check if the queue is available */
if (vp->use_modern) {
num = vp_read(>common, virtio_pci_common_cfg, queue_size);
if (num > MAX_QUEUE_NUM) {
vp_write(>common, virtio_pci_common_cfg, queue_size,
MAX_QUEUE_NUM);
num = vp_read(>common, virtio_pci_common_cfg, queue_size);
}
} else {
num = vp_read(>legacy, virtio_pci_legacy, queue_num);
}
if (!num) {
dprintf(1, "ERROR: queue size is 0\n");
goto fail;
}
if (num > MAX_QUEUE_NUM) {
dprintf(1, "ERROR: queue size %d > %d\n", num, MAX_QUEUE_NUM);
goto fail;
}
...
}
If the device queue num is greater then the max queue size supported by seabios,
seabios tries to reduce the queue size, then re-read it again, I suppose to
check if the setting actually happens, and then checks the virtqueue size again,
to deside whether it is satisfied with the vaule.
In this case, if device's virtqueue size is 512 and seabios max supported queue
size is 256, seabios tries to set 256 but than read 512 again and can't proceed
with that vaule, preventing the guest from successful booting.
The root case was investigated by Roman Kagan
The patch fixes the problem, by propagating the queue size to the device right
away, so the written value could be read on the next step, if the value was
ok for the device.
Suggested-by: Roman Kagan
Suggested-by: Michael S. Tsirkin
Signed-off-by: Denis Plotnikov
---
hw/virtio/virtio-pci.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index c6b47a9c73..e5c759e19e 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -1256,6 +1256,8 @@ static void virtio_pci_common_write(void *opaque, hwaddr
addr,
break;
case VIRTIO_PCI_COMMON_Q_SIZE:
proxy->vqs[vdev->queue_sel].num = val;
+virtio_queue_set_num(vdev, vdev->queue_sel,
+ proxy->vqs[vdev->queue_sel].num);
break;
case VIRTIO_PCI_COMMON_Q_MSIX:
msix_vector_unuse(>pci_dev,
--
2.17.0
[PATCH v1] hw: fix using 4.2 compat in 5.0 machine types for i440fx/q35
5.0 machine type uses 4.2 compats. This seems to be incorrect, since the latests machine type by now is 5.0 and it should use its own compat or shouldn't use any relying on the defaults. Seems, like this appeared because of some problems on merge/rebase. Signed-off-by: Denis Plotnikov --- hw/i386/pc_piix.c | 1 - hw/i386/pc_q35.c | 1 - 2 files changed, 2 deletions(-) diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c index ffb30c32ce..846e70bc55 100644 --- a/hw/i386/pc_piix.c +++ b/hw/i386/pc_piix.c @@ -431,7 +431,6 @@ static void pc_i440fx_5_0_machine_options(MachineClass *m) m->alias = "pc"; m->is_default = 1; pcmc->default_cpu_version = 1; -compat_props_add(m->compat_props, hw_compat_4_2, hw_compat_4_2_len); } DEFINE_I440FX_MACHINE(v5_0, "pc-i440fx-5.0", NULL, diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c index 7398d7baa2..ddd485d608 100644 --- a/hw/i386/pc_q35.c +++ b/hw/i386/pc_q35.c @@ -354,7 +354,6 @@ static void pc_q35_5_0_machine_options(MachineClass *m) pc_q35_machine_options(m); m->alias = "q35"; pcmc->default_cpu_version = 1; -compat_props_add(m->compat_props, hw_compat_4_2, hw_compat_4_2_len); } DEFINE_Q35_MACHINE(v5_0, "pc-q35-5.0", NULL, -- 2.17.0
Re: [PATCH v5 0/2] virtio: make seg_max virtqueue size dependent
PLEASE, IGNORE THIS PATCH SET On 20.12.2019 17:04, Denis Plotnikov wrote: > v5: >* rebased on the recent master [MST] >* NOTE: the test doesn't pass because 5.0 machine type use 4.2 compat >instead of it's own or no compat at all. The test will pass >once the new 5.0 compat is used. > > v4: >* rebased on 4.2 [MST] > > v3: >* add property to set in machine type [MST] >* add min queue size check [Stefan] >* add avocado based test [Max, Stefan, Eduardo, Cleber] > > v2: >* the standalone patch to make seg_max virtqueue size dependent >* other patches are postponed > > v1: >the initial series > > Denis Plotnikov (2): >virtio: make seg_max virtqueue size dependent >tests: add virtio-scsi and virtio-blk seg_max_adjust test > > hw/block/virtio-blk.c | 9 +- > hw/core/machine.c | 3 + > hw/scsi/vhost-scsi.c | 2 + > hw/scsi/virtio-scsi.c | 10 +- > include/hw/virtio/virtio-blk.h| 1 + > include/hw/virtio/virtio-scsi.h | 1 + > tests/acceptance/virtio_seg_max_adjust.py | 134 ++ > 7 files changed, 158 insertions(+), 2 deletions(-) > create mode 100755 tests/acceptance/virtio_seg_max_adjust.py >
[PATCH v5 1/2] virtio: make seg_max virtqueue size dependent
Before the patch, seg_max parameter was immutable and hardcoded
to 126 (128 - 2) without respect to queue size. This has two negative effects:
1. when queue size is < 128, we have Virtio 1.1 specfication violation:
(2.6.5.3.1 Driver Requirements) seq_max must be <= queue_size.
This violation affects the old Linux guests (ver < 4.14). These guests
crash on these queue_size setups.
2. when queue_size > 128, as was pointed out by Denis Lunev
,
seg_max restrics guest's block request length which affects guests'
performance making them issues more block request than needed.
https://lists.gnu.org/archive/html/qemu-devel/2017-12/msg03721.html
To mitigate this two effects, the patch adds the property adjusting seg_max
to queue size automaticaly. Since seg_max is a guest visible parameter,
the property is machine type managable and allows to choose between
old (seg_max = 126 always) and new (seg_max = queue_size - 2) behaviors.
Not to change the behavior of the older VMs, prevent setting the default
seg_max_adjust value for older machine types.
Reviewed-by: Stefan Hajnoczi
Signed-off-by: Denis Plotnikov
---
hw/block/virtio-blk.c | 9 -
hw/core/machine.c | 3 +++
hw/scsi/vhost-scsi.c| 2 ++
hw/scsi/virtio-scsi.c | 10 +-
include/hw/virtio/virtio-blk.h | 1 +
include/hw/virtio/virtio-scsi.h | 1 +
6 files changed, 24 insertions(+), 2 deletions(-)
diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
index d62e6377c2..0f6f8113b7 100644
--- a/hw/block/virtio-blk.c
+++ b/hw/block/virtio-blk.c
@@ -908,7 +908,8 @@ static void virtio_blk_update_config(VirtIODevice *vdev,
uint8_t *config)
blk_get_geometry(s->blk, );
memset(, 0, sizeof(blkcfg));
virtio_stq_p(vdev, , capacity);
-virtio_stl_p(vdev, _max, 128 - 2);
+virtio_stl_p(vdev, _max,
+ s->conf.seg_max_adjust ? s->conf.queue_size - 2 : 128 - 2);
virtio_stw_p(vdev, , conf->cyls);
virtio_stl_p(vdev, _size, blk_size);
virtio_stw_p(vdev, _io_size, conf->min_io_size / blk_size);
@@ -1133,6 +1134,11 @@ static void virtio_blk_device_realize(DeviceState *dev,
Error **errp)
error_setg(errp, "num-queues property must be larger than 0");
return;
}
+if (conf->queue_size <= 2) {
+error_setg(errp, "invalid queue-size property (%" PRIu16 "), "
+ "must be > 2", conf->queue_size);
+return;
+}
if (!is_power_of_2(conf->queue_size) ||
conf->queue_size > VIRTQUEUE_MAX_SIZE) {
error_setg(errp, "invalid queue-size property (%" PRIu16 "), "
@@ -1262,6 +1268,7 @@ static Property virtio_blk_properties[] = {
true),
DEFINE_PROP_UINT16("num-queues", VirtIOBlock, conf.num_queues, 1),
DEFINE_PROP_UINT16("queue-size", VirtIOBlock, conf.queue_size, 128),
+DEFINE_PROP_BOOL("seg-max-adjust", VirtIOBlock, conf.seg_max_adjust, true),
DEFINE_PROP_LINK("iothread", VirtIOBlock, conf.iothread, TYPE_IOTHREAD,
IOThread *),
DEFINE_PROP_BIT64("discard", VirtIOBlock, host_features,
diff --git a/hw/core/machine.c b/hw/core/machine.c
index 023548b4f3..bfa320387e 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -29,6 +29,9 @@
GlobalProperty hw_compat_4_2[] = {
{ "virtio-blk-device", "x-enable-wce-if-config-wce", "off" },
+{ "virtio-blk-device", "seg-max-adjust", "off"},
+{ "virtio-scsi-device", "seg_max_adjust", "off"},
+{ "vhost-blk-device", "seg_max_adjust", "off"},
};
const size_t hw_compat_4_2_len = G_N_ELEMENTS(hw_compat_4_2);
diff --git a/hw/scsi/vhost-scsi.c b/hw/scsi/vhost-scsi.c
index c693fc748a..26f710d3ec 100644
--- a/hw/scsi/vhost-scsi.c
+++ b/hw/scsi/vhost-scsi.c
@@ -275,6 +275,8 @@ static Property vhost_scsi_properties[] = {
DEFINE_PROP_UINT32("num_queues", VirtIOSCSICommon, conf.num_queues, 1),
DEFINE_PROP_UINT32("virtqueue_size", VirtIOSCSICommon, conf.virtqueue_size,
128),
+DEFINE_PROP_BOOL("seg_max_adjust", VirtIOSCSICommon, conf.seg_max_adjust,
+ true),
DEFINE_PROP_UINT32("max_sectors", VirtIOSCSICommon, conf.max_sectors,
0x),
DEFINE_PROP_UINT32("cmd_per_lun", VirtIOSCSICommon, conf.cmd_per_lun, 128),
diff --git a/hw/scsi/virtio-scsi.c b/hw/scsi/virtio-scsi.c
index e8b2b64d09..405cb6c953 100644
--- a/hw/scsi/virtio-scsi.c
+++ b/hw/scsi/virtio-scsi.c
@@ -654,7 +654,8 @@ static void virtio_scsi_get_config(VirtIODevice *vdev,
VirtIOSCSICommon *s = VIRTIO_SCSI_COMMON(vdev);
virtio_stl_p(vdev, >num_queues, s->conf.num_queu
[PATCH v5 0/2] virtio: make seg_max virtqueue size dependent
v5: * rebased on the recent master [MST] * NOTE: the test doesn't pass because 5.0 machine type use 4.2 compat instead of it's own or no compat at all. The test will pass once the new 5.0 compat is used. v4: * rebased on 4.2 [MST] v3: * add property to set in machine type [MST] * add min queue size check [Stefan] * add avocado based test [Max, Stefan, Eduardo, Cleber] v2: * the standalone patch to make seg_max virtqueue size dependent * other patches are postponed v1: the initial series Denis Plotnikov (2): virtio: make seg_max virtqueue size dependent tests: add virtio-scsi and virtio-blk seg_max_adjust test hw/block/virtio-blk.c | 9 +- hw/core/machine.c | 3 + hw/scsi/vhost-scsi.c | 2 + hw/scsi/virtio-scsi.c | 10 +- include/hw/virtio/virtio-blk.h| 1 + include/hw/virtio/virtio-scsi.h | 1 + tests/acceptance/virtio_seg_max_adjust.py | 134 ++ 7 files changed, 158 insertions(+), 2 deletions(-) create mode 100755 tests/acceptance/virtio_seg_max_adjust.py -- 2.17.0
[PATCH v5 2/2] tests: add virtio-scsi and virtio-blk seg_max_adjust test
It tests proper seg_max_adjust settings for all machine types except
'none', 'isapc', 'microvm'
Signed-off-by: Denis Plotnikov
---
tests/acceptance/virtio_seg_max_adjust.py | 134 ++
1 file changed, 134 insertions(+)
create mode 100755 tests/acceptance/virtio_seg_max_adjust.py
diff --git a/tests/acceptance/virtio_seg_max_adjust.py
b/tests/acceptance/virtio_seg_max_adjust.py
new file mode 100755
index 00..5458573138
--- /dev/null
+++ b/tests/acceptance/virtio_seg_max_adjust.py
@@ -0,0 +1,134 @@
+#!/usr/bin/env python
+#
+# Test virtio-scsi and virtio-blk queue settings for all machine types
+#
+# Copyright (c) 2019 Virtuozzo International GmbH
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+import sys
+import os
+import re
+
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
+from qemu.machine import QEMUMachine
+from avocado_qemu import Test
+
+#list of machine types and virtqueue properties to test
+VIRTIO_SCSI_PROPS = {'seg_max_adjust': 'seg_max_adjust'}
+VIRTIO_BLK_PROPS = {'seg_max_adjust': 'seg-max-adjust'}
+
+DEV_TYPES = {'virtio-scsi-pci': VIRTIO_SCSI_PROPS,
+ 'virtio-blk-pci': VIRTIO_BLK_PROPS}
+
+VM_DEV_PARAMS = {'virtio-scsi-pci': ['-device', 'virtio-scsi-pci,id=scsi0'],
+ 'virtio-blk-pci': ['-device',
+'virtio-blk-pci,id=scsi0,drive=drive0',
+'-drive',
+'driver=null-co,id=drive0,if=none']}
+
+
+class VirtioMaxSegSettingsCheck(Test):
+@staticmethod
+def make_pattern(props):
+pattern_items = ['{0} = \w+'.format(prop) for prop in props]
+return '|'.join(pattern_items)
+
+def query_virtqueue(self, vm, dev_type_name):
+query_ok = False
+error = None
+props = None
+
+output = vm.command('human-monitor-command',
+command_line = 'info qtree')
+props_list = DEV_TYPES[dev_type_name].values();
+pattern = self.make_pattern(props_list)
+res = re.findall(pattern, output)
+
+if len(res) != len(props_list):
+props_list = set(props_list)
+res = set(res)
+not_found = props_list.difference(res)
+not_found = ', '.join(not_found)
+error = '({0}): The following properties not found: {1}'\
+ .format(dev_type_name, not_found)
+else:
+query_ok = True
+props = dict()
+for prop in res:
+p = prop.split(' = ')
+props[p[0]] = p[1]
+return query_ok, props, error
+
+def check_mt(self, mt, dev_type_name):
+with QEMUMachine(self.qemu_bin) as vm:
+vm.set_machine(mt["name"])
+for s in VM_DEV_PARAMS[dev_type_name]:
+vm.add_args(s)
+vm.launch()
+query_ok, props, error = self.query_virtqueue(vm, dev_type_name)
+
+if not query_ok:
+self.fail('machine type {0}: {1}'.format(mt['name'], error))
+
+for prop_name, prop_val in props.items():
+expected_val = mt[prop_name]
+self.assertEqual(expected_val, prop_val)
+
+@staticmethod
+def seg_max_adjust_enabled(mt):
+# machine types >= 5.0 should have seg_max_adjust = true
+# others seg_max_adjust = false
+mt = mt.split("-")
+
+# machine types with one line name and name like pc-x.x
+if len(mt) <= 2:
+return False
+
+# machine types like pc--x.x[.x]
+ver = mt[2]
+ver = ver.split(".");
+
+# versions >= 5.0 goes with seg_max_adjust enabled
+major = int(ver[0])
+
+if major >= 5:
+return True
+return False
+
+def test_machine_types(self):
+# collect all machine types except 'none', 'isapc', 'microvm'
+with QEMUMachine(self.qemu_bin) as vm:
+vm.launch()
+machines = [m['name'] for m in vm.command('query-machines')]
+vm.shutdown()
+machines.remove('none')
+machines.remove('isapc')
+machines.remove('microvm')
+
+for dev_type in DEV_TYPES:
+# create the list of machine types and their parameters.
+
[PATCH v5 0/2] virtio: make seg_max virtqueue size dependent
v5: * rebased on the recent master [MST] * NOTE: the test doesn't pass because 5.0 machine type use 4.2 compat instead of it's own or no compat at all. The test will pass once the new 5.0 compat is used. v4: * rebased on 4.2 [MST] v3: * add property to set in machine type [MST] * add min queue size check [Stefan] * add avocado based test [Max, Stefan, Eduardo, Cleber] v2: * the standalone patch to make seg_max virtqueue size dependent * other patches are postponed v1: the initial series Denis Plotnikov (2): virtio: make seg_max virtqueue size dependent tests: add virtio-scsi and virtio-blk seg_max_adjust test hw/block/virtio-blk.c | 9 +- hw/core/machine.c | 3 + hw/scsi/vhost-scsi.c | 2 + hw/scsi/virtio-scsi.c | 10 +- include/hw/virtio/virtio-blk.h| 1 + include/hw/virtio/virtio-scsi.h | 1 + tests/acceptance/virtio_seg_max_adjust.py | 134 ++ 7 files changed, 158 insertions(+), 2 deletions(-) create mode 100755 tests/acceptance/virtio_seg_max_adjust.py -- 2.17.0
[PATCH v5 1/2] virtio: make seg_max virtqueue size dependent
Before the patch, seg_max parameter was immutable and hardcoded
to 126 (128 - 2) without respect to queue size. This has two negative effects:
1. when queue size is < 128, we have Virtio 1.1 specfication violation:
(2.6.5.3.1 Driver Requirements) seq_max must be <= queue_size.
This violation affects the old Linux guests (ver < 4.14). These guests
crash on these queue_size setups.
2. when queue_size > 128, as was pointed out by Denis Lunev
,
seg_max restrics guest's block request length which affects guests'
performance making them issues more block request than needed.
https://lists.gnu.org/archive/html/qemu-devel/2017-12/msg03721.html
To mitigate this two effects, the patch adds the property adjusting seg_max
to queue size automaticaly. Since seg_max is a guest visible parameter,
the property is machine type managable and allows to choose between
old (seg_max = 126 always) and new (seg_max = queue_size - 2) behaviors.
Not to change the behavior of the older VMs, prevent setting the default
seg_max_adjust value for older machine types.
Reviewed-by: Stefan Hajnoczi
Signed-off-by: Denis Plotnikov
---
hw/block/virtio-blk.c | 9 -
hw/core/machine.c | 3 +++
hw/scsi/vhost-scsi.c| 2 ++
hw/scsi/virtio-scsi.c | 10 +-
include/hw/virtio/virtio-blk.h | 1 +
include/hw/virtio/virtio-scsi.h | 1 +
6 files changed, 24 insertions(+), 2 deletions(-)
diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
index d62e6377c2..0f6f8113b7 100644
--- a/hw/block/virtio-blk.c
+++ b/hw/block/virtio-blk.c
@@ -908,7 +908,8 @@ static void virtio_blk_update_config(VirtIODevice *vdev,
uint8_t *config)
blk_get_geometry(s->blk, );
memset(, 0, sizeof(blkcfg));
virtio_stq_p(vdev, , capacity);
-virtio_stl_p(vdev, _max, 128 - 2);
+virtio_stl_p(vdev, _max,
+ s->conf.seg_max_adjust ? s->conf.queue_size - 2 : 128 - 2);
virtio_stw_p(vdev, , conf->cyls);
virtio_stl_p(vdev, _size, blk_size);
virtio_stw_p(vdev, _io_size, conf->min_io_size / blk_size);
@@ -1133,6 +1134,11 @@ static void virtio_blk_device_realize(DeviceState *dev,
Error **errp)
error_setg(errp, "num-queues property must be larger than 0");
return;
}
+if (conf->queue_size <= 2) {
+error_setg(errp, "invalid queue-size property (%" PRIu16 "), "
+ "must be > 2", conf->queue_size);
+return;
+}
if (!is_power_of_2(conf->queue_size) ||
conf->queue_size > VIRTQUEUE_MAX_SIZE) {
error_setg(errp, "invalid queue-size property (%" PRIu16 "), "
@@ -1262,6 +1268,7 @@ static Property virtio_blk_properties[] = {
true),
DEFINE_PROP_UINT16("num-queues", VirtIOBlock, conf.num_queues, 1),
DEFINE_PROP_UINT16("queue-size", VirtIOBlock, conf.queue_size, 128),
+DEFINE_PROP_BOOL("seg-max-adjust", VirtIOBlock, conf.seg_max_adjust, true),
DEFINE_PROP_LINK("iothread", VirtIOBlock, conf.iothread, TYPE_IOTHREAD,
IOThread *),
DEFINE_PROP_BIT64("discard", VirtIOBlock, host_features,
diff --git a/hw/core/machine.c b/hw/core/machine.c
index 023548b4f3..bfa320387e 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -29,6 +29,9 @@
GlobalProperty hw_compat_4_2[] = {
{ "virtio-blk-device", "x-enable-wce-if-config-wce", "off" },
+{ "virtio-blk-device", "seg-max-adjust", "off"},
+{ "virtio-scsi-device", "seg_max_adjust", "off"},
+{ "vhost-blk-device", "seg_max_adjust", "off"},
};
const size_t hw_compat_4_2_len = G_N_ELEMENTS(hw_compat_4_2);
diff --git a/hw/scsi/vhost-scsi.c b/hw/scsi/vhost-scsi.c
index c693fc748a..26f710d3ec 100644
--- a/hw/scsi/vhost-scsi.c
+++ b/hw/scsi/vhost-scsi.c
@@ -275,6 +275,8 @@ static Property vhost_scsi_properties[] = {
DEFINE_PROP_UINT32("num_queues", VirtIOSCSICommon, conf.num_queues, 1),
DEFINE_PROP_UINT32("virtqueue_size", VirtIOSCSICommon, conf.virtqueue_size,
128),
+DEFINE_PROP_BOOL("seg_max_adjust", VirtIOSCSICommon, conf.seg_max_adjust,
+ true),
DEFINE_PROP_UINT32("max_sectors", VirtIOSCSICommon, conf.max_sectors,
0x),
DEFINE_PROP_UINT32("cmd_per_lun", VirtIOSCSICommon, conf.cmd_per_lun, 128),
diff --git a/hw/scsi/virtio-scsi.c b/hw/scsi/virtio-scsi.c
index e8b2b64d09..405cb6c953 100644
--- a/hw/scsi/virtio-scsi.c
+++ b/hw/scsi/virtio-scsi.c
@@ -654,7 +654,8 @@ static void virtio_scsi_get_config(VirtIODevice *vdev,
VirtIOSCSICommon *s = VIRTIO_SCSI_COMMON(vdev);
virtio_stl_p(vdev, >num_queues, s->conf.num_queu
[PATCH v5 2/2] tests: add virtio-scsi and virtio-blk seg_max_adjust test
It tests proper seg_max_adjust settings for all machine types except
'none', 'isapc', 'microvm'
Signed-off-by: Denis Plotnikov
---
tests/acceptance/virtio_seg_max_adjust.py | 134 ++
1 file changed, 134 insertions(+)
create mode 100755 tests/acceptance/virtio_seg_max_adjust.py
diff --git a/tests/acceptance/virtio_seg_max_adjust.py
b/tests/acceptance/virtio_seg_max_adjust.py
new file mode 100755
index 00..5458573138
--- /dev/null
+++ b/tests/acceptance/virtio_seg_max_adjust.py
@@ -0,0 +1,134 @@
+#!/usr/bin/env python
+#
+# Test virtio-scsi and virtio-blk queue settings for all machine types
+#
+# Copyright (c) 2019 Virtuozzo International GmbH
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+import sys
+import os
+import re
+
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
+from qemu.machine import QEMUMachine
+from avocado_qemu import Test
+
+#list of machine types and virtqueue properties to test
+VIRTIO_SCSI_PROPS = {'seg_max_adjust': 'seg_max_adjust'}
+VIRTIO_BLK_PROPS = {'seg_max_adjust': 'seg-max-adjust'}
+
+DEV_TYPES = {'virtio-scsi-pci': VIRTIO_SCSI_PROPS,
+ 'virtio-blk-pci': VIRTIO_BLK_PROPS}
+
+VM_DEV_PARAMS = {'virtio-scsi-pci': ['-device', 'virtio-scsi-pci,id=scsi0'],
+ 'virtio-blk-pci': ['-device',
+'virtio-blk-pci,id=scsi0,drive=drive0',
+'-drive',
+'driver=null-co,id=drive0,if=none']}
+
+
+class VirtioMaxSegSettingsCheck(Test):
+@staticmethod
+def make_pattern(props):
+pattern_items = ['{0} = \w+'.format(prop) for prop in props]
+return '|'.join(pattern_items)
+
+def query_virtqueue(self, vm, dev_type_name):
+query_ok = False
+error = None
+props = None
+
+output = vm.command('human-monitor-command',
+command_line = 'info qtree')
+props_list = DEV_TYPES[dev_type_name].values();
+pattern = self.make_pattern(props_list)
+res = re.findall(pattern, output)
+
+if len(res) != len(props_list):
+props_list = set(props_list)
+res = set(res)
+not_found = props_list.difference(res)
+not_found = ', '.join(not_found)
+error = '({0}): The following properties not found: {1}'\
+ .format(dev_type_name, not_found)
+else:
+query_ok = True
+props = dict()
+for prop in res:
+p = prop.split(' = ')
+props[p[0]] = p[1]
+return query_ok, props, error
+
+def check_mt(self, mt, dev_type_name):
+with QEMUMachine(self.qemu_bin) as vm:
+vm.set_machine(mt["name"])
+for s in VM_DEV_PARAMS[dev_type_name]:
+vm.add_args(s)
+vm.launch()
+query_ok, props, error = self.query_virtqueue(vm, dev_type_name)
+
+if not query_ok:
+self.fail('machine type {0}: {1}'.format(mt['name'], error))
+
+for prop_name, prop_val in props.items():
+expected_val = mt[prop_name]
+self.assertEqual(expected_val, prop_val)
+
+@staticmethod
+def seg_max_adjust_enabled(mt):
+# machine types >= 5.0 should have seg_max_adjust = true
+# others seg_max_adjust = false
+mt = mt.split("-")
+
+# machine types with one line name and name like pc-x.x
+if len(mt) <= 2:
+return False
+
+# machine types like pc--x.x[.x]
+ver = mt[2]
+ver = ver.split(".");
+
+# versions >= 5.0 goes with seg_max_adjust enabled
+major = int(ver[0])
+
+if major >= 5:
+return True
+return False
+
+def test_machine_types(self):
+# collect all machine types except 'none', 'isapc', 'microvm'
+with QEMUMachine(self.qemu_bin) as vm:
+vm.launch()
+machines = [m['name'] for m in vm.command('query-machines')]
+vm.shutdown()
+machines.remove('none')
+machines.remove('isapc')
+machines.remove('microvm')
+
+for dev_type in DEV_TYPES:
+# create the list of machine types and their parameters.
+
Re: [PATCH v0 2/2] block: allow to set 'drive' property on a realized block device
On 16.12.2019 18:38, Kevin Wolf wrote: > Am 16.12.2019 um 15:51 hat Denis Plotnikov geschrieben: >> On 13.12.2019 13:32, Kevin Wolf wrote: >>> Am 18.11.2019 um 11:50 hat Denis Plotnikov geschrieben: >>>> Another problem here, is that the "size" of the device dev may not match >>>> after setting a drive. >>>> So, we should update it after the drive setting. >>>> It was found, that it could be done by calling >>>> BlockDevOps.bdrv_parent_cb_resize. >>>> >>>> But I have some concerns about doing it so. In the case of virtio scsi >>>> disk we have the following callstack >>>> >>>> bdrv_parent_cb_resize calls() -> >>>> scsi_device_report_change(dev, SENSE_CODE(CAPACITY_CHANGED)) -> >>>> virtio_scsi_change -> >>>> virtio_scsi_push_event(s, dev, >>>> VIRTIO_SCSI_T_PARAM_CHANGE, >>>> sense.asc | >>>> (sense.ascq << 8)); >>> I think the safest option for now (and which should solve the case you >>> want to address) is checking whether old and new size match and >>> returning an error otherwise. >>> >>>> virtio_scsi_change pushes the event to the guest to make the guest >>>> ask for size refreshing. If I'm not mistaken, here we can get a race >>>> condition when some another request is processed with an unchanged >>>> size and then the size changing request is processed. >>> I think this is actually a problem even without resizing: We need to >>> quiesce the device between removing the old root and inserting the new >>> one. They way to achieve this is probably by splitting blk_drain() into >>> a blk_drain_begin()/end() and then draining the BlockBackend here while >>> we're working on it. >>> >>> Kevin >> Why don't we use bdrv_drained_begin/end directly? This is what >> blk_drain does. >> If we want to split blk_drain we must keep track if blk's brdv isn't >> change otherwise we can end up with drain_begin one and drain end >> another bdrv if we do remove/insert in between. > Hmm, true, we would have to keep track of draining at the BlockBackend > level and consider it in blk_remove_bs() and blk_insert_bs(). Maybe > that's not worth it. > > If we use bdrv_drained_begin/end directly, I think we need to drain both > the old and the new root node during the process. > >> Another thing is should we really care about this if we have VM >> stopped and the sizes matched? > How do we know that the VM is stopped? And why would we require this? I implied the scenario of VM migration over a shared storage with an exclusive file access model. The VM is stopped on drive changing phase. If there is no use to require it, than ok. Denis > Your patch doesn't implement or at least check this, and it seems a bit > impractical for example when all you want is inserting a filter node. > > Kevin
Re: [PATCH v0 2/2] block: allow to set 'drive' property on a realized block device
On 13.12.2019 13:32, Kevin Wolf wrote:
> Am 18.11.2019 um 11:50 hat Denis Plotnikov geschrieben:
>>
>> On 10.11.2019 22:08, Denis Plotnikov wrote:
>>> On 10.11.2019 22:03, Denis Plotnikov wrote:
>>>> This allows to change (replace) the file on a block device and is useful
>>>> to workaround exclusive file access restrictions, e.g. to implement VM
>>>> migration with a shared disk stored on some storage with the exclusive
>>>> file opening model: a destination VM is started waiting for incomming
>>>> migration with a fake image drive, and later, on the last migration
>>>> phase, the fake image file is replaced with the real one.
>>>>
>>>> Signed-off-by: Denis Plotnikov
>>>> ---
>>>> hw/core/qdev-properties-system.c | 89 +++-
>>>> 1 file changed, 77 insertions(+), 12 deletions(-)
>>>>
>>>> diff --git a/hw/core/qdev-properties-system.c
>>>> b/hw/core/qdev-properties-system.c
>>>> index c534590dcd..aaab1370a4 100644
>>>> --- a/hw/core/qdev-properties-system.c
>>>> +++ b/hw/core/qdev-properties-system.c
>>>> @@ -79,8 +79,55 @@ static void set_pointer(Object *obj, Visitor *v,
>>>> Property *prop,
>>>> /* --- drive --- */
>>>> -static void do_parse_drive(DeviceState *dev, const char *str, void
>>>> **ptr,
>>>> - const char *propname, bool iothread,
>>>> Error **errp)
>>>> +static void do_parse_drive_realized(DeviceState *dev, const char *str,
>>>> + void **ptr, const char *propname,
>>>> + bool iothread, Error **errp)
>>>> +{
>>>> + BlockBackend *blk = *ptr;
>>>> + BlockDriverState *bs = bdrv_lookup_bs(NULL, str, NULL);
>>>> + int ret;
>>>> + bool blk_created = false;
>>>> +
>>>> + if (!bs) {
>>>> + error_setg(errp, "Can't find blockdev '%s'", str);
>>>> + return;
>>>> + }
>>>> +
>>>> + if (!blk) {
>>>> + AioContext *ctx = iothread ? bdrv_get_aio_context(bs) :
>>>> + qemu_get_aio_context();
>>>> + blk = blk_new(ctx, BLK_PERM_ALL, BLK_PERM_ALL);
>>>> + blk_created = true;
>>> Actually, I have concerns about situation where blk=null.
>>>
>>> Is there any case when scsi-hd (or others) doesn't have a blk assigned
>>> and it's legal?
> No, block devices will always have a BlockBackend, even if it doesn't
> have a root node inserted.
>
>>>> + } else {
>>>> + if (blk_bs(blk)) {
>>>> + blk_remove_bs(blk);
>>>> + }
>>>> + }
>>>> +
>>>> + ret = blk_insert_bs(blk, bs, errp);
>>>> +
>>>> + if (!ret && blk_created) {
>>>> + if (blk_attach_dev(blk, dev) < 0) {
>>>> + /*
>>>> + * Shouldn't be any errors here since we just created
>>>> + * the new blk because the device doesn't have any.
>>>> + * Leave the message here in case blk_attach_dev is changed
>>>> + */
>>>> + error_setg(errp, "Can't attach drive '%s' to device '%s'",
>>>> + str, object_get_typename(OBJECT(dev)));
>>>> + } else {
>>>> + *ptr = blk;
>>>> + }
>>>> + }
>> Another problem here, is that the "size" of the device dev may not match
>> after setting a drive.
>> So, we should update it after the drive setting.
>> It was found, that it could be done by calling
>> BlockDevOps.bdrv_parent_cb_resize.
>>
>> But I have some concerns about doing it so. In the case of virtio scsi
>> disk we have the following callstack
>>
>> bdrv_parent_cb_resize calls() ->
>> scsi_device_report_change(dev, SENSE_CODE(CAPACITY_CHANGED)) ->
>> virtio_scsi_change ->
>> virtio_scsi_push_event(s, dev, VIRTIO_SCSI_T_PARAM_CHANGE,
>> sense.asc |
>> (sense.ascq << 8));
> I think the safest option for now (and which should solve the case you
> want to address) is checking whether old and new size match and
> returning an error otherwise.
>
>> virtio_scsi_change pushes the event to the guest to make the guest
>> ask for size refreshing. If I'm not mistaken, here we can get a race
>> condition when some another request is processed with an unchanged
>> size and then the size changing request is processed.
> I think this is actually a problem even without resizing: We need to
> quiesce the device between removing the old root and inserting the new
> one. They way to achieve this is probably by splitting blk_drain() into
> a blk_drain_begin()/end() and then draining the BlockBackend here while
> we're working on it.
>
> Kevin
Why don't we use bdrv_drained_begin/end directly? This is what blk_drain
does.
If we want to split blk_drain we must keep track if blk's brdv isn't
change otherwise we can end up with drain_begin one and drain end
another bdrv if we do remove/insert in between.
Another thing is should we really care about this if we have VM stopped
and the sizes matched?
Denis
>
[PATCH v4 1/2] virtio: make seg_max virtqueue size dependent
Before the patch, seg_max parameter was immutable and hardcoded
to 126 (128 - 2) without respect to queue size. This has two negative effects:
1. when queue size is < 128, we have Virtio 1.1 specfication violation:
(2.6.5.3.1 Driver Requirements) seq_max must be <= queue_size.
This violation affects the old Linux guests (ver < 4.14). These guests
crash on these queue_size setups.
2. when queue_size > 128, as was pointed out by Denis Lunev
,
seg_max restrics guest's block request length which affects guests'
performance making them issues more block request than needed.
https://lists.gnu.org/archive/html/qemu-devel/2017-12/msg03721.html
To mitigate this two effects, the patch adds the property adjusting seg_max
to queue size automaticaly. Since seg_max is a guest visible parameter,
the property is machine type managable and allows to choose between
old (seg_max = 126 always) and new (seg_max = queue_size - 2) behaviors.
Not to change the behavior of the older VMs, prevent setting the default
seg_max_adjust value for older machine types.
Signed-off-by: Denis Plotnikov
---
hw/block/virtio-blk.c | 9 -
hw/core/machine.c | 3 +++
hw/scsi/vhost-scsi.c| 2 ++
hw/scsi/virtio-scsi.c | 10 +-
include/hw/virtio/virtio-blk.h | 1 +
include/hw/virtio/virtio-scsi.h | 1 +
6 files changed, 24 insertions(+), 2 deletions(-)
diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
index d62e6377c2..0f6f8113b7 100644
--- a/hw/block/virtio-blk.c
+++ b/hw/block/virtio-blk.c
@@ -908,7 +908,8 @@ static void virtio_blk_update_config(VirtIODevice *vdev,
uint8_t *config)
blk_get_geometry(s->blk, );
memset(, 0, sizeof(blkcfg));
virtio_stq_p(vdev, , capacity);
-virtio_stl_p(vdev, _max, 128 - 2);
+virtio_stl_p(vdev, _max,
+ s->conf.seg_max_adjust ? s->conf.queue_size - 2 : 128 - 2);
virtio_stw_p(vdev, , conf->cyls);
virtio_stl_p(vdev, _size, blk_size);
virtio_stw_p(vdev, _io_size, conf->min_io_size / blk_size);
@@ -1133,6 +1134,11 @@ static void virtio_blk_device_realize(DeviceState *dev,
Error **errp)
error_setg(errp, "num-queues property must be larger than 0");
return;
}
+if (conf->queue_size <= 2) {
+error_setg(errp, "invalid queue-size property (%" PRIu16 "), "
+ "must be > 2", conf->queue_size);
+return;
+}
if (!is_power_of_2(conf->queue_size) ||
conf->queue_size > VIRTQUEUE_MAX_SIZE) {
error_setg(errp, "invalid queue-size property (%" PRIu16 "), "
@@ -1262,6 +1268,7 @@ static Property virtio_blk_properties[] = {
true),
DEFINE_PROP_UINT16("num-queues", VirtIOBlock, conf.num_queues, 1),
DEFINE_PROP_UINT16("queue-size", VirtIOBlock, conf.queue_size, 128),
+DEFINE_PROP_BOOL("seg-max-adjust", VirtIOBlock, conf.seg_max_adjust, true),
DEFINE_PROP_LINK("iothread", VirtIOBlock, conf.iothread, TYPE_IOTHREAD,
IOThread *),
DEFINE_PROP_BIT64("discard", VirtIOBlock, host_features,
diff --git a/hw/core/machine.c b/hw/core/machine.c
index 023548b4f3..bfa320387e 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -29,6 +29,9 @@
GlobalProperty hw_compat_4_2[] = {
{ "virtio-blk-device", "x-enable-wce-if-config-wce", "off" },
+{ "virtio-blk-device", "seg-max-adjust", "off"},
+{ "virtio-scsi-device", "seg_max_adjust", "off"},
+{ "vhost-blk-device", "seg_max_adjust", "off"},
};
const size_t hw_compat_4_2_len = G_N_ELEMENTS(hw_compat_4_2);
diff --git a/hw/scsi/vhost-scsi.c b/hw/scsi/vhost-scsi.c
index c693fc748a..26f710d3ec 100644
--- a/hw/scsi/vhost-scsi.c
+++ b/hw/scsi/vhost-scsi.c
@@ -275,6 +275,8 @@ static Property vhost_scsi_properties[] = {
DEFINE_PROP_UINT32("num_queues", VirtIOSCSICommon, conf.num_queues, 1),
DEFINE_PROP_UINT32("virtqueue_size", VirtIOSCSICommon, conf.virtqueue_size,
128),
+DEFINE_PROP_BOOL("seg_max_adjust", VirtIOSCSICommon, conf.seg_max_adjust,
+ true),
DEFINE_PROP_UINT32("max_sectors", VirtIOSCSICommon, conf.max_sectors,
0x),
DEFINE_PROP_UINT32("cmd_per_lun", VirtIOSCSICommon, conf.cmd_per_lun, 128),
diff --git a/hw/scsi/virtio-scsi.c b/hw/scsi/virtio-scsi.c
index e8b2b64d09..405cb6c953 100644
--- a/hw/scsi/virtio-scsi.c
+++ b/hw/scsi/virtio-scsi.c
@@ -654,7 +654,8 @@ static void virtio_scsi_get_config(VirtIODevice *vdev,
VirtIOSCSICommon *s = VIRTIO_SCSI_COMMON(vdev);
virtio_stl_p(vdev, >num_queues, s->conf.num_queues);
-virtio_stl_p(vdev, >seg_
[PATCH v4 2/2] tests: add virtio-scsi and virtio-blk seg_max_adjust test
It tests proper seg_max_adjust settings for all machine types except
'none', 'isapc', 'microvm'
Signed-off-by: Denis Plotnikov
---
tests/acceptance/virtio_seg_max_adjust.py | 135 ++
1 file changed, 135 insertions(+)
create mode 100755 tests/acceptance/virtio_seg_max_adjust.py
diff --git a/tests/acceptance/virtio_seg_max_adjust.py
b/tests/acceptance/virtio_seg_max_adjust.py
new file mode 100755
index 00..00cf2565d9
--- /dev/null
+++ b/tests/acceptance/virtio_seg_max_adjust.py
@@ -0,0 +1,135 @@
+#!/usr/bin/env python
+#
+# Test virtio-scsi and virtio-blk queue settings for all machine types
+#
+# Copyright (c) 2019 Virtuozzo International GmbH
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+import sys
+import os
+import re
+
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
+from qemu.machine import QEMUMachine
+from avocado_qemu import Test
+
+#list of machine types and virtqueue properties to test
+VIRTIO_SCSI_PROPS = {'seg_max_adjust': 'seg_max_adjust'}
+VIRTIO_BLK_PROPS = {'seg_max_adjust': 'seg-max-adjust'}
+
+DEV_TYPES = {'virtio-scsi-pci': VIRTIO_SCSI_PROPS,
+ 'virtio-blk-pci': VIRTIO_BLK_PROPS}
+
+VM_DEV_PARAMS = {'virtio-scsi-pci': ['-device', 'virtio-scsi-pci,id=scsi0'],
+ 'virtio-blk-pci': ['-device',
+'virtio-blk-pci,id=scsi0,drive=drive0',
+'-drive',
+'driver=null-co,id=drive0,if=none']}
+
+
+class VirtioMaxSegSettingsCheck(Test):
+@staticmethod
+def make_pattern(props):
+pattern_items = ['{0} = \w+'.format(prop) for prop in props]
+return '|'.join(pattern_items)
+
+def query_virtqueue(self, vm, dev_type_name):
+query_ok = False
+error = None
+props = None
+
+output = vm.command('human-monitor-command',
+command_line = 'info qtree')
+props_list = DEV_TYPES[dev_type_name].values();
+pattern = self.make_pattern(props_list)
+res = re.findall(pattern, output)
+
+if len(res) != len(props_list):
+props_list = set(props_list)
+res = set(res)
+not_found = props_list.difference(res)
+not_found = ', '.join(not_found)
+error = '({0}): The following properties not found: {1}'\
+ .format(dev_type_name, not_found)
+else:
+query_ok = True
+props = dict()
+for prop in res:
+p = prop.split(' = ')
+props[p[0]] = p[1]
+return query_ok, props, error
+
+def check_mt(self, mt, dev_type_name):
+with QEMUMachine(self.qemu_bin) as vm:
+vm.set_machine(mt["name"])
+for s in VM_DEV_PARAMS[dev_type_name]:
+vm.add_args(s)
+vm.launch()
+query_ok, props, error = self.query_virtqueue(vm, dev_type_name)
+
+if not query_ok:
+self.fail('machine type {0}: {1}'.format(mt['name'], error))
+
+for prop_name, prop_val in props.items():
+expected_val = mt[prop_name]
+self.assertEqual(expected_val, prop_val)
+
+@staticmethod
+def seg_max_adjust_enabled(mt):
+# machine types > 4.2 should have seg_max_adjust = true
+# others seg_max_adjust = false
+mt = mt.split("-")
+
+# machine types with one line name and name like pc-x.x
+if len(mt) <= 2:
+return False
+
+# machine types like pc--x.x[.x]
+ver = mt[2]
+ver = ver.split(".");
+
+# all versions greater than 4.2 goes with seg_max_adjust enabled
+major = int(ver[0])
+minor = int(ver[1])
+
+if major > 4 or (major == 4 and minor > 2):
+return True
+return False
+
+def test_machine_types(self):
+# collect all machine types except 'none', 'isapc', 'microvm'
+with QEMUMachine(self.qemu_bin) as vm:
+vm.launch()
+machines = [m['name'] for m in vm.command('query-machines')]
+vm.shutdown()
+machines.remove('none')
+machines.remove('isapc')
+machines.remove('microvm')
+
+for dev_type in DEV_TYPES:
+# create the list of machine
[PATCH v4 0/2] virtio: make seg_max virtqueue size dependent
v4: * rebased on 4.2 [MST] v3: * add property to set in machine type [MST] * add min queue size check [Stefan] * add avocado based test [Max, Stefan, Eduardo, Cleber] v2: * the standalone patch to make seg_max virtqueue size dependent * other patches are postponed v1: the initial series Denis Plotnikov (2): virtio: make seg_max virtqueue size dependent tests: add virtio-scsi and virtio-blk seg_max_adjust test hw/block/virtio-blk.c | 9 +- hw/core/machine.c | 3 + hw/scsi/vhost-scsi.c | 2 + hw/scsi/virtio-scsi.c | 10 +- include/hw/virtio/virtio-blk.h| 1 + include/hw/virtio/virtio-scsi.h | 1 + tests/acceptance/virtio_seg_max_adjust.py | 135 ++ 7 files changed, 159 insertions(+), 2 deletions(-) create mode 100755 tests/acceptance/virtio_seg_max_adjust.py -- 2.17.0
