Re: [Qemu-devel] [PATCH 0/3] migration: add sztd compression
ping ping ping ping ping! On 18.03.2019 10:53, Denis Plotnikov wrote: > ping ping ping ping! > > On 11.03.2019 11:20, Denis Plotnikov wrote: >> ping ping ping! >> >> On 04.03.2019 18:10, Denis Plotnikov wrote: >>> ping! >>> >>> On 26.02.2019 16:15, Denis Plotnikov wrote: zstd date compression algorithm shows better performance on data compression. It might be useful to employ the algorithm in VM migration to reduce CPU usage. A user will be able to choose between those algorithms, therefor compress-type migration parameter is added. Here are some results of performance comparison zstd vs gzip: host: i7-4790 8xCPU @ 3.60GHz, 16G RAM migration to the same host VM: 2xVCPU, 8G RAM total 5G RAM used, memory populated with postgreqsl data produced by pgbench performance benchmark Threads: 1 compress – 1 decompress zstd provides slightly less compression ratio with almost the same CPU usage but copes with RAM compression roghly 2 times faster compression type zlib | zstd - compression level 1 5 | 1 5 compression ratio 6.927.05 | 6.696.89 cpu idle, %82 83| 86 80 time, sec 49 71| 26 31 time diff to zlib, sec -25 -41 Threads: 8 compress – 2 decompress zstd provides the same migration time with less cpu consumption compression type none |gzip(zlib)| zstd -- compression level- | 1 5 9| 1 5 15 compression ratio- | 6.94 6.997.14 | 6.646.89 6.93 time, sec154 | 22 23 27 | 23 23 25 cpu idle, % 99| 45 30 12 | 70 52 23 cpu idle diff to zlib | | -25%-22% -11% Denis Plotnikov (3): migration: rework compression code for adding more data compressors hmp: add compress-type parameter to migration parameters migration: add zstd compression configure | 26 hmp.c | 8 ++ migration/migration.c | 45 ++- migration/migration.h | 1 + migration/qemu-file.c | 39 ++ migration/qemu-file.h | 18 ++- migration/ram.c | 291 ++ qapi/migration.json | 26 +++- 8 files changed, 369 insertions(+), 85 deletions(-) >>> >> > -- Best, Denis
Re: [Qemu-devel] [PATCH 0/3] migration: add sztd compression
ping ping ping ping! On 11.03.2019 11:20, Denis Plotnikov wrote: > ping ping ping! > > On 04.03.2019 18:10, Denis Plotnikov wrote: >> ping! >> >> On 26.02.2019 16:15, Denis Plotnikov wrote: >>> zstd date compression algorithm shows better performance on data >>> compression. >>> It might be useful to employ the algorithm in VM migration to reduce CPU >>> usage. >>> A user will be able to choose between those algorithms, therefor >>> compress-type >>> migration parameter is added. >>> >>> Here are some results of performance comparison zstd vs gzip: >>> >>> host: i7-4790 8xCPU @ 3.60GHz, 16G RAM >>> migration to the same host >>> VM: 2xVCPU, 8G RAM total >>> 5G RAM used, memory populated with postgreqsl data >>> produced by pgbench performance benchmark >>> >>> >>> Threads: 1 compress – 1 decompress >>> >>> zstd provides slightly less compression ratio with almost the same >>> CPU usage but copes with RAM compression roghly 2 times faster >>> >>> compression type zlib | zstd >>> - >>> compression level 1 5 | 1 5 >>> compression ratio 6.927.05 | 6.696.89 >>> cpu idle, %82 83| 86 80 >>> time, sec 49 71| 26 31 >>> time diff to zlib, sec -25 -41 >>> >>> >>> Threads: 8 compress – 2 decompress >>> >>> zstd provides the same migration time with less cpu consumption >>> >>> compression type none |gzip(zlib)| zstd >>> -- >>> compression level- | 1 5 9| 1 5 15 >>> compression ratio- | 6.94 6.997.14 | 6.646.89 >>> 6.93 >>> time, sec154 | 22 23 27 | 23 23 25 >>> cpu idle, % 99| 45 30 12 | 70 52 23 >>> cpu idle diff to zlib | | -25%-22% >>> -11% >>> >>> >>> Denis Plotnikov (3): >>> migration: rework compression code for adding more data compressors >>> hmp: add compress-type parameter to migration parameters >>> migration: add zstd compression >>> >>> configure | 26 >>> hmp.c | 8 ++ >>> migration/migration.c | 45 ++- >>> migration/migration.h | 1 + >>> migration/qemu-file.c | 39 ++ >>> migration/qemu-file.h | 18 ++- >>> migration/ram.c | 291 ++ >>> qapi/migration.json | 26 +++- >>> 8 files changed, 369 insertions(+), 85 deletions(-) >>> >> > -- Best, Denis
Re: [Qemu-devel] [PATCH 0/3] migration: add sztd compression
ping ping ping! On 04.03.2019 18:10, Denis Plotnikov wrote: > ping! > > On 26.02.2019 16:15, Denis Plotnikov wrote: >> zstd date compression algorithm shows better performance on data compression. >> It might be useful to employ the algorithm in VM migration to reduce CPU >> usage. >> A user will be able to choose between those algorithms, therefor >> compress-type >> migration parameter is added. >> >> Here are some results of performance comparison zstd vs gzip: >> >> host: i7-4790 8xCPU @ 3.60GHz, 16G RAM >> migration to the same host >> VM: 2xVCPU, 8G RAM total >> 5G RAM used, memory populated with postgreqsl data >> produced by pgbench performance benchmark >> >> >> Threads: 1 compress – 1 decompress >> >> zstd provides slightly less compression ratio with almost the same >> CPU usage but copes with RAM compression roghly 2 times faster >> >> compression type zlib | zstd >> - >> compression level 1 5 | 1 5 >> compression ratio 6.927.05 | 6.696.89 >> cpu idle, %82 83| 86 80 >> time, sec 49 71| 26 31 >> time diff to zlib, sec -25 -41 >> >> >> Threads: 8 compress – 2 decompress >> >> zstd provides the same migration time with less cpu consumption >> >> compression type none |gzip(zlib)| zstd >> -- >> compression level- | 1 5 9| 1 5 15 >> compression ratio- | 6.94 6.997.14 | 6.646.89 >> 6.93 >> time, sec154 | 22 23 27 | 23 23 25 >> cpu idle, % 99| 45 30 12 | 70 52 23 >> cpu idle diff to zlib | | -25%-22%-11% >> >> >> Denis Plotnikov (3): >> migration: rework compression code for adding more data compressors >> hmp: add compress-type parameter to migration parameters >> migration: add zstd compression >> >>configure | 26 >>hmp.c | 8 ++ >>migration/migration.c | 45 ++- >>migration/migration.h | 1 + >>migration/qemu-file.c | 39 ++ >>migration/qemu-file.h | 18 ++- >>migration/ram.c | 291 ++ >>qapi/migration.json | 26 +++- >>8 files changed, 369 insertions(+), 85 deletions(-) >> > -- Best, Denis
Re: [Qemu-devel] [PATCH 0/3] migration: add sztd compression
ping ping On 04.03.2019 18:10, Denis Plotnikov wrote: > ping! > > On 26.02.2019 16:15, Denis Plotnikov wrote: >> zstd date compression algorithm shows better performance on data compression. >> It might be useful to employ the algorithm in VM migration to reduce CPU >> usage. >> A user will be able to choose between those algorithms, therefor >> compress-type >> migration parameter is added. >> >> Here are some results of performance comparison zstd vs gzip: >> >> host: i7-4790 8xCPU @ 3.60GHz, 16G RAM >> migration to the same host >> VM: 2xVCPU, 8G RAM total >> 5G RAM used, memory populated with postgreqsl data >> produced by pgbench performance benchmark >> >> >> Threads: 1 compress – 1 decompress >> >> zstd provides slightly less compression ratio with almost the same >> CPU usage but copes with RAM compression roghly 2 times faster >> >> compression type zlib | zstd >> - >> compression level 1 5 | 1 5 >> compression ratio 6.927.05 | 6.696.89 >> cpu idle, %82 83| 86 80 >> time, sec 49 71| 26 31 >> time diff to zlib, sec -25 -41 >> >> >> Threads: 8 compress – 2 decompress >> >> zstd provides the same migration time with less cpu consumption >> >> compression type none |gzip(zlib)| zstd >> -- >> compression level- | 1 5 9| 1 5 15 >> compression ratio- | 6.94 6.997.14 | 6.646.89 >> 6.93 >> time, sec154 | 22 23 27 | 23 23 25 >> cpu idle, % 99| 45 30 12 | 70 52 23 >> cpu idle diff to zlib | | -25%-22%-11% >> >> >> Denis Plotnikov (3): >> migration: rework compression code for adding more data compressors >> hmp: add compress-type parameter to migration parameters >> migration: add zstd compression >> >>configure | 26 >>hmp.c | 8 ++ >>migration/migration.c | 45 ++- >>migration/migration.h | 1 + >>migration/qemu-file.c | 39 ++ >>migration/qemu-file.h | 18 ++- >>migration/ram.c | 291 ++ >>qapi/migration.json | 26 +++- >>8 files changed, 369 insertions(+), 85 deletions(-) >> > -- Best, Denis
Re: [Qemu-devel] [PATCH 0/3] migration: add sztd compression
Patchew URL:
https://patchew.org/QEMU/20190226131535.30361-1-dplotni...@virtuozzo.com/
Hi,
This series failed the docker-mingw@fedora build test. Please find the testing
commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.
=== TEST SCRIPT BEGIN ===
#!/bin/bash
time make docker-test-mingw@fedora SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===
Configure options:
--enable-werror --target-list=x86_64-softmmu,aarch64-softmmu
--prefix=/tmp/qemu-test/install --python=/usr/bin/python3
--cross-prefix=x86_64-w64-mingw32- --enable-trace-backends=simple
--enable-gnutls --enable-nettle --enable-curl --enable-vnc --enable-bzip2
--enable-guest-agent
ERROR: zstd check failed
Make sure to have the zstd libs and headers installed.
# QEMU configure log Wed Mar 6 11:31:58 UTC 2019
---
funcs: do_compiler do_cc compile_object check_define main
lines: 92 122 619 636 0
x86_64-w64-mingw32-gcc -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE
-Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings
-Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -c -o
config-temp/qemu-conf.o config-temp/qemu-conf.c
config-temp/qemu-conf.c:2:2: error: #error __linux__ not defined
#error __linux__ not defined
^
---
funcs: do_compiler do_cc compile_object check_define main
lines: 92 122 619 688 0
x86_64-w64-mingw32-gcc -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE
-Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings
-Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -c -o
config-temp/qemu-conf.o config-temp/qemu-conf.c
config-temp/qemu-conf.c:2:2: error: #error __i386__ not defined
#error __i386__ not defined
^
---
funcs: do_compiler do_cc compile_object check_define main
lines: 92 122 619 691 0
x86_64-w64-mingw32-gcc -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE
-Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings
-Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -c -o
config-temp/qemu-conf.o config-temp/qemu-conf.c
config-temp/qemu-conf.c:2:2: error: #error __ILP32__ not defined
#error __ILP32__ not defined
^
---
lines: 92 128 923 0
x86_64-w64-mingw32-gcc -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64
-D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef
-Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv
-std=gnu99 -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -g -liberty
/usr/lib/gcc/x86_64-w64-mingw32/8.2.0/../../../../x86_64-w64-mingw32/bin/ld:
cannot find -liberty
collect2: error: ld returned 1 exit status
funcs: do_compiler do_cc compile_object main
lines: 92 122 1825 0
---
funcs: do_compiler do_cc compile_prog cc_has_warning_flag main
lines: 92 128 1906 1910 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE
-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes
-Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes
-fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Werror -Wstring-plus-int
-o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g
x86_64-w64-mingw32-gcc: error: unrecognized command line option
'-Wstring-plus-int'; did you mean '-Wstrict-aliasing'?
funcs: do_compiler do_cc compile_prog cc_has_warning_flag main
lines: 92 128 1906 1910 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE
-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes
-Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes
-fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Werror
-Winitializer-overrides -o config-temp/qemu-conf.exe config-temp/qemu-conf.c
-m64 -g
x86_64-w64-mingw32-gcc: error: unrecognized command line option
'-Winitializer-overrides'; did you mean '-Wno-suggest-override'?
funcs: do_compiler do_cc compile_prog cc_has_warning_flag main
lines: 92 128 1906 1910 0
---
lines: 92 122 2165 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE
-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes
-Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes
-fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined
-Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body
-Wnested-externs -Wformat-security -Wformat-y2k -Winit-self
-Wignored-qualifiers -Wold-style-declaration -Wold-style-definition
-Wtype-limits -fstack-protector-strong -Werror -c -o config-temp/qemu-conf.o
config-temp/qemu-conf.c
config-temp/qemu-conf.c: In function 'main':
config-temp/qemu-conf.c:4:21: error: unknown conversion type character 'z' in
format [-Werror=format=]
return printf("%zu", SIZE_MAX);
^
config-temp/qemu-conf.c:4:19: error: too many arguments for format
[-Werror=format-extra-args]
return printf("%zu", SIZE_MAX);
^
config-temp/qemu-conf.c:4:21: error: unknown
Re: [Qemu-devel] [PATCH 0/3] migration: add sztd compression
ping! On 26.02.2019 16:15, Denis Plotnikov wrote: > zstd date compression algorithm shows better performance on data compression. > It might be useful to employ the algorithm in VM migration to reduce CPU > usage. > A user will be able to choose between those algorithms, therefor compress-type > migration parameter is added. > > Here are some results of performance comparison zstd vs gzip: > > host: i7-4790 8xCPU @ 3.60GHz, 16G RAM > migration to the same host > VM: 2xVCPU, 8G RAM total > 5G RAM used, memory populated with postgreqsl data > produced by pgbench performance benchmark > > > Threads: 1 compress – 1 decompress > > zstd provides slightly less compression ratio with almost the same > CPU usage but copes with RAM compression roghly 2 times faster > > compression type zlib | zstd > - > compression level 1 5 | 1 5 > compression ratio 6.927.05 | 6.696.89 > cpu idle, %82 83| 86 80 > time, sec 49 71| 26 31 > time diff to zlib, sec -25 -41 > > > Threads: 8 compress – 2 decompress > > zstd provides the same migration time with less cpu consumption > > compression type none |gzip(zlib)| zstd > -- > compression level- | 1 5 9| 1 5 15 > compression ratio- | 6.94 6.997.14 | 6.646.896.93 > time, sec154 | 22 23 27 | 23 23 25 > cpu idle, % 99| 45 30 12 | 70 52 23 > cpu idle diff to zlib | | -25%-22%-11% > > > Denis Plotnikov (3): >migration: rework compression code for adding more data compressors >hmp: add compress-type parameter to migration parameters >migration: add zstd compression > > configure | 26 > hmp.c | 8 ++ > migration/migration.c | 45 ++- > migration/migration.h | 1 + > migration/qemu-file.c | 39 ++ > migration/qemu-file.h | 18 ++- > migration/ram.c | 291 ++ > qapi/migration.json | 26 +++- > 8 files changed, 369 insertions(+), 85 deletions(-) > -- Best, Denis
