Re: [Qemu-devel] [Qemu-block] [PATCH v1] [RFC] qcow2: add compression type feature
On 5/17/19 4:05 AM, Denis Plotnikov wrote: > > > On 17.05.2019 2:25, John Snow wrote: >> >> >> On 5/16/19 9:48 AM, Denis Plotnikov wrote: >>> The patch adds some preparation parts for incompatible compression type >>> feature into QCOW2 header that indicates that *all* compressed clusters >>> must be (de)compressed using a certain compression type. >>> >>> It is implied that the compression type is set on the image creation and >>> can be changed only later by image conversion, thus the only compression >>> algorithm is used for the image. >>> >>> The plan is to add support for ZSTD and then may be something more effective >>> in the future. >>> >>> ZSTD compression algorithm consumes 3-5 times less CPU power with a >>> comparable compression ratio with zlib. It would be wise to use it for >>> data compression e.g. for backups. >>> >>> The default compression is ZLIB. >>> >> >> (Merely a curiosity:) >> >> Since this is coming from Virtuozzo, I trust that you've had good luck >> with ZSTD already in R What do the compression ratios look like in >> practice? It's touted as "comparable to zlib" which certainly does sound >> quite nice for streaming compression of backups. >> >> I suppose in the worst case it ought to be faster than bandwidth speeds, >> so no harm in utilizing it. > Yes, we did some research on it. Actually, there is a patch in the > mailing list (please, take a look: ) which applies ZSTD compression in > the migration. > Here are the numbers from there: > > host: i7-4790 8xCPU @ 3.60GHz, 16G RAM > migration to the same host > VM: 2xVCPU, 8G RAM total > 5G RAM used, memory populated with postgreqsl data > produced by pgbench performance benchmark > > Threads: 1 compress – 1 decompress > > zstd provides slightly less compression ratio with almost the same > CPU usage but copes with RAM compression roughly 2 times faster > > compression type zlib | zstd > - > compression level 1 5 | 1 5 > compression ratio 6.927.05 | 6.696.89 > cpu idle, %82 83| 86 80 > time, sec 49 71| 26 31 > time diff to zlib, sec | -25 -41 > time diff to zlib, % | -47%-56% > > I general ZSTD provides better compression ratio on big dependent chunks > of data. Than bigger the data size then better ZSTD it compresses. > > Since, in our cases (migration: 4K RAM block, qcow2: cluster) we > have to compress independent chunks the ability of ZSTD to find better > compression solution is restricted. > > Although, the compression ratio is pretty much the same in both cases > the experiments shown that ZSTD does the compression much faster (x2). > > Which is obviously good for us. > > Here is other comparison of ZLIB vs ZSTD without any application in qemu > which shows that ZSTD works faster > > zlib = max compression level > zstd = compression level 5 (max 22) > > cycles consumed for compression: > > 4k txt 4K ram 4K bin 64K txt 64K ram 64K bin > zlib400K344K1.3M 13M 5M92.3M > zstd350K235K312K 3.3M 1.3M 2.4M > Diff,% -12 -32 -77 -75 -73 -97 > Wow, the 4k bin one is drastic. The text is even more prominent. wow! > size after compression in bytes: > > 4k txt 4K ram 4K bin 64K txt 64K ram 64K bin > zlib154235991403 17386 64735 20609 > zstd156835401250 17656 65546 20023 > Diff,%-2 2 11 -2-1 3 > Yeah, that's pretty close. Seems like absolutely a great tradeoff for the speed gain. If the little bit of difference matters to you, you can always do some more heavy-duty compression of your choice in another layer of the storage stack. > Data sources for test files [we took 4K and 64K chunks from there]: > > txt = linux/Documentation/memory-barriers.txt > ram = /boot/initramfs-4.20.0-rc6+.img > bin = x86_64-softmmu/qemu-system-x86_64 > > Increasing of ZSTD compression ratio didn't give any significant > improvements of the out size but slowed down the pace of ZSTD > > Denis > >> >>> Signed-off-by: Denis Plotnikov > Very useful data, thanks for sharing! Seems like this would indeed be a great thing to have for qcow2.
Re: [Qemu-devel] [Qemu-block] [PATCH v1] [RFC] qcow2: add compression type feature
On 17.05.2019 2:25, John Snow wrote: > > > On 5/16/19 9:48 AM, Denis Plotnikov wrote: >> The patch adds some preparation parts for incompatible compression type >> feature into QCOW2 header that indicates that *all* compressed clusters >> must be (de)compressed using a certain compression type. >> >> It is implied that the compression type is set on the image creation and >> can be changed only later by image conversion, thus the only compression >> algorithm is used for the image. >> >> The plan is to add support for ZSTD and then may be something more effective >> in the future. >> >> ZSTD compression algorithm consumes 3-5 times less CPU power with a >> comparable compression ratio with zlib. It would be wise to use it for >> data compression e.g. for backups. >> >> The default compression is ZLIB. >> > > (Merely a curiosity:) > > Since this is coming from Virtuozzo, I trust that you've had good luck > with ZSTD already in R What do the compression ratios look like in > practice? It's touted as "comparable to zlib" which certainly does sound > quite nice for streaming compression of backups. > > I suppose in the worst case it ought to be faster than bandwidth speeds, > so no harm in utilizing it. Yes, we did some research on it. Actually, there is a patch in the mailing list (please, take a look: ) which applies ZSTD compression in the migration. Here are the numbers from there: host: i7-4790 8xCPU @ 3.60GHz, 16G RAM migration to the same host VM: 2xVCPU, 8G RAM total 5G RAM used, memory populated with postgreqsl data produced by pgbench performance benchmark Threads: 1 compress – 1 decompress zstd provides slightly less compression ratio with almost the same CPU usage but copes with RAM compression roughly 2 times faster compression type zlib | zstd - compression level 1 5 | 1 5 compression ratio 6.927.05 | 6.696.89 cpu idle, %82 83| 86 80 time, sec 49 71| 26 31 time diff to zlib, sec | -25 -41 time diff to zlib, % | -47%-56% I general ZSTD provides better compression ratio on big dependent chunks of data. Than bigger the data size then better ZSTD it compresses. Since, in our cases (migration: 4K RAM block, qcow2: cluster) we have to compress independent chunks the ability of ZSTD to find better compression solution is restricted. Although, the compression ratio is pretty much the same in both cases the experiments shown that ZSTD does the compression much faster (x2). Which is obviously good for us. Here is other comparison of ZLIB vs ZSTD without any application in qemu which shows that ZSTD works faster zlib = max compression level zstd = compression level 5 (max 22) cycles consumed for compression: 4k txt 4K ram 4K bin 64K txt 64K ram 64K bin zlib 400K344K1.3M 13M 5M92.3M zstd 350K235K312K 3.3M 1.3M 2.4M Diff,% -12 -32 -77 -75 -73 -97 size after compression in bytes: 4k txt 4K ram 4K bin 64K txt 64K ram 64K bin zlib 154235991403 17386 64735 20609 zstd 156835401250 17656 65546 20023 Diff,% -2 2 11 -2-1 3 Data sources for test files [we took 4K and 64K chunks from there]: txt = linux/Documentation/memory-barriers.txt ram = /boot/initramfs-4.20.0-rc6+.img bin = x86_64-softmmu/qemu-system-x86_64 Increasing of ZSTD compression ratio didn't give any significant improvements of the out size but slowed down the pace of ZSTD Denis > >> Signed-off-by: Denis Plotnikov -- Best, Denis
Re: [Qemu-devel] [Qemu-block] [PATCH v1] [RFC] qcow2: add compression type feature
On 5/16/19 9:48 AM, Denis Plotnikov wrote: > The patch adds some preparation parts for incompatible compression type > feature into QCOW2 header that indicates that *all* compressed clusters > must be (de)compressed using a certain compression type. > > It is implied that the compression type is set on the image creation and > can be changed only later by image conversion, thus the only compression > algorithm is used for the image. > > The plan is to add support for ZSTD and then may be something more effective > in the future. > > ZSTD compression algorithm consumes 3-5 times less CPU power with a > comparable compression ratio with zlib. It would be wise to use it for > data compression e.g. for backups. > > The default compression is ZLIB. > (Merely a curiosity:) Since this is coming from Virtuozzo, I trust that you've had good luck with ZSTD already in R What do the compression ratios look like in practice? It's touted as "comparable to zlib" which certainly does sound quite nice for streaming compression of backups. I suppose in the worst case it ought to be faster than bandwidth speeds, so no harm in utilizing it. > Signed-off-by: Denis Plotnikov
