Zstandard (zstd) [0] is a data compression algorithm, in addition to gzip, lzo for our backup/restore. It can utilize multiple core CPUs. But by default it has one compression and one writer thread.
Here some quick tests I made on my workstation. The files where placed on a ram disk. And with dd filled from /dev/urandom and /dev/zero. __Compression__ file size: 1073741824 bytes = urandom = = zero = 995ms 1073766414 328ms 98192 zstd -k 732ms 1073766414 295ms 98192 zstd -k -T4 906ms 1073791036 562ms 4894779 lzop -k 31992ms 1073915558 5594ms 1042087 gzip -k 30832ms 1074069541 5776ms 1171491 pigz -k -p 1 7814ms 1074069541 1567ms 1171491 pigz -k -p 4 __Decompression__ file size: 1073741824 bytes = urandom = = zero = 712ms 869ms zstd -d 685ms 872ms zstd -k -d -T4 841ms 2462ms lzop -d 5417ms 4754ms gzip -k -d 1248ms 3118ms pigz -k -d -p 1 1236ms 2379ms pigz -k -d -p 4 And I used the same ramdisk to move a VM onto it and run a quick backup/restore. __vzdump backup__ INFO: transferred 34359 MB in 69 seconds (497 MB/s) zstd -T1 INFO: transferred 34359 MB in 37 seconds (928 MB/s) zstd -T4 INFO: transferred 34359 MB in 51 seconds (673 MB/s) lzo INFO: transferred 34359 MB in 1083 seconds (31 MB/s) gzip INFO: transferred 34359 MB in 241 seconds (142 MB/s) pigz -n 4 __qmrestore__ progress 100% (read 34359738368 bytes, duration 36 sec) total bytes read 34359738368, sparse bytes 8005484544 (23.3%) zstd -d -T4 progress 100% (read 34359738368 bytes, duration 38 sec) total bytes read 34359738368, sparse bytes 8005484544 (23.3%) lzo progress 100% (read 34359738368 bytes, duration 175 sec) total bytes read 34359738368, sparse bytes 8005484544 (23.3%) pigz -n 4 v4 -> v5: * fixup, use File::stat directly without overwriting CORE::stat, thanks Dietmar for pointing this out https://pve.proxmox.com/pipermail/pve-devel/2020-April/043134.html * rebase to current master v3 -> v4: * fixed styling issues discovered by f.ebner (thanks) * incorporated tests of d.jaeger into patches * added fixes discovered by tests (f.ebner thanks) v2 -> v3: * split archive_info into decompressor_info and archive_info * "compact" regex pattern is now a constant and used in multiple modules * added tests for regex matching * bug fix for ctime of backup files v1 -> v2: * factored out the decompressor info first, as Thomas suggested * made the regex pattern of backup files more compact, easier to read (hopefully) * less code changes for container restores Thanks for any comment or suggestion in advance. [0] https://facebook.github.io/zstd/ Alwin Antreich (17): __pve-storage__ storage: test: split archive format/compressor storage: replace build-in stat with File::stat test: parse_volname test: list_volumes Fix: backup: ctime was from stat not file name test: path_to_volume_id Fix: path_to_volume_id returned wrong content Fix: add missing snippets subdir backup: compact regex for backup file filter Fix: #2124 storage: add zstd support test: get_subdir test: filesystem_path test/Makefile | 5 +- PVE/Diskmanage.pm | 9 +- PVE/Storage.pm | 91 ++++-- PVE/Storage/Plugin.pm | 47 ++- test/archive_info_test.pm | 127 ++++++++ test/filesystem_path_test.pm | 91 ++++++ test/get_subdir_test.pm | 44 +++ test/list_volumes_test.pm | 537 +++++++++++++++++++++++++++++++++ test/parse_volname_test.pm | 253 ++++++++++++++++ test/path_to_volume_id_test.pm | 274 +++++++++++++++++ test/run_plugin_tests.pl | 18 ++ 11 files changed, 1440 insertions(+), 56 deletions(-) create mode 100644 test/archive_info_test.pm create mode 100644 test/filesystem_path_test.pm create mode 100644 test/get_subdir_test.pm create mode 100644 test/list_volumes_test.pm create mode 100644 test/parse_volname_test.pm create mode 100644 test/path_to_volume_id_test.pm create mode 100755 test/run_plugin_tests.pl __guest_common__ Fix: #2124 add zstd support PVE/VZDump/Common.pm | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) __qemu-server__ restore: replace archive format/compression Fix #2124: Add support for zstd PVE/QemuServer.pm | 38 +++++++------------------------------- 1 file changed, 7 insertions(+), 31 deletions(-) __pve-container__ Fix: #2124 add zstd src/PVE/LXC/Create.pm | 1 + 1 file changed, 1 insertion(+) __pve-manager__ Fix #2124: Add support for zstd PVE/VZDump.pm | 11 +++++++++-- debian/control | 1 + www/manager6/form/CompressionSelector.js | 3 ++- 3 files changed, 12 insertions(+), 3 deletions(-) -- 2.20.1 _______________________________________________ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel