Zstandard (zstd) [0] is a data compression algorithm, in addition to
gzip, lzo for our backup/restore. It can utilize multiple core CPUs. But
by default it has one compression and one writer thread.
Here some quick tests I made on my workstation. The files where placed
on a ram disk. And with dd filled from /dev/urandom and /dev/zero.
__Compression__
file size: 1073741824 bytes
= urandom = = zero =
995ms 1073766414328ms 98192zstd -k
732ms 1073766414295ms 98192zstd -k -T4
906ms 1073791036562ms 4894779 lzop -k
31992ms 1073915558 5594ms 1042087 gzip -k
30832ms 1074069541 5776ms 1171491 pigz -k -p 1
7814ms 1074069541 1567ms 1171491 pigz -k -p 4
__Decompression__
file size: 1073741824 bytes
= urandom = = zero =
712ms 869ms zstd -d
685ms 872ms zstd -k -d -T4
841ms 2462ms lzop -d
5417ms 4754ms gzip -k -d
1248ms 3118ms pigz -k -d -p 1
1236ms 2379ms pigz -k -d -p 4
And I used the same ramdisk to move a VM onto it and run a quick
backup/restore.
__vzdump backup__
INFO: transferred 34359 MB in 69 seconds (497 MB/s) zstd -T1
INFO: transferred 34359 MB in 37 seconds (928 MB/s) zstd -T4
INFO: transferred 34359 MB in 51 seconds (673 MB/s) lzo
INFO: transferred 34359 MB in 1083 seconds (31 MB/s) gzip
INFO: transferred 34359 MB in 241 seconds (142 MB/s) pigz -n 4
__qmrestore__
progress 100% (read 34359738368 bytes, duration 36 sec)
total bytes read 34359738368, sparse bytes 8005484544 (23.3%) zstd -d -T4
progress 100% (read 34359738368 bytes, duration 38 sec)
total bytes read 34359738368, sparse bytes 8005484544 (23.3%) lzo
progress 100% (read 34359738368 bytes, duration 175 sec)
total bytes read 34359738368, sparse bytes 8005484544 (23.3%) pigz -n 4
v3 -> v4:
* fixed styling issues discovered by f.ebner (thanks)
* incorporated tests of d.jaeger into patches
* added fixes discovered by tests (f.ebner thanks)
v2 -> v3:
* split archive_info into decompressor_info and archive_info
* "compact" regex pattern is now a constant and used in
multiple modules
* added tests for regex matching
* bug fix for ctime of backup files
v1 -> v2:
* factored out the decompressor info first, as Thomas suggested
* made the regex pattern of backup files more compact, easier to
read (hopefully)
* less code changes for container restores
Thanks for any comment or suggestion in advance.
[0] https://facebook.github.io/zstd/
Alwin Antreich (12):
__pve-storage__
storage: test: split archive format/compressor
storage: replace build-in stat with File::stat
test: parse_volname
test: list_volumes
Fix: backup: ctime was from stat not file name
test: path_to_volume_id
Fix: path_to_volume_id returned wrong content
Fix: add missing snippets subdir
backup: compact regex for backup file filter
Fix: #2124 storage: add zstd support
test: get_subdir
test: filesystem_path
test/Makefile | 5 +-
PVE/Diskmanage.pm | 9 +-
PVE/Storage.pm | 91 --
PVE/Storage/Plugin.pm | 47 ++-
test/archive_info_test.pm | 127
test/filesystem_path_test.pm | 91 ++
test/get_subdir_test.pm| 44 +++
test/list_volumes_test.pm | 537 +
test/parse_volname_test.pm | 253
test/path_to_volume_id_test.pm | 274 +
test/run_plugin_tests.pl | 18 ++
11 files changed, 1440 insertions(+), 56 deletions(-)
create mode 100644 test/archive_info_test.pm
create mode 100644 test/filesystem_path_test.pm
create mode 100644 test/get_subdir_test.pm
create mode 100644 test/list_volumes_test.pm
create mode 100644 test/parse_volname_test.pm
create mode 100644 test/path_to_volume_id_test.pm
create mode 100755 test/run_plugin_tests.pl
__qemu-server__
restore: replace archive format/compression
Fix #2124: Add support for zstd
PVE/QemuServer.pm | 38 +++---
1 file changed, 7 insertions(+), 31 deletions(-)
__pve-container__
Fix: #2124 add zstd
src/PVE/LXC/Create.pm | 1 +
1 file changed, 1 insertion(+)
__guest_common__
Fix: #2124 add zstd support
PVE/VZDump/Common.pm | 9 -
1 file changed, 8 insertions(+), 1 deletion(-)
__pve-manager__
Fix #2124: Add support for zstd
PVE/VZDump.pm| 11 +--
debian/control | 1 +
www/manager6/form/CompressionSelector.js | 3 ++-
3 files changed, 12 insertions(+), 3 deletions(-)
--
2.20.1
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
