Re: [pve-devel] [PATCH v3 0/13] Fix: #2124 zstd
On Thu, Apr 09, 2020 at 02:29:46PM +0200, Fabian Ebner wrote: > Hi, > series looks mostly good to me. Some comments on individual patches. > Backup/restore seems to work, also still with the other compression formats. > The tests for path_to_volume_id actually uncover a bug and a missing feature > in the implementation in Storage.pm, which should be fixed. I'll look into it. > > For the patches that won't change (much): > Reviewed-By: Fabian Ebner > Tested-By: Fabian Ebner Thanks for review and testing. > > For a potential follow-up, I feel like the parsing of the backup filename > might need its own method, rather than have all those pattern matchings in > different places. That was my intent with archive_info, it could be extend later on. :) ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH v3 0/13] Fix: #2124 zstd
Hi, series looks mostly good to me. Some comments on individual patches. Backup/restore seems to work, also still with the other compression formats. The tests for path_to_volume_id actually uncover a bug and a missing feature in the implementation in Storage.pm, which should be fixed. For the patches that won't change (much): Reviewed-By: Fabian Ebner Tested-By: Fabian Ebner For a potential follow-up, I feel like the parsing of the backup filename might need its own method, rather than have all those pattern matchings in different places. On 08.04.20 12:25, Alwin Antreich wrote: Zstandard (zstd) [0] is a data compression algorithm, in addition to gzip, lzo for our backup/restore. It can utilize multiple core CPUs. But by default it has one compression and one writer thread. Here some quick tests I made on my workstation. The files where placed on a ram disk. And with dd filled from /dev/urandom and /dev/zero. __Compression__ file size: 1073741824 bytes = urandom = = zero = 995ms 1073766414328ms 98192zstd -k 732ms 1073766414295ms 98192zstd -k -T4 906ms 1073791036562ms 4894779 lzop -k 31992ms 1073915558 5594ms 1042087 gzip -k 30832ms 1074069541 5776ms 1171491 pigz -k -p 1 7814ms 1074069541 1567ms 1171491 pigz -k -p 4 __Decompression__ file size: 1073741824 bytes = urandom = = zero = 712ms 869ms zstd -d 685ms 872ms zstd -k -d -T4 841ms 2462ms lzop -d 5417ms 4754ms gzip -k -d 1248ms 3118ms pigz -k -d -p 1 1236ms 2379ms pigz -k -d -p 4 And I used the same ramdisk to move a VM onto it and run a quick backup/restore. __vzdump backup__ INFO: transferred 34359 MB in 69 seconds (497 MB/s) zstd -T1 INFO: transferred 34359 MB in 37 seconds (928 MB/s) zstd -T4 INFO: transferred 34359 MB in 51 seconds (673 MB/s) lzo INFO: transferred 34359 MB in 1083 seconds (31 MB/s) gzip INFO: transferred 34359 MB in 241 seconds (142 MB/s) pigz -n 4 __qmrestore__ progress 100% (read 34359738368 bytes, duration 36 sec) total bytes read 34359738368, sparse bytes 8005484544 (23.3%) zstd -d -T4 progress 100% (read 34359738368 bytes, duration 38 sec) total bytes read 34359738368, sparse bytes 8005484544 (23.3%) lzo progress 100% (read 34359738368 bytes, duration 175 sec) total bytes read 34359738368, sparse bytes 8005484544 (23.3%) pigz -n 4 v2 -> v3: * split archive_info into decompressor_info and archive_info * "compact" regex pattern is now a constant and used in multiple modules * added tests for regex matching * bug fix for ctime of backup files v1 -> v2: * factored out the decompressor info first, as Thomas suggested * made the regex pattern of backup files more compact, easier to read (hopefully) * less code changes for container restores Thanks for any comment or suggestion in advance. [0] https://facebook.github.io/zstd/ Alwin Antreich (13): __pve-storage__ storage: test: split archive format/compressor test: parse_volname test: list_volumes Fix: backup: ctime taken from stat not file name test: path_to_volume_id backup: more compact regex for backup file filter Fix: #2124 storage: add zstd support test/Makefile | 5 +- PVE/Storage.pm | 85 ++--- PVE/Storage/Plugin.pm | 9 +- test/run_parser_tests.pl | 17 ++ test/test_archive_info.pm | 57 ++ test/test_list_volumes.pm | 313 + test/test_parse_volname.pm | 98 +++ test/test_path_to_volume_id.pm | 104 +++ 8 files changed, 661 insertions(+), 27 deletions(-) create mode 100755 test/run_parser_tests.pl create mode 100644 test/test_archive_info.pm create mode 100644 test/test_list_volumes.pm create mode 100644 test/test_parse_volname.pm create mode 100644 test/test_path_to_volume_id.pm __pve-container__ Fix: #2124 add zstd src/PVE/LXC/Create.pm | 1 + 1 file changed, 1 insertion(+) __qemu-server__ restore: replace archive format/compression regex Fix #2124: Add support for zstd PVE/QemuServer.pm | 38 +++--- 1 file changed, 7 insertions(+), 31 deletions(-) __pve-manager__ Fix #2124: Add support for zstd PVE/VZDump.pm| 11 +-- debian/control | 1 + www/manager6/form/CompressionSelector.js | 3 ++- 3 files changed, 12 insertions(+), 3 deletions(-) __pve-guest-common__ Fix: #2124 add zstd support PVE/VZDump/Common.pm | 9 - 1 file changed, 8 insertions(+), 1 deletion(-) ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v3 0/13] Fix: #2124 zstd
Zstandard (zstd) [0] is a data compression algorithm, in addition to gzip, lzo for our backup/restore. It can utilize multiple core CPUs. But by default it has one compression and one writer thread. Here some quick tests I made on my workstation. The files where placed on a ram disk. And with dd filled from /dev/urandom and /dev/zero. __Compression__ file size: 1073741824 bytes = urandom = = zero = 995ms 1073766414328ms 98192zstd -k 732ms 1073766414295ms 98192zstd -k -T4 906ms 1073791036562ms 4894779 lzop -k 31992ms 1073915558 5594ms 1042087 gzip -k 30832ms 1074069541 5776ms 1171491 pigz -k -p 1 7814ms 1074069541 1567ms 1171491 pigz -k -p 4 __Decompression__ file size: 1073741824 bytes = urandom = = zero = 712ms 869ms zstd -d 685ms 872ms zstd -k -d -T4 841ms 2462ms lzop -d 5417ms 4754ms gzip -k -d 1248ms 3118ms pigz -k -d -p 1 1236ms 2379ms pigz -k -d -p 4 And I used the same ramdisk to move a VM onto it and run a quick backup/restore. __vzdump backup__ INFO: transferred 34359 MB in 69 seconds (497 MB/s) zstd -T1 INFO: transferred 34359 MB in 37 seconds (928 MB/s) zstd -T4 INFO: transferred 34359 MB in 51 seconds (673 MB/s) lzo INFO: transferred 34359 MB in 1083 seconds (31 MB/s) gzip INFO: transferred 34359 MB in 241 seconds (142 MB/s) pigz -n 4 __qmrestore__ progress 100% (read 34359738368 bytes, duration 36 sec) total bytes read 34359738368, sparse bytes 8005484544 (23.3%) zstd -d -T4 progress 100% (read 34359738368 bytes, duration 38 sec) total bytes read 34359738368, sparse bytes 8005484544 (23.3%) lzo progress 100% (read 34359738368 bytes, duration 175 sec) total bytes read 34359738368, sparse bytes 8005484544 (23.3%) pigz -n 4 v2 -> v3: * split archive_info into decompressor_info and archive_info * "compact" regex pattern is now a constant and used in multiple modules * added tests for regex matching * bug fix for ctime of backup files v1 -> v2: * factored out the decompressor info first, as Thomas suggested * made the regex pattern of backup files more compact, easier to read (hopefully) * less code changes for container restores Thanks for any comment or suggestion in advance. [0] https://facebook.github.io/zstd/ Alwin Antreich (13): __pve-storage__ storage: test: split archive format/compressor test: parse_volname test: list_volumes Fix: backup: ctime taken from stat not file name test: path_to_volume_id backup: more compact regex for backup file filter Fix: #2124 storage: add zstd support test/Makefile | 5 +- PVE/Storage.pm | 85 ++--- PVE/Storage/Plugin.pm | 9 +- test/run_parser_tests.pl | 17 ++ test/test_archive_info.pm | 57 ++ test/test_list_volumes.pm | 313 + test/test_parse_volname.pm | 98 +++ test/test_path_to_volume_id.pm | 104 +++ 8 files changed, 661 insertions(+), 27 deletions(-) create mode 100755 test/run_parser_tests.pl create mode 100644 test/test_archive_info.pm create mode 100644 test/test_list_volumes.pm create mode 100644 test/test_parse_volname.pm create mode 100644 test/test_path_to_volume_id.pm __pve-container__ Fix: #2124 add zstd src/PVE/LXC/Create.pm | 1 + 1 file changed, 1 insertion(+) __qemu-server__ restore: replace archive format/compression regex Fix #2124: Add support for zstd PVE/QemuServer.pm | 38 +++--- 1 file changed, 7 insertions(+), 31 deletions(-) __pve-manager__ Fix #2124: Add support for zstd PVE/VZDump.pm| 11 +-- debian/control | 1 + www/manager6/form/CompressionSelector.js | 3 ++- 3 files changed, 12 insertions(+), 3 deletions(-) __pve-guest-common__ Fix: #2124 add zstd support PVE/VZDump/Common.pm | 9 - 1 file changed, 8 insertions(+), 1 deletion(-) -- 2.20.1 ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
