Hello again, after archange and Baptiste mentioned that the numbers look a little odd, I took some more time and re-ran the tests with additional parameters. Most notably, this includes -T2 - to show behavior on lower-spec machines, and it fixes the higher compression levels by appending --ultra.
Here are the new results: Compressor Package Name Size (MiB) Comp. Size (MiB) Ratio Time (mm:ss) Max. RSS in MiB Decomp. Time (mm:ss) Decomp. RSS in MiB xz -c -z - cuda 3038,58 1316,93 43,34% 19:03.44 95,32 1:19.74 10,18 zstd -c -T2 -18 - cuda 3038,58 1375,41 45,26% 7:10.76 373,53 0:04.49 10,70 zstd -c -T0 -18 - cuda 3038,58 1375,41 45,26% 1:17.23 2646,19 0:04.41 10,76 zstd -c -T2 -19 - cuda 3038,58 1371,94 45,15% 9:08.09 420,43 0:04.43 10,68 zstd -c -T0 -19 - cuda 3038,58 1371,94 45,15% 1:34.74 3415,77 0:04.51 10,75 zstd -c -T2 --ultra -20 - cuda 3038,58 1286,91 42,35% 10:05.19 1255,64 0:04.46 34,78 zstd -c -T0 --ultra -20 - cuda 3038,58 1286,91 42,35% 1:57.94 8192,42 0:04.43 34,76 zstd -c -T2 --ultra -21 - cuda 3038,58 1141,94 37,58% 10:37.84 2404,56 0:04.11 66,73 zstd -c -T0 --ultra -21 - cuda 3038,58 1141,94 37,58% 2:58.45 8035,52 0:04.08 66,77 xz -c -z - gcc 135,54 33,11 24,43% 0:54.54 95,34 0:02.59 10,11 zstd -c -T2 -18 - gcc 135,54 35,87 26,47% 0:23.39 255,13 0:00.27 10,76 zstd -c -T0 -18 - gcc 135,54 35,87 26,47% 0:12.42 419,35 0:00.24 10,81 zstd -c -T2 -19 - gcc 135,54 35,66 26,31% 0:30.34 319,03 0:00.24 10,45 zstd -c -T0 -19 - gcc 135,54 35,66 26,31% 0:16.07 579,00 0:00.24 10,73 zstd -c -T2 --ultra -20 - gcc 135,54 24,38 17,99% 0:51.69 484,32 0:00.20 34,63 zstd -c -T0 --ultra -20 - gcc 135,54 24,38 17,99% 0:51.79 484,66 0:00.20 34,73 zstd -c -T2 --ultra -21 - gcc 135,54 22,89 16,89% 1:10.22 481,77 0:00.22 66,71 zstd -c -T0 --ultra -21 - gcc 135,54 22,89 16,89% 1:10.39 482,17 0:00.21 66,65 xz -c -z - go 484,10 122,10 25,22% 3:19.11 95,35 0:08.78 10,16 zstd -c -T2 -18 - go 484,10 132,69 27,41% 1:20.42 292,36 0:00.78 10,75 zstd -c -T0 -18 - go 484,10 132,69 27,41% 0:15.42 1402,87 0:00.78 10,79 zstd -c -T2 -19 - go 484,10 131,84 27,23% 1:46.85 352,77 0:00.79 10,75 zstd -c -T0 -19 - go 484,10 131,84 27,23% 0:20.13 1914,13 0:00.80 10,75 zstd -c -T2 --ultra -20 - go 484,10 121,87 25,17% 1:58.00 879,29 0:00.83 34,68 zstd -c -T0 --ultra -20 - go 484,10 121,87 25,17% 1:07.37 1252,75 0:00.84 34,71 zstd -c -T2 --ultra -21 - go 484,10 112,18 23,17% 2:09.79 1240,84 0:00.82 66,73 zstd -c -T0 --ultra -21 - go 484,10 112,18 23,17% 2:09.70 1241,08 0:00.81 66,80 xz -c -z - intellij-* 772,46 384,37 49,76% 4:53.01 95,31 0:28.69 10,18 zstd -c -T2 -18 - intellij-* 772,46 392,44 50,80% 1:51.91 342,91 0:00.94 10,71 zstd -c -T0 -18 - intellij-* 772,46 392,44 50,80% 0:20.50 2341,05 0:00.93 10,70 zstd -c -T2 -19 - intellij-* 772,46 391,04 50,62% 2:40.44 407,06 0:00.94 10,82 zstd -c -T0 -19 - intellij-* 772,46 391,04 50,62% 0:28.37 3107,88 0:00.95 10,76 zstd -c -T2 --ultra -20 - intellij-* 772,46 380,38 49,24% 3:19.46 1182,29 0:01.04 34,73 zstd -c -T0 --ultra -20 - intellij-* 772,46 380,38 49,24% 1:28.10 2282,25 0:01.03 34,72 zstd -c -T2 --ultra -21 - intellij-* 772,46 374,19 48,44% 4:07.18 1788,19 0:01.06 66,81 zstd -c -T0 --ultra -21 - intellij-* 772,46 374,19 48,44% 2:31.77 2433,28 0:01.04 66,73 xz -c -z - linux 80,15 70,66 88,17% 0:31.27 95,35 0:03.85 10,11 zstd -c -T2 -18 - linux 80,15 70,22 87,62% 0:09.94 250,32 0:00.06 10,56 zstd -c -T0 -18 - linux 80,15 70,22 87,62% 0:07.52 299,25 0:00.06 10,72 zstd -c -T2 -19 - linux 80,15 70,18 87,56% 0:12.90 314,25 0:00.05 10,57 zstd -c -T0 -19 - linux 80,15 70,18 87,56% 0:09.32 395,17 0:00.06 10,71 zstd -c -T2 --ultra -20 - linux 80,15 70,18 87,56% 0:22.64 313,47 0:00.08 34,61 zstd -c -T0 --ultra -20 - linux 80,15 70,18 87,56% 0:22.69 313,82 0:00.08 34,68 zstd -c -T2 --ultra -21 - linux 80,15 70,17 87,55% 0:27.01 473,56 0:00.09 66,64 zstd -c -T0 --ultra -21 - linux 80,15 70,17 87,55% 0:26.97 473,89 0:00.09 66,71 xz -c -z - linux-headers 103,85 17,02 16,39% 0:42.24 95,35 0:01.45 10,15 zstd -c -T2 -18 - linux-headers 103,85 18,92 18,22% 0:19.51 218,35 0:00.17 10,48 zstd -c -T0 -18 - linux-headers 103,85 18,92 18,22% 0:12.74 320,92 0:00.17 10,47 zstd -c -T2 -19 - linux-headers 103,85 18,88 18,18% 0:24.42 282,43 0:00.16 10,67 zstd -c -T0 -19 - linux-headers 103,85 18,88 18,18% 0:16.28 448,92 0:00.17 10,59 zstd -c -T2 --ultra -20 - linux-headers 103,85 18,77 18,08% 0:43.86 286,13 0:00.19 34,74 zstd -c -T0 --ultra -20 - linux-headers 103,85 18,77 18,08% 0:44.00 286,32 0:00.19 34,79 zstd -c -T2 --ultra -21 - linux-headers 103,85 18,70 18,00% 1:03.41 445,96 0:00.20 66,61 zstd -c -T0 --ultra -21 - linux-headers 103,85 18,70 18,00% 1:03.29 446,32 0:00.20 66,66 xz -c -z - tensorflow 303,10 55,58 18,34% 1:59.56 95,40 0:04.78 10,27 zstd -c -T2 -18 - tensorflow 303,10 61,83 20,40% 0:54.04 277,06 0:00.48 10,72 zstd -c -T0 -18 - tensorflow 303,10 61,83 20,40% 0:15.64 856,86 0:00.47 10,61 zstd -c -T2 -19 - tensorflow 303,10 61,49 20,29% 1:15.56 340,99 0:00.48 10,75 zstd -c -T0 -19 - tensorflow 303,10 61,49 20,29% 0:20.82 1176,75 0:00.49 10,68 zstd -c -T2 --ultra -20 - tensorflow 303,10 60,63 20,00% 1:30.19 678,34 0:00.53 34,67 zstd -c -T0 --ultra -20 - tensorflow 303,10 60,63 20,00% 1:11.32 849,60 0:00.54 34,68 zstd -c -T2 --ultra -21 - tensorflow 303,10 59,98 19,79% 2:42.81 1007,56 0:00.54 66,47 zstd -c -T0 --ultra -21 - tensorflow 303,10 59,98 19,79% 2:43.03 1007,95 0:00.55 66,70 The new results show that -20 is actually more beneficial for our goals, as it: - Actually reduces the size compared to xz more often - While not as fast, still beats xz in compression time - Only increases the decompressor memory usage negligibly - Maintains similar decompression speed to the other levels TL;DR: Benefits: - Faster - Often smaller or similar to xz in size, an improvement over -18 either way - Still reproducible :) Trade-offs: - Minimal increase in decompressor memory usage, but we're talking 50 MiB here. - Increase in memory usage during compression, however the important part is that memory usage scales with the amount of threads used. Given that low-end systems can simply change the thread allocation to 1 or 2 to slash the compressor memory usage as a trade-off on speed, i don't think that is a problem. New changeset: PKGEXT='.pkg.tar.zst' COMPRESSZST=(zstd -c -T0 -20 -) This would hopefully address the concerns over the filesize increase, while still maintaining most of the benefits. Regards, Rob (coderobe)
signature.asc
Description: OpenPGP digital signature
