Package: debootstrap Version: 1.0.142 Severity: wishlist Tags: patch Hi!
What do you think about support for zstd compression of Packages files? The performance increase for compression with zstd instead of xz is significant. Speed comparison for doing 5 compressions of the trixie Packages file of ~54MB on my laptop: jas@frallan:~$ du -h Packages; time for i in $(seq 1 5); do cat Packages | /usr/bin/time xz > Packages.xz; done; du -h Packages.xz 54M Packages 13.35user 0.06system 0:06.19elapsed 216%CPU (0avgtext+0avgdata 336528maxresident)k 0inputs+19240outputs (0major+17574minor)pagefaults 0swaps 13.20user 0.05system 0:06.12elapsed 216%CPU (0avgtext+0avgdata 336660maxresident)k 0inputs+19240outputs (0major+17572minor)pagefaults 0swaps 13.22user 0.07system 0:06.21elapsed 213%CPU (0avgtext+0avgdata 335796maxresident)k 0inputs+19240outputs (0major+17379minor)pagefaults 0swaps 13.27user 0.09system 0:06.16elapsed 216%CPU (0avgtext+0avgdata 334456maxresident)k 0inputs+19240outputs (0major+17509minor)pagefaults 0swaps 13.38user 0.06system 0:06.25elapsed 214%CPU (0avgtext+0avgdata 335836maxresident)k 0inputs+19240outputs (0major+17811minor)pagefaults 0swaps real 0m31,027s user 1m6,461s sys 0m0,444s 9,4M Packages.xz jas@frallan:~$ du -h Packages; time for i in $(seq 1 5); do cat Packages | /usr/bin/time zstd > Packages.zst; done; du -h Packages.zst 54M Packages 0.17user 0.03system 0:00.06elapsed 304%CPU (0avgtext+0avgdata 77344maxresident)k 0inputs+24704outputs (0major+2565minor)pagefaults 0swaps 0.13user 0.02system 0:00.04elapsed 340%CPU (0avgtext+0avgdata 75008maxresident)k 0inputs+24704outputs (0major+1920minor)pagefaults 0swaps 0.17user 0.02system 0:00.05elapsed 332%CPU (0avgtext+0avgdata 76868maxresident)k 0inputs+24704outputs (0major+1919minor)pagefaults 0swaps 0.19user 0.02system 0:00.06elapsed 324%CPU (0avgtext+0avgdata 79672maxresident)k 0inputs+24704outputs (0major+2520minor)pagefaults 0swaps 0.14user 0.02system 0:00.05elapsed 322%CPU (0avgtext+0avgdata 75044maxresident)k 0inputs+24704outputs (0major+1911minor)pagefaults 0swaps real 0m0,323s user 0m0,826s sys 0m0,221s 13M Packages.zst jas@frallan:~$ That is going from over a minute to below a second! What matters for users is transfer time and decompression time though. Transfer time depends on network speed, and sometimes the size difference of 13M vs 9.4M can be significant. But let's look at decompression speed: jas@frallan:~$ du -h Packages.zst; time for i in $(seq 1 10); do cat Packages.zst | /usr/bin/time unzstd > Packages; done; du -h Packages 13M Packages.zst 0.04user 0.00system 0:00.03elapsed 148%CPU (0avgtext+0avgdata 6700maxresident)k 0inputs+110384outputs (0major+1235minor)pagefaults 0swaps 0.03user 0.01system 0:00.03elapsed 143%CPU (0avgtext+0avgdata 6260maxresident)k 0inputs+110384outputs (0major+1204minor)pagefaults 0swaps 0.02user 0.01system 0:00.03elapsed 143%CPU (0avgtext+0avgdata 6188maxresident)k 0inputs+110384outputs (0major+1203minor)pagefaults 0swaps 0.02user 0.01system 0:00.03elapsed 143%CPU (0avgtext+0avgdata 6312maxresident)k 0inputs+110384outputs (0major+1173minor)pagefaults 0swaps 0.04user 0.00system 0:00.03elapsed 146%CPU (0avgtext+0avgdata 6372maxresident)k 0inputs+110384outputs (0major+1237minor)pagefaults 0swaps 0.03user 0.01system 0:00.03elapsed 142%CPU (0avgtext+0avgdata 6352maxresident)k 0inputs+110384outputs (0major+1207minor)pagefaults 0swaps 0.02user 0.02system 0:00.03elapsed 150%CPU (0avgtext+0avgdata 6692maxresident)k 0inputs+110384outputs (0major+759minor)pagefaults 0swaps 0.02user 0.01system 0:00.03elapsed 146%CPU (0avgtext+0avgdata 6496maxresident)k 0inputs+110384outputs (0major+1207minor)pagefaults 0swaps 0.03user 0.02system 0:00.03elapsed 157%CPU (0avgtext+0avgdata 6664maxresident)k 0inputs+110384outputs (0major+1270minor)pagefaults 0swaps 0.03user 0.01system 0:00.03elapsed 154%CPU (0avgtext+0avgdata 6368maxresident)k 0inputs+110384outputs (0major+1206minor)pagefaults 0swaps real 0m0,484s user 0m0,341s sys 0m0,312s 54M Packages jas@frallan:~$ du -h Packages.xz; time for i in $(seq 1 10); do cat Packages.xz | /usr/bin/time unxz > Packages; done; du -h Packages 9,4M Packages.xz 0.25user 0.03system 0:00.13elapsed 223%CPU (0avgtext+0avgdata 87876maxresident)k 0inputs+110384outputs (0major+3654minor)pagefaults 0swaps 0.25user 0.03system 0:00.12elapsed 226%CPU (0avgtext+0avgdata 87468maxresident)k 0inputs+110384outputs (0major+3653minor)pagefaults 0swaps 0.24user 0.04system 0:00.12elapsed 223%CPU (0avgtext+0avgdata 87396maxresident)k 0inputs+110384outputs (0major+3652minor)pagefaults 0swaps 0.27user 0.01system 0:00.13elapsed 222%CPU (0avgtext+0avgdata 88012maxresident)k 0inputs+110384outputs (0major+4205minor)pagefaults 0swaps 0.28user 0.03system 0:00.14elapsed 217%CPU (0avgtext+0avgdata 88840maxresident)k 0inputs+110384outputs (0major+3655minor)pagefaults 0swaps 0.27user 0.01system 0:00.13elapsed 225%CPU (0avgtext+0avgdata 84768maxresident)k 0inputs+110384outputs (0major+3658minor)pagefaults 0swaps 0.26user 0.03system 0:00.13elapsed 235%CPU (0avgtext+0avgdata 85596maxresident)k 0inputs+110384outputs (0major+4257minor)pagefaults 0swaps 0.26user 0.03system 0:00.12elapsed 230%CPU (0avgtext+0avgdata 88008maxresident)k 0inputs+110384outputs (0major+4257minor)pagefaults 0swaps 0.26user 0.02system 0:00.13elapsed 223%CPU (0avgtext+0avgdata 88828maxresident)k 0inputs+110384outputs (0major+4293minor)pagefaults 0swaps 0.27user 0.03system 0:00.13elapsed 227%CPU (0avgtext+0avgdata 87400maxresident)k 0inputs+110384outputs (0major+4253minor)pagefaults 0swaps real 0m1,565s user 0m2,677s sys 0m0,461s 54M Packages jas@frallan:~$ That is three times as fast. Regardless of performance, having support for this in debootstrapp allows for further real-world testing. Attached patch has also been submitted as a debootstrap merge request, for the purpose of having an informed discussion: https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/148 /Simon
From 599de36b8abd65419b2bfe5fc8af4f2118ef5d3d Mon Sep 17 00:00:00 2001 From: Simon Josefsson <[email protected]> Date: Sat, 3 Jan 2026 19:19:43 +0100 Subject: [PATCH] Add zstd support for Packages files --- functions | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/functions b/functions index 85ba9b5..4da7628 100644 --- a/functions +++ b/functions @@ -391,6 +391,7 @@ get () { iters="0" case "$typ" in + zst) from="$from_base.zst"; dest="$dest_base.zst" ;; xz) from="$from_base.xz"; dest="$dest_base.xz" ;; bz2) from="$from_base.bz2"; dest="$dest_base.bz2" ;; gz) from="$from_base.gz"; dest="$dest_base.gz" ;; @@ -431,6 +432,7 @@ get () { gz) gunzip "$dest" ;; bz2) bunzip2 "$dest" ;; xz) unxz "$dest" ;; + zst) unzstd --quiet --rm "$dest" ;; esac return 0 else @@ -723,7 +725,7 @@ download_release_sig () { download_release_indices () { local m1 inreldest reldest relsigdest totalpkgs \ - subpath xzi bz2i gzi normi i ext \ + subpath zsti xzi bz2i gzi normi i ext \ donepkgs pkgdest acquirebyhash s c a m m1="${MIRRORS%% *}" for s in $SUITE $EXTRA_SUITES; do @@ -743,12 +745,15 @@ download_release_indices () { totalpkgs=0 for c in $COMPONENTS; do subpath="$c/binary-$a/Packages" + zsti="$(get_release_checksum "$reldest" "$subpath.zst")" xzi="$(get_release_checksum "$reldest" "$subpath.xz")" bz2i="$(get_release_checksum "$reldest" "$subpath.bz2")" gzi="$(get_release_checksum "$reldest" "$subpath.gz")" normi="$(get_release_checksum "$reldest" "$subpath")" if [ "$normi" != "" ]; then i="$normi" + elif in_path unzstd && [ "$zsti" != "" ]; then + i="$zsti" elif in_path bunzip2 && [ "$bz2i" != "" ]; then i="$bz2i" elif in_path unxz && [ "$xzi" != "" ]; then @@ -770,6 +775,7 @@ download_release_indices () { subpath="$c/binary-$a/Packages" path="dists/$s/$subpath" xzi="$(get_release_checksum "$reldest" "$subpath.xz")" + zsti="$(get_release_checksum "$reldest" "$subpath.zst")" bz2i="$(get_release_checksum "$reldest" "$subpath.bz2")" gzi="$(get_release_checksum "$reldest" "$subpath.gz")" normi="$(get_release_checksum "$reldest" "$subpath")" @@ -781,6 +787,10 @@ download_release_indices () { ext="$ext $normi ." i="$normi" fi + if in_path unzstd && [ "$zsti" != "" ]; then + ext="$ext $zsti zst" + i="${i:-$zsti}" + fi if in_path unxz && [ "$xzi" != "" ]; then ext="$ext $xzi xz" i="${i:-$xzi}" -- 2.52.0
signature.asc
Description: PGP signature
