On 02/09/2025 5:15 pm, Andrew Cooper wrote: > On 25/08/2025 2:54 pm, Jan Beulich wrote: >> Other projects have long switched to xz and/or lzip. >> >> Tidy things some as well: With the removal of qemu from the tarball, >> intermediately extracting the tarball again has become wasteful. Drop >> that. Invoke compressors using asynchronous lists, to reduce overall >> latency. Drop the -v option from the (previously implicit) gzip >> invocation. >> >> Signed-off-by: Jan Beulich <jbeul...@suse.com> >> --- >> v2: Don't expand intermediate uncompressed tarball. Don't check for >> commands' availablity. Don't request statistics. Use async lists. >> >> --- a/docs/process/release-technician-checklist.txt >> +++ b/docs/process/release-technician-checklist.txt >> @@ -119,7 +119,7 @@ RELEASE TARBALL >> make src-tarball # uses git-describe (best for RCs) >> # ^find some way to add git-cache-proxy to this (done in >> ~iwj/.gitconfig) >> mkdir /volatile/iwj/website-thing/xen.org/oss-xen/release/$v >> - mv dist/xen-$v.tar.gz >> /volatile/iwj/website-thing/xen.org/oss-xen/release/$v/. >> + mv dist/xen-$v.tar.[glx]z >> /volatile/iwj/website-thing/xen.org/oss-xen/release/$v/. >> >> # website-thing/xen.org is cvs -d >> mail.xenproject.org:/home/downloads-cvs/cvs-repos co xen.org >> cd /volatile/iwj/website-thing/xen.org >> @@ -139,9 +139,12 @@ RELEASE TARBALL >> cvs add -kb oss-xen/release/$v/ >> >> cd oss-xen/release/$v >> - gpg --digest-algo=SHA256 --detach-sign -u 'xen tree' xen-$v.tar.gz >> - cvs add -kb xen-$v.tar.gz >> - cvs add -kb xen-$v.tar.gz.sig >> + for t in xen-$v.tar.[glx]z >> + do >> + gpg --digest-algo=SHA256 --detach-sign -u 'xen tree' $t >> + cvs add -kb $t >> + cvs add -kb $t.sig >> + done >> cd ../../.. >> >> cvs ci -m $v >> @@ -152,6 +155,10 @@ RELEASE TARBALL >> # should show something like >> # U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.gz >> # U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.gz.sig >> + # U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.lz >> + # U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.lz.sig >> + # U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.xz >> + # U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.xz.sig >> >> After a .0 release, update XEN_EXTRAVERSION again (to .1-pre, see above). >> >> --- a/docs/process/xen-release-management.pandoc >> +++ b/docs/process/xen-release-management.pandoc >> @@ -274,10 +274,10 @@ Xen X.Y rcZ is tagged. You can check tha >> https://xenbits.xen.org/git-http/xen.git X.Y.0-rcZ >> >> For your convenience there is also a tarball at: >> -https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.gz >> +https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.[glx]z >> >> And the signature is at: >> -https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.gz.sig >> +https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.[glx]z.sig >> >> Please send bug reports and test reports to xen-devel@lists.xenproject.org. >> When sending bug reports, please CC relevant maintainers and me >> --- a/tools/misc/mktarball >> +++ b/tools/misc/mktarball >> @@ -5,14 +5,6 @@ >> # Takes 2 arguments, the path to the dist directory and the version >> set -ex >> >> -function git_archive_into { >> - mkdir -p "$2" >> - >> - git --git-dir="$1"/.git \ >> - archive --format=tar HEAD | \ >> - tar Cxf "$2" - >> -} >> - >> if [[ -z "$1" || -z "$2" ]] ; then >> echo "usage: $0 path-to-XEN_ROOT xen-version" >> exit 1 >> @@ -21,14 +13,20 @@ fi >> xen_root="$1" >> desc="$2" >> >> -tdir="$xen_root/dist/tmp.src-tarball" >> +tdir="$xen_root/dist" >> >> -rm -rf $tdir >> +rm -f $tdir/xen-$desc.tar.[glx]z > This is asymmetric with the rm at the end. I'd remove > $tdir/xen-$desc.tar* here and remove the final rm. > > Looking at the uncompressed tarball is part of my process, and it was > preserved previously. > > With something along these lines, Reviewed-by: Andrew Cooper > <andrew.coop...@citrix.com> > >> >> mkdir -p $tdir >> >> -git_archive_into $xen_root $tdir/xen-$desc >> +git --git-dir="$xen_root/.git" archive --format=tar HEAD >> --prefix=xen-$desc/ \ >> + >"$tdir/xen-$desc.tar" >> + >> +gzip -9k "$tdir/xen-$desc.tar" & >> +xz -9k "$tdir/xen-$desc.tar" & >> +lzip -9k "$tdir/xen-$desc.tar" & >> +wait > Interestingly, this wasn't fatal for not having lzip, but the error was > clear on the console given the reduced verbosity, and doing 3 at the > same time worked very nicely. > >> >> -GZIP=-9v tar cz -f $xen_root/dist/xen-$desc.tar.gz -C $tdir xen-$desc >> +rm -f $tdir/xen-$desc.tar >> >> -echo "Source tarball in $xen_root/dist/xen-$desc.tar.gz" >> +echo "Source tarball in" $tdir/xen-$desc.tar.[glx]z > This was grammatically awkward to begin with, but is now pretty useless, > especially combined with the set -x so it gets printed twice. > > Something like this: > > echo "Source tarballs:" > ls -lah $tdir/xen-$desc.tar* > > generates: > > -rw-rw-r-- 1 andrew andrew 32M Sep 2 17:13 > /home/andrew/xen.git/dist/xen-4.21-unstable.tar > -rw-rw-r-- 1 andrew andrew 6.8M Sep 2 17:13 > /home/andrew/xen.git/dist/xen-4.21-unstable.tar.gz > -rw-rw-r-- 1 andrew andrew 4.7M Sep 2 17:13 > /home/andrew/xen.git/dist/xen-4.21-unstable.tar.lz > -rw-rw-r-- 1 andrew andrew 4.7M Sep 2 17:13 > /home/andrew/xen.git/dist/xen-4.21-unstable.tar.xz > > > on my system and is rather more useful IMO.
And just for reference, here is what zstd looks like: -rw-rw-r-- 1 andrew andrew 32M Sep 2 17:24 /home/andrew/xen.git/dist/xen-4.21-unstable.tar -rw-rw-r-- 1 andrew andrew 6.8M Sep 2 17:24 /home/andrew/xen.git/dist/xen-4.21-unstable.tar.gz -rw-rw-r-- 1 andrew andrew 4.7M Sep 2 17:24 /home/andrew/xen.git/dist/xen-4.21-unstable.tar.lz -rw-rw-r-- 1 andrew andrew 4.7M Sep 2 17:24 /home/andrew/xen.git/dist/xen-4.21-unstable.tar.xz -rw-rw-r-- 1 andrew andrew 4.9M Sep 2 17:24 /home/andrew/xen.git/dist/xen-4.21-unstable.tar.zst using -19kq (zstd defaults to giving an interactive progress bar, and needs quietening). With a bit of `time` hacking, gzip is 2.4s, zx is 11.8s, zstd is 13s and lzip is 17.3s on my system. ~Andrew
