Hi,
Ludovic Courtès <[email protected]> skribis:
> From that, we could deduce that about 1% of our users who take
> substitutes from ci.guix are still using a pre-1.1.0 daemon without
> support for lzip compression.
>
> I find it surprisingly low: 1.1.0 was released “only” 9 months ago,
> which is not a lot for someone used to the long release cycles of
> “stable” distros.
(See
<https://lists.gnu.org/archive/html/guix-devel/2021-01/msg00378.html>
for the initial message.)
Here’s an update, 1.5 month later. This time I’m looking at nginx logs
covering Feb 8th to Mar 17th and using a laxer regexp than in the
message above, here are the gzip/lzip download ratio for several
packages:
--8<---------------cut here---------------start------------->8---
ludo@berlin ~$ ./nar-download-stats.sh /tmp/sample3.log
gtk%2B-3: gzip/lzip ratio: 37/3255 1%
glib-2: gzip/lzip ratio: 97/8629 1%
coreutils-8: gzip/lzip ratio: 81/2306 3%
python-3: gzip/lzip ratio: 120/7177 1%
r-minimal-[34]: gzip/lzip ratio: 8/302 2%
openmpi-4: gzip/lzip ratio: 19/236 8%
hwloc-2: gzip/lzip ratio: 10/43 23%
gfortran-7: gzip/lzip ratio: 6/225 2%
--8<---------------cut here---------------end--------------->8---
(Script attached.)
The hwloc/openmpi outlier is intriguing. Is it one HPC web site running
an old daemon, or several of them? Looking more closely, it’s 22 of
them on 8 different networks (looking at the first three digits of the
IP address):
--8<---------------cut here---------------start------------->8---
ludo@berlin ~$ grep -E
'/gzip/[[:alnum:]]{32}-(hwloc-2|openmpi-4)\.[[:digit:]]+\.[[:digit:]]+ ' <
/tmp/sample3.log | cut -f1 -d- | sort -u | wc -l
22
ludo@berlin ~$ grep -E
'/gzip/[[:alnum:]]{32}-(hwloc-2|openmpi-4)\.[[:digit:]]+\.[[:digit:]]+ ' <
/tmp/sample3.log | cut -f1 -d- | cut -f 1-3 -d. | sort -u | wc -l
8
--8<---------------cut here---------------end--------------->8---
Conclusion? It still sounds like we can’t reasonably remove gzip
support just yet.
I’d still like to start providing zstd-compressed substitutes though.
So I think what we can do is:
• start providing zstd substitutes on berlin right now so that when
1.2.1 comes out, at least some substitutes are available as zstd;
• when 1.2.1 is announced, announce that gzip substitutes may be
removed in the future and invite users to upgrade;
• revisit this issue with an eye on dropping gzip within 6–18 months.
Thoughts?
Ludo’.
#!/bin/sh
if [ ! "$#" = 1 ]
then
echo "Usage: $1 NGINX-LOG-FILE"
exit 1
fi
set -e
sample="$1"
items="gtk%2B-3 glib-2 coreutils-8 python-3 r-minimal-[34] openmpi-4 hwloc-2
gfortran-7"
for i in $items
do
# Tweak the regexp so we don't catch ".drv" substitutes as these
# usually compress better with gzip.
lzip="$(grep -E "/lzip/[[:alnum:]]{32}-$i\\.[[:digit:]]+(\\.[[:digit:]]+)?
" < "$sample" | wc -l)"
gzip="$(grep -E "/gzip/[[:alnum:]]{32}-$i\\.[[:digit:]]+(\\.[[:digit:]]+)?
" < "$sample" | wc -l)"
echo "$i: gzip/lzip ratio: $gzip/$lzip $(($gzip * 100 / $lzip))%"
done
