Re: Guix v1.1.0 (~2020) and Software Heritage & friends

[Simon Tournier]

Hi Ludo,

Thanks for giving a look and for your insights.

On Mon, 6 Mar 2023 at 21:40, Ludovic Courtès  wrote:

> Maybe we should write down a road map and “recruit” on these items?

Yeah, I totally agree. :-)  For what it is worth, in that mood, for instance,

 + GSoC 
 + Patch #62008 
 + SWH #4538< https://gitlab.softwareheritage.org/swh/meta/-/issues/4538>

Well, I will try to summarize the current status of Disarchive.

Cheers,
simon

Re: Guix v1.1.0 (~2020) and Software Heritage & friends

[Ludovic Courtès]

Hello!

Simon Tournier  skribis:

> Here I would like to report my attempt to simulate a “network collapse”
> – as discussed with Mathieu at Guix Days. :-)

Great experiment!

> Using this,
>
> archive.softwareheritage.org
> disarchive.guix.gnu.org
> git.savannah.gnu.org
>
> is not enough.  It fails for many.  For instance,
>
> ed-1.15
> gcc-core-2.95.3
> ghostscript-9.27
> guile-2.2.6
> linux-libre-4.19.56-gnu
> linux-libre-headers-stripped-4.14.67-i686-linux
> mes-0.19
> mes-minimal-stripped-0.19-i686-linux
>
> do not seem backed up with SWH + Disarchive.

It’d be great to identify the root cause for each of them, with a script
to do that.  But note that 1.1.0 did *not* support Disarchive; it can
only look up SWH or one of the content-addressed mirrors.

That said, it’s true that disarchive.guix lacks coverage for some of
these.  For ed, that’s because it’s lzip:

--8<---cut here---start->8---
$ guix time-machine --commit=v1.1.0 -- build -S ed
guile: warning: failed to install locale
0.1 MB will be downloaded:
   /gnu/store/0xgcp3mgq3nql22ak97dacvs3k18c4zg-ed-1.15.tar.lz
substituting /gnu/store/0xgcp3mgq3nql22ak97dacvs3k18c4zg-ed-1.15.tar.lz...
downloading from 
https://ci.guix.gnu.org/nar/0xgcp3mgq3nql22ak97dacvs3k18c4zg-ed-1.15.tar.lz...
 ed-1.15.tar.lz  66KiB  1.3MiB/s 00:00 
[##] 100.0%
/gnu/store/0xgcp3mgq3nql22ak97dacvs3k18c4zg-ed-1.15.tar.lz

$ wget  https://disarchive.guix.gnu.org/sha256/$(guix hash -f hex $(guix 
time-machine --commit=v1.1.0 -- build -S ed))
guile: warning: failed to install locale
--2023-03-06 21:27:53--  
https://disarchive.guix.gnu.org/sha256/ad4489c0ad7a108c514262da28e6c2a426946fb408a3977ef1ed34308bdfd174
Resolving disarchive.guix.gnu.org (disarchive.guix.gnu.org)... 141.80.181.40
Connecting to disarchive.guix.gnu.org 
(disarchive.guix.gnu.org)|141.80.181.40|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-03-06 21:27:53 ERROR 404: Not Found.
--8<---cut here---end--->8---

For Guile it’s probably because it’s xz and predates xz support in
Disarchive:

--8<---cut here---start->8---
$ guix time-machine --commit=v1.1.0 -- build -Sd guile@2.2.6
guile: warning: failed to install locale
/gnu/store/19mwlfc1092vipdka1y0l6v92ahkhl52-guile-2.2.6.tar.xz.drv
$ guix gc --references 
/gnu/store/19mwlfc1092vipdka1y0l6v92ahkhl52-guile-2.2.6.tar.xz.drv |grep 
tar.xz.drv
/gnu/store/pf9acfsb3zkmvfqcny4b8hrjif9a1kb8-guile-2.2.6.tar.xz.drv
$ wget -O-  https://disarchive.guix.gnu.org/sha256/$(guix hash -f hex $(guix 
build /gnu/store/pf9acfsb3zkmvfqcny4b8hrjif9a1kb8-guile-2.2.6.tar.xz.drv))
10.9 MB will be downloaded:
  /gnu/store/zfp7d4wr5hbl5lrnzs8c15bsc3ygq74g-guile-2.2.6.tar.xz
substituting /gnu/store/zfp7d4wr5hbl5lrnzs8c15bsc3ygq74g-guile-2.2.6.tar.xz...
downloading from 
https://ci.guix.gnu.org/nar/zfp7d4wr5hbl5lrnzs8c15bsc3ygq74g-guile-2.2.6.tar.xz 
...
 guile-2.2.6.tar.xz  10.4MiB 24.3MiB/s 00:00 ▕██▏ 100.0%

--2023-03-06 21:30:41--  
https://disarchive.guix.gnu.org/sha256/b33576331465a60b003573541bf3b1c205936a16c407bc69f8419a527bf5c988
Resolving disarchive.guix.gnu.org (disarchive.guix.gnu.org)... 141.80.181.40
Connecting to disarchive.guix.gnu.org 
(disarchive.guix.gnu.org)|141.80.181.40|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-03-06 21:30:41 ERROR 404: Not Found.

$ wget -qO-  https://disarchive.ngyro.com/sha256/$(guix hash -f hex $(guix 
build /gnu/store/pf9acfsb3zkmvfqcny4b8hrjif9a1kb8-guile-2.2.6.tar.xz.drv)) 
|head -4
(disarchive
  (version 0)
  (xz-file
(name "guile-2.2.6.tar.xz")
--8<---cut here---end--->8---

> Well, basically after these, I stop because it’s boring and I add
> ftp.gnu.org to the list.  It still fails,
>
> nyacc-0.86.0
> static-binaries
>
> First lesson: we have holes in sources.json and maybe Disarchive too; I
> will give a look.

Some are known issues: lack of lzip support in Disarchive, lack of
SVN/SWH integration, missing entries in the Disarchive database for old
packages, etc.

One thing that’s easily done is retroactively populating the Disarchive
database.  It’s a job that Timothy did on their side:
 has more “old” entries, which we should
copy at disarchive.guix.

We should also reuse the tools that Timothy wrote:
.  In particular, we should use project
infrastructure to compute a new Preservation of Guix report, and ideally
automate the whole process.

Maybe we should write down a road map and “recruit” on these items?

Thanks,
Ludo’.

Guix v1.1.0 (~2020) and Software Heritage & friends

[Simon Tournier]

Hi,

Here I would like to report my attempt to simulate a “network collapse”
– as discussed with Mathieu at Guix Days. :-)

What is the minimal set of machines for running “guix time-machine” from
a current Guix to an older one?

Using this,

archive.softwareheritage.org
disarchive.guix.gnu.org
git.savannah.gnu.org

is not enough.  It fails for many.  For instance,

ed-1.15
gcc-core-2.95.3
ghostscript-9.27
guile-2.2.6
linux-libre-4.19.56-gnu
linux-libre-headers-stripped-4.14.67-i686-linux
mes-0.19
mes-minimal-stripped-0.19-i686-linux

do not seem backed up with SWH + Disarchive.  Well, basically after
these, I stop because it’s boring and I add ftp.gnu.org to the list.  It
still fails,

nyacc-0.86.0
static-binaries

First lesson: we have holes in sources.json and maybe Disarchive too; I
will give a look.

To be continued…

Cheers,
simon

- - -

Using Guix b8f6ead (~February 2023) and trying to go to v1.1.0 (~April
2020) corresponding to this commit,

--8<---cut here---start->8---
$ cat /tmp/v1.1.0.scm
(list (channel
(name 'guix)
(url "https://git.savannah.gnu.org/git/guix.git;)
(commit
  "d62c9b2671be55ae0305bebfda17b595f33797f2")))
--8<---cut here---end--->8---

All is fine when substitutes is allowed and the network is fine.  So let
garbage collect all and restart reaching the minimal set of network.

Now, let disable the network name resolution and only resolve for
archive.softwareheritage.org. Then using the exact same command line as
previously, I get:

--8<---cut here---start->8---
$ guix time-machine -C /tmp/v1.1.0.scm -- help
Updating channel 'guix' from Git repository at 
'https://git.savannah.gnu.org/git/guix.git'...
substitute: updating substitutes from 'https://ci.guix.gnu.org'...   0.0%guix 
substitute: warning: ci.guix.gnu.org: host not found: Name or service not known
substitute:
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'...   
0.0%guix substitute: warning: bordeaux.guix.gnu.org: host not found: Name or 
service not known
substitute:
substitute: updating substitutes from 'https://ci.guix.gnu.org'...   0.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'...   0.0%
retrying download of '/gnu/store/8kv98fjwjp26ry1ixjdz28i1vjj9xfqi-config.scm' 
with other substitute URLs...
guix substitute: warning: bordeaux.guix.gnu.org: host not found: Name or 
service not known
guix substitute: error: failed to find alternative substitute for 
'/gnu/store/8kv98fjwjp26ry1ixjdz28i1vjj9xfqi-config.scm'
substitution of /gnu/store/8kv98fjwjp26ry1ixjdz28i1vjj9xfqi-config.scm failed
building /gnu/store/4a2p4vylpw1l0wz5szvyh4xgm0x5pl7f-config.scm.drv...
guix time-machine: error: some substitutes for the outputs of derivation 
`/gnu/store/lf70hiwqyz5qx418zjz7chn2lwq7nhic-module-import-compiled.drv' failed 
(usually happens due to networking issues); try `--fallback' to build 
derivation from source
--8<---cut here---end--->8---

which is a bit unexpected.  I miss why Guix does not fallback by itself
but that’s another story I guess.

Let re-run using --fallback as explained,

--8<---cut here---start->8---
$ guix time-machine -C /tmp/v1.1.0.scm --fallback -- help
Updating channel 'guix' from Git repository at 
'https://git.savannah.gnu.org/git/guix.git'...
substitute: updating substitutes from 'https://ci.guix.gnu.org'...   0.0%guix 
substitute: warning: ci.guix.gnu.org: host not found: Name or service not known
substitute:
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'...   
0.0%guix substitute: warning: bordeaux.guix.gnu.org: host not found: Name or 
service not known
substitute:
substitute: updating substitutes from 'https://ci.guix.gnu.org'...   0.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'...   0.0%
retrying download of '/gnu/store/8kv98fjwjp26ry1ixjdz28i1vjj9xfqi-config.scm' 
with other substitute URLs...
guix substitute: warning: bordeaux.guix.gnu.org: host not found: Name or 
service not known
guix substitute: error: failed to find alternative substitute for 
'/gnu/store/8kv98fjwjp26ry1ixjdz28i1vjj9xfqi-config.scm'
substitution of /gnu/store/8kv98fjwjp26ry1ixjdz28i1vjj9xfqi-config.scm failed
building /gnu/store/4a2p4vylpw1l0wz5szvyh4xgm0x5pl7f-config.scm.drv...
building /gnu/store/0xpdpf1az8p2i1zppn4s1w407f5b2ngg-git.scm.drv...
building /gnu/store/c70kbbbw6v9vj10fab7jnbjl9yv3s23j-hash.scm.drv...
building /gnu/store/cq303xhw6g86b5nyn1xc1gqr2i5bgn91-module-import.drv...
building /gnu/store/z4f6cy8lp1c90wvjzh738kdznxbii2vf-module-import.drv...
building 
/gnu/store/wjbqf99lhpyx84c6nqdvvl1lyhgi7igw-module-import-compiled.drv...
building 
/gnu/store/lf70hiwqyz5qx418zjz7chn2lwq7nhic-module-import-compiled.drv...
building