Re: Substitute retention

[zimoun]

Hey,

On Tue, 12 Oct 2021 at 18:04, Ludovic Courtès  wrote:

> (Moving to guix-devel from .)

I was preparing a report.  You have been faster than me. :-)

Two questions are raising, IIUC:

 1. the “modular” derivations for all the commits
 2. long-term support for substitutes


>> Other said 3388/6996= ~50% of commits are pushed at the same time, i.e.,
>> missed by both build farms using 2 different strategies to collect the
>> thing to build (fetch every 5 minutes or fetch from guix-commits).  It
>> is a quick back to envelope so keep that with some salt. :-)
>
> OK.

To make it explicit of #1, I was talking about the “modular” Guix, i.e.,
when running “guix pull” or “guix time-machine” it leads to build the
derivations module-import.drv, guix-.drv, guix-command.drv,
guix-module-union.drv, guix--modules.drv,
guix-packages-modules.drv, guix-system-tests-modules.drv,
guix-packages-base-modules.drv, etc.  On slow machines, it can be
unpleasant; not to say unpractical.  Even for recent commits.

The recent addition of ’channel-with-substitutes-available’ helps when
going forward (guix pull) if the build farm does not have yet these.

The issue is going backward (guix time-machine).

Basically, commit 59d10c3112 is from March 14, 2020 and it takes ~29min
on my slow laptop.  And to compare apple to apple, let take another
commit one year later from March 14, 2021, e.g., commit 7327295462.  It
takes ~5min on the same machine.

--8<---cut here---start->8---
$ time guix time-machine --commit=59d10c3112 -- --version
Updating channel 'guix' from Git repository at 
'https://git.savannah.gnu.org/git/guix.git'...
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
   /gnu/store/zvy89f9xb53fbqvfrm7lql8mbfrsfk1b-compute-guix-derivation.drv
   /gnu/store/7y80kn1bypnbm869hvcq8841mr6nqvfm-module-import-compiled.drv
   /gnu/store/amwvgaf45722k6jn4r39983zsgmbyp2g-module-import.drv
   /gnu/store/h3h0qfiw5100zkwfb919r7vn0q06ksqy-config.scm.drv
   /gnu/store/jkwhdilsbxb18hx6gi4i2rj0v06mfbab-module-import.drv
   /gnu/store/sixfy4sazai667n99pxa5h7wzzaabw79-module-import-compiled.drv

[...]

Computing Guix derivation for 'x86_64-linux'...  WARNING: (guix build 
emacs-build-system): imported module (guix build utils) overrides core binding 
`delete'
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%

[...]

substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
   /gnu/store/50ymxym19h8whzg3ajcl6kdjmq3p7qrg-profile.drv
   /gnu/store/l0znp7g83lbylbv97nd3ahz8rnrvxfrf-guix-59d10c311.drv
   /gnu/store/bvxrnp8bydl3zsbcg7j8j7m0qfygdhfs-guix-command.drv
   /gnu/store/xmsciylxx1j7nbry5cv1lm7595a8rilr-guix-module-union.drv
   /gnu/store/yvkw65kv9bhvx41750dchxw98qqxv64b-guix-59d10c311-modules.drv
   /gnu/store/6hrn3bpvcg8571mckzcj519xf2kqn2sl-guix-packages-base-modules.drv
   /gnu/store/nwl8z8cx9pdwn0sx5i5j5mp0bkdi64mm-guix-packages-base.drv
   /gnu/store/0j0271nbm2l526m5xs7zpd686qqrjz7w-guix-core.drv
   /gnu/store/2134jzhhckh871vlscw6dmwqqhny9zxg-guix-core-source.drv
   /gnu/store/mhik7ggrf4z5f38nsg7g0gbijm916b98-config.scm.drv
   /gnu/store/53zlyv96nrrm4h5ns7nmmndj8jys38f7-guix-extra.drv
   /gnu/store/7dn3f27i48jp6zvlwanzk8mfy828k6cm-guix-config-modules.drv
   /gnu/store/6xy3yyvjqvjyrlrwk1lzs12knbvilqpy-guix-config-source.drv
   /gnu/store/15ihwkqzyz4r4b4rppb92qcawha6a7p7-config.scm.drv
   /gnu/store/cacawv4yib8pa2ajzw0kyaihgym72mww-guix-config.drv
   /gnu/store/baivfv20hzm799v0wvdrcfaimh4aw22a-guix-extra-modules.drv
   /gnu/store/cxpkd0jkxapzbmg5vfmn4fy30yd7vlhm-guix-core-modules.drv
   /gnu/store/g3nqbybggh7dc2qd9gkj7swfmgmiigpp-guix-packages-modules.drv
   /gnu/store/nq9mzr00ny3nrsldvcq9r4va4fhb26sq-guix-packages.drv
   /gnu/store/i9dfjh5mf3r83447x7fa75hv1hnp9myv-guix-cli-modules.drv
   /gnu/store/xdsld8rhrawgngv93qx6lk9cgmql908c-guix-cli.drv
   /gnu/store/qnzm0a1gcwnfvkii7vk93rimzzp3mcf9-guix-system.drv
   /gnu/store/x2g2s3rhw0bv9qdp21yghgl2sij15dkr-guix-system-modules.drv
   /gnu/store/zandkjnylznvdj8jfsfgssvmvs6jfyph-guix-system-tests-modules.drv
   /gnu/store/45bs7y1h3gx2m7qry6r621klgkmv47wl-guix-system-tests.drv
   /gnu/store/caj7qjxvhrksk3jrkpsxqnx4kg7mlj9d-guix-daemon.drv
   /gnu/store/mah24wyy6bd51c27ww45hsqnjxhcn0yx-guix-manual.drv
   /gnu/store/002i672yl1192x7wvhkdbih94qffmcdk-guix-translated-texinfo.drv
   /gnu/store/5sf9aalvic81qlm06lq3a1pwdb2b3bm0-inferior-script.scm.drv
   /gnu/store/k612gnziiy3hn2dnrj33w4mw84kcnynm-profile.drv

11.7 MB will be downloaded

[...]

real29m14.588s
user0m56.126s
sys 0m1.032s
--8<---cut here---end--->8---

--8<---cut here---start->8---
$ time guix time-machine --commit=7327295462 -- --version
Updating channel 'guix' from Git repository at 
'https://git.savannah.gnu.org/git/guix.git'...

Re: Test parallelism with CMake

[Greg Hogan]

Hi Liliana,

The packages I am building do not disable parallel tests.

Greg

On Sat, Oct 9, 2021 at 3:56 AM Liliana Marie Prikler <
liliana.prik...@gmail.com> wrote:

> Hi Greg,
>
> for the purposes of GNU Guix, #:parallel-build? and #:parallel-tests?
> are distinct flags and the latter (if implemented) would apply to the
> check phase.  Our cmake-build-system in this case defers to gnu-build-
> system, which ought to insert -jN as you described.
>
> It could be that the package(s) you build have disabled parallel tests
> for some reason (breakage, reproducibility, ...).  If not there would
> be issues passing those arguments, which I don't believe the case from
> static analysis however.
>
> Cheers,
> Liliana
>
>

Substitute retention

[Ludovic Courtès]

Hi!

(Moving to guix-devel from .)

zimoun  skribis:

>> For the record, the ‘guix publish’ config on berlin is here:
>>
>>   
>> https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/modules/sysadmin/services.scm#n485
>>
>> If I read that correctly, nars have a TTL of 180 days (this is the time
>> a nar is retained after the last time it has been requested, so it’s a
>> lower bound.)

[...]

> Just for the record, a back to envelope computations.  180 days before
> today was April 15th (M-x calendar C-u 180 C-b).  It means 6996 commits
> (35aaf1fe10 is my current last commit).
>
> git log --format="%cd" --after=2021-04-15 | wc -l
> 6996
>
> However, these commits are pushed by batch.  Roughly, it reads:
>
> git log --format="%cd" --after=2021-04-15 --date=unix \
> | awk 'NR == 1{old= $1; next}{print old - $1; old = $1}' \
> | sort -n | uniq -c | grep -e "0$" | head
>   1 -1542620
>3388 0
>  14 10
>   6 20
>   5 30
>   2 40
>   4 50
>   1 60
>   2 70
>   2 80
>
> (Take the ’awk’ with care, I am not sure of what I am doing. :-)  And,
> it is rough because timezone etc.)
>
> Other said 3388/6996= ~50% of commits are pushed at the same time, i.e.,
> missed by both build farms using 2 different strategies to collect the
> thing to build (fetch every 5 minutes or fetch from guix-commits).  It
> is a quick back to envelope so keep that with some salt. :-)

OK.

> On that number, after 180 days (6 months), it is hard to evaluate the
> rate of the time-machine queries.  And from my experience (no number to
> back), running time-machine on a commit older than this 180 days implies
> to build derivations.  Or it is a lucky day. :-)

Right.

So what can we do to address this issue?  I *think* we could use a
higher TTL on berlin, and we can try that right away (9 months to being
with?).

However, there is an upper bound anyway.  To make informed decisions on
the retention policy, we should monitor storage space on berlin/bayfront
to better estimate what can be done.  We have Zabbix but it’s not
accessible from the outside; maybe we could graph storage space
somewhere so people can grab the data and work on those estimates?

What if we decide that we need to provide substitutes for 2y old
commits?  In that case, we need a plan to scale up.  That could be
renting storage space somewhere.  That’s largely non-technical work that
needs attention.

There are also technical tweaks that could help: distinguishing between
“important” substitutes that we want to keep, and less important
substitutes (how?); identifying “equivalence classes” for builds of a
given package; etc.  The outcome is unclear and it’ll take time.

Thoughts?

Ludo’.

Re: Disarchive update

[zimoun]

Hi Ludo,

On Sat, 09 Oct 2021 at 12:05, Ludovic Courtès  wrote:

> If you run:
>
>   guix build 
> /gnu/store/nnl67m8c2x9rwqbnych1agc6p7g5473g-disarchive-collection.drv

Oh, cool!

> and if you’re patient :-), you eventually get a 579 MB directory
> containing Disarchive metadata for 8,413 tarballs out of 9,113 (the
> missing tarballs are those that “disarchive disassemble” fails to
> handle, for instance because it couldn’t guess what compression method
> is being used.)

Timothy made this table months ago:

tar+gz9090  52.0%
git   5294  30.3%
tar+xz1184  06.8%
tar+bz2775  04.4%
tar393  02.2%
zip273  01.6%
svn-multi  175  01.0%
svn125  00.7%
file51  00.3%
computed38  00.2%
hg  36  00.2%
unknown-uri 20  00.1%
tar+gz? 15  00.1%
tar+lz  13  00.1%
tar+Z4  00.0%
cvs  3  00.0%
bzr  3  00.0%
tar+lzma 2  00.0%
total17494 100.0%

What is really missing is XZ and Bzip2 support in Disarchive, I guess.


> Where to go from here?  Timothy Sample had already set up a Disarchive
> database at , which (guix download) uses
> as a fallback; I’m not sure exactly how it’s populated.  The goal here
> would be for the Guix project to set up infrastructure populating a
> database automatically and creating backups, possibly via SWH (we’ll
> have to discuss it with them).

Timothy was working on feeding the database using each release.  Well,
you can give a look at:



Then something along these lines:

$ sqlite3 /tmp/pog.db < schema.sql
$ guix repl -L . <(echo '
  (use-modules (pog))
  (ingest "6298c3ffd9654d3231a6f25390b056483e8f407c"
  "/tmp/pog.db")
  ')

for where the commit hash corresponds to v1.0.0.  I do not know if it
would be equivalent to run:

   guix time-machine --commit=6298c3ffd9654d3231a6f25390b056483e8f407c \
-- build -m etc/disarchive-manifest.scm


> A plan we can already deploy would be:
>
>   1. Add the disarchive.guix.gnu.org DNS entry, pointing to berlin.
>
>   2. On berlin, add an mcron job that periodically copies the output of
>  the latest “disarchive-collection” build to a directory, say
>  /srv/disarchive.  Thus, the database would accumulate tarball
>  metadata over time.
>
>   3. Add an nginx route so that /srv/disarchive is served at
>  https://disarchive.guix.gnu.org.
>
>   4. Add disarchive.guix.gnu.org to (guix download).

To replace (or add to) the current ’%disarchive-mirrors’ right?

Going this road (use Cuirass), why not generating the sources.json
similarly?   Instead of the hack using the website builder.


On my side, I will try to resume what I started months ago: knowing the
SWH coverage.  For instance, on this ~92% of tarballs, how many are
currently stored into SWH?  Well, do not take your breath and I would be
happy if someone beats me. ;-)


Cheers,
simon

Re: Disarchive update

[Mathieu Othacehe]

Hey,

> That’d be nice!  How do we do that again?

The build-outputs field of the  record must be used as
explained here:
https://guix.gnu.org/cuirass/manual/html_node/Specifications.html#Specifications.

This field cannot be manipulated via the web interface yet.

I think the easier way to proceed would be to create the "disarchive"
specification in the (maintenance sysadmin services) module, this way:

--8<---cut here---start->8---
(specification
 (name "disarchive")
 (build '(manifests "etc/disarchive-manifest.scm"))
 (build-outputs
  (list
   (build-output
(job "disarchive-collection*")
(type "archive")
(path ""
 (notifications #$(cuirass-notifications))
 (period 43200)
 (priority 7)
 (systems '("x86_64-linux")))
--8<---cut here---end--->8---

I can take care of that if it's ok for you.

Thanks,

Mathieu