Start of 2022 update on bordeaux.guix.gnu.org (over 1,000,000 successful builds!)

[Christopher Baines]

Hey!

It's not quite been a year since bordeaux.guix.gnu.org started
operating, but I thought I'd send out an update email towards the start
of 2022 looking back in to 2021 and looking forward to the rest of this
year.

If you're missing some context, this blog post should be useful, and has
lots of useful links:

  
https://guix.gnu.org/en/blog/2021/substitutes-now-also-available-from-bordeauxguixgnuorg/

The last update was back in December [1] and talked about addressing the
nar capacity, the machines building things, and IPv6 support (something
I'm glad to see).

1: https://lists.gnu.org/archive/html/guix-devel/2021-12/msg00140.html

### Looking back

First, some data.

The first build was created back on the 9th of April, 2021.

So far the build coordinator behind bordeaux.guix.gnu.org has been asked
to build nearly 2 million things (> 1,939,000).

Of those builds, around 824,100 haven't been processed, and > 1,114,900
have been processed.

Of those processed builds, > 113,338 failed, and > 1,001,567 succeeded.

From those successful builds, bordeaux.guix.gnu.org provides > 914586
nars. That's 3.6TiB of lzip compressed data (~11TiB's uncompressed).

The Guix Build Coordinator project began with two use cases in mind,
building things to provide substitutes, and building things to perform
quality assurance tasks. The substitutes use case is the relevant one
here, so I'm just focusing on that. Regarding substitutes, there were a
few different aims in mind: things like making operating substitute
servers easier, increasing hardware utilisation and improving substitute
availability.

I guess the baseline for comparison has changed since ci.guix.gnu.org no
longer uses offloading to perform most builds, but I think there's
reason to think that progress has been made and is being made on the
above aims.

Since bordeaux.guix.gnu.org came in to existence, not all that much has
changed with the Guix Build Coordinator, I think most of the changes
have been tweaks and optimisations.

### Looking forward

I think the most important area for improvement is data.guix.gnu.org
which has been struggling to keep up with new revisions lately. I think
as Guix grows, both in terms of packages and supported architectures,
this additional data has slowed the revision loading process. I've got
some ideas on improving this, and I'm hoping to work on it soon.

Something I've been working on recently is neatening up the
configuration for the machines involved, there's a bit more to do on
this. It would also be beneficial to add more machines and expand the
supported architectures. Also in terms of the operation, getting regular
automated database backups in place, and increasing the redundancy and
capacity for the nar storage would be good.

Specifically on the nar capacity issue, assuming that the retention is
kept at 100%, I think it's likely that the available storage (currently
there's a rough limit of 5.5TiB) will need expanding this year. Now that
the nar-herder exists, this should be much less stressful than the
previous work to expand the available storage (which involved writing
the nar-herder from scratch).

On the subject of the nar-herder, I think continuing to work towards
having mirrors in different geographical locations is important to
improve substitute performance for everyone. I sent out an email
recently to resume discussion of the nar-herder design [2]. I also want
to look at gathering metrics about substitute use, probably through the
nar-herder, and that's somewhat dependent on mirrors, as I want any
metrics gathered to be across several mirrors.

2: https://lists.gnu.org/archive/html/guix-devel/2022-02/msg00033.html

It would also be nice to make the inner workings of the build
coordinator behind bordeaux.guix.gnu.org more visible, so everyone can
see what builds are happening.

Finally, it would be good to work towards bordeaux.guix.gnu.org building
packages for non-master branches. This would help when preparing for
staging/core-updates merges. I think this mostly involves properly
setting up a data.qa.guix.gnu.org data service and enabling substitutes
from this on the build agents.

Please let me know if you have any comments or questions. I think there
will also be time for discussion at the upcoming Guix Days event.

Thanks,

Chris


signature.asc
Description: PGP signature

Re: Maven Build System Dependency Issue

[Phil]

Julien Lepiller writes:

> Hi Phil,
>
> I have already seen the issue previously, and I believe it is now fixed
> on master. As you can see here,
> https://github.com/guix-mirror/guix/blob/master/gnu/packages/java.scm#L7899
> we now propagate the correct parent pom.

Thanks Julien that looks exactly as I'd expect now.  Out of curiosity
was this change only merged to master in the last few days?  When I
looked on Guix mirror I ended up seeing the old 50 version just a few
days ago?  It's always possible I just looked at the wrong branch by
accident!

One last question - I note your change was committed on 21st Dec, which
is bad luck for me as the Python 3.9 upgrade went in on 17th Dec, and
I'm stuck for non-Guix reasons on 3.8.2 (so my Guix is a bit stale for
now).  I'll fix this medium-term on my side, but was wondering if there
is any way around this?

None of my previous e-mail's attempts worked, but using Guix Inferiors
this morning to make "guix" itself the inferior (rather than the maven
package), my pacakge then actually built.  Adopting the example given
in the manual, I was able to create a profile out of the below manifest,
which if I then run "guix build" inside it downloaded and built the older
build system on the fly, including java-commons-codec@1.14 - which is
pretty cool.

I could then continue to add packages from current guix alongside this,
although I'm not sure if this is a particularly good idea!

Given that I want to rewind the build system itself rather than a
specific package is this the right/canonical way to do this?  There
seems no way of using --with-inputs or --with-source to do this,
presumably because you need to make the change *before* guix itself is
called?


(use-modules (guix inferior) (guix channels)
 (srfi srfi-1))   ;for 'first'

(define channels
  ;; This is the commit id just before java-commons-codec was upgraded 1.14 -> 
1.15
  (list (channel
 (name 'guix)
 (url "https://git.savannah.gnu.org/git/guix.git;)
 (commit
  "a3b6e904484e23db65990be70d76ba32c15fd03f" ;; old fix

;;"a348520e2a253bc81fa92566e74f8b3e60fea058" ;; future fix

(define inferior
  ;; An inferior representing the above revision.
  (inferior-for-channels channels))

;; Now create a manifest contain guix itself rewound to where 
java-commons-codec=1.14
(packages->manifest
 (list (first (lookup-inferior-packages inferior "guix"

;;(specification->package "maven"))) ;; add other packages as required

Re: Dropping gzip-compressed substitutes

[Efraim Flashner]

On Tue, Feb 08, 2022 at 02:34:49PM +0100, Mathieu Othacehe wrote:
> 
> Hey Maxim,
> 
> Sound like a fine plan.
> 
> > 1.  Promptly set up both a blog post and a NEWS entry announcing the
> > support for gzip substitutes is about to be phased out from the build
> > farm (1 month notice), urging users to upgrade their daemon to a version
> >>= 1.1.0.
> 
> I addition, we could warn users with old daemons as proposed by Ludo &
> Ricardo. There's an attached patch here.
> 
> > -   ;; TODO: Eventually, disable gzip, as discussed at
> > -   ;; 
> > .
> > -   (compression '(("gzip" 9) ("lzip" 9) ("zstd" 19)))
> > +   (compression '(("lzip" 9) ("zstd" 19)))
> > (cache-bypass-threshold cache-bypass-threshold)
> > (workers publish-workers)))
> 
> Nice.
> 
> > 3. Come up with a Guile script that is able to
> >
> >   a) Strip gzip-related metadata from the .narinfo guix-publish metadata
> >   files
> >   b) recompute and update their 'Signature' field.
> >
> > 4. Finally, 'rm -r /var/guix/publish/gzip' and free about 6.5 TiB of data.
> 
> I hope the script will be able to complete within a reasonable amount of
> time, but we cannot know before trying it out :).

I'm sure the internet is full of people who know if this is fastest, or
using find /var/guix/publish/gzip -delete or find ... -print0 | xargs -0
rm or something else :)


-- 
Efraim Flashner  רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted


signature.asc
Description: PGP signature