Re: Cuirass upgrade on ci.guix

[Mathieu Othacehe]

Hey,

>   1. Better throughput.  I fixed a bug that would lead “workers” to
>  eventually get ignored by ‘cuirass remote-server’, meaning they
>  would become idle until either the server or the worker is
>  restarted¹.

Wow, that's a nice catch!

>   2. Evaluation can be triggered via POST requests, which should make
>  things a bit more reactive and will allow us to poll repos less
>  frequently.  I submitted a support request on Savannah:
>  .
>
>   3. Improved resilience against “evaluation storms” since ‘cuirass
>  register’ now has an upper limit on the number of evaluations that
>  may occur in parallel.
>
>   4. Build priority handling has been restored (it had been lost
>  recently).

Nice as well! Thanks for your awesome contributions to Cuirass :)

Mathieu

Re: Implementing the guix-dameon in Guile

[Mathieu Othacehe]

Hey Chris,

Congrats on getting the grant! I'm sure it will lead to a lot of
positive things. I tried it myself a couple years ago, for
the exact same subject, without success. 

> I imagine the daemon could be structured as a set of actors (it’s really
> my thing these days ;-)), with an eye on facilitating code sharing and
> interaction with the Coordinator, Cuirass, and all that.

I think this point is really important here. If the offload mechanism of
the existing guix-daemon was scaling correctly, with a better API, we
would probably never had to develop the Cuirass remote worker and the
Build Coordinator mechanisms.

Perhaps, part of the Build Coordinator code could be one day integrated
inside the new guix-daemon so that we can only use one offload mechanism
for all use cases. Users of the current "guix offload" as well as CI
tools could rely upon a unique offload mechanism.

The three other points I had in mind when proposing the subject to NLNet
were to:

- Fix the sqlite contention issues that have been observed on Berlin and
  other machines with huge local databases.

- Remove the GC lock: I think it has been done on recent nix-daemon.

- Add support for unprivileged daemon.

I don't know if it fits your plans but thought it could be interesting
to share that.

Good luck,

Mathieu

Cross compilation status

[Mathieu Othacehe]

Hello,

In order for Guix to become an alternative to tools such as Yocto and
Buildroot, having most or all our packages cross-compiling is a
prerequisite.

Here is a status of cross-compilation in Guix. For cross-compilation to
work, the build-system needs to support cross-compilation.

The following build-systems explicitly refuse cross-compiling packages:

haskell, agda, waf, chicken, rakudo, julia, python, emacs, rebar, cargo, ruby, 
renpy, dub, android-ndk, scons, dune, ant, pyproject, maven, asdf, r, ocaml, 
node

while the rest of the build-systems do accept cross-compiling packages:

clojure, qt, copy, minetest-mod, tree-sitter, raw, linux-module, glib-or-gtk, 
asdf/source, go, cmake, minify, perl, trivial, guile, elm, font, gnu, asdf/ecl, 
asdf/sbcl, meson, mozilla, texlive

It does not mean that all the packages relying on those build-systems
will cross-compile OK. It just means that we will at least try to build
those.

In term of numbers, the attached script produces the following output:

--8<---cut here---start->8---
Total packages: 28076

clojure: 13
qt: 317
copy: 134
minetest-mod: 22
tree-sitter: 35
raw: 2
linux-module: 17
glib-or-gtk: 139
asdf/source: 794
go: 632
cmake: 1278
minify: 12
perl: 839
trivial: 250
guile: 49
elm: 30
font: 101
gnu: 4178
asdf/ecl: 753
asdf/sbcl: 814
meson: 504
mozilla: 2
texlive: 4150
Cross-compilation OK: 15065

ocaml: 61
haskell: 812
maven: 2
chicken: 12
node: 57
emacs: 1370
dune: 289
android-ndk: 12
waf: 21
julia: 300
pyproject: 433
r: 2441
cargo: 3535
ruby: 597
rebar: 22
scons: 15
rakudo: 22
agda: 6
ant: 559
python: 2488
Cross-compilation KO: 13054
--8<---cut here---end--->8---

Over the 28076 packages in Guix, 15065 are part of build-systems with
cross-compilation support and 13054 are part of build-systems without
cross-compilation support.

Overall 46.5% of our packages will refuse to cross-compile with errors
such as:

--8<---cut here---start->8---
mathieu@meije ~$ guix build --target=aarch64-linux-gnu librsvg
guix build: error: gnu/packages/gnome.scm:3500:2: librsvg@2.54.5: build system 
`cargo' does not support cross builds
--8<---cut here---end--->8---

I'd like to help converge towards the situation where all build-systems
support cross-compilation.

I have CC'ed members of the python, java, ruby, rust, r, haskell and
emacs teams. Any plans adding cross-compilation support to your
build-system, barriers to overcome?

Thanks,

Mathieu


cross.scm
Description: Binary data

Re: 01/09: services: base: Add extra-env support to guix-configuration.

[Mathieu Othacehe]

Hey,

> Nitpick: To stick to existing naming conventions, could we rename it to
> ‘environment-variables’ or ‘environment’?

I changed to environment and pushed.

Thanks,

Mathieu

Re: 04/08: gnu: QEMU: Unbundle iPXE.

[Mathieu Othacehe]

Hello Marius,

> glib
> gtk+
> +   ipxe-qemu
> libaio
> libcacard;smartcard support
> attr libcap-ng   ;VirtFS support

I just noticed that:

--8<---cut here---start->8---
./pre-inst-env guix build --system=aarch64-linux grub
/home/mathieu/guix/gnu/packages/bootloaders.scm:101:2: warning: package 
grub@2.06 does not support aarch64-linux
--8<---cut here---end--->8---

it seems to be because of the following dependency path:

--8<---cut here---start->8---
grub <- qemu-minimal <- ipxe-qemu <- ipxe <- syslinux
--8<---cut here---end--->8---

with syslinux that only supports x86_64-linux and i686-linux.

I'm not sure how to break that path. It looks like qemu-minimal is
required by grub for test purposes. Maybe we could restrict that
dependency and the tests to x86_64-linux and i686-linux?

WDYT?

Thanks,

Mathieu

Re: Has ci.guix.gnu.org stopped building?

[Mathieu Othacehe]

> This was fixed with commit 658c09333da095edf6e1b3c5e351a7bfa3c87354

Builds have restarted!

Good catch, thanks!

Mathieu

Re: Has ci.guix.gnu.org stopped building?

[Mathieu Othacehe]

Hello Leo,

> According to the web interface, ci.guix.gnu.org stopped building things
> on December 31, with evaluation 79343.
>
> Is this true? What's wrong?

Yes the master specification has not been evaluated since December 30.
It looks like all subsequent evaluations are never finishing which
causes a database slots starvation causing even more evaluations to
fail.

A good way to investigate that is often to run `make cuirass-jobs`
locally. I'll try to do that today or tomorrow.

Thanks,

Mathieu

Re: Release progress, week 8

[Mathieu Othacehe]

Hey Ludo,

>   fe563a87ad gnu: texinfo, info-reader: Do not run tests when cross-compiling.

Wow, that's a lot of fixes!

> Yesterday on IRC we discussed an installer crash received at
> dump.guix.gnu.org.  Did you or will you have time to look into it?

Yes and I think that for some reason we are not detecting the
installation device properly. However, as I cannot reproduce it, we
would need to find someone kind enough to run an instrumented installer
to understand this issue.

We can also add a few traces to the syslog and fix this issue later on,
depending on our schedule.

Thanks,

Mathieu

Re: Release progress, week 8

[Mathieu Othacehe]

Hey,

Thanks for your hard work to publish the RC1!

> Now we need reports from users to act upon.  I’d say we can decide next
> week whether we need an RC2 or not.  I can handle the next release
> candidate or the release itself, but I’ll be unavailable on Dec. 12–15.

I noticed that we have failing tests on the version-1.4.0-tests
specification.

- docker-system (https://ci.guix.gnu.org/build/215380/details)
- prosody (https://ci.guix.gnu.org/build/215424/details)

that fail both on Berlin and on my machine and,

- fail2ban-extension (https://ci.guix.gnu.org/build/215447/details)

that only fails on Berlin.

There is also an issue with the system images. The following images are
failing on Berlin:

- novena-barebones-raw-image (https://ci.guix.gnu.org/build/215363/details)
- pine64-barebones-raw-image (https://ci.guix.gnu.org/build/215361/details)
- pinebook-pro-barebones-raw-image 
(https://ci.guix.gnu.org/build/215362/details)

It seems that those failures are caused by an issue during info-reader
build. I cannot reproduce it locally though.

Having an RC2 with all the tests passing and images buildable could be
nice.

On the other hand, I don't have much bandwidth to fix those issues and
they should not prevent us from releasing I guess.

Mathieu

Re: Release progress, week 6

[Mathieu Othacehe]

Hello,

>It would seem that a lot of aarch64 builds are queued but not
>getting processed.  Any idea how to investigate that, Mathieu?

That would be because we only had a single ARM machine available till
this week (kreuzberg). It seems that overdrive1 and grunewald are back
though!

I also noticed that my recent remote worker fix was not working as
expected. The workers are not blocked anymore after a remote server
disconnection but some builds are flagged as submitted even though they
are not received by the worker.

>  Should we send out a call for testing the installer?  Looks like
>  we’re ready, no?

Sure, I'll take care of that on Monday.

>   • Release issue  blocked by 2 open
> bugs, down from 5!
>
> 58221 nautilus: Crashes loading KgxNautilus plugin twice (problems with 
> NAUTILUS_EXTENSION_PATH)
> 53594 no matching pattern #  - Shepherd 0.9.3 release fixes two bugs!
>
>

Thanks for your work on that topic. It looks like we are getting closer :)

Mathieu

Re: i586-gnu builds on ci.guix

[Mathieu Othacehe]

Hello,

> Alas Cuirass depends on Fibers, which is currently Linux-only¹, so we
> cannot {cross-,}build it for GNU/Hurd.  The good news is that
> ‘remote-worker.scm’ itself doesn’t seem to use Fibers.  So I wonder if
> we could arrange to build a stripped-down package that contains nothing
> but ‘cuirass-remote-worker’.  WDYT?

Yes, Fiber is not used much anymore, except for the web server.

> Cuirass also depends on Avahi, which is currently not buildable for
> GNU/Hurd due to libcap requiring on Linux-specific headers, and it seems
> to be a hard requirement².  Unfortunately ‘remote-worker.scm’ requires
> it.

The remote worker is able to discover the remote server using Avahi but
can also connect directly to the server using its IP address.  We could
definitely make the Avahi dependency optional.

Thanks,

Mathieu

Re: Release progress, week 5

[Mathieu Othacehe]

Hey,

>Probably we’ll postpone that post-release and drop i586-gnu from
>‘etc/release-manifest.scm’ in the meantime.  Objections?

No.

>  - armhf-linux: No progress; can we arrange so that ci.guix at least
>builds these core subset of packages?

We already have the 'core' specification setup to build armhf-linux
packages. I force restarted the latest builds for that specification
this way:

--8<---cut here---start->8---
update builds set status = -2 where id in (select build from jobs where 
evaluation = 814784 and system = 'armhf-linux');
--8<---cut here---end--->8---

Thanks,

Mathieu

Installer for 1.4.0

[Mathieu Othacehe]

Hello,

The installer has been hardened during the past months when it comes to
error reporting:

- Guile exceptions are caught, displayed and possibly uploaded to
  dump.guix.gnu.org if the user agrees to do so.

- Failing system commands can be run again.

I think that it would be nice to go a bit further as people tend to use
the stable Guix installer image and given our current release rate, the
future 1.4.0 could last for long.

It would be awesome if we could tackle the following issues before the
release:

- Guile and library segfaults could be reported as proposed here:
  https://issues.guix.gnu.org/58733.

- There is a nasty finalizer bug that has probably bitten a lot of users
  here: https://issues.guix.gnu.org/58732.

- The backtrace page is not displayed correctly if an exception occurs
  withing the final PTY: https://issues.guix.gnu.org/58734. This one is
  maybe not as important but it would be great to have it fixed so that
  every part of the installer is well covered by the dump mechanism.

- Displaying upfront that the WiFi, video card or any other hardware
  won't be supported by the installer/linux-libre in the spirit of:
  https://issues.guix.gnu.org/58549 would be great.

  For instance, I tested recently the installer on a laptop with an
  unsupported WiFi card. The WiFi page is skipped altogether by the
  installer which forces the user to plug an ethernet cable without
  explaining why. I think we could be a bit more didactic.

Any volunteers?

Thanks,

Mathieu

Re: Status of armhf-linux and powerpc64le-linux

[Mathieu Othacehe]

Hey,

> How frequently does that machine become unreachable?
>
> Its uptime right now is “only” 51 days, but it seems to have been
> reliably building things so far (surprisingly so!).

Oh so it must be available from the Cuirass point of view but not from
the guix offload point of view. I'll try to fix it today.

> In Cuirass, we should arrange to support partial evaluations or
> per-system evaluations so that a single missing offload machine doesn’t
> cause the whole evaluation to fail.

We can define a guix specification for x86_64-linux, i686-linux and
aarch64-linux and a different one for powerpc64le-linux. That can be
done really quickly. That can break some mechanisms relying on the fact
that the guix specification name is "guix" though.

> That’s radical, but maybe that’s the most reasonable option.
>
> How about a plan like this: until next Thursday, we try to address the
> infrastructure issues discussed above to estimate feasibility.  Then we
> decide on the way forward.  WDYT?

Alright, let's try that :). I agree that it is a pity not to release for
those architectures but on the other hand, we cannot offer fresh
substitutes reliably for them.

The recent outages of ci.guix.gnu.org have shown once again that the
infrastructure is maybe one of the most important Guix aspect. Without
it, Guix is almost unusable, in particular on architectures for which
it's hard to find powerful hardware.

Given the limited amount of people willing to help for
powerpc64le-linux and armhf-linux and the limited amount of hardware
resources available for those, I think it could be reasonable to focus
on a smaller set of architectures and provide stronger guarantees on
those in term of substitutes availability.

But let's discuss that again next week depending on our progress.

Thanks,

Mathieu

Re: Release progress, week 2

[Mathieu Othacehe]

Hey,

Thanks for the update!

>  - armhf-linux is disabled on ci.guix due to improper offloading
>setup (probably along the lines of
>).  Should we try and reenable
>it, or should we drop it?
>
>  - powerpc64le-linux is disabled on ci.guix since today
>(maintenance.git commit
>d641115e20973731555b586985fa81fbe293aeca).  However it did work
>until recently and we have one machine to offload to.  Should we
>fix it or drop it?  Mathieu?

Yeah, we only have a single machine to offload to and each time it is
not reachable, the "guix" specification fails on Cuirass.

That's because we need to offload to a powerpc64le-linux machine in
order to evaluate the guix derivation for that specific architecture
(that's true for all the other architectures).

Given the lack of workers for powerpc64le-linux I think we should drop
it. Regarding armhf-linux we can in theory rely on the overdrives but
we are already struggling on aarch64-linux, we I think we should also
drop it for now.

Focusing on x86_64-linux, i686-linux and aarch64-linux for this release
seems more pragmatic.

>  - ‘guix system init’ will print progress bars instead of dots
>again (patch available): .

Just merged the fix that you reviewed :)

>   • Release issue  still blocked
> by 8 open bugs that we should review!

I'll take care of #53194 and #53541. I just closed #52943. It means that
we will have to take care of:

57068 important, Ludovic CourtèsResizing mcron job in vm-image.tmpl 
interferes with settings
53594 important, Guu, Jin-Cheng no matching pattern #

Maintainers meeting notes.

[Mathieu Othacehe]

Hello,

For the last six months or so, the maintainers have been conducting
monthly meetings on Jami.  They are scheduled every first Tuesday of the
month. It is often the occasion to plan some actions and discuss the
administration of the build farm.

The meeting notes are available here:
https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/maintainers/meetings

and the notes for the October meeting are here:
https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/maintainers/meetings/2022-10-04.org

Nothing super fancy, but we will try to post a small note here every
month in case some people might be interested.

Thanks,

Mathieu

Re: branch master updated: services: Add file system utilities to profile.

[Mathieu Othacehe]

Hey,

> So it would seem like e2fsprogs is not being added to the profile of the
> installation image.

I think that's because there is no ext4 file-system needed for boot, as
root is an iso9660 file-system. I added e2fsprogs to the
operating-system packages for this specific test with:
0f66ef9aa99d2043abccbc80d858bdeca57534ac.

Thanks,

Mathieu

Pick your team!

[Mathieu Othacehe]

Hello,

We recently introduced a teams.scm[1] file defining different teams such
as "Julia team", "Lisp team" or "Mentors".

If you are a newcomer seeking for some guidance to contribute to a Lisp
package, you can run the following command:

--8<---cut here---start->8---
mathieu@meije ~/guix [env]$ ./etc/teams.scm list-members lisp
Guillaume Le Vaillant 
jgart 
--8<---cut here---end--->8---

so that you know who can help you with you endeavor.

Now, we are going a step further, and by running:

--8<---cut here---start->8---
mathieu@meije ~/guix [env]$ git send-email --to XXX@@debbugs.gnu.org 
$(./etc/teams.scm cc-members HEAD~2 HEAD) *.patch
--8<---cut here---end--->8---

the teams concerned by the two last commits you are sending to review
will automatically by put in CC.

This is automatic but not magic! It works by defining a scope of files
that are mentored by a team. For instance, the installer team has the
following scope:

--8<---cut here---start->8---
(define-team installer
  (team 'installer
#:name "Installer script and system installer"
#:scope
(list (make-regexp "^guix/installer(\\.scm$|/)"
--8<---cut here---end--->8---

So, we can all improve this mechanism by:

1. Defining your team scope, if you are already part of a team

2. Joining a team

To do that, you can send patches to the etc/teams.scm.in file.

Thanks,

Mathieu

[1]: https://git.savannah.gnu.org/cgit/guix.git/tree/etc/teams.scm.in

Re: 02/04: services: Add samba service.

[Mathieu Othacehe]

Hello,

Thanks for this new service :) I noticed that while the "samba" test
works fine on my machine, it fails on the CI:

https://ci.guix.gnu.org/build/1495525/log/raw

any idea why?

Mathieu

Re: 04/07: guix: Add compression module.

[Mathieu Othacehe]

Hey Ludo,

> While I agree this should be factorize, I wonder if this should be
> promoted to (guix compression).  The module name intuitively is a better
> fit for the compression-related things currently in (guix utils).
>
> Not sure, maybe (gnu compression) would make more sense?

Thanks for noticing that. I moved this module to (gnu compression),
checked that make as-derivation is working and updated the wip-image
branch accordingly.

Mathieu

Re: 02/02: gnu: shepherd: Update to 0.9.2.

[Mathieu Othacehe]

Hey Ludo,

The installation system tests have fail this night and I suspect this is
related with the shepherd update. The problem seems to occur when we
restart the guix-daemon service before running the installation command,
in the install-system procedure. Any idea what could cause it?

The cgit and gitile tests also fail, but with the following error:

--8<---cut here---start->8---
  result-kind: fail
  actual-value: #f
  actual-error: (read-error #f "socket:5:14: Unknown # object: ~S" ("#<") #f)
--8<---cut here---end--->8---

Thanks,

Mathieu

Re: branch master updated: gnu: gitolite: Wrap programs instead of using propagated inputs.

[Mathieu Othacehe]

Hello David,

I noticed that the gitolite test is now failing. Any chance it is
related to the recent work on the package?

https://ci.guix.gnu.org/build/1333185/details

Thanks

Mathieu

Re: 'staging' freeze

[Mathieu Othacehe]

Hey Marius,

> The 'staging' branch is in a pretty good shape, let's get it merged!

Nice work!

> I'm fairly rusty when it comes to Cuirass, and don't see a button to
> start the jobset here even when authenticated:
>
>   https://ci.guix.gnu.org/jobset/staging
>
> Can someone remind me how to enable a disable jobset?  :-)

No easy way for now:

--8<---cut here---start->8---
psql -d cuirass -c "update specifications set is_active = 1 where name = 
'staging';"
--8<---cut here---end--->8---

does the trick.

Thanks,

Mathieu

Re: branch master updated: gnu: python-lxml: Update to 4.6.5.

[Mathieu Othacehe]

Hey

> * gnu/packages/xml.scm (python-lxml): Update to 4.6.5.

I had to revert it as it caused >15k rebuilds. I also cancelled all the
triggered builds in the CI.

Thanks,

Mathieu

Re: Teams

[Mathieu Othacehe]

Hello Ricardo,

I would suggest to extend a bit the scope of this idea. What about
creating an etc/tutors.scm file as the one attached.

This way people would run something like:

--8<---cut here---start->8---
git send-email $(get-tutors.scm HEAD^^) *.patches
--8<---cut here---end--->8---

The get-tutors.scm command would take a git reference as argument. From
this git reference the list of edited files would be extracted. Once
matched with the modules field of tutors.scm, the ML & tutors that
should be CC'ed would be returned.

For the previous example, if the patches are modifying (gnu packages
bioconductor), the command would be:

--8<---cut here---start->8---
git send-email --to guix-patc...@gnu.org --cc guix-r-patc...@gnu.org
--cc rek...@elephly.net --cc zimoun.touto...@gmail.com *.patches
--8<---cut here---end--->8---

People could subscribe to the relevant ML if they want to follow
specific subjects. If someone wants to become a tutor, a patch could be
sent against the tutors.scm file.

Being listed in the tutors.scm file would imply some duties, such as
trying to review part of the traffic for the tutored parts.

We could then turn this tutors.scm file into a website page,
transforming the tutors SEXP into an HTML table.

WDYT?

Thanks,

Mathieu


tutors.scm
Description: Binary data

Re: Donation of used Softiron Overdrive 1000

[Mathieu Othacehe]

Hello Julien and Reza,

Thanks for the donation! We already have two Overdrive 1000 connected to
the build farm behind ci.guix.gnu.org (overdrive1 and dover
machines). Adding an extra one, and have it hosted at your place Julien
would be great :).

You can then follow the procedure described here:
https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/cuirass.org,
to add it to the build farm. I will be happy to help you, if needed at
that time.

Mathieu

Re: Dropping gzip-compressed substitutes

[Mathieu Othacehe]

Hey Ludo,

> As discussed on IRC, I’m skeptical about this because:
>
>   1. It requires the development and testing of a custom tool that’s
>  easy to get wrong—e.g., it removes a gzipped nar for something that
>  had nothing but gzip available, etc.
>
>   2. That code would have to run with privileges that give it access to
>  the signing key on berlin.
>
>   3. Those 6.5 TB are an initial constant factor; growth of the storage
>  requirements going forward probably matters more and
>   will give us more flexibility
>  on that.

While those are valid points, we need to keep in mind that it is
important that we manage to move the store to the new SSD array quite
quickly to start GCing again.

If we cannot manage to remove those gzip nars then, I see only two
alternatives:

* Host the nars on the HDD array only.

* Host the nars elsewhere, on a VPS as you are proposing.

> I like Chris Baines’ idea of decoupling nar distribution from nar
> building.  If we want to keep nars long enough so that ‘time-machine’ is
> usable, then storage requirements will keep growing.
>
> Perhaps that means we can regularly copy nars “elsewhere” for long-term
> storage, using nar-herder, rsync, or whatever.  The machine that stores
> nars long-term has low requirements compared to the build farm because
> we don’t need to trust it for anything other than storage.  If that
> makes things easier (and financially viable), a VPS is good enough.

Sure, the VPS would also allow us to have a less European-centric
hosting. I did not follow closely the development of the
nar-herder. Chris what improvements this tool would bring compared to a
rsync based approach?

Thanks,

Mathieu

Re: Dropping gzip-compressed substitutes

[Mathieu Othacehe]

Hey Maxim,

Sound like a fine plan.

> 1.  Promptly set up both a blog post and a NEWS entry announcing the
> support for gzip substitutes is about to be phased out from the build
> farm (1 month notice), urging users to upgrade their daemon to a version
>>= 1.1.0.

I addition, we could warn users with old daemons as proposed by Ludo &
Ricardo. There's an attached patch here.

> -   ;; TODO: Eventually, disable gzip, as discussed at
> -   ;; 
> <https://lists.gnu.org/archive/html/guix-devel/2021-03/msg00333.html>.
> -   (compression '(("gzip" 9) ("lzip" 9) ("zstd" 19)))
> +   (compression '(("lzip" 9) ("zstd" 19)))
> (cache-bypass-threshold cache-bypass-threshold)
> (workers publish-workers)))

Nice.

> 3. Come up with a Guile script that is able to
>
>   a) Strip gzip-related metadata from the .narinfo guix-publish metadata
>   files
>   b) recompute and update their 'Signature' field.
>
> 4. Finally, 'rm -r /var/guix/publish/gzip' and free about 6.5 TiB of data.

I hope the script will be able to complete within a reasonable amount of
time, but we cannot know before trying it out :).

Thanks,

Mathieu
>From bd306a8b20f2033d67755dd332a5d33b2f6b822d Mon Sep 17 00:00:00 2001
From: Mathieu Othacehe 
Date: Tue, 8 Feb 2022 14:28:56 +0100
Subject: [PATCH 1/1] store: Warn about daemon deprecation.

* guix/deprecation.scm (warn-about-old-daemon): New procedure.
* guix/store.scm (build-things): Use it to warn about old daemons.
---
 guix/deprecation.scm | 7 +++
 guix/store.scm   | 2 ++
 2 files changed, 9 insertions(+)

diff --git a/guix/deprecation.scm b/guix/deprecation.scm
index c66c9367f6..666e178d75 100644
--- a/guix/deprecation.scm
+++ b/guix/deprecation.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2019, 2020, 2021 Ludovic Courtès 
+;;; Copyright © 2021 Mathieu Othacehe 
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -24,6 +25,8 @@ (define-module (guix deprecation)
 
 define-deprecated/public
 define-deprecated/alias
+
+warn-about-old-daemon
 warn-about-deprecation))
 
 ;;; Commentary:
@@ -32,6 +35,10 @@ (define-module (guix deprecation)
 ;;;
 ;;; Code:
 
+(define (warn-about-old-daemon)
+  (warning (G_ "Your Guix daemon is seriously outdated, please consider
+ updating it by following the 'Upgrading Guix' documentation section.~%")))
+
 (define* (warn-about-deprecation variable properties
  #:key replacement)
   (let ((location (and properties (source-properties->location properties
diff --git a/guix/store.scm b/guix/store.scm
index a93e9596d9..11dfc095b1 100644
--- a/guix/store.scm
+++ b/guix/store.scm
@@ -1442,6 +1442,8 @@ (define build-things
  things)))
 (parameterize ((current-store-protocol-version
 (store-connection-version store)))
+  (when (< (current-store-protocol-version) 355) ;0x163
+(warn-about-old-daemon))
   (if (>= (store-connection-minor-version store) 15)
   (build store things mode)
   (if (= mode (build-mode normal))
-- 
2.34.0

Re: License of your contributions to the blog at guix.gnu.org

[Mathieu Othacehe]

Hey Ludo,

> Do you agree with the proposed licensing terms for your contributions to
> the blog?

I agree.

Thanks,

Mathieu

Re: File search

[Mathieu Othacehe]

Hello Ludo!

> Lately I found myself going several times to
>  to look for packages providing a given
> file and I thought it’s time to do something about it.

Yeah, I'm also thinking regularly about it but giving up because setting
up this mechanism properly turns out to be much more complex than
initially expected.

> The script below creates an SQLite database for the current set of
> packages, but only for those already in the store:
>
>   guix repl file-database.scm populate
>
> That creates /tmp/db; it took about 25mn on berlin, for 18K packages.
> Then you can run, say:
>
>   guix repl file-database.scm search boot-9.scm

Nice proof of concept :).

> I think accuracy (making sure you get results that correspond precisely
> to, say, your current channel revisions and your current system) is not
> a high priority: some result is better than no result.  Likewise for
> freshness: results for an older version of a given package may still be
> valid now.

Agreed.

> In terms of privacy, I think it’s better if we can avoid making one
> request per file searched for.  Off-line operation would be sweet, and
> it comes with responsiveness; fast off-line search is necessary for
> things like ‘command-not-found’ (where the shell tells you what package
> to install when a command is not found).

Yeah, that's the tricky part. In term of maintenance, it would probably
be easier to have Cuirass index the packages it's building, store the
results in the PostgreSQL database and serve them using the Cuirass web
server. The pros are that we only rely on one database which is very
important in my opinion. It's also relatively easy to setup. The cons
are that you need to be online to access this API.

If we instead decide to build periodically an sqlite database indexing
all the packages in a cronjob or so, it would still be needed for the
users to download it, which would be an expensive operation as you
mentioned. It would also be difficult to index custom Guix channels with
that approach.

Another solution could be to have guix publish index the files from the
NAR in its cache and provide a file searching API. That would still
require to be online, but it would allow to search from multiple publish
servers hence possibly multiple Guix channels. The packages that do not
have substitutes couldn't be searched which is a strong cons. I would
still maybe have a preference for that option.

WDYT?

Thanks,

Mathieu

Preparing the 1.4.0 release.

[Mathieu Othacehe]

Hello,

As you may know we have started working actively on the next Guix
release.

* The release branch is available here:
  https://git.savannah.gnu.org/cgit/guix.git/log/?h=version-1.4.0
  
* It is built by Cuirass here:
  https://ci.guix.gnu.org/jobset/version-1.4.0

It would be great if we could coordinate and distribute the
effort. We need support to:

* Fix the bugs in this meta issue:
  https://issues.guix.gnu.org/53214

* Check the release machinery is still working by following those steps:
  https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/release.org.
  
* Gather the list of changes between 1.3.0 and 1.4.0 in order to
  complete the NEWS file.

* It is planned to merge the wip-harden-installer branch in the
  version-1.4.0. Once done, we will need some help to test the
  installer.

Who's available to help :) ?

Thanks,

Mathieu

Re: "make dist" fails ... en@boldquote: warning: PO file headder missing or invalid

[Mathieu Othacehe]

Hello,

>> > I've also in the past implored to set up a ci job for "make dist" to
>> > make it at least easier to figure out which commits trigger these
>> > issues... how can I help move that forward? :)
>
> Mathieu, what does Cuirass need to make that happens?

That would require to have a derivation running an equivalent of the
"make dist" command, in the same spirit as the "tarball-jobs" procedure
of the (gnu ci) module I guess.

Thanks,

Mathieu

Re: Branch with only available substitutes ?

[Mathieu Othacehe]

Hello,

> Short answer: yes, but someone has to not just ask whether it's
> possible or tell it ‘should be easy’, and actually implement all this.
> Also, have a look at channel-with-substitutes-available.
> It's not sufficiently general for your use case though.

There's also a pending patch extending this procedure to support
manifests: https://issues.guix.gnu.org/47929.

It still needs some work but it could be conceptually close from what
Nicolas is asking.

Mathieu

Re: Solstice infrastructure hackathon

[Mathieu Othacehe]

Hello,

> That's not to say there's not something to be gained by upgrading the
> bayfront hardware, some SSD storage would be ideal to speed up the
> coordinator and builds.

While we are talking infrastructure, a short digression on the current
situation. The Berlin build farm is providing substitutes for all
branches and development branches. It is providing system images that
are made available on the Guix website, on Gnome Boxes and probably
elsewhere.

It is giving a frequent status on the system tests. It is also checking
that all our sources are available and feeding the Disarchive
database. When this build farm is experiencing troubles, all those
important services are suspended or limited.

As you may know, Berlin is also experiencing storage issues that will
soon be really problematic. Having Bordeaux as another build farm
monopolizing resources and requiring new hardware while maintaining
Berlin is already a huge burden seems untenable to me.

I personally think that having Bordeaux as an alternative build farm,
running alternative software is a wrong direction for the project. I
would personally prefer to put a stop to that situation.

We should clearly host some services such as the Guix website on
Berlin & Bordeaux to bring some redundancy. However, as far as
substitutes building is concerned, redundancy is premature when
maintaining a single system, with our limited human and hardware
resources, proves to be so complex.

What do other people think?

Thanks,

Mathieu

Re: CI status

[Mathieu Othacehe]

> * The cuirass-remote-server Avahi service is no longer visible when
>   running "avahi-browse -a". I strongly suspect that this is related to
>   the static-networking update, even if I don't have a proof for
>   now. This means that the remote-workers using Avahi for discovering
>   (hydra-guix-*) machines can no longer connect. The
>   ci.guix.gnu.org/workers list is thus quite empty.

This is caused by: https://issues.guix.gnu.org/52520. I worked-around it
by enabling multicast manually on the berlin eno1 network interface.

Cuirass is building again, yay!

Mathieu

CI status

[Mathieu Othacehe]

Hello,

You must have noticed that the CI is currently struggling a bit. Here is
a small recap of the situation.

* The IO operations on Berlin are mysteriously slow. Removing files from
  /gnu/store/trash is taking ages. This is reported here:
  https://issues.guix.gnu.org/51787.

  We have to kill the garbage collect frequently to keep things
  going. The bad side is obviously that we can't do that forever, as we
  only have 9.3T and decreasing, while we aim to stay at 10T available.

* The PostgreSQL database behind ci.guix.gnu.org also became super slow
  and I decided to drop it. I don't know if there's a connection with
  the above point. I'm missing the appropriate tools/knowledge to
  monitor the IO & file-system performances.

* The php package isn't building anymore, reported here:
  https://issues.guix.gnu.org/52513. This means that we cannot
  reconfigure zabbix. I removed it from the berlin configuration
  temporarily.
  
* The cuirass-remote-server Avahi service is no longer visible when
  running "avahi-browse -a". I strongly suspect that this is related to
  the static-networking update, even if I don't have a proof for
  now. This means that the remote-workers using Avahi for discovering
  (hydra-guix-*) machines can no longer connect. The
  ci.guix.gnu.org/workers list is thus quite empty.

* Facing those problems, I tried to rollback to a previous system
  generation, but this is bringing even more issues, as for instance the
  older Cuirass package, is struggling with the new database structure and
  other niceties. I think out best course of action is to stick to
  master and fix the above problems.

Thanks,

Mathieu

Re: core-updates-frozen branch merged

[Mathieu Othacehe]

Hey,

> That's it!  Enjoy the latest additions and improvements, and report any
> issues you encounter!

That's great news! Thanks to all the people that have been involved and
special thanks to you Maxim for your commitment.

Mathieu

Re: branch master updated: remote-server: Fix fetch worker crash.

[Mathieu Othacehe]

Hey Ludo,

> I think it’s be safer (less prone to TOCTTOU races) to write it as:
>
>   (catch 'system-error
> (lambda ()
>   (register-gc-roots drv))
> (lambda args
>   (unless (= ENOENT (system-error-errno args)) ;collected in the meantime
> (apply throw args

Oh, indeed! Fixed with 6dcf2f65cea920b9b1c265de3e2b0abe0048a08e.

Thanks,

Mathieu

Re: 04/05: gnu: petsc: Update to 3.16.1.

[Mathieu Othacehe]

Hey,

> * gnu/packages/maths.scm (petsc): Update to 3.16.1.
> [native-inputs]: Use PYTHON instead of PYTHON-2.  Add WHICH.
> [arguments]: Rewrite using gexps.  Pass '--with-openblas-dir'.  In
> 'configure' phase, modify "config/example_template.py".

I noticed that petsc-* packages are now part of every evaluation of
c-u-f in Cuirass. Any chance this is related to this patch?

Thanks,

Mathieu

Re: 01/03: gnu: freecad: Switch to vtk-9.

[Mathieu Othacehe]

Hey Thiago!

> This duplicates the line for ‘freedink-engine-fix-sdl-hints.patch’.

Thanks for the heads up, fixed with:
854120d01ffe8b209fa8d897ba0fcf39ce2cdf32.

Mathieu

Re: Disarchive update

[Mathieu Othacehe]

Hey,

> That’d be nice!  How do we do that again?

The build-outputs field of the  record must be used as
explained here:
https://guix.gnu.org/cuirass/manual/html_node/Specifications.html#Specifications.

This field cannot be manipulated via the web interface yet.

I think the easier way to proceed would be to create the "disarchive"
specification in the (maintenance sysadmin services) module, this way:

--8<---cut here---start->8---
(specification
 (name "disarchive")
 (build '(manifests "etc/disarchive-manifest.scm"))
 (build-outputs
  (list
   (build-output
(job "disarchive-collection*")
(type "archive")
(path ""
 (notifications #$(cuirass-notifications))
 (period 43200)
 (priority 7)
 (systems '("x86_64-linux")))
--8<---cut here---end--->8---

I can take care of that if it's ok for you.

Thanks,

Mathieu

Re: Disarchive update

[Mathieu Othacehe]

Hey Ludo,

>   https://ci.guix.gnu.org/eval/29213?status=succeeded

Nice! It looks like an expensive operation, maybe we should increase its
period to 24 hours or so?

>   2. On berlin, add an mcron job that periodically copies the output of
>  the latest “disarchive-collection” build to a directory, say
>  /srv/disarchive.  Thus, the database would accumulate tarball
>  metadata over time.

We could add the result as a "build-product" so that it is available at:
https://ci.guix.gnu.org/search/latest/disarchive-collection. The mcron
job could use this URL to fetch the latest archive.

> How does that sound?  Thoughts?

Sounds great, happy to see more use-cases for Cuirass :)

Thanks,

Mathieu

Re: core-updates-frozen: Planning for the last world rebuild

[Mathieu Othacehe]

Hey Ricardo,

> It should be fine now.  I didn’t run the test myself, but I ran most of the
> commands and found a few more hardcoded tool file names that referenced /sbin
> or /usr/bin. 

Yes, I confirm it passes! I also fixed the childhurd test. Only 9 tests
to go :).

Thanks,

Mathieu

Re: guix weather -m etc/sources-manifest.scm and CI?

[Mathieu Othacehe]

Hey Ludo,

> -package system))
> - packages)))
> +  (let ((manifests (arguments->manifests rest channels)))
> +(manifests->jobs store manifests)))

This sounds great to me :)

Thanks,

Mathieu

Re: guix weather -m etc/sources-manifest.scm and CI?

[Mathieu Othacehe]

Hey!

> Mathieu, could it have to do with the fact that the manifest is within
> the ‘guix’ channel?

Using the attached little reproducer, I get a more interesting
backtrace:

--8<---cut here---start->8---
Backtrace:
In ice-9/boot-9.scm:
  1752:10 11 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
  10 (apply-smob/0 #)
In ice-9/boot-9.scm:
724:2  9 (call-with-prompt _ _ #)
In ice-9/eval.scm:
619:8  8 (_ #(#(#)))
In ice-9/boot-9.scm:
   2835:4  7 (save-module-excursion _)
  4380:12  6 (_)
In gnu/ci.scm:
496:4  5 (cuirass-jobs _ _)
In srfi/srfi-1.scm:
   673:15  4 (append-map _ _ . _)
   586:17  3 (map1 (list "x86_64-linux"))
   586:17  2 (map1 (#https://www.brain-dump.org/projects…> …))
In gnu/ci.scm:
   575:38  1 (_ #https://www.brain-dump.org/projects/abduco…>)
In guix/packages.scm:
441:0  0 (%package-name-procedure #https://www.brain-du…>)

guix/packages.scm:441:0: In procedure %package-name-procedure:
In procedure package-name: Wrong type argument: #https://www.brain-dump.org/projects/abduco/abduco-0.6.tar.gz; # () 7f84f252ec00>
--8<---cut here---end--->8---

The issue is that (gnu ci) expects manifests of packages and not
manifests of origins. Hence, the "job-name" procedure that calls
"package-name" on an origin fails.

We could maybe generalize the manifests->packages to support packages
and origins. The package-job procedure would also need some adjustment
to call origin->derivation instead of package-derivation.

WDYT?

Thanks,

Mathieu


cuirass.scm
Description: Binary data

Re: core-updates-frozen: Planning for the last world rebuild

[Mathieu Othacehe]

Hello Ricardo,

> I fixed the build.  Also had to upgrade 389-ds-base to get around a linker
> error.

Thanks for your support here! The ldap test is now run, but sadly there
are some failures: https://ci.guix.gnu.org/build/928047/log/raw.

Mathieu

Re: core-updates-frozen: Planning for the last world rebuild

[Mathieu Othacehe]

Hey,

> For example, this evaluation from Sep 17 has thousands of scheduled 
> evaluations:
>
> https://ci.guix.gnu.org/eval/19271?status=pending

Mmmh, not nice. Any chance you could share the cuirass-remote-worker
logs on the guixp9 machine?

Thanks,

Mathieu

Re: core-updates-frozen: Planning for the last world rebuild

[Mathieu Othacehe]

Hey Thiago,

> I was waiting for this world rebuild to happen to see the results in 
> Cuirass now that powerpc64le-linux builds are only done on guixp9, but in 
> hindsight it would have been better if I requested that those builds were 
> restarted instead. Restarting this build should remove the main roadblock 
> though:
>
> https://ci.guix.gnu.org/build/703194/details

Now restarted!

>
> When that build is restarted and succeeds, does Cuirass also restart the 
> builds that failed because they depended on it, or do we need to explicitly 
> restart those as well?

Yes, on successful build completion Cuirass looks if some builds that
were marked as "failed-dependency" (failing because some of the
dependencies are also failing) can be resumed.

That being said, Cuirass is not performing nicely these days because of:
this issue https://issues.guix.gnu.org/48468.

Thanks,

Mathieu

Re: core-updates-frozen: Planning for the last world rebuild

[Mathieu Othacehe]

Hey!

> have more system tests passing. Currently 16 out of 87 are
> broken. Except for the nfs-root-fs that has always been broken, the
> other tests probably cover user configurations out there.

Made some progress here but there are still a bit more work needed:

* childhurd (https://ci.guix.gnu.org/build/901219/details)
 => gnumach build appears to be broken.

* dovecot (https://ci.guix.gnu.org/build/901189/details)
 => dovecot build appears to be broken.

* patchwork (https://ci.guix.gnu.org/build/901227/details)
 => python-django-pipeline build (dependency of patchwork) appears to be
  broken.

* ganeti-kvm (https://ci.guix.gnu.org/build/901158/details)
 => ganeti build appears to be broken.

* ganeti-lxc (https://ci.guix.gnu.org/build/901157/details)
 => same reason.

* getmail (https://ci.guix.gnu.org/build/901187/details)
 => dovecot build appears to be broken.

* gui-installed-desktop-os-encrypted
  (https://ci.guix.gnu.org/build/923078/details)
 => emacs-exwm build is broken presumably because of
  https://issues.guix.gnu.org/50066.

* jami-provisioning (https://ci.guix.gnu.org/build/901214/details) and
 jami (https://ci.guix.gnu.org/build/901213/details)
 => sobjectizer build appears to be broken.

* ldap (https://ci.guix.gnu.org/build/901179/details)
 => 389-ds-base build appears to be broken.

The overall status can be seen here:
https://ci.guix.gnu.org/eval/24933/dashboard. Let's get those tests fixed
so that we can start this branch on good grounds!

Thanks,

Mathieu

Re: core-updates-frozen: Planning for the last world rebuild

[Mathieu Othacehe]

Hey!

> Wouldn’t it be nice to get ‘core-updates-frozen’ merged?  :-)

That would be great :) Before merging to master, it would be nice to
have more system tests passing. Currently 16 out of 87 are
broken. Except for the nfs-root-fs that has always been broken, the
other tests probably cover user configurations out there.

I'm currently working on the installation tests.

Thanks,

Mathieu

Re: branch core-updates-frozen updated: services: database: Change postgresql default socket.

[Mathieu Othacehe]

Hello,

> The manual should also be updated to reflect this change.

Fixed, thanks for the reminder.

> Also, is there any reason for making this change on
> ‘core-updates-frozen’ rather than ‘master’?

Yes, that's a follow-up of 86cf4c039631cdf15c4c428bf4ffe52d6cbd7d4c that
was pushed on core-updates a while ago.

Thanks,

Mathieu

Re: OBS substitutes

[Mathieu Othacehe]

Hello Chris,

> On Guix Systems, if the substitute keys are left as the default, for
> recent revisions of Guix, the bordeaux.guix.gnu.org signing key will be
> included in the ACL by default. With Guix on foreign distributions, it
> needs adding manually.
>
> I think there's some room for improvement in the UI/UX here, in terms of
> telling users that there are substitutes available, if they trust a
> specific key (all the relevant information is in the narinfo).

I totally agree here, as I also faced this issue several times. When
fetching a narinfo which is signed with a key that's not in the ACL, we
should probably issue a warning.

Mathieu

Re: OBS substitutes

[Mathieu Othacehe]

Hello zimoun,

> well, I do not understand why it is not on Berlin.  But why not.  Then,

Looks like Cuirass built it the 15th of September:
https://ci.guix.gnu.org/build/785837/details.

When did you experienced this issue? What was your guix revision?

> Why?  How is it possible that “guix weather” reports that the
> substitute is available but then “guix build” does not fetch it?

That is strange. Did you run those two commands in that specific
order? How much time elapsed between the commands?

Thanks,

Mathieu

Re: core-updates weather reports!

[Mathieu Othacehe]

> Oh, nice!  Did increasing the nginx timeouts help?

Yes, it definitely helped, but there are still some request timeouts. I
hope that the publish patches we are discussing here[1] can fix this
situation.

Thanks,

Mathieu

[1]: https://issues.guix.gnu.org/50040

Re: Distributing Guix System Pinebook Pro images

[Mathieu Othacehe]

Hello Vagrant,

> I wonder how much diskspace savings you might get by leaving out the
> display manager... I typically just login from the console and run "exec
> sway". I *think* the last image I generated for that was more like 2GB,
> though my memory is fallable, and maybe misses various other standard
> desktop packages (e.g. web browser).

Yeah SDDM drags Xorg between other things, I could definitely get rid of
it.

> I have since figured out how to use the regular "linux-libre" kernel and
> which modules to add to the initrd... I'll try to dig those up.

That would be nice. I also came across to the issue you reported here:
https://issues.guix.gnu.org/48266. We definitely need to do something
about it. Building this initrd modules list is quite painful. I'll try
to think about it.

> Have fully native builds been fixed yet with "guix system image" ? That
> has been a blocker forme from trying it out. I haven't been booting my
> x86 guix installation lately, mostly just using the pinebook pro, so it
> is ironic I haven't been able to make a pinebook pro image. :)

Hehe, that's a shame indeed. I'm working on it as a follow-up of
https://issues.guix.gnu.org/45020. I'll propose a patch soon.

Thanks,

Mathieu

Distributing Guix System Pinebook Pro images

[Mathieu Othacehe]

Hello,

The Guix website now offers Pinebook Pro Guix System images[1]. Those
images can be directly copied on a micro-SD card and booted from. More
details here[2].

Thanks,

Mathieu

[1]: https://guix.gnu.org/en/download/latest/
[2]: https://othacehe.org/distributing-guix-system-pinebook-pro-images.html.

Re: core-updates weather reports!

[Mathieu Othacehe]

Hello,

The situation is getting better here:

--8<---cut here---start->8---
https://ci.guix.gnu.org
  81.1% substitutes available (16,148 out of 19,921)
  at least 112,494.5 MiB of nars (compressed)
  17,592,186,161,392.5 MiB on disk (uncompressed)
  0.006 seconds per request (29.4 seconds in total)
  162.1 requests per second
--8<---cut here---end--->8---

I came up with a small script to pick randomly a failing package on the
core-updates-frozen branch:

--8<---cut here---start->8---
(use-modules (guix ci)
 (srfi srfi-1))

(define url "https://ci.guix.gnu.org;)
(define system "x86_64-linux")

(define evals
  (latest-evaluations "https://ci.guix.gnu.org; 5
  #:spec "core-updates-frozen"))

(define (last-eval evals)
  (first
   (filter-map (lambda (eval)
 (and (evaluation-complete? eval)
  (evaluation-id eval)))
   evals)))

(define eval
  (last-eval evals))

(define failed-builds
  (filter-map (lambda (job)
(and (eq? (job-status job) 'failed)
 (job-name job)))
  (evaluation-jobs url eval)))

(define %seed
  (seed->random-state
   (logxor (getpid) (car (gettimeofday)

(define failed-for-system
  (filter-map (lambda (job)
(and (string-suffix? system job)
 job))
  failed-builds))

(format #t "~a~%"
(list-ref failed-for-system
  (random (length failed-for-system)
  %seed)))
--8<---cut here---end--->8---

For instance:

--8<---cut here---start->8---
mathieu@meije ~/guix [env]$ guix repl ~/tmp/fix.scm 
rust-test-case.x86_64-linux
mathieu@meije ~/guix [env]$ guix repl ~/tmp/fix.scm 
rust-tungstenite.x86_64-linux
--8<---cut here---end--->8---

That's a fun little game :)

Mathieu

Re: core-updates weather reports!

[Mathieu Othacehe]

Hey,

> Okay, are you confident in the #50040 patch? It looks like there are a
> lot of spurious failures from #48468.

I just discovered that many spurious failures were also caused by short
Nginx timeout values. Added a comment about that on #50040. I'm also
currently experimenting directly on Berlin on that topic.

Mathieu

Re: core-updates weather reports!

[Mathieu Othacehe]

Hello Leo,

> As far as I can tell, the build farms are not actually trying to build
> for core-updates-frozen, at least not in a systematic way. Am I
> mistaken? Or are we only building the core subset of packages for this
> branch?

Thanks for the reports! Cuirass is actually trying to build all packages
on top of the core-updates-frozen branch.

A few weeks ago, I tried to work around the substitute timeout issue[1]
by using the main berlin publish server on all the workers. This caused
another problem, that should be fixed with this patch[2].

Mathieu

[1]: https://issues.guix.gnu.org/48468
[2]: https://issues.guix.gnu.org/50040

Re: Hundreds of failed build on master following git package update

[Mathieu Othacehe]

Hello Brice & Sarah,

> That sounds awfully familiar.  I wonder if this is related to the
> following issue?

Yes, this is indeed completely related to the pointed issue. I'd like to
proceed with the patch soon so that Cuirass can recover a working
state. However, as this is a rather delicate change, it's probably
better to wait for some feedback.

Thanks,

Mathieu

Re: branch master updated: services: cuirass: Add a no-publish argument.

[Mathieu Othacehe]

Hey,

> So I’d suggest turning that into ‘publish?’, defaulting to #t.

Yeah, you're completely right, I reversed the logic with:
cfd2442488971420289a12d5ca8f07816e1149bf.

Thanks,

Mathieu

Re: Substitute timeouts

[Mathieu Othacehe]

Hey Ludo,

Thanks for taking the time to read my wall of text :D.

> Yeah, it’s a double-edged sword.  If this is a problem on the main ‘guix
> publish’ server, we can lower the bypass threshold, which is currently
> 50 MiB:
>
>   
> https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/modules/sysadmin/services.scm#n450
>
> WDYT?

That would maybe help, but on the other hand, I would prefer to find a
more definitive solution :).

> First, in terms of UI, you’d have a command sitting there and doing
> nothing, which can be off-putting.  Second, clients have no idea how
> long they’re going to wait; it could be that the nar is going to be
> baked within seconds, or it could take 20mn if the baking queue is
> already crowded or if the user is asking for a big store item like
> libreoffice.  Third, in many cases, building locally is likely to be
> faster than waiting for substitutes to be available (the majority of
> packages build very quickly, though the few most popular leaf packages
> take a long time to build).

It would be interesting to monitor the status of the baking
workers. Could it really take 20 minutes to bake a substitute from your
experience?

Personally, I have always found this baking 404 and bypass cache a bit
misleading. When substituting libreoffice, I would much rather wait a
few minutes than trying to build it while there's an almost ready
substitute. I get that this is a personal choice and maybe it should be
an optional behaviour.

>> It will also allow the Cuirass build farm to use directly the main guix
>> publish server, simplifying the current CI setup.
>
> The only reason why Cuirass runs its own publish server is to avoid
> overloading the main one?

No, the main reason is that with the use of a publish cache, the Cuirass
workers would probably hit 404 errors while the substitutes are being
baked. Using a publish server without cache was a way to work around it.

The motivation of the 202 waiting patch was to solve both problems at
once. Maybe I should explore the narinfo dedicated thread solution as a
short term solution, while starting to think about a more long term
solution based on Fiber/Nginx.

A Cuirass dedicated solution could also be to declare a build successful
only when a nar is available and stop using a non-caching publish
server.

Thanks,

Mathieu

Re: Project direction with testing changes (branches and patches)

[Mathieu Othacehe]

Hello Chris,

> I think trying to change up how branches (staging/core-updates) are
> tested is a good place to start. The concrete change I'm proposing is to
> use an instance of the Guix Data Service plus an instance of the Guix
> Build Coordinator to do the testing and builds, rather than Cuirass on
> ci.guix.gnu.org which is the current approach.
>
> The main advantages of that would be the comparison support from the
> Guix Data Service, and the build performance and reliability that the
> Guix Build Coordinator brings. The main disadvantage is probably the
> lack of an admin like interface similar to that of Cuirass (I think this
> can be remedied in the medium term though).

We indeed desperately need some more automation. For each new patch
series, it would be great to have the following information:

* Status of the linter.
* Status of the depending derivations.
* Status of the unit tests (in the tests/ directory).
* Status of the system tests (in the gnu/tests/ directory).

I would like to stay focused on the existing, well adopted solutions and
build upon them.  With Cuirass we already have most of the machinery to
provide those information.

In addition, we have dashboards, RSS and email notification support
which could allow us to extend Cuirass functionalities to provide the
expected CI features.

We could imagine a new API such as /api/changeset/new that would create
temporary Cuirass specifications. The output result could be consulted
using a new /api/changeset/status API or directly through the Cuirass
web interface. Cuirass could also send email and RSS notifications on
that changeset.

Those APIs could be used by Mumi to trigger the /api/changeset/new API
notification on one hand and to display some status using the
/api/changeset/status API on the other hand.

I think that this whole infrastructure is just a few Guile modules away
and if we could involve more fellow hackers in the process, it would be
just great.

Thanks,

Mathieu

Substitute timeouts

[Mathieu Othacehe]

Hello,

I have been investigating a problem that is visible both on the main
guix publish server at https://ci.guix.gnu.org[1] and on the Cuirass
build farm[2].

This error comes from the fact that the publish server does not accept
the "guix substitute" connection requests within the %fetch-timeout
duration of 5 seconds.

The main guix publish server is using a cache. If a requested narinfo is
not in the cache, it will be baked and the client receives a 404
error. Since ecaa102a58ad3ab0b42e04a3d10d7c761c05ec98 and the
introduction of the bypass mechanism, small store items are directly
returned.

This means that the "narinfo-string" procedure can be called directly in
the main publish thread. Running perf on the main publish server reveals
that this procedure can be really expensive under IO pressure (GC
running for example) because it opens a lot of files. I have observed
that the "read-derivation-from-file" call can take up to 600 ms.

If multiple clients were to ask narinfo of several items not yet cached,
under IO pressure, I think that the publish server could become
unresponsive and cause the timeout errors.

The fact that Cuirass triggers the baking of successfully built
derivations probably doesn't help here.

Now regarding the timeout errors that are much more frequent on the
Cuirass build farm, the cause varies a bit. The Cuirass publish server
running on Berlin does not use a cache. This means that the
"narinfo-string" procedure is called for each request, in the main
thread.

To fix those issues, a solution could be to run the "narinfo-string" in
a separate thread, but it will make the publish server code even harder
to understand. My proposition would be to get rid of the bypass
mechanism and instead implement a retry when some substitutes are
reported as being baked, as proposed by Miguel[3].

I think this is the most reasonable solution. This way, users won't
receive 404 errors and start building substitutes that are being
baked[4].

It will also allow the Cuirass build farm to use directly the main guix
publish server, simplifying the current CI setup.

There's a proposed patch attached, WDYT?

Thanks,

Mathieu

[1]: https://issues.guix.gnu.org/49089
[2]: https://issues.guix.gnu.org/48468
[3]: https://issues.guix.gnu.org/44193#2
[4]: http://issues.guix.gnu.org/33370



patch
Description: Binary data

Core updates frozen.

[Mathieu Othacehe]

Hello,

A new core-updates-frozen branch is available here:
https://git.savannah.gnu.org/cgit/guix.git/log/?h=core-updates-frozen.

This means that the core-updates branch remains open, while the
core-updates-frozen branch will only accept bug fixes.

This branch contains exciting new features, such as:

* Switch to GCC 10.
* Update to TexLive 2021.
* Simplified package inputs[1].
* Build system Gexp support[2].
* Meson cross-compilation support[3].

between lots of other things.

Cuirass has started evaluating this branch here:
https://ci.guix.gnu.org/jobset/core-updates-frozen. According to the
related dashboard, there's still a bit of work required to stabilize
this branch: https://ci.guix.gnu.org/eval/68124/dashboard.

Your help is welcome :).

Thanks,

Mathieu

[1]: https://lists.gnu.org/archive/html/guix-devel/2021-05/msg00343.html
[2]: https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00269.html
[3]: https://lists.gnu.org/archive/html/guix-patches/2021-07/msg00923.html

Re: branch master updated: ci: Add bootstrap packages to the core subset.

[Mathieu Othacehe]

Hey Ludo!

> Should we arrange so that jobs for the commencement.scm packages get a
> custom name, for example prefixed by “commencement.”?

The primary key for the Builds table is on the derivation field, so
duplicated job-name shouldn't be an issue. If you search for
"gcc-toolchain.aarch64-linux" here:
https://ci.guix.gnu.org/eval/59586?paginate=0, you will find 9
occurrences.

Did you find a specific job missing in that evaluation?

I'll see if I can reproduce the issue you are describing here:
https://lists.gnu.org/archive/html/guix-devel/2021-06/msg00222.html.

Thanks,

Mathieu

Re: Zile and minimal systems

[Mathieu Othacehe]

Hey,

> I suppose this line could be removed now.

Pushed, thanks for having a look.

Mathieu

Re: Zile and minimal systems

[Mathieu Othacehe]

Hey,

> ‘zile-on-guile’ is tempting but currently unmaintained.  ‘mg’ could work too.

mg seems to work fine. I'm currently testing the attached patch. Any
objections?

Thanks,

Mathieu
>From 0cff5b30711a8987bc0457c23a73982888ec802e Mon Sep 17 00:00:00 2001
From: Mathieu Othacehe 
Date: Fri, 9 Jul 2021 10:20:39 +0200
Subject: [PATCH] system: Provide mg instead of zile.

Since the update to the 2.6.2 release, the closure size of zile has
increased. Switch to mg which is lighter.

* gnu/system.scm (%base-packages-interactive): Replace zile by mg.
* doc/guix.texi (Proceeding with the Installation,
Using the Configuration System): Adapt those sections.
---
 doc/guix.texi  | 4 ++--
 gnu/system.scm | 3 ++-
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 18bc600440..46cec73d11 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -2543,7 +2543,7 @@ provide the declaration of the operating system to be installed.  To
 that end, the installation system comes with three text editors.  We
 recommend GNU nano (@pxref{Top,,, nano, GNU nano Manual}), which
 supports syntax highlighting and parentheses matching; other editors
-include GNU Zile (an Emacs clone), and
+include mg (an Emacs clone), and
 nvi (a clone of the original BSD @command{vi} editor).
 We strongly recommend storing that file on the target root file system, say,
 as @file{/mnt/etc/config.scm}.  Failing to do that, you will have lost your
@@ -13485,7 +13485,7 @@ environment variable---in addition to the per-user profiles
 (@pxref{Invoking guix package}).  The @code{%base-packages} variable
 provides all the tools one would expect for basic user and administrator
 tasks---including the GNU Core Utilities, the GNU Networking Utilities,
-the GNU Zile lightweight text editor, @command{find}, @command{grep},
+the @command{mg} lightweight text editor, @command{find}, @command{grep},
 etc.  The example above adds GNU@tie{}Screen to those,
 taken from the @code{(gnu packages screen)}
 module (@pxref{Package Modules}).  The
diff --git a/gnu/system.scm b/gnu/system.scm
index 8a3ae27d04..3951a6806f 100644
--- a/gnu/system.scm
+++ b/gnu/system.scm
@@ -63,6 +63,7 @@
   #:use-module (gnu packages package-management)
   #:use-module (gnu packages pciutils)
   #:use-module (gnu packages texinfo)
+  #:use-module (gnu packages text-editors)
   #:use-module (gnu packages wget)
   #:use-module (gnu packages zile)
   #:use-module (gnu services)
@@ -775,7 +776,7 @@ of PROVENANCE-SERVICE-TYPE to its services."
 
 (define %base-packages-interactive
   ;; Default set of common interactive packages.
-  (list less zile nano
+  (list less mg nano
 nvi
 man-db
 info-reader ;the standalone Info reader (no Perl)
-- 
2.32.0

Re: Wrap %default-guix-channel inside channel-with-substitutes-available by default?

[Mathieu Othacehe]

Hello,

> I think we do have the obligation to clearly document channels-with-
> substitutes-available and make it so that this particular piece of
> documentation can easily be found by everyone who is potentially
> affected by slow-downs.  Perhaps we might even want to include an
> option in the installer to set up /etc/guix/channels.scm to only use
> channels with available substitutes.  But using it as a default without
> communicating this fact to the users would be a poor idea in my
> opinion.

I fear that a lot of users are not aware of this mechanism and do not
really understand why "guix pull" sometimes starts building stuff. 

We could maybe proceed this way:

1. Enable the channel-with-substitutes-available snippet by default.

2. Add a command line option --without-available-substitutes.

3. Add a new info message when the fetched commit is not the latest
commit, to minimize the risk you are mentioning above. Something like:

Skipping N new commits that do not have any available substitutes, use
the --without-available-substitutes option to update anyway.

WDYT?

Thanks,

Mathieu

Cuirass 1.1 released

[Mathieu Othacehe]

We are pleased to announce the release of Cuirass 1.1.

This is the second official release of Cuirass, the GNU Guix continuous
integration software.  For the last six months, this project has been
funded through the NGI0 PET Fund, a fund established by NLNet[1].

Thanks to this support, Cuirass has seen numerous improvements, such as
a switch to PostgreSQL and the introduction of a distributed build
mechanism.  Cuirass is now providing substitutes to all Guix users in a
faster and more reliable way while providing better monitoring.

• About

Cuirass is the GNU Guix continuous integration software. It's a general
purpose build automation server written in GNU Guile that checks out
sources from VCS repositories, execute build jobs and store build
results in a database. Cuirass also provides a web interface to monitor
the build results.

Cuirass is running on GNU Guix build farm[2].

• Download

  Here are the compressed sources and a GPG detached signature:
https://guix.gnu.org/cuirass/releases/cuirass-1.1.0.tar.gz
https://guix.gnu.org/cuirass/releases/cuirass-1.1.0.tar.gz.sig

  Here are the SHA1 checksums:

  90a4ddbbb255353a1a90807b5dcc4ec7ed7e  cuirass-1.1.0.tar.gz
  fe46ad674be0c3d1578d2c3950717f1dcf775e1f  cuirass-1.1.0.tar.gz.sig

• Changes since version 1.1.0 (excerpt from the NEWS file)

** Database
*** Add Jobs table
*** Add BuildDependencies table
*** Add Dashboards table

** Remote building
*** Add GC roots for the build outputs
*** Increase the fetch workers count to 8
*** Build derivations in a topological order using the BuildDependencies table
*** Resume dependent builds when a build is successful after a restart
*** Display the remote-server fetch queue size

** Specifications
*** Add period support
*** Add "images", "system-tests" and tarball build types

** Web
*** Add a footer with the Cuirass version
*** Add table order buttons
*** Add a pagination button on the evaluation page
*** Add a build dashboard page
*** Improve accessibility and add the Accessiblity Foundation report
*** Add badges support
*** Display build dependencies in the build details page
   
Please report bugs to bug-g...@gnu.org
Join guix-devel@gnu.org and #guix on Freenode for discussions.

Mathieu

[1]: https://nlnet.nl/
[2]: https://ci.guix.gnu.org


smime.p7s
Description: S/MIME cryptographic signature

Re: Cuirass badges

[Mathieu Othacehe]

Hey Ludo,

> This is not yet deployed in the current ‘cuirass’ package, right?

Yes it is :) For instance, to get the badge of the guix-hpc
specification at https://guix.bordeaux.inria.fr, you can use the
following URL: https://guix.bordeaux.inria.fr/jobset/guix-hpc/badge.svg.

There are some additional details in the Cuirass documentation.

Thanks,

Mathieu

Re: Ungrafting CI jobs

[Mathieu Othacehe]

Hey Leo,

> Okay. Can you let me know when you do? I'll delay the ungrafting builds
> until then.

I deployed the upgrade yesterday. I've been monitoring it for 12 hours
without any sign of substitutes timeout. I think you can proceed whenever
you want.

Thanks,

Mathieu

Re: Ungrafting CI jobs

[Mathieu Othacehe]

Hello Leo,

I monitored a previous evaluation of the ungrafting Cuirass
specification that was more successful, with more than 17000 builds
performed in less than 24 hours, a new record! The recent evaluation is
sadly less victorious.

> The latest iteration failed en masse due to network timeouts between
> build farm nodes.

Yes, I described this issue here: https://issues.guix.gnu.org/48468 and
hopefully fixed it with https://issues.guix.gnu.org/48556.

I need to perform a reconfigure / deploy but I'm waiting to have a few
days ahead of me to fix potential issues.

> It lets us see that the Audacity build is not attempted due to the
> failure of wxwidgets. And in turn, the failure of webkitgtk and libsoup.
> However, there is no information (logs, duration) available about those
> failed builds.

When a build is cancelled ("Cancel pending builds" button), all
its dependents are automatically updated to "Failed dependency". It
looks like this is the case for https://ci.guix.gnu.org/build/515005/details.

> What can we do to clear all these "spurious" build failures and re-try
> building the branch? I already tried using the "restart all builds"
> button, but there were still a lot of spurious failures.

I think the easiest way to recover from this situation is to remove the
ungrafting specification and recreate it. It ensures that no build will
fail because of previously failing/cancelled build dependencies.

Note that when a build B is restarted in Cuirass ("Restart" button), any
other build B' depending from B, will automatically be restarted if B
becomes successful and all the other dependencies of B' are already
successful.

I used this mechanism to restart a few builds failing on the master
branch because of substitutes timeout. I hope that when the keep-alive
patchset will solve this situation.

Thanks,

Mathieu

Re: Adding Substitute Mirrors page to installer

[Mathieu Othacehe]

Hello,

Sorry again for the long delay.

> Of course, it feels somewhat dirty, as there is a need to read the 
> configuration file.
> It would be cleaner if a page could affect the field created by a previous 
> page.

Yes I agree. In the meantime, I think that the configuration file read
is an acceptable solution.

> I think Less is More, and deploying an installer with a simple "select one
> mirror" page *now* will cover 90% of use-cases, and we can add the more
> complicated page later when there is more time.

Seems fair.

I ran the installer tests with this command: make check-system
TESTS="gui-installed-os". The following error is reported:

--8<---cut here---start->8---
conversation expecting pattern ((quote list-selection) ((quote title) 
"Partitioning method") ((quote multiple-choices?) #f) ((quote items) 
(not-encrypted encrypted _ ...)))
/gnu/store/h38i2hvaqy9bd0sbn9isdfzl7m56mngr-shepherd-marionette.scm:1:1718: 
ERROR:
  1. :
  pattern: ((quote list-selection) ((quote title) "Partitioning method") 
((quote multiple-choices?) #f) ((quote items) (not-encrypted encrypted _ ...)))
  sexp: (list-selection (title "Substitute mirror") (multiple-choices? #f) 
(items ("https://ci.guix.gnu.org (Berlin, official Guix substitute server)" 
"https://mirror.sjtu.edu.cn/guix (China, SJTU)")))
Backtrace:
   2 (primitive-load "/gnu/store/n6z0h9gxfkzrvsfcsbpd05kxxcx?")
In ice-9/eval.scm:
   191:35  1 (_ #f)
619:8  0 (_ #(# #))

ice-9/eval.scm:619:8: Throw to key `marionette-eval-failure' with args `((quote 
(choose-partitioning installer-socket #:encrypted? #f #:passphrase 
"thepassphrase" #:uefi-support? #f)))'.
note: keeping build directory `/tmp/guix-build-installation.drv-0'
builder for `/gnu/store/wp95pl7l9jfjwj6dpizhmbsfb85alzi1-installation.drv' 
failed with exit code 1
build of /gnu/store/wp95pl7l9jfjwj6dpizhmbsfb85alzi1-installation.drv failed
View build log at 
'/var/log/guix/drvs/wp/95pl7l9jfjwj6dpizhmbsfb85alzi1-installation.drv.bz2'.
--8<---cut here---end--->8---

As explained here:
https://lists.gnu.org/archive/html/guix-patches/2021-05/msg00819.html,
you need to introduce a new "list-selection" conversation item in the
"choose-services" procedure of the (gnu installer tests) module to fix
this issue.

Thanks,

Mathieu

Re: linux-libre source tarballs

[Mathieu Othacehe]

Hello,

>> Not knowing exactly how ci.guix.gnu.org works, would it make sense to
>> create a tarball package instead of the ... computed origin(?) tarball,
>> so it could be better represented in the package dependency graph, and
>> the various linux-libre-* packages can wait till it is available rather
>> than all trying to recreate the same thing?
>
> If I understand correctly, this was recently fixed in Cuirass:
>
> https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/commit/?id=d1a95e8b33b454a45bda506a22a8b9d9d2c8b16e

With the recent Cuirass commits, the notion of build dependencies have
been introduced. If you have a look to this page:
https://ci.guix.gnu.org/build/497096/details, you will see that the
"git" build won't start until all the other builds it depends are
successfully completed (xmlto, subversion, tk ...).

The problem with linux-libre and a few other packages is that some of
the derivations they depend do not correspond to proper packages that
are identified and built by Cuirass. You will see here:
https://ci.guix.gnu.org/build/512266/details that linux-libre has no
build dependencies.

If we could translate the computed origin method into a package, I think
it could solve this issue. I also noticed a similar issue with
ublock-origin and the make-chromium-extension method.

Thanks,

Mathieu

Re: Cuirass badges

[Mathieu Othacehe]

Hey Luis,

> And the source file:
>
> https://luis-felipe.gitlab.io/media/2021/05/cuirass-badges-proposal-2021-05-28.svg
>
> I think I like d and d*.

Wooh, that's great! I updated Cuirass to use the d badge proposal.

Many thanks for your help,

Mathieu

Cuirass badges

[Mathieu Othacehe]

Hello,

Cuirass is now able to generate badges like any respectable CI
system. For instance:

* https://ci.guix.gnu.org/jobset/master/badge generates a badge showing
  the percentage of successful jobs for the master specification.

* https://ci.guix.gnu.org/jobset/guix/badge generates a badge showing
  the percentage of successful jobs for the guix specification.

You can integrate those badges in your Git forge. There's a screenshot
of a Gitlab integration attached.

I think that the badge design[1] could be significantly improved by
anyone with artistic abilities. It would also be nice if the badge could
display the specification name so that badges corresponding to multiple
specifications can be displayed together.

Thanks,

Mathieu

[1]:
https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/tree/src/static/images/badge-per.svg

Re: Bringing substitutes from the Guix Build Coordinator to users

[Mathieu Othacehe]

Hey Chris,

> That sounds sensible. On the specific name, given this is just about
> substitutes, and at least in my opinion has nothing to do with
> continuous integration, maybe picking just another word would avoid
> thinking too much, it could be bordeaux, or hippo, or anything
> really. As you say, stability and not being tied to a particular machine
> is the important thing.

The substitutes coverage is one indicator to take into account but there
are many others. For instance, the evaluation speed, the failed
evaluation count, the average evaluation builds completion time, the
availability of the connected build machines between other things.

Deploying a solution that builds substitutes is fine, but as soon as it
is deployed and accessible to all Guix users, the system administrators
will have to monitor it and maintain it in the long run.

Having two heterogeneous build infrastructures on two sets of machines,
providing different metrics will make the update and maintenance of
those machines harder.

I hear your point about K-out-of-N policy and it also makes sense to
me. However, we should maybe consider doing it using two similar
infrastructures.

Thanks,

Mathieu

Re: branch master updated: gnu: Add emacs-wucuo.

[Mathieu Othacehe]

Hello Chris,

> The n on this line means that this package can't be built, and also
> breaks the linter.
>
> I've removed the n in fd2abc2a51e2cc39ac67dcef1d21a8037147e798.

It also broke Cuirass master evaluation.

Thanks for fixing it,

Mathieu

Cuirass accessibility

[Mathieu Othacehe]

Hello,

As part of the Cuirass NLNet grant[1], the website at
https://ci.guix.gnu.org has been audited by the Accessibility
Foundation[2].

I have made the report public here[3]. Although I have already applied a
bunch of accessibility related fixes, there's still some work left. The
report if full of interesting teachings, and if some people want to join
me, they are more than welcome :).

Thanks,

Mathieu

[1]: https://othacehe.org/gnu-guix-continuous-integration---nlnet-grant.html
[2]: https://www.accessibility.nl/en/english
[3]: 
https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/commit/?id=a9925ec3b5c3be1ee1fe16d9fd51225a8be32a1d

Re: GNU Guix 1.3.0 released

[Mathieu Othacehe]

Hey,

>> This release corresponds to 8,300 commits over almost 6 months by 212
>> people.  Support for the POWER9 platform is now offered as technological
>> preview.  This release adds new features, refines the user experience
>> and improves performance.  It also includes many new packages and
>> services along bug fixes--see below for a list of changes.

Congrats! It's amazing to see how much has been achieved in six
months. Special thanks to Ludo and Maxim for your dedication and hard
work that made this release possible :).

Mathieu

Re: New blog post on the Guix Build Coordinator: Building derivations, how complicated can it be?

[Mathieu Othacehe]

Hello,

> On Sat, Apr 24, 2021 at 11:10 AM Christopher Baines  wrote:
>> I did think about trying to include something about Cuirass, but I don't
>> have a clear picture of it's scope or purpose, so I'm not really the
>> right person to attempt to write authoritatively about it.
>
> OK, fair enough, Matthieu, Ludo, or anyone else can shed some light
> here ?

You can learn more about Cuirass by reading its online manual[1].
The Cuirass server at https://ci.guix.gnu.org currently:

* Evaluates new Guix derivations on several branches (master,
  core-updates, ...).

* Builds those derivations on a build farm of 29 machines, some of them
  connected using a Wireguard VPN.

* Reports the build status on the Web interface, by email at
  guix...@gnu.org, though a Web API used by "guix pull" and "guix
  weather".
  
Maintaining and improving this whole architecture has been keeping me
busy for the last year. It involved a constant monitoring of the build
farm, upgrading the database, and rewriting Cuirass almost completely,
between other things.

The situation is now much better. Cuirass offers a nice substitutes
coverage, at least for Intel architectures, it is stable, well covered
by unit tests and documented.

We have already discussed the Cuirass vs Build coordinator situation in
the past. I haven't changed by mind on that subject. The Guix Build
Coordinator is more or less the equivalent of the Cuirass remote build
mechanism[2].

Integrating the Build coordinator in the Berlin build farm would mean
hooking in to Cuirass as an alternative to the remote build mechanism. I
don't think it would be wise because:

* It wouldn't bring any new features as far as I can tell.

* It would mean maintaining a new SQLite database.

  When Cuirass was using an SQLite database, maintaining it was a
  nightmare. I had to dive into SQLite sources, apply a wide variety of
  hacks[3] to make it scale.

  Even if the Build coordinator is updated to use a PostgreSQL database,
  that would mean having two databases, sitting next to each other,
  with a very similar content, which is a nonsense in my opinion.

* The Build coordinator has no documentation, no unit tests and a large
  code base that would drastically increase the system administrator
  burden.

It really makes me sad that we have two pieces of software that are
stepping on each other toes. The Build coordinator has a nice concept
and represents a huge amount of work. However, integrating it to Berlin
is not an option for me.

I think that the next challenges on the CI front are:

* Increase the number of ARM machines in the build farm.

* Work on substitutes mirrors.

* Find a way to make Cuirass and the Data Service work together.

and we should face those issues together, rather than having competing
software sharing the few build machines we own.

Thanks,

Mathieu

[1]: https://guix.gnu.org/cuirass/manual/html_node/index.html
[2]: 
https://guix.gnu.org/cuirass/manual/html_node/Build-modes.html#With-the-remote-build-mechanism_002e
[3]: 
https://wiki.mozilla.org/Performance/Avoid_SQLite_In_Your_Next_Firefox_Feature

Re: wip-ungrafting builds stuck

[Mathieu Othacehe]

Hello,

> Any idea what could be wrong, Mathieu?  What would you suggest to do
> when investigating such issues?

Yes I noticed it. The main problem here is that almost all workers are
stuck building Rust.

I see two actions here:

1. Understand why Rust is taking so long to build.

2. Implement the BuildDependencies table to avoid this kind of
situations where all workers are stuck building the same thing, as
discussed here: https://issues.guix.gnu.org/46402.

Thanks,

Mathieu

Re: wip-ungrafting builds stuck

[Mathieu Othacehe]

Hello,

> Any idea what could be wrong, Mathieu?  What would you suggest to do
> when investigating such issues?

Yes I noticed it. The main problem here is that almost all workers are
stuck building Rust.

I see two possible actions here.

1. Understand why Rust is taking so long to build.

2. Implement the BuildDependencies table to avoid this kind of
situations where all workers are stuck building the same thing, as
discussed here: https://issues.guix.gnu.org/46402.

Thanks,

Mathieu

Re: Following Guix weather.

[Mathieu Othacehe]

Hello Vincent,

Thanks for your feedback!

> The column sorting icon is overlapped with the columns title

Oh, didn't notice it on my large screen!

> The jobs column:
> * is there really a need for decimals for the %age, an integer would
>   be good enough ? (no more ".00%" nor "100.00%")

Yeah, I hesitated a bit about that. As the master specification has 45k+
items, those extra digits give the impression that something is going
on. But I'll think more about that.

> * could you give a bit more width so that it always fits in two lines ?

I think you are still using a cached CSS. You can reload the page using
"Ctrl - F5" to fix it. The Jobs column displays either the percentage or
the build count but not both.

Thanks,

Mathieu

Following Guix weather.

[Mathieu Othacehe]

Hello,

I have deployed a bunch of changes at https://ci.guix.gnu.org/. I'd like
to introduce some of them here.

First, there are some new specifications: "images", "tarball" and
"tests".  Those specifications are configured to build respectively, the
%guix-system-images declared in (gnu ci), the Guix binary tarball and
the Guix System tests.

Each Guix commit potentially triggers a rebuilt of all those
specifications. We have previously established that this is too resource
consuming.

That's why I have added a "period" field to the specification
definition.  This field only allows a new evaluation when at least X
seconds have elapsed since the last evaluation.  Right now, the three
aforementioned specifications have a period field set to 86400 seconds
(24 hours).

I have also improved the visibility of latest state of a
specification. For instance, if you want to know what's the current
status of the "master" branch or the "staging" branch, you will find
some statistics directly on the home page. For a more detailed view, you
can consult dashboards for each specification.

master dashboard: https://ci.guix.gnu.org/eval/19534/dashboard
staging dashboard: https://ci.guix.gnu.org/eval/19488/dashboard
test dashboard: https://ci.guix.gnu.org/eval/19464/dashboard

By having a look at those dashboards, it's obvious that we still have a
lot of work before all those circles turn to green.

Finally, in the evaluation view there's a "Toggle" button to switch
between:

- The number of newly fixed/broken/pending builds for each evaluation
  (default)
- The total number of fixed/broken/pending builds for each evaluation.

Don't hesitate to comment on those new features :)

Thanks,

Mathieu

Re: branch master updated: hydra: berlin: Accept new languages.

[Mathieu Othacehe]

Hello Julien,

> * hydra/nginx/berlin.scm (%extra-content): Autoredirect 'eo', 'ko' and 
> 'ru'
> to the translated website.

I had to revert this one that causes the following nginx error:

2021/04/07 17:05:08 [emerg] 94058#0: variable already defined: "lang" in
/gnu/store/ajvqgc205hvrfab7plbwds2a9wiqj52f-nginx.conf:4666

Mathieu

Re: Finding the channel in which a package is defined

[Mathieu Othacehe]

Hello Konrad,

> Is there a simple way to find out in which channel a given package was
> defined? I tried "guix edit" to see the source code, but it shows a file
> from a "module-union" directory in the store.

You can run something like:

--8<---cut here---start->8---
,use (guix describe) (gnu packages) (gnu packages linux)
(%package-module-path)
(package-channels strace)
--8<---cut here---end--->8---

in a "guix repl" to determine the channel providing "strace". If you
replace "strace" by a package provided by my-channel, "package-channels"
should return a list containing the default Guix channel as well as
my-channel.

Mathieu

Re: Adding Substitute Mirrors page to installer

[Mathieu Othacehe]

Hello,

Thanks for this patch, it seems to work fine!

> + ;; Extract the substitute URLs of the user configuration.
> + (os  (read-operating-system 
> (%installer-configuration-file)))
> + (substitute-urls (and=> (find
> +   (lambda (service)
> + (eq? guix-service-type
> +  (service-kind service)))
> +   (operating-system-services os))
> + (compose guix-configuration-substitute-urls
> +  service-value)))

We could make the mirror selection a proper step, adding it to (gnu
installer record). Then in (gnu installer), you could add:

--8<---cut here---start->8---
  (installer-step
   (id 'mirrors)
   (description (G_ "Mirror substitute server"))
   (compute (lambda _
  ((installer-mirrors-page current-installer)
--8<---cut here---end--->8---

This way, you should be able to select the step result in
"run-final-page" this way:

--8<---cut here---start->8---
(let* ((configuration   (format-configuration prev-steps result))
   (user-partitions (result-step result 'partition))
   (locale  (result-step result 'locale))
   (users   (result-step result 'user))
   (mirrors (result-step result 'mirrors))
   (install-ok?
(with-mounted-partitions
 user-partitions
 (configuration->file configuration)
 (run-config-display-page #:locale locale)
 (run-install-shell locale #:users users #:mirrors mirrors
  ...)
--8<---cut here---end--->8---

That would avoid the need to parse the resulting configuration file.

> +(define (run-substitute-mirror-page)
> +  (let ((title (G_ "Substitute mirror")))
> +(run-listbox-selection-page
> +  #:title title
> +  #:info-text (G_ "Choose a server to get substitutes from.
> +
> +Depending on your location, the official substitutes server can be slow; \
> +in that case, using a mirror may be faster.")

I wonder if it would make sense to select multiple mirrors, as fallback
if the preferred mirror is not up to date. We could also add the
possibility for the user to add a mirror manually.

In that case, the mirror page could look like the "User creation" page,
with an "Add" button opening a popup proposing to type a mirror URL or
to select one from an existing list.

WDYT?

Thanks,

Mathieu

Cuirass 1.0 released

[Mathieu Othacehe]

We are pleased to announce the release of Cuirass 1.0.

This is the first official release of Cuirass, the GNU Guix continuous
integration software. Since January, this project is funded through the
NGI0 PET Fund, a fund established by NLNet[1]. Thanks to this support,
we were able to speed up the developments and finally propose this
release.

Read more about today’s announcement at:

  https://gnu.org/software/guix/blog/2021/cuirass-10-released/

• About

Cuirass is the GNU Guix continuous integration software. It's a general
purpose build automation server written in GNU Guile that checks out
sources from VCS repositories, execute build jobs and store build
results in a database. Cuirass also provides a web interface to monitor
the build results.

Cuirass is running on GNU Guix build farm[2].

• Download

  Here are the compressed sources and a GPG detached signature:
https://guix.gnu.org/cuirass/releases/cuirass-1.0.0.tar.gz
https://guix.gnu.org/cuirass/releases/cuirass-1.0.0.tar.gz.sig

  Here are the SHA1 checksums:

  90a4ddbbb255353a1a90807b5dcc4ec7ed7e  cuirass-1.0.0.tar.gz
  fe46ad674be0c3d1578d2c3950717f1dcf775e1f  cuirass-1.0.0.tar.gz.sig

• Changes since version 0.0.1 (excerpt from the NEWS file)

  ** Database
  *** Switch from SQLite to PostgreSQL
  *** New test cases covering most of the SQL queries
  ** Notifications
  *** New notification mechanism with Email and Mastodon backends
  *** Add RSS events support
  
  ** Remote building
  *** Add a remote building mechanism
  *** Add a live build log mechanism
  *** Honor timeout and max-silent-time package properties
  *** Add specification and package priorities support
  *** Add Workers monitoring support with Zabbix
  ** Evaluation
  *** Rewrite evaluation mechanism to rely on Guix inferiors.
  *** Change the specifications format from an association list to a record.
  *** Use Guix Channels instead of custom Inputs.
  ** Web
  *** Add a Workers status page to monitor the remote workers status
  *** Add actions
  - Add a specification
  - Edit a specification
  - Delete a specification
  - Cancel an evaluation pending builds
  - Retry all build of an evaluation
  - Retry an evaluation
  - Restart a build
  *** Improve the specification display
  *** Fix pagination
  *** Add build weather support
  *** Add build history support

Please report bugs to bug-g...@gnu.org
Join guix-devel@gnu.org and #guix on Freenode for discussions.

Thanks to everyone who contributed to this release:

19  Christopher Baines
22  Clément Lassieur
10  Danny Milosavljevic
 8  Jan Nieuwenhuizen
 1  Jonathan Brielmaier
 1  Leo Famulari
   177  Ludovic Courtès
   143  Mathieu Lirzin
   296  Mathieu Othacehe
36  Ricardo Wurmus
 1  Robert Vollmert
 1  Roel Janssen
 6  TSholokhova
 1  Tobias Geerinckx-Rice
 1  Vincent Legoll

Mathieu

[1]: https://nlnet.nl/
[2]: https://ci.guix.gnu.org


smime.p7s
Description: S/MIME cryptographic signature

Re: armhf-linux substitutes

[Mathieu Othacehe]

Hello,

> Mathieu, what’s preventing us from doing armhf-linux builds again?  We
> could use the OverDrives for that (with an upper bound though), along
> with the SBCs in machines-for-berlin.scm.
>
> That won’t be enough to keep up, so perhaps we’ll have to restrict
> armhf-linux builds to the “core” subset or the release-manifest.scm.

With emulating builds performing badly, I'm considering building only the
'core subset for armhf and aarch64 and only on native hardware. That
would require to have the other overdrives up & running I guess.

Thanks,

Mathieu

Re: Adding Substitute Mirrors page to installer

[Mathieu Othacehe]

Hello,

I plan to have a look to your patch in the next coming days. Sorry for
the long delay.

Thanks,

Mathieu

Re: Cuirass not processing core-updates (or weirdly)

[Mathieu Othacehe]

Hello Leo,

> If you look at https://ci.guix.gnu.org/eval/13652 you can see that the
> evaluation of the derivation seems completed but there's no pending
> builds.

If you have a look to
https://ci.guix.gnu.org/specification/edit/core-updates you will see
that this specification is configured to build to "core"[1] subset of
the "core-updates" branch with the "4" priority.

The 13652 evaluation does not trigger any new derivation in the core
packages subset, hence there are no triggered builds.

When there are some new derivations to build, like for the 12656
evaluation, because of the low priority, they can stay pending for a
long time.

Thanks,

Mathieu

[1]: 
http://guix.gnu.org/cuirass/manual/html_node/Specifications.html#Specifications

Re: cuirass

[Mathieu Othacehe]

Hello,

> * The "Home" & "Guix logo" buttons bothlink back to the home page (duplicate).

Yes, but I'm not sure what to do about that. Maybe remove the logo link.

>
> * The search combobox help popup window
> is not on top level, probably on all pages.
> So other page elements are displayed over
> it.

I applied your patch, thanks!

> * Overdirve1 does not have infos (missing
> zabbix)

Yes, that's on purpose, the Overdrive resources are limited, and running
Zabbix and its PostgreSQL database wouldn't help.

> I have collected a buch of ideas for additions.
>
> Should I create bugs for those or just the
> above list will suffice ?

We can discuss them here I think. Don't hesitate if you have other
improvements ideas.

Thanks,

Mathieu

Re: [opinion] CVE-patching is not sufficient for package security patching

[Mathieu Othacehe]

Hello,

> Concretely, this would mean a Honeycomb LX2 or Ampere ALTRA workstation,
> since I don't believe there are any other aarch64 workstations available
> for sale.
>
> https://www.solid-run.com/arm-servers-networking-platforms/honeycomb-workstation/
> https://store.avantek.co.uk/ampere-altra-64bit-arm-workstation.html

I recently added a new metric in Cuirass: "Builds count per machine
during the last day". Turns out the overdrive1 with its two workers
seems to outperform the hydra-guix-X running emulated builds on four
workers.

As soon as the other overdrives are back online, the situation will
hopefully be a tiny bit better. Buying and hosting other machines such
as the ones you mentioned could also help here.

The Wireguard tunnel between berlin an the overdrive1 works fine and
configuring those machines with something similar to
"hydra/modules/sysadmin/overdrive.scm" should be enough to add them as
Cuirass workers.

Thanks,

Mathieu

Re: 01/09: gnu: Add python-sphobjinv.

[Mathieu Othacehe]

Hello,

> +   (patches (search-patches "python-sphobjinv-system-ca.patch"

Looks like this patch is missing, causing an evaluation failure.

Thanks,

Mathieu

Re: 01/09: gnu: Add python-sphobjinv.

[Mathieu Othacehe]

Hello,

> +   (patches (search-patches "python-sphobjinv-system-ca.patch"

Looks like this patch is missing, causing an evaluation failure.

Thanks,

Mathieu

Re: 04/07: inferior: Break cached-channel-instance into two procedures.

[Mathieu Othacehe]

Hey,

> We need to keep ‘cached-channel-instance’ because it’s part of the
> public API and used outside Guix (in Guix-Jupyter at least).  We can use
> ‘define-deprecated’.
>
> Also, I think ‘cached-profile’ doesn’t quite capture what this is
> about.  :-)  The docstring should mention what COMMITS is.  The fact
> that INSTANCES is a thunk is a bit awkward and IMO not great for a
> public interface.
>
> WDYT?

Yeah you're right the abstraction is really not great. I'll tweak
Cuirass to use the existing API instead.

Reverted with: 8898eaec57f6294221888e6dca1802abdd3d5868.

Thanks,

Mathieu

Re: 05/07: inferior: Fix concurrent cached-profile calls.

[Mathieu Othacehe]

Hey Ludo,

> However, there’s already a (file-exists? cached) call a few lines
> above.  So what you need instead is a TOCTTOU-free ‘symlink’, along
> these lines:
>
>   (define (symlink/safe old new)
> (catch 'system-error
>   (lambda ()
> (symlink old new))
>   (lambda args
> (unless (= EEXIST (system-error-errno args))
>   (apply throw args)
>
>   ;; …
>
>   (define symlink*
> (lift2 symlink/safe %store-monad))
>
> WDYT?

Fixed with a831ff6bc3f92ab4ecf6135e4d6386f14189ad06.

Thanks,

Mathieu

Re: [Cuirass] Hanging(?) restarted build with negative Duration

[Mathieu Othacehe]

Hello Tobias,

> I restarted  almost exactly 24h
> ago, and it's still in the ‘Started’ state.  Is it stuck?  Maybe I'm just
> overestimating our newly coordinated build farm.

The build timestamps were not reset on build restart. This is fixed
with: 37de70546bcc09f2e5f59463ed4fb1b4c3078c00.

The CI farm has been performing badly this week-end probably because of
https://issues.guix.gnu.org/46796.

I have restarted Cuirass yesterday and the build queues are now empty.

Thanks for reporting,

Mathieu