[PATCH 00/48] Extend bag-build to gexps.

[Development of GNU Guix and the GNU System distribution.]

Rationale:
Almost all build-systems are defined with gexpressions in functions
that return derivations. Derivations are not easily extensible while
gexps are. An example usage is given below.

This is a pretty big rewrite that should recompile almost all packages,
but a lot of grafting happens such as I could rebuild my system quickly.

I was trying to get the build-phases of an existing package to apply to
a local repository, because guix as a development tool for heavy packages
(emacs, ungoogled-chromium) is tedious, and there are precious info in
build-phases that can be applied in a local repository. I'm not aware of
prior work on this particular issue.

These patches allow to do extensions such as:

(build-system
  (name 'local-gnu)
  (description "GNU Build System applied in the current directory")
  (lower
   (lambda* args
 (let ((old-bag (apply
 (build-system-lower
  (package-build-system emacs-pgtk))
 args)))
   (bag
 (inherit old-bag)
 (build
  (lambda* build-args
(mlet %store-monad
((builder (apply (bag-build old-bag) build-args)))
  (return (with-imported-modules '((guix build utils))
#~(begin
(use-modules (guix build utils))
(with-directory-excursion #$(getcwd)
  #$builder

Of course this type of build-system isn't directly applicable because of
the chroot of the builder, but this other trick makes it happen :

  ;; We can't use package->derivation directly because we want the user rather
  ;; than the daemon to build the derivation.
  (with-store store
(run-with-store store
  (mlet* %store-monad ((bag -> (package->bag pkg))
   (drv(bag->derivation bag pkg)))
;; ensure inputs are in the store.
(built-derivations (derivation-inputs drv))
(with-environment-excursion
 (apply invoke (derivation-builder (pk 'd drv))
(derivation-builder-arguments drv))

This isn't polished yet, but could serve as an handy way to develop
heavy packages locally while taking advantage of the code that's
already in guix build phases.


Nicolas Graves (48):
  guix: packages: Extend bag-build to support gexp.
  build-system: gnu: Improve gnu-cross-build style.
  build-system: gnu: Redefine gnu-build and gnu-cross-build.
  build-system: agda: Redefine agda-build.
  build-system: android-ndk: Redefine gnu-build.
  build-system: ant: Redefine ant-build.
  build-system: asdf: Redefine asdf-build.
  build-system: cargo: Redefine cargo-build and cargo-cross-build.
  build-system: chicken: Redefine chicken-build.
  build-system: clojure: Redefine clojure-build.
  build-system: cmake: Redefine cmake-build and cmake-cross-build.
  build-system: composer: Redefine composer-build.
  build-system: copy: Redefine copy-build.
  build-system: dub: Redefine dub-build.
  build-system: dune: Redefine dune-build.
  build-system: elm: Redefine elm-build.
  build-system: emacs: Redefine emacs-build.
  build-system: font: Redefine font-build.
  build-system: glib-or-gtk: Improve glib-or-gtk-cross-build style.
  build-system: glib-or-gtk: Redefine glib-or-gtk-build functions.
  build-system: go: Redefine go-build and go-cross-build.
  build-system: guile: Redefine guile-build and guile-cross-build.
  build-system: haskell: Redefine haskell-build.
  build-system: julia: Redefine julia-build.
  build-system: linux-module: Redefine linux-module-build functions.
  build-system: maven: Redefine maven-build.
  build-system: meson: Redefine meson-build and meson-cross-build.
  build-system: minify: Redefine minify-build.
  build-system: mix: Redefine mix-build.
  build-system: node: Redefine node-build.
  build-system: ocaml: Redefine ocaml-build.
  build-system: perl: Redefine perl-build and perl-cross-build.
  build-system: pyproject: Redefine pyproject-build.
  build-system: python: Redefine python-build.
  build-system: qt: Redefine qt-build and qt-cross-build.
  build-system: r: Redefine r-build.
  build-system: rakudo: Redefine rakudo-build.
  build-system: rebar: Redefine rebar-build.
  build-system: renpy: Redefine renpy-build.
  build-system: ruby: Improve ruby-cross-build style.
  build-system: ruby: Redefine ruby-build.
  build-system: scons: Redefine scons-build.
  build-system: texlive: Redefine texlive-build.
  build-system: tree-sitter: Redefine tree-sitter-build functions.
  build-system: vim: Redefine vim-build.
  build-system: waf: Improve waf-build style.
  build-system: zig: Redefine zig-build.
  build-system: trivial: Redefine trivial-build functions.

 guix/build-system.scm  |   2 +-
 guix/build-system/agda.scm |   8 +-
 

Re: xwayland security updates, to mesa- or core-updates or ?

[John Kehayias]

Hi all,

Forgive the top post and please see below/previous messages for
previous updates.

TL;DR: I plan to merge mesa-updates into master today-ish (well,
tomorrow for me at this point).

I've been checking in with Efraim who's been very helpful at trying to
nudge along substitute coverage on non-x86_64 platforms. Unfortunately
looks like we have plateaued a bit on, e.g., aarch64. We haven't been
getting stats from QA for this round, and Berlin looks good for what
it covers (x86) but other architectures are down from what we can
tell.

I don't think there are any fundamental failures at this point but
just lots of "missing derivation" errors (I've restarted so many
manually for x86_64/i686) and builds not completing without restarts.
Or unknown reasons. Given the few weeks I've given this and the risk
of just perpetually doing rebuilds to keep catching up (with then more
updates to push) I think it would be best to merge to master. Mesa and
other bits will continue to move forward as well, so I think it is
time so we can be somewhat timely.

I'd rather not without complete substitute coverage, but given recent
build farm difficulties, and the tools we do have for users (pinning,
weather checks, etc.) I think it is best to call this branch so we can
move on. Gnome has some updates that will need (re)building as well as
trying to move forward with core-updates now too.

This is a case where having some better sense of our users and actual
substitute needs/wants would be helpful (yes, Guix survey!) as well as
recognizing our current infrastructure limits. Here's another vote for
prioritizing infrastructure and making sure QA lives and expands.

Feel free to object to this merge timing, though with the relative
silence in each previous message I take it I can make a call here.

Thanks everyone and hope 2024 is off to a good start! Enjoy the new
mesa with curl and xwayland security updates (no new grafts!).

John

On Thu, Jan 04, 2024 at 12:09 AM, John Kehayias wrote:

> Hi Efraim and guix-devel
>
> On Mon, Dec 25, 2023 at 08:44 AM, Efraim Flashner wrote:
>
>> On Fri, Dec 22, 2023 at 09:19:27AM +0200, Efraim Flashner wrote:
>>> On Thu, Dec 21, 2023 at 09:18:50PM +, John Kehayias wrote:
>>> > Hi all,
>>> >
>>> > On Mon, Dec 18, 2023 at 12:57 AM, John Kehayias wrote:
>>> >
> [snip]
>>> >
>>> > I haven't seen QA process this branch, so I'm just going with what I
>>> > see on Berlin. From the branches overview it shows about 61% last I
>>> > saw, compared to 72% for master. Unfortunately, non x86 architectures
>>> > are usually better covered by Bordeaux, but I don't know where to get
>>> > a sense of that coverage. For what it is worth, Efraim has manually
>>> > built xorgproto and mesa at least on powerpc64le, riscv64, without
>>> > issues.
>>>
>>> I had berlin build for powerpc64le and that went without any problems.
>>> Locally I built for riscv64 and powerpc and those both built fine.  I
>>> ran into an issue locally with curl on aarch64 and test 1477(?) which is
>>> weird since it's supposed to be skipped but I'm sending it through
>>> again.  Haven't started armhf yet.
>>>
>>> > Coverage on x86_64 and i686 seems good from what I can tell. I also
>>> > don't think there are any other branches ready to merge, and would
>>> > like to give them time to rebuild once these changes hit.
>>> >
>>> > Any thoughts on when to merge?
>>> >
>>> > Thanks everyone!
>>> > John
>>
>
> Coming back to this point, seems Berlin is doing better with building
> but I don't see mesa-updates on QA so I'm not sure about non
> x86_64/i686-linux coverage. Anyone have any thoughts?
>
> I don't know that I've seen real new failures, as still lots of
> "missing derivation" or other transient issues that resolve on forcing
> a rebuild.
>
> I don't want to merge to master and have issues with substitute
> coverage, but do have to call it at some point or will end up keep
> scheduling/waiting for rebuilds to happen anyway.
>
> Thoughts?
>
>> I've been having trouble with curl on aarch64 again. Looking at this
>> snippet from the build log:
>>
>> test 1477...[Verify that error codes in headers and libcurl-errors.3 are in 
>> sync]
>>
>>  1477: stdout FAILED:
>> --- log/1/check-expected2023-12-22 10:53:51.658667071 +
>> +++ log/1/check-generated   2023-12-22 10:53:51.658667071 +
>> @@ -1 +0,0 @@
>> -Result[LF]
>>
>>  - abort tests
>> test 1475...[-f and 416 with Content-Range: */size]
>> --pd---e--- OK (1247 out of 1472, remaining: 00:45, took 5.310s, duration: 
>> 04:11)
>> test 1474...[HTTP PUT with Expect: 100-continue and 417 response during 
>> upload]
>> --pd---e--- OK (1246 out of 1472, remaining: 00:48, took 22.794s, duration: 
>> 04:29)
>> Warning: test1474 result is ignored, but passed!
>> ...
>> TESTFAIL: These test cases failed: 1477
>>
>> It looks like 1474 is passing locally and the ~1474 is telling the test
>> suite to ignore the result.  If that's how ~ is interpreted then
>> I'd suggest that 1477 is 

Introduce Cuirass and remove Hydra on https://www.gnu.org/software/devel.html

[Jing Luo]

Hi Guix,

I am Jing Luo, a new member from the GNU webmaster team. I noticed that 
gnu.org has a page on "GNU Development Resources" [1], which has a 
section about "Hydra: Continuous builds and portability testing". As I 
know, Guix now uses Cuirass for CI. The text [1] has not been updated 
since 2012. Would anyone on this list be interested in writing something 
about Cuirass to replace the Hydra section? Or does anyone have other 
ideas about this page? You can send an email to webmasters@gnu (plural!) 
or just reply to me.



[1] https://www.gnu.org/software/devel.html

--
Jing Luo
About me: https://jing.rocks/about/
PGP Fingerprint: 4E09 8D19 00AA 3F72 1899 2614 09B3 316E 13A1 1EFC


signature.asc
Description: OpenPGP digital signature

Re: GNU Shepherd 0.10.3 released

[Wilko Meyer]

Hi Ludo,

Ludovic Courtès  writes:

> We are pleased to announce the GNU Shepherd version 0.10.3, a bug-fix
> release of the new 0.10.x series, representing 51 commits over 6 months.

Congratulations on the release & thanks to everyone that have worked on
this for their work!

>   ** New #:respawn-delay parameter to ‘service’
>  ()
>
>   This specifies a delay before a service is respawned.  Its default value is
>   given by ‘default-respawn-delay’ and defaults to 100ms.  Until now, services
>   were respawned immediately.

I've a couple of services defined that benefits from defining a respawn
delay, thanks for this!

>   ** Fix portability issues to GNU/Hurd
>
>   Previous versions in the 0.10.x and 0.9.x series did not work on GNU/Hurd.
>   This is now fixed, although some features are still implemented in a
>   suboptimal way.

This also sounds great, always happy to read news about the hurd.

-- 
Kind regards,

Wilko Meyer
w...@wmeyer.eu

Re: GNU Shepherd 0.10.3 released

[Tomas Volf]

On 2024-01-07 15:08:59 +0100, Ludovic Courtès wrote:
> We are pleased to announce the GNU Shepherd version 0.10.3, a bug-fix
> release of the new 0.10.x series, representing 51 commits over 6 months.

Congratulations on the release :)

>
>   ** Do not accidentally wait for Linux kernel thread completion
>  ()
>
>   In cases a PID file contained a bogus PID or one that’s only valid in a
>   separate PID namespace, shepherd could end up waiting for the termination of
>   what’s actually a Linux kernel thread, such as PID 2 (“kthreadd”).  This
>   situation is now recognized and avoided.

This is great, I will not have to remember to run `modprobe -r mt7921e' before
each shutdown anymore.  I hope.  Looking forward to getting it in the Guix :)

Have a nice 2024,
Tomas Volf

--
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.


signature.asc
Description: PGP signature

GNU Shepherd 0.10.3 released

[Ludovic Courtès]

We are pleased to announce the GNU Shepherd version 0.10.3, a bug-fix
release of the new 0.10.x series, representing 51 commits over 6 months.

The 0.10.x series is a major overhaul towards 1.0, addressing shortcomings
and providing new features that help comprehend system state.


• About

  The GNU Shepherd is a service manager written in Guile that looks
  after the herd of daemons running on the system.  It can be used as an
  “init” system (PID 1) and also by unprivileged users to manage
  per-user daemons—e.g., tor, privoxy, mcron.  It supports several
  daemon startup mechanisms, including inetd and systemd-style socket
  activation.  The GNU Shepherd is configured in Guile Scheme and can be
  extended in the same language.  It builds on a simple memory-safe and
  callback-free programming model.

  The GNU Shepherd is developed jointly with the GNU Guix project; it is
  used as the init system of Guix, GNU’s advanced GNU/Linux distribution.

  https://www.gnu.org/software/shepherd/


• Download

  Here are the compressed sources and a GPG detached signature:
https://ftp.gnu.org/gnu/shepherd/shepherd-0.10.3.tar.gz
https://ftp.gnu.org/gnu/shepherd/shepherd-0.10.3.tar.gz.sig

  Use a mirror for higher download bandwidth:
https://ftpmirror.gnu.org/shepherd/shepherd-0.10.3.tar.gz
https://ftpmirror.gnu.org/shepherd/shepherd-0.10.3.tar.gz.sig

  Here are the SHA1 and SHA256 checksums:

  aac0f3f11245fca5b13119dc8e25d366fc53df23  shepherd-0.10.3.tar.gz
  jsb+vrAwu1LoTGI/ECnKccLCElFRVhxZFF2qbjuFr+8  shepherd-0.10.3.tar.gz

  The SHA256 checksum is base64 encoded, instead of the
  hexadecimal encoding that most checksum tools default to.

  Use a .sig file to verify that the corresponding file (without the
  .sig suffix) is intact.  First, be sure to download both the .sig file
  and the corresponding tarball.  Then, run a command like this:

gpg --verify shepherd-0.10.3.tar.gz.sig

  The signature should match the fingerprint of the following key:

pub   rsa4096 2014-08-11 [SC]
  3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
uid   [ unknown] Ludovic Courtès 
uid   [ unknown] Ludovic Courtès 
uid   [ unknown] Ludovic Courtès (Inria) 

  If that command fails because you don't have the required public key,
  or that public key has expired, try the following commands to retrieve
  or refresh it, and then rerun the 'gpg --verify' command.

gpg --recv-keys 3CE464558A84FDC69DB40CFB090B11993D9AEBB5

  As a last resort to find the key, you can try the official GNU
  keyring:

wget -q https://ftp.gnu.org/gnu/gnu-keyring.gpg
gpg --keyring gnu-keyring.gpg --verify shepherd-0.10.3.tar.gz.sig


  This release was bootstrapped with the following tools:
Autoconf 2.71
Automake 1.16.5
Makeinfo 7.0.3
Help2man 1.49.2


• Changes since version 0.10.2 (excerpt from the NEWS file)

  ** Fix a bug that could lead shepherd to hang after loading replacements
 ()

  After loading replacements with ‘herd load’ or ‘guix system reconfigure’,
  shepherd could eventually hang.  Specifically, the replaced service would no
  longer respond to messages, so it would be impossible to start it, to stop it,
  or to unload it.  This is now fixed.

  ** Fix ownership and permissions on Unix-domain sockets
 ()

  When using an AF_UNIX endpoint with systemd- and inetd-style services, the
  socket file itself would remain owned by root (when shepherd is running as
  root) with permissions 755.  This is now fixed, with ownership set according
  to #:socket-owner and #:socket-group of the endpoint, and permissions on the
  socket set to 666.

  Likewise, #:socket-directory-permissions was previously ignored when the
  socket’s directory already existed prior to creating the endpoint, potentially
  leading to unexpectedly wide access to the socket.  This is now fixed.

  ** New #:respawn-delay parameter to ‘service’
 ()

  This specifies a delay before a service is respawned.  Its default value is
  given by ‘default-respawn-delay’ and defaults to 100ms.  Until now, services
  were respawned immediately.

  ** Non-blocking ‘sleep’ replacement provided

  Until now, user code could call (@ (guile) sleep), the core Guile binding for
  ‘sleep’, instead of ‘sleep’ as provided by (fibers).  The former would have
  caused ‘shepherd’ to actually sleep for that time, instead of performing other
  on-going tasks.  ‘sleep’ is now replaced by (@ (fibers) sleep) to avoid that.

  ** Ensure termination of services that failed to produce a PID file

  When a service started with the #:pid-file argument of
  ‘make-forkexec-constructor’ or similar would fail to produce its PID file, the
  process that was spawned would be sent SIGTERM.  Now, it is additionally sent
  SIGKILL after ‘default-process-termination-grace-period’ has expired, as is
  the case 

Re: How/where/when to ask for a patch review?

[Development of GNU Guix and the GNU System distribution.]

Hello,

Felix Lechner via "Development of GNU Guix and the GNU System
distribution."  writes:

> On Sat, Jan 06 2024, Tomas Volf wrote:
>
>> what is the proper channel to ask for a patch review and how long
>> should one wait before doing so.

I think it is reasonable to bump the bug report associated to your patch
after a week, or possibly two during a winter break.

Note that committers generally wait for a green flag in the QA page
. Unfortunately, I noticed it had some
hiccups recently.

> Please don't waste your time waiting. The process of patch review and
> acceptance here is erratic. I won't go into reasons; the project
> understands the problem and is trying several solutions but I'm not sure
> they are working. Please go ahead and maintain your own fork of Guix.

It is certainly a pragmatic answer, but hopefully, we can do better.

> No chance. I run about a hundred patches on top of stock and build half
> of all packages myself (mainly due to a defective eudev in Guix). In
> another month, I'll be at two hundred.

IMHO, at this point, you should ask for write access to the project,
This would certainly help everyone involved.

Regards,
-- 
Nicolas Goaziou