Re: Core-updates merge

[John Kehayias]

Dear Andreas and fellow Guix-ers,

On Tue, Apr 25, 2023 at 04:09 PM, Andreas Enge wrote:

> Hello all,
>
> I have just merged core-updates into master and deleted the branch!
> This has been a long adventure, which became particularly intensive
> after the last Guix Days in February. First and foremost many thanks to
> everyone who contributed to the branch, be it by commits, discussions or
> by working on the infrastructure.
>

As others have said, a big thank you to you once again for leading the
charge! Much appreciated!

> Each and every package is not yet in shape; please feel free to submit
> patches for your favourite packages that fail to build. In particular:
> - python-yubikey-manager does not build currently; work to correct this
>   is underway.

I've just submitted  which does a bit
more than just fix that package and would be good for a Python feature
branch. I'll send the cover letter to this list for wider visibility,
though the Python team was cc'ed on the series. I suppose I should add
myself to the team after this.

> - R on powerpc does not build; this will also be corrected soon;
> - aarch64 has very few substitutes; I think this is mainly due to the build
>   farm catching up and not so much to packages not building, but this is
>   difficult to know.
> If any of these are essential to you, you may wish to wait a little bit
> longer before a "guix pull", or for the time being pull to a commit just
> before the merge by issuing
>guix pull --commit=472706ae2f9160833951a4e4bcc4c206e03097b0
>
> Every end of a story is the beginning of many new ones. Several areas of
> work have already been identified; I will summarise what came up on the
> mailing list and who expressed interest to work on it - please feel free
> to join if a topic interests you, or launch another initiative if you
> want to work on a different subject!
> - rust-team already has a branch that is almost ready to be built and
>   merged, as a precursor to the team based workflow that we need to invent
>   (Efraim Flashner).
> - R on powerpc64le needs to be built by changes to valgrind and lz4
>   (Simon Tournier, I).
> - Many Python packages need updates, in particular with the aim of building
>   python-yubikey-manager (John Kehayias, Lars-Dominik Braun, Brian Cully).

I'll have to see what other patches we want to add to my series for
the branch and creating/building that as soon as possible. There is
some feedback needed on the deeper changes in my patch series, but
hopefully won't take much to finalize.

> - There is still work to do to bootstrap GHC until the latest version on
>   i686, and to potentially shorten the bootstrap chain (Lars-Dominik Braun).
> - OCaml could be simplified by dropping version 4.07 (Julien Lepiller).
> - After the mesa update is before the mesa update, and it looks like more
>   features can be enabled (Kaelyn Takata, John Kehayias).

When I get a chance next I'll be pulling the relevant patches locally
and building as a first check. And then we'll want this feature branch
to be created and built for wider testing.

> - Too much in Guix depends on too much else, which makes building things
>   needlessly entangled; in particular time zone data should not be referred
>   to by packages, but be loaded at runtime (Leo Famulari).
>
> All the best in your Guix endeavours,
>
> Andreas

Thanks everyone!
John

ci.guix.gnu.org proxy blocked

[Andrew Tropin]

Hello guix!

ci.guix.gnu.org is not available in some countries and I maintain a
nginx proxy for it at ci.guix.trop.in.

From time to time the proxy host get rate limited with following message
from ci.guix.gnu.org:

--8<---cut here---start->8---
Blocked because of DoS Attack
Your computer has been blocked because a DoS attack
originating from your system was detected. For more
information, contact the system administrator.
--8<---cut here---end--->8---

and becomes very slow.

Can we workaround it somehow?


Also, we provide ci.guix.ygg.trop.in, which make it available over
yggdrasil network (which is censorship-resistant and quite fast and also
packaged/serviced in guix).  https://yggdrasil-network.github.io/

Maybe we can setup yggdrasil node on ci.guix.gnu.org, to make it
possible to access it directly and not via proxy?  We have a tor node on
ci.guix.gnu.org, but it's very slow comparing to yggdrasil.

-- 
Best regards,
Andrew Tropin


signature.asc
Description: PGP signature

Re: Build dependency inflation (was: Re: Core-updates merge)

[bokr]

Hi,
TL;DR:

If we have 22k sources, how about expressing dependency as a
22k x 22k matrix of weights Wij and dependency by
multiplication of 22k x 1 matrix os sources variables
ordered Sj by immediate dependency first. I guess the Sj
column matrix is transposed to do the multiplication, so the
Wij j's and Sj j's match.

Start with weights of only 1 0r 0. Later maybe measures of
compile times?

The diagonal Wjj would be all ones, and counts of ones in
vertical columns Wkj would be maximal if that Sj was
directly depended on for a lot of k's

Reordering the Wjj and Sj sort of like solving simultaneous
equations and colorizing somehow to show the clusters of Wij
dependency bits vs a backgrond of zero bits might make an
interesting display. WDYT?

Any GnuPlot whizzes out there? :-)

Regards,
Bengt Richter



On +2023-04-27 19:56:38 +0200, Simon Josefsson via Development of GNU Guix and 
the GNU System distribution. wrote:
> Andreas Enge  writes:
> 
> > - Too much in Guix depends on too much else, which makes building things
> >   needlessly entangled; in particular time zone data should not be referred
> >   to by packages, but be loaded at runtime (Leo Famulari).
> 
> This is an important open problem -- is there any way to attack this
> problem in a systematic way?  I guess it is hard to understand which
> packages ends up depending on what since it is a large graph with long
> cycles, and also to understand which build depencies are essential and
> which are superficial, and thus consequently challenging to know where
> to start working to reduce build dependencies?
> 
> I wonder if it is possible to graph all the build dependencies, and do
> something like a monte-carlo what-if simulation: randomly pick one
> build-dependency from the entire build-dependency graph and remove it,
> and recompute the graph.  If the difference between these two graphs
> leads to a significantly lower total build computational cost than
> before, we may be on to something.  My theory is that "true" build
> dependencies will show up in so many places that removing just one
> instance of it will not affect total build time.  But "needless" build
> dependencies may occur just once or few times, and this approach would
> catch low-hanging fruit to work on.  Maybe the simulation needs to be
> done on more than just removing one build-dependency, it could play
> what-if games removing two build-dependencies etc, or three random
> build-dependencies, and so on.  Maybe my idea is flawed, and this will
> only lead to a list of build-dependencies that are impossible to get rid
> off anyway.
> 
> Is there some other method to understand what build dependencies would
> be important to remove, to speed up total rebuild time?
> 
> Maybe we could analyze how much of a particular build-dependency
> actually is used by a particular build?  By looking into file-access
> patterns etc.
> 
> /Simon

Build dependency inflation (was: Re: Core-updates merge)

[Development of GNU Guix and the GNU System distribution.]

Andreas Enge  writes:

> - Too much in Guix depends on too much else, which makes building things
>   needlessly entangled; in particular time zone data should not be referred
>   to by packages, but be loaded at runtime (Leo Famulari).

This is an important open problem -- is there any way to attack this
problem in a systematic way?  I guess it is hard to understand which
packages ends up depending on what since it is a large graph with long
cycles, and also to understand which build depencies are essential and
which are superficial, and thus consequently challenging to know where
to start working to reduce build dependencies?

I wonder if it is possible to graph all the build dependencies, and do
something like a monte-carlo what-if simulation: randomly pick one
build-dependency from the entire build-dependency graph and remove it,
and recompute the graph.  If the difference between these two graphs
leads to a significantly lower total build computational cost than
before, we may be on to something.  My theory is that "true" build
dependencies will show up in so many places that removing just one
instance of it will not affect total build time.  But "needless" build
dependencies may occur just once or few times, and this approach would
catch low-hanging fruit to work on.  Maybe the simulation needs to be
done on more than just removing one build-dependency, it could play
what-if games removing two build-dependencies etc, or three random
build-dependencies, and so on.  Maybe my idea is flawed, and this will
only lead to a list of build-dependencies that are impossible to get rid
off anyway.

Is there some other method to understand what build dependencies would
be important to remove, to speed up total rebuild time?

Maybe we could analyze how much of a particular build-dependency
actually is used by a particular build?  By looking into file-access
patterns etc.

/Simon


signature.asc
Description: PGP signature

Re: [Rust Team] Status post core-updates merge

[Efraim Flashner]

On Thu, Apr 27, 2023 at 07:06:29PM +0200, Josselin Poiret wrote:
> Hi Efraim (and the rest of the Rust team),
> 
> Efraim Flashner  writes:
> 
> > Now that core-updates has been merged into master it's time for a status
> > update from the rust team.
> >
> > Currently rust supports (in the order of gaining support) x86_64,
> > aarch64 and riscv64. GDB-11 doesn't currently build on aarch64 or
> > riscv64, so we're removing gdb-11 entirely and using a new gdb-12 as
> > gdb/pinned for the test suite. This is expected to fix rust building on
> > aarch64.
> 
> Is there any hope for the other architectures?  I'm thinking i686
> mostly, but s390x, powerpc and mips are all on the radar.

There hasn't been much development in the mrustc repo for the past few
months and I believe the issues blocking i686 (needs too much memory at
once) is still present. As far as powerpc64le, it does pretty well with
mrustc but doesn't get through the whole bootstrap yet.

Another option would be to work on the cross-compiling ability of the
cargo-build-system, then you could at least cross compile to them.

I will point out that s390x and mips aren't actually supported in Guix
currently.

> > rust-bootstrap currently fails to build on riscv64. I've been testing
> > solutions by building on an x86_64 machine to see what builds pass and
> > what fails, and I think I've found a solution with minimal changes.
> >
> > Once I'm able to successfully build rust-bootstrap on x86_64 and riscv64
> > I'll push the update to the rust-team branch and let the CI build it
> > out.
> >
> > As a quick overview, this push moves our rust package to rust-1.67 and
> > updates many of our rust packages. We've had some more unbundling of
> > bundled library sources and precompiled libraries and updated a number
> > of rust programs, mostly so they build with the newer version of rust.
> 
> Very impressive!  I only watch from afar and complain about the rust
> situation whenever I can, but this is giving me hope.  With Rust getting
> further and further down into the dependency chain, this is vital work,
> and you all seem to be handling it very well.
> 
> I'm also curious, since you're "early adopters": how are you managing
> with the team workflow?  Any tips, remarks or impressions that you could
> share with other teams?

Speak in the plural and no one will realize it's a team of one ;)

I'm thinking of tagging the rust stuff with [rust team] so it stands out
and is easily searchable. if/when I send patches for other teams I'll
probably tag those too and hope it catches on.


-- 
Efraim Flashner  אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted


signature.asc
Description: PGP signature

Re: [Rust Team] Status post core-updates merge

[Josselin Poiret]

Hi Efraim (and the rest of the Rust team),

Efraim Flashner  writes:

> Now that core-updates has been merged into master it's time for a status
> update from the rust team.
>
> Currently rust supports (in the order of gaining support) x86_64,
> aarch64 and riscv64. GDB-11 doesn't currently build on aarch64 or
> riscv64, so we're removing gdb-11 entirely and using a new gdb-12 as
> gdb/pinned for the test suite. This is expected to fix rust building on
> aarch64.

Is there any hope for the other architectures?  I'm thinking i686
mostly, but s390x, powerpc and mips are all on the radar.

> rust-bootstrap currently fails to build on riscv64. I've been testing
> solutions by building on an x86_64 machine to see what builds pass and
> what fails, and I think I've found a solution with minimal changes.
>
> Once I'm able to successfully build rust-bootstrap on x86_64 and riscv64
> I'll push the update to the rust-team branch and let the CI build it
> out.
>
> As a quick overview, this push moves our rust package to rust-1.67 and
> updates many of our rust packages. We've had some more unbundling of
> bundled library sources and precompiled libraries and updated a number
> of rust programs, mostly so they build with the newer version of rust.

Very impressive!  I only watch from afar and complain about the rust
situation whenever I can, but this is giving me hope.  With Rust getting
further and further down into the dependency chain, this is vital work,
and you all seem to be handling it very well.

I'm also curious, since you're "early adopters": how are you managing
with the team workflow?  Any tips, remarks or impressions that you could
share with other teams?

Best,
---
Josselin Poiret


signature.asc
Description: PGP signature

Re: Blog post on the Full-Source Bootstrap

[Simon Tournier]

Hi Janneke,

On mer., 26 avril 2023 at 16:12, Janneke Nieuwenhuizen  wrote:

>   
> https://gnu.org/software/guix/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/

Really cool!


Maybe I misread the wording:

If you run guix pull today, you get a package graph of more than
22,000 nodes rooted in a 357-byte program
or
First, while the package graph is rooted in a 357-byte program,

Well, without being the accountant, considering the “more than 22,000
nodes”, the revision fa685c8 contains ~23,000 packages and more than
1700 packages depends on GHC (Haskell) which is not bootstrapped from
this 357-byte program, AFAIK.

--8<---cut here---start->8---
$ guix refresh -l ghc@9.2 | cut -f1 -d':'
Building the following 569 packages would ensure 1712 dependent packages are 
rebuilt
--8<---cut here---end--->8---

Because Pandoc, it appears in many non-Haskell packages, here an example
about R and bioinformatics:

--8<---cut here---start->8---
$ guix show r-catalyst | recsel -p synopsis
synopsis: Cytometry data analysis tools  

$ guix graph --path r-catalyst ghc@9.2 -t bag-emerged
r-catalyst@1.22.0
r-dplyr@1.1.1
r-pillar@1.9.0
r-utf8@1.2.3
r-rmarkdown@2.21
pandoc@2.19.2
ghc@9.2.5
--8<---cut here---end--->8---

In addition, that’s similar for ~450 packages relying on OCaml.

Bah, I am nitpicking.  Am I? :-)


What achievement this bootstrap story!  Thanks to all the people
involved.  Back on FOSDEM 2017, I remember “Mes -- Maxwell's Equations
of Software An attempt at dissolving bootstrap binaries” [1].  Thanks to
this talk, I had this kind of revelation:

Yes, that was the big revelation to me when I was in graduate
school—when I finally understood that the half page of code on
the bottom of page 13 of the Lisp 1.5 manual was Lisp in
itself. These were “Maxwell’s Equations of Software!”

– Alan Kay – 

Then, I got the point with Yogurt metaphor [2] on FOSDEM 2019.  Anyway!

Thank you.

Cheers,
simon

1: https://archive.fosdem.org/2017/schedule/event/guixsdbootstrap/
2: https://archive.fosdem.org/2019/schedule/speaker/jan_janneke_nieuwenhuizen/

[Rust Team] Status post core-updates merge

[Efraim Flashner]

Now that core-updates has been merged into master it's time for a status
update from the rust team.

Currently rust supports (in the order of gaining support) x86_64,
aarch64 and riscv64. GDB-11 doesn't currently build on aarch64 or
riscv64, so we're removing gdb-11 entirely and using a new gdb-12 as
gdb/pinned for the test suite. This is expected to fix rust building on
aarch64.

rust-bootstrap currently fails to build on riscv64. I've been testing
solutions by building on an x86_64 machine to see what builds pass and
what fails, and I think I've found a solution with minimal changes.

Once I'm able to successfully build rust-bootstrap on x86_64 and riscv64
I'll push the update to the rust-team branch and let the CI build it
out.

As a quick overview, this push moves our rust package to rust-1.67 and
updates many of our rust packages. We've had some more unbundling of
bundled library sources and precompiled libraries and updated a number
of rust programs, mostly so they build with the newer version of rust.

-- 
Efraim Flashner  אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted


signature.asc
Description: PGP signature

Re: [PATCH] gnu: guitarix: Update to 0.44.1.

[Development of GNU Guix and the GNU System distribution.]

Hi John,

On Wed, Apr 26, 2023 at 3:28 PM John Kehayias
 wrote:
>
> a quick patch that
> updates the package which builds fine for me locally; I didn't test
> running it though.

Thanks for looking into it! An untested executable strikes me as
superior to one that does not build. I would commit the patch,
regardless of form, at the earliest opportunity.

Kind regards
Felix

P.S. You may find the X-Debbugs-CC header helpful. [1]

[1] https://debbugs.gnu.org/Reporting.html#xcc

Re: `mumi send-email' means no more debbugs dance to send multiple patches

[Josselin Poiret]

Hi Arun,

Arun Isaac  writes:

> Could you link to the specific email message that you tried sending with
> `mumi send-email'? Debbugs provides the complete email message in mbox
> format. The email headers should have timestamps we can look at to
> ascertain how long it took. I am hoping that this was a one-off spurious
> delay somewhere in the email system. I would certainly encourage you to
> try using `mumi send-email' again to see if the problem persists. If it
> does, we should certainly try and do something about it.

Thanks for looking into it.  I have tried twice now, with the same issue
both times, with [1] and [2].

[1] mid:cover.1682364024.git@jpoiret.xyz
https://yhetil.org/guix/cover.1682364024.git@jpoiret.xyz/

[2] mid:cover.1682452738.git@jpoiret.xyz
https://yhetil.org/guix/cover.1682452738.git@jpoiret.xyz/

Best,
-- 
Josselin Poiret


signature.asc
Description: PGP signature