Re: Managing patches and branches, retrospective and futher changes?

[Christopher Baines]

Steve George  writes:

> I think we should strongly recommend against long-running unmerged branches.
>
> Perhaps there could be a recommendation to merge every 3 months.

My hope is that with these process changes, we won't end up with
long-running branches.

Maybe we could add a recommendation, but I'm not really sure if that
would help. We still want to merge these changes when they and related
things are ready, so it's not really something that can be willed or
commanded to happen.

> Could we add any automation to remind people if:
>
> 1. a branch has been unmerged for more than 3 months

We can have QA highlight how long the issues have been open for, that's
quite easy to do.

> 2. an odd merge takes places (e.g. the core-updates merges)

I'm not quite sure what you mean here?

Thanks,

Chris


signature.asc
Description: PGP signature

Managing patches and branches, retrospective and futher changes?

[Christopher Baines]

Hey!

Almost a year ago, the branching strategy was changed [1][2].

1: https://issues.guix.gnu.org/63459
2: https://lists.gnu.org/archive/html/guix-devel/2023-06/msg00024.html

I think these changes have gone OK, we've had ~27 [3] branches merged in
this manor and I think looking back these changes have been
helpful. Coordination and visibility has improved, and we've been able
to build and test more prior to merging.

3: https://issues.guix.gnu.org/search?query=%22Request+for+merging%22

There's still room for more improvement though. The current guidance is
here [4], and I've sent a patch for improvements here [5].

4: 
https://guix.gnu.org/en/manual/devel/en/html_node/Managing-Patches-and-Branches.html
5: https://issues.guix.gnu.org/70549

Let me know if you have any thoughts or questions!

Thanks,

Chris


signature.asc
Description: PGP signature

Re: Status of ‘core-updates’

[Christopher Baines]

Ludovic Courtès  writes:

> What’s the status of ‘core-updates’?  What are the areas where help is
> needed?
>
> I know a lot has happened since the last update¹, which is roughly when
> I dropped the ball due to other commitments, but I’m not sure where we
> are now.

I haven't really been following core-updates, but I have had a look
since there's a request to merge it now [1].

I'm really concerned by the commits on the branch though, assuming I'm
using Git right, there are 6351 commits on the branch:

  git log --pretty=oneline core-updates ^master | wc -l

Somehow, I think there's been a couple of pushes of commits to
core-updates that have partially duplicated lots of commits from master,
I put some more details in:

  https://issues.guix.gnu.org/70456#3

I think keeping the Git commit history clean and representative is
really important, so to me at least this means core-updates can't be
merged to master in it's current form, even if the changes overall from
these 6351 commits are reasonable.

I'm really not sure how to move forward though, I had a go at trying to
rebuild the branch without introducing the thousands of duplicate
commits and that produced a branch with 765 commits over master, which
still seems a lot, but a big improvement over 6351:

  https://git.cbaines.net/guix/log/?h=chris-core-updates-no-duplicates-attempt

That was really hard going though, as there's plenty of merge conflicts
along the way, and I'm pretty sure I solved some of them
incorrectly. The resulting branch also differs from core-updates.

Maybe someone with more time, care and attention could do a better job,
but it might be more worthwhile just starting fresh and rather than
trying to produce a like for like branch just without the thousands of
duplicate commits, effectively manually rebase the branch (without the
duplicate commits) on master and try to get the commits in to a usable
state.

Any ideas?

Thanks,

Chris


signature.asc
Description: PGP signature

Re: Error handling when 'guix substitute' dies

[Christopher Baines]

Ludovic Courtès  writes:

> Hi,
>
> Philip McGrath  skribis:
>
>> I don't know if the root cause is related, but this reminded me of
>> some networking errors I sometimes get accessing substitutes. I had
>> the luck (good or bad?) to get an example while building
>> , so I thought I'd report.
>>
>>> ice-9/boot-9.scm:1685:16: In procedure raise-exception:
>>> ERROR:
>>>   1. :
>>>   uri: #< scheme: https userinfo: #f host: 
>>> "bordeaux-us-east-mirror.cbaines.net" port: #f path: 
>>> "/nar/lzip/r8w68mhsvqlkvazncvfn36jcpvz404x6-chez-fmt-0.8.11.tar.gz" query: 
>>> #f fragment: #f>
>>>   code: 404
>>>   reason: "Not Found"
>>>   headers: ((server . "nginx") (date . #>> minute: 20 hour: 6 day: 1 month: 4 year: 2024 zone-offset: 0>) 
>>> (content-type application/json (charset . "utf-8")) (content-length . 35) 
>>> (connection keep-alive))
>>>   2. : 
>>> "https://bordeaux-us-east-mirror.cbaines.net/nar/lzip/r8w68mhsvqlkvazncvfn36jcpvz404x6-chez-fmt-0.8.11.tar.gz:
>>>  HTTP download failed: 404 (\"Not Found\")"
>>> substitution of 
>>> /gnu/store/r8w68mhsvqlkvazncvfn36jcpvz404x6-chez-fmt-0.8.11.tar.gz failed
>>> guix build: error: corrupt input while restoring archive from #>> file 7f9f7deaad90>
>
> No, that’s another problem: bordeaux.guix would delete nars before their
> advertised TTL has expired.  See .
> Chris explained that this was solved recently I believe.

This only applied to zstd nars, whereas this is an lzip nar that was
apparently missing at the time.

Maybe this is/was some problem with the mirror.


signature.asc
Description: PGP signature

Re: Emacs and Gnome branches are merged now

[Christopher Baines]

Liliana Marie Prikler  writes:

> I've now pushed the merge commits for both emacs-team and gnome-team.

Thank you to everyone involved in getting these changes through :D

> If you have a weak machine, PLEASE DO NOT PULL IMMEDIATELY AND WAIT FOR
> CI TO CATCH UP!  Despite efforts to prebuild things on the respective
> branches, the merge commit carries with it a rebuild of the most nasty
> package to have to compile locally.

I've had a look in to this as it would be good to work out how this
happened, and how we might avoid it in the future.

I had a look at QEMU and gtk, diffing the derivations from the
gnome-team branch prior to the merge to the master branch after the
merge.

QEMU was affected by the usbutils update in
855fecf00f7df8bf878a8ecd47800ea9bdfebea5, which was pushed to master on
the 26th of March. For gtk, it was affected by the psmisc update in
a2d82fbec4254cf6127b15aa3e827d22da235c30 which was pushed at the same
time.

I think the last merge of master in to gnome-team was pushed on the 20th
of March, with the merge of gnome-team in to master being pushed today
on the 30th.

Looking at those dates, it seems like bringing changes from master in to
branches more frequently, or at least just before considering whether
it's ready to merge (and merging if it is) would help to avoid this. 10
days is a long window in which changes can be pushed to master. QA is
meant to pick up when a branch has diverged from master, but the
mechanism is crude and the threshold it was using was very high. I've
now reduced it [1] so QA might warn in the future about this situation.

1: 
https://git.savannah.gnu.org/cgit/guix/qa-frontpage.git/commit/?id=b1d7225636d7c1946f4718d6441a621d51f8cd6a

While I don't think it's directly relevant, I think it's worth noting
that both changes mentioned above (usbutils and psmisc), we didn't
follow our own guidance on managing patches. psmisc affects more than
300 dependent packages, and while I think that's less of an issue if
changes go through QA, I don't think either of these changes did.



signature.asc
Description: PGP signature

Re: March update on bordeaux.guix.gnu.org

[Christopher Baines]

Ludovic Courtès  writes:

> Hi!
>
> Christopher Baines  skribis:
>
>> Related to this, I've added options to the nar-herder to help change the
>> TTL being used, and reduced the TTL for bordeaux.guix.gnu.org to 10
>> minutes (from 180 days) [4]. This will at least mean that in the future,
>> the nar-herder on bordeaux will be able to delete zstd compressed nars
>> that it's generated more quickly.
>
> It’s not 10mn right now:
>
> $ wget --debug -qO/dev/null 
> https://bordeaux.guix.gnu.org/yr39rh6wihd1wv6gzf7w4w687dwzf3vb.narinfo 2>&1 
> |grep Cache
> Cache-Control: max-age=15502374
>
>
> Or maybe that’s just for newly created nars?

The max-age of that narinfo is currently based on the scheduled removal
of the zstd compressed nar, which is going to happen quite far in the
future.

I did think of a number of ways to approach this, and I'm not sure I've
settled on the right one yet. Maybe the TTL should be capped at 600, and
then drop to 0 as the time to remove the zstd nar approaches?

This narinfo for example currently has a max-age of 600:

  https://bordeaux.guix.gnu.org/ganfjbgy75r31bwzgddpnpswwjrrffvj.narinfo

> But then again, that’s the advertised TTL; the real TTL is still
> infinite, right?

As you probably know, the situation is more complex.

The problems caused when the nar-herder started removing zstd compressed
nars shows the difference between retention of the nar in some form, and
whether a cached narinfo response can be considered fresh or stale.

Users might also not notice the availability of zstd nars if they cache
responses forever, since currently there will be a lag between the nar
becoming available, and a zstd compression being created (although we
could generate zstd compressed nars for everything).

As described below, I also do want to start removing some nars, and that
requires not having an infinite TTL.

>> I'm really unsure about the need/usefulness of narinfo caching in
>> general, the cost of storing all these narinfos locally is quite high I
>> think and I don't really know why it's done.
>
> It’s a cache.  It’s useful to have this cache because in “typical” Guix
> usage you’re likely to ask repeatedly for the same substitutes.
>
> Regarding the cost, 3f5e14182931f123c10513a3e1e2abaebfb52279 made things
> more reasonable by putting a higher bound on narinfo retention.  On my
> laptop, I have:
>
> $ ls -lrt 
> /var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjefq7kiciw2pfhaf26a/
>  |wc -l
> 11549
> $ du -h 
> /var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjefq7kiciw2pfhaf26a/
>  
> 50M 
> /var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjefq7kiciw2pfhaf26a/
>
> Maybe that’s still excessive and we could further reduce the maximum
> caching time.

Having played around with this a bit (e.g. hacking guix weather not to
cache), I'm a bit sceptical. Given maintaining the cache takes time that
could be spent doing network I/O, and does potentially slow disk I/O, I
think it would be interesting to try and work out in what situations the
cache speeds things up overall, and in what situations it slows things
down overall..

>> 6: https://lists.gnu.org/archive/html/guix-devel/2023-05/msg00290.html
>
> BTW, should we document this mirror somewhere (and also ensure that Guix
> Foundation pays the bills), or do you view it more as an experiment for
> now?

If the project does want to provide mirrors, I think that would be
great. From this experiment, I think we have some evidence that there
are people using Guix outside of Europe, and in some cases they struggle
with the European based infrastructure. It also seems like these mirrors
do help, and the monetary cost isn't too high in my view.

I think we should probably wait until the project starts managing them
before documenting/advertising them more widely though.

>> Apart from that, the main thing on my mind for the next year regarding
>> bordeaux substitutes specifically is storage space. We're at 90%
>> capacity on hatysa (one of the two machines storing all the nars) so
>> this will need looking at shortly. I'd also like to finally get removing
>> nars that don't relate to the guix master branch happening, as that
>> should free up a little bit of space at least.
>
> Nice (the difficulty, I guess, is that some substitutes that we not
> initially for ‘master’ eventually get used on ‘master’).

Yep, my plan is to wait some long amount of time (say 6 months) before
scheduling things for removal, to check that they haven't started being
used by the master branch in this time.

We could also add other criteria as well, like tracking which nars are
generated by fixed output derivations and never removing them.

> On this issue, I think we should learn from fellow NixOS

March update on bordeaux.guix.gnu.org

[Christopher Baines]

Hey!

I'm think the last update I sent out was back in April 2023 [1], but
it's coming up to 3 years since bordeaux was added as a default
substitute server [2]. It's scary how much time has passed!

1: https://lists.gnu.org/archive/html/guix-devel/2023-04/msg00319.html
2: 
https://guix.gnu.org/en/blog/2021/substitutes-now-also-available-from-bordeauxguixgnuorg/

I've finally got around to starting to address the problems with
disappearing nars discussed in [3]. The nar-herder now schedules nars
which it's generated for removal and the time for removal is based on
the TTL in use.

3: https://issues.guix.gnu.org/63634

Related to this, I've added options to the nar-herder to help change the
TTL being used, and reduced the TTL for bordeaux.guix.gnu.org to 10
minutes (from 180 days) [4]. This will at least mean that in the future,
the nar-herder on bordeaux will be able to delete zstd compressed nars
that it's generated more quickly.

4: 
https://git.savannah.gnu.org/cgit/guix/maintenance.git/commit/?id=1fc5a1af488cb8838c2f196e07a43f37f2fbf509

I'm really unsure about the need/usefulness of narinfo caching in
general, the cost of storing all these narinfos locally is quite high I
think and I don't really know why it's done.

The mishandling of these zstd nars available from bordeaux was the last
thing on my mind blocking proposing to switch the default substitute
server ordering, so I've now gone ahead and done that in [5].

5: https://issues.guix.gnu.org/70028

Obviously the differences to having ci.guix.gnu.org first in the list of
substitute URLs is subtle, but I'm hoping this will help with substitute
speed issues (as discussed in [6]) plus increase the impact of mirroring
work for the bordeaux substitutes in the future.

6: https://lists.gnu.org/archive/html/guix-devel/2023-05/msg00290.html

Apart from that, the main thing on my mind for the next year regarding
bordeaux substitutes specifically is storage space. We're at 90%
capacity on hatysa (one of the two machines storing all the nars) so
this will need looking at shortly. I'd also like to finally get removing
nars that don't relate to the guix master branch happening, as that
should free up a little bit of space at least.

Let me know if you have any comments or questions!

Thanks,

Chris


signature.asc
Description: PGP signature

Re: qa.guix.gnu.org: Returns a plain Guile error for a merged issue

[Christopher Baines]

"Artyom V. Poptsov"  writes:

> Hello,
>
> I just found out that Guix QA returns a plain Guile error message when I
> try to open an issue that was merged:
>   https://qa.guix.gnu.org/issue/69654
>
> The error looks like this:
>
> An error occurred
>
> Sorry about that!
> wrong-type-arg
>
> #fWrong type to apply: ~S#f#f
>
> Please find a screenshot of the error attached.
>
> I suppose that the service should return something more meaningful than
> that.

I think this was happening because QA had problems talking to Mumi
(issues.guix.gnu.org). That's now resolved and it's back to the still
not very useful "Issue not found" message.


signature.asc
Description: PGP signature

Re: Concerns/questions around Software Heritage Archive

[Christopher Baines]

MSavoritias  writes:

> On 3/16/24 19:50, Christopher Baines wrote:
>> Ian Eure  writes:
>>
>>> Hi Guixy people,
>>>
>>> I’d never heard of SWH before I started hacking on Guix last fall, and
>>> it struck me as rather a good idea.  However, I’ve seen some things
>>> lately which have soured me on them.
>>>
>>> They appear to be using the archive to build LLMs:
>>> https://www.softwareheritage.org/2024/02/28/responsible-ai-with-starcoder2/
>>>
>>> I was also distressed to see how poorly they treated a developer who
>>> wished to update their name:
>>> https://cohost.org/arborelia/post/4968198-the-software-heritag
>>> https://cohost.org/arborelia/post/5052044-the-software-heritag
>>>
>>> GPL’d software I’ve created has been packaged for Guix, which I assume
>>> means it’s been included in SWH.  While I’m dealing with their (IMO:
>>> unethical) opt-out process, I likely also need to stop new copies from
>>> being uploaded again in the future.
>>>
>>> Is there a way to indicate, in a Guix package, that it should *never*
>>> be included in SWH?
>> Not currently, and I don't really see the point in such a mechanism. If
>> you really never want them to store your code, then you need to license
>> it accordingly (and not make it free software).
>
> You are talking about legal tho. Yes legally they can copy the code.
>
> But what can Guix do socially to give people the choice? For reasons
> of consent that is.

...

>>> I was also distressed to see how poorly they treated a developer who
>>> wished to update their name:
>>> https://cohost.org/arborelia/post/4968198-the-software-heritag
>>> https://cohost.org/arborelia/post/5052044-the-software-heritag
>> This is probably worth thinking about as Guix is in a similar situation
>> regarding publishing source code, and people potentially wanting to
>> change historical source code both in things Guix packages and Guix
>> itself.
>>
>> Like Software Heritage, there's cryptographical implications for
>> rewriting the Git history and modifying source tarballs or nars that
>> contain source code.
>>
>> We have 17TiB of compressed source code and built software stored for
>> bordeaux.guix.gnu.org now and we should probably work out how to handle
>> people asking for things to be removed or changed (for any and all
>> reasons).
>>
>> It's probably worth working out our position on this in advance of
>> someone asking.
>
> I would go a step further actually. Software Heritage is effectively
> breaking CoC of Guix now.
>
> Im not proposing removing all code or something obviously that
> connects to Software Heritage, but there should be some social action
> we can take.
>
>
> For example until the matter is resolved and Software Heritage
> implements a process that respects trans rights Software Heritage
> should not be welcome in Guix Spaces.

As I say, Guix is in a very similar situation as a project to Software
Heritage, we publish artefacts containing peoples personal details and
there are technical implications in changing the personal details in
those artefacts.

The only difference as far as I'm aware is that no one is currently
asking Guix as a project to update their personal details in the
artefacts we store and publish.

As a project, we should sort out our stuff before jumping to judge
others.


signature.asc
Description: PGP signature

Re: Concerns/questions around Software Heritage Archive

[Christopher Baines]

Ian Eure  writes:

> Hi Guixy people,
>
> I’d never heard of SWH before I started hacking on Guix last fall, and
> it struck me as rather a good idea.  However, I’ve seen some things
> lately which have soured me on them.
>
> They appear to be using the archive to build LLMs:
> https://www.softwareheritage.org/2024/02/28/responsible-ai-with-starcoder2/
>
> I was also distressed to see how poorly they treated a developer who
> wished to update their name:
> https://cohost.org/arborelia/post/4968198-the-software-heritag
> https://cohost.org/arborelia/post/5052044-the-software-heritag
>
> GPL’d software I’ve created has been packaged for Guix, which I assume
> means it’s been included in SWH.  While I’m dealing with their (IMO:
> unethical) opt-out process, I likely also need to stop new copies from
> being uploaded again in the future.
>
> Is there a way to indicate, in a Guix package, that it should *never*
> be included in SWH?

Not currently, and I don't really see the point in such a mechanism. If
you really never want them to store your code, then you need to license
it accordingly (and not make it free software).

> Is there a way to tell Guix to never download source from SWH?

Also no, and it's probably best to do this at the network level on your
systems/network if you want this to be the case.

Skipping back to this though:

> I was also distressed to see how poorly they treated a developer who
> wished to update their name:
> https://cohost.org/arborelia/post/4968198-the-software-heritag
> https://cohost.org/arborelia/post/5052044-the-software-heritag

This is probably worth thinking about as Guix is in a similar situation
regarding publishing source code, and people potentially wanting to
change historical source code both in things Guix packages and Guix
itself.

Like Software Heritage, there's cryptographical implications for
rewriting the Git history and modifying source tarballs or nars that
contain source code.

We have 17TiB of compressed source code and built software stored for
bordeaux.guix.gnu.org now and we should probably work out how to handle
people asking for things to be removed or changed (for any and all
reasons).

It's probably worth working out our position on this in advance of
someone asking.


signature.asc
Description: PGP signature

Re: February update on the Guile guix-daemon

[Christopher Baines]

Ludovic Courtès  writes:

> Christopher Baines  skribis:
>
>>  - Then there's the big areas to work on next:
>>
>>- I think I'm going to need to use thread pools for SQLite operations
>>  in the daemon, as the build coordinator does.
>
> I think we should refrain from using POSIX threads directly and instead
> use Fibers to its full extent.  In this case, I’d use a resource pool as
> done in Cuirass (and in the Coordinator too, no? maybe that’s what you
> meant?).
>
>>- There's the low level work of setting up the build environment, the
>>  work on the guile-daemon branch helps a lot with this, but as
>>  pointed out by Ludo, there might be some issues with fork and
>>  similar operations in a Guile program using threads.
>
> Right.  I think one of the things we discussed in Brussels is that,
> since SQLite operations might block for a while, the daemon will have to
> be multithreaded.  Which means no fork/clone, which in turn probably
> means delegating forking/cloning to a separate helper process.  (That’s
> also what bubblewrap does, IIUC.)

I've had a bit more of a think and have thought of some more things we
can try, but I'm still not sure if it can be made to work.

We can call (sleep 0) after every sqlite-step to allow other fibers to
run if they're able to [1] and if we find places where sqlite-step takes
too long, maybe we can use a short lived thread for this, but then make
sure these threads have stopped before doing anything where they would
cause problems (e.g. fork/clone).

1: https://github.com/wingo/fibers/wiki/Manual#31-blocking


signature.asc
Description: PGP signature

February update on the Guile guix-daemon

[Christopher Baines]

Hey!

Here's an overdue update on rewriting the Guix daemon in Guile,
following on from the earlier thread on guix-devel [1] and the blog post
[2].

1: https://lists.gnu.org/archive/html/guix-devel/2023-09/msg00328.html
2: https://guix.gnu.org/en/blog/2023/a-build-daemon-in-guile/

Unfortunately I was quite slow to actually get stuck in with the
code. Among other things, I've been distracted by trying to at least
work around long running issues with the build coordinator and data
service. I wanted to try and get QA back up and running and reduce the
number of times that manual intervention is required, and this has been
successful to some extent. I'm going to try harder to focus on the
guix-daemon in the coming months however.

With the time I have managed to spend working on the daemon though, I
think I've made some progress. I started by getting a process to listen
on a socket, then worked through implementing operations used by guix
build (with many simplifying assumptions that is).

This has given me some assurances about the viability of the technical
plan I outlined in the blog post, as well as filling in some of the
details.

These are my current thoughts about the work ahead:

 - I've sent a couple of patch series [3][4] to adapt parts of Guix for
   use in the daemon, there'll need to be more changes in the database
   and substitute areas though.

 - I'm starting to understand, test and adapt the work on the
   guile-daemon branch, but there's more that needs doing on this.

 - Then there's the big areas to work on next:

   - I think I'm going to need to use thread pools for SQLite operations
 in the daemon, as the build coordinator does.

   - There's the low level work of setting up the build environment, the
 work on the guile-daemon branch helps a lot with this, but as
 pointed out by Ludo, there might be some issues with fork and
 similar operations in a Guile program using threads.

   - While I've implemented some of the server side protocol used by the
 daemon, I'd like to extract that code in to a module so that it's
 not wrapped up inside the daemon script

3: https://issues.guix.gnu.org/69291
4: https://issues.guix.gnu.org/69292

I think once I've made more progress in the above areas, I will
hopefully have something that people could attempt to use in a limited
capacity.

Let me know if you have any comments or questions,

Thanks,

Chris


signature.asc
Description: PGP signature

Re: Guix Days: Patch flow discussion

[Christopher Baines]

Clément Lassieur  writes:

> On Fri, Feb 16 2024, Andreas Enge wrote:
>
>> Am Fri, Feb 16, 2024 at 11:56:50AM +0100 schrieb Clément Lassieur:
>>> Would it makes sense to have a "does-not-apply" tag too?
>>
>> Should this not appear in the QA page, assuming that once all the new
>> issues are closed, older ones will bubble to the top and be treated by QA?
>> (I am not sure if just looking at the n latests issues is how QA works,
>> but I think so.)
>
> I think it would be useful (at least for me) if I browse a list of
> patches that I want to review, to document why a review can't be done.
>
> Also, so that other people won't try to apply it.
>
> It would be great that QA does that job, I imagine when it does it it
> can also add that tag.

There's the filter issues form which allows finding patches which can't
be applied:

  https://qa.guix.gnu.org/patches?failed-to-apply-patches=on

I think QA tagging issues might be useful for people to benefit from
that data in other places, and should be possible, it just needs QA to
send emails I think.


signature.asc
Description: PGP signature

Re: QA is back, who wants to review patches?

[Christopher Baines]

Vivien Kraus  writes:

> Hello Chris,
>
> Le vendredi 09 février 2024 à 10:44 +, Christopher Baines a écrit :
>> Let me know if you have any comments or questions!
>
> Thank you for all your work on QA.
>
> I can’t help but notice QA is missing a few patches. For instance,
> issues.guix.gnu.org lists 7 open issues with patches for gnome-team
> (#67623, #67493, #67273, #6648, #68937, #68716, #68911) but if I search
> for gnome-team on qa.guix.gnu.org, it only shows 2: #68937 and #68716.
> Do you know why the others were lost?

QA only looks at a fixed number of recent issues (by the time the latest
patches were sent).

This is mostly a disk space and memory limitation on beid which runs
data.qa.guix.gnu.org, so hopefully we can get more resources and
increase the number of issues to look at


signature.asc
Description: PGP signature

Re: QA is back, who wants to review patches?

[Christopher Baines]

Vivien Kraus  writes:

> Dear QA wizards,
>
> Le vendredi 09 février 2024 à 10:44 +, Christopher Baines a écrit :
>> You just need to not be involved (so you can't review your
>> own patches)
>
> I interpret this as it’s OK to review patches if you asked for a change
> in the thread, am I correct? Or is this too much involvement?

That sounds fine. It's more you shouldn't review your own patches or
someone elses patches where you were significantly involved in the work.


signature.asc
Description: PGP signature

Re: QA is back, who wants to review patches?

[Christopher Baines]

Andreas Enge  writes:

> I see a few "Failed to process revision", for instance here:
>https://qa.guix.gnu.org/issue/68778
> While I am not sure why, these look like transient (?) build failures,
> at least failures not related to the patch in question. What is there to do?

Long term it would be nice for Guile to segfault less, in the short term
though sending the patch again will trigger the data service to try
again.


signature.asc
Description: PGP signature

Re: QA is back, who wants to review patches?

[Christopher Baines]

Tanguy LE CARROUR  writes:

> Hi Chris,
>
> First of, thanks (again) for everything that you’ve done with QA!
> It looks great!
>
>
> Quoting Christopher Baines (2024-02-09 11:44:11)
>> Let me know if you have any comments or questions!
>
> Unfortunately, I have some (stupid) questions!
>
> I decided to give it a try and I picked at random a patch that
> was supposed to be an easy one:
>
> ```
> [bug#68590] gnu: notmuch: update to version 0.38.2
> ```
>
> It’s mark as "green" *ie* important checks passing, but…
> it does not even apply?! Actually, it’s for a good reason:
> the exact same patch has been applied 2 weeks ago by
> Nicolas Goaziou as #9b65b60b97.
>
> The patch is still open on Debbugs. I guess it should be closed, right?
>
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=68590
>
> I guess it got is "green" status on QA before other patch made it to
> master.
>
> Can I safely close it?!

Yep, this unfortunately looks like a case where there was a duplication
of effort and the original patch got ignored.

It looks like the issue has been closed now.

QA can spot when patches don't apply, but it doesn't test for that
regularly at the moment.


signature.asc
Description: PGP signature

QA is back, who wants to review patches?

[Christopher Baines]

Hey!

After substitute availability taking a bit of a dive recently, the
bordeaux build farm has finally caught back up and QA is back submitting
builds for packages changed by patches.

QA also has a feature to allow easily tagging patches (issues) as having
been reviewed and ready to merge (reviewed-looks-good). You can do this
via sending an email and QA has a form ("Mark patches as reviewed") on
the page for each issue to help you do this.

I'd encourage anyone and everyone to review patches, there's no burden
on you to spot every problem and you don't need any special
knowledge. You just need to not be involved (so you can't review your
own patches) and take a good look at the changes, mentioning any
questions that you have or problems that you spot. If you think the
changes look good to be merged, you can tag the issue accordingly.

When issues are tagged as reviewed-looks-good, QA will display them in
dark green at the top of the list of patches, so it's on those with
commit access to prioritise looking at these issues and merging the
patches if indeed they are ready.

Let me know if you have any comments or questions!

Thanks,

Chris


signature.asc
Description: PGP signature

Re: Collecting Guix talks at FOSDEM

[Christopher Baines]

Steve George  writes:

> Hi,
>
> Did anyone take a photo of the discussion whiteboard?
>
> And/or anyone get a photo of the room generally that could be used in
> the blog post? Be nice to have a visual ...

I took a picture of the infrastructure discussion whiteboard:

https://www.cbaines.net/guix-days-2024-infrastructure-discussion-whiteboard.jpg


signature.asc
Description: PGP signature

Re: Meeting in Brussels on Wednesday night?

[Christopher Baines]

Ludovic Courtès  writes:

> To those traveling to Brussels tomorrow: who’s in to meet in our lair,
> namely Au Bon Vieux Temps, on Wednesday evening/night?  :-)
>
>   
> https://www.openstreetmap.org/?mlat=50.84832=4.35237#map=20/50.84832/4.35237=Y
>
> I should be able to be there around 9:30–10PM.

I'm in, I'll probably be there a little earlier as well, maybe around
20:00 to 20:30.


signature.asc
Description: PGP signature

Re: [Guix-europe-sac] Shutting down qa.guix?

[Christopher Baines]

Andreas Enge  writes:

> Am Sat, Dec 09, 2023 at 11:54:59AM +0100 schrieb Ludovic Courtès:
>> I think this underlines a collective failure to get our act together.
>
> indeed, and besides what Simon mentioned about the bank situation I think
> there was a certain lack of consistency between deciding on the technical
> and on the financial questions. And of course the lack of urgency, since
> anyway things continued thanks to Chris... So thank you for all you have
> done, Chris, and thank you for taking action now to force us to get our act
> together! Indeed QA has become a central part of our project infrastructure.
>
> I suggest the following, which resolves the lockstep between technology and
> finance:
> - Decide that the funds at the FSF pay for the hosting in its current form.
>   Ideally move the billing to Guix Foundation, and then, as we use to do
>   for bayfront, periodically ask the FSF to reimburse the hosting cost.
>   I think we have an informal consensus for this, and just require a formal
>   vote at the Guix spending committee and at the Guix Foundation SAC.
> - Reimburse Chris for the costs incurred for hosting before this move.
>   As Simon has said, we have a vote for this already, but could also
>   start over with the exact amount once the first point is handled, so
>   that Chris does not pay for future hosting any more.
>
> Then in a separate step, other people can discuss about potential
> technical changes to the hosting situation. It would probably be good
> to have a small group of people, including Chris at least for a
> transitory period, who take care of the sysadmin part.
>
> Any thoughts on this proposal?

Sounds good to me.


signature.asc
Description: PGP signature

December/January update on QA and related things

[Christopher Baines]

Hey,

I sent out the last update [1] back near the start of December.

1: https://lists.gnu.org/archive/html/guix-devel/2023-12/msg00021.html

In summary, QA has been not really working since mid December as
data.qa.guix.gnu.org wasn't keeping up with processing revisions for
patches and branches. I was also looking to shutdown
data.qa.guix.gnu.org but this didn't happen, there still seems to be
some interest in managing the infrastructure though the project (and
shutting it down also takes more time and effort than leaving it
running). Things are still running though, and I'm hopeful that QA will
be back processing patches in the next few weeks.

Specifically for the QA frontpage, little has changed in the last
month. I have got around to making a diagram to put QA in context and
it's now included in the README [2]. Let me know if any more diagrams
would be useful as I've now got a process.

2: https://qa.guix.gnu.org/README#orgd169767

For the Guix Data Service, I've put some time in to speeding up the
processing of revisions. I replaced all uses of delete-duplicates with a
sort and pair-fold alternative and parallelised computing the guix
derivations, computing the package derivations and a few other things
that happen through inferiors. This still needs a bit more testing, but
the changes are deployed on data.qa.guix.gnu.org and I think it's sped
up processing individual revisions at least.

Since data.qa.guix.gnu.org had less revisions in the database, I took
advantage of this to do some maintenance and managed to reduce the size
of the database considerably. Hopefully the frequent cleanup tasks will
prevent it from getting this large again.

Finally, I made various small fixes and speedups in the Guix Build
Coordinator. I hopefully mitigated the port encoding issue [3] by
switching from using the display procedure to log to using string->utf8
and put-bytevector. I opened a bug for a Guile segfault I hadn't seen
before [4]. I also hopefully reduced the impact of the build coordinator
stopping listening for connections by checking this internally and
exiting if there's an issue. Unfortunately I've been quite slow in
tracking down and trying to fix or at least mitigate these frequent but
hard to reproduce bugs, but I think I've made some progress recently.

3: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62590
4: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=68221

Looking forward, there's still the issue of me hosting various parts of
QA. Andreas has sent a proposal around on this in the last couple of
days though.

There's been some discussion in the past about using the data service to
monitor performance related things, but maybe this is important for just
keeping QA running as well. Failing to spot these issues before they're
introduced can cause significant disruption, so maybe we need the data
service to start monitoring and reporting on how particular performance
characteristics change between revisions so that this can be reported by
QA.

While the bordeaux build farm is still doing well I think, I still
haven't got around to implementing a way of pruning the nars that aren't
for the master branch from being stored indefinitely. I've got some
design ideas, but they need implementing and testing. There's also the
ongoing issue of build hardware for current and up and coming
architectures.

Let me know if you have any comments or questions! I'm also planning to
be at the Guix Days event and FOSDEM in a couple of weeks.

Thanks,

Chris


signature.asc
Description: PGP signature

Re: Performance of computing cross derivations

[Christopher Baines]

Ludovic Courtès  writes:

> Christopher Baines  skribis:
>
>> I think you're right, while I send some other changes in #68266, I think
>> it's this change around make-rust-sysroot that has pretty much all the
>> effects on performance.
>>
>> I think the tens of thousands of duplicated packages from cross-base
>> that I was looking at are almost entirely coming from
>> make-rust-sysroot. As Ludo mentions in [1], maybe this has something to
>> do with use of cross- procedures in native-inputs, although I'm not sure
>> that moving those calls out of native-inputs is a correct thing to do.
>>
>> I don't know what the correct approach here is, but I think something
>> needs doing here to address the performance regression.
>
> I probably missed it in the thread: what commit caused the regression,
> and how can I test any changes?  I’m willing to help but I missed some
> of the context.

It's not a pure performance regression, more that in it's current form,
rust cross derivations are very expensive to compute. It's been this way
since cross-compiling was enabled in [1].

1: 
https://git.savannah.gnu.org/cgit/guix.git/patch/?id=e604972d9c697302691aeb22e9c50c933a1a3c72

I've been looking at data service slowness in processing revisions over
the last few weeks, and I think it's mostly down to this. Looking at the
revision prior to the change [2], computing all the derivations took
around 3 hours, which is ages, but still quick compared to the nearly 9
hours it took after this change [3].

2: https://data.guix.gnu.org/revision/58bbb38c5bd2e42aab9e9408d8c9d8da3409f178
3: https://data.guix.gnu.org/revision/c9e1a72cc27925484635ae01bc4de28bf232689d

Obviously having more derivations is good and that usually means more
work for the data service, but in this case it seems like things can be
sped up quite a bit.

For testing locally, I've been computing all the derivations for
i586-pc-gnu, but Efraim also posted a concise command to look at
computing some cross derivations for a subset of rust packages [4].

4: https://lists.gnu.org/archive/html/guix-devel/2024-01/msg00053.html


signature.asc
Description: PGP signature

Re: Performance of computing cross derivations

[Christopher Baines]

Efraim Flashner  writes:

> [[PGP Signed Part:Signature made by expired key 41AAE7DCCA3D8351 Efraim 
> Flashner ]]
> On Fri, Jan 05, 2024 at 04:41:14PM +, Christopher Baines wrote:
>> 
>> Ludovic Courtès  writes:
>> 
>> > Hi,
>> >
>> > Christopher Baines  skribis:
>> >
>> >> When asked by the data service, it seems to take Guix around 3 minutes
>> >> to compute cross derivations for all packages (to a single
>> >> target). Here's a simple script that replicates this:
>> 
>> ...
>> 
>> > One idiom that defeats caching is:
>> >
>> >   (define (make-me-a-package x y z)
>> > (package
>> >   …))
>> >
>> > Such a procedure returns a fresh package every time it’s called,
>> > preventing caching from happening (because cache entries are compared
>> > with ‘eq?’).  That typically leads to lower hit rates.
>> >
>> > Anyway, lots of words to say that I don’t see anything immediately
>> > obvious with cross-compilation, yet I wouldn’t be surprised if some of
>> > these cache-defeating idioms were used because we’ve payed less
>> > attention to this.
>> 
>> I've got a feeling that performance has got worse since I looked at this
>> originally, I've finally got around to having a further look.
>> 
>> I spent some time looking at various metrics, but it was most useful to
>> just write the cache keys of various types to files and have a read.
>> 
>> The cross-base module was causing many issues, as all but one of the
>> procedures there produced new package records each time. There is also
>> make-rust-sysroot which showed up.
>> 
>> I've sent some patches as #68266 to add memoization to avoid this, and
>> that seems to speed things up.
>> 
>> Looking at other things in the cache, I think there are some issues with
>> file-append and local-file. The use of file-append in svn-fetch and
>> local-file in the lower procedure in the python build system both bloat
>> the cache for example, although I'm less sure about how to address these
>> cases.
>> 
>> One thing I am sure about though, is that these problems will come
>> back. Maybe we could add some reporting in to Guix to look through the
>> cache at the keys, lower them all and check for equivalence. That way it
>> should be possible to automate saying that having [1] in the cache
>> several thousand times is unhelpful. The data service could then run
>> this reporting and store it.
>> 
>> 1: #> gnu/packages/version-control.scm:2267 7f294d908840> "/bin/svn">
>
> I grabbed the patch for make-rust-sysroot to try it out:
> Native builds:
> time GUIX_PROFILING="object-cache" ./pre-inst-env guix build --no-grafts 
> $(./pre-inst-env ~/list-all-cargo-build-system-packages | grep rust- | head 
> -n 100) -d

...

> That's a massive drop in the size of the cache and a big decrease in the
> amount of time it took to calculate those 100 items.

I think you're right, while I send some other changes in #68266, I think
it's this change around make-rust-sysroot that has pretty much all the
effects on performance.

I think the tens of thousands of duplicated packages from cross-base
that I was looking at are almost entirely coming from
make-rust-sysroot. As Ludo mentions in [1], maybe this has something to
do with use of cross- procedures in native-inputs, although I'm not sure
that moving those calls out of native-inputs is a correct thing to do.

I don't know what the correct approach here is, but I think something
needs doing here to address the performance regression.

1: https://lists.gnu.org/archive/html/guix-patches/2024-01/msg00733.html


signature.asc
Description: PGP signature

Re: Performance of computing cross derivations

[Christopher Baines]

Ludovic Courtès  writes:

> Hi,
>
> Christopher Baines  skribis:
>
>> When asked by the data service, it seems to take Guix around 3 minutes
>> to compute cross derivations for all packages (to a single
>> target). Here's a simple script that replicates this:

...

> One idiom that defeats caching is:
>
>   (define (make-me-a-package x y z)
> (package
>   …))
>
> Such a procedure returns a fresh package every time it’s called,
> preventing caching from happening (because cache entries are compared
> with ‘eq?’).  That typically leads to lower hit rates.
>
> Anyway, lots of words to say that I don’t see anything immediately
> obvious with cross-compilation, yet I wouldn’t be surprised if some of
> these cache-defeating idioms were used because we’ve payed less
> attention to this.

I've got a feeling that performance has got worse since I looked at this
originally, I've finally got around to having a further look.

I spent some time looking at various metrics, but it was most useful to
just write the cache keys of various types to files and have a read.

The cross-base module was causing many issues, as all but one of the
procedures there produced new package records each time. There is also
make-rust-sysroot which showed up.

I've sent some patches as #68266 to add memoization to avoid this, and
that seems to speed things up.

Looking at other things in the cache, I think there are some issues with
file-append and local-file. The use of file-append in svn-fetch and
local-file in the lower procedure in the python build system both bloat
the cache for example, although I'm less sure about how to address these
cases.

One thing I am sure about though, is that these problems will come
back. Maybe we could add some reporting in to Guix to look through the
cache at the keys, lower them all and check for equivalence. That way it
should be possible to automate saying that having [1] in the cache
several thousand times is unhelpful. The data service could then run
this reporting and store it.

1: # "/bin/svn">


signature.asc
Description: PGP signature

Re: Shutting down qa.guix?

[Christopher Baines]

Maxim Cournoyer  writes:

> Hi,
>
> Christopher Baines  writes:
>
>> Tobias Geerinckx-Rice  writes:
>>
>>> Christopher Baines 写道：
>>>> it's not the most cost effective setup
>>>
>>> Has this been explained in more detail before?
>>
>> Probably not, beid is currently a CPX51 Hetzner cloud server costing
>> €65.33 a month. This has been useful as it's enabled scaling the
>> resources dynamically, but it would be possible to reduce the costs and
>> still have sufficient RAM/disk space by using a Hetzner server auction
>> machine for example.
>>
>> It's not all about cost though, given the data service is one of the
>> slow points of QA, if we want QA to get faster at giving feedback, it's
>> probably important to not try and cut costs on this part of the system.
>
> Isn't QA mostly slow because of the lack of x86 build machines?  Does
> the head node needs to be powerful itself?  What kind of resources does
> it likes having the most?  CPU?  RAM?  Storage?

There are two key bottlenecks, processing the revisions in the data
service, then the build coordinator performing the builds.

For the data service, lots of RAM helps as computing and building the
derivations for Guix (similar to pull, time-machine, ...) is quite
expensive in CPU and RAM. Also computing all the derivations for each
revision takes a lot of RAM.

Storage is also an issue as beid currently is working with 340G of total
storage and that's almost full, and this doesn't leave any space for
maintenance. More storage means being able to store data about more
patch series at once.

For the build coordinator, the machine doesn't need to be powerful, it
has quite low requirements. While bayfront's storage isn't particularly
fast, it's more than sufficient in terms of hardware. More build
machines, including x86 ones would speed up the test results for patches
and branches though.


signature.asc
Description: PGP signature

Re: Shutting down qa.guix?

[Christopher Baines]

Tobias Geerinckx-Rice  writes:

> Christopher Baines 写道：
>> it's not the most cost effective setup
>
> Has this been explained in more detail before?

Probably not, beid is currently a CPX51 Hetzner cloud server costing
€65.33 a month. This has been useful as it's enabled scaling the
resources dynamically, but it would be possible to reduce the costs and
still have sufficient RAM/disk space by using a Hetzner server auction
machine for example.

It's not all about cost though, given the data service is one of the
slow points of QA, if we want QA to get faster at giving feedback, it's
probably important to not try and cut costs on this part of the system.


signature.asc
Description: PGP signature

Re: Shutting down qa.guix?

[Christopher Baines]

Ludovic Courtès  writes:

> Hello,
>
> Christopher Baines  skribis:
>
>> I am still planning to shutdown data.qa.guix.gnu.org and
>> QA which depends on it within the next couple of weeks. I do hope it can
>> return some point though, and hopefully sooner rather than later.
>>
>> On this like most decisions I'm indecisive, I could try and keep the
>> current server going, but it's not the most cost effective setup and
>> it's very low on disk space. I could replace the server with some
>> slightly better setup, but this would still mean I'm managing a key part
>> of the infrastructure, which is something I'm trying to move away
>> from. There was some discussion of the project taking over the hosting,
>> and maybe that will happen at some point, but it hasn't happened yet. So
>> while not having qa.guix.gnu.org for a time isn't ideal, I'm still going
>> with this approach.
>
> I think this underlines a collective failure to get our act together.
>
> I believe there’s consensus that qa.guix is useful and has been a boost
> for reviewers and contributors; we’d probably all want it to provide
> quicker feedback, which is a sign of success: we’ve come to rely on it.
>
> I know this has been discussed several times and it remains unclear to
> me why as a project we never managed to move forward—maybe the comfort
> of the status quo?

In addition, it's also unclear to me who should be making decisions on
things like this.

I also think that fundamentally I may think that processes and tooling
to make changes is more important than others regard it to be. While it
has no inherent value to users, personally I see it as so much more
important than actual Guix features or packages since the value to users
comes through Guix getting better faster, because of the increased pace
of changes and reduced number of regressions.

> Anyway, would it be possible for you to transfer billing of the hardware
> (Hetzner?) to Guix Foundation?  Does Guix Foundation know what it would
> cost them?

I believe so, at least I think that's possible. The costs have also been
discussed previously.

> The “spending committee” (Tobias, Ricardo, and myself), which oversees
> expenditure from the funds held at the FSF, can also be in the loop to
> provide additional financial support.
>
> As for system administration, is there documentation that people willing
> to help could look at?  Very concrete things like: what services are
> running on which machines, what do I do if one of them is stuck or if I
> get this error message, etc.

The configuration for beid, the machine that runs data.qa.guix.gnu.org
and Patchwork is in maintenance.git. It could probably use some more
comments to provide some context for the configuration.

There's also probably a benefit from making some high level architecture
diagrams for QA and the bordeaux build farm, and I can try and make a
start on these.

As for monitoring and responding to problems, that's a bit more
complicated, but in most cases a herd restart of the relevant bit will
temporarily resolve the issue. I'm still working on mitigating some of
the underlying problems that cause things to break.


signature.asc
Description: PGP signature

Re: Discontinuing data.guix.gnu.org?

[Christopher Baines]

Ludovic Courtès  writes:

> Hello!
>
> Christopher Baines  skribis:
>
>> As previously set out, I'm planning to stop hosting the data service
>> instances this year. While I would like to stop hosting the server for
>> data.guix.gnu.org,
>
> I forgot the outcome of previous discussions, but it seems to me that
> the service itself and all the data it accumulated over the years are
> super valuable.  I would be sad to see it go!

There was a discussion back in April, but no action came directly from
it.

Just having data.qa.guix.gnu.org was discussed, and at least
concentrating on getting to a sustainable hosting situation there seemed
like a sensible priority. The longer history provided by
data.guix.gnu.org does have value though in my view.

> Is there something we can do to not lose it all?  It could be
> distributing responsibility, reducing the scope, ensuring hosting is
> managed collectively by the project, etc.  WDYT?

Since that discussion, I have disabled the database dumps and backups,
which has reduced (to 62€ per month) the hosting costs (although
obviously not having backups isn't ideal). It's possible to further
reduce the hosting costs as well by switching away from a VM to a
physical machine at Hetzner.

But yeah, given that having at least one data service instance is a key
part of keeping the bordeaux build farm running, managing the hosting
through the project seems to be the way to go. I'm just not sure how we
can get there, or what I can do to move things in that direction.


signature.asc
Description: PGP signature

November/December update on qa.guix.gnu.org and related things

[Christopher Baines]

Hey!

Not much has changed since the last update. There's a new "waiting for
builds" status, I tweaked some code around build cancelation, I put in
place a mitigation for #67194 affecting the build coordinator, and did
some investigation of the hurd locale issue (#67507).

As previously set out, I'm planning to stop hosting the data service
instances this year. While I would like to stop hosting the server for
data.guix.gnu.org, I am going to keep that going at least for the time
being as that will allow me to test out pruning the bordeaux nars among
other things. I am still planning to shutdown data.qa.guix.gnu.org and
QA which depends on it within the next couple of weeks. I do hope it can
return some point though, and hopefully sooner rather than later.

On this like most decisions I'm indecisive, I could try and keep the
current server going, but it's not the most cost effective setup and
it's very low on disk space. I could replace the server with some
slightly better setup, but this would still mean I'm managing a key part
of the infrastructure, which is something I'm trying to move away
from. There was some discussion of the project taking over the hosting,
and maybe that will happen at some point, but it hasn't happened yet. So
while not having qa.guix.gnu.org for a time isn't ideal, I'm still going
with this approach.

Given that the first prototype setup came (and went I think) sometime
back in 2019 [1], this stuff has been going on for a while now. From
that prototyping over the last few years, I think the current software
setup is something that can work long term and be iterated on. To take
this technical setup forward though, it would ideally happen more
through the project and no longer have key parts be reliant on me for
hosting.

1: https://lists.gnu.org/archive/html/guix-devel/2019-02/msg00089.html

Thanks,

Chris


signature.asc
Description: PGP signature

RISC-V builds and substitutes

[Christopher Baines]

Hey,

I have a RISC-V board that is hooked up to the bordeaux build farm, and
there is another board connected, although there's some issues with the
connection/agent.

1 to 2 boards though isn't enough to get good substitute availability
and build packages for patches and branches though.

There's now HiFive Unmatched Rev B boards available [1], so is there
interest in buying additional boards for the project so that we can
improve substitute availability and get QA doing RISC-V things?

1: 
https://www.sifive.com/blog/unlock-the-possibilities-with-hifive-unmatched-risc-v

Chris


signature.asc
Description: PGP signature

Re: Failed to build in QA

[Christopher Baines]

Reza Housseini  writes:

> Hi Christopher
>
> I submitted a new revision to the issue, but the QA link shows
>
> Issue not found
> This could mean the issue does not exist, it has no patches or has
> been closed.
>
> do you know what the problem is here?

There's two issues, one is the machine running Patchwork is low on disk
space, and that keeps stopping new messages being processed. I've
resolved that for now.

The other issue with the v4 series is that Patchwork has got confused
and only picked out the first of the v4 patches. The threading also
looks weird to me in my email client, but I'm not quite sure why. How
did you send the v4 patches?

Thanks,

Chris


signature.asc
Description: PGP signature

Re: mesa-updates: call for patches

[Christopher Baines]

John Kehayias  writes:

> Hi everyone,
>
> Update below:
>
> On Sun, Nov 05, 2023 at 11:47 PM, John Kehayias wrote:
> [snippy snip snip]
>>>
>>> Happy to! Substitutes will eventually become available, but there's
>>> quite a few builds to be done. This takes care of some ungrafts and
>>> updates with I hope minimal disruption. I'll be keeping an eye out and
>>> using locally as well. Please test and report, thanks everyone!
>>>
>>> John
>>
>> An issue was created to track merging the mesa-updates branch here:
>> . Please use that bug number as
>> needed (and cc me or use wide-reply in emacs debbugs).
>
> At this point I feel we are just about ready to go, unless there are
> objections?
>
> Substitute coverage, according to
>  is good on x86_64 and
> i686 (about 95% and 83%, respectively) while, as usual, other
> architectures are behind. The next best is aarch64 at 54% on bordeaux,
> and then falling to 24% for armhf, with others we build in the teens.
> I think this is to be expected? In any event builds continue very
> slowly and in the past I think this is about where we merge.

I think some changes have been pushed since this email, since the
aarch64 substitute availability has dropped from 54% to 25%.

> So, shall I merge this to master in the next couple of days? I've been
> merging master into mesa-updates smoothly so far. Please do check and
> feel free to object if this needs more time.

I guess this has been held up by the changes on the 15th, but still, I
think we need to wait for substitute availability to improve more prior
to merging, unless there's a specific and significant reason why we
don't want to wait.

Targets are arbitrary, but guix weather defines ☀ as 80%+, so I think
that's what we should aim for at least for x86_64-linux, i686-linux,
aarch64-linux and armhf-linux. This isn't just about substitute
availability though as this is key for discovering what things fail to
build.

Obviously delays in merging aren't ideal, but we should tackle the
problems around this, maybe by deciding that testing and providing
substitutes for ARM isn't a priority and thus isn't something we should
wait for, or look at getting more ARM hardware to speed up the process
(we also have a lack of x86_64 hardware on the bordeaux build farm).


signature.asc
Description: PGP signature

Re: Patch Review Flow

[Christopher Baines]

"jgart"  writes:

> Does anyone follow this workflow for reviewing patches?
>
> git clone https://git.guix-patches.cbaines.net/guix-patches/
> git checkout issue-x
> git format-patch ...
> # then in the development checkout of Guix:
> git am ...; make; ./pre-inst-env guix build
>
> Should we document it in the manual?
>
> Does anyone follow a workflow that is more efficient than the above?

I have the guix-patches repository as a remote, which I then cherry pick
from (via Magit), which makes it quick and easy to apply patches.


signature.asc
Description: PGP signature

Re: Reproducible Builds 2023 in Hamburg

[Christopher Baines]

Christopher Baines  writes:

> Hey!
>
> This event is now finished! Efraim and Julien and I had a discussion
> about Guix things relating to Reproducible Builds and there are some
> notes available here [1].
>
> 1: https://reproducible-builds.org/events/hamburg2023/guix-todo/
>
> Maybe the others can mention what things they've been looking at, but
> I've been reminded about my previous investigation in to Reproducible
> builds and the ACL [2]. I also started working on a QA topic page [3] to
> pull together data and issues.
>
> 2: https://lists.gnu.org/archive/html/guix-devel/2020-06/msg00179.html
> 3: https://qa.guix.gnu.org/reproducible-builds

Oh, and I forgot to say with the help of Bernhard I added Guix support
to [3].

3: https://ismypackagereproducibleyet.org/

It's probably going to be useful to compare reproducibility status
across distros, and this is a start.


signature.asc
Description: PGP signature

Reproducible Builds 2023 in Hamburg

[Christopher Baines]

Hey!

This event is now finished! Efraim and Julien and I had a discussion
about Guix things relating to Reproducible Builds and there are some
notes available here [1].

1: https://reproducible-builds.org/events/hamburg2023/guix-todo/

Maybe the others can mention what things they've been looking at, but
I've been reminded about my previous investigation in to Reproducible
builds and the ACL [2]. I also started working on a QA topic page [3] to
pull together data and issues.

2: https://lists.gnu.org/archive/html/guix-devel/2020-06/msg00179.html
3: https://qa.guix.gnu.org/reproducible-builds

Chris


signature.asc
Description: PGP signature

Re: August/November update on qa.guix.gnu.org and related things

[Christopher Baines]

Suhail  writes:

> "Christopher Baines"  writes:
>
>> There isn't much documentation for QA
>
> Understood.
>
> Is the preferred place to ask questions regd the QA service this mailing
> list?

Yep.

>> I think it's fair to say that these shouldn't be styled the same as
>> failed builds, so I've changed the styling now.
>
> The neutral blue works better; thank you. On a related note, the
> specific build status on data.qa.guix.gnu.org for the "now blue" entries
> is "Scheduled". Why does that get presented as "Unknown" in QA? IMO,
> either "Scheduled" or "Pending" (in case it's important to maintain a
> distinction from the build status of individual jobs as on
> data.qa.guix.gnu.org) would be clearer than "Unknown".

I guess the result of the builds is unknown when they're scheduled.

>> I've also added a new issue status for when QA is waiting on builds to
>> happen to provide more information.
>
> This being "Investigate"? Out of curiosity, and in a similar vein as
> above, why not simply "Scheduled" or "Pending"? Or is it that it has had
> "Scheduled" build jobs for far too long and thus requires someone else
> with more privileged access (than myself) to investigate the cause?
>
> I.e., Investigate is a verb and thus makes me wonder what the object is
> (what needs to be investigated) and who the subject is (by whom)?
> Shouldn't the QA issue status be an adjective instead?

Yeah, this is just me copying and pasting the code for the badge. I've
changed the text to pending now. That's not as specific as "QA is
waiting on the results of builds" but at least it hints at that.

>> There's also some content in the manual that might be useful when
>> reviewing patches:
>>
>>   https://guix.gnu.org/en/manual/devel/en/html_node/Packaging-Guidelines.html
>>   https://guix.gnu.org/en/manual/devel/en/html_node/Submitting-Patches.html
>
> Perhaps linking to these from the "Mark patches as reviewed" section on
> QA would be helpful?

Yes, although I think the documentation maybe needs looking at a bit to
make it a bit more useful for patch review.

>> But there's no pre-requisites to reviewing Guix patches, so the best
>> way to learn is to start looking to review things.
>
> I imagine some of the "common things to check" will get automated in the
> near future (e.g. whether or not the changes are adding to the lint
> warnings), whereas some others will stay manual (e.g. are things "well
> written"). Personally, for such subjective measures (i.e., the latter) I
> find having some examples of "what good looks like" readily available
> quite helpful. In case the intent is to make it easier for newcomers to
> the project (i.e., those who've not yet internalized this knowledge) to
> contribute, providing such prototypical examples by linking to commits,
> descriptions etc in the existing source tree would help.

Yes, and while I think working on the docs is important, maybe QA can
link and show various examples.


signature.asc
Description: PGP signature

Re: August/November update on qa.guix.gnu.org and related things

[Christopher Baines]

Suhail  writes:

> Christopher Baines  writes:
>
>>  - The README is published at https://qa.guix.gnu.org/README
>
> The README seems more focused on task planning and TODO items than
> explaining how to use the qa.guix.gnu.org website. Could you please
> provide a reference for the latter?

There isn't much documentation for QA, mostly because I want to see it
improve so that you don't need documentation to use it.

> Specifically, I submitted a patch some while ago:
> <https://qa.guix.gnu.org/issue/66644>. Its QA status is marked as
> unknown with a few items highlighted in red. While the UI helps draw
> attention to those items, it's not clear (to me) how to remedy them and
> who is responsible for doing what. I.e., what are the next steps? I
> would like to get that patch reviewed and merged in some way, but I
> don't know what, if anything, I can do to help with the matter.

I had a look at the QA page for #66644 and yeah, the red highlighted
bits where builds which hadn't happened yet.

I think it's fair to say that these shouldn't be styled the same as
failed builds, so I've changed the styling now. I've also added a new
issue status for when QA is waiting on builds to happen to provide more
information.

So yeah, QA isn't currently pointing out anything for you to do on this
issue.

>> I'd also really like to see some testing of the patch review feature
>> in QA, since I think trying to get people without commit access
>> reviewing patches will really help speed up getting things reviewed
>> and merged.
>
> Is there a document that outlines how to get started and/or
> pre-requisites that one must have before reviewing certain aspects?

On the page for each issue on qa.guix.gnu.org, there's a list of common
things to check. That form gives a way to record a review when you think
the patches look good to merge.

There's also some content in the manual that might be useful when
reviewing patches:

  https://guix.gnu.org/en/manual/devel/en/html_node/Packaging-Guidelines.html
  https://guix.gnu.org/en/manual/devel/en/html_node/Submitting-Patches.html

But there's no pre-requisites to reviewing Guix patches, so the best way
to learn is to start looking to review things.


signature.asc
Description: PGP signature

August/November update on qa.guix.gnu.org and related things

[Christopher Baines]

Hey,

The last update I sent out was back in July [1].

1: https://lists.gnu.org/archive/html/guix-devel/2023-07/msg00144.html

There have been quite a few changes over the last few months, here's a
summary of the changes:

 - QA now stores when it's submitted builds for a issue/branch and
   cancels them when they're no longer needed

 - The README is published at https://qa.guix.gnu.org/README

 - The output when applying patches is stored and displayed if there was
   a failure

 - There's a review feature for marking patches as looking good

 - There are more issue statuses and support filtering by status

 - Merged issues are now handled

 - Patches can be applied to non-master branch

 - The latest patch series are processed, rather than the latest issues

 - There's a page specifically about reproducible builds
   (qa.guix.gnu.org/reproducible-builds)


I've also been doing little bit of work towards speeding up the data
service processing revisions. I've disabled computing the system test
derivations on data.qa.guix.gnu.org as that takes a significant amount
of time, and they aren't being used yet.

The formatting linter takes quite a bit of time, and I've got an open
patch to speed it up:

  https://issues.guix.gnu.org/66796

The performance of computing cross derivations also seems like it could
be improved, I've sent some details here:

  https://lists.gnu.org/archive/html/guix-devel/2023-10/msg00257.html

# Next steps

I'm still planning to stop hosting both data.guix.gnu.org and
data.qa.guix.gnu.org at the end of the year, which is only a few weeks
away now. The qa-frontpage which runs qa.guix.gnu.org is dependent on
data.qa.guix.gnu.org. There's been some discussion of potentially moving
some or all of this to hosting organised through the project/Guix
Europe, but I'm not sure anything has happened on this yet.

I've added a few TODO's to the list in the README off the back of
discussion at the Reproducible Builds summit.

In my mind, while there's still lots to do in the qa-frontpage,
addressing problems in the data service is probably still the most
important thing to do. There's still some reliability issues to
investigate further as well as improving the performance of processing
revisions. If the data service can be sped up, QA will be able to give
feedback faster, and it'll scale to handle more patches.

I'd also really like to see some testing of the patch review feature in
QA, since I think trying to get people without commit access reviewing
patches will really help speed up getting things reviewed and merged.

Let me know if you have any comments or questions!

Thanks,

Chris


signature.asc
Description: PGP signature

Performance of computing cross derivations

[Christopher Baines]

Hey!

When asked by the data service, it seems to take Guix around 3 minutes
to compute cross derivations for all packages (to a single
target). Here's a simple script that replicates this:

  (use-modules (srfi srfi-34)
   (gnu packages)
   (guix grafts)
   (guix packages)
   (guix store)
   (statprof))
  
  (define (all-cross system target)
(with-store store
  (%graft? #f)
  (fold-packages
   (lambda (package result)
 (with-exception-handler
 (lambda (exn)
   (unless (package-cross-build-system-error? exn)
 (peek exn))
   result)
   (lambda ()
 (package-cross-derivation store
   package
   target
   system)
 (+ 1 result))
   #:unwind? #t))
   0)))
  
  (statprof
   (lambda ()
 (peek "COUNT"
   (all-cross "x86_64-linux"
  "i586-pc-gnu")))
   #:count-calls? #t)


Here's some relevant output:

  % cumulative   self 
  time   secondsseconds   calls   procedure
   50.48126.68102.40
ice-9/vlist.scm:502:0:vhash-foldq*
   11.49 23.31 23.31hashq
5.16 10.52 10.47write
2.79 14.28  5.65
ice-9/vlist.scm:494:0:vhash-fold*
2.28  4.63  4.63equal?
2.14  4.35  4.35hash
1.85  4.67  3.75
guix/packages.scm:1874:0:input=?
1.78  3.68  3.61put-string
1.77  7.16  3.59
guix/derivations.scm:736:0:derivation/masked-inputs
0.93  1.90  1.90get-bytevector-n
0.78  1.58  1.58put-char
0.67  1.36  1.36search-path

  ...
  
  Total time: 202.872232073 seconds (30.927648399 seconds in GC)


Over 3 minutes seems like a long time for this, especially since it only
computes around 1 derivations.

I don't know how to use statprof, but looking at vhash-foldq* being at
the top of the output, is this suggesting that around a third of the CPU
time is being spent looking for things in various caches?

I had a go at using the Guix profiling stuff and I did get some output,
but I couldn't figure out how to get it to show all the caching going
on.

Any ideas?

Chris


signature.asc
Description: PGP signature

Re: branch master updated: gnu: Add passff.

[Christopher Baines]

guix-comm...@gnu.org writes:

> This is an automated email from the git hooks/post-receive script.
>
> snape pushed a commit to branch master
> in repository guix.
>
> The following commit(s) were added to refs/heads/master by this push:
>  new 6c894b7a1a gnu: Add passff.
> 6c894b7a1a is described below
>
> commit 6c894b7a1a6a7a0f7c51a44136bba00dc0b5250c
> Author: Clément Lassieur 
> AuthorDate: Tue Oct 17 16:53:57 2023 +0200
>
> gnu: Add passff.
>
> * gnu/packages/browser-extensions.scm (passff-host): New variable.
> (passff): New variable.
>
> Change-Id: I0f6f4b0c319e5cffd0940421a4d8bdc73d8d806b
> ---
>  gnu/packages/browser-extensions.scm | 76 
> +
>  1 file changed, 76 insertions(+)

This passff-host package looks a bit odd to me, one thing to mention is
that guix show says it has no dependencies, but I don't think that's
correct:

  ./pre-inst-env guix show passff-host
  name: passff-host
  version: 1.2.3
  outputs:
  + out: everything
  systems: x86_64-linux mips64el-linux aarch64-linux powerpc64le-linux 
riscv64-linux
  + i686-linux armhf-linux i586-gnu powerpc-linux
  dependencies: 

Was this change sent as a patch to guix-patches?

Thanks,

Chris


signature.asc
Description: PGP signature

Re: [PATCH qa-frontpage] Apply incoming patches onto the correct feature branch

[Christopher Baines]

Christopher Baines  writes:

> Vivien Kraus  writes:
>
>> It seems that the only available information of the feature branch is
>> in the patch name.
>> ---
>>
>> Hello guix!
>>
>> Thank you for making the QA tool. It seems to me that feature branches
>> are ignored for patches. The patches seem to always be applied on top
>> of master. I get that idea because the base-for-issue-* tag is always
>> put on HEAD, and all my patches targetting gnome-team recently get
>> applied to master. I understand that the latter could be a problem
>> with me.
>>
>> If patches are indeed applied on top of master, then the question
>> arises, what to do. The available information from patchwork is
>> scarce: the base-commit is not available, and the optional feature
>> branch only appears in the name of the patches of the series.
>>
>> I think it is possible to parse the patch names. This way, we get some
>> useful information: the feature branch, the series revision, and the
>> patch index. The patch index can be used to make sure the patches are
>> in correct order (this can be the case if a server in the path
>> re-orders the emails).
>>
>> What do you think?
>
> Hi Vivien,
>
> Thanks for trying to implement this and sending a patch, I think it's a
> good feature to add but there's a few things needed to make this work.
>
> Firstly, you need to actually change which branch the patch is applied
> to, and the place to do that is here [1] in the call to
> with-git-worktree. You can probably use get-commit from
> (guix-qa-frontpage git-repository) to find the commit hash for a
> branch. You'll also probably need to move around some of the code in
> create-branch-for-issue so that you have the data about the patches to
> work out what the desired branch is before you call with-git-worktree.
>
> 1: 
> https://git.cbaines.net/guix/qa-frontpage/tree/guix-qa-frontpage/manage-patch-branches.scm#n250
>
> The second thing is that there's other places in the codebase that
> assume the patches have been applied to the master branch. In
> start-manage-patch-branches-thread, there's
> get-changes-compared-to-master which is used to decide if a branch needs
> recreating if there's too many changes between the base revision and a
> recent master branch revision. That'll need changing to at least skip
> over any patches been applied to a non-master branch, or perform the
> comparison against the tip of the relevant branch.
>
> The third thing that will need to change to allow this to happen is
> adding clear messaging on to the issue page to indicate where patches
> have been applied to non-master branches. That'll help to avoid people
> merging these patches in to the master branch rather than the branch
> they were intended for.
>
> Maybe what would be helpful is a procedure in (guix-qa-frontpage issue)
> that takes the data for the issue from latest-patchwork-series-by-issue,
> and gives you back the branch name that the patches should be applied
> to. That can then be used to get the branch information for all the
> above 3 use cases.
>
> As for trying to order the patches, do you know of a case where the
> ordering from Patchwork is wrong? I think it's only worth changing the
> ordering in the qa-frontpage if we know we're doing it for a reason.
>
> There's now a create-issue-branch command to the guix-qa-frontpage
> script as well, which will be a good way of testing the code out for
> applying patches to various branches. It'll fail when it comes to
> pushing the branch, but it'll still be useful for testing the code up to
> that point.
>
> Would you be able to look at sending some revised patches?

I've now gone ahead and applied a slimmed down version of this patch,
and made the other changes to get a basic form of applying patches to
specific branches in place. There's probably going to be some problems
and more error handling to add, but this should be a good starting
point.

Thanks,

Chris


signature.asc
Description: PGP signature

Re: How to send revision patches to old issues in DEBBUGS so Guix QA can look at them?

[Christopher Baines]

Andy Tai  writes:

> Hi, per Chris's recent comments, Guix QA only looks at (a window of)
> recently created issues due to storage space limitations, which is
> understandable.
>
> However, if I have update (package definition) patches to not very
> recent issues, such as some stale issues sitting there for some time,
> Guix QA does not pick these revisions up.  Then these revisions will
> not be reviewed and would not be merged.
>
> Ideally, Guix QA should look at recent patch submissions to the
> guix-patches mailing list but not sure if that is easy to implement.
>
> Then what is the best way to make recent patches to old issues covered
> by Guix QA?  I tried to create new issues and then merge old issues
> into the new ones but that is probably not a good way to go or good
> thing to do.

As part of changing QA to use the Patchwork series endpoint (rather than
the patches endpoint), I've now changed it to select the most recent N
(currently 200) series.

This should mean that QA will pick up patches sent to an old issue,
which makes it possible to just resend the patches to "bump" the issue.

I think that's a good option to have, but obviously the best approach
here would be to not have patches waiting around to get old and to be
merging patches faster. My current view is that this is possible if we
can get more people involved in patch review, and I've been working on
features in QA (like the "Mark patches as reviewed" form) to help with
that.


signature.asc
Description: PGP signature

Re: [PATCH] web: Include merged_with in graphql .

[Christopher Baines]

Arun Isaac  writes:

>> Sorry, I missed this patch. I will apply it tonight.
>
> Done!
>
> https://git.savannah.gnu.org/cgit/guix/mumi.git/commit/?id=a89c5109963de0b5fd1f500ed2e2ede47728fdb4
> https://git.savannah.gnu.org/cgit/guix/mumi.git/commit/?id=025fc600f1cb4c73042bf920aee3e07d5fb9c53a
>
> I have also reconfigured berlin with the latest mumi.

THanks Arun, QA is now using this data to group merged issues which is
very useful.

Chris


signature.asc
Description: PGP signature

Re: Guix Data Service can now poll for new revisions/branches

[Christopher Baines]

Maxim Cournoyer  writes:

> Hi Christopher,
>
> Christopher Baines  writes:
>
>> Hey,
>>
>> As has happened before in the past, at the moment we're not getting
>> emails to guix-commits for the main guix.git repository. This impacts
>> the data service instances as they use these emails to know when there
>> are new revisions to process.
>>
>> I've now added the ability to configure the data service to poll the
>> repository for changes and I've enabled this on both data.guix.gnu.org
>> and data.qa.guix.gnu.org, which should mean they'll be shortly back up
>> to date.
>>
>> This sounds easy, but it was a little tricky to implement, as I didn't
>> want to remove the email feature in favour of polling, at least not
>> yet. It's also had a large impact on data.qa.guix.gnu.org, as now it's
>> learned about lots of older branches that it didn't know about before.
>
> Ludovic recently implemented a mechanism to notify Cuirass of new
> commits via the Savannah server-side git hook; perhaps you could use
> that as well?  It should be cheaper than polling on the git repo.

Yeah, there's also this UDP thing that Giovanni mentioned:

  https://lists.gnu.org/archive/html/guix-devel/2023-09/msg00153.html

Both approaches should be easier to implement now that polling is
implemented.


signature.asc
Description: PGP signature

Re: [PATCH] web: Include merged_with in graphql .

[Christopher Baines]

Simon Tournier  writes:

> Hi,
>
> Out of my curiosity, is this patch
>
> On Tue, 26 Sep 2023 at 09:59, Christopher Baines  wrote:
>> * mumi/web/graphql.scm (): Include merged_with.
>> ---
>>  mumi/web/graphql.scm | 12 +++-
>>  1 file changed, 11 insertions(+), 1 deletion(-)
>
> merged to Mumi?

Not that I'm aware of, do you see it merged somewhere?

> Is it still relevant?

I think so.


signature.asc
Description: PGP signature

Re: Failed to build in QA

[Christopher Baines]

Reza Housseini  writes:

>> This is probably down to a top level circular dependency. In particular,
>> trying to paraview to compute the version to form part of the
>> native-search-path at the top level causes problems.
>
> I'm wondering why it builds fine locally but causes problems in QA
> have you any pointers what might be the difference?

You should be able to reproduce this locally, I see the issue when
running make.


signature.asc
Description: PGP signature

Re: Guix QA "Unable to apply patches"

[Christopher Baines]

Andy Tai  writes:

> Hi, for one of my package patches, Guix QA shows
>
> "Unable to apply patches"
>
> in https://qa.guix.gnu.org/issue/63095
>
> even though I double checked the patch applies cleanly on the current
> head of the master branch.
>
> Guix QA shows no hint as what the failure is (no log, etc.)   Curious
> how to investigate the problem?

That message was appearing incorrectly, which I've now fixed. The
qa-frontpage hadn't attempted to apply the patch, which is why the log
showed #f.

The reason why QA hasn't looked at this issue is that it's too old. To
avoid resource problems, QA only currently looks at a fixed number of
recent issues. The limiting factor is currently data service storage
space.

Unfortunately none of that is very helpful in getting this patch looked
at though.


signature.asc
Description: PGP signature

Guix Data Service can now poll for new revisions/branches

[Christopher Baines]

Hey,

As has happened before in the past, at the moment we're not getting
emails to guix-commits for the main guix.git repository. This impacts
the data service instances as they use these emails to know when there
are new revisions to process.

I've now added the ability to configure the data service to poll the
repository for changes and I've enabled this on both data.guix.gnu.org
and data.qa.guix.gnu.org, which should mean they'll be shortly back up
to date.

This sounds easy, but it was a little tricky to implement, as I didn't
want to remove the email feature in favour of polling, at least not
yet. It's also had a large impact on data.qa.guix.gnu.org, as now it's
learned about lots of older branches that it didn't know about before.

Chris


signature.asc
Description: PGP signature

Re: Failed to build in QA

[Christopher Baines]

"reza.housse...@gmail.com"  writes:

> On October 5, 2023 10:49:06 AM GMT+02:00, Christopher Baines 
>  wrote:
>>
>>"reza.housse...@gmail.com"  writes:
>>
>>> I submitted an issue to guix. But QA refuses to build it [1]. I have
>>> no clue what the problem is, can anyone shed light on a possible
>>> resolution?
>>
>>You pretty much found the problem, the relevant line on the page you
>>linked to is:
>>
>>[  6/ 50] loading...   24.0% of 25 filesbuilder for 
>>`/gnu/store/qhvpjfn3d9cwz5zxadblbnbqa92a8i27-guix-cli-core.drv' failed due to 
>>signal 11 (Segmentation fault)
>>
>>So the data service wasn't able to build Guix. This probably isn't due
>>to your changes, and it doesn't happen very often, so the thing to do
>>here is just retry.
>>
>>I've triggered QA to reapply the patches now (by deleting the
>>issue-66262 branch), so hopefully things will work better this time.
>>
>>Thanks,
>>
>>Chris
>
> Thanks very much, it seems to have worked, but now it's stuck with
> paraview undefined symbol, although the necessary module
> (gnu/packages/image-processing) is imported?

This is probably down to a top level circular dependency. In particular,
trying to paraview to compute the version to form part of the
native-search-path at the top level causes problems.

Making openfoam have LD_LIBRARY_PATH as a search path seems like the
incorrect use of search paths though, since you're searching for
something in the same package. Replacing this with wrapping would be an
improvement, although still I'm unsure why LD_LIBRARY_PATH would need
setting in this case.


signature.asc
Description: PGP signature

Re: Failed to build in QA

[Christopher Baines]

"reza.housse...@gmail.com"  writes:

> I submitted an issue to guix. But QA refuses to build it [1]. I have
> no clue what the problem is, can anyone shed light on a possible
> resolution?

You pretty much found the problem, the relevant line on the page you
linked to is:

[  6/ 50] loading... 24.0% of 25 filesbuilder for 
`/gnu/store/qhvpjfn3d9cwz5zxadblbnbqa92a8i27-guix-cli-core.drv' failed due to 
signal 11 (Segmentation fault)

So the data service wasn't able to build Guix. This probably isn't due
to your changes, and it doesn't happen very often, so the thing to do
here is just retry.

I've triggered QA to reapply the patches now (by deleting the
issue-66262 branch), so hopefully things will work better this time.

Thanks,

Chris


signature.asc
Description: PGP signature

Re: Guix related things in Germany at the end of October/start of November

[Christopher Baines]

Efraim Flashner  writes:

> [[PGP Signed Part:Signature made by expired key 41AAE7DCCA3D8351 Efraim 
> Flashner ]]
> On Thu, Sep 28, 2023 at 11:58:10PM +0200, Ricardo Wurmus wrote:
>> 
>> Christopher Baines  writes:
>> 
>> > PackagingCon [1] is happening in Berlin on the 26th to the 28th of
>> > October […]
>> > Is anyone else planning to attend these events, or otherwise interested
>> > in meeting up in Germany around these dates?
>> 
>> I don’t seem to have anything planned around the time of PackagingCon,
>> so should a little Guix meeting happen in Berlin I’d try to make time to
>> stop by for a while.
>
> I think I got everyone in the list.
>
> It looks like I won't be traveling with wife & kids this trip.
>
> PackageCon is Thursday & Friday in Berlin with a hackday on Saturday.
> Then Reproducible Builds is Tuesday, Wednesday, Thursday in Hamburg.
> Saturday hackathon entrance fee is 5€ I think but it comes with the
> location and I assume outlets and internet and seems like a good idea to
> repurpose it for a meetup. Or we can meetup for food at some point.
>
> I don't have any ideas yet for Hamburg, except that I don't know if
> Chris has any plans on when he's heading over from Berlin and I haven't
> booked anything yet.

I'm planning to arrive in Berlin on Tuesday the 24th, then head from
Berlin to Hamburg on Monday the 30th then back to the UK on Monday the
6th.

I haven't thought about the Saturday sprint day yet, but maybe that is a
good opportunity to meet up in Berlin.


signature.asc
Description: PGP signature

Re: [workflow] Automatically close bug report when a patch is committed

[Christopher Baines]

Giovanni Biscuolo  writes:

> [[PGP Signed Part:Signature made by expired key D37D0EA7CECC3912 Giovanni 
> Biscuolo (Xelera) ]]
> Hi,
>
> Giovanni Biscuolo  writes:
>
> [...]
>
 The first thing we need is a server side git post-receive hook on
 Savannah, I've opened the sr#110928 support request:
 https://savannah.nongnu.org/support/index.php?110928
>>>
>>> It's something that the Savannah folks would need to maintain
>>> themselves, right?
>>
>> Forgot to mention that I'm pretty sure a post-receive server side hook
>> is already running (and maintained) for our guix.git repo on Savannah,
>> it's the one that sends notifications to guix-commits mailing list
>> https://lists.gnu.org/mailman/listinfo/guix-commits
>
> Regarding server side git hooks, I forgot to mention that on 2023-08-31
> a new commit-hook is available on Savannah (installation must be
> requested per-project):
>
> git post-receive UDP syndication
> https://savannah.gnu.org/news/?id=10508
>
>
> A new commit-hook is available to install for git repositories that will send 
> a single Datagram via UDP after each successful commit.  This can be useful 
> for continuous integration (CI) schemes and elsewise when a push driven model 
> is prefered to (e.g) regularly repolling upstream when changes may or may not 
> have occured. 
>
> To request installation please open a ticket with the Savannah Administration 
> project:
>
> [...]
>
> The (sh, GPLv3+) post-receive script source, detail on how the Datagram is 
> structured, and example "receiver" scripts (in perl) can be found here:
>
>   https://git.sr.ht/~mplscorwin/git-udp-syndicate
>
> Maybe this hook is useful for comminication with the QA service(s).

Not QA directly, but the Guix Data Service would benefit from supporting
this, as that could supplement or replace listening for emails to
determine when new revisions have been pushed.

I've added an entry to the TODO list in the README.


signature.asc
Description: PGP signature

Re: Process for reviewing patches as someone without commit access

[Christopher Baines]

Christopher Baines  writes:

> I've been reviewing the list of ideas on and around QA ([1]) recently,
> and got thinking again about how to support people without commit access
> reviewing patches. Obviously you don't need commit access to review
> patches, but where I think we need some process is how to expedite
> someone with commit access taking a look at the patches that have been
> reviewed, and merging them if appropriate.
>
> 1: https://qa.guix.gnu.org/README
>
> Maybe we can use debbugs tags for this? It looks like this has already
> been done in the past, I found some issues tagged with the usertag
> "reviewed" for example [1]. Some were also tagged with
> "reviewed-looks-good". I guess my primary concern is to have a tag (or
> combination of tags) which indicate that a committer should have a look
> at the issue as it's been reviewed by someone and should be ready to
> merge. I don't really use debbugs much, but does anyone have any
> opinions on appropriate tags?
>
> 1: https://debbugs.gnu.org/cgi-bin/pkgreport.cgi?tag=reviewed=guix
>
> Once we know what tags to use, I can have the QA frontpage do something
> similar to the "Mark as moreinfo" links, so it's easy to just click a
> button then send the email to change the state of a issue.

I've gone ahead and implemented an initial version of this. On the page
for an issue now there's a form to mark patches as reviewed. This
replaces the previous review checklist and notes field.

That form then takes you to a page to submit the review. This can be
done through the mailto link, or by following the manual
instructions. Of course people using other ways of interacting with
debbugs can also just add the reviewed-looks-good usertag for the guix
user.

Once QA has picked up that the reviewed-looks-good usertag is present,
it'll display a dark green status and the issue will appear at the top
of the /patches page. My hope here is that people with commit access can
prioritise merging patches that have been reviewed and should be ready
to merge.

The implementation is a bit iffy, Mumi doesn't have access to the
usertags as far as I can tell, so I'm getting them from the debbugs soap
interface (luckily Ricardo has already figured that out), but it gives
you every issue tagged for every usertag for that user, which probably
won't scale well if lots of people start reviewing patches. But if that
happens we can always look at how to address it.

Thanks,

Chris


signature.asc
Description: PGP signature

[PATCH] web: Include merged_with in graphql .

[Christopher Baines]

* mumi/web/graphql.scm (): Include merged_with.
---
 mumi/web/graphql.scm | 12 +++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/mumi/web/graphql.scm b/mumi/web/graphql.scm
index 6dcb8ce..2c7c676 100644
--- a/mumi/web/graphql.scm
+++ b/mumi/web/graphql.scm
@@ -69,7 +69,17 @@
   (issue-messages (bug-num parent
   (blocked_by (non-nullable-type (list-type ))
   (lambda (parent . _)
-(map bug-status (bug-blockedby parent)
+(map bug-status (bug-blockedby parent
+  (merged_with (non-nullable-type (list-type ))
+   (lambda (parent . _)
+ (map
+  bug-status
+  (match (bug-mergedwith parent)
+   ((? string? str)
+(string-split str #\space))
+   ((? number? n)
+(list (number->string n)))
+   (#f '()))
 
 (define-object-type 
   (name  (lambda (parent . _)
-- 
2.41.0

Re: [PATCH qa-frontpage WIP] Add a library to parse patchwork json data

[Christopher Baines]

Vivien Kraus  writes:

> Hi!
>
> Here is a small library that exports 3 types:
> −  is the collection of metadata that is present
>   in the square brackets in the patch names;
> −  is an individual item of the patch series;
> −  is a whole series of patches;
>
> And a set of functions to parse and serialize these.
>
> A fun experiment is to run the following script:
>
> (use-modules (guix-qa-frontpage patchwork patch-series))
> (use-modules (rnrs bytevectors))
> (use-modules (web client))
> (use-modules (ice-9 receive))
> (use-modules (json))
>
> (define patchwork-data
>   (receive (r body)
>   (http-get 
> "https://patches.guix-patches.cbaines.net/api/patches/?order=-id;)
> (json-string->scm (utf8->string body
>
> (define patchwork-series
>   (map scm->patch-series (vector->list patchwork-data)))
>
> (for-each
>  (lambda (correct-series)
>(display correct-series)
>(newline))
>  (map patch-series->scm patchwork-series))
>
> You will see that patchwork has quite a lot of creativity when it
> comes to breaking my expectations. I made sure to add as much
> information in exceptions so that we can understand what is happening.

This looks good, but would it be possible to adapt this to work with the
series endpoint [1], rather than the patches one?

1: https://patches.guix-patches.cbaines.net/api/series/?order=-id

I think the use of the patches endpoint currently is just because
previously the Patchwork checks information was used. Now that Patchwork
checks aren't used, I think the series endpoint can be used instead, and
this should map much more closely to the data structures you're trying
to construct.

Thanks,

Chris


signature.asc
Description: PGP signature

Re: July update on qa.guix.gnu.org

[Christopher Baines]

Simon Tournier  writes:

> On Wed, 26 Jul 2023 at 09:44, Christopher Baines  wrote:
>
>> The queue is very large at [3].
>
> Some patches trigger a lot of rebuilds.  And the threshold is about 300
> or something, I guess.  Would it be possible to mark these items?
>
> For instance, #65636 [1] is marked as ?
>
> 65636 ?   [bug#65636] gnu: git: Update to 2.42.0.
>
> when it implies 799 rebuilds.  Somehow, that’s not the same Unknown as
> the one not-yet processed.
>
> Would it be possible to mark these items with ’TooMany’ instead of
> ’Unknown’?  Or whatever term better than TooMany?  And instead of a grey
> ’?’, maybe a cross grey.
>
> 1: https://qa.guix.gnu.org/issue/65636

Yep, it would be nice to add more statuses [1] for issues to make more
of the data easily available.

1: 
https://git.cbaines.net/guix/qa-frontpage/tree/guix-qa-frontpage/issue.scm#n43


signature.asc
Description: PGP signature

Re: guix QA "fails to process revision"

[Christopher Baines]

Andy Tai  writes:

> For a submitted patch, if Guix QA "fails to process revision" as in this log
> https://data.qa.guix.gnu.org/job/49399
>
> [ 32/ 40] compiling... 60.0% of 20 filesmadvise failed: Cannot allocate memory
> builder for `/gnu/store/j3hy5gymlfrdrhm8aj2brnsa2pix16n2-guix-home.drv'
> failed due to signal 11 (Segmentation fault)
> @ build-failed /gnu/store/j3hy5gymlfrdrhm8aj2brnsa2pix16n2-guix-home.drv
> - 1 builder for
> `/gnu/store/j3hy5gymlfrdrhm8aj2brnsa2pix16n2-guix-home.drv' failed due
> to signal 11 (Segmentation fault)
> cannot build derivation
> `/gnu/store/yh45c2z642xv2m752i3mj4lahj7q4qxm-guix-cli.drv': 1
> dependencies couldn't be built
> ...
>
> what can be done about it?

I think in this case the machine (beid) was short on memory, it has 32GB
of RAM and 16GB of swap.

I've reduced the max processes for the data service processing new
revisions which should help, but there's probably more that can be done.

There looks to be regular spikes in the memory used by the data service
process, so it would be good to look in to why that happens, and what
can be done to avoid it.


signature.asc
Description: PGP signature

Re: [PATCH qa-frontpage] Apply incoming patches onto the correct feature branch

[Christopher Baines]

Vivien Kraus  writes:

> It seems that the only available information of the feature branch is
> in the patch name.
> ---
>
> Hello guix!
>
> Thank you for making the QA tool. It seems to me that feature branches
> are ignored for patches. The patches seem to always be applied on top
> of master. I get that idea because the base-for-issue-* tag is always
> put on HEAD, and all my patches targetting gnome-team recently get
> applied to master. I understand that the latter could be a problem
> with me.
>
> If patches are indeed applied on top of master, then the question
> arises, what to do. The available information from patchwork is
> scarce: the base-commit is not available, and the optional feature
> branch only appears in the name of the patches of the series.
>
> I think it is possible to parse the patch names. This way, we get some
> useful information: the feature branch, the series revision, and the
> patch index. The patch index can be used to make sure the patches are
> in correct order (this can be the case if a server in the path
> re-orders the emails).
>
> What do you think?

Hi Vivien,

Thanks for trying to implement this and sending a patch, I think it's a
good feature to add but there's a few things needed to make this work.

Firstly, you need to actually change which branch the patch is applied
to, and the place to do that is here [1] in the call to
with-git-worktree. You can probably use get-commit from
(guix-qa-frontpage git-repository) to find the commit hash for a
branch. You'll also probably need to move around some of the code in
create-branch-for-issue so that you have the data about the patches to
work out what the desired branch is before you call with-git-worktree.

1: 
https://git.cbaines.net/guix/qa-frontpage/tree/guix-qa-frontpage/manage-patch-branches.scm#n250

The second thing is that there's other places in the codebase that
assume the patches have been applied to the master branch. In
start-manage-patch-branches-thread, there's
get-changes-compared-to-master which is used to decide if a branch needs
recreating if there's too many changes between the base revision and a
recent master branch revision. That'll need changing to at least skip
over any patches been applied to a non-master branch, or perform the
comparison against the tip of the relevant branch.

The third thing that will need to change to allow this to happen is
adding clear messaging on to the issue page to indicate where patches
have been applied to non-master branches. That'll help to avoid people
merging these patches in to the master branch rather than the branch
they were intended for.

Maybe what would be helpful is a procedure in (guix-qa-frontpage issue)
that takes the data for the issue from latest-patchwork-series-by-issue,
and gives you back the branch name that the patches should be applied
to. That can then be used to get the branch information for all the
above 3 use cases.

As for trying to order the patches, do you know of a case where the
ordering from Patchwork is wrong? I think it's only worth changing the
ordering in the qa-frontpage if we know we're doing it for a reason.

There's now a create-issue-branch command to the guix-qa-frontpage
script as well, which will be a good way of testing the code out for
applying patches to various branches. It'll fail when it comes to
pushing the branch, but it'll still be useful for testing the code up to
that point.

Would you be able to look at sending some revised patches?

Thanks,

Chris


signature.asc
Description: PGP signature

Re: Implementing the guix-dameon in Guile

[Christopher Baines]

Mathieu Othacehe  writes:

>> I imagine the daemon could be structured as a set of actors (it’s really
>> my thing these days ;-)), with an eye on facilitating code sharing and
>> interaction with the Coordinator, Cuirass, and all that.
>
> I think this point is really important here. If the offload mechanism of
> the existing guix-daemon was scaling correctly, with a better API, we
> would probably never had to develop the Cuirass remote worker and the
> Build Coordinator mechanisms.

Indeed.

> Perhaps, part of the Build Coordinator code could be one day integrated
> inside the new guix-daemon so that we can only use one offload mechanism
> for all use cases. Users of the current "guix offload" as well as CI
> tools could rely upon a unique offload mechanism.

Yeah, I think the ideal situation would be to have a daemon that's
flexible and extensible enough to allow this, without being too
complicated or fragile.

> The three other points I had in mind when proposing the subject to NLNet
> were to:
>
> - Fix the sqlite contention issues that have been observed on Berlin and
>   other machines with huge local databases.

Yeah, I think there's some simple things that can be done to improve the
use of SQLite, and then some instrumentation can be added so that any
performance issues can be better understood.

> - Remove the GC lock: I think it has been done on recent nix-daemon.

I'm hoping that with some careful design of how to do garbage
collection, this should be possible.

> - Add support for unprivileged daemon.

I'm not quite sure what is required for this, but maybe it would be
possible without too much effort.

Thanks for your input!

Chris


signature.asc
Description: PGP signature

Re: Implementing the guix-dameon in Guile

[Christopher Baines]

Mekeor Melire  writes:

> 2023-09-13 16:36 m...@cbaines.net:
>
>> I think this has been talked about for a while [1], but I want to
>> make it happen. Currently the guix-daemon is still similar to the
>> nix-daemon that it was forked from, and is implemented in C++. I
>> think that a Guile implementation of the guix-daemon will simplify
>> Guix and better support hacking on and around the daemon to add new
>> features and move Guix forward.
>
> Yippie! I'm really excited about this; and convinced that it'll
> greatly impact and improve the Guix project.
>
>> I'd like to hear what people think about which direction the
>> implementation should go, and what features they'd like to see.
>
> Here's a feature-request (in the long-run). But I'm not sure if I
> understand correctly that a new, improved Guix-daemon enables us to
> implement this. Sorry if I'm wrong. I'd like to be able to step
> through the build-process of a package phase-by-phase and see what
> changed after each phase by looking at the file-tree.

This definitely isn't in the minimal scope, but it's something that I
think having the Guile daemon will maybe make easier to do.

Given the phases are running inside the builder script, I think you'd
need to have that cooperate somehow, and with the help of the daemon
connect somehow to the user so that they can pause and resume the build
process, and inspect the internal state of the builder. The daemon could
also provide access to process/filesystem information as well.


signature.asc
Description: PGP signature

Re: Implementing the guix-dameon in Guile

[Christopher Baines]

Caleb Ristvedt  writes:

>> Still though, I'd like to hear what people think about which direction
>> the implementation should go, and what features they'd like to see. Even
>> if those are not essential to make the Guile implementation viable, it
>> still might inform the direction to take.
> Okay, brain dump time:
>
> I think that using fibers has a lot of potential, but there are
> obstacles that need to be worked around.  In the single-threaded case,
> we risk a big slowdown if multiple clients are active at once, since
> we're doing what used to be done by n processes with one single thread.
> It would be especially noticeable during big disk reads and writes,
> since those basically ignore O_NONBLOCK, and most procedures that act on
> entire files at once would therefore probably not hit many yield points.
> The worst situation would be where multiple worker fibers are attempting
> to do reference scanning at the same time.  Posix asynchronous disk IO
> could be used, but glibc currently implements it... using a thread pool.
> There is the RWF_NOWAIT flag to preadv2, though it's only available on
> newer linuxes and has bugs in 5.9 and 5.10.
>
> Additionally, being single-threaded means use of F_SETLKW is a no-go, so
> you're stuck with polling there.  Granted, that's not such a big issue if
> in 99% of use cases there is only one process doing the locking, so it
> can all be managed internally.

I think a thread pool is essential for using SQLite, so we're already
going to have thread pools interacting with fibers, so if more cases
come up where this is needed, then I think it's an option.

> Speaking of file locks, the risk of accidental clobbering of locks jumps
> way up once it's all moved in to one process, and IIRC we already have
> bugs with accidental clobbering of locks.  You can get a half-decent
> interface by doing what sqlite does, which is a combination of
> intra-process locking and holding on to open file descriptors until all
> locks on the underlying file are released.  There are some subtle
> pathological cases there that are a lot more likely in the guix daemon
> than in sqlite, though.  For example, suppose you open a file twice to
> get ports p1 and p2, acquire read locks on both of them, then close p1,
> then open the file again to get p3, acquire a read lock on it, close p2,
> get p4, acquire a read lock on it, close p3, get p5... and so on.  This
> will cause unbounded file descriptor usage, and eventually you'll run
> out.  There is no workaround in this model other than "hope that usage
> pattern doesn't come up much".  Additionally, you need to ensure that
> every close of a potentially-locked file goes through a special
> close-wrapper.
>
> I'm actually in the middle of working on a solution for this that
> involves a separate locker process that gets passed file descriptors to
> lock via a unix socket.
>
> Speaking of file descriptors, running the entire daemon in one process
> is going to mean much higher pressure on file descriptor resource usage.
> IIRC, while building a derivation, the closure of its inputs needs to be
> locked, and that means a file descriptor for each and every store item
> in its input closure, simultaneously.  The separate locker process would
> make it possible to retain those locks while not having them open in the
> main process.

Maybe that will still need to happen, however I think it might be
possible to replace the IPC through locking files with inter-fiber
communication inside of the dameon.

Backwards compatibility is the priority though, so this can only happen
in cases where that's unaffected.

> In the multithreaded case, fork() and clone() become concerns, since
> they can no longer be safely run from guile.  One way around this would
> be to use posix_spawn to produce a single-threaded guile process, then
> have that do the fork or clone as necessary.  The fork case shouldn't
> actually be necessary, though, as the child process can just exec
> directly.  In the clone case, CLONE_PARENT can be used to make the
> resulting process a child of the original, main process, though I don't
> know how portable that is to hurd (really, I don't know how namespace
> setup works in general on hurd).  Instead of doing this
> spawn-two-processes-to-spawn-one routine every time we want to set up a
> container, we could create a spawner-helper process once and just keep
> it around.  If we can do that before any threads are created, we don't
> even need posix_spawn (though it is nice to have around, and I do have
> bindings to it).  I remember reading that that's what the apache web
> server did.
>
> This would however mean some places would need to use interfaces like
> "eval-with-container" instead of "call-with-container", which is
> somewhat less convenient.  But code staging shouldn't be a terribly
> foreign concept to most guixers.
>
> Another concern is child process management; a blocking waitpid will of
> course block the calling 

Re: Implementing the guix-dameon in Guile

[Christopher Baines]

Josselin Poiret  writes:

>> 2: https://nlnet.nl/project/GuixDaemon-Guile/
>>
>> Rewrites are risky because you only get the value right at the end,
>> therefore the priority is to get a minimal but viable implementation in
>> Guile that can be switched to, and not to get distracted on adding or
>> improving functionality unnecessarily. That is better done once the new
>> implementation has been adopted.
>>
>> While I think there's a substantial amount of work to do, progress
>> towards a Guile guix-daemon has already been made. There was a least one
>> GSoC project which did make progress, and there's Guile implementations
>> of some of the functionality in Guix already.
>>
>> Still though, I'd like to hear what people think about which direction
>> the implementation should go, and what features they'd like to see. Even
>> if those are not essential to make the Guile implementation viable, it
>> still might inform the direction to take.
>
> I think the #1 feature for me would be to have it completely
> unpriviledged using mount namespaces, so that you could still build
> software without needing to run the daemon on the system.  You won't be
> able to run the built software without using namespaces as well though,
> but that still a step in the right direction imo.
>
> WDYT?

Thanks for the suggestion :)

I'm not quite sure what this would involve, but it sounds like it might
not be that hard to do.


signature.asc
Description: PGP signature

Re: branch master updated: gnu: qemu: Reinstate the iothreads-commit-active test.

[Christopher Baines]

Christopher Baines  writes:

> [[PGP Signed Part:Good signature from 5E28A33B0B84F577 Christopher Baines 
>  (trust ultimate) created at 2023-09-15T12:40:53+0100 using 
> RSA]]
>
> guix-comm...@gnu.org writes:
>
>> commit 076b3384dfa29ae118d0375d516376a7fe98a197
>> Author: Maxim Cournoyer 
>> AuthorDate: Tue Sep 12 09:29:22 2023 -0400
>>
>> gnu: qemu: Reinstate the iothreads-commit-active test.
>>
>> * gnu/packages/virtualization.scm (qemu) [arguments]: Add set-SOCK_DIR 
>> phase.
>> (qemu-minimal) [arguments]: Delete the disable-extra-tests phase.
>> ---
>>  gnu/packages/virtualization.scm | 17 +
>>  1 file changed, 9 insertions(+), 8 deletions(-)
>
> I cannot seem to get qemu-minimal to build for x86_64-linux with this
> change [1].
>
> 1: 
> https://data.guix.gnu.org/gnu/store/zn390q1p6alvklsymfjs7kk286dg0jmn-qemu-minimal-8.1.0.drv
>
> It could be that it just happened to build previously, but this is the
> kind of issue that using QA should help to catch (and ironically, now I
> can't update QA because qemu-minimal fails to build).

12th times the charm apparently.


signature.asc
Description: PGP signature

Re: branch master updated: gnu: qemu: Reinstate the iothreads-commit-active test.

[Christopher Baines]

guix-comm...@gnu.org writes:

> commit 076b3384dfa29ae118d0375d516376a7fe98a197
> Author: Maxim Cournoyer 
> AuthorDate: Tue Sep 12 09:29:22 2023 -0400
>
> gnu: qemu: Reinstate the iothreads-commit-active test.
>
> * gnu/packages/virtualization.scm (qemu) [arguments]: Add set-SOCK_DIR 
> phase.
> (qemu-minimal) [arguments]: Delete the disable-extra-tests phase.
> ---
>  gnu/packages/virtualization.scm | 17 +
>  1 file changed, 9 insertions(+), 8 deletions(-)

I cannot seem to get qemu-minimal to build for x86_64-linux with this
change [1].

1: 
https://data.guix.gnu.org/gnu/store/zn390q1p6alvklsymfjs7kk286dg0jmn-qemu-minimal-8.1.0.drv

It could be that it just happened to build previously, but this is the
kind of issue that using QA should help to catch (and ironically, now I
can't update QA because qemu-minimal fails to build).


signature.asc
Description: PGP signature

Re: Implementing the guix-dameon in Guile

[Christopher Baines]

Ludovic Courtès  writes:

>> Rewrites are risky because you only get the value right at the end,
>> therefore the priority is to get a minimal but viable implementation in
>> Guile that can be switched to, and not to get distracted on adding or
>> improving functionality unnecessarily. That is better done once the new
>> implementation has been adopted.
>
> In the past I wondered, as Maxim wrote, whether we could move
> incrementally—after all, a fair bit of “the daemon” is already in Scheme
> (there’s substitute support and other helps, plus (guix store …) etc.)
> I’m not sure that’s feasible or desirable though.
>
> My take today :-) is that ‘wip-guile-daemon’ is a great starting point.
> We could aim towards having a minimal ‘guix daemon’ (space) command that
> would coexist with ‘guix-daemon’ and that people could try out soonish
> (I think Caleb got it to build derivations back then).  Eventually less
> adventurous people will use it, and at some point it’ll be sufficiently
> mature that we can default to ‘guix daemon’ instead of ‘guix-daemon’.

Yep, my hope is that this can happen over the next year.

> Technically, I think it should be a single-threaded Fibers program,
> building on the experience we got from the Coordinator, shepherd, etc.
> That should allow us to do everything in one process (in contrast, the
> C++ implementation forks for every incoming connection, which then
> significantly complicates the implementation of locking, etc.)

Yep, that sounds good to me.

> I imagine the daemon could be structured as a set of actors (it’s really
> my thing these days ;-)), with an eye on facilitating code sharing and
> interaction with the Coordinator, Cuirass, and all that.

I'm not very familiar with actors, but I guess that's similar to having
a bunch of cooperating fibers which handle different things.

And maybe not in the short term, but if the Guile daemon implementation
is flexible and extensible enough, it should be possible to replace the
build coordinator with some extensions in and around the guix daemon.


signature.asc
Description: PGP signature

Re: Implementing the guix-dameon in Guile

[Christopher Baines]

Vagrant Cascadian  writes:

> On 2023-09-13, Christopher Baines wrote:
>> I think this has been talked about for a while [1], but I want to make it
>> happen. Currently the guix-daemon is still similar to the nix-daemon
>> that it was forked from, and is implemented in C++. I think that a Guile
>> implementation of the guix-daemon will simplify Guix and better support
>> hacking on and around the daemon to add new features and move Guix
>> forward.
>>
>> 1: 
>> https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/ROADMAP.org#n71
> ...
>> Let me know if you've got any comments or questions!
>
> Sounds great!
>
> My only real concern, as someone maintaining guix packages in Debian, is
> to make sure that we do not break compatibility with being able to use
> an older daemon, as Debian stable/bookworm is still at guix 1.4.x and it
> would be nice to not have to force people to manually upgrade the daemon
> (e.g. and even if a newer version lands in a future Debian stable
> release, in general it will stuck using that version for some years as
> well).
>
> I have noticed occasional issues with the Debian packages of guix having
> compatibility issues when newer versions of guile-git/libgit2,
> guile-ssh/libssh2, etc. get introduced, and wonder if the same would
> hold true of a daemon?
>
> In Guix, by design you wouldn't really notice these sorts of problems as
> it is always generally built with the current version, but Debian does
> rely on ABI compatibility for package upgrades... I might be able to
> keep better track of these types of issues in Debian, although various
> guile-* modules that depend on C libraries seem to avoid the normal
> detection mechanisms to trigger rebuilds in Debian.
>
> That is a bit of a tangent, but it reminded me about that issue...

I think compatibility is a priority, and although I haven't looked too
much in to the details yet, I think it's quite realistic.

I think it's very important for Guix to keep compatibility with older
daemons, and on the daemon side, I'd want to see the Guile
implementation work with older Guix's, as well as it being possible to
switch back and forth between the implementations without issue.


signature.asc
Description: PGP signature

Re: Implementing the guix-dameon in Guile

[Christopher Baines]

Maxim Cournoyer  writes:

>> The transformation toward a Guile daemon is a point of consistency and
>> pride for the project and therefore unlikely to be second-guessed or
>> reverted. My recommendation is to replace the daemon gradually—working
>> from (apply system* (command-line) downward—and mainline your
>> incremental changes as quickly as possible.
>
> I had some musing about the daemon recently; I was thinking libguile
> could be added to our old C++ daemon, which could then replace its
> functions piece-wise with Scheme implemented ones?
>
> At the end, we'd have almost everything in Scheme, at which point we
> could rewrite the main loop to Scheme and say farewell to the old
> daemon.
>
> Would that approach makes any sense?  I know of at least one C++ project
> integrating libguile, and that's the jami-daemon, so it could perhaps
> provide some clues as to how to proceed to do so.

I think it's a viable approach, but probably not one for me. Since I
don't really write C++, I think that trying to replace parts of the C++
code and keep things working would take more work than just writing
Guile.


signature.asc
Description: PGP signature

Implementing the guix-dameon in Guile

[Christopher Baines]

Hey!

I think this has been talked about for a while [1], but I want to make it
happen. Currently the guix-daemon is still similar to the nix-daemon
that it was forked from, and is implemented in C++. I think that a Guile
implementation of the guix-daemon will simplify Guix and better support
hacking on and around the daemon to add new features and move Guix
forward.

1: 
https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/ROADMAP.org#n71

My plan is to focus on this over the next year. I left my previous day
job quite a few months ago now to take a bit of a break, that's the main
reason I've been able to spend more time trying to push forward some of
the QA stuff. With some monetary support from NLNet [2], I'm planning to
continue this break and focus for the next year on getting a Guile
implementation of the guix-daemon written and adopted.

2: https://nlnet.nl/project/GuixDaemon-Guile/

Rewrites are risky because you only get the value right at the end,
therefore the priority is to get a minimal but viable implementation in
Guile that can be switched to, and not to get distracted on adding or
improving functionality unnecessarily. That is better done once the new
implementation has been adopted.

While I think there's a substantial amount of work to do, progress
towards a Guile guix-daemon has already been made. There was a least one
GSoC project which did make progress, and there's Guile implementations
of some of the functionality in Guix already.

Still though, I'd like to hear what people think about which direction
the implementation should go, and what features they'd like to see. Even
if those are not essential to make the Guile implementation viable, it
still might inform the direction to take.

The Guile rewrite of the guix-dameon was on my mind over 3 years ago
when I was thinking about the build coordinator [3]. As part of writing
the build coordinator, I got experienced using SQLite and the Guile
bindings and that'll come in very useful. The build coordinator also
uses fibers, and I'll probably look to use fibers as well in the
guix-daemon. I think the work Ludo has done on the shepherd has made
this possible.

3: https://lists.gnu.org/archive/html/guix-devel/2020-04/msg00323.html

Let me know if you've got any comments or questions!

Thanks,

Chris


signature.asc
Description: PGP signature

Re: How can we decrease the cognitive overhead for contributors?

[Christopher Baines]

Maxim Cournoyer  writes:

> Hi Vagrant,
>
> Vagrant Cascadian  writes:
>
>> On 2023-09-06, Liliana Marie Prikler wrote:
>>> Am Dienstag, dem 05.09.2023 um 19:41 -0400 schrieb brian
 ‘* foo/bar.scm new-package (inputs): add input’
 
 stuff. I literally can never remember this format, no matter how many
 times I do it. I'm reasonably sure square brackes go in there some
 where. It can take me quite a while to put together all that stuff,
 even with magit's help.
>>> It's 
>>>
>>> * file (variable)[field]{do you need 4 levels?}
>>
>> Honestly, not knowing the difference between a variable and field and
>> selector... this comment is of little help to me.
>>
>> I always get tripped up with phases, modify-phases, etc. as there seem
>> to be potentially four or more levels deep in some common code
>> patterns... for example, a recent commit mentioning phases:
>
> The ChangeLog Style section suggests there should be spaces between the
> "markers".  It also says the square brackets should be used to denote a
> change made in a conditional code path, so we're abusing the meaning of
> it in Guix to just a 'field' (which is, a record field, or slot, for the
>  record in Guix).  < > is to precise even more the place
> modified.  They're just shortcuts to avoid less typing while remaining
> readable.
>
> Here's an example in the Guix "dialect":
>
> * gnu/packages/file.scm (package-symbol)
> [arguments] <#:phases>: New patch-paths phase.
>
>
> It could also have been:
>
> * gnu/packages/file.scm (package-symbol) [arguments]: Add patch-paths
> phase.
>
> It doesn't really matter, as long as it's clear and you did the exercise
> of reviewing the code you touched and writing down the changes summary
> for the reviewer (and yourself).

I think I have a similar feeling to Vagrant, and on whether this
matters, I think it matters a lot.

In terms of encouraging people to contribute to Guix, we want the
documentation to set people up for success, or at least feeling like
they're doing things correctly.

The docs say:

  Please write commit logs in the ChangeLog format (see Change Logs in
  GNU Coding Standards); you can check the commit history for examples.

Which is expecting a lot of people who haven't encountered this
before. My view on this is that asking people to perform this task that
isn't very well documented or defined, and which people reviewing and
committing the changes will comment on or revise (maybe to be
objectively better, or maybe just in their own style) discourages people
to continue to contribute, especially those that care more about not
making mistakes/getting things right, or that lack confidence.

I'm indifferent about how we write the commit messages, I'm strongly in
favour of the expectations being clear and documented so that if people
new to the project want to attempt to write a good commit message,
they're setup for success with good documentation, and if something they
write can be improved, there should be some bit of documentation you can
point them to.


signature.asc
Description: PGP signature

Re: Process for reviewing patches as someone without commit access

[Christopher Baines]

Felix Lechner  writes:

> On Wed, Sep 6, 2023 at 9:47 AM Christopher Baines  wrote:
>>
>> Maybe we can use debbugs tags for this?
>
> Instead of pushing people into reviews and then again making the same
> committers a bottleneck, I would offer some entry-level contributors
> commit rights but require that they obtain approval for some steps. It
> can be done on a trust basis.

It's an idea, although one I'd discount based on how many breaking
changes (including ones with wide impact like breaking guix pull) happen
with the current criteria for granting commit access.

I don't want to make reviewing changes more difficult, and I think
setting up more people with commit access and continuing the trend that
it's mostly people with commit access that review changes would increase
the difficulty, compared to what I'm proposing here, which is trying to
empower people who just do review whilst avoiding any of the complexity
of merging and pushing the changes without breaking things.

Now of course you could argue that it being easy to break things is a
problem, and maybe it is, but often it's not strictly someone breaking
something but simply a commit not being signed, or not being signed in a
way that guix accepts. Here I think pushing changes is complicated for
good reason.

> That way, you can train a new generation of committers while getting
> the work done.

In my opinion, I want to see review and committing things become more
separated, because the valuable thing is having good review. Committing
and pushing someone elses changes isn't adding much value to Guix in and
of itself, but good review of those changes does.

If we end up with a big backlog of changes that are reviewed and ready
to merge, then I'm all for training and helping more people do the
committing and pushing, but I have a suspicion that it's the review bit
that takes the time.

Thanks,

Chris


signature.asc
Description: PGP signature

Process for reviewing patches as someone without commit access

[Christopher Baines]

Hey!

I've been reviewing the list of ideas on and around QA ([1]) recently,
and got thinking again about how to support people without commit access
reviewing patches. Obviously you don't need commit access to review
patches, but where I think we need some process is how to expedite
someone with commit access taking a look at the patches that have been
reviewed, and merging them if appropriate.

1: https://qa.guix.gnu.org/README

Maybe we can use debbugs tags for this? It looks like this has already
been done in the past, I found some issues tagged with the usertag
"reviewed" for example [1]. Some were also tagged with
"reviewed-looks-good". I guess my primary concern is to have a tag (or
combination of tags) which indicate that a committer should have a look
at the issue as it's been reviewed by someone and should be ready to
merge. I don't really use debbugs much, but does anyone have any
opinions on appropriate tags?

1: https://debbugs.gnu.org/cgi-bin/pkgreport.cgi?tag=reviewed=guix

Once we know what tags to use, I can have the QA frontpage do something
similar to the "Mark as moreinfo" links, so it's easy to just click a
button then send the email to change the state of a issue.

Chris


signature.asc
Description: PGP signature

Re: [workflow] Automatically close bug report when a patch is committed

[Christopher Baines]

Giovanni Biscuolo  writes:

> Hello,
>
> often bug reports related to patches are left open even after the
> patch/patchset have been applied, the last example is a batch of Debbugs
> manual gardening from Vagrant last Fri and Sat when he closed more than
> 20 bugs with messages similar to this one:
>
>
>  rofi-wayland was added in:
>
>  04b5450ad852735dfa50961d3afc789b2e52b407 gnu: Add rofi-wayland.
>
>  And updated to a newer version in:
>
>  19c042ddf80533ba7a615b424dedf9647ca65b0f gnu: rofi-wayland: Update to 
> 1.7.5+wayland2.
>
>  Marking as done.
>
> (https://yhetil.org/guix/87zg25r0id.fsf@wireframe/)
>
> IMO we need a way automatically close this kind of bug reports... or am
> I missing something?

I think the example you give doesn't relate to what you're looking at
below (a post-receive hook).

There were at least two different issues with patches for adding
rofi-wayland [1] and [2].

1: https://issues.guix.gnu.org/53717
2: https://issues.guix.gnu.org/59241

One improvement I can think of here is that QA should highlight that
some of the changes in each of those patch series can be found in
another patch series.

That would then make it easier to both issues to be closed if that's
appropriate.


signature.asc
Description: PGP signature

Guix related things in Germany at the end of October/start of November

[Christopher Baines]

Hey,

PackagingCon [1] is happening in Berlin on the 26th to the 28th of
October, and the Reproducible Builds summit is happening in Hamburg on
the 31st of October to the 2nd of November.

1: https://packaging-con.org/
2: https://reproducible-builds.org/events/hamburg2023/

I've had a talk proposal on QA and Guix accepted for PackagingCon, and
I'm planning to attend the Reproducible Builds summit as well. Plus I'll
try to make some time to do some tourist stuff while I'm in Germany as
well.

Is anyone else planning to attend these events, or otherwise interested
in meeting up in Germany around these dates?

Chris


signature.asc
Description: PGP signature

Re: Current Issues with Patch Review Workflow Using git.guix-patches.cbaines.net

[Christopher Baines]

"jgart"  writes:

> Hi Guixers,
>
> Andreas' detailed a nice workflow for reviewing patches in a previous thread*:
>
> ```
> git clone https://git.guix-patches.cbaines.net/guix-patches/
> git checkout issue-x
> git format-patch ...
> then in the development checkout of Guix:
> git am ...; make; ./pre-inst-env guix build
> ```
>
> I noticed that there is one issue with this approach after trying it.
>
> Old tickets are not kept around.
>
> For example, A branch for ticket 51810* does not exist anymore.
>
> But that ticket is still open 
>
> Hi Christopher,
>
> Would it be possible to keep around branches for any open tickets?

Yes and no.

Currently the QA frontpage creates branches for the latest 350 [1] patch
series, and it'll remove them if they slip outside of the latest 350.

1: 
https://git.cbaines.net/guix/qa-frontpage/tree/scripts/guix-qa-frontpage.in#n238

Increasing that number is possible, but not really feasible at the
moment since the branch existing feeds in to data.qa.guix.gnu.org
potentially keeping data around for that revision, which has a cost in
disk space, and beid (the machine running data.qa.guix.gnu.org) is
really low on disk space at the moment (23G available out of 338G).

There's also an ongoing cost in that the QA frontpage tries to reapply
patches when big changes happen in the master branch, which involves
more work for the data serivce the more branches you're trying to
manage.

Having more hardware for data.qa.guix.gnu.org, and speeding it up would
help. Another approach would be to have the QA frontpage push these
older patch branches with a slightly different name (that the data
service would ignore), or push them to a different remote. That would
mean that the branch would still be somewhere, but the data service
would ignore it.

> Should stale tickets like 51810 be automatically closed so that
> git.guix-patches.cbaines.net* gets comprehensive coverage of
> ticket/branch pairings?

I don't think automatically closing older issues is helpful, they still
need looking at.


signature.asc
Description: PGP signature

July update on qa.guix.gnu.org

[Christopher Baines]

Hey,

It's been a while since I sent out the last update on QA things that
I've been working on, there was a small update back in May [1] and one
before that in March [2].

1: https://lists.gnu.org/archive/html/guix-devel/2023-05/msg00142.html
2: https://lists.gnu.org/archive/html/guix-devel/2023-03/msg00222.html

## Changes since the last update

The QA frontpage processing branches is now automatically handled
through creating issues. This seems to be working really well, both as a
process for making it clear what's going on with which branch, and also
feeding in to the automation and testing.

Old builds for patches and branches are now cancelled. This helps the
bordeaux build farm work on the most relevant builds.

Branch comparisons now are done against the merge base, rather than
master, this just helps give relevant feedback.

There's also a new package changes page which is linked to from the
package changes table on the issue and branch pages. This starts to
address the long standing issue where you can't see what the numbers in
the package changes table relate to.

## Current state

While the software is getting better, the service overall is struggling
a bit and that looks to continue for the short term at least.

The queue is very large at [3]. The branches for lots of patches could
have been rebased due to the tex-team-next merge, although it's possible
that the missing i686-linux and armhf-linux derivations recently are
causing some patches to be repeatedly rebased. It'll take a few weeks to
get back to a point where there's data on lots of patch series.

3: https://data.qa.guix.gnu.org/jobs/queue

Discussions ongoing about hosting for data.qa.guix.gnu.org going
forward. So far I've been organising and paying for a virtual machine to
run this, but I'd like to stop by the end of the year.

On the bordeaux build farm side, there's been quite a lot of work
recently on growing the available storage for nars. With the recent work
on branches though, I think it's easier to feel the impact of not having
many available machines. While it's possible to just wait longer for
builds to complete, that has it's own disadvantages. harbourfront (one
of the 3 x86_64 machines is going away at the end of August) which will
further reduce the already limited capacity for x86_64-linux and
i686-linux builds.

I'm in the progress of drafting a blog post about the qa-frontpage.

## Next steps

The qa-frontpage still could do with giving clearer and more actionable
feedback. The other big issue on my mind is getting the data service to
process revisions in less time, as that'll help give quicker feedback
and cope better when rebasing large numbers of patch branches. Apart
from that, I'd like to get back to actually using qa.guix.gnu.org to
review patches, that's a little difficult at the moment as data is
available for very view patches.

Also, as mentioned above, more work needs doing on longer term hosting
for some of the services, as well as growing the bordeaux build farm so
that it can keep up with the growing load.

Let me know if you have any questions or comments!

Thanks,

Chris

Re: openshot patch: Guix QA shows 14000 rebuilds

[Christopher Baines]

Andy Tai  writes:

> This is a bug report on Guix QA:
> I submitted a patch updating Openshot to the latest version.  As
> Openshot is an app, it should have few, if any, packages depending on
> it.  but Guix QA seems confused and was rebuilding 14000 packages:
>
> https://qa.guix.gnu.org/issue/64722
>
> This is probably a problem for Guix QA and deserves a look. Thanks

Thanks for raising this Andy. I've been aware of the issue but haven't
been making much progress in tracking it down.

I've created a bug now [1] and I've made some good progress today.

1: https://issues.guix.gnu.org/64762

Thanks,

Chris


signature.asc
Description: PGP signature

Re: 06/24: gnu: mig: Update to 1.8+git20230520.

[Christopher Baines]

guix-comm...@gnu.org writes:

> jpoiret pushed a commit to branch master
> in repository guix.
>
> commit 999a6ac0cfd9339e138007ed9e4e544a55e92e3e
> Author: Josselin Poiret 
> AuthorDate: Mon May 22 11:04:17 2023 +0200
>
> gnu: mig: Update to 1.8+git20230520.
>
> * gnu/packages/hurd.scm (mig)[source]: Update to 1.8+git20230520.
> * 
> gnu/packages/patches/gnumach-add-missing-const_mach_port_name_array_t-type.patch:
> Drop patch.
> * gnu/local.mk (dist_patch_DATA): Unregister it.
> ---
>  gnu/local.mk   |  1 -
>  gnu/packages/hurd.scm  | 20 ++
>  ...missing-const_mach_port_name_array_t-type.patch | 32 
> --
>  3 files changed, 8 insertions(+), 45 deletions(-)

This seemed to be causing the data service some problems, and locally I
was able to reproduce this by trying to build hello for i586-gnu. Guix
would just use more and more memory.

> --- a/gnu/packages/hurd.scm
> +++ b/gnu/packages/hurd.scm
> @@ -88,24 +88,20 @@
>  (define-public mig
>(package
>  (name "mig")
> -(version "1.8+git20220827")
> +(version "1.8+git20230520")
>  (source (origin
> -  (method url-fetch)
> -  ;; XXX: Version 2.35 of glibc can only be built with an
> -  ;; unreleased version of MiG:
> -  ;; 
> .
> -  ;; It cannot be fetched from Git though, as the extra 
> dependency
> -  ;; on Autoconf/Automake would complicate bootstrapping.
> -  (uri (string-append "mirror://gnu/guix/mirror/mig-"
> -  version ".tar.gz"))
> +  (method git-fetch)

I did check to see if the new inputs were the cause, but that seemed
fine. Although I'm guessing that maybe using git-fetch here is probably
the cause.


signature.asc
Description: PGP signature

Re: Changes to the branching/commit policy

[Christopher Baines]

John Kehayias  writes:

>>> Or does the section about branch building for once patches are pushed
>>> to a branch on Savannah?
>>
>> I'm not sure what you're asking here.
>
> I'm confused myself over my wording, but I remember what I was trying
> to get at:
>
> Currently QA doesn't build patches if they cause a large number of
> rebuilds correct? So for building a branch that requires this, will
> that only happen on Cuirass with a new jobspec for the branch? Or will
> this be addressed through changes to how QA builds? (Maybe this
> answered below actually.)

Yep, for issues there's a cap of 300 builds per system.

QA does build the branch at the front of the queue to be merged though,
currently that's ruby-team [1].

1: https://qa.guix.gnu.org/

>>> Does that mean pushing to a branch should follow the same 1-2 week
>>> review allowing QA builds? I guess patch series are always built
>>> together on QA but wondering if there is anything else to be aware of
>>> or needs mentioning to keep things tidy and clear.
>>
>> Those durations mentioned in the commit policy [1] are intended to allow
>> for human review. For QA, it doesn't need to be time based as it can
>> report it's progress. Obviously it does need a bit of time to check
>> things, but I prefer to leave it up to people to assess the changes and
>> any information provided by QA and decide when it's appropriate to push.
>>
>> 1: 
>> 
>>
>
> A separate issue which I wanted to get at was about pushing changes to
> any branch on Savannah. Do we expect those to be at the same state as
> anything that would go directly to master, just pending the testing of
> builds basically? So, after the usual review period? Or can those be
> more WIP and not polished yet? I suppose this is for a team/people
> working on a branch to discuss and how it will then be merged to
> master.

The latter. Non-master branches can be in whatever state is useful.

>> On keeping things clear, I think with branches I'm hoping the issue
>> tracker can help with this, which is why there's now a strong
>> requirement to create a guix-patches issue when you want to merge a
>> branch, and use those issues to manage the timing.
>>
>> For those issues, there's currently a convention of using the following
>> title: `Request for merging "" branch` e.g. [2]. That helps QA find
>> these issues and act accordingly, but of course if someone comes up with
>> a better way of naming these issues, we can just adjust the code in the
>> qa-frontpage.
>>
>
> I see, that's a nice way to then group up discussion if a branch has a
> bunch of separate patch threads. Looks like a good idea!
>
> So, to be concrete, with the mesa patch I just sent,
>  I can open a merge request issue for
> QA to process the branch, once the branch is actually created on
> Savannah? I assume with the pretty trivial version change here I could
> do that to see how the builds go, but I'll hold off pending the thread
> I just started about this branch/team.

Currently QA just builds the branch at the front of the queue, but yes.


signature.asc
Description: PGP signature

Re: Changes to the branching/commit policy

[Christopher Baines]

John Kehayias  writes:

>> I've now merged these changes as
>> 0ea096ae23fa81f05ce97e5e61c15647c0a475ec.
>>
>> You can now see the updated Commit Policy on the website [1] (you might
>> need to force a refresh), as well as the new section on managing patches
>> and branches [2].
>>
>> 1: 
>> 
>> 2: 
>> 
>>
>
> Thanks for these changes! Question on branches (sorry if this was
> covered in a previous thread, but now that we have new language in the
> manual I figure this is a good place): do we have a convention on
> branch names and subject headers for emailing patches for the branch?
> e.g. does [PATCH  1/3] do anything on the QA end?

On the names of branches, there's some typical names like -team, or
wip- but really it's up to you.

QA doesn't currently support applying patches to anything other than the
master branch, but that's something that should probably be addressed at
some point. I'd encourage you to do whatever you think is useful here,
then hopefully QA can use that to act appropriately.

> Or does the section about branch building for once patches are pushed
> to a branch on Savannah?

I'm not sure what you're asking here.

> Does that mean pushing to a branch should follow the same 1-2 week
> review allowing QA builds? I guess patch series are always built
> together on QA but wondering if there is anything else to be aware of
> or needs mentioning to keep things tidy and clear.

Those durations mentioned in the commit policy [1] are intended to allow
for human review. For QA, it doesn't need to be time based as it can
report it's progress. Obviously it does need a bit of time to check
things, but I prefer to leave it up to people to assess the changes and
any information provided by QA and decide when it's appropriate to push.

1: 
https://guix.gnu.org/manual/devel/en/html_node/Commit-Access.html#Commit-Policy

On keeping things clear, I think with branches I'm hoping the issue
tracker can help with this, which is why there's now a strong
requirement to create a guix-patches issue when you want to merge a
branch, and use those issues to manage the timing.

For those issues, there's currently a convention of using the following
title: `Request for merging "" branch` e.g. [2]. That helps QA find
these issues and act accordingly, but of course if someone comes up with
a better way of naming these issues, we can just adjust the code in the
qa-frontpage.

2: https://issues.guix.gnu.org/63713
3: 
https://git.savannah.gnu.org/cgit/guix/qa-frontpage.git/tree/guix-qa-frontpage/branch.scm#n63

Thanks for your questions :)

Chris


signature.asc
Description: PGP signature

Re: Changes to the branching/commit policy

[Christopher Baines]

Christopher Baines  writes:

> The changes in #63459 have strayed now in to touching the commit policy
> [1]. My intent was to simplify the guidance by grouping it better, but I
> think the significant change here is that the commit policy now
> references the entire branching strategy, rather than just talking about
> sending patches for review.
>
> 1: 
> https://guix.gnu.org/manual/devel/en/html_node/Commit-Access.html#Commit-Policy
>
> That new branching strategy makes some "should" requirements on sending
> patches for review and pushing to topic branches for larger changes. It
> also makes a "must" requirement on opening guix-patches issues to track
> and manage merging branches.
>
> I'd like to merge these changes next week since they've been up for a
> few weeks, so do comment if you have any thoughts or if you'd like more
> time to review them.

I've now merged these changes as
0ea096ae23fa81f05ce97e5e61c15647c0a475ec.

You can now see the updated Commit Policy on the website [1] (you might
need to force a refresh), as well as the new section on managing patches
and branches [2].

1: 
https://guix.gnu.org/manual/devel/en/html_node/Commit-Access.html#Commit-Policy
2: 
https://guix.gnu.org/manual/devel/en/html_node/Managing-Patches-and-Branches.html

Thanks,

Chris


signature.asc
Description: PGP signature

Re: Changes to the branching/commit policy

[Christopher Baines]

Andreas Enge  writes:

> thanks for taking up this issue! I agreed with Ludovic's comments, so
> things look good now for me. A very minor point: In the section on
> "trivial" changes, I would drop this sentence (which was already there
> before):
> "This is subject to being adjusted, allowing individuals to commit directly
> on non-controversial changes on parts they’re familiar with."
> The sentence is meaningless, as everything is all the time subject to being
> adjusted; and we do not have immediate plans to adjust it.

My reading of this line is that "adjusted" is probably not the right
word to use, but I think the intent here is to talk about how currently
it's accepted that people can and will push non-controversial changes on
parts they’re familiar with directly to master.

I'm not sure if others read this similarly.


signature.asc
Description: PGP signature

Re: 01/03: gnu: wxwidgets: Add libxtst to inputs.

[Christopher Baines]

Maxim Cournoyer  writes:

> Hi Christopher,
>
> Christopher Baines  writes:
>
>> guix-comm...@gnu.org writes:
>>
>>> apteryx pushed a commit to branch master
>>> in repository guix.
>>>
>>> commit ec9f15b158300da3a77ce02cd2267222f435e80f
>>> Author: Maxim Cournoyer 
>>> AuthorDate: Tue Jun 6 12:12:40 2023 -0400
>>>
>>> gnu: wxwidgets: Add libxtst to inputs.
>>>
>>> WxWidgets was already built with XTest support, but mostly by luck, via
>>> propagation of libxtst from GTK's propagated at-spi2-core package.  
>>> Make it an
>>> explicit input.
>>>
>>> * gnu/packages/wxwidgets.scm (wxwidgets) [inputs]: Add libxtst.
>>> ---
>>>  gnu/packages/wxwidgets.scm | 1 +
>>>  1 file changed, 1 insertion(+)
>>
>> Did this need to go straight to the master branch?
>>
>> → guix refresh -l wxwidgets
>> Building the following 217 packages would ensure 456 dependent packages are 
>> rebuilt: ...
>>
>> That to me says this should go to staging.
>
> Correct.  Except there's no staging branch anymore.  I guess we should
> create one?  :-)

You're the second person to mention this. I guess I view the branch not
existing as a minor technical detail that doesn't really change what
you'd do.

Maybe we need to do more in the guidance to reassure people though, you
should still push to a branch if that's what you should do, even if that
branch doesn't currently exist on the remote. This situation will
probably come up more often in the case of team branches.


signature.asc
Description: PGP signature

Changes to the branching/commit policy

[Christopher Baines]

Hey!

The changes in #63459 have strayed now in to touching the commit policy
[1]. My intent was to simplify the guidance by grouping it better, but I
think the significant change here is that the commit policy now
references the entire branching strategy, rather than just talking about
sending patches for review.

1: 
https://guix.gnu.org/manual/devel/en/html_node/Commit-Access.html#Commit-Policy

That new branching strategy makes some "should" requirements on sending
patches for review and pushing to topic branches for larger changes. It
also makes a "must" requirement on opening guix-patches issues to track
and manage merging branches.

I'd like to merge these changes next week since they've been up for a
few weeks, so do comment if you have any thoughts or if you'd like more
time to review them.

Thanks,

Chris


signature.asc
Description: PGP signature

Re: 01/03: gnu: sbcl: Update to 2.3.5.

[Christopher Baines]

Christopher Baines  writes:

> guix-comm...@gnu.org writes:
>
>> glv pushed a commit to branch master
>> in repository guix.
>>
>> commit b019b49c74e51e42230da471f39bff9f642fbc24
>> Author: Guillaume Le Vaillant 
>> AuthorDate: Fri Jun 2 13:32:55 2023 +0200
>>
>> gnu: sbcl: Update to 2.3.5.
>>
>> * gnu/packages/lisp.scm (sbcl): Update to 2.3.5.
>> ---
>>  gnu/packages/lisp.scm | 4 ++--
>>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> Did this change need to go straight to master?

Joining this thread with a different issue, it looks like this change
has broken sbcl on armhf-linux [1].

1: 
https://data.guix.gnu.org/repository/1/branch/master/package/sbcl/output-history?output=out=armhf-linux=none

Maybe you tested this locally and knew of this breakage, but it would
still be good to send this for review (either as some patches or pushed
to a branch) so that it's visible to others what the effect of the
change will be.


signature.asc
Description: PGP signature

Re: 02/03: gnu: openblas: Update architectures we provide substitutes for.

[Christopher Baines]

Efraim Flashner  writes:

> On Sat, Jun 03, 2023 at 08:12:48PM +0100, Christopher Baines wrote:
>> 
>> Efraim Flashner  writes:
>> 
>> > On Fri, Jun 02, 2023 at 11:03:42PM +0100, Christopher Baines wrote:
>> >> 
>> >> guix-comm...@gnu.org writes:
>> >> 
>> >> > efraim pushed a commit to branch master
>> >> > in repository guix.
>> >> >
>> >> > commit 076688fa1e41a09f034a80e1a593bac43f1f1482
>> >> > Author: Efraim Flashner 
>> >> > AuthorDate: Thu Jun 1 11:06:00 2023 +0300
>> >> >
>> >> > gnu: openblas: Update architectures we provide substitutes for.
>> >> >
>> >> > * gnu/packages/maths.scm (openblas)[arguments]: Adjust the 
>> >> > substitutable?
>> >> > flag to only not provide substitutes when building for 
>> >> > powerpc-linux.
>> >> > Adjust the comment accordingly.
>> >> > ---
>> >> >  gnu/packages/maths.scm | 11 ++-
>> >> >  1 file changed, 2 insertions(+), 9 deletions(-)
>> >> 
>> >> I've been looking at why armhf-linux substitute availability has been
>> >> dropping recently, and I think this change triggered a lot of
>> >> rebuilds. Could this have gone to core-updates?
>> >> 
>> >> → guix refresh -l openblas
>> >> Building the following 2282 packages would ensure 5596 dependent packages 
>> >> are rebuilt: ...
>> >
>> > It's not that it's triggered rebuilds, but that it's triggered builds.
>> > It's also triggered builds on powerpc64le and riscv64. Before any
>> > package which had openblas as a transitive dependency wasn't built by
>> > the CI because it wasn't substitutable¹. People still have the option of
>> > using package transformations to use openblas tuned for the cortex a7 or
>> > a15 on armhf, but in reality this just unlocks substitutes for those
>> > ~5600 packages which wasn't available before.
>> >
>> > ¹ We saw this in the past briefly in the past when openzfs made its way
>> > as a dependency to qemu and through that to Gnome.
>> 
>> Ok, so the documentation does mention "rebuilding", and I do see that
>> indeed ci.guix.gnu.org doesn't build not substitutable things.
>> 
>> Although I think it doesn't apply recursively. Take qjson, guix refresh
>> -l tells me it's dependent on openblas, and looking back at say this
>> output [1] for powerpc64le-linux, that's available from both
>> ci.guix.gnu.org. Which makes sense, as that derivation is substitutable,
>> even though one of it's inputs isn't.
>> 
>> 1: 
>> https://data.guix.gnu.org/gnu/store/fibiwzyz8s899ccpix5zs6r2pcdpxk5b-qjson-0.9.0
>> 
>> Maybe on the client side this works differently, and guix won't
>> substitute things which have a non substitutable input?
>> 
>> Assuming ci.guix.gnu.org was building things for armhf-linux, I think
>> this would have still caused ~5596 rebuilds, and as I say, I think for
>> systems like powerpc64le-linux, I think it did cause ~5596
>> rebuilds.
>
> I looked into it more. First I ran on master:
> './pre-inst-env guix build --no-grafts --system=armhf-linux openblas -d'
> /gnu/store/whi4yhiw2b0c0i3n6l8s0qfcphkvbzg4-openblas-0.3.20.drv
>
> Then I locally reverted the patch expanding the architectures where we
> provided substitutes:
> /gnu/store/1m57z8jkbf6gz7qlbw3ws4ayl0ln9602-openblas-0.3.20.drv
>
> Then I locally reverted the patch adjusting the make-flags:
> /gnu/store/1m57z8jkbf6gz7qlbw3ws4ayl0ln9602-openblas-0.3.20.drv
>
> It seems I was wrong, changing the #:substitutable? flag _does_ change
> the derivation of the package. I also checked the substitutes and saw
> that bordeaux did (and does) have substitutes for the non-substitutable
> version, showing that it was built before. I also checked for
> powerpc64le, to see if perhaps cuirass worked a different way and
> honored the #:substitutable flag by not building it there, and it too
> has substitutes for both versions of openblas.
>
> I didn't check for riscv64 but I assume the case is the same with a
> changing derivation.
>
> So obviously if I had realized this would cause ~5596 rebuilds per
> affected arch I wouldn't have pushed the patch. I should've checked the
> derivation before and after to make sure it didn't change.

Thanks for investigating further.

There's maybe a secondary issue here about why changing the
substitutibility of a package affects it's outputs. That doesn't make
much sense to me, but maybe there's a reason.


signature.asc
Description: PGP signature

Re: 01/03: gnu: sbcl: Update to 2.3.5.

[Christopher Baines]

Guillaume Le Vaillant  writes:

> Christopher Baines  skribis:
>
>> guix-comm...@gnu.org writes:
>>
>>> glv pushed a commit to branch master
>>> in repository guix.
>>>
>>> commit b019b49c74e51e42230da471f39bff9f642fbc24
>>> Author: Guillaume Le Vaillant 
>>> AuthorDate: Fri Jun 2 13:32:55 2023 +0200
>>>
>>> gnu: sbcl: Update to 2.3.5.
>>>
>>> * gnu/packages/lisp.scm (sbcl): Update to 2.3.5.
>>> ---
>>>  gnu/packages/lisp.scm | 4 ++--
>>>  1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> Did this change need to go straight to master?
>>
>> → guix refresh -l sbcl
>> Building the following 314 packages would ensure 1615 dependent packages are 
>> rebuilt: ...
>>
>> Above 300 but below 1800 means staging according to [1].
>>
>> 1: 
>> https://guix.gnu.org/manual/devel/en/html_node/Submitting-Patches.html#index-rebuild-scheduling-strategy
>
> Currently there is no staging branch, as it got deleted because of the
> changes in the branching strategy ([1]). If someone creates a staging
> branch, will cuirass see that and start building it? Or could it just
> stay in limbo for months?

Deleting the staging branch when it's merged seems sensible to me
regardless of whether you want to use that branch in the future. As far
as I'm concerned, while there have been disucssions of changing the
branching strategy, the guidance as written down hasn't changed yet
(there are some suggested changes in [1]).

The changes in [1] still keep the 300 dependent limit on pushing changes
directly to master though.

As for the Cuirass configuration, it might need changing. You can either
ask on guix-devel/guix-sysadmin for someone to do this, or ask on
guix-sysadmin to get access to Cuirass so that you can use the admin
interface yourself [2].

2:  https://lists.gnu.org/archive/html/guix-devel/2023-06/msg9.html

> Concerning the specific case of sbcl dependents, they build pretty fast.
> On my machine, building all of them took less than one hour. So the
> period during which some substitutes are missing is pretty short.
>
> [1] https://issues.guix.gnu.org/63459

That's still time people will be without substitutes, and because these
changes in effect "jumped the queue", that's also time the build farm
won't be building changes from patches or the branch(s) that it's meant
to be working on (since the master branch builds have priority).


signature.asc
Description: PGP signature

Re: 01/03: gnu: sbcl: Update to 2.3.5.

[Christopher Baines]

guix-comm...@gnu.org writes:

> glv pushed a commit to branch master
> in repository guix.
>
> commit b019b49c74e51e42230da471f39bff9f642fbc24
> Author: Guillaume Le Vaillant 
> AuthorDate: Fri Jun 2 13:32:55 2023 +0200
>
> gnu: sbcl: Update to 2.3.5.
>
> * gnu/packages/lisp.scm (sbcl): Update to 2.3.5.
> ---
>  gnu/packages/lisp.scm | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)

Did this change need to go straight to master?

→ guix refresh -l sbcl
Building the following 314 packages would ensure 1615 dependent packages are 
rebuilt: ...

Above 300 but below 1800 means staging according to [1].

1: 
https://guix.gnu.org/manual/devel/en/html_node/Submitting-Patches.html#index-rebuild-scheduling-strategy


signature.asc
Description: PGP signature

Re: branch master updated: gnu: sbcl: fix build on riscv64-linux.

[Christopher Baines]

guix-comm...@gnu.org writes:

> This is an automated email from the git hooks/post-receive script.
>
> efraim pushed a commit to branch master
> in repository guix.
>
> The following commit(s) were added to refs/heads/master by this push:
>  new 2a6d2fc1d8 gnu: sbcl: fix build on riscv64-linux.
> 2a6d2fc1d8 is described below
>
> commit 2a6d2fc1d8e3434e283d6d2e559529b41a242fea
> Author: Efraim Flashner 
> AuthorDate: Wed Jun 7 09:55:35 2023 +0300
>
> gnu: sbcl: fix build on riscv64-linux.
>
> * gnu/packages/patches/sbcl-riscv-Make-contribs-build-again.patch: New
> file.
> * gnu/local.mk(dist_patch_DATA): register it.
> * gnu/packages/lisp.scm (sbcl): [source]: Use it here.
> ---
>  gnu/local.mk   |  1 +
>  gnu/packages/lisp.scm  |  3 +
>  .../sbcl-riscv-Make-contribs-build-again.patch | 71 
> ++
>  3 files changed, 75 insertions(+)

Did this change need to go straight to master?

→ guix refresh -l sbcl
Building the following 314 packages would ensure 1615 dependent packages are 
rebuilt: ...

To me that says to be that this would be appropriate for staging [1].

1: 
https://guix.gnu.org/manual/devel/en/html_node/Submitting-Patches.html#index-rebuild-scheduling-strategy

Maybe this would be OK if the patch was applied in such a way that it
only affects riscv64-linux, but it isn't, so this affects all systems.


signature.asc
Description: PGP signature

Re: 01/03: gnu: wxwidgets: Add libxtst to inputs.

[Christopher Baines]

guix-comm...@gnu.org writes:

> apteryx pushed a commit to branch master
> in repository guix.
>
> commit ec9f15b158300da3a77ce02cd2267222f435e80f
> Author: Maxim Cournoyer 
> AuthorDate: Tue Jun 6 12:12:40 2023 -0400
>
> gnu: wxwidgets: Add libxtst to inputs.
>
> WxWidgets was already built with XTest support, but mostly by luck, via
> propagation of libxtst from GTK's propagated at-spi2-core package.  Make 
> it an
> explicit input.
>
> * gnu/packages/wxwidgets.scm (wxwidgets) [inputs]: Add libxtst.
> ---
>  gnu/packages/wxwidgets.scm | 1 +
>  1 file changed, 1 insertion(+)

Did this need to go straight to the master branch?

→ guix refresh -l wxwidgets
Building the following 217 packages would ensure 456 dependent packages are 
rebuilt: ...

That to me says this should go to staging.


signature.asc
Description: PGP signature

The weather (substitute availability) is looking good today

[Christopher Baines]

Hey!

Since the core-updates merge, substitute availability has been quite
changeable. This has been more due to the large changes merged since the
core-updates merge than the changes in core-updates itself.

Substitute availability was quite high for a brief period around a
couple of weeks ago, but then there were more unexpected large changes
that caused it to drop.

You can see an overview of the substitute availability for various
systems here [1]. This page uses data from data.qa.guix.gnu.org [2].

1: https://qa.guix.gnu.org/branch/master
2: 
https://data.qa.guix.gnu.org/repository/2/branch/master/latest-processed-revision/package-substitute-availability

My hope is still that we can make small (and large) changes without
negatively affecting the substitute availability. For small changes,
qa.guix.gnu.org is back building packages affected by patches. For
larger changes, I've got some proposed changes to the process here [3]
and qa.guix.gnu.org is currently working on building the tex-team branch
[4][5].

3: https://issues.guix.gnu.org/63459
4: https://issues.guix.gnu.org/63521
5: https://qa.guix.gnu.org/branch/tex-team

Thanks,

Chris


signature.asc
Description: PGP signature

Re: 02/03: gnu: openblas: Update architectures we provide substitutes for.

[Christopher Baines]

Efraim Flashner  writes:

> On Fri, Jun 02, 2023 at 11:03:42PM +0100, Christopher Baines wrote:
>> 
>> guix-comm...@gnu.org writes:
>> 
>> > efraim pushed a commit to branch master
>> > in repository guix.
>> >
>> > commit 076688fa1e41a09f034a80e1a593bac43f1f1482
>> > Author: Efraim Flashner 
>> > AuthorDate: Thu Jun 1 11:06:00 2023 +0300
>> >
>> > gnu: openblas: Update architectures we provide substitutes for.
>> >
>> > * gnu/packages/maths.scm (openblas)[arguments]: Adjust the 
>> > substitutable?
>> > flag to only not provide substitutes when building for powerpc-linux.
>> > Adjust the comment accordingly.
>> > ---
>> >  gnu/packages/maths.scm | 11 ++-
>> >  1 file changed, 2 insertions(+), 9 deletions(-)
>> 
>> I've been looking at why armhf-linux substitute availability has been
>> dropping recently, and I think this change triggered a lot of
>> rebuilds. Could this have gone to core-updates?
>> 
>> → guix refresh -l openblas
>> Building the following 2282 packages would ensure 5596 dependent packages 
>> are rebuilt: ...
>
> It's not that it's triggered rebuilds, but that it's triggered builds.
> It's also triggered builds on powerpc64le and riscv64. Before any
> package which had openblas as a transitive dependency wasn't built by
> the CI because it wasn't substitutable¹. People still have the option of
> using package transformations to use openblas tuned for the cortex a7 or
> a15 on armhf, but in reality this just unlocks substitutes for those
> ~5600 packages which wasn't available before.
>
> ¹ We saw this in the past briefly in the past when openzfs made its way
> as a dependency to qemu and through that to Gnome.

Ok, so the documentation does mention "rebuilding", and I do see that
indeed ci.guix.gnu.org doesn't build not substitutable things.

Although I think it doesn't apply recursively. Take qjson, guix refresh
-l tells me it's dependent on openblas, and looking back at say this
output [1] for powerpc64le-linux, that's available from both
ci.guix.gnu.org. Which makes sense, as that derivation is substitutable,
even though one of it's inputs isn't.

1: 
https://data.guix.gnu.org/gnu/store/fibiwzyz8s899ccpix5zs6r2pcdpxk5b-qjson-0.9.0

Maybe on the client side this works differently, and guix won't
substitute things which have a non substitutable input?

Assuming ci.guix.gnu.org was building things for armhf-linux, I think
this would have still caused ~5596 rebuilds, and as I say, I think for
systems like powerpc64le-linux, I think it did cause ~5596
rebuilds.


signature.asc
Description: PGP signature

Re: 02/03: gnu: openblas: Update architectures we provide substitutes for.

[Christopher Baines]

guix-comm...@gnu.org writes:

> efraim pushed a commit to branch master
> in repository guix.
>
> commit 076688fa1e41a09f034a80e1a593bac43f1f1482
> Author: Efraim Flashner 
> AuthorDate: Thu Jun 1 11:06:00 2023 +0300
>
> gnu: openblas: Update architectures we provide substitutes for.
>
> * gnu/packages/maths.scm (openblas)[arguments]: Adjust the substitutable?
> flag to only not provide substitutes when building for powerpc-linux.
> Adjust the comment accordingly.
> ---
>  gnu/packages/maths.scm | 11 ++-
>  1 file changed, 2 insertions(+), 9 deletions(-)

I've been looking at why armhf-linux substitute availability has been
dropping recently, and I think this change triggered a lot of
rebuilds. Could this have gone to core-updates?

→ guix refresh -l openblas
Building the following 2282 packages would ensure 5596 dependent packages are 
rebuilt: ...


signature.asc
Description: PGP signature

Re: Do substitutes download slowly for you? / Speeding up substitute delivery/mirrors

[Christopher Baines]

Luis Felipe  writes:

> Hi Chris,
>
> El 31/05/23 a las 10:48, Christopher Baines escribió:
>> Luis Felipe  writes:
>>
>>> Another (faster) test from a different machine with Guix System 020184f,
>>> same place and network (60 Mbps):
>> Thanks for the information, although it looks like you're not well
>> connected to any of the three machines.
>>
>> The Singapore mirror is hosted by Vultr, and they do have some locations
>> closer to you, I'd be interested to see what speeds you get downloading
>> these files?
>>
>> https://sao-br-ping.vultr.com/vultr.com.100MB.bin
>> https://mex-mx-ping.vultr.com/vultr.com.100MB.bin
>> https://scl-cl-ping.vultr.com/vultr.com.100MB.bin
>
> It feels faster downloading from those locations.

Good to know, so to summarise a US mirror is helpful for some people at
least in South America, but closer mirrors help more (with Mexico being
the best in this test).


signature.asc
Description: PGP signature

Re: Do substitutes download slowly for you? / Speeding up substitute delivery/mirrors

[Christopher Baines]

Luis Felipe  writes:

> Another (faster) test from a different machine with Guix System 020184f, 
> same place and network (60 Mbps):

Thanks for the information, although it looks like you're not well
connected to any of the three machines.

The Singapore mirror is hosted by Vultr, and they do have some locations
closer to you, I'd be interested to see what speeds you get downloading
these files?

https://sao-br-ping.vultr.com/vultr.com.100MB.bin
https://mex-mx-ping.vultr.com/vultr.com.100MB.bin
https://scl-cl-ping.vultr.com/vultr.com.100MB.bin


signature.asc
Description: PGP signature

Re: branch master updated: gnu: eudev: Use new package style.

[Christopher Baines]

Liliana Marie Prikler  writes:

> Am Montag, dem 29.05.2023 um 20:29 +0100 schrieb Christopher Baines:
>> 1:
>> https://guix.gnu.org/manual/devel/en/html_node/Submitting-Patches.html#index-rebuild-scheduling-strategy
>>
>> As for reverting it, I'm somewhat indifferent. I'm more interested in
>> the longer term cost of making changes like this than the temporary
>> drops in substitute availability.
>
> You mean as a precedent for similar commits in the future or as a way
> of involuntarily breaking other packages?  As already stated, I only
> pushed the commit because I was quite sure that all rebuilds would
> succeed.

I'm more thinking of "cost" to do with capacity on the build farm. Even
though nothing should fail to build, the builds still need to happen,
and there are other builds that will be delayed.


signature.asc
Description: PGP signature

Re: [PATCH] web: Include blocked-by in graphql .

[Christopher Baines]

Arun Isaac  writes:

> Thanks for the patch! I have pushed it after renaming blocked-by to
> blocked_by. Underscores are a bit cringey, but unfortunately GraphQL
> doesn't like hyphens.
>
> To see the effects of this change, we need to reconfigure berlin to run
> the latest mumi. I will figure that out and do it soon.

Awesome, thanks for fixing my rough untested changes!

Chris


signature.asc
Description: PGP signature

Re: [PATCH] web: Include blocked-by in graphql .

[Christopher Baines]

Christopher Baines  writes:

> * mumi/web/graphql.scm (): Include blocked-by.
> ---
>  mumi/web/graphql.scm | 5 -
>  1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/mumi/web/graphql.scm b/mumi/web/graphql.scm
> index aa4f38c..3cc19cd 100644
> --- a/mumi/web/graphql.scm
> +++ b/mumi/web/graphql.scm
> @@ -63,7 +63,10 @@
>(list
>(messages (non-nullable-type (list-type ))
>  (lambda (parent . _)
> -  (issue-messages (bug-num parent)
> +  (issue-messages (bug-num parent
> +  (blocked-by (non-nullable-type (list-type ))
> +  (lambda (parent . _)
> +(map bug-status (bug-blockedby parent)
>  
>  (define-object-type 
>(name  (lambda (parent . _)

I'm looking at this so that the qa-frontpage can see what issues are
blocked.

I'm not sure how to test this though.


signature.asc
Description: PGP signature

[PATCH] web: Include blocked-by in graphql .

[Christopher Baines]

* mumi/web/graphql.scm (): Include blocked-by.
---
 mumi/web/graphql.scm | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/mumi/web/graphql.scm b/mumi/web/graphql.scm
index aa4f38c..3cc19cd 100644
--- a/mumi/web/graphql.scm
+++ b/mumi/web/graphql.scm
@@ -63,7 +63,10 @@
   (list
   (messages (non-nullable-type (list-type ))
 (lambda (parent . _)
-  (issue-messages (bug-num parent)
+  (issue-messages (bug-num parent
+  (blocked-by (non-nullable-type (list-type ))
+  (lambda (parent . _)
+(map bug-status (bug-blockedby parent)
 
 (define-object-type 
   (name  (lambda (parent . _)
-- 
2.40.1

Re: branch master updated: gnu: eudev: Use new package style.

[Christopher Baines]

Liliana Marie Prikler  writes:

> Am Montag, dem 29.05.2023 um 19:28 +0100 schrieb Christopher Baines:
>> 
>> guix-comm...@gnu.org writes:
>> 
>> > This is an automated email from the git hooks/post-receive script.
>> > 
>> > lilyp pushed a commit to branch master
>> > in repository guix.
>> > 
>> > The following commit(s) were added to refs/heads/master by this
>> > push:
>> >  new 7ff003bcbf gnu: eudev: Use new package style.
>> > 7ff003bcbf is described below
>> > 
>> > commit 7ff003bcbf388677c7c85b1709c58f41f84b9947
>> > Author: Felix Lechner 
>> > AuthorDate: Sun May 28 16:28:20 2023 -0700
>> > 
>> >     gnu: eudev: Use new package style.
>> >     
>> >     * gnu/packages/linux.scm (eudev)[arguments]: Convert to list of
>> > G-Expressions.
>> >     [native-inputs]: Drop labels.
>> >     
>> >     Signed-off-by: Liliana Marie Prikler
>> > 
>> > ---
>> >  gnu/packages/linux.scm | 94 --
>> > 
>> >  1 file changed, 45 insertions(+), 49 deletions(-)
>> 
>> These changes do look to affect the derivation for eudev, and eudev
>> also has too many dependents to update on the master branch.
>> 
>> → guix refresh -l eudev
>> Building the following 1913 packages would ensure 4138 dependent
>> packages are rebuilt: ...
> Should I revert it?  Even with the mass rebuild, my intuition was that
> it'd be okay since it's no functional change and core-updates was
> dropped in favour of teams (with no particular team being responsible
> for udev afaik).

I think there is disagreement about this (including at least some
maintainers thinking differently), but I'm of the opinion that while
there has been discussion about stopping using core-updates, people
should follow the currently documented process, and for now that's still
using staging/core-updates [1].

1: 
https://guix.gnu.org/manual/devel/en/html_node/Submitting-Patches.html#index-rebuild-scheduling-strategy

As for reverting it, I'm somewhat indifferent. I'm more interested in
the longer term cost of making changes like this than the temporary
drops in substitute availability.


signature.asc
Description: PGP signature