Branch and release process (was: gnu: inetutils: Update to 2.4.)

[Maxim Cournoyer]

Hi,

Leo Famulari  writes:

> On Tue, Mar 14, 2023 at 09:10:33PM -0400, Maxim Cournoyer wrote:
>> Felix Lechner  writes:
>> > With the core-updates process now abandoned, I retitled the issue to
>> 
>> Could you share the reference of that?  I'm not against it, but our
>> currently documented process still mention the good old staging and
>> core-updates branches.
>
> At the Guix Days in February, we discussed the branching workflow and
> reached a rough consensus that for non-core packages (defined in
> %core-packages), we should try to adopt a more targeted "feature branch"
> workflow. That's actually what we used to do, before we outgrew our old
> build farm, after which we were barely able to build one branch at a
> time (IIRC, we would stop building master in order to build core-updates
> or staging).
>
> The discussion was summarized by Andreas here:
>
> https://lists.gnu.org/archive/html/guix-devel/2023-02/msg00066.html

Thanks!  I had missed it.  It sounds promising!

> Currently we are demo-ing this workflow in the wip-go-updates branch and
> go-team Cuirass jobset.

So the review happens first on the ML, then the changes land to the team
branch, and then finally the feature branch gets merged to master?  If
the review has already happened and the package been tested (and built
by QA), why is a feature branch needed?

> My hope is that we can rewrite the relevant documentation in the coming
> months, as we learn from these early efforts.

OK!  Thanks for allowing me to catch up!

-- 
Thanks,
Maxim

Re: gnu: inetutils: Update to 2.4.

[Leo Famulari]

On Tue, Mar 14, 2023 at 09:10:33PM -0400, Maxim Cournoyer wrote:
> Felix Lechner  writes:
> > With the core-updates process now abandoned, I retitled the issue to
> 
> Could you share the reference of that?  I'm not against it, but our
> currently documented process still mention the good old staging and
> core-updates branches.

At the Guix Days in February, we discussed the branching workflow and
reached a rough consensus that for non-core packages (defined in
%core-packages), we should try to adopt a more targeted "feature branch"
workflow. That's actually what we used to do, before we outgrew our old
build farm, after which we were barely able to build one branch at a
time (IIRC, we would stop building master in order to build core-updates
or staging).

The discussion was summarized by Andreas here:

https://lists.gnu.org/archive/html/guix-devel/2023-02/msg00066.html

Currently we are demo-ing this workflow in the wip-go-updates branch and
go-team Cuirass jobset.

My hope is that we can rewrite the relevant documentation in the coming
months, as we learn from these early efforts.

I don't think the core-updates process is abandoned, but we should
reduce its scope. For the core of Guix, both the packages and Guix
itself, it makes sense to alter things in tandem.

But as we have >22000 packages, there are many packages that would
currently qualify for core-updates but aren't core, and can be
relatively easily tested in smaller themed batches.

I would suggest abandoning the staging branch approach after its current
patches are merged to master. The staging branch was always a kludge
from when our build farm was struggling.

For something like inetutils, I would suggest borrowing from its package
module name (gnu packages admin), and attempt to update the
administrative packages as a group. Those who are interested in system
administration should lead the effort.

Re: gnu: inetutils: Update to 2.4.

[Maxim Cournoyer]

Hi Felix,

Felix Lechner  writes:

> Hi Maxim,
>
> On Tue, Mar 14, 2023 at 8:53 AM Maxim Cournoyer
>  wrote:
>>
>> In the future, try to remember using the --subject-prefix='PATCH
>> core-updates', to avoid committers mistakenly pushing this to master
>
> Thank you for your response! I wrote for guidance on that issue. I am
> sorry that was unclear from my message.
>
> With the core-updates process now abandoned, I retitled the issue to

Could you share the reference of that?  I'm not against it, but our
currently documented process still mention the good old staging and
core-updates branches.

> "[PATCH wip-inetutils 0/4] gnu: inetutils: Update to 2.4".
>
> Does that work for you?
>
> I am happy to retitle as "core-updates" as you suggested, if that's
> what you want. Of course, you can also retitle it in any way you think
> is best.

Until everybody has a good grasp of the new process, I think sticking to
the documented one may be best for now, as it makes it clear that this
cause a mass rebuild and shouldn't land to master.

-- 
Thanks,
Maxim

Re: State of core-updates

[Maxim Cournoyer]

Hi Andreas,

Andreas Enge  writes:

> Hello Maxim,
>
> Am Tue, Mar 14, 2023 at 11:50:12AM -0400 schrieb Maxim Cournoyer:
>> That looks promising.  Should we spun a differently named branch to
>> avoid people sending core-updates change?  This was discussed in the
>> past and agreed to (main branches do not *freeze* themselves), instead
>> we use git to branch to our will.
>
> well, I consider core-updates to be frozen, people should not send any
> more patches there unless they repair things that are currently broken
> (and, one may add, a regression to master). I do not expect this to
> include any world rebuilding changes any more.
>
> And then the goal will be to not have a core-updates branch in the future,
> but separate feature branches as discussed at the Guix days.
>
> So the aim is to merge core-updates to master, and then to delete this
> branch once and for all. (Of course, there can then be a new feature
> branch "core-team" or something like this, for changes concerning the
> core of the system.)

Thanks for the explanations.  I'm out of the loop, not having been able
to attend physically the last Guix Days event.  Was there a recap of the
discussions posted somewhere?  Was the freeze announce somewhere?  I
wholly missed that, and I try to pay attention.

> So I think there is no need to branch from the branch!
> (And just as a reminder, let us not forget the staging branch that needs
> a similar treatment.)

OK!  We could probably merge staging into master and be done already.

-- 
Thanks,
Maxim

Re: State of core-updates

[Andreas Enge]

Am Mon, Mar 13, 2023 at 03:14:09PM +0100 schrieb Simon Tournier:
> Well, maybe it could be helpful if now Berlin or Bordeaux starts to
> build etc/release-manifest.scm.  WDYT?

That sounds like a great next step. Someone just has to do it™...

Andreas

Re: Core-updates and cross-compilation

[Andreas Enge]

Hello Josselin,

Am Tue, Mar 14, 2023 at 06:48:16PM +0100 schrieb Josselin Poiret:
> I'm working on building Hurd atm, and it's been taking longer than I
> would've liked.  We needed an update to mig, which cascaded into updates
> to gnumach and hurd, but they also shouldn't be too recent as we'd need
> a newer libc.  I thought the git tags in the repos would be a safe bet,
> but they don't build by themselves, and lots of patches need to be
> backported as well :) maybe after this upgrade is over we could offer
> some help with releases/CI.

thanks for your update and working on the hurd integration! And many thanks
also for trying to find a core-updates friendly solution... If it turns out
to be too much work, maybe you could also postpone it until after the merge
and try to get a feature branch going quickly thereafter.

Suggesting help to the hurd people concerning CI sounds like it could be
beneficial to both our communities.

Andreas

Re: gnu: inetutils: Update to 2.4.

[Andreas Enge]

Hello Felix,

Am Tue, Mar 14, 2023 at 09:37:23AM -0700 schrieb Felix Lechner via Development 
of GNU Guix and the GNU System distribution.:
> With the core-updates process now abandoned, I retitled the issue to
> "[PATCH wip-inetutils 0/4] gnu: inetutils: Update to 2.4".

apologies for not replying earlier - this was because I did not have the
answer... I think this is a place where the feature branch idea is not yet
fully thought out. It is quite clear how branches closer to the leaves
could work (language branches, a qt/kde branch, such things); and probably
also "core things". There I see two kinds of cores: First, things that are
very close to Guile/Guix (switching to gexps, rewriting the daemon in Guile,
these kind of things). And maybe something more like "bootstrap" and "tool-
chain" (changing the Guile version, updating gcc/mpfr/mpc, probably rust,
and so on).

I do not yet have a clear picture for things on a "middle level", but which
cause a lot of rebuilds. Which kind of team would be responsible for
inetutils?

Suggestions would be very welcome!

Something completely different: I am not sure whether inetutils should have
so many dependencies. It only contains binaries and no libraries. For
instance, it is sometimes used as native input for tests, but tests do not
have access to the inet anyway. So maybe it would make sense to identify
the "root" where this package is pulled in, drop the input and modify
the test suite?

Andreas

Re: State of core-updates

[Andreas Enge]

Hello Maxim,

Am Tue, Mar 14, 2023 at 11:50:12AM -0400 schrieb Maxim Cournoyer:
> That looks promising.  Should we spun a differently named branch to
> avoid people sending core-updates change?  This was discussed in the
> past and agreed to (main branches do not *freeze* themselves), instead
> we use git to branch to our will.

well, I consider core-updates to be frozen, people should not send any
more patches there unless they repair things that are currently broken
(and, one may add, a regression to master). I do not expect this to
include any world rebuilding changes any more.

And then the goal will be to not have a core-updates branch in the future,
but separate feature branches as discussed at the Guix days. So the aim
is to merge core-updates to master, and then to delete this branch once
and for all. (Of course, there can then be a new feature branch "core-team"
or something like this, for changes concerning the core of the system.)

So I think there is no need to branch from the branch!
(And just as a reminder, let us not forget the staging branch that needs
a similar treatment.)

Andreas

Re: Core-updates and cross-compilation

[Josselin Poiret]

-- 

Andreas Enge  writes:

> Hello,
>
> Am Sat, Mar 11, 2023 at 01:56:27PM +0100 schrieb Josselin Poiret:
>> I've been looking at the state of most failures for the CI jobset for
>> core-updates, and we have a couple of problems:
>> - gcc < 9 and gcc == 12 never cross-compile.
>> - we can't build the cross toolchain for the hurd, because the glibc
>>   upgrade to 2.35 would require newer gnumach headers
>
> are these new issues compared to master? There we also have gcc@9 and @12,
> so my guess would be no, at least for the first one.

The first one no, the second one yes.

>> Should we consider these blockers for a core-updates merge?
>
> If the situation is not worse than on master, my answer will be a firm "no".
> Otherwise I am less sure. I think we might still want to merge core-updates
> first and handle these cross compilation issues in a later feature branch
> of the core team.

I'm working on building Hurd atm, and it's been taking longer than I
would've liked.  We needed an update to mig, which cascaded into updates
to gnumach and hurd, but they also shouldn't be too recent as we'd need
a newer libc.  I thought the git tags in the repos would be a safe bet,
but they don't build by themselves, and lots of patches need to be
backported as well :) maybe after this upgrade is over we could offer
some help with releases/CI.

Best,
-- 
Josselin Poiret


signature.asc
Description: PGP signature

Re: gnu: inetutils: Update to 2.4.

[Development of GNU Guix and the GNU System distribution.]

Hi Maxim,

On Tue, Mar 14, 2023 at 8:53 AM Maxim Cournoyer
 wrote:
>
> In the future, try to remember using the --subject-prefix='PATCH
> core-updates', to avoid committers mistakenly pushing this to master

Thank you for your response! I wrote for guidance on that issue. I am
sorry that was unclear from my message.

With the core-updates process now abandoned, I retitled the issue to

"[PATCH wip-inetutils 0/4] gnu: inetutils: Update to 2.4".

Does that work for you?

I am happy to retitle as "core-updates" as you suggested, if that's
what you want. Of course, you can also retitle it in any way you think
is best.

Kind regards,
Felix Lechner

Re: bug#61894: [PATCH RFC] Team approval for patches

[Maxim Cournoyer]

Hi,

Peter Polidoro  writes:

> There is a phenomenon in manufacturing quality control where sometimes
> adding inspectors decreases the number of defects that get past
> inspection unnoticed, because one inspector catches a defect that
> another inspector missed, but other times the number of unnoticed
> defects actually goes UP, presumably because if inspectors know others
> are also looking for defects, they, perhaps subconciously, think they
> do not need to look as carefully, because another inspector will catch
> whatever they miss. One inspector looking carefully can be better than
> two inspectors looking less carefully.

Haha!  That seems very human.

> It would be nice if packages that pull from a "trusted source" and
> that need only a bump in the version number and hash could be approved
> by only one person or, more ideally, zero people, if it could be
> tested and automated somehow. Although perhaps that would always be a
> security risk.

That'd be cool.  I think it's not too far fetched that in the future
this may be possible with the QA tooling.

> Is there documentation or a roadmap somewhere online for people new
> the community who submit patches, but someday aspire to arise to
> committer status? The roadmap might be a list of books to read,
> tutorials to complete, packages to create, in order to learn enough to
> be able to help with the committer shortage?

There are some tips in the manual: info '(guix) Commit Access', which
reads like:

Everyone can contribute to Guix without having commit access (*note
Submitting Patches::).  However, for frequent contributors, having write
access to the repository can be convenient.  As a rule of thumb, a
contributor should have accumulated fifty (50) reviewed commits to be
considered as a committer and have sustained their activity in the
project for at least 6 months.  This ensures enough interactions with
the contributor, which is essential for mentoring and assessing whether
they are ready to become a committer.  Commit access should not be
thought of as a “badge of honor” but rather as a responsibility a
contributor is willing to take to help the project.

The most important part in my opinion is having been around long enough
to have had enough interactions to gain the trust of the other
participants, and shown a rationale, positive response to feedback
provided (which can admittedly be difficult at times!).

-- 
Thanks,
Maxim

Disarchive database synchronization

[Ludovic Courtès]

Hello Guix!

As you may know, there are currently two different Disarchive databases:
the one at  that Timothy Sample set up a
few years back, and the one at  that we
set up later, with a continuous integration job to populate it¹.

The database at ngyro.com has more historical metadata (metadata about
tarballs that older Guix revisions referred to) because Timothy worked
hard to populate it with tarballs from all the packages Guix refers to
starting from 1.0—which is crucial for long-term reproducibility.

Thanks to Timothy, I have now copied over things from
disarchive.ngyro.com to disarchive.guix.gnu.org.  The stats are as
follows:

  disarchive.ngyro.com had 28,396 entries
  12,905 (45%) entries were missing from disarchive.guix.gnu.org
  15,491 (the rest: 55%) entries were present in both yet different.
  3,444 entries of disarchive.guix were missing from disarchive.ngyro²

I copied over the 12K entries that were missing from
disarchive.guix.gnu.org.  (Note that there are currently only two copies
of the database: one at/in [bB]erlin, and one at/in [Bb]ordeaux.)
disarchive.guix.gnu.org now weighs in at 1.8 GiB for 31,839 entries.

For the remaining entries, it’s trickier.  Sometimes it’s just the
gzip compression parameters that differ, which could be addressed with a
little bit more work:

--8<---cut here---start->8---
$ file ffdc77f5e5cb2390b9309de63eb7be68d9fe631e898f4da6c04a8159daefc2c0.gz 
../../disarchive/sha256/ffdc77f5e5cb2390b9309de63eb7be68d9fe631e898f4da6c04a8159daefc2c0.gz
ffdc77f5e5cb2390b9309de63eb7be68d9fe631e898f4da6c04a8159daefc2c0.gz:
 gzip compressed data, max compression, from Unix, original size 
modulo 2^32 446731
../../disarchive/sha256/ffdc77f5e5cb2390b9309de63eb7be68d9fe631e898f4da6c04a8159daefc2c0.gz:
 gzip compressed data, max speed, from Unix, original size modulo 2^32 446731
--8<---cut here---end--->8---

Sometimes it’s trickier:

# diff -u <(gunzip -d < 0001f025c1425ffe36270a81cb091eade87dd8d29ac773735ae47e1a8c8066c9.gz) <(gunzip -d < ../../disarchive/sha256/0001f025c1425ffe36270a81cb091eade87dd8d29ac773735ae47e1a8c8066c9.gz)
--- /dev/fd/63  2023-03-14 16:13:21.635733426 +0100
+++ /dev/fd/62  2023-03-14 16:13:21.635733426 +0100
@@ -1,7 +1,7 @@
 (disarchive
   (version 0)
   (gzip-member
-(name "webview-sys-0.6.2.tar.gz")
+(name "rust-webview-sys-0.6.2.tar.gz")
 (digest
   (sha256
 "0001f025c1425ffe36270a81cb091eade87dd8d29ac773735ae47e1a8c8066c9"))
@@ -13,7 +13,7 @@
 (footer (crc 1807070134) (isize 121344))
 (compressor zlib-best)
 (input (tarball
- (name "webview-sys-0.6.2.tar")
+ (name "rust-webview-sys-0.6.2.tar")
  (digest
(sha256
  "4fb18f3206838e11f7f8caba6fad9e0f796109428b502793b9f2f0613fe0f275"))
@@ -78,7 +78,7 @@
  (padding 0)
  (input (directory-ref
   (version 0)
-  (name "webview-sys-0.6.2")
+  (name "rust-webview-sys-0.6.2")
   (addresses
 (swhid "swh:1:dir:fa41df38bf639ada28c900b0915661e787fe6d15"))
   (digest

As Tim pointed out, Disarchive disassembly is not fully deterministic
and/or might change a bit over time as Disarchive evolves, and that’s
prolly what we’re seeing here.

The admins among us can see the remaining files in
/gnu/disarchive.ngyro.com on berlin.  That directory also contains two
files: ‘files-present-in-both-yet-different.txt’ and
‘files-that-were-missing.txt’.

Kudos to Timothy for making it possible.

Feedback welcome!

Ludo’.

¹ https://lists.gnu.org/archive/html/guix-devel/2021-10/msg00060.html

² Some of these showed up at disarchive.ngyro.com since I copied the
  database ~16h ago.  Example missing entry is “samplv1-0.9.24.tar.gz”:
  
.


signature.asc
Description: PGP signature

Re: gnu: inetutils: Update to 2.4.

[Maxim Cournoyer]

Hi Felix,

Felix Lechner via "Development of GNU Guix and the GNU System
distribution."  writes:

> Hi,
>
> With core-updates possibly being replaced by feature branches, I am
> not sure where to place this update to GNU Inetutils 2.4.
>
> The patch series [1] will rebuild 3,416 packages.

In the future, try to remember using the --subject-prefix='PATCH
core-updates', to avoid committers mistakenly pushing this to master, as
hinted in the manual: info '(guix) Sending a Patch Series' :-).

Thanks for the heads-up!

-- 
Maxim

Re: State of core-updates

[Maxim Cournoyer]

Hi Andreas,

Andreas Enge  writes:

> Hello all,
>
> let me start with a call for help! I realise that it takes me about one
> week and something close to 100GB on my poor 2-core laptop to rebuild
> the bulk of core-updates up to the packages in my profile, and that is not
> sustainable. It also forces me to do a "guix gc" between two runs, with
> the danger of either doing it too late and having to restart the builds
> (lived experience, one week lost), or losing and having to recompile
> store items that effectively have not changed.

Some things that may help that I use:

- Offloading
- Btrfs file system with zstd compression (to make that 100 GiB appear
  as 50 GiB or less on your drive)

[...]

> Since the bootstrapping seems to have stabilised, that would allow more
> people to work on packages closer to the leaves, since most of what
> currently builds would be available as substitutes from the build farm
> without everybody needing to go through a one-week compilation project.
>
> Here is my eclectic selection of packages I would add to the job:
> - guix (builds)
> - icecat (builds)
> - ungoogled-chromium (probably also builds)
> - openjdk (pulls in rust!, and builds)
> - unison (pulls in ocaml, and builds)
> - calibre (pulls in qt@5 and python; the former builds, the latter still
>   has some problems, among which the python bindings to qt, and packages
>   failing their tests even when updating to the latest release)
> - pandoc (pulls in ghc, which currently fails its tests @9.2.5)
> Please suggest more leaf packages that exercise your favourite missing
> language or application domain!

That looks promising.  Should we spun a differently named branch to
avoid people sending core-updates change?  This was discussed in the
past and agreed to (main branches do not *freeze* themselves), instead
we use git to branch to our will.

We could perhaps then add a 'core-updates-fixes-only' branch to the CI,
in which we'd try to restrict core changes as much as possible, keeping
the rebuild stress low for Berlin.

How does that sound?

-- 
Thanks,
Maxim

Re: The  Shepherd gets a service collection

[Efraim Flashner]

On Mon, Mar 13, 2023 at 05:11:34PM +0100, Ludovic Courtès wrote:
> Hello Guix!
> 
> I pushed some changes yesterday that confirm that the Shepherd paves the
> way for init system innovation, synergistic cross-domain fertilization,
> and delimited continuations:
> 
>   
> https://git.savannah.gnu.org/cgit/shepherd.git/log/?id=31d21fa083872d500c016b6b3b2587d25510702d
> 
>   31d21fa * Add REPL service.
>   cd6f3fb * comm: Add 'open-server-socket'.
>   c64804f * Add resource monitoring service.
> 
> These new services are built into shepherd, allowing users to control it
> and to fiddle with it.  The REPL is functional but of course a bit
> bumpy: you’d better know what you’re doing.
> 
> I imagine we could develop more convenient services like this, such as a
> basic command scheduler similar to the ‘at’ command, and a syslogd
> implementation.  The latter could be nice for a couple of reasons:
> logging would happen from the start and till the end (an improvement
> over the external syslogd process), and it could let us provide a nicer
> user interface to view logs (taking inspiration from that of
> ‘journalctl’).
> 
> Thoughts?  Ideas?

I always imagined an 'at' command being part of mcron. Does this mean we
should look forward to shepherdd-mcron?

> PS: It just occurred to me that we might as well rename the new
> (shepherd service …) hierarchy to (shepherd sheep …) or even
> (shepherd  …).

How does shepherd handle emojis for system symbols?

-- 
Efraim Flashner  אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted


signature.asc
Description: PGP signature

Re: The  Shepherd gets a service collection

[Maxim Cournoyer]

Hi Ludo!

Ludovic Courtès  writes:

> Hello Guix!
>
> I pushed some changes yesterday that confirm that the Shepherd paves the
> way for init system innovation, synergistic cross-domain fertilization,
> and delimited continuations:
>
>   
> https://git.savannah.gnu.org/cgit/shepherd.git/log/?id=31d21fa083872d500c016b6b3b2587d25510702d
>
>   31d21fa * Add REPL service.
>   cd6f3fb * comm: Add 'open-server-socket'.
>   c64804f * Add resource monitoring service.
>
> These new services are built into shepherd, allowing users to control it
> and to fiddle with it.  The REPL is functional but of course a bit
> bumpy: you’d better know what you’re doing.
>
> I imagine we could develop more convenient services like this, such as a
> basic command scheduler similar to the ‘at’ command, and a syslogd
> implementation.  The latter could be nice for a couple of reasons:
> logging would happen from the start and till the end (an improvement
> over the external syslogd process), and it could let us provide a nicer
> user interface to view logs (taking inspiration from that of
> ‘journalctl’).
>
> Thoughts?  Ideas?

While I also find the journalctl interface to be convenient, the
underlying database logs is costly in terms of storage and complexity (I
remember comparing it to a simple, non-compressed text file, and to my
surprise the later used less space even the systemd logs are supposed to
be compressed, if I remember correctly).  I think it had to do with all
the keys having to be stored for each message, even when they don't
contain anything useful.

It'd be nice if Shepherd didn't end up with that same caveat, should
database logs be implemented.

Thanks for improving Shepherd!

-- 
Maxim

Re: Preservation of Guix (PoG) report 2023-03-13

[Simon Tournier]

Hi,

On Mon, 13 Mar 2023 at 19:37, Timothy Sample  wrote:

> Note that you can link to the most recent version of the report using
> .

Awesome! \o/

Well, I do not remember if you consider also the ’origin’
(fixed-outputs) as ’inputs’ or ’patches’.  Do you?

Basically, ’package-direct-sources’ from (guix packages).

For instance, see the package ’ntp’,

--8<---cut here---start->8---
(source
 (origin
   (method url-fetch)
   (uri (list (string-append
   "https://www.eecis.udel.edu/~ntp/ntp_spool/ntp4/ntp-;
   (version-major+minor version)
[...]
   (sha256
(base32 "06cwhimm71safmwvp6nhxp6hvxsg62whnbgbgiflsqb8mgg40n7n"))
   ;; Add an upstream patch to fix build with GCC 10.  Taken from
   ;; .
   (patches (list (origin
(method url-fetch)
(uri "https://bugs.ntp.org/attachment.cgi?id=1760\
=diff=patch==1=raw")
(file-name "ntp-gcc-compat.patch")
(sha256
 (base32
  
"13d28sg45rflc7kqiv30asrhna8n69wlpwx16l65rravgpvp90h2")))
--8<---cut here---end--->8---

or see the package ’tensorflow’,

--8<---cut here---start->8---
(native-inputs
 `(("pkg-config" ,pkg-config)
[...]
   ("boringssl-src"
,(let ((commit "ee7aa02")
   (revision "1"))
   (origin
 (method git-fetch)
 (uri (git-reference
   (url "https://boringssl.googlesource.com/boringssl;)
   (commit commit)))
 (file-name (string-append "boringssl-0-" revision
   (string-take commit 7)
   "-checkout"))
 (sha256
  (base32
   "1jf693q0nw0adsic6cgmbdx6g7wr4rj4vxa8j1hpn792fqhd8wgw")
--8<---cut here---end--->8---


> Over the whole set, 77.1% are known to be safely tucked away in the
> Software Heritage archive.  But it’s actually much better than that.  If
> we only look at the most recent sampled commit (from Sunday the 5th),
> that number becomes 87.4%, which is starting to look pretty good!

Just to be point the new nixguix loader [1] is still in SWH staging and
not yet deployed, IIRC.  It will not change much the coverage on our
side but it should be fix some corner-cases.

1: 


>  This is kinda like an automated version of Simon’s recent
> investigation.

Neat!  Note that I also wanted to check the SWH capacity for cooking,
not only checking the end points.  For instance, it allowed to discover
mismatch due to uncovered CR/LF normalization; now fixed with:
58f20fa8181bdcd4269671e1d3cef1268947af3a.


> Here’s a rough road map for that based on a glance at the script’s
> output:
>
> • Subversion support (for TeX-based documentation stuff, I guess)

For the interested reader, details for helping in the implementation:

https://issues.guix.gnu.org/issue/43442#9
https://issues.guix.gnu.org/issue/43442#11

However, it would ease all the dance if SWH would consider to store and
expose NAR hashes on their side.  As discussed here:

https://gitlab.softwareheritage.org/swh/meta/-/issues/4538


>  However, 42% of them are old Bioconductor packages.  They
> seem to be lost.  It looks like Bioconductor now stores multiple package
> versions per Bioconductor version [2], but before version 3.15 that was
> not the case.  As an example, take “ggcyto” from Bioconductor 3.10 [3].
> We packaged version 1.14.0, and then at some point Bioconductor 3.10
> switched to version 1.14.1.  We packaged that, too, but now 1.14.0 is
> gone.

Well, I have not investigated much because it is between December 2019
and March 2020 thus “guix time-machine” is not smooth for this old time.

First question, does we have the source tarball in Berlin or Bordeaux or
somewhere else?  If yes, there is a hope. :-) Else, it is probably gone
forever.

The hope is: https://git.bioconductor.org/packages/ggcyto

If we have the tarball with the correct checksum from commit
f5f440312d848e12463f0c6f7510a86b623a9e27

--8<---cut here---start->8---
+(version "1.14.0")
+(source
+ (origin
+   (method url-fetch)
+   (uri (bioconductor-uri "ggcyto" version))
+   (sha256
+(base32
+ "165qszvy5z176h1l3dnjb5dcm279b6bjl5n5gzz8wfn4xpn8anc8"
--8<---cut here---end--->8---

then we can disassemble it and then using the Git repository, we can try
to assemble the content from SWH and the meta from Disarchive DB.

For sure, it is again another example why we should augment by 

Re: Google Summer of Code 2023 Inquiry

[Simon Tournier]

Hi,

On Sat, 11 Mar 2023 at 14:32, Simon Tournier  wrote:

> and probably more in the bug
> tracker.

For instance, you might be interested by:

https://issues.guix.gnu.org/issue/43442#9
https://issues.guix.gnu.org/issue/43442#11

as discussed in the recent message [1].

1: https://lists.gnu.org/archive/html/guix-devel/2023-03/msg00175.html

Feel free to ask questions in guix-devel mailing list or IRC libera.chat
#guix or #guix-hpc.

Cheers,
simon

Re: GSoC 2023

[Simon Tournier]

Hi,

On Wed, 08 Mar 2023 at 18:32, Simon Tournier  wrote:

> and probably more in the bug
> tracker. 

For instance, you might be interested by:

https://issues.guix.gnu.org/issue/43442#9
https://issues.guix.gnu.org/issue/43442#11

as discussed in the recent message [1].

1: https://lists.gnu.org/archive/html/guix-devel/2023-03/msg00175.html

Feel free to ask questions in guix-devel mailing list or IRC libera.chat
#guix or #guix-hpc.

Cheers,
simon

Re: Caching test results separately?

[Simon Tournier]

Hi,

On Mon, 13 Mar 2023 at 23:21, Josselin Poiret  wrote:

>> (⇒ keep the test result (boolean) longer than the build result)

[...]

> As it stands it's really not possible, as
>
> 1) testing is part of the build process itself and
> 2) we can't look-up any stateful info like this from the building
> process (of course!)

Indeed, the builder derivation is not the same and so the item is not
the same.

--8<---cut here---start->8---
$ guix build hello --no-grafts
/gnu/store/s5pd3rnzymliafb4la5sca63j86xs0y0-hello-2.12.1

$ guix build hello --no-grafts --without-tests=hello
/gnu/store/0h3d2z53xx2idy6pnqa8k781hlf40zmx-hello-2.12.1
--8<---cut here---end--->8---

Compare,
/gnu/store/s5y2w04jiydki5wb0zb9189x88v5288s-hello-2.12.1-builder
/gnu/store/d5asvb7mla28mf93hjrk6fffnx14n8b1-hello-2.12.1-builder

   (quote
())
   #:out-of-source? #f #:tests? #t 

   (quote
())
   #:out-of-source? #f #:tests? #f 



Last, note the self reference,

--8<---cut here---start->8---
$ diff -r --no-dereference \
  $(guix build hello --no-grafts)  \
  $(guix build hello --no-grafts --without-tests=hello)
Binary files
/gnu/store/s5pd3rnzymliafb4la5sca63j86xs0y0-hello-2.12.1/bin/hello and
/gnu/store/0h3d2z53xx2idy6pnqa8k781hlf40zmx-hello-2.12.1/bin/hello
differ

$ grep s5pd3rnzymliafb4la5sca63j86xs0y0 $(guix build hello 
--no-grafts)/bin/hello
grep: /gnu/store/s5pd3rnzymliafb4la5sca63j86xs0y0-hello-2.12.1/bin/hello: 
binary file matches

$ grep 0h3d2z53xx2idy6pnqa8k781hlf40zmx $(guix build hello --no-grafts 
--without-tests=hello)/bin/hello
grep: /gnu/store/0h3d2z53xx2idy6pnqa8k781hlf40zmx-hello-2.12.1/bin/hello: 
binary file matches
--8<---cut here---end--->8---


> But I would really like for tests to move out of build phases, the
> advantages would be twofold: less build time for a lot of packages, and
> better environment management for tests (we could rely on better
> interaction with other packages, more complicated mocking, use linux
> namespaces to their fullest extent, etc.).  That would require a huge
> change to Guix though, so it's more of a dream than anything concrete.

I agree:

 + it would simplify many things if we were able to run or re-run
   specific phases – Nix does that I guess.
 + it would be huge complex change. :-)

Well, similarly of the ’without-tests’ transformation, maybe we could
imagine another transformation for just run the test suite.  I mean, it
would not solve Arne’s issue but the converse: sometimes a package
requiring many compilation is failing because of one test and debugging
that is for some cases painful.


Cheers,
simon

Re: State of core-updates

[Roman Scherer]

Ok, cool. Looking forward to it.

Josselin Poiret  writes:

> [[PGP Signed Part:Undecided]]
> Hi Andreas,
>
> Roman Scherer  writes:
>
>> Hi Andreas,
>>
>> could we please add the jemalloc package that supports transparent huge
>> page sizes on aarch64 (which is currently waiting in core-updates) to
>> this list? At the moment any substitute which involves jemalloc isn't
>> useable on Linux kernels with a page size > 4K and requires building the
>> whole rust chain to Icecat working.
>
> Since jemalloc is indeed a dependency of icecat, it would also get built
> by the CI job if the latter is added.
>
> Best,


signature.asc
Description: PGP signature

Re: State of core-updates

[Josselin Poiret]

Hi Andreas,

Roman Scherer  writes:

> Hi Andreas,
>
> could we please add the jemalloc package that supports transparent huge
> page sizes on aarch64 (which is currently waiting in core-updates) to
> this list? At the moment any substitute which involves jemalloc isn't
> useable on Linux kernels with a page size > 4K and requires building the
> whole rust chain to Icecat working.

Since jemalloc is indeed a dependency of icecat, it would also get built
by the CI job if the latter is added.

Best,
-- 
Josselin Poiret


signature.asc
Description: PGP signature

Re: State of core-updates

[Roman Scherer]

Hi Andreas,

could we please add the jemalloc package that supports transparent huge
page sizes on aarch64 (which is currently waiting in core-updates) to
this list? At the moment any substitute which involves jemalloc isn't
useable on Linux kernels with a page size > 4K and requires building the
whole rust chain to Icecat working.

Please let me know if I can help with this somehow?

Thanks, Roman.

Andreas Enge  writes:

>
> Here is my eclectic selection of packages I would add to the job:
> - guix (builds)
> - icecat (builds)
> - ungoogled-chromium (probably also builds)
> - openjdk (pulls in rust!, and builds)
> - unison (pulls in ocaml, and builds)
> - calibre (pulls in qt@5 and python; the former builds, the latter still
>   has some problems, among which the python bindings to qt, and packages
>   failing their tests even when updating to the latest release)
> - pandoc (pulls in ghc, which currently fails its tests @9.2.5)
> Please suggest more leaf packages that exercise your favourite missing
> language or application domain!
>
> Andreas


signature.asc
Description: PGP signature