Re: API for rewriting package fields ?

[Ricardo Wurmus]

Nicolas Graves via "Development of GNU Guix and the GNU System distribution." 
 writes:

> Is there an interface to rewrite / update a field from a series of
> packages easily?

We have "update-package-inputs" in (guix upstream), which defines a
local procedure "update-field".  Perhaps you can use that?

What it cannot do yet is *create* a field, e.g. add a native-inputs
field to add an input when the package currently doesn't have a
native-inputs field.

-- 
Ricardo

Re: Is git the best tool for pulling packages?

[Ricardo Wurmus]

Adam  writes:

> As I see, first guix pull running too long for a lot of people.

Note that this is not due to download speeds but often due to
compilation.  When updating Guix you are not just fetching new data, but
a new version of Guix itself (which happens to come with a library
encoding package relationships).  That new version may need to be
compiled locally, which is slow.  We aim to provide pre-built components
that Guix can download instead, but computing what needs to be fetched
in the first place *also* takes a decent amount of time.

The problem would be the same if the transport mechanism was a
compressed archive instead of an update to a git repository.

> Probably there are ways to cache all of this.

We are caching things in ~/.cache/guix.

-- 
Ricardo

Re: Python's native-inputs

[Ricardo Wurmus]

>  TL;DR :
>  - patch series in big progress, not done yet because I don't really
>  know where to stop and massive rebuilds.

Please take a look at the python-team branch, which contains changes to
the build system.

-- 
Ricardo

Re: Distributed GNU Shepherd NLNet Grant

[Ricardo Wurmus]

Hi Juli,

> I am happy to announce that this grant application was approved! [2]

Congratulations!

> Materially, it would allow Shepherd dæmons running on
> different machines to securely communicate and interact with each
> other, going so far as to control one machine's dæmons from another
> machine.

I'm very interested in this.  Back when I worked on the Swineherd I
wanted to eventually extend it so that it can communicate with other
Swineherd instances to migrate Guix System containers between
cooperating machines.  Having this feature at the foundation of the
implementation of the Shepherd is much preferable to implementing a pale
compromise of this in an extension.

It also reminds me of Mach's cross-node message passing that could be
used to turn the Hurd into a Single System Image cluster.

Exciting stuff!  I wish you success and lots of fun working on this
project.

-- 
Ricardo

Re: Adding plumbing subcommand 'derivation'?

[Ricardo Wurmus]

Hi Simon,

> So I propose to add the plumbing command ’derivation’.  Any objection?

I think it's useful to have.  To avoid proliferation of sub-commands, do
you think we could put this under "inspect", a generic sub-command for
all sorts of as yet to be invented introspection tools?

-- 
Ricardo

Re: [python-team] Weird python-notebook test failures

[Ricardo Wurmus]

[resending as a wide reply, sorry]

Felix Lechner  writes:

> Hi Ricardo,
>
> On Sat, Apr 06 2024, Ricardo Wurmus wrote:
>
>> Any ideas what might be going on here?
>
> Could it be "ModuleNotFoundError: No module named 'jupyter_server'"?

I don't think so.  There is no jupyter_server module in the build
environment on the master branch either.

-- 
Ricardo

[python-team] Weird python-notebook test failures

[Ricardo Wurmus]

Hi Guix,

on the python-team branch I'm slowly fixing build errors, but I can't
figure out what's wrong with python-notebook:

https://ci.guix.gnu.org/build/3872654/details

Nothing big changed that would explain the build failure.  It looks like
the notebook server doesn't start and all the errors we get are due to
"Connection refused".

Any ideas what might be going on here?

-- 
Ricardo

Re: backdoor injection via release tarballs combined with binary artifacts (was Re: Backdoor in upstream xz-utils)

[Ricardo Wurmus]

[mu4e must have changed the key bindings for replies, so here is my mail
again, this time as a wide reply.]

Giovanni Biscuolo  writes:

> So AFAIU using a fixed "autoreconf -fi" should mitigate the risks of
> tampered .m4 macros (and other possibly tampered build configuration
> script)?
>
> IMHO "ignoring" (deleting) pre-built build scripts in Guix
> build-system(s) should be considered... or is /already/ so?

The gnu-build-system has a bootstrap phase, but it only does something
when a configure script does not already exist.  We sometimes force it
to bootstrap the build system when we patch configure.ac.

In previous discussions there were no big objections to always
bootstrapping the build system files from autoconf/automake sources.

This particular backdoor relied on a number of obfuscations:

- binary test data.  Nobody ever looks at binaries.

- incomprehensibility of autotools output.  This one is fundamentally a
  social problem and easily extends to other complex build systems.  In
  the xz case, the instructions for assembling the shell snippets to
  inject the backdoor could hide in plain sight, just because configure
  scripts are expected to be near incomprehensible.  They contain no
  comments, are filled to the brim with portable (lowest common
  denominator) shell magic, and contain bizarrely named variables.

Not using generated output is a good idea anyway and removes the
requirement to trust that the release tarballs are faithful derivations
from the autotools sources, but given the bland complexity of build system
code (whether that's recursive Makefiles, CMake cruft, or the infamous
gorilla spit[1] of autotools) I don't see a good way out.

[1] 
https://www.gnu.org/software/autoconf/manual/autoconf-2.65/autoconf.html#History

> Given the above observation that < to peer review a tarball prepared in this manner>>, I strongly doubt that
> a possible Makefile tampering _in_the_release_tarball_ is easy to peer
> review; I'd ask: is it feaseable such an "automated analysis" (see
> above) in a dedicated build-system phase?

I don't think it's feasible.  Since Guix isn't a regular user (the
target audience of configure scripts) it has no business depending on
generated configure scripts.  It should build these from source.

> In other words: what if the backdoor was injected directly in the source
> code of the *official* release tarball signed with a valid GPG signature
> (and obviously with a valid sha256 hash)?

A malicious maintainer can sign bad release tarballs.  A malicious
contributor can push signed commits that contain backdoors in code.

> Do upstream developer communities peer review release tarballs or they
> "just" peer review the code in the official DVCS?

Most do neither.  I'd guess that virtually *nobody* reviews tarballs
beyond automated tests (like what the GNU maintainers' GNUmakefile /
maint.mk does when preparing a release).

> Also, in (info "(guix) origin Reference") I see that Guix packages can have a
> list of uri(s) for the origin of source code, see xz as an example [7]:
> are they intended to be multiple independent sources to be compared in
> order to prevent possible tampering or are they "just" alternatives to
> be used if the first listed uri is unavailable?

They are alternative URLs, much like what the mirror:// URLs do.

> If the case is the first, a solution would be to specify multiple
> independent release tarballs for each package, so that it would be
> harder to copromise two release sources, but that is not something under
> Guix control.

We have hashes for this purpose.  A tarball that was modified since the
package definition has been published would have a different hash.  This
is not a statement about tampering, but only says that our expectations
(from the time of packaging) have not been met.

> All in all: should we really avoid the "pragmatically impossible to be
> peer reviewed" release tarballs?

Yes.

-- 
Ricardo

Re: Backdoor in upstream xz-utils

[Ricardo Wurmus]

Tomas Volf <~@wolfsden.cz> writes:

> On 2024-03-29 13:39:59 -0700, Felix Lechner via Development of GNU Guix and 
> the GNU System distribution. wrote:
>> > Is there a way we can blacklist known bad versions?
>>
>> Having said all that, I am not sure Guix is affected.
>>
>> On my systems, the 'detect.sh' script shows no referece to liblzma in
>> sshd.  Everyone, please send additional reports.
>
> If nothing else, our xz is at 5.2.8.  I think the question was if there is a 
> way
> to blacklist specific known tarball to ensure no-one updates to it by 
> accident.

The properties field on a package definition can be used to record
arbitrary information, which could be read by `guix lint`.

-- 
Ricardo

Re: guix --container is RAM hungry

[Ricardo Wurmus]

raingl...@riseup.net writes:

> On 2024-03-21 22:44, Edouard Klein wrote:
>> Dear Guixers,
>> 
>> I'm a huge fan of guix --container, and I created a system to use those
>> by default for network services. But the VPS these services run on has
>> only 2GB of RAM, and I just realized that a container, by default,
>> requires at least 200MB.
>> 
>> Try it:
>> guix shell time which -- bash -c "$(which time) -v guix shell
>> --container"
>> 
>> Then Ctrl-D, and look for this line:
>>  Maximum resident set size (kbytes): 291300
>> 
>> 291MB of RAM to run bash.
>> 
>> 
>> By contrast, removing the --container option from above yields:
>>  Maximum resident set size (kbytes): 64496
>> 
>> 64MB, still a lot, but I can live with that.
[...]
> My first guess is that the Guile process is hanging around waiting for
> the process it launched, instead of exec-ing into it directly.

In (@@ (guix scripts environment) launch-environment) we use execlp;
launch-environment is used by both launch-environment/fork as well as
launch-environment/container.  There is, however, an awful lot of
forking going on.

launch-environment/container uses call-with-container, which ends in
using launch-environment.  call-with-container forks, and
launch-environment also forks.

The Guile process that was started when launching Guix probably doesn't
get replaced in this case, as raingloom guessed.

-- 
Ricardo

Re: PyTorch with ROCm

[Ricardo Wurmus]

Hi David,

> after seeing that ROCm packages [1] are available in the Guix-HPC
> channel, I decided to try and package PyTorch 2.2.1 with ROCm 6.0.2.

Excellent initiative!

> For this, I first unbundled the (many) remaining dependencies of the
> python-pytorch package and updated it to 2.2.1, the patch series for
> which can be found here [2,3].

Oh, commit 8429f25ecd83594e80676a67ad9c54f0d6cf3f16 added
python-pytorch2 at version 2.2.1.  Do you think you could adjust your
patches to modify that one instead?

> It would be really great to have these packages in Guix proper, but
> first of course the base ROCm packages need to be added after deciding
> how to deal with the different architectures. Also, are several ROCm
> versions necessary or would only one (the current latest) version
> suffice?

As for the ROCm-specific work, I'm not qualified to comment.  I do
support a move of the ROCm packages from Guix HPC to Guix proper.

I think it is sufficient to only have the current version of ROCm; other
versions could be added if there is reasonable demand.

-- 
Ricardo

Re: How would you feel about this derivative logo for Nonguix?

[Ricardo Wurmus]

Luis Felipe  writes:

> https://gitlab.com/nonguix/nonguix/-/issues/317
>
> That page also mentions the reasoning behind the derivative logo.
>
> I don't see any issue if Nonguix uses such a logo (specially variant
> D2), but what do you think about it? They don't want to upset the GNU
> Guix community.

Variant D2 is my favourite.

I think it would be best to avoid the all-caps look, though, because
we're already losing the fight in correcting people who insist on
writing "GUIX" as if it was an acronym.  "RTOS GUIX" is the name of the
Microsoft thing.

Perhaps the tail could be a little curved towards the end?

-- 
Ricardo

Re: Contribute or create a channel?

[Ricardo Wurmus]

Hartmut Goebel  writes:

> In both cases I need some tooling to fetch the current bug-fix version
> of the series in question. This can not be done using "guix refresh"
> only AFAIU, as this would use the next release-series if this is
> already released while our packages are not yet updated.

"guix refresh" can be modified, and more specific importers / updaters
can be added to better suit the needs of groups of packages.

-- 
Ricardo

Re: GSoC 2024

[Ricardo Wurmus]

Gábor Boskovits  writes:

> Continue rewrite build daemon in Guile Scheme
> Ludovic Courtès (civodul)

Since this is an ongoing NLnet project I think we should not offer it as
a GSoC project this time.

-- 
Ricardo

Re: Contribute or create a channel?

[Ricardo Wurmus]

Hi Hartmut,

> I'm currently updating Tryton to version 7.0 and am wondering whether it's 
> better to contribute the change to Guix or to set up a
> channel for Tryton.
>
> WDYT? I'm eager to learn about your thoughts.
>
> Here is why I'm wondering:
>
> * Tryton consists of a client, a server and about 200 module/add-on providing 
> business logic.
> * Tryton publishes a LTS version every 2.5 years. Two LTS versions are 
> supported (currently 6.0 and 7.0) and bugfixes are backported
>  there for 5 years.

I think it would be preferable to have one LTS version in Guix.  If you
want to commit to maintaining the packages I think it would be worth
discussing whether to grant you commit rights to handle these upgrades
by yourself.

> OTOH in Guix, maintaining several version seems laborious.

What makes you say this?  What exactly is the obstacle here?

> Anyhow I'm unsure whether it's worth the effort maintaining three versions 
> and whether I'll be able to keep three version up to date -
> esp. given that I don't have much automation for this.

Personally, I don't think it makes sense to commit to maintaining there
separate versions.  It seems like a really big committment.

-- 
Ricardo

Re: Contribute or create a channel?

[Ricardo Wurmus]

Attila Lendvai  writes:

>> WDYT? I'm eager to learn about your thoughts.
>
> the patch inflow to the guix repo is currently overwhelming the
> available capacity for review and pushing.

With an email like the one sent by Hartmut we can better arrange for
shepherding this large submission.  (Nothing is to be gained from
repeatedly bemoaning well-known issues in the patch review processes
here and in other threads on the mailing list.)

-- 
Ricardo

Re: Building container images with nix2container

[Ricardo Wurmus]

Antoine Eiche  writes:

> Ricardo Wurmus  writes:
>
>> We have "guix pack" as part of Guix.  It builds Docker or squashfs
>> images as well as various other formats.  What does nix2container offer
>> beyond what we have?
>
> I acutally don't know how you currently build Docker images. But if
> nix2container brings something, i think it would mainly be
> optimizations (time and space).
>
> Does your built images contains several layers?

Yes.  See
https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-pack.html

-- 
Ricardo

Re: Building container images with nix2container

[Ricardo Wurmus]

Antoine Eiche  writes:

> In case you would like to try nix2container with Guix, in theory, you
> would need to add the support of another input reference graph format
> [6] and a write simple Guix derivations [7] calling the nix2container
> binary.

We have "guix pack" as part of Guix.  It builds Docker or squashfs
images as well as various other formats.  What does nix2container offer
beyond what we have?

-- 
Ricardo

Re: Simple design question for schemers

[Ricardo Wurmus]

Hartmut Goebel  writes:

> Using a custom function "extend":
>
> (native-inputs
>  (extend %standard-trytond-native-inputs
>  trytond-account-invoice
>  trytond-purchase
>  trytond-sale))

We have a macro called MODIFY-INPUTS, which you could use, but CONS* is
probably enough in your case.

-- 
Ricardo

Re: [Cuirass] JavaScript work

[Ricardo Wurmus]

Janneke Nieuwenhuizen  writes:

> I'm wondering though what the net gain of minification is
> with current bandwiths.  On the few sites that I use javascript on I just
> ship the preferred readable code.

JavaScript libraries can weigh several megabytes unminified.  With
minification applied to the concatenation of all JavaScript code we can
reduce the number of requests that need to be made and the weight of the
request.

[I remember that accessing anything JavaScript heavy (by byte-count) was
no fun in China.]

But minification may not be worth it anyway once we get rid of all
unnecessary JavaScript like jQuery, bootstrap, and DT.

-- 
Ricardo

Re: Supporting sssd, preparing for nscd sunset

[Ricardo Wurmus]

Hi Ludo,

thanks for getting the discussion started.  This problem has been
weighing on me for the past months and I don't see a good way forward.

Ludovic Courtès  writes:

> For the record, glibc maintainer Carlos O’Donell brought up our use case
> a while back on the glibc mailing list but it wasn’t particularly well
> received:
>
>   https://sourceware.org/pipermail/libc-alpha/2022-February/136741.html

DJ Delorie's response makes it sound like they would prefer that all of
the host system's NSS *configuration* (alongside the NSS modules) be
duplicated in the Guix "container", because that's not guaranteed to be
compatible.  That's not a way forward for us, because that would make
running Guix-built software much harder and require significant
configuration on the side of the user.

> [...] If sssd becomes dominant, we
> might just as well arrange to have NSS always load libnss_sss.so.

What are the other contenders here?  I assumed that sssd was what
everyone™ used nowadays.

At the MDC we've been using sssd on RHEL for a long time and we've seen
some problems with the naïve approach of just having the Guix System's
NSS load libnss_sss.so:

- In a Guix System VM we also had to explicitly set LD_PRELOAD or
  LD_LIBRARY_PATH in the current environment (not just nscd's
  environment) to make things like "id" work.  Having just NSS load the
  library was not enough to work with user accounts that are defined in
  LDAP, for example.

- A different Guix System VM was configured to use nss-pam-ldapd.  Upon
  reconfigure it attempted (and predictably failed) to delete all user
  accounts on LDAP, because none of them were defined locally.  I guess
  Guix System needs to be a little less eager to manage remote accounts
  that NSS plugins (including sssd) say exist.

- PAM in a container.

  While this is not strictly related to the disappearance of nscd, it is
  another obstacle relating to sssd.  On foreign distros I use Guix
  containers that bind-mount the host system's /var/lib/sss/pipes
  directory.  The hope was to use a native Guix build of libsss to talk
  to the socket at /var/lib/sss/pipes/pam (and potentially others) for
  authorizing users via the host system's sssd infrastructure.

  In "guix shell -C" this won't work, because sssd refuses to talk over
  that socket if the ownership is not root:root.  In the Guix shell
  container there is only ever *one* user account, so root-owned files
  are assigned to the overflow user account, and thus communication with
  the host system's pam socket is impossible.  I failed to fix this with
  subuids, because the "guix shell -C" process does not have sufficient
  privileges to map more than one user id.

I'll note that /var/lib/sss/pipes also contains an "nss" socket; I don't
know if it checks ownership on that socket, but it's possible that it
also cannot be used in "guix shell -C".

-- 
Ricardo

“guix package -d” on empty profile downloads stuff

[Ricardo Wurmus]

Hi Guix,

I have a little i686-linux laptop to which I deployed the Sugar desktop.
The only user account on that laptop (other than root) has no Guix
profile.

I switched to that user account and ran “guix package -d”.  Here is what
happened:

--8<---cut here---start->8---
$ guix package -d
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
25.8 MB will be downloaded
 glibc-2.35  8.5MiB 1.0MiB/s 00:08 ▕██▏ 
100.0%
 bash-minimal-5.1.16  603KiB2.0MiB/s 00:00 ▕██▏ 
100.0%
 gcc-11.3.0-lib  4.8MiB 977KiB/s 00:05 ▕██▏ 
100.0%
 libffi-3.4.4  57KiB588KiB/s 00:00 ▕██▏ 
100.0%
 libunistring-1.0  671KiB   1.1MiB/s 00:01 ▕██▏ 
100.0%
 pkg-config-0.29.2  218KiB  1.1MiB/s 00:00 ▕██▏ 
100.0%
 libgc-8.2.2  219KiB767KiB/s 00:00 ▕██▏ 
100.0%
 guile-3.0.9  8.1MiB1.3MiB/s 00:06 ▕██▏ 
100.0%
 guile-3.0.9-debug  7.5MiB  1.4MiB/s 00:05 ▕██▏ 
100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
0.4 MB will be downloaded
 glibc-utf8-locales-2.35  382KiB266KiB/s 00:01 ▕██▏ 
100.0%
guix package: error: profile '/var/guix/profiles/per-user/xxx/guix-profile' 
does not exist
--8<---cut here---end--->8---

Why do we need to download 25.8 MB to decide that we can’t do anything
here?

My usual excuse when faced with questions of this kind what always that
profile hooks may need to run, and they may require more recent versions
of tools to be built first.  Here we don’t seem to have a good reason
for that though.

-- 
Ricardo

[Cuirass] JavaScript work

[Ricardo Wurmus]

Hi,

I noticed that Cuirass bundles minified JavaScript.  I’ve started a new
branch that replaces the minified JavaScript with readable source code
and minifies the files as part of the build.

I also tried to remove the need for jQuery, at least in our own
JavaScript code.  (Datatables.js still uses jQuery.)

Eventually, I’d like to remove Bootstrap (both CSS and JS) all together
and switch to Pico.css for conventional default styles without the need
for bootstrap classes everywhere.  I’d like to remove JS for thing that
HTML+CSS can do just fine.

(Pico is also used in mumi.)

What do you think?

The branch is here:
https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/log/?h=wip-js%2bcss

-- 
Ricardo

Re: Debugging missing architecture support

[Ricardo Wurmus]

Efraim Flashner  writes:

> As far as tracking back from git-annex to ghc to see that it's not
> supported on aarch64, I'm not sure how you would find that information.

You can try this in guix repl:

--8<---cut here---start->8---
(import (srfi srfi-1)
(guix packages)
(gnu packages))

(define p (specification->package "git-annex"))
(define deps (package-development-inputs p))
(find (lambda (pkg)
(not (member "aarch64-linux" (package-supported-systems pkg
  (map cadr deps))
--8<---cut here---end--->8---


-- 
Ricardo

Re: guix refresh --update Removes Needed Dependencies

[Ricardo Wurmus]

"jgart"  writes:

> Hi Guixers,
>
> `guix refresh --update` removes Tex Dependencies that are needed. This makes 
> it more tedious to update packages :(

“guix refresh” considers the importers’ output to be authoritative.  Can
the Python importer be improved to discover the need for TeX things?
(If not the whole list of packages perhaps a collection package.)

In the CRAN importer we routinely check the contents of Makefiles,
configure scripts, etc to detect otherwise undeclared dependencies.
Perhaps the pypi importer could do something similar.

-- 
Ricardo

Re: Guix Days: Patch flow discussion

[Ricardo Wurmus]

Hi Josselin,

>>> b4/lei is a nice example (we already have yhetil.org as a back-end,
>>> but maybe a more blessed one would be better) of a tool that lets you
>>> completely automate applying a patchset to a branch.
>>>
>>> patchwork is a nice tool to gather up and track patchsets, with status
>>> indicators like "under review", "accepted", etc.  Chris already
>>> deploys one as part of QA, more integration with it would be nice.
>>
>> It seems (based on above) that "patchwork" can co-exist with debbugs.
>> Is that also the case with b4/lei?  Specifically, are the
>> users/reviewers able to benefit from using the above tools at present?
>> Or are there some reasons (over and above their lack of familiarity with
>> the above tools) that would prevent them from doing so?
>
> They both can co-exist with debbugs, and for now the patchwork instance
> of QA is not usable for status tracking (because it is not meant to be
> used as such for now).  One can already use both of them, but using both
> supercedes debbugs, and gets rid of its limitations.  I've been using
> b4/lei with the yhetil public-inbox instance, with piem.el as an
> interface, and it's really useful.  With a properly configured b4, one
> could simply run `b4 shazam some-msg-id` and it would automatically
> apply the corresponding patchset.

I’m interested in adopting this workflow.  Where can I find more
information on how to configure this?

-- 
Ricardo

Re: Debbugs update (Was: Guix Days: Patch flow discussion)

[Ricardo Wurmus]

Hi Felix,

> I also packaged and deployed on GNU Guix
>
>   (A) the fifteen-year old Debbugs version deployed at gnu.org [2][3][4]
>   (B) the modern Debbugs version deployed at debian.org [5][6][7]
>   (C) and a custom version of Mumi for my own bug fixes [8][9]
>
> Together with the official debbugs.gnu.org, which runs on Debian 8, and
> issues.guix.gnu.org, I am now working to fold all five deployments into
> one.

Let me just pop in to say thank you for that.  I appreciate you taking
the time to talk to Debbugs / GNU folks to work through the obstacles
that the current situation with the GNU fork of Debbugs presents us
with.

I don’t want to add weight to the scales on either side (a future with
or without Debbugs), but I think it is important to identify the actual
limitations and opportunities of the existing platform and chart the
shortest route around them.

It’s how we ended up with mumi in the first place, and I’m hopeful that
this approach can get us to a better place when it comes to tooling.

(The biggest chunk of the problem may well be a social issue, that can
only be overcome if we find a narrative we all can comfortably enact.)

-- 
Ricardo

Re: Introducing Guix "Features"! (Was: Syntactic Diabetes)

[Ricardo Wurmus]

> for an average unix user a service is a process that is running in the
> backgroud, doing stuff mostly without any user interaction. you can
> try to argue this away, but i'm afraid that this is the state of
> things.

I don’t think it’s a good idea to aim to satisfy some presumed “average
unix user”, because such a user would not be familiar with many concepts
introduced by Guix (e.g. “guix shell” or “guix system”).

The manual defines system services by referencing users’ expectations:

--8<---cut here---start->8---
11.18.1 Service Composition
---

Here we define a “service” as, broadly, something that extends the
functionality of the operating system.  Often a service is a process—a
“daemon”—started when the system boots: a secure shell server, a Web
server, the Guix build daemon, etc.  Sometimes a service is a daemon
whose execution can be triggered by another daemon—e.g., an FTP server
started by ‘inetd’ or a D-Bus service activated by ‘dbus-daemon’.
Occasionally, a service does not map to a daemon.  For instance, the
“account” service collects user accounts and makes sure they exist when
the system runs; the “udev” service collects device management rules and
makes them available to the eudev daemon; the ‘/etc’ service populates
the ‘/etc’ directory of the system.
--8<---cut here---end--->8---

Shepherd takes care of monitoring daemons and the like, but services
provided by the system (in the sense of system facilities) don’t have to
be daemons.

-- 
Ricardo

Re: 07/07: gnu: Add python-docspec.

[Ricardo Wurmus]

Hi Maxim,

> I'm not in the Python team, but I thought I'd give some feedback on
> recent Python packages added.
>
> guix-comm...@gnu.org writes:
>
>> gnu: Add python-docspec.
>> 
>> * gnu/packages/python-xyz.scm (python-docspec): New variable.
>> 
>> Change-Id: I3103bde3483273a335156b38de742f493fd366f1
>> ---
>>  gnu/packages/python-xyz.scm | 25 +
>>  1 file changed, 25 insertions(+)
>>
>> diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
>> index 61958d5eac..58a1a2b3cd 100644
>> --- a/gnu/packages/python-xyz.scm
>> +++ b/gnu/packages/python-xyz.scm
>> @@ -622,6 +622,31 @@ Python dataclasses.")
>>  from JSON payloads using the @code{databind.core} framework.")
>>  (license license:expat)))
>>  
>> +(define-public python-docspec
>> +  (package
>> +(name "python-docspec")
>> +(version "2.2.1")
>> +(source
>> + (origin
>> +   (method url-fetch)
>> +   (uri (pypi-uri "docspec" version))
>> +   (sha256
>> +(base32 "1zqmdrc6k8pprra8p3wpzq2ml2gph1cfjmsyg07f8b8fvizffm28"
>> +(build-system pyproject-build-system)
>> +(arguments (list #:tests? #false))  ;there are none
>
> This commit and a few others mention there are no tests; perhaps that's
> true of the PyPI archive, but in this case it's worth fetching from Git
> in my opinion to run the tests.  Here for example, there appear to be a
> Pytest test suite:
> https://github.com/NiklasRosenstein/python-docspec/tree/develop/docspec/test
>
> Fetching from git and adding pytest no native-inputs may be enough to
> have it run.
>
> Something to keep in mind for future submissions!

Yes, I agree.  These recent Python additions are a little special in
that they had been misplaced and forgotten in the guix-science channel.
I’ve recently taken some time to clean up the guix-science channel and
move things that don’t belong there to the default Guix channel,
reviving and updating packages as I went along.

It’s true that it would be a good idea to revisit these packages to
check if we could get a test suite from the source repository and run
it.

-- 
Ricardo

Re: python importers as an alternative to propagated-inputs

[Ricardo Wurmus]

Justin Veilleux  writes:

> Hi everyone. I was thinking about the propagated-inputs field in package
> definitions. As I understand it, it is useful as a way to replace RPATHs
> in packages that aren't compiled or don't support them.
>
> I was reading the documentation on
> https://docs.python.org/3/reference/import.html. It looks like we can
> define custom objects to do module resolution, possibly bypassing
> PYTHON_PATH lookup. I think it would be possible to write a very simple
> importer object that looks up module paths from an environment variable
> that looks like this:
>
> PYTHON_GUIX_MODULE_PATH=numpy=/gnu/store/...,pandas=/gnu/store/...

How would these compose?  “pandas” has dependencies, too.  Where would
those be read from?  How can be guaranteed that these are compatible
with what we had at compile time?

> Does someone know if this has been tried before? What potential problems
> would I encounter if I tried to implement this?

In 2018 we discussed the problems of PYTHONPATH extensively.  Hartmut
Goebel provided a three part analysis of the problem and potential
fixes.

The discussion didn’t lead to a generic solution, but it provided the
minimal reasons to replace our use of PYTHONPATH with GUIX_PYTHONPATH,
which we enable with a generated “sitecustomize.py” file — see (gnu
packages python) for details.

It would be great if we could do without GUIX_PYTHONPATH, which forces
us to harmonize *all* Python packages whenever we upgrade any of them.
Something that doesn’t require a global search path at all would be
ideal.  Perhaps the venv feature is how we can get there.  I recommend
reading the “PYTHONPATH issue analysis” messages for ideas and a list of
open questions.

-- 
Ricardo

Re: Separating test inputs?

[Ricardo Wurmus]

Felix Lechner via "Development of GNU Guix and the GNU System distribution." 
 writes:

> In Debian, test prerequisites are annotated awkwardly with  in
> the build prerequisites. (I think Guix calls them native-inputs.) You
> can see some of Debian's funny notations here [1] and here. [2]

Native inputs are those that in the context of cross-building need to
run on the host, not the target.  This is not just for test inputs.

Note also that the presence of test inputs has an impact on the build
itself, such as the configure step, the number of build phases, and even
the output.  In short: without tests this would be a different
derivation.  That’s what we do with the “--without-tests”
transformation: we generate new derivations where tests are not run.

We probably could enhance “--without-tests” to also remove some inputs.

-- 
Ricardo

Re: Question on python-pydantic update, rust and maturin build tool

[Ricardo Wurmus]

Sharlatan Hellseher  writes:

> It might be addressed to rust/python teams.
> https://github.com/pydantic/pydantic-core
>
> During update of some packages from (gnu packages astronomy) I've faced with 
> requirement for the fresh version of pydantic which
> depends on pydantic-core built with rust and maturin.

In python-blake3 I avoided the use of maturin with a little bit of
manual work.

-- 
Ricardo

Re: Are declarative app configs worth it?

[Ricardo Wurmus]

Nguyễn Gia Phong  writes:

> [[PGP Signed Part:Undecided]]
> [1. text/plain]
> On 2023-12-26 at 15:56+01:00, Ricardo Wurmus wrote:
>> On 2023-12-26 at 14:53+01:00, Sergey Trofimov wrote:
>> > - adding it to guix increases maintenance burden: new versions
>> >   could add or remove config options
>>
>> This is why there should be automated tests.
>> There are too few of them.
>
> This is easier said than done.  A program could […]

Yes, it’s easy to say a lot of things.  This is not an argument against
providing tests *where feasible*.

> I would also want to share my experience as a user, that having
> to run guix home reconfigure iteratively is not exactly pleasant
> due to the high delay.

What’s a high delay?  Is a one second reconfiguration too long for what
it does?  I don’t think so.

-- 
Ricardo

Re: Are declarative app configs worth it?

[Ricardo Wurmus]

Attila Lendvai  writes:

> if you demand that e.g. all services accepted into guix have a
> configuration entry for every possible config field, and that the
> documentation of these fields are duplicated into the guix
> codebase... then whatever is included into guix will have a 100%
> coverage. this is what is seen.

This is not what I argued for.  We also have had a couple of service
configurations that only provided options for a handful of options,
relegating all other possible options to a free text escape hatch.

There’s no point in talking percentages.

-- 
Ricardo

Re: Are declarative app configs worth it?

[Ricardo Wurmus]

Sergey Trofimov  writes:

> - adding it to guix increases maintenance burden: new versions   could
>  add or remove config options

This is why there should be automated tests.  There are too few of them.

> - it requires documentation/translation, another hidden cost

We should only accept configuration procedures that have proper
documentation, yes.

> - it bloats guix: imagine if we add configs for every
>   user-configurable app

That would be nice.

If we started to accept the term bloat we could easily apply it to
anything in Guix: all that R stuff?  Bloat!  All that bioinfo stuff?
Bloat!

> - such configs are not easily transferrable: if I were to use the
>   same app in non-guix env, I'd have to maintain 2 configs

We are generating configuration files from our config languages.  So you
would only need to generate them and copy them for your non-guix
environment.

> Another recent example is `oci-container-configuration` which defines
> a subset of docker-cli startup arguments. The problem is that `docker
> run` command has 96 options and the configuration only uses a handful,
> lacking a way to provide the remaining ones.

All config bindings need to have an escape hatch.

-- 
Ricardo

Re: arm-none-eabi toolchain and compiling C++ stuff

[Ricardo Wurmus]

Attila Lendvai  writes:

> is that a bug in (gnu packages embedded)? shall i look into fixing it?
>
> or am i the one who has invalid expectations?

FWIW I have successfully been using
make-arm-none-eabi-nano-toolchain-4.9 with the Axoloti DSP board (via
the axoloti-patcher package).  The 2018 toolchain you are using should
also just work (perhaps after setting *_CROSS_* variables).  So this
could very well be a bug in the package.

-- 
Ricardo

Re: Request-For-Comment process: concrete implementation

[Ricardo Wurmus]

Hi Simon,

> Well, more than 7 weeks later… Hum, does it mean that the Guix project
> is not interested in formalizing some RFC?
>
> WDYT about the proposal?

I just got back from travels and finally caught up with important email.
I read the proposal and it looks good to me.  Thank you for working on
this!

This would be the first project I contribute to that has an RFC process,
so I don’t know what to look out for.  My concerns may thus be
completely out of touch with reality, so beware as you read on.

It seems to me that the exact process is a little vague, especially with
regard to how long the comment period should be, and what expectations
there are during this period.  There is a chance that the open comment
period will lead to derailing discussions of tangents that make it hard
for the submitter to answer to real issues (because it would become
increasingly difficult to read all messages).

I’m thinking of some of the big discussions on the devel list in the
past that became too big to follow, and resulted in “consensus by
attrition”.  Do you know how other projects avoid needlessly dragging on
discussions about RFCs?

Will *any* disagreement have to be addressed, or will there be an
implicit weighing of opinions?  As the project grows bigger there can be
a problem of having inexperienced contributors (or those with
qualifications that are irrelevant to the proposal) block the RFC
without malicious intent by essentially requiring to be tutored on areas
outside of their expertise.

I wouldn’t trust myself to write an RFC without having played with an
implementation first.  I have doubts whether RFCs that are written
without a proof of concept could reasonably be evaluated.  Often details
and subtle problems are discovered only when playing with a patch, and
this may happen only after an RFC has been accepted.  Can we take back
approval in this RFC process?

And lastly a typo:

* “subtilities” should probably be “subtleties”.

-- 
Ricardo

Re: Computing a derivation for a profile

[Ricardo Wurmus]

Ricardo Wurmus  writes:

> I would like to compute a derivation for a profile specified by a
> manifest.  I do not (yet) want to actually build that derivation.

“guix build -d -m manifest.scm” also doesn’t quite do what I want.  It
returns the derivations for all the listed packages, but it doesn’t
generate a profile derivation and doesn’t include any of the hooks
either.

-- 
Ricardo

Computing a derivation for a profile

[Ricardo Wurmus]

Hi Guix,

I would like to compute a derivation for a profile specified by a
manifest.  I do not (yet) want to actually build that derivation.

This seems like a simple thing to do, but I haven’t been able to
accomplish this.  Here is one of the things I’ve tried in “guix repl”:

--8<---cut here---start->8---
(begin
  (use-modules (gnu packages) (guix) (guix profiles))
  (derivation-file-name
   (with-store s
 (run-with-store s
   (lower-object
(profile
 (content
  (specifications->manifest
   (list "texlive-collection-latex"
 "texlive-collection-latexextra"
 "texlive-collection-latexrecommended"
 "texlive-collection-fontsrecommended"
 "texlive-pgf")
--8<---cut here---end--->8---

Unfortunately, lowering this profile object appears to invariably result
in substitutes for all these texlive packages to be downloaded, even
though I only want to get the derivation.

Am I doing something wrong here?  Is this supposed to work?  Is this a
regression?

-- 
Ricardo

Re: RFI: Guix XMPP service. paid service?

[Ricardo Wurmus]

jbra...@dismail.de writes:

> I would like to pay $5 a month to have an xmpp account 
> coolawesomeusern...@guix.gnu.org
>
> Are there other interested parties?  It might be a possible way to generate 
> $$ to continue developing guix.

I’d rather not commercialize stuff like this, especially not for project
communication.  I wouldn’t want our servers to cache image,
video, and audio uploads by “coolawesomeusername” that aren’t related to
discussing Guix.  Nor would I want to give the impression that
“coolawesomeusername”’s activities are in any way endorsed by
guix.gnu.org.  We don’t ache for $5 a month, and giving away autonomy
over the name and what it is associated with is worth far more than a
trickle of $5.

In my personal opinion (as an experienced curmudgeon) I think that
setting things up for commercialization is to invite unspeakable dread
and rot.

-- 
Ricardo

Re: Should commits rather be buildable or small

[Ricardo Wurmus]

Attila Lendvai  writes:

> i myself also had headaches multiple times when i fixed something that
> needed to touch several different packages, and they would only work
> when applied in one transaction:
>
> how many debbugs issues? multiple issues and record the dependencies?
> little gain for much more effort on both sides... but if one issue,
> then what should be the name of the debbugs issue? etc...

We often have just one debbugs issue per transaction, i.e. multiple
patches that all belong to the same issue.  The free form cover letter
should describe what the relation between the patches is.

This can be done for dependencies (e.g. an R package and all the
unpackaged things it needs) or for a feature with several related but
atomically recorded changes.

A misconception I read earlier is that there’s a desire to have one
commit per touched file, but that’s certainly not true.  Atomicity
doesn’t care about file boundaries, and we don’t care about them either.
Commits should tell a story and often it is easier to tell it by making
consecutive independent changes.

-- 
Ricardo

Re: Heisenbug

[Ricardo Wurmus]

Felix Lechner via "Development of GNU Guix and the GNU System distribution." 
 writes:

> While executing meta-command:
> error: label: unbound variable
>
>> ,reload
>
> While this gives
>
> While executing meta-command:
> unknown file name for module #
>
> Also, what is a meta-command, please?

The Guile manual doesn’t define the term but consistently uses it for
things like “,reload” or “,bt”, i.e. any command that is not supposed to
be evaluated directly as program code but in the meta environment of the
REPL itself.

-- 
Ricardo

Re: Should commits rather be buildable or small

[Ricardo Wurmus]

Attila Lendvai  writes:

>> I guess "required" here means that in some cases Guix's policy is to
>> prefer small commits over buildable commits (with the previous
>> definition). I at least don't see any technical reasons why it would be
>> required. The question then becomes whether that policy applies in this
>> case.
>
>
> FWIW, this commit policy has always bothered me as a newcomer to
> Guix. pretty much everywhere else it's a major offence against your
> colleagues to commit something that breaks the build in any way.

In the last few months I’ve repeatedly seen assertions in a similar
style as this one.  They always genuinely surprise me, and it’s probably
not just because I’m oblivious and out of touch.

Also in Guix it is not okay to commit intermediate things that break
stuff.

Commits should, however, also tell a reviewable story and not be a big
blob of thousands of lines of non-trivial changes.  (The one exception
has always been fixes of repeated typos, but this is not what this
discussion is about.)

For what it’s worth: when I do bulk R upgrades I generally have one
commit per package upgrade, because it’s easy to review in a git log in
the future and easy to revert individual changes with fine grain access.
Of course I group upgrades that depend on one another (e.g. r-arrow
together with apache-arrow and the inheriting python-arrow).

-- 
Ricardo

Re: Better support remote deployment

[Ricardo Wurmus]

Ludovic Courtès  writes:

> That said, I wonder if this would really be more convenient than SSH’ing
> into the target machine and running the commands right there.  Perhaps
> I’m missing something about the use case?

The use case is to have a development host where packages are built and
then pushed / installed on a remote that lacks the prerequisites to
build the packages (e.g. checkout of development source repositories).

I feel it would be a good fit for “guix deploy”, which has a similar set
of assumptions but for deploying Guix System rather than mere package
profiles.

-- 
Ricardo

Building and caching old Guix derivations for a faster time machine

[Ricardo Wurmus]

Hi Guix,

to me the biggest downside of using “guix time-machine” is that it has
to do a lot of boring work before the interesting work begins.  The
boring work includes building Guix derivations for the given channels,
most of which have long been collected as garbage on ci.guix.gnu.org.

It would be helpful, I think, to more aggressively cache these
derivations and their outputs, and to go back in time and build the
derivatinons for past revisions of Guix.  I would expect there to be a
lot of overlap in the produced files, so perhaps it won’t cost all that
much in terms of storage.

What do you think?

-- 
Ricardo

Re: Meet Guix at Capitole du Libre in Toulouse, nov. 18-19

[Ricardo Wurmus]

Luis Felipe  writes:

> El 26/10/23 a las 14:03, Luis Felipe escribió:
>> El 25/10/23 a las 21:17, Julien Lepiller escribió:
>>> The print service I usually use has a lot of options for flyers,
>>> {10,15,20,30}*{10,15,20,30}, 12*12, 21*29.7 and 30*40 cm. I think
>>> 15*20 would be best.
>>>
>>> Visit cards would be 3.5*7.5, 5.5*8.5 or 5*9 cm
>> Thanks, Julien, I'll hopefully have something ready for next
>> week. I'll let you all know.
>
> Here's a visit card in the same style of the kakemono:
>
> https://luis-felipe.gitlab.io/downloads/pdf/gnu-guix-visit-card-A01-EN-90x50mm.pdf
>
> The source is in https://codeberg.org/luis-felipe/guix-graphics/
> (promotional/cards).

Wonderful work, Luis Felipe!  I’d love to have some of these for future
presentations or conferences.

-- 
Ricardo

Re: Better support remote deployment

[Ricardo Wurmus]

Hi Felix,

> On Wed, Nov 01 2023, Ricardo Wurmus wrote:
>
>> What do you think about changing “guix package” and/or “guix copy” to
>> better support deployment of remote profiles?
>
> As someone who uses 'guix deploy' all the time. I believe remote
> administration is one of our core strengths and selling points. Other
> remote administration tools like Ansible or Puppet are probably no
> longer needed when using GNU Guix System.
>
> I currently have only x86_64 equipment deployed, so there is no
> cross-building happening in my setup. Is that the main purpose of your
> proposal?

No, both machines are x86_64.  I think “guix deploy” could benefit from
a way to not only deploy Guix System (which makes the tool primarily
useful for managing hosts, not applications) but also profiles and
containers.

-- 
Ricardo

Better support remote deployment

[Ricardo Wurmus]

Hi Guix,

I build software locally and deploy the result to a remote system with
“guix copy”.  This works pretty well but has a few rough edges:

1. “guix build -m manifest.scm” does not generate a profile.  It only
builds the list of packages.  To build a profile from a manifest file we
need to resort to something like this:

guix shell -m $(PWD)/etc/container-server-manifest.scm -- sh -c 'echo 
$GUIX_ENVIRONMENT'

2. “guix package” cannot install an existing profile store item as the
current generation of the profile.  It can, however, install individual
package items into a profile.

3. “guix package --remove” does not support regular expressions, so
removing packages that were installed with “guix install /gnu/store/…”
cannot easily be removed.

Because of these limitations I cannot make use of a Guix profile symlink
forest on the target system.  Instead I build a profile locally (with
the “guix shell” trick above), copy it to the remote with “guix copy
--to=remote /gnu/store/…-profile”, and then link that profile to a fixed
location on the remote system.

I would like to change this workflow so that I can benefit from roll
backs without having to manually mess with symlinks.

What do you think about changing “guix package” and/or “guix copy” to
better support deployment of remote profiles?

-- 
Ricardo

[cuirass] Typo?

[Ricardo Wurmus]

Hi,

In (cuirass base) there is this definition:

--8<---cut here---start->8---
(define (exception-reporter . results)
  "Return an exception handler that reports the exception on the error port
and returns the values RESULTS."
  (lambda (key . args)
(false-if-exception
 (let* ((stack (make-stack #t))
(depth (stack-length stack))
(frame (or (and (> depth 1) (stack-ref stack 1))
   (and (> depth 0)) (stack-ref stack 0
   (print-exception (current-error-port) frame key args)
   (apply values results)
--8<---cut here---end--->8---

The parentheses for the binding of “frame” look wrong to me:

(frame (or (and (> depth 1) (stack-ref stack 1))
   (and (> depth 0))
   (stack-ref stack 0)))

-- 
Ricardo

Re: doc: Mention the responsibilities that blocking comes with.

[Ricardo Wurmus]

Simon Tournier  writes:

> +concerns are actively resolved with proposals that work for everyone.  A
> +contributor (which may or may not have commit access) wishing to

Should be “who” instead of “which”.

> +understand its finer details, you are encouraged to read
> +.

Isn’t this URL also supposed to be wrapped in @url?

-- 
Ricardo

Re: Re-evaluating the practice of automating user configuration

[Ricardo Wurmus]

Maxim Cournoyer  writes:

> All in all, I guess my position is unchanged: despite the potential for
> surprises, automating and enforcing these configs provide benefits that
> outweigh the cons, in my experience/opinion.

I concur.

In light of efforts to reduce cognitive overhead, I think it is a good
idea to automatically use the default configuration.  Contributors can
opt out, but I prefer not to have to think about yet another important
piece of configuration here.

-- 
Ricardo

Re: Proposal: Differentiate products more clearly (Cycle 01)

[Ricardo Wurmus]

Hi,

> 2018-01-17 · website: say what Guix is at the very top
> https://lists.gnu.org/archive/html/guix-devel/2018-01/msg00232.html

Hey, that’s me!  Naturally, I do agree that we need clearer product
differentiation and your drafts are pretty and clear as always.

On the other hand I’ve always felt that the term “package manager” is so
limiting.  It primes people to expect a package manager, but none of
the features in the “guix system” family of commands match that.
Likewise, all that container stuff we do with “guix shell -C” lies just
outside of what one would think of when hearing “package manager”.

In my presentations I try to drive home the point that Guix is a
solution for reproducible deployment at every level of scale.  I have a
set of slides similar to the “Reproducible Outputs” section in your
home-page.pdf, going from package to environments to containers to
systems to sites (via “guix deploy”).

I wonder if we should find a way to phase out the use of the term
“package manager”, which seems to fire the wrong synapses and awaken
associations with things that aren’t at all like Guix.

(Imagine we came from the build system side and positioned Guix as “A
Better GNU Make”.  That would be similarly flawed and evoke images that
fail to match what Guix actually is.)

-- 
Ricardo

Re: performance issue with TeX Live

[Ricardo Wurmus]

Emmanuel Beffara  writes:

> De Nicolas Goaziou le 09/10/2023 à 16:03:
>> Emmanuel Beffara  writes:
>> > I do want to contribute to a solution, because right now texlive is
>> > practically unusable in Guix.
>> 
>> FWIW, I use modular TeX Live regularly, so "unusable" is probably a bit
>> strong. However, I agree the current situation is frustrating.
>
> Ok, "unusable" may be an overstatement. But when compiling a simple document
> takes one or two minutes […]

Oh, that sounds bad.  I only ever use pretty small collections of TeX
Live packages, so I never noticed.

In the past we used to create a union directory and generated a
configuration file to only search for packages in that single directory
instead of searching a large number of directories.  I wonder if we
could do something like that.

-- 
Ricardo

Re: Is this a bug in guix refresh with respect to Common Lisp packages?

[Ricardo Wurmus]

"jgart"  writes:

>> These are the 7 packages:
>
> I didn't get that output from `./pre-inst-env guix refresh -l 
> sbcl-enhanced-eval-when`
>
> I get the following:
>
> [jgart@fedora guix]$ guix refresh -l sbcl-enhanced-eval-when
> Building the following 1 package would ensure 7 dependent packages are 
> rebuilt: cl-definitions-systems@2.0.1

This prints the one package, not the 7 dependent packages.

> [jgart@fedora guix]$ guix refresh -l python-requests
> Building the following 633 packages would ensure 1362 dependent packages are 
> rebuilt: sdkmanager@0.6.5 …protonup-ng@0.2.1

These are the 633 packages, not 1362 packages.

I don’t see any difference in behavior here.

-- 
Ricardo

Re: Is this a bug in guix refresh with respect to Common Lisp packages?

[Ricardo Wurmus]

"jgart"  writes:

> make -j6 && ./pre-inst-env guix refresh -l sbcl-enhanced-eval-when
>
> I don't see 7 dependent packages that would be rebuilt... Just three.
>
> Building the following 1 package would ensure 7 dependent packages are 
> rebuilt: cl-definitions-systems@2.0.1

These are the 7 packages:

cl-enhanced-eval-when@1.0
cl-enhanced-defclass@2.1
cl-canonicalized-initargs@2.0
cl-definitions-systems@2.0.1

sbcl-enhanced-defclass@2.1
sbcl-canonicalized-initargs@2.0
sbcl-definitions-systems@2.0.1

-- 
Ricardo

Re: guix refresh -u weird behaviour

[Ricardo Wurmus]

"jgart"  writes:

> I've experienced this same issue with quite a number of Python
> packages that I've tried to import though...

Yes, this indicates a defect in the importer.

-- 
Ricardo

Re: guix refresh -u weird behaviour

[Ricardo Wurmus]

"jgart"  writes:

> guix refresh -u does some weird stuff. See the git diff below.
>
> Why did it remove the native inputs?

Because Faker doesn’t list them in a way that the importer can add them.
The quality of the importer determines the quality of the updater.  The
updater only does what the importer tells it to do.

-- 
Ricardo

Re: GUI for Guix

[Ricardo Wurmus]

Adam Faiz  writes:

>> The last message in that issue discussion is mine:
>
>>FWIW M-x guix-installed-packages (and all the other stuff) works for me
>
>> Is it *actually* broken?  If it is and you can provide information on
> how to trigger the broken behavior we might be a step closer to fixing
> it.
>
> I assumed that M-x guix-installed-packages already worked, so I wanted to 
> test the main menu instead.
>
> I did the following steps:
> 1. M-x guix
> 2. Chose 'packages' by pressing p
> 3. Go to the search by pressing s

This works for me.  It starts the REPL in the background and shows me a
long list of packages.

-- 
Ricardo

Re: GUI for Guix

[Ricardo Wurmus]

Adam Faiz  writes:

> 4. Emacs-Guix, currently broken (https://issues.guix.gnu.org/55013)

The last message in that issue discussion is mine:

FWIW M-x guix-installed-packages (and all the other stuff) works for me

Is it *actually* broken?  If it is and you can provide information on
how to trigger the broken behavior we might be a step closer to fixing
it.

-- 
Ricardo

Re: Guix related things in Germany at the end of October/start of November

[Ricardo Wurmus]

Christopher Baines  writes:

> PackagingCon [1] is happening in Berlin on the 26th to the 28th of
> October […]
> Is anyone else planning to attend these events, or otherwise interested
> in meeting up in Germany around these dates?

I don’t seem to have anything planned around the time of PackagingCon,
so should a little Guix meeting happen in Berlin I’d try to make time to
stop by for a while.

-- 
Ricardo

Re: more than 1,800 dependent packages: website out of date

[Ricardo Wurmus]

Nathan Dehnel  writes:

> I feel there should be a "guix manual" command that opens the latest
> version of your local copy in a new tab of whatever your XDG web
> browser is

Across the wider GNU system the command for accessing manuals is “info”
followed by the name of the manual, e.g. “info guix” or “info
guix-cookbook”.

-- 
Ricardo

Re: The e(macs)lephant in the room and the Guix Bang

[Ricardo Wurmus]

Csepp  writes:

> Ricardo Wurmus  writes:
>
>> Nathan Dehnel  writes:
>>
>>> heard about guile-studio, but it doesn't appear to have a dark mode,
>>> and I imagine trying to add one would require a bunch of emacs-style
>>> screwing around with it.
>>
>> M-x modus-themes-toggle RET
>>
>> i.e. hold Alt, press x, then type “modus-themes-toggle” and hit the
>> return key.
>>
>> We can add a little toggle button to Guile Studio that does this.
>
> Kinda tangential, but could Emacs be made to just use the system (GTK)
> theme?

Yes, that’s my plan, but I don’t have enough time this month to work on
it.  My notes:

+ when =dconf read /org/gnome/desktop/interface/color-scheme= returns
  ’prefer-dark’ enable dark mode by default.

+ use =xprop= to change the theme variant *after* the frame has been created; 
see
  https://gist.github.com/itoshkov/850c46746705e32e2039fb0112a75ec7
  
+ we may also need to set =GTK_THEME= to =Adwaita:dark= before actually
  launching Guile Studio.  Or rather the value at
  =/org/gnome/desktop/wm/preferences/theme= followed by =:dark=.  This
  can be done in shell wrapper.

-- 
Ricardo

Re: The e(macs)lephant in the room and the Guix Bang

[Ricardo Wurmus]

Ricardo Wurmus  writes:

> Nathan Dehnel  writes:
>
>> heard about guile-studio, but it doesn't appear to have a dark mode,
>> and I imagine trying to add one would require a bunch of emacs-style
>> screwing around with it.
>
> M-x modus-themes-toggle RET
>
> i.e. hold Alt, press x, then type “modus-themes-toggle” and hit the
> return key.
>
> We can add a little toggle button to Guile Studio that does this.

Actually, I just noticed that Guile Studio already tells you how to do
this in the help pane that is displayed on startup:

Toggle between dark and light mode with F5.

-- 
Ricardo

Re: The e(macs)lephant in the room and the Guix Bang

[Ricardo Wurmus]

Ricardo Wurmus  writes:

> Nathan Dehnel  writes:
>
>> heard about guile-studio, but it doesn't appear to have a dark mode,
>> and I imagine trying to add one would require a bunch of emacs-style
>> screwing around with it.
>
> M-x modus-themes-toggle RET
>
> i.e. hold Alt, press x, then type “modus-themes-toggle” and hit the
> return key.
>
> We can add a little toggle button to Guile Studio that does this.

To affect the context menu and menu bar you’d also need to do set the
GTK_THEME variable when starting Guile Studio, e.g.

GTK_THEME=Adwaita:dark guile-studio

I’ll try to change Guile Studio so that it will automatically pick the
dark or light variant of the theme dependent on system preferences, but
for now this is the workaround that should get you most of the way to a
dark theme.

-- 
Ricardo

Re: The e(macs)lephant in the room and the Guix Bang

[Ricardo Wurmus]

Nathan Dehnel  writes:

> heard about guile-studio, but it doesn't appear to have a dark mode,
> and I imagine trying to add one would require a bunch of emacs-style
> screwing around with it.

M-x modus-themes-toggle RET

i.e. hold Alt, press x, then type “modus-themes-toggle” and hit the
return key.

We can add a little toggle button to Guile Studio that does this.

-- 
Ricardo

Re: OCI-backed Guix System Services

[Ricardo Wurmus]

Hi,

> I was recently inspired from Nix's oci-container feature and wrote a thin 
> wrapper around the docker CLI to enable the management of
> docker containers through Shepherd [0]. This enables handling of non packaged 
> services through guix system reconfigure and herd
> start/stop/status . 

Neat!

> To conclude, I'm not advocating for adding OCI-backed services to Guix 
> mainstream: in my opinion they should be bootstrapped and
> built from source, but I believe the actual "backend" implementation for such 
> services could be useful to have in Guix. What do you
> think?

I think this could still be a valuable addition to Guix, because it
integrates with Shepherd and thus unifies management of system services
independent of whether they came from Guix or from elsewhere.

Because integration with Shepherd is nice I wrote the Swineherd which
serves a related need: https://github.com/BIMSBbioinfo/swineherd

-- 
Ricardo

Re: The e(macs)lephant in the room and the Guix Bang

[Ricardo Wurmus]

MSavoritias  writes:

> On 9/20/23 11:45, Nguyễn Gia Phong via Development of GNU Guix and the
> GNU System distribution. wrote:
>> On 2023-09-20 at 10:21+02:00, Csepp wrote:
>>> It's better if we have at least one *well documented* developer setup,
>>> than if we have a bunch of (sometimes conflicting) partial docs
>>> for setting up certain subsystems.
>>>
>>> Emacs can be pretty good, once you do (setq make-defaults-not-suck 1)
>>> a bunch of times.
>> Or even more outrageous, an overriden Emacs package
>> with all the good stuff for Guix development.
>> We already have guix shell that spawns a shell
>> and guix edit that spawns an editor, why no guix boot
>> that spawns an OS^W^W Emacs with appropriate defaults?
>>
>> Disclaimer: I use the devil editor that goes by the number
>> of VI VI VI, so take this suggestion with a grain of salt.
>>
>
> Can't the guile editor be that?
>
> It kind of tries to be already. We just need to promote it and dogfood
> it more.

Do you mean Guile Studio?[1]

It was really only intended to be a pre-configured editor for new
Guilers, which is why it includes the picture language and Geiser with
picture display.

But as I wrote there

   There are many more things that can be done to make Emacs less
   confusing for people who use it just as a Guile IDE. I would be happy
   to hear your suggestions or apply patches to improve Guile Studio!

This is still true today.

[1]: https://elephly.net/guile-studio/

-- 
Ricardo

Re: Swineherd: Guix System container manager

[Ricardo Wurmus]

Simon Tournier  writes:

>> Of course the Swineherd is also available as a Guix package called
>> “swineherd”.
>
> Hum, I did (or adding guile):
>
> guix time-machine -q -- shell swineherd
>
> Then I do not know what to do.  What are the basic steps for testing it?

There are no executables.

The basic steps for setting things up are in the included manual.

You’ll first need to set up a root disk for all your disposable
container home directories.  See the “Getting Started” chapter in the
manual on instructions how to get this set up for testing.

You then launch a new Shepherd process with the included
shepherd-config.scm (it’s at $out/share/swineherd/shepherd-config.scm),
which registers the “swineherd” and “swineherd-http-api” services, and
starts the former.

You can then try the Shepherd actions documented in the “Command Line
Interface” section of the manual.

I haven’t tried loading the included shepherd-config.scm into my PID 1,
because I don’t need it and I don’t want to freeze my PID 1 due to bugs
:) But with a properly set GUILE_LOAD_PATH (including the Swineherd
modules) it should also work with PID 1.

-- 
Ricardo

Re: How can we decrease the cognitive overhead for contributors?

[Ricardo Wurmus]

Fannys  writes:

>> But again, even if this is a great option for you, it might be a really bad
>> option for some other people. Everybody does not have the time to spend
>> learning emacs, or other specific tool. It's ok if the workflow suggests that
>> but it's not great if we have no other alternative.
>>
>> It's not accessible and imposes a barrier in some people.
>
> Yeah agreed. And we should be consious of that.
> Ironically by mandating Emacs and Email we force people to use specific
> tools while at the same time even though the same people will complain(!) 
> against vendor lock-in
> like github.

We don’t *mandate* the use of Emacs.  It’s just a common recommendation
because it works so well with text and is trivially extensible, so it’s
a common target for helper tools.  Surely we also wouldn’t call a
recommendation to use a shell script “vendor lock-in” just because it
needs Bash.

Emacs works well with text, and text is all that’s needed in a
patch-based workflow, which is in fact vendor agnostic.

Of course this doesn’t mean that it is as accessible as we’d want.

-- 
Ricardo

Re: How can we decrease the cognitive overhead for contributors?

[Ricardo Wurmus]

MSavoritias  writes:

> Simon Tournier  writes:
>
>> Hi,
>>
>> On Tue, 29 Aug 2023 at 12:53, MSavoritias  wrote:
>>
>>> Do you know if there are any plans to write a scheme bug/patching
>>> system? Because looking a bit into it, it doesn't seem like its that
>>> actively developed so maybe we would be better served by one in scheme.
>>> Or Sourcehut of course as somebody wrote in another email.
>>> Since not much would change with sr.ht anyways.
>>
>> The only work I know is named tissue by Arun Isaac.  Arun also
>> contributes to Mumi (one web front-end of the venerable Debbugs instance
>> of the GNU project).
>>
>> https://tissue.systemreboot.net/
>> https://archive.fosdem.org/2023/schedule/event/tissue/
>>
>> Well, I am not convinced by the argument « we would be better served by
>> one in scheme ».  For instance, Cuirass or Build Coordinator or even
>> Mumi, all are implemented in Scheme and they are receiving very few
>> contributions.
>>
>> https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git
>> https://git.savannah.gnu.org/cgit/guix/build-coordinator.git/
>> https://git.savannah.gnu.org/cgit/guix/mumi.git/
>>
>>
>> Cheers,
>> simon
>
> Its more about:
> If debbugs has issues and we dont fully control it and can make those
> improvements, why not something in scheme instead of sr.ht?

It’s an independent axis.  Limitations in Debbugs cannot be fixed
centrally because we don’t control the server.  If we did decide to
operate a bug tracker by ourselves it would suit us to pick one that
didn’t also require a lot of effort to work around annoyances.

The fact that we’ve used Scheme for a number of tools that don’t see
many contributions aside from those who initially wrote them is a good
reminder to adjust our expectations.


-- 
Ricardo

Re: Swineherd: Guix System container manager

[Ricardo Wurmus]

> Can you share any ways you're currently using this?

We had been using this at work as part of a much bigger training
platform to launch customizable (but still reproducible) interactive
environments for workshops and courses.

An example is pre-configured RStudio installations with shared course
materials.  With guix.install (provided by r-guix-install) participants
could install any Guix packages from within their running R session.  We
also provided a web interface to customize the environment ahead of
time, and to save/restore the container root fs.

> Any plans/interest in hooking it into `guix deploy`?

Plans no, but interest yes!

-- 
Ricardo

Swineherd: Guix System container manager

[Ricardo Wurmus]

Hi there,

you know the Shepherd: it is an elegant service manager looking after a
herd of daemons.  Since it can be extended with Guile, I decided to do
just that to add an extra skill to the Shepherd, turning it into the
Swineherd.

The Swineherd is a manager of Guix System containers.  It is implemented
as a Shepherd service called “swineherd” that lets you manage Guix
System containers as Shepherd services.  These system container services
implement a few common features to bring up bridge networking, execute
commands inside the containers, peek at files, and get simple usage
stats.

Swineherd also provides an optional HTTP API server to launch, manage,
and destroy containers through an HTTP client.  This makes the Swineherd
useful for building web applications that launch Guix System containers
on demand.  It comes with an Info manual describing the assumptions, the
required setup, as well as the implemented commands and the HTTP API.

The Swineherd was designed to be used with Shepherd on foreign distros,
so it does not assume to be running on top of Guix System (for better or
worse).

You can find the code here:

https://github.com/BIMSBbioinfo/swineherd

Of course the Swineherd is also available as a Guix package called
“swineherd”.

If this tickles your interests I’d be very happy to hear your comments
and incorporate patches to extend it beyond my original requirements.

-- 
Ricardo

Re: Mumi CLI client (was: How can we decrease the cognitive overhead for contributors?)

[Ricardo Wurmus]

Giovanni Biscuolo  writes:

> AFAIU mumi does not (still?) have ad authentication/authorization,
> right?
>
> If so how do you plan to deal with users posting SPAM or similar
> unappropriate content?

It only sends email on behalf of commenters, so we’re using the same
email mechanism to deal with spam.

-- 
Ricardo

Re: How can we decrease the cognitive overhead for contributors?

[Ricardo Wurmus]

Simon Tournier  writes:

> On Fri, 08 Sep 2023 at 17:27, Ricardo Wurmus  wrote:
>
>> Now, this is no longer a problem for me because I’ve been writing so
>> many commit messages over the years (and because I no longer try to
>> adhere to some poorly specified format), but it *is* a problem for
>> people that I’ve mentored.
>
> Well, from my point of view, you cannot know if another way to write
> commit message format would not be a problem either. :-)

Let’s not write them at all and have them generated.  People are still
expected to read them before sending them off, to be sure that they say
what was intended.

> I think we have a consensus about the complexity for contributing.  From
> my point of view, all the “cognitive complexity” are not equal.  Some
> cognitive loads are totally useless and just pure friction – the number
> of steps for checking and submitting for example – and it brings no
> value at all for the project.  Other, as commit message format, also
> leads to some friction but it also brings value for the project.

Agreed.

-- 
Ricardo

Re: How can we decrease the cognitive overhead for contributors?

[Ricardo Wurmus]

Liliana Marie Prikler  writes:

>> Must we force a single workflow on everyone, even if our track record
>> in reviewing and merging doesn’t clearly show that our way is
>> superior?
> Again, define superior.

No, I won’t.  I think it’s obvious that our review process isn’t working
*well*.  So the argument that our current workflow allows for effective
review is dubious.  Not saying that you made that claim, just that it’s
hard to convince others of adopting our ways when the results just
aren’t great.

>> Recall that the reason for my response at this point in the thread is
>> your statement:
>> 
>> > Hiding obsolete versions of a pull request is in practice
>> > implemented either by pushing more commits on top of the existing
>> > one, often with dubious commit messages or by force-pushing a
>> > branch, neither of which is an acceptable solution for Guix.
>> 
>> I’m trying to convey that this workflow is similar to how we would
>> push to wip-* branches and informally discuss open issues out of
>> band.  (And even in that scenario, we are rather limited by the way
>> our shared repository with all-or-nothing permission management
>> works.)
> I think this is a bit of an apples to oranges comparison.

Exactly.  Apples and oranges are both fruit, full of sugar and fiber,
and some people prefer one over the other.

> Even for our
> wip branches, we mostly adhere to the guidelines that govern master,
> it's mostly the rule regarding breaking changes that is relaxed.  We
> wouldn't have well-functioning wip branches if those forewent *all*
> communication efforts.  The only common ground I can see is that all
> feature branches – or at least those that don't die – get merged to the
> main branch in either workflow.

Yes, that’s my point.

-- 
Ricardo

Re: How can we decrease the cognitive overhead for contributors?

[Ricardo Wurmus]

Liliana Marie Prikler  writes:

> Am Freitag, dem 08.09.2023 um 17:27 +0200 schrieb Ricardo Wurmus:
>> I have the same positive view on our faux ChangeLogs commit messages,
>> though I also would like to have them generated.  The benefit is
>> still there: I still get to *review* an effective summary of the
>> changes before pushing or sending them off for review.  But at least
>> I don’t have to write them myself.
>> 
>> Now, this is no longer a problem for me because I’ve been writing so
>> many commit messages over the years (and because I no longer try to
>> adhere to some poorly specified format), but it *is* a problem for
>> people that I’ve mentored.
>> 
>> etc/committer.scm and the yasnippets are supposed to alleviate some
>> of the pain, but I don’t need to think for a long time to come up
>> with a number of improvements in this area.
> Can I assume this to mean it'd take you some short time to think of
> snippets that we're currently lacking?  If so, please do contribute
> them.  If not, what do you mean then?

I mean that they have plenty of defects.

When I wrote the first few iterations of etc/committer.scm it was only
really meant and good for bulk package updates (= lots of changes across
files, all upgrades).  It couldn’t (and maybe still can’t) reliably
detect added or removed package definitions.  It doesn’t handle changes
to the arguments field.  It’s also terribly slow because it naively
recomputes information for every hunk in the diff, reading package
definitions from the old vs the changed file after every commit.

The update yasnippet repeatedly gets the order of lines wrong when
adding a patch to dist_patch_DATA in gnu/local.mk; it also doesn’t do
what etc/committer.scm is already able to do: detecting changes to
inputs.  Configuring yasnippet is also not trivial for people who don’t
regularly use Emacs (the snippets are tied to modes set by magit).

I think in light of these defects “Uhm, we have snippets?” isn’t a
satisfying response.

-- 
Ricardo

Re: How can we decrease the cognitive overhead for contributors?

[Ricardo Wurmus]

Liliana Marie Prikler  writes:

>> On Github, Pull Request branches are like our WIP branches.  They are
>> how we arrive at acceptable changes.  Picky people like me would then
>> go back and write new atomic commits for the effective diff, but in
>> my role as a reviewer I usually rebase, squash, and merge.
>> 
>> This workflow is more familiar to some and alienating to others, but
>> both of these workflows would work fine for Guix.  But today our
>> tools can only accommodate *one* workflow.  
> I'd imagine that rebase, squash and merge would exacerbate the workload
> on the committer side and I think that most popular projects on those
> forges already experience similar effects to us despite folks just
> merging the requests as-is and in part even getting paid by big tech
> for doing so.

Look, I’m relating merely my work experience here as someone who
regularly reviews pull requests on Github.  Github has buttons for
“Rebase and merge”, “Squash and merge”, and “Create a merge commit”, so
that part is automated.

I’m not saying that this is the workflow we should adopt.  I’m saying
that these platforms — for better or worse — encourage *different*
workflows, and for some this is what they are most comfortable with.

Must we force a single workflow on everyone, even if our track record in
reviewing and merging doesn’t clearly show that our way is superior?

Recall that the reason for my response at this point in the thread is
your statement:

> Hiding obsolete versions of a
> pull request is in practice implemented either by pushing more commits
> on top of the existing one, often with dubious commit messages or by
> force-pushing a branch, neither of which is an acceptable solution for
> Guix.

I’m trying to convey that this workflow is similar to how we would push
to wip-* branches and informally discuss open issues out of band.  (And
even in that scenario, we are rather limited by the way our shared
repository with all-or-nothing permission management works.)

-- 
Ricardo

Re: Mumi CLI client (was: How can we decrease the cognitive overhead for contributors?)

[Ricardo Wurmus]

Giovanni Biscuolo  writes:

> […] actually Debbugs or Mumi web
> interfaces are read-only: you cannot open a bug report or comment it,
> you have to send an email; this is a _feature_, not a bug since we don't
> need a _complex_ web based authentication+authorization system for bug
> reporters/commenters. […]

Mumi actually does support commenting on the web:

   
https://git.savannah.gnu.org/cgit/guix/mumi.git/tree/mumi/web/controller.scm#n145

It’s just been disabled because messages ended up being stuck in the
queue and nobody could make enough time to debug this.

-- 
Ricardo

Re: Cadence For Merging python-team into master

[Ricardo Wurmus]

"jgart"  writes:

> What is the cadence for merging the python-team branch into master?
>
> Should you do it, me, or someone else?

Another thing that needs fixing is sphinx.  The upgrade of
python-pygments and python-sphinx did not go over too well.

-- 
Ricardo

Re: How can we decrease the cognitive overhead for contributors?

[Ricardo Wurmus]

Liliana Marie Prikler  writes:

> To be completely honest, mumi recalling everything at 0 precision is my
> biggest pet peeve at the moment

Can you please make a test case for this and add it to the mumi tests?
We have tests for the search feature there.

-- 
Ricardo

Re: How can we decrease the cognitive overhead for contributors?

[Ricardo Wurmus]

Maxim Cournoyer  writes:

> I used to think the same about ChangeLogs style commit messages, but I
> have to admit that it forces some discipline on me by having me review
> my changes in details and write out what I did.  It'll sometimes expose
> something that'd be better kept in a separate commit, or something I did
> and forgot about in the early development, that should be dropped.

I have the same positive view on our faux ChangeLogs commit messages,
though I also would like to have them generated.  The benefit is still
there: I still get to *review* an effective summary of the changes
before pushing or sending them off for review.  But at least I don’t
have to write them myself.

Now, this is no longer a problem for me because I’ve been writing so
many commit messages over the years (and because I no longer try to
adhere to some poorly specified format), but it *is* a problem for
people that I’ve mentored.

etc/committer.scm and the yasnippets are supposed to alleviate some of
the pain, but I don’t need to think for a long time to come up with a
number of improvements in this area.

> There's also no stopping you adding a more conventional rationale as a
> paragraph between the topic and the ChangeLog content (which should
> appear at the bottom of the commit message).

Yes, this is something that might not be clear to all.  Before the
somewhat mechanical summary you can write a paragraph or two to give
context.

-- 
Ricardo

Re: How can we decrease the cognitive overhead for contributors?

[Ricardo Wurmus]

Liliana Marie Prikler  writes:

>> one fundamental issue with the email based workflow is that its
>> underlying data model simply does not formally encode enough
>> information to be able to implement a slick workflow and frontend.
>> e.g. with a PR based model the obsolete versions of a PR is hidden
>> until needed (rarely). the email based model is just a flat list of
>> messages that includes all the past mistakes, and the by now
>> irrelevant versions.
> What the?  If anything, emails are like a tree and discussions in most
> forges are a single long list that's rarely well organized.  Virtually
> every mail client supports threads, whereas a certain one of the more
> popular forges still refuses to do so.  Hiding obsolete versions of a
> pull request is in practice implemented either by pushing more commits
> on top of the existing one, often with dubious commit messages or by
> force-pushing a branch, neither of which is an acceptable solution for
> Guix.

I often have to review Github Pull Requests, and I don’t go commit by
commit by go through the diff and annotate the changes.  I *read* the
commits to get a sense for how the feature evolved and why changes were
made, but the fact that new commits are pushed on top of the branch is
not an obstacle in practice, because the commits don’t matter.

(I know, it hurts me too, but I don’t make the rules, okay?)

And in these review interfaces we can mark individual comments as
resolved.  So the flat list of changes with annotations *does* in fact
provide a clearer organization than a tree of emails.

Note also that we don’t usually review commits by starting one new
thread for each issue we address, so we don’t benefit from automatic
branching of sub-discussions.

On Github, Pull Request branches are like our WIP branches.  They are
how we arrive at acceptable changes.  Picky people like me would then go
back and write new atomic commits for the effective diff, but in my role
as a reviewer I usually rebase, squash, and merge.

This workflow is more familiar to some and alienating to others, but
both of these workflows would work fine for Guix.  But today our tools
can only accommodate *one* workflow.  It happens to be the one I’m used
to, but let’s please not pretend that it’s inherently *better* than the
other.

-- 
Ricardo


FWIW: Mumi is the worst of both worlds as it flattens the email tree
while not providing any of the review features that popular forges
have.  So the problem of outdated and repeated mistakes in a flat list
becomes more pronounced there.

Re: How can we decrease the cognitive overhead for contributors?

[Ricardo Wurmus]

Josselin Poiret  writes:

> Regarding the “mom argument”, I would disagree and say that this is
> completely related: interruptions are more costly, you're more likely to
> have less attention span, and overall you probably don't want to commit
> to 20 steps just to send a contribution.

That’s exactly it.

I’m a parent working a full-time job and I’m easily overwhelmed by what
I would have considered to be simple processes 20 years ago.  All these
simple and totally “easy” processes add up and – oops it’s 11:30pm
already and I had been feeling exhausted since 10am…

There’s an hour before I pass out — what will I do with it?  This thread
is about how we can reduce overhead; for me that would mean to have
certain checks automated, or to have commit messages be generated so I
only need to read them to sign off on them, etc.

We all may gain from streamlining our processes.  Personally, I see the
biggest problem in a lack of effective tools for reviewers, which
results in stretching out contribution interactions over weeks, months,
or years.  This, to me, would be a way to increase the weight of the
rewards on the unbalanced scales that are overwhelmed by chores on the
other side.

-- 
Ricardo

Re: How can we decrease the cognitive overhead for contributors?

[Ricardo Wurmus]

Giovanni,

> You are obviously free not to contribute your patches upstream but the
> fact that you decided not to because it's "too hard" (my executive
> summary about your complaints about Change Log content rules) to write
> commit messages suitable for contribution it _not_ a Guix maintainers
> fault, not at all.

As a former Guix co-maintainer I disagree with this take.  (Nobody even
brought up the word “fault”, which is a particularly unhelpful lens for
understanding social issues, in my opinion.)

“too hard” sounds (perhaps unintentionally) derisive.  It’s a real issue
for the projects when very capable contributors like Katherine encounter
numerous little obstacles that in aggregate lead to a very rational
decision to be selective about what contribution to send off on this
unnecessarily long journey.

It’s not that writing commit messages is hard.  It’s one of many
obstacles, and the lack of clear objective guidelines (fun fact: we
aren’t actually following the Changelog rules) that means that even
something as trivial (compared to the rest of the work) as the commit
message must be placed on the pile of chores.  Add on top of that that
there’s a low probability of gratification, because committers like
myself are getting burned out and so patches simply go unacknowledged or
only ever see a first cursory review.

We can’t blame anyone for seeing these two piles and come to the
conclusion that it’s not worth the hassle — especially when operating
your own channel is so easy in comparison.

Katherine, I’m very happy you brought this up and continue to respond in
this thread to clarify and steer the discussion into a fruitful
direction.  I know I couldn’t do it.  I thank you for this work, and I
hope that the project can come up with ways to lower the barriers to
entry.

-- 
Ricardo


PS: It’s probably no exaggeration to say that I’m only still
contributing to Guix because I already *have* commit access and I’m
committed to ensuring that support for R and science packages doesn’t
degrade.  That alone keeps me busy.  While commit messages aren’t really
an obstacle for me personally (probably because nobody judges my
messages), I probably wouldn’t be able to justify spending my now very
limited free time to contributing to Guix now that the reward/effort
scale is so unbalanced.

Re: How can we decrease the cognitive overhead for contributors?

[Ricardo Wurmus]

Maxim Cournoyer  writes:

> Hi,
>
> Ricardo Wurmus  writes:
>
>> "Danny Milosavljevic"  writes:
>>
>>> I agree that automating the technical steps of contributing to Guix
>>> would be nice.
>>
>> We could provide a VM.  Guix can create systems well enough to make this
>> feasible, I think.
>
> I'm not convinced.  I typically uses VM for testing quick hacks on
> foreign systems, not as my main development environment.  I doubt the
> experience would be helped much.  We have 'guix shell' :-).

Or “guix system container” if we want to meet in the middle :)

-- 
Ricardo

Replacing Mumi+Debbugs?

[Ricardo Wurmus]

Maxim Cournoyer  writes:

>> I’ll say that many of my gripes with (the GNU instance of) Debbugs are
>> due to the fact that we can’t customize it to better suit our needs — it
>> is a shared resource with a complicated maintenance story.  So all
>> changes went into Mumi as crude workarounds.  I think this is a dead end
>> and we’d be better off leaving the shared GNU instance of Debbugs
>> behind.
>
> I'd be sad to loose at least two good things from Debbugs:
>
> 1. It's hosted by the GNU/FSF for us.  It always work, and the rare
> times it doesn't, the folks in #savannah are hard at work resolving the
> problems. While hosting sourcehut is probably not too difficult, keeping
> it up to date (Go...) and running would be yet another weigh on our
> meager sysadmin team.

Yes, this is a real concern.  Infrastructure at scale is always a little
messier than one would assume.

> 2. Integration with Emacs.  emacs-debbugs is useful.  I think it's the
> only successful thing we have at keeping track of old tickets and
> resuming discussion or acting on these.
>
> I like how clean Mumi looks, compared to most forge issue trackers.  I'm
> not convinced by its search results (perhaps I'd need to get to know
> what Xapian is).

Xapian is a search engine for documents.  It lets us create search
indices over keywords extracted from documents and retrieve matching
documents.  It’s what “mu” uses for its database; that’s the origin of
“mu” in “mumi” as we used to just leave all the search business to “mu”
in early versions.

All the things related to search are here:

https://git.savannah.gnu.org/cgit/guix/mumi.git/tree/mumi/xapian.scm

“index-files” opens all emails, extracts fields such as author, subject,
status, body (for full text search), etc, and builds the search
index/database.

“search” converts the query string into a query object and requests
matching documents from the database.

We’re constantly updating the index to match the growing collection of
Debbugs emails.

We are testing the basic search interface here:

https://git.savannah.gnu.org/cgit/guix/mumi.git/tree/tests/xapian.scm

If there are bad results (and mumi should be used for a while longer to
justify changing it) we should create test cases for them, so we get a
chance to update the way we index the documents and create a query.

Going forward we could also think about augmenting the search with text
embeddings, e.g. by computing word vectors over the whole collection of
emails, turning the search query into a word vector, and then search for
the most similar vectors in the collection.  This would give us very
fuzzy matching of synonyms and related terms.  fasttext and faiss can be
used for these purposes.

-- 
Ricardo

Re: How can we decrease the cognitive overhead for contributors?

[Ricardo Wurmus]

"Danny Milosavljevic"  writes:

> I agree that automating the technical steps of contributing to Guix
> would be nice.

We could provide a VM.  Guix can create systems well enough to make this
feasible, I think.

> This definitely should be automated. Why not generate a UUID locally and send
> a mail to @debbugs.gnu.org ? That may require changes to debbugs,
> though.

We cannot change the shared debbugs instance, which is why I think we
should take a closer look at self-hosting sourcehut.

-- 
Ricardo

Re: How can we decrease the cognitive overhead for contributors?

[Ricardo Wurmus]

Vagrant Cascadian  writes:

> The only thing clunky about this particular aspect of the workflow
> described is the fact that the guix community (maintainers?) have
> decided on a one patch per mail policy with a cover letter, rather than
> submitting the patches as attachments in the initial mail.

You are right.  When I started contributing I actually did attach all
patches in one email.  I wonder why we stopped doing that.

I’ll say that many of my gripes with (the GNU instance of) Debbugs are
due to the fact that we can’t customize it to better suit our needs — it
is a shared resource with a complicated maintenance story.  So all
changes went into Mumi as crude workarounds.  I think this is a dead end
and we’d be better off leaving the shared GNU instance of Debbugs
behind.

-- 
Ricardo

Re: How can we decrease the cognitive overhead for contributors?

[Ricardo Wurmus]

Csepp  writes:

> I'll try to pick up the Sourcehut
> packaging, now that I know there is a chance it would be accepted.

Excellent.  Thank you!

I started work on the issue tracker a while ago, but only had enough
time for the first two packages which are in gnu/packages/sourcehut.scm.

-- 
Ricardo

Re: How can we decrease the cognitive overhead for contributors?

[Ricardo Wurmus]

paul  writes:

>> Please go ahead and package the rest of Sourcehut, so we can host it and
>> finally forget about Mumi and Debbugs.
>
> Again, this is not a competition between Sourcehut and Mumi. This is a
> discussion between adults that care very much about the same
> philosophical and political points about technological control and
> autonomy.

You seem to have managed to miss my point.  I emphatically agree that
there is no competition.

As the original author of Mumi I want it to become obsolete as it
clearly has outlived its usefulness and better alternatives exist.  The
way to get there is to add the replacement to Guix so we can host it.

I won’t contribute to Mumi any more.  Giving it up doesn’t hurt my
feelings.  I’d be glad to see it gone.

-- 
Ricardo

Re: How can we decrease the cognitive overhead for contributors?

[Ricardo Wurmus]

Csepp  writes:

> Mumi currently fails at its main function of being a search engine for
> issues.  I can't even find my own messages with keyword searches,
> because for some reason Mumi connects each words with a logical or and
> doesn't rank search results based on how many hits there are, not to
> mention handling fuzzy hits, or synonyms, etc.

Mumi commit bda10fa1b37fed8ea313ea8ad49fa6146e65b4c0 changed the AND to
the Xapian default of OR.  Here’s the rationale from the commit message:

xapian: Do not override the default OR implicit query operator.

An implicit AND operator is overly restrictive. It was only necessary
because prefixes that should have been indexed as boolean prefixes
were not.

Xapian *does* rank search results by score.  It also supports fuzzy
search.  But let me move a little so I won’t stand in the way of a good
rant.

> As I see it, this is an issue of developer resources.  Mumi and Debbugs
> have no chance of catching up to any alterntive that already has a
> sustainable financial model and professional developers working on it.
>
> Why are we trying to compete with Sourcehut?  What's the end goal?

I can’t make myself even clearer about this, but I’ll repeat myself
again: we are not.

“Competition” is a foreign concept to me.  Mumi exists only because I
found the Debbugs web interface kinda ugly and alienating.  It’s a hack
that obviously outlived its welcome.  I have to admit that messages like
this make me regret scratching this itch that gave rise to Mumi in 2016.
(Sourcehut started in 2018.)

Please go ahead and package the rest of Sourcehut, so we can host it and
finally forget about Mumi and Debbugs.

-- 
Ricardo

Re: How can we decrease the cognitive overhead for contributors?

[Ricardo Wurmus]

Maxim Cournoyer  writes:

> In the simplest case, it can be as simple as:
>
> $ ./pre-inst-env guix refresh -u some-package
> [build it, try it, fix if needed]
> $ ./etc/committer.scm
> $ git send-email
>
> Since it's a single patch, there's no jumping through hoops to create
> the original issue, and the auto-configured git will CC teams people
> subscribed to the scope of your change (if there are any).

Sadly the not-so-simple case is much more annoying.

I like an email-based workflow, don’t get me wrong.  At the same time
I’m really not a fan of Debbugs.  It makes simple things more difficult
than they should be (send a cover letter, wait for your mail to pass the
anti-spam measures, remember that you wanted to send a couple of
patches, etc), and no amount of sugar coating (whether that be a
different frontend like Mumi or a hell of a lot of client-side
scripting) can overcome this inherent clunkiness.

I’m no stranger to clunky or quirky interfaces.  I put up with them
because they unlock workflows and features or flexibility that would
otherwise be unattainable.  Emacs and Guix are great examples of quirky
interfaces that brim with flexibility.  I just think that Debbugs
doesn’t make the cut.

If only we can find a suitable baby sieve, I’d be happy to see the
turbid bathwater drained.  If Mumi and Debbugs went down the drain to
reveal a new hassle-free web/email hybrid patch workflow I’d be
ecstatic.

-- 
Ricardo

Re: How can we decrease the cognitive overhead for contributors?

[Ricardo Wurmus]

Katherine Cox-Buday  writes:

> * We could support a managed web-based workflow

I started packaging some requirements of sourcehut.  I’d be happy to
retire mumi + debbugs and use whatever sourcehut offers if we can host
it.

I haven’t ever used sourcehut things, but I hear they are good for
web-based and email-based workflows alike.

-- 
Ricardo

Re: IPv6 access for ci.guix.gnu.org

[Ricardo Wurmus]

Guillaume Le Vaillant  writes:

> […] not ready for IPv6 […]
> Is it still the case today?

I don’t think anything has changed in that regard as the switches have
not been replaced.

-- 
Ricardo

Re: Binary descriptors for OpenCV

[Ricardo Wurmus]

Nathan Dehnel  writes:

> Perhaps such greyzone objects that can't be fully regenerated should
> be put in their own channel so users know where they are and it
> doesn't become a mystery how many they have installed on their
> systems.

I would want put a variant of the opencv package in the guix-science
channel if we decide not to include them, because xfeatures2d is useful
for feature detection in scientific applications (that’s the only reason
why I’m looking into this).

However, it would be good to come up with a policy anyway.

And of course it would be nicer if we could avoid further fragmentation,
as we’d need to push all packages that depend on this variant of opencv
to that separate channel, too.

-- 
Ricardo

Binary descriptors for OpenCV

[Ricardo Wurmus]

Hi Guix,

I’d like to draw your attention to https://issues.guix.gnu.org/64945.
It’s a patch that adds binary descriptors to OpenCV.

These descriptors are the result of a very expensive computation, which
could be performed with lots of memory and GPUs.  The result is a small
number of very small descriptors in binary format, which OpenCV can use
as an input to a feature detection algorithm.

This is probably one of the simplest cases of machine learning output;
the output is small and compared to other machine learning models
require only a small amount of computation.  But it’s above the
threshold for our build farm and not something we can have users
recompute on install.

The software used to generate these descriptors is freely licensed, and
the descriptors are living in the twilight zone of assets that are not
quite software but clearly not just decorative either.  They are large
arguments to image feature detection algorithms, much like an image mask
would be.

What shall we do with this patch?  Can we accept it or does it cross a
line we don’t want to cross?

-- 
Ricardo

Re: A Forum for Guix Users

[Ricardo Wurmus]

Nguyễn Gia Phong via "Development of GNU Guix and the GNU System distribution." 
 writes:

> On 2023-07-13 at 16:40+02:00, Andrea Rossi wrote:
>> However, I do wonder whether the features in Mailman 3,
>> namely Postorius and HyperKitty, might make it easier for new users
>> to get started without losing the ability to hone their skills
>> in information retrieval and knowledge representation models.
>
> Were I not subscribed, I'd defo prefer HyperKitty to Mailman basic (?)
> web UI.  HyperKitty was created precisely for this purpose:
> https://blog.linuxgrrl.com/2012/02/29/7750-pixels-of-mailing-list-thread

HyperKitty is very nice looking.  We don’t host Mailman ourselves,
though, so I don’t know how we’d go about setting it up.

-- 
Ricardo

Re: Adding GNAT/GCC-Ada to Guix

[Ricardo Wurmus]

Denis 'GNUtoo' Carikli  writes:

> - The haskell is not and was never bootstrapped from source. Though
>   the binaries used to bootstrap it changed over time.

We do, however, build GHC 4 from source; there is only a small amount of
*generated* C files that we currently can’t avoid.  We build up from GHC
4 to version 7.

Someone needs to close the link between that build output and the
more recent GHCs we still hold.

-- 
Ricardo

Re: A Forum for Guix Users

[Ricardo Wurmus]

Pjotr Prins  writes:

> If the xapian indexer also analysed mailing list output (through
> publicinbox, for example) and maybe debbugs and IRC logs it would be
> complete. 

It does index the IRC logs, but there’s a bug in the deployment:


https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/goggles.scm#n56

>> 5. Substitute availability is good, but download speeds can be poor
>> due to peering issues.
>
> That is a matter of money, I guess.

Peering issues can only be avoided by distributing the artifacts to many
different locations.  We’ve previously paid for distribution via AWS,
which was rather expensive.

It might be better to revive the distribution over IPFS.

-- 
Ricardo

Re: npm has irreproducible install behavior

[Ricardo Wurmus]

Hi Jelle,

>> I’ve tested this little addition to the build phases of node-lts on an
>> affected system:
>>
>> --8<---cut here---start->8---
>>(add-after 'install 'do-not-ignore-links
>>  (lambda* (#:key outputs #:allow-other-keys)
>>(with-directory-excursion
>>(string-append (assoc-ref outputs "out")
>>   "/lib/node_modules/npm/node_modules")
>>  (substitute* "pacote/lib/fetcher.js"
>>   (("\\/Link\\$\\/.test\\(entry.type\\)") 
>> "false")
>>   (("\\/File\\$\\/.test\\(entry.type\\)")
>>"/(File|Link)$/.test(entry.type)"))
>> --8<---cut here---end--->8---
>
> The proposed change seems fine to me, provided our QA shows a pretty
> green button at some point. Could you send a patch that also includes a
> comment referencing the upstream bug report?

I changed it to patch node-tar instead, because the patch to fetcher.js
would cause *some* files to be copied to the wrong location.

I’ll send the updated patch to guix-patches today.

-- 
Ricardo