Re: Error handling when 'guix substitute' dies

[Philip McGrath]

Hi,

On 3/29/24 11:10, Ludovic Courtès wrote:
>
> [...]


OK, I’ll push it shortly, but…

Lars Bilke  skribis:


thanks Ada for bringing this issue up again. I get the same error on
`guix pull` almost always when I am on my enterprise
network. Re-running `guix pull` a second time also almost always then
runs fine. I checked with our IT: nothing suspicious on the network,
i.e. no firewall blocking.

I never experienced the error on my home network.


… your reports make me think there’s a bug lurking somewhere that
perhaps only manifests under some precise networking or timing
conditions.

Could the two of you run the following command in a loop to see whether
it’s easy to reproduce that GnuTLS error?

   guile -c '(use-modules (guix http-client) (ice-9 binary-ports))
 (get-bytevector-all (http-fetch "https://ci.guix.gnu.org/nix-cache-info;))'

If you can reproduce it, could you capture the strace output of the
process?  You would run the command above prefixed by:

   strace -o log.strace -s 300 …



I don't know if the root cause is related, but this reminded me of some 
networking errors I sometimes get accessing substitutes. I had the luck 
(good or bad?) to get an example while building 
, so I thought I'd report.



ice-9/boot-9.scm:1685:16: In procedure raise-exception:
ERROR:
  1. :
  uri: #< scheme: https userinfo: #f host: "bordeaux-us-east-mirror.cbaines.net" 
port: #f path: "/nar/lzip/r8w68mhsvqlkvazncvfn36jcpvz404x6-chez-fmt-0.8.11.tar.gz" query: #f 
fragment: #f>
  code: 404
  reason: "Not Found"
  headers: ((server . "nginx") (date . #) (content-type application/json (charset . 
"utf-8")) (content-length . 35) (connection keep-alive))
  2. : 
"https://bordeaux-us-east-mirror.cbaines.net/nar/lzip/r8w68mhsvqlkvazncvfn36jcpvz404x6-chez-fmt-0.8.11.tar.gz:
 HTTP download failed: 404 (\"Not Found\")"
substitution of 
/gnu/store/r8w68mhsvqlkvazncvfn36jcpvz404x6-chez-fmt-0.8.11.tar.gz failed
guix build: error: corrupt input while restoring archive from #


The last time this happened, I kept rerunning the command a bunch of 
times. When it succeeded with the first substitute with the problem, it 
hit the problem with another substitute, for several rounds, but 
eventually the whole thing succeeded.


In this incident, the command failed a few times, so I ran:

guix build --no-substitutes --source chez-fmt

and then ran my original command again, successfully.

Hope this is useful,
Philip

Re: the right to rewrite history to rectify the past (was Re: Concerns/questions around Software Heritage Archive)

[Philip McGrath]

On Thu, Mar 21, 2024, at 11:11 AM, MSavoritias wrote:
> On 3/21/24 17:08, Giovanni Biscuolo wrote:
>> […]
>> I don't understand how using petnames, uuids or even a re:claimID
>> identity (see below) could solve the problem with "rewriting history" in
>> case a person wishes to change his or her previous _published_ name
>> (petname, uuid...) in an archived content-addressable storage system.
>
> It doesnt solve the problem of rewriting history. It solves the bug of 
> having names part of the git history.
>
> see also https://gitlab.com/gitlab-org/gitlab/-/issues/20960 for Gitlab 
> doing the same thing.
>

Unless I’m missing something, the linked Gitlab issue seems to be a proposal by 
someone in February 2018 that Gitlab adopt some system of using UUIDs instead 
of author information. There was fairly limited discussion, with the last 
comment in May 2018. There does not seem to have been a consensus supporting 
the proposal, and I’m not seeing any indication that Gitlab plans to implement 
the proposal.

Furthermore, the author and committer metadata are not the only places where 
people’s names appear in Guix. For example, I know some font packages that 
mention the name of the font designer in the package’s description. More 
broadly, Guix also refers to package sources by their content hashes: most 
sources probably contain some people’s names, and any of these could face the 
same problems as names directly included in the Guix Git repository.

I strongly believe in the importance of protecting trans people from 
harassment. I don’t know how to solve the tension with long-term bit-for-bit 
reproducibility. 

Philip

Re: Git-LFS or Git Annex?

[Philip McGrath]

Hi,

On Sun, Jan 28, 2024, at 5:33 AM, Nicolas Graves via Development of GNU Guix 
and the GNU System distribution. wrote:
> I've left git-annex for git-lfs, I'll just add a few points about
> git-lfs.
>
>
> On 2024-01-24 18:39, Giovanni Biscuolo wrote:
>
>> Hi Ludo’
>>
>> Ludovic Courtès  writes:
>>
>> [...]
>>
>>> The question boils down to: Git-LFS or Git Annex?
>>>
>>> From a quick look (I haven’t used them), Git-LFS seems to assume a
>>> rather centralized model where there’s an LFS server sitting next to the
>>> Git server¹.  Git Annex looks more decentralized, allowing you to have
>>> several “remotes”, to check the status of each one, to sync them, etc.²
>>> Because of this, Git Annex seems to be a better fit.
>
> This is not always true. Git-LFS also has the concept of Custom Transfer
> Agents, which in some cases do not need a running server. One example is
> lfs-folderstore, which can simply use a remote directory as a LFS remote.
>

This is very interesting and could have me look at Git LFS again.

>>
>> I've never used Git-LFS for my media repository (and will never use it,
>> never).
>>
>> AFAIK this two advantages of git-annex vs Git-LFS are still valid today:
>>
>> --8<---cut here---start->8---
>>
>> A major advantage of git annex is that you can choose which file you
>> want to download.
>>
>> You still know which files are available thanks to the symlinks.
>>
>> For example suppose that you have a directory full of ISO files. You can
>> list the files, then decide which one you want to download by typing:
>> git annex get my_file.
>
> This is true, but
> 1) you can still adapt your filters to ignore certain files, although
> more inconvenient, it's not impossible
> 2) in practice, I think most uses don't need to. I just now that all .lz
> files in a directory are to LFS, no questions asked.
>

I think you could probably use the fairly new “sparse checkout” feature of Git 
to get only some Git LFS files.

>>
>> Another advantage is that the files are not duplicated in your
>> checkout. With LFS, lfs files are present as git objects both in
>> .git/lfs/objects and in your working repository. So If you have 20 GB of
>> LFS files, you need 40 GB on your disk. While with git annex, files are
>> symlinked so in this case only 20 GB is required.
>
> True.

This raises a question for me about Git Annex: if the files are symlinks, if I 
edit a file, is the change detected and tracked? Could the old version of the 
file potentially be lost, if I don’t take care to have it synced elsewhere 
before editing?

Philip

Re: Git-LFS or Git Annex?

[Philip McGrath]

Hi,

On 1/24/24 10:22, Ludovic Courtès wrote:


The question boils down to: Git-LFS or Git Annex?

[...]

What’s your experience?  What would you suggest?



I have a few times had a problem for which I thought Git LFS might be a 
solution, and each time I have ended up ripping out Git LFS in 
frustration before long.


I have not used Git Annex. I have looked into it a few times, but each 
time I decided it was too complex or not quite suitable for my use-case 
in some way. On the other hand, I have heard good things about it from 
people who have used it: in particular, I believe Morgan Lemmer-Webber 
(CC'ed) used it to manage a large set of art history images.


The main thing in this context that still isn't clear to me from by 
reading so far is how sharing lists of remotes works with Git Annex. In 
plain Git, remotes are part of the local state of a particular clone, 
not distributed as part of the repository. For the objectives here, 
though, a lot of the benefit would seem to be having many copies in 
synchronized, possibly "special" remotes so that anyone trying to get 
the videos would have plenty of ways to get them. I'm not sure to what 
extent Git Annex does that out of the box.


I did see that Git Annex can use Git LFS as a "special remote".

There are also two other approaches I think would be worth at least 
considering:


1. Just use Git

While the limitations of Git for storing large media files are well 
known, I have found it to be good enough for several use-cases, and it 
has the strong advantage of not requiring additional tools. My 
impression is that a significant factor in people using Git LFS, in 
particular, is the limit on repository size imposed by the popular 
hosting providers. There are strategies within Git to avoid having to 
download unwanted artifacts, including creating branches with unrelated 
histories, shallow clones (e.g. --depth=1 --single-branch), partial 
clones [1][2][3] (e.g. --filter=blob:none), and sparse checkouts [4][5], 
with the later two being fairly new features.


[1]: https://git-scm.com/docs/partial-clone
[2]: 
https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---filterltfilter-specgt
[3]: 
https://git-scm.com/docs/git-rev-list#Documentation/git-rev-list.txt---filterltfilter-specgt

[4]: https://git-scm.com/docs/git-sparse-checkout
[5]: https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---sparse

2. Mirror URLs

Another approach would be just to make each video available at a few 
URLs and have Guix origins with the list. If one of the available URLs 
were the Internet Archive, it would have a high degree of assurance of 
long-term preservation. I think the biggest downside is that this might 
not help much with managing the collection of videos.


Philip

Re: Mixing GPL and non-copyleft code in source files

[Philip McGrath]

On Wed, Jan 3, 2024, at 3:32 PM, Vagrant Cascadian wrote:
> On 2024-01-03, Wojtek Kosior via wrote:
>> Before getting back to the discussion, please let me ask 1 question.
>> Assume I submit a patch series that adds some useful and needed code and
>> includes a copyright notice with a promise, like this
>>
>> ;;; Copyright © 2023 Wojtek Kosior 
>> ;;; Wojtek Kosior promises not to sue for violations of this file's license.
>>
>> Will this weirdness be considered minor enough to tolerate?  I made
>> sure the promise line takes below 78 chars.
>
> I am not at all a lawyer, but this seems like an entirely different
> license, and at the very least a pragmatic headache.
>

In the spirit of sticking to the concrete issue, there is precedent for GNU 
software accepting changes that the contributor has placed in the public 
domain: see in particular 
. In fact, 
since Guix leaves copyright ownership with individual contributors, it seems 
like there's no way to stop contributors from also licensing their 
contributions under additional licenses.

I think it could make sense to include in the commit message something like, 
"I, Alyssa P. Hacker, dedicate my contribution in this commit to the public 
domain under CC0-1.0.". That would make it clear exactly what changes are 
covered.

Philip

Re: Should commits rather be buildable or small

[Philip McGrath]

On 12/10/23 18:20, Attila Lendvai wrote:

FWIW, this commit policy has always bothered me as a newcomer to
Guix. pretty much everywhere else it's a major offence against your
colleagues to commit something that breaks the build in any way.



In the last few months I’ve repeatedly seen assertions in a similar
style as this one. They always genuinely surprise me, and it’s probably
not just because I’m oblivious and out of touch.



well, both point of views are reasonable. they just make different tradeoffs.



I find it hard to see any benefit to anyone from making commits so small 
that they are known to break things that will be fixed later in the same 
series. Even aside from `guix time-machine`, substitute building, and 
the like, a human reading the diff won't be able to see what the true 
impact of the change is.


On the concrete issue:


On 12/10/23 10:28, Saku Laesvuori wrote:
>> However, in each commit at least the package touched in that
>> commit ought to build.
> This should, of course, be theoretically possible with at least one
> update order but I don't know how would I discover that order (more
> efficiently than by trial and error. I don't want to try ~800² different
> combinations).

Preparing a large set of updates like this is already a great deal of 
work. It does not seem to me like a good use of volunteers' time to ask 
them to break such an update into hundreds of tiny pieces, especially 
not if the result is hundreds of broken commits to Guix.


Philip

Re: git send-email

[Philip McGrath]

Hi,

On Sun, Nov 19, 2023, at 12:27 PM, Felix Lechner wrote:
> Hi Philip,
>
> On Sun, Nov 19 2023, Philip McGrath wrote:
>
>>  1. Can we reorganize things on the website so that the non-"devel" manual 
>> is much more prominent than the other one?
>
> Was the "non" possibly a typo? In my opinion, the non-devel manual
> should no longer be published.
>

Yes, a typo, or perhaps another example of how confusing this is!

> Please feel free to chime in under
>
>   https://issues.guix.gnu.org/51000
>
>>  2. I wish there were a Guix package I could install to read the HTML 
>> documentation locally.
>
> As an Emacs user, I select "guix" after hitting "C h R". I can search
> the index with "i".
>
> A standalone 'info' executable may provide a similarly convenient kind
> of local access to the manual.
>

I have used the local info version, but I like the HTML version better.

Philip

Re: git send-email

[Philip McGrath]

On Sun, Nov 19, 2023, at 10:59 AM, Nicolas Débonnaire wrote:
> Well, I just realised there is two version of the manual available and I was 
> looking at the wrong one from the beginning. (https://guix.gnu.org/en/manual/)
> It looks like it's been fixed in the latest version. 
> (https://guix.gnu.org/en/manual/devel/)
> (sorry for the duplicate mail, I used reply instead of reply all in the 
> previous mail)
> 

This seems to regularly cause confusion for a lot of people (and has confused 
me). AIUI, the "devel" manual seems like a misnomer: it is the manual for Guix 
from the "master" branch, which, unless you never run `guix pull`, is the 
relevant Guix for almost anything. It's the one that ends up in 
~/.config/guix/current.

It's possible to be running daemon that is different from your "current" Guix. 
If so, AND if you happen to have the daemon from the latest Guix release (as 
opposed to an older or newer Guix), then the non-"devel" manual would be 
relevant **for functionality specific to the daemon**. That seems to me like a 
very specific scenario, and AFAIK it's the only scenario when the non-"devel" 
manual is relevant.

 1. Can we reorganize things on the website so that the non-"devel" manual is 
much more prominent than the other one?

 2. I wish there were a Guix package I could install to read the HTML 
documentation locally.

Re: Do substitutes download slowly for you? / Speeding up substitute delivery/mirrors

[Philip McGrath]

Hi,

On Thu, May 25, 2023, at 5:04 PM, Felix Lechner wrote:
> Hi Philip,
>
> On Thu, May 25, 2023 at 1:49 PM Philip McGrath  
> wrote:
>>
>> Here are results from Florida, US:
>>
>> 2023-05-25 16:28:07 (14.9 MB/s)
>> 2023-05-25 16:28:53 (52.9 MB/s)
>> 2023-05-25 16:29:37 (10.5 MB/s)
>
> Wow, those are the best Guix speeds I have seen yet, perhaps aside
> from folks connected to DFN. [1] Would you please share some details
> about your ISP? From a peering perspective, I think the most
> interesting part is your ASN. [2] Thanks!
>

This was on a residential Comcast connection which seems to be AS7922. I tried 
to avoid any local slowdowns by doing this with a wired connection at a time 
when there was little other activity on the machine or the LAN and downloading 
to a tmpfs.

Anecdotally, the speeds for servers *other* than bordeaux-us-east-mirror were 
faster in this trial than feels typical here. The results I reported at 
https://lists.gnu.org/archive/html/guix-devel/2022-07/msg00320.html (e.g. 
8.17MB/s for bordeaux.guix.gnu.org) might be more usual. Subjectively, my 
impression of the non-mirrored substitute servers is that often the speeds is 
ok, but when they are slow it is very, very slow.

Philip

Re: Do substitutes download slowly for you? / Speeding up substitute delivery/mirrors

[Philip McGrath]

Hi,

On Thu, May 25, 2023, at 9:52 AM, Christopher Baines wrote:
> Hey!
>
> I was reminded again about substitute download speeds outside of Europe
> again today.
>
> There was some feedback when I sent out a message to guix-devel [1], [2]
> a while back. I think the rough summary is that there's anecdotal
> evidence that a US mirror helps.
>
> 1: https://lists.gnu.org/archive/html/guix-devel/2022-05/msg00203.html
> 2: https://lists.gnu.org/archive/html/guix-devel/2022-07/msg00163.html

Thanks again for setting this up! The us-east mirror makes a big difference for 
me. The wget output is below, but anecdotally, last week, when I installed Guix 
on a new Debian installation, my first `guix pull` failed maybe three times due 
to substitute timeouts before I enabled the mirror. After enabling the mirror, 
it succeed on the first attempt.

Tangentially, a few months ago I was using a very poorly connected network that 
had the additional annoying property of disconnecting every 10 minutes or so. I 
got stuck downloading the beginning of one large nar enough times that 
eventually I gave up. I found, though, that `wget -c` was able to complete the 
download (though it took several attempts). Adding some similar resume 
functionality might at least help to mitigate the impact of slow downloads.

>
> France:wget 
> https://bordeaux.guix.gnu.org/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
> US:wget 
> https://bordeaux-us-east-mirror.cbaines.net/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
> Singapore: wget 
> https://bordeaux-singapore-mirror.cbaines.net/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
>
>
> So please share the output from wget and if you're comfortable doing so,
> the rough real world location of where the computer doing the
> downloading is.
>

Here are results from Florida, US:

philip@avalon:/tmp$ wget 
https://bordeaux.guix.gnu.org/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
--2023-05-25 16:27:53--  
https://bordeaux.guix.gnu.org/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
Resolving bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)... 185.233.100.56, 
2a0c:e300::58
Connecting to bordeaux.guix.gnu.org 
(bordeaux.guix.gnu.org)|185.233.100.56|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 208615205 (199M) [text/plain]
Saving to: ‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0’

078vr3r8mn3yrwzwxw 100%[=>] 198.95M  10.1MB/sin 13s 

2023-05-25 16:28:07 (14.9 MB/s) - 
‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0’ saved [208615205/208615205]

philip@avalon:/tmp$ wget 
https://bordeaux-us-east-mirror.cbaines.net/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
--2023-05-25 16:28:49--  
https://bordeaux-us-east-mirror.cbaines.net/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
Resolving bordeaux-us-east-mirror.cbaines.net 
(bordeaux-us-east-mirror.cbaines.net)... 5.161.49.48
Connecting to bordeaux-us-east-mirror.cbaines.net 
(bordeaux-us-east-mirror.cbaines.net)|5.161.49.48|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 208615205 (199M) [text/plain]
Saving to: ‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0.1’

078vr3r8mn3yrwzwxw 100%[=>] 198.95M  55.8MB/sin 3.8s

2023-05-25 16:28:53 (52.9 MB/s) - 
‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0.1’ saved 
[208615205/208615205]

philip@avalon:/tmp$ wget 
https://bordeaux-singapore-mirror.cbaines.net/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
--2023-05-25 16:29:17--  
https://bordeaux-singapore-mirror.cbaines.net/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
Resolving bordeaux-singapore-mirror.cbaines.net 
(bordeaux-singapore-mirror.cbaines.net)... 64.176.80.78, 
2401:c080:1400:71df:5400:4ff:fe73:757d
Connecting to bordeaux-singapore-mirror.cbaines.net 
(bordeaux-singapore-mirror.cbaines.net)|64.176.80.78|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 208615205 (199M) [text/plain]
Saving to: ‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0.2’

078vr3r8mn3yrwzwxw 100%[=>] 198.95M  11.4MB/sin 19s 

2023-05-25 16:29:37 (10.5 MB/s) - 
‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0.2’ saved 
[208615205/208615205]

Re: tracking /etc/profile.d/*guix.sh in downstream distributions

[Philip McGrath]

On Friday, May 19, 2023 6:10:53 PM EDT Vagrant Cascadian wrote:
> Philip McGrath recently pointed out that the /etc/profile.d/guix.sh
> snippet had not been updating in the Guix packaging for Debian:
> 
>   https://bugs.debian.org/1036304
> 
> (and as bonus complication, was also renamed to zzz-guix.sh)
> 
> The reason I never noticed before is because it is actually a manual
> process, with this script embedded in the sys_create_init_profile()
> function of etc/guix-install.sh ... and thus easy to miss when updating
> packaging.
> 
> What would be the impact of separating the /etc/profile.d/zzz-guix.sh
> into a separate file upstream? Would guix-install.sh need to be adjusted
> to guix-install.sh.in so as to embed the contents of this file?
> 

I think this would also make it easier for Guix contributors to edit the file. 
When I sent a patch for it, Ludo’ and I both got to learn about the obscure 
feature of Bash here documents it uses: https://issues.guix.gnu.org/56050#11
Ludo’s informal poll results suggest that others would be confused, too:
https://toot.aquilenet.fr/@civodul/108590952454715930

> Alternately, maybe guix-install.sh could be updated to optionally output
> or generate the zzz-guix.sh file either by passing commandline arguments
> or some other conditional mechanism?
> 

One reason I use Vagrant's packaging for Debian is that otherwise, Guix has no 
mechanism at all for updating the /etc/profile.d/ script, the default 
authorized substitute keys, etc. That's not exactly the same problem, but a 
solution that works well for downstream packagers seems like it make updates 
for other users an easier problem to address.

Philip

signature.asc
Description: This is a digitally signed message part.

Re: Potential removal of unmaintained node-openzwave-shared

[Philip McGrath]

Hi,

On Wed, Mar 22, 2023, at 8:08 AM, Jelle Licht wrote:
> Hey guix,
>
> In getting `Node.js' updated to a more recent LTS version[0], I found
> out that node-openzwave-shared no longer builds with modern versions of
> node [1]; random people on the Internet seem to indicate that
> the hip new thing is Z-Wave JS [2].
>
> Long story short, what is our de facto policy here?
>
> 1) Keep around a copy of Node 14 and all node-openzwave-shared deps,
>even after the End Of Life of 2023-04-30
> 2) Remove node-openzwave-shared, and move to Node 18 whenever possible
>without this package.
> 3) Patch node-openzwave-shared' so it builds with newer versions of
>Node, and move to Node 18.
> 4) Remove node-openzwave-shared, move to Node 18, package the relevant
>parts of Z-Wave JS.
>
> I don't have the time nor means for anything but option 2) myself, so if
> consensus deems any of the other options a better way forward,
> volunteers are invited to apply :-)
>
> [0]: https://issues.guix.gnu.org/59188
> [1]: https://github.com/OpenZWave/node-openzwave-shared/issues/398
> [2]: https://github.com/zwave-js?type=source

I added this package, so I have some interest in trying options 3 or 4, but I 
don't think this should block Node 18 in any case.

Is there a log message for the build failure somewhere? I don't see any details 
at [2], and I'm a bit surprised by the failure.

It's true that open-zwave and, by extension, node-openzwave-shared currently 
have no active maintainers. On the other hand, AIUI, it still works fine, and 
e.g. Debian Bookworm still has libopenzwave1.6 packaged: my impression is that 
the only changes in several years, even before the loss of maintainers, have 
been adding XML definitions for new hardware.

On the other hand, while I've heard good things about Z-Wave JS and on 
principle would favor a memory-safe language, the large number of NPM 
dependencies seem likely to make it difficult to package. (The person who 
recommended it in [2] is one of its maintainers, FYI.)

-Philip

Re: Qt in core-updates (was: KDE in core-updates)

[Philip McGrath]

Hi,

On Sunday, February 26, 2023 7:44:20 AM EST Andreas Enge wrote:
> 
> In any case, I realised that we are still compiling most packages (including
> KDE) with Qt 5, which is seriously outdated (not maintained any more in the
> free version since May 2021). Qt 6.3 support will end in April 2023, that
> of the current version Qt 6.4 in September 2023. So we have the work carved
> out for a (yet to be created) Qt/KDE team.
> 

Note that KDE maintains a patch collection for Qt 5.15:
https://community.kde.org/Qt5PatchCollection

There was an announcement here:
https://dot.kde.org/2021/04/06/announcing-kdes-qt-5-patch-collection

Patches are exclusively backports of bugfixes that have already been committed 
to upstream Qt 6, except that patches for Qt 5.15 components removed in Qt 6 
are also accepted. The patches are curated by a small group of KDE developers 
who also have commit privileges in the upstream Qt Project (as distinct from 
the Qt Company).

I think out Qt 5 packages should be based on the KDE patch collection.

The patches are maintained in Git repositories with the same structure as 
upstream Qt. They could be extracted with `git format-patch v5.15.3-lts-
lgpl..origin/kde/5.15` and added to "gnu/packages/patches/", or we could just 
change the origins for Qt 5 to point to KDE's repositories, e.g. this one for 
qt-base: https://invent.kde.org/qt/qt/qtbase/-/tree/kde/5.15

Most KDE stuff can build against Qt 6, but it will continue targeting Qt 5.15 
until KDE Frameworks 6 and Plasma 6 are released (maybe as soon as the end of 
this year?).

I'm not a Qt or C++ developer, but I'm a long-time KDE user, and I'm really 
excited that people have been working to get KDE into Guix: thank you!

-Philip

signature.asc
Description: This is a digitally signed message part.

Re: Can zig-build-system be an alternative to the gnu-build-system?

[Philip McGrath]

Hi Pjotr,

On Wed, Feb 1, 2023, at 5:11 AM, Pjotr Prins wrote:
>
> There have been some older discussions about creating our own
> replacement for autotools, cmake and others. I often fight these make
> make systems - and meson and/or language specific build systems. It is
> a time waster for programmers and none of these systems leverages the
> power of Guix itself. I kinda settled for cmake because, even though
> it is an effing dragon, at least I can make it work (pun intended).
>
> We need someone with deep experience in build systems to write a guile
> replacement - generating ninja is one idea. That is my opinion :). I
> would love a simple way to describe a project in guile. It should not
> be too hard actually (famously that is how these projects start and
> turn out to be a real time sink). Maybe someone wants to try with
> guidance from us, or maybe we can do it when we get bored some day.
>

You might be interested in Zuo, which was created to replace the work of `make` 
in the build system for Racket and Racket's branch of Chez Scheme. (The name 
comes from a Chinese word for "make".) It's packaged for Guix: 
https://packages.guix.gnu.org/packages/zuo/

The documentation is at: https://docs.racket-lang.org/zuo/

One important aspect is that the primary implementation is *not* in Racket or 
Guile: a single C source file implements the tiny Scheme-like language, and 
most of the functionality is implemented in the Zuo language. This simplifies 
bootstrapping, and it also means that, if your project already needs a C 
compiler, you can add Zuo without adding a build dependency on another 
scripting language. (On unfriendly systems like Windows, even requiring Python 
can be a significant complication.)

By design, Zuo does not try to replace the `configure` part of your build 
system: Racket uses Autoconf, and Chez has its own custom `configure` script. 
Similarly, it works well to have a stub Makefile that just runs Zuo, and Zuo 
even cooperates with the GNU Make jobserver. When Racket adopted Zuo, almost 
all make-based workflows continued to work unchanged.

The build system part of Zuo is modeled on Shake (used by GHC). The paper 
"Build Systems à la Carte" is a nice overview of the design space: 
https://dl.acm.org/doi/pdf/10.1145/3236774

The thread about replacing Racket's build scripts is here: 
https://github.com/racket/racket/pull/4179

-Philip

Re: RFC: libgit2 is slow/inefficient; switch to git command?

[Philip McGrath]

Hi,

On Tue, Nov 22, 2022, at 10:39 AM, zimoun wrote:
> Hi,
>
> On Mon, 21 Nov 2022 at 21:21, Maxim Cournoyer  
> wrote:
>
>> Given that:
>>
>> * the git CLI doesn't suffer from such poor performance;
>> * This kind of performance problem has been known for years in libgit2
>>   [0] with no fix in sight;
>> * other projects such as Cargo support using the git CLI and that
>>   projects are using it for that reason [1];
>
> And I would add the lack of «Support for shallow repositories» [1].
>
> 1: 
>

>
> PS: For the record, Software Heritage, which ingests *a lot* of Git
> repositories, relies on Dulwhich [2] (pure Python implementation), IIUC.
>
> 2: 

Along those lines, there’s an implementation of clone/checkout in pure Racket 
(for the package manager) that could probably be ported to Guile relatively 
easily. I’d expect libgit2 to be faster for the things that it supports, but 
the Racket implementation does support shallow checkout, so it might pay off if 
that skips a lot of work.

Code: 
https://github.com/racket/racket/blob/master/racket/collects/net/git-checkout.rkt
Docs: https://docs.racket-lang.org/net/git-checkout.html

(More broadly, I haven’t investigated performance issues, but my basic 
inclination would be toward improving libgit2 over running the git executable.)

-Philip

Re: What 'sh' should 'system' use?

[Philip McGrath]

On Sunday, October 16, 2022 3:04:45 AM EDT Liliana Marie Prikler wrote:
> Am Samstag, dem 15.10.2022 um 19:23 -0400 schrieb Philip McGrath:
> > On Saturday, October 1, 2022 12:54:27 PM EDT Ludovic Courtès wrote:
> > > Hello!
> > > 
> > > Philip McGrath  skribis:
> > > > 1) If we want to continue to hard-code a specific shell into
> > > > Glibc, I
> > > > think we should document the decision (for example, why 'bash-
> > > > static' vs.
> > > > 'bash- minimal'?) […]
> > > 
> > > The choice of ‘bash-static’ rather than ‘bash-minimal’ is motivated
> > > by
> > > the fact that, in (gnu packages commencement), we want to make sure
> > > ‘glibc-final’ does not retain references to its build-time
> > > environment.
> > > See #:allowed-references in ‘glibc-final’.
> > 
> > This makes sense as far as using 'bash-static' in Glibc. The aspects
> > I'm unsure of are:
> > 
> >  1. If I'm packaging software that implements a function like
> > 
> > 'system' (e.g. Racket, SML/NJ, Chez Scheme, etc.), should I use
> > 'bash-minimal' or 'bash-static'?
> > 
> >  2. Do we really need 'bash-minimal' at all? Why not just replace it
> > 
> > with 'bash-static'?
> 
> We already explained those two to you. Racket, SML/NJ, Chez Scheme et
> al. are not bootstrap-relevant, thus they can use bash-minimal.  Unlike
> bash-static, bash-minimal can be grafted, i.e. a security bug in bash(-
> minimal) that necessitates a version bump or similar does not cause a
> world rebuild.  A security bug in bash-static does.
> 

I don't think I understand this. Does it mean that, in the following, I am 
running a Bash that wouldn't have security bugs fixed? If so, that seems quite 
bad!

--8<---cut here---start->8---
philip@bastet:/tmp$ cat run-bshell.scm 
(use-modules
 (guix build-system gnu)
 (guix gexp)
 ((guix licenses) #:prefix license:)
 (guix packages))
(define src
  (plain-file "run-bshell.c"
  "
#include 
#include 
#include 
int main(void)
{
  execl(_PATH_BSHELL,
_PATH_BSHELL,
\"-c\",
\"echo \" _PATH_BSHELL,
(char *)NULL);
}
"))
(package
  (name "run-bshell")
  (version "0")
  (source src)
  (build-system gnu-build-system)
  (arguments
   (list
#:phases
#~(modify-phases %standard-phases
(delete 'configure)
(replace 'build
  (lambda args
(invoke "gcc" "-o" "run-bshell" #$src)))
(delete 'check)
(replace 'install
  (lambda args
(install-file "run-bshell" (string-append #$output "/bin")))
  (home-page #f)
  (synopsis #f)
  (description #f)
  (license license:cc0))
philip@bastet:/tmp$ guix shell --rebuild-cache --container --no-cwd -f run-
bshell.scm -- run-bshell
substitute: updating substitutes from 
'https://bordeaux-us-east-mirror.cbaines.nsubstitute: updating substitutes from 
'https://bordeaux-us-east-mirror.cbaines.net'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 
100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
  /gnu/store/q5bib9dgaxzag29a2l4b833mm5l12dx3-profile.drv
  /gnu/store/r45khn1mq17fc0xsab1yszii85ynsm2j-run-bshell-0.drv

building /gnu/store/r45khn1mq17fc0xsab1yszii85ynsm2j-run-bshell-0.drv...
building CA certificate bundle...
listing Emacs sub-directories...
building fonts directory...
building directory of Info manuals...
building profile with 1 package...
/gnu/store/720rj90bch716isd8z7lcwrnvz28ap4y-bash-static-5.1.8/bin/sh
philip@bastet:/tmp$ 
--8<---cut here---end--->8---

signature.asc
Description: This is a digitally signed message part.

Re: What 'sh' should 'system' use?

[Philip McGrath]

On Saturday, October 1, 2022 12:54:27 PM EDT Ludovic Courtès wrote:
> Hello!
> 
> Philip McGrath  skribis:
> > 1) If we want to continue to hard-code a specific shell into Glibc, I
> > think we should document the decision (for example, why 'bash-static' vs.
> > 'bash- minimal'?) […]
> 
> The choice of ‘bash-static’ rather than ‘bash-minimal’ is motivated by
> the fact that, in (gnu packages commencement), we want to make sure
> ‘glibc-final’ does not retain references to its build-time environment.
> See #:allowed-references in ‘glibc-final’.
> 

This makes sense as far as using 'bash-static' in Glibc. The aspects I'm unsure
of are:

 1. If I'm packaging software that implements a function like 'system'
(e.g. Racket, SML/NJ, Chez Scheme, etc.), should I use 'bash-minimal' or
'bash-static'?

 2. Do we really need 'bash-minimal' at all? Why not just replace it with
'bash-static'?

In particular, AFAICT, 'bash-minimal' currently has a reference to
'bash-static' via Glibc:

--8<---cut here---start->8---
$ guix size bash-minimal 
store item   totalself
/gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33  38.3
36.6  50.4%
/gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib  71.7
33.4  45.9%
/gnu/store/720rj90bch716isd8z7lcwrnvz28ap4y-bash-static-5.1.81.7 
1.7   2.3%
/gnu/store/chfwin3a4qp1znnpsjbmydr2jbzk0d6y-bash-minimal-5.1.8  72.7 
1.0   1.4%
total: 72.7 MiB
--8<---cut here---end--->8---

> > 2) If we want to make 'sh' a weak/dynamic reference, I think we should
> > strongly consider arranging to make it available at '/bin/sh' when
> > present. I expect this option would require less patching of other
> > packages *by far* than any other approach.
> 
> This is not a viable option because build containers lack /bin/sh.
> 

Right, this option would depend on making /bin/sh exist in the build
environment.

I'd hoped this might be possible without having to change the daemon, but the
ways I've tried so far haven't worked. I tried `(mkdir-p "/bin")`, but the
build user apparently doesn't have sufficient permissions. Then I tried
creating a nested container using `call-with-container` in which I could
bind-mound the directory from 'bash-static' at '/bin', but I hit permissions
errors that way, too. I also thought there might be a way to pass the daemon
options like 'build-chroot-dirs' to have it set up /bin/sh before it drops
privileges, but I couldn't figure out how to do that.

> Overall, I think the current situation is a reasonable tradeoff.  It
> forces us to do some patching, indeed, but I think that’s acceptable:
> we’re talking about a handful of packages.
> 
> WDYT?
> 
> Ludo’.

The patching itself isn't so bad, and, as you say, it's limited to at least
a relatively small number of packages. However, the fact that Glibc retains a
reference to 'bash-static' affects nearly every package. It doesn't affect them
very much, to be sure! But I think it does prevent using
`guix shell --container` to create containers without a shell, and it likewise
seems difficult to experiment with different shells. Or maybe it's really just
that it disturbs my sense of aesthetics.

-Philip

signature.asc
Description: This is a digitally signed message part.

Re: Proposal: A new build-system API

[Philip McGrath]

Hi,

On 9/26/22 14:07, Mája Tomášek wrote:

For example, there's currently no guarantee
that #:phases will contain list of functions and the error from that
will be cryptic. Contrast that with the service-configuration API. There
are field sanitizers that ensure that the configuration is propper and
easily one can introspect what the build system accepts and what are the
defaults.



This in particular is absolutely a valid point. It reminds me that there 
was some discussion a while ago about error checking/reporting generally 
in Guix in which I advocated for adopting contracts à la Racket. I'd 
started putting together a minimal implementation at the time, but I got 
diverted by other things: I'll try to get back to it.


-Philip

Re: What 'sh' should 'system' use?

[Philip McGrath]

Hi,

On 9/19/22 03:07, Liliana Marie Prikler wrote:

Am Sonntag, dem 18.09.2022 um 20:13 -0400 schrieb Philip McGrath:

On the other hand, even Nix puts '/bin/sh' at its usual path: we
are really quite an outlier in this respect. (IIUC, Nix also has 
'/usr/bin/env', but no other utilities at FHS paths.)
We are not.  We provide both /bin/sh and /usr/bin/env.  If you're 
talking about the build container then that's a much smaller 
distinction.




Yes, I'm talking about the build container. But for the build container,
programs/libraries that use "/bin/sh" would work unmodified.

More broadly, I now think it would be better in we embedded zero 
references to copies of Bash in libc.

I don't think we can do that without breaking system.



When "/bin/sh" is not available at runtime, I think libc's `system`
ought to return 127, and other `system`-like functions should raise
exceptions or whatever the idiomatic way is to signal failure. Of
course, we will presumably need to make "/bin/sh" available in many more
places, but don't think it's surprising for programs that need to run
shell commands to fail in the absence of a shell.

However, giving every program using Glibc a hard dependency on 
Bash—and on a particular Bash executable—seems like a much bigger 
imposition.

We're talking 1.7 MiB here.  Certainly a "big" imposition, but
nothing in comparison to the things you need in the store for
bootstrapping purposes.  Also note that bash-minimal, while only
taking up 1.0 MiB for itself, requires both glibc and gcc:lib, which
apart from creating a cycle does blow up its closure size quite a
bit.



I'm less concerned with the literal size than with the significance of
putting a specific shell so near the root of most dependency graphs: I
tried to give examples in my reply to Maxime, like creating containers
without a shell.


In versions of glibc before 2.1.3, [...] system() always returned
1 [...].

Note that always returning non-zero is required by POSIX 2017.



To quote the whole paragraph from
<https://pubs.opengroup.org/onlinepubs/9699919799/functions/system.html>:


Note that, system(NULL) is required to return non-zero, indicating
that there is a command language interpreter. At first glance, this
would seem to conflict with the ISO C standard which allows
system(NULL) to return zero. There is no conflict, however. A system
must have a command language interpreter, and is non-conforming if
none is present. It is therefore permissible for the system()
function on such a system to implement the behavior specified by the
ISO C standard as long as it is understood that the implementation
does not conform to POSIX.1-2017 if system(NULL) returns zero.


I understand that to mean that `system(NULL)` returning zero indicates 
that the program is not (currently) running in a POSIX.1-2017 
environment. Guix creates many environments that do not conform to 
POSIX.1-2017: for example, any environment without `vi`.


-Philip

Re: What 'sh' should 'system' use?

[Philip McGrath]

Hi,


On 9/19/22 08:55, Maxime Devos wrote:
(4) Stop using 'system' in applications -- instead use whatever the 
language's equivalent of Guile's system*, execl ... or Guix' 
'invoke'. Why?  Because 'system'-like functions requires quoting the

 command line arguments whereas in 'system*'-like functions you could
 just pass a list of command line arguments, and it's easy to get the
 quoting wrong, especially if some of the arguments are generated 
dynamically.


As a bonus, this could often remove a dependency on 
bash{-minimal,-static,}.




I definitely advocate 'system*'-like functions in general. Still,
'system'-like functions exist: I'm advocating that Guix should should
have a consistent answer for how such functions should behave.

(Very occasionally, a program really does want to invoke the shell, such
as when shell expansion is part of an existing API.)

From a different perspective, this is part of why I've recently been
thinking we should find 'sh' dynamically: most programs/environments
don't, and shouldn't, need bash{-minimal,-static}, so it seems wrong to
make it a mandatory dependency of libc.

On 9/19/22 08:55, Maxime Devos wrote:


On 19-09-2022 02:13, Philip McGrath wrote:
1) If we want to continue to hard-code a specific shell into 
Glibc,


We do, for reproducibility -- otherwise, the behaviour of the 
'system' function depends on whatever is the current /bin/sh, and 
sometimes /bin/sh is updated (and on some foreign systems it might 
not even be the bash we are used to).


[...]



2) If we want to make 'sh' a weak/dynamic reference, I think we 
should strongly consider arranging to make it available at 
'/bin/sh' when present. I expect this option would require less 
patching of other packages*by far*  than any other approach.


See (1) (reproducibility) -- also, you would need to modify the 
daemon for that, so there are compatibility concerns, and then we're
 stuck with the /bin/sh special case forever (unless breaking 
compatibility would later be considered acceptable).




I don't think there's a reproducibility problem. Guix already can create
reproducible containers with "/bin/sh" (e.g. 'guix shell coreutils
--container -- ls /bin') and without "/bin/sh" (as in package build
environments).

I haven't investigated whether adding the ability to create "/bin/sh" in
build containers would require modifying the daemon or just sending the
daemon different instructions. However, AIUI, Nix *always* creates
"/bin/sh" in build containers, which makes me further expect that any
change needed to the daemon would be small.

To be clear, I'm not proposing that we always create "/bin/sh" in build
containers. At a low level, I'm suggesting that we add the ability to
create "/bin/sh" when desired. I can imagine one possibility for a
high-level interface would be to create "/bin/sh" by default when an
input provides "bin/sh", and it might turn out that we end up wanting
"/bin/sh" in most or all build containers in practice, but I see those
as secondary questions.

and recommendations for how packages should use it: '_PATH_BSHELL' 
is the best mechanism I've heard of so far, though I wish it were 
standardized, and the fact that it can't be portably assumed to be

a string constant could be surprising.


I consider _not_ using it, and using (4) instead, to be best. If not
 suitable (for example, because a shell is needed to run an actual 
shell script), then a plain "sh" looked up in the $PATH (like other 
binaries) and substitute*-ed by Guix should suffice.




As I said above, I agree that 'system*' should be preferred over
'system' when possible.

There are a few dimensions here that I want to try to pick apart.

When you say:

a plain "sh" looked up in the $PATH (like other binaries) and 
substitute*-ed by Guix should suffice


there are a few different things that might mean.

I think you're probably referring to the status quo, where "sh" is
looked up in the 'inputs' or a G-expression equivalent and an absolute
reference to that particular "sh" is embedded into the package being
built. (But, when cross-compiling, that "sh" would not be in the $PATH
in the build environment.)

There's a different question about $PATH vs. _CS_PATH that I'll address
later.

I see at least two reasons to prefer finding "sh" dynamically at run-time.

First, we have other POSIX-like shells packaged in Guix, such as dash,
zsh, and gash. Currently, to create an environment where one of these
shells is used to run 'system'-like functions (e.g. because dash is
supposed to be faster than bash), you would have to recompile everything
that depends on glibc. (Maybe you could craft a very ugly graft.)

Second, sometimes people may want to create environments, images, etc.
without an "sh" available. In some sense this is a special case of using
an alternate shel

What 'sh' should 'system' use?

[Philip McGrath]

Hi Guix,

The C standard library includes a function 'system' to run a string as a shell 
command. Other languages provide similar functions, including Guile and many 
other Schemes and the Standard ML Basis Library.[1][2] Even without a 
dedicated library function, a program might want to run 'sh' using a general-
purpose mechanism for launching subprocesses.

How should the implementation of a function like 'system' find an 'sh' 
executable?

This question came up most recently with the patch series updating to Racket 
8.6 [3]: we already had patched in a workaround for Racket, but Chez Scheme 
and Zuo also needed workarounds to find 'sh'. I'm going to try to summarize the 
context I found in the course of that discussion and explain my current 
thinking, which changed over the course of that thread.

I think Guix should decide on an approach for functions like 'system' that can 
be applied consistently across languages. In particular, I don't think what 
our 'glibc' package is currently doing makes sense under any sort of approach.

First, an overview of three ways of trying to answer this question:

1) Many programs assume 'sh' can be found at '/bin/sh', but this is not true 
in Guix build environments, and it is not portable in general. Some systems 
have historically had a non-POSIX shell at '/bin/sh' and a POSIX shell at '/
usr/xpg4/bin/sh'.

More significantly, on Android, the shell is never at '/bin/sh'! (It is 
typically at '/system/bin/sh', except it is '/vendor/bin/sh' for "vendor 
code", whatever that is.[4]) That may be relevant for upstreams' willingness 
to reconsider this assumption.

In recent Python, the implementation of 'subprocess.Popen' uses [5]:

unix_shell = ('/system/bin/sh' if
  hasattr(sys, 'getandroidapilevel') else '/bin/sh')

which accounts for Android while being completely non-general.

On the other hand, even Nix puts '/bin/sh' at its usual path: we are really 
quite an outlier in this respect. (IIUC, Nix also has '/usr/bin/env', but no 
other utilities at FHS paths.)

In Glibc, 'sysdeps/posix/system.c' assumes '/bin/sh', and Guix currently 
patches it to refer to a store path as a string constant (but see below for 
issues).

2) There is a non-standard but ubiquitous macro '_PATH_BSHELL' in 'paths.h' 
which is supposed to give the path of a Bourne-like shell. In Guix, we patch 
this to refer to a store path as a string constant (but again, see below for 
issues). Notablty, on Android, it is not a compile-time constant: it is

#define _PATH_BSHELL __bionic_get_shell_path()

where the function returns '/system/bin/sh' or '/vendor/bin/sh' as appropriate 
(but, in any case, it returns a `const char*` to a compile-time constant, so 
no manual memory management is needed).

3) POSIX actually has an answer to the question of how to find 'sh', but, 
unfortunately, its answer doesn't work in Guix build environments.

The POSIX spec for 'system' [6] says, in the informative section "Application 
Usage":

> There is no defined way for an application to find the specific path for the
> shell. However, confstr() can provide a value for PATH that is guaranteed
> to find the sh utility.

Likewise, the spec for 'sh' [7] says in the corresponding section:

> Applications should note that the standard PATH to the shell cannot be
> assumed to be either /bin/sh or /usr/bin/sh, and should be determined by
> interrogation of the PATH returned by getconf PATH, ensuring that the
> returned pathname is an absolute pathname and not a shell built-in.

Most emphatically, the spec for 'confstr' [8] says in the normative section 
"Description":

> If the implementation supports the POSIX shell option, the string stored in
> buf after a call to:
> 
> confstr(_CS_PATH, buf, sizeof(buf))
> 
> can be used as a value of the PATH environment variable that accesses all of
> the standard utilities of POSIX.1-2017, that are provided in a manner
> accessible via the exec family of functions, if the return value is less
> than or equal to sizeof(buf).

It's worth noting here that 'PATH' is explicitly not consulted. Likewise, from 
the rationale section of [6]:

> One reviewer suggested that an implementation of system() might want to use
> an environment variable such as SHELL to determine which command
> interpreter to use. The supposed implementation would use the default
> command interpreter if the one specified by the environment variable was
> not available. This would allow a user, when using an application that
> prompts for command lines to be processed using system(), to specify a
> different command interpreter. Such an implementation is discouraged. If
> the alternate command interpreter did not follow the command line syntax
> specified in the Shell and Utilities volume of POSIX.1-2017, then changing
> SHELL would render system() non-conforming. This would affect applications
> that expected the specified behavior from system(), and since the Shell 

Re: secure boot

[Philip McGrath]

On Sun, Aug 21, 2022, at 4:46 AM, Josselin Poiret wrote:
> Hi Antonio,
>
> Antonio Carlos Padoan Junior  writes:
>
>> As far as I understand, Guix doesn't provide means to automatically sign
>> bootloaders and kernels in order to use UEFI secure boot after each system
>> reconfigure (assuming a PKI is properly implemented).  Hence, using
>> secure boot with Guix is currently not viable (am i correct?).
>
> You're right, we don't really have any means to do that.  It would have
> to be done outside of the store, again, so that the private key doesn't
> leak into it.
>

I could imagine a process like this:

 1. Build the binary that needs to be signed.
 2. Outside of the Guix build environment, create a detached signature
for the binary using your secret key.
 3. Add the detached signature to the Guix store, perhaps with 'local-file'.
 4. Use Guix to attach the signature to the built binary.
 5. Use the signed binary in your operating-system configuration.

IIUC, executables that run in the UEFI environment need "secure boot" 
signatures to be attached, but you may be able to use detached signatures 
directly for other things that they want to verify by means other than "secure 
boot".

I expect the things that need to be signed are small, build reproducibly, and 
change rarely, which might make this especially practical.

-Philip

Re: A proposal of a consistent set of clear rules and guidelines involving snippets, phases and patches.

[Philip McGrath]

On Sun, Jul 24, 2022, at 11:17 PM, Maxime Devos wrote:
>  * In principle, you can apply a patch from a phase. However, this causes the 
> result of "guix build --source" to not correspond to the actual source code 
> anymore (i.e., it doesn't act as corresponding source anymore), so consider 
> this a last resort for situations such as avoiding causing a world-rebuild 
> for a patch fixing a target-specific bug by making the patching conditional 
> upon target-foo?. If you apply a patch from a phase, make sure that the patch 
> appears in the inputs or native-inputs, such that "guix build --source=all" 
> will include the patch.

Should we have an option for "guix build --source=all" to also include the Guix 
"scripts used to control compilation and installation"?

-Philip

Re: A proposal of a consistent set of clear rules and guidelines involving snippets, phases and patches.

[Philip McGrath]

On Sun, Jul 24, 2022, at 11:17 PM, Maxime Devos wrote:
>  * Patches must not be used to remove non-free files, because a patch by 
> construction contains the non-free file itself so the patch would be 
> non-free, which would not be acceptable to Guix. Likewise, patches should not 
> be used to remove bundled libraries, to avoid large space usage, but this is 
> not an absolute rule unlike as for non-free files.

It is possible to create patches that do not contain the deleted file, e.g. 
with `git format-patch --irreversible-delete`. That said, I don't know if the 
version of `patch` we use to patch origins is able to apply such patches—but 
maybe it would be a useful feature?

-Philip

Re: Test US mirror for bordeaux.guix.gnu.org and slow downloading of sub

[Philip McGrath]

Hi,

On Tue, Jul 12, 2022, at 1:34 PM, John Kehayias wrote:
> Hi Chris,
>
> Thanks for setting up some more mirrors, here is what I just got (in a 
> previous run the main Bordeaux server was a bit slower, more like 18 
> MB/s) on a wired connection that maxes out at about 320 Mbps.
>

I also get a nice speedup: results below.

I wondered why, given that, I still seemed to be getting substitutes primarily 
from https://ci.guix.gnu.org, until someone just mentioned on IRC 
() that:

> you also need to make sure that the public key is authorised,
> or Guix will not trust the server even if present in the list of URLs.
> This is unfortunately the case with bordeaux if you installed from
> 1.3.0. If ‘grep 7D602902D3 /etc/guix/acl’ returns nothing, you're
> missing out on FREE substitutes included in your subscription,
> and you should add it

I checked, and it was indeed missing! Somehow I thought it was updated 
automatically when it was added to `%default-authorized-guix-keys`. Not only 
was this true on the foreign system where I installed manually about a year 
ago, the key also was missing on a machine where I more recently installed 
using the Debian (well, Kubuntu 22.04) package.

Just thought I'd mention it in case anyone else was as confused as I was.

-Philip

philip@bastet:/tmp/subs$ wget 
https://bordeaux.guix.gnu.org/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
--2022-07-31 07:58:16--  
https://bordeaux.guix.gnu.org/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
Resolving bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)... 185.233.100.56, 
2a0c:e300::58
Connecting to bordeaux.guix.gnu.org 
(bordeaux.guix.gnu.org)|185.233.100.56|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 208615205 (199M) [text/plain]
Saving to: ‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0’

078vr3r8mn3yrwzwxw64h 100%[==>] 198.95M  8.17MB/sin 27s 


2022-07-31 07:58:44 (7.45 MB/s) - 
‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0’ saved [208615205/208615205]

philip@bastet:/tmp/subs$ wget 
https://bordeaux-us-east-mirror.cbaines.net/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
--2022-07-31 07:58:51--  
https://bordeaux-us-east-mirror.cbaines.net/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
Resolving bordeaux-us-east-mirror.cbaines.net 
(bordeaux-us-east-mirror.cbaines.net)... 5.161.49.48
Connecting to bordeaux-us-east-mirror.cbaines.net 
(bordeaux-us-east-mirror.cbaines.net)|5.161.49.48|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 208615205 (199M) [text/plain]
Saving to: ‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0.1’

078vr3r8mn3yrwzwxw64h 100%[==>] 198.95M  25.2MB/sin 
8.5s

2022-07-31 07:59:00 (23.4 MB/s) - 
‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0.1’ saved 
[208615205/208615205]

philip@bastet:/tmp/subs$ wget 
https://ci.guix.gnu.org/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
--2022-07-31 07:59:25--  
https://ci.guix.gnu.org/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
Resolving ci.guix.gnu.org (ci.guix.gnu.org)... 141.80.181.40
Connecting to ci.guix.gnu.org (ci.guix.gnu.org)|141.80.181.40|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 208615205 (199M) [application/octet-stream]
Saving to: ‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0.2’

078vr3r8mn3yrwzwxw64h 100%[==>] 198.95M  6.39MB/sin 27s 


2022-07-31 07:59:53 (7.34 MB/s) - 
‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0.2’ saved 
[208615205/208615205]

Re: The case for moving raw binaries

[Philip McGrath]

Hi,

On Fri, Apr 29, 2022, at 12:59 PM, Liliana Marie Prikler wrote:
> Am Freitag, dem 29.04.2022 um 11:27 +0200 schrieb Maxime Devos:
>> [...]
>> 
>> I thought that
>> 
>>   (if already-wrapped?
>>   ;; PROG is already a wrapper: add the new "export VAR=VALUE"
>>   ;; lines just before the last line.
>>   [...])
>> 
>> in 'wrap-program' would avoid creating ..foo-real-real?
> You are correct, I was going on old info that I haven't checked since.
>
> This leaves us with
>> That said, the proposed new behaviour seems reasonable to me --
>> "pidof emacs" would then actually find Emacs.
> and the annoyance that "." shell-completes to all the wrapped binaries.
> For the former, there is IIRC still a bug in tramp (and I'm sure other
> emacs packages), because a process name doesn't match the expected
> regexp.
>
> As for where to move things, I'm starting to lean a little closer
> towards having an own output.  That way, we don't need to worry about
> stuff from different directories (e.g. bin and sbin) shadowing each
> other (even though that shouldn't occur), but more importantly, if we
> need to copy data into rawbin so that it's correctly resolved, we can
> do that.  The only thing that doesn't quite work is relative resolution
> of commands, which would go through the wrapper-less binaries instead.
> However, given that the wrapperless binary is invoked from a wrapped
> binary, I am 73.69% certain, that this ought not to create too much of
> a problem w.r.t. the set environment variables.
>
> WDYT?

I was mildly annoyed recently with several programs that use the ".foo-real" 
name in their `--help` output, for example:

```
$ guix shell --pure reuse -- reuse -h
usage: .reuse-real [-h] [--debug] [--include-submodules]
```

I wondered about just changing `wrap-program` to put the real program at 
`.real/foo` instead of `.foo-real`. One advantage is that it wouldn't need any 
special cooperation like setting up an output or an environment variable.

-Philip

Re: Release v1.4?

[Philip McGrath]

On Fri, Jun 17, 2022, at 11:37 AM, Brian Cully via Development of GNU Guix and 
the GNU System distribution. wrote:
> Ludovic Courtès  writes:
>
>> So plain ‘emacs’ package doesn’t work on Wayland?  That sounds 
>> like a
>> recipe for a poor user experience, no?
>
> The mainline Emacs is not Wayland-native, but it (along with just 
> about everything else) will run fine under XWayland. It's how I've 
> been running it for some time now. The user experience is almost 
> indistinguishable from either the ‘pgtk’ branch or the mainline, 
> X-only branch.
>

I'm on a foreign distro, but FWIW I've been using plain Emacs 27 with the KDE 
Plasma Wayland session on HiDPI displays for over a year, and I have no 
complaints. In fact, I didn't even realize Emacs wasn't Wayland-native until 
this thread. I took a look just now at the 'emacs-next-pgtk' package in Guix: 
it does look a bit nicer in a side-by-side comparison, but it's subtle. Either 
way, mostly Emacs just looks like Emacs. So I don't think Emacs should be a 
blocker for enabling Wayland by default.

-Philip

Re: FSDG-compatibility of APSL-2.0

[Philip McGrath]

Hi,

On Friday, June 17, 2022 10:37:07 AM EDT zimoun wrote:
>  
> On the other hand, I refuse to judge the intent behind a software.  It
> appears to me a slippery slope.  The only way is to set a clear frame
> and then scrutinize using this very frame.  Debian defines a frame, GNU
> defines another frame, etc. and each project qualifies via this frame.
> 
> Guix is part of GNU and the GNU project lists the acceptable and
> unacceptable licenses.
> 
> Again, if the Guix project would like to apply a more stringent license
> policy than the GNU one, then the question is according to which frame.
> 

Yes: I think the most ideal license policy has yet to be invented, and I'm 
glad the Debian Free Software Guidelines and the Free System Distribution 
Guidelines both exist and provide some valuable critiques of each other. 
(Neither is a subset of the other!) I do think the guidelines should be open 
to (good faith) discussion and debate. But forking the license standards 
caries big costs: maybe even some literal ones, but certainly in terms of 
effort—from idealists who think about these principles, from distributions who 
have a smaller set of collaborators among whom to share the work of evaluating 
license issues, and from freedom-loving upstream developers who want to 
publish software everyone can accept without patching. At least for me, 
whatever marginal improvement could be made (with respect to my idiosyncratic 
opinion) just doesn't seem worth it.

>
> On Fri, 17 Jun 2022 at 11:39, Maxime Devos  wrote:
> > ... currently Guix isn't using the APSL2.0 anywhere (according to git
> > grep -F aspl), so it seems quite practical and effortless to just
> > remove apsl2 from (guix licenses).
> 
> It is maybe used by Chromium.
> 
> I have nothing against removing APSL2.0 if it is not currently used.
> However, I think the Guix project should continue to accept all software
> using the licenses on GNU list [1].
> 
> About patch#55998, the question is about dependencies and linking
> because APSL is incompatible with the GPL.  If all is fine, then let
> include ’cctools’ in Guix.
> 

I'll say just a bit about the practical motivation for this patch.

Assuming we agree that it's beneficial for at least some free software to be 
able to run on non-free operating systems, that means some free software 
developers have to build software for those systems. For Windows, MinGW lets 
us do so without having to leave the free world. For Apple platforms, the 
status quo is that someone has to buy or get access to a piece of hardware 
sold the nonfree os. Maybe, with a lot of effort, you can cross-compile by 
downloading a nonfree binary bundle.

Patch 55998 doesn't solve that problem, but it at least takes a step toward 
being *less* reliant on a nonfree toolchain. The particular utility that I've 
used so far [1] is significant enough that part of its functionality 
(unfortunately not the part I needed) had been implemented in Racket [2], and 
I believe other compilers or package managers have done similarly.

(I plan to write more about how this experiment worked out once various loose 
ends have been tied up.)

This is possible because we *do* have the freedom to use, study, modify, and 
distribute major components of the toolchain and operating system kernel. 
Requiring something other than those freedoms would perversely increase the 
reliance on nonfree systems for building free software.

I think the dependency and linking issues, to whatever extent they existed, 
are resolved by v2 of the patch. The only GPL file was a header from GDB which 
seemed to be left over from before Apple switched to an LLVM/Clang toolchain: 
it wasn't actually used by anything, so just deleting the file worked out fine. 
(Not being a lawyer, I can only speculate about whether it might have been 
fair use anyway under Google v. Oracle.) There are a few files or parts of them 
under miscellaneous permissive Expat- or BSD-like licenses. The only linking 
is with things like libc that have sufficiently permissions.

-Philip

[1]: https://github.com/LiberalArtist/native-libgit2-pkgs/tree/build-scripts
[2]: https://github.com/racket/racket/blob/master/racket/src/mac/
install_name_tool.rkt
[3]: https://issues.guix.gnu.org/55998#23



signature.asc
Description: This is a digitally signed message part.

Re: FSDG-compatibility of APSL-2.0

[Philip McGrath]

Hi,

On Thursday, June 16, 2022 3:43:39 AM EDT Liliana Marie Prikler wrote:
> Hi Philip,
> 
> Am Donnerstag, dem 16.06.2022 um 02:21 -0400 schrieb Philip McGrath:
> > Hi Guix,
> > 
> > Is the Apple Public Source License 2.0 (APSL-2.0 [1]) a free license
> > according to Guix's standards?
> 
> While it isn't included in the free licenses list the FSF publishes,
> from your note [6] I would guess that it is a GPL-incompatible free
> software license.
> 

In fact, it is on the list at 
<https://www.gnu.org/licenses/license-list.html#apsl2>:

> Apple Public Source License (APSL), version 2 (#apsl2)
> 
> This is a free software license, incompatible with the GNU GPL. We
> recommend that you not use this license for new software that you
> write, but it is ok to use and improve the software released under this
> license.

I don't want to speak for Maxime, but AIUI the question was whether Guix ought 
to continue to accept all licenses on that list, or instead ought adopt some 
different (more stringent?) license policy.

-Philip

signature.asc
Description: This is a digitally signed message part.

FSDG-compatibility of APSL-2.0

[Philip McGrath]

Hi Guix,

Is the Apple Public Source License 2.0 (APSL-2.0 [1]) a free license 
according to Guix's standards?


In , I sent a patch adding a package 
under this license, and Maxime Devos pointed out this choice-of-forum 
provision, which I agree is quite one-sided:


> 13.6 Dispute Resolution. Any litigation or other dispute resolution
> between You and Apple relating to this License shall take place in the
> Northern District of California, and You and Apple hereby consent to
> the personal jurisdiction of, and venue in, the state and federal
> courts within that District with respect to this License. The
> application of the United Nations Convention on Contracts for the
> International Sale of Goods is expressly excluded.

We thought this list was a better place for any discussion of Guix's 
policy that needs to happen.


As I understand it, Guix's current policy is the Free System 
Distribution Guidelines published at [2], which links to [3] for its 
definition of "free license". That definition says (at [4]), "It is 
acceptable for a free license to specify which jurisdiction's law 
applies, or where litigation must be done, or both."


The revision notes [5] say that paragraph was added in version 1.129, 
from 2012, but that "this was always our policy".


The FSF has issued an opinion [6] that APSL-2.0 is a free software 
license: they say that "Apple's lawyers worked with the FSF to produce a 
license that would qualify" (after problems with earlier versions of the 
license).


Is this satisfactory for Guix? Or does Guix want to forbid such 
choice-of-forum provisions? In the latter case `apsl2`, and maybe other 
definitions, presumable would need to be removed from `(guix licenses)`.


My personal view:

I wouldn't recommend using this license: indeed, even Apple seems to 
have moved away from it for newer projects (often to Apache-2.0). If 
established guidelines *hadn't* allowed this kind of one-sided 
choice-of-forum provision, I wouldn't have found it particularly 
surprising. I think there are important community governance questions 
around how questions like this ought to be answered (basically, I agree 
with [7]).


Still, I'm in favor of the status quo. I think fragmentation over 
license policies has a significant cost for the community, and this does 
not seem to be sufficiently problematic to be worth a schism.


I'm not a lawyer, so take this paragraph lease seriously, but I also 
think the concrete impact is less than it might first seem. We accept 
choice-of-forum provisions like the one in MPL-2.0 ("Any litigation 
relating to this License may be brought only in the courts of a 
jurisdiction where the defendant maintains its principal place of 
business and such litigation shall be governed by laws of that 
jurisdiction, without reference to its conflict-of-law provisions.") [8] 
which would require you to sue Apple in California. We also accept 
licenses like the GPL that don't have any choice-of-forum provisions: 
the law of "personal jurisdiction" and venue is complex, but I would not 
be shocked if Apple could sue you in California in this case. My 
impression is that it would be very difficult to require something like 
a "freedom not to litigate in California" (especially so for all 
possible values of "California") without rejecting many 
currently-accepted licenses.


-Philip

[1]: https://spdx.org/licenses/APSL-2.0.html
[2]: https://www.gnu.org/distros/free-system-distribution-guidelines.html
[3]: https://www.gnu.org/philosophy/free-sw.html
[4]: https://www.gnu.org/philosophy/free-sw.html#legal-details
[5]: 
https://web.cvs.savannah.gnu.org/viewvc/www/www/philosophy/free-sw.html?r1=1.128=1.129

[6]: https://www.gnu.org/philosophy/apsl.html
[7]: https://guix.gnu.org/blog/2019/joint-statement-on-the-gnu-project/
[8]: https://spdx.org/licenses/MPL-2.0.html

Re: maradns reproducibility fixes and the merits of picking a random number

[Philip McGrath]

Hi,

On 6/8/22 16:25, Vagrant Cascadian wrote:

On 2022-06-09, Arun Isaac wrote:

Hi Vagrant,


But there's one nervous-making issue this revealed; maradns embeds a
random number at build time ... allegedly for systems that don't have
/dev/urandom... see
maradns-3.5.0020/deadwood-3.5.0020/src/Makefile.ubuntu2004:

   # Since some systems may not have /dev/urandom (Windows, *cough* *cough*), we
   # keep a randomly generated prime around

So it's got some code to generate a random number at build time and
embed it in the binary. Now, if there's anything I know about good
practices about random numbers, this sort of thing is generally a very
large red flag! It also makes the package build differently every
time!


Wow, great find! Has this issue been reported to maradns upstream? If
upstream fixes it or provides us a compile flag to disable this
"feature", it would be even better in the long run.


That does sound like the best long-term approach, definitely!

Will take the issue upstream...



Not sure if this is useful—I've never used MaraDNS, and I know just 
barely enough about Windows to try to keep free software portable—but, 
on Windows XP and later, you can get cryptographic-quality randomness by 
dynamically loading SystemFunction036 from Advapi32.dll, sometimes known 
as RtlGenRandom. Racket uses this to implement crypto-random-bytes, 
which uses /dev/urandom on Unix-like systems. It's also apparently used 
by Chromium, Firefox, and many others.


Racket example: 
https://github.com/racket/racket/blob/master/racket/collects/racket/private/windows-rand.rkt


C example from Microsoft: 
http://web.archive.org/web/20180929235240/https://blogs.msdn.microsoft.com/michael_howard/2005/01/14/cryptographically-secure-random-number-on-windows-without-using-cryptoapi/


Documentation, such as it is: 
https://docs.microsoft.com/en-us/windows/win32/api/ntsecapi/nf-ntsecapi-rtlgenrandom


Mozilla thread explaining why the warnings in that documentation are 
irrelevant: https://bugzilla.mozilla.org/show_bug.cgi?id=504270#c4


-Philip

Re: Finding a “good” OpenPGP key server

[Philip McGrath]

Hi,

On 4/18/22 16:24, Ludovic Courtès wrote:

Hi,

Tanguy LE CARROUR  skribis:


gpgv: Signature made Wed 16 Sep 2020 22:30:16 CEST
gpgv:using RSA key 6115012DEA3026F62A98A556D6B570842F7E7F8D
gpgv: Can't check signature: No public key
Would you like to add this key to keyring 
'/home/tanguy/.config/guix/upstream/trustedkeys.kbx'?
yes
gpg: keyserver receive failed: No data


This indicates that ‘guix refresh’ failed to download the relevant GPG
key from the default key server, the one that appears in
~/.gnupg/dirmngr.conf (if it exists).

That’s unfortunately often the case these days.  :-/ This key appears to
be on keys.openpgp.org, but it lacks a “user ID” packet and so gpg
ignores it (for no good reason):

--8<---cut here---start->8---
$ gpg --no-default-keyring --keyring 
/home/ludo/.config/guix/upstream/trustedkeys.kbx --keyserver keys.openpgp.org 
--recv-keys 6115012DEA3026F62A98A556D6B570842F7E7F8D
gpg: key D6B570842F7E7F8D: no user ID
gpg: Total number processed: 1
$ gpg --no-default-keyring --keyring 
/home/ludo/.config/guix/upstream/trustedkeys.kbx --list-keys 
6115012DEA3026F62A98A556D6B570842F7E7F8D
gpg: error reading key: No public key
--8<---cut here---end--->8---

I’m not sure what a good solution is (other than looking for the key
manually on Savannah or on some random key server).



Many distributions of GnuPG include a patch to handle keys without “user 
ID” packets.[1] In fact, it may well be *most* distributions: Debian, 
Fedora, Nix, OpenSUSE[2], and at least one commonly-recommended 
installation option for Mac. Debian packagers have argued [3]:



I think GnuPG's inability to receive these
kinds of cryptographic updates to OpenPGP certificates that it knows
about is at core a security risk (it makes it more likely that users
will use a revoked key; or will be unable to use any key at all, and
will send plaintext).


Unfortunately, the upstream GnuPG maintainer has rejected the patch, I 
guess because strict conformance to the OpenPGP standards requires user 
ids.[4]


I am by no means an expert on PGP or GPG issues, but I'd be in favor of 
Guix adopting this patch.


-Philip

[1]: https://keys.openpgp.org/about/faq#older-gnupg
[2]: https://build.opensuse.org/package/show/openSUSE:Factory/gpg2
[3]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=930665#10
[4]: https://dev.gnupg.org/T4393#133689

Re: better error messages through assertions

[Philip McGrath]

Hi,

On 3/30/22 05:37, Ludovic Courtès wrote:



What is the preferred mechanism for exceptions?


For Guix code, SRFI-34/35.


Likewise, what record system should I use?


SRFI-9.

(Perhaps we should put answers to these questions in the “Coding Style”
section of the manual.)



As I've looked more closely, I think I'd be porting a fair amount of 
code that looks like this:


```
(define-struct (blame-no-swap blame) ()
  #:property prop:equal+hash
  (list blame=? blame-hash blame-secondary-hash))
```

That code creates a new opaque struct type `blame-no-swap` with parent 
type `blame` and customizes the behavior of `equal?` and `equal?`-based 
hashing (but the customization only applies when the two 
potentially-equal values are both immediate instances of 
`blame-no-swap`, not a supertype or subtype).


It looks like SRFI 9 doesn't have a notion of opaque or parent types, so 
it seems like it would be easier to use R6RS records as the basis for 
porting.


(I think it should still work to use SRFI 9 records for the interface. 
But I know the representation of blame, among other details, has gotten 
a lot of attention over the years, so it seems like it would be good to 
keep the port as close as reasonably possible to the Racket implementation.)


I'm less sure about `equal?` and hashing. I see in [1] that `equal?` can 
be controlled via GOOPS, and I have the impression that there's some 
kind of bridging between basic types and GOOPS, but I'm not clear on the 
details. I don't see anything explicitly about customizing `hash`[2], 
but maybe that would work the same way?


-Philip

[1]: 
https://www.gnu.org/software/guile/manual/html_node/GOOPS-Object-Miscellany.html
[2]: 
https://www.gnu.org/software/guile/manual/html_node/Hash-Table-Reference.html#index-hash

Re: better error messages through assertions

[Philip McGrath]

Hi,

On 3/30/22 09:28, Andy Wingo wrote:


Too bad about all that other crap about checking whether the index is in
range and the field is boxed or not, though :-/  Probably there is a
better design...

Andy


For the index-out-of-range part, when I saw `record-accessor`, I thought 
of it as similar to Racket's `make-struct-field-accessor`[1], which can 
check the index just once, when the accessor is created, rather than 
each time the accessor is used. That's (part of) what Racket's `struct` 
form expands to.


Would it be reasonable to use `record-accessor` in the implementation of 
SRFI 9?


-Philip

[1]: 
https://docs.racket-lang.org/reference/creatingmorestructs.html#%28def._%28%28quote._~23~25kernel%29._make-struct-field-accessor%29%29

Re: The Shepherd on Fibers

[Philip McGrath]

Hi,

On 3/29/22 05:36, Maxime Devos wrote:

Philip McGrath schreef op ma 28-03-2022 om 20:14 [-0400]:
Maybe it would be enough for this case for Fibers to provide 
variants of `dynamic-wind` and/or `call-with-continuation-barrier` 
that cooperate with the Fibers implementation. I don't know if

that would be possible of not—in addition to my limited familiarity
with Fibers, I have not read all of the related literature that
Alexis, Matthew, and Matthias Felleisen discuss in [5] and
[6]—again, I am not an expert!


Fibers' context switching is based on continuations.  By definition,
 ‘call-with-continuation-barrier’ must create a continuation barrier
 (and as a consequence, interfere with Fibers' context switching).

It can be important to let 'call-with-continuation-barrier' (*) 
actually create a continuation barrier, even when fibers is used, 
e.g. to not create deadlocks (or livelocks or whatever, I don't know 
the terminology) when POSIX mutexes are used.  As such, as-is, I 
don't think so.


[...]

(*) Actually, 'call-with-continuation-barrier' and 'dynamic-wind' are
already a bit of a lie, since the kernel ignores them when context
switching ... Maybe continuation barriers and {un,re}winding is a
matter of perspective?


Yes, that's the crux of the problem.

All of the references I know of are discussed in mailing list threads
[1] and [2], plus the more recent Flatt & Dybvig, "Compiler and Runtime
Support for Continuation Marks" (PLDI 2020) [3], which discusses the
Racket-on-Chez implementation of delimited control. In [1], Matthew
Flatt wrote:

For an implementation that relies on a representation choice instead 
of tracing or analysis, Racket CS's implementation of delimited 
control is the state of the art --- mostly by building on Chez 
Scheme's implementation of continuations.


Again, I am very far from an expert, but I'll try to distill the
relevant parts here.

To quote from the abstract and introduction of "Adding Delimited and
Composable Control to a Production Programming Environment" [4]
(internal citations omitted),

Operators for delimiting control and for capturing composable 
continuations litter the landscape of theoretical programming 
language research.  Numerous papers explain their advantages, how

the operators explain each other (or don’t), and other aspects of
the operators’ existence.  Production programming languages, however,
do not support these operators, partly because their relationship to 
existing and demonstrably useful constructs—such as exceptions and 
dynamic binding—remains relatively unexplored.


[...]

Due to this semantic interference, simulations of delimited control 
do not immediately yield production-quality implementations.  For 
example, a Scheme library can use `call/cc` to simulate delimited 
continuations, but other libraries that use `call/cc` directly or 
that use `dynamic-wind` can interfere with the simulation.


Over the past year, we have integrated a full set of delimited- 
control operators within PLT Scheme, ensuring that all of them 
interact properly with the rest of the Scheme programming language
as well as pre-existing extensions in PLT Scheme. Specifically, PLT 
Scheme’s prompts and composable continuations have a well-defined

and useful interaction with `call/cc`, `dynamic-wind`, dynamic
binding via continuation marks, and exceptions.


Racket's Concurrent ML subsystem also falls into that category.

The result of this paper is `racket/control` library[5].

To take Racket CS as an example, Chez Scheme doesn't provide delimited
continuations or "green" threads/fibers, but it does provide an
efficient and powerful implementation of continuations (even including
support for `equal?`!). The Racket CS runtime system implementation uses
Chez's `call/cc`, `dynamic-wind`, etc. to implement Racket's primitive
control operators (from the built-in module '#%kernel). Then, a larger
set of control operators can be implemented as a library in terms of the
primitives.

But, as the above paper says, this means that Chez's `call/cc`,
`dynamic-wind`, etc. are *unsafe* from the perspective of Racket's
control primitives. From the docs for Racket's `ffi/unsafe/vm` library [6]:

Beware of directly calling a Chez Scheme primitive that uses Chez 
Scheme parameters or `dynamic-wind` internally. Note that `eval`, in 
particular, is such a primitive. The problem is that Chez Scheme’s 
`dynamic-wind` does not automatically cooperate with Racket’s 
continuations or threads. To call such primitives, use the 
`call-with-system-wind primitive`, which takes a procedure of no 
arguments to run in a context that bridges Chez Scheme’s

`dynamic-wind` and Racket continuations and threads.


Anything that has the potential to block Racket's scheduler (as opposed 
to a fiber/thread), like POSIX mutexes, is likewise unsafe and should be 
encapsulated in a safe abstraction. For more on this, see the docs for 
`ffi/unsafe/atomic`[7], `ff

Re: The Shepherd on Fibers

[Philip McGrath]

On 3/28/22 20:14, Philip McGrath wrote:
I'm not sure how useful any of that is, but take it for whatever it's 
worth. My overall impression is that the Shepherd on Fibers is a big 
step in the right direction. Congrats on the great work!


-Philip



P.S.: If you can get access to a copy of John Reppy's book "Concurrent 
Programming in ML" (Cambridge University Press, 1999, reprinted with 
corrections in 2007), chapter 7 presents an extended example of a 
software build system. In addition to the other interesting features of 
the problem domain, it provides an example of controlling external 
programs in Concurrent ML style.

Re: The Shepherd on Fibers

[Philip McGrath]

Hi,

On 3/23/22 18:36, Ludovic Courtès wrote:

Hello Guix!

I have pushed a ‘wip-fibers’ branch of the Shepherd:

   https://git.savannah.gnu.org/cgit/shepherd.git/log/?h=wip-fibers

The goal is to make shepherd (the daemon) use Fibers¹ for concurrency.



Exciting!

As I wrote in 
, 
"I haven't programmed with it in Guile at all, but, from my experience 
using Racket's Concurrent ML system, I think it is the right foundation 
for a supervisor daemon."


I still haven't programmed with Guile Fibers, and looking briefly at the 
diff of the "wip-fibers" branch was the first time I'd looked at the 
Shepherd's implementation, but I had a couple thoughts.


First,


Fibers is used in a single-threaded fashion, which is the main
constraint for shepherd since it forks.  That also means that fibers
cannot be preempted, so it’s fully cooperative scheduling.



Would it be feasible for shepherd *not* to fork? Or only to fork in a 
way that cooperates with fibers?


Obviously forking is pretty ubiquitous, but I the 2019 paper "A fork() 
in the road"[1] fairly convincing in its argument that



fork has lost its classic simplicity; it today impacts all the
other operating system abstractions with which it was once
orthogonal. Moreover, a fundamental challenge with fork is
that, since it conflates the process and the address space in
which it runs, fork is hostile to user-mode implementation
of OS functionality, breaking everything from buffered IO
to kernel-bypass networking. Perhaps most problematically,
fork doesn’t compose—every layer of a system from the kernel
to the smallest user-mode library must support it.


I consider a Concurrent ML system a "user-mode implementation of OS 
functionality": indeed, an early version of Racket's thread system 
(where thread = fiber) is one of the examples in Flatt, Findler, 
Krishnamurthi, & Felleisen, "Programming Languages as Operating Systems 
(or Revenge of the Son of the Lisp Machine)" (ICFP 1999)[2].


More concretely, preemption is a big benefit of fibers.

Racket's approach also addresses this part:


There’s one catch: Fibers is currently Linux-only.  The good news is
that work has been done to port it to other kernels via libevent².
Until it is merged, we could keep using the Shepherd 0.8 on GNU/Hurd.



Racket has a cross-platform C library, "rktio"[3], for accessing 
os-specific functionality. It was refactored into its current form in 
2017 as an early step toward Racket-on-Chez: while it happens to provide 
exactly the functionality Racket needs, it no longer is specific to 
Racket or any particular implementation thereof. That includes 
everything needed to implement the Concurrent ML system and nonblocking 
IO on a variety of OSes and kernels.


In particular, it implements—IIUC primarily in 
"rktio_process.c"—abstractions (over `fork` or alternatives) to start a 
new process running something, with environment, stdin, stdout, and 
stderror wired up ports in the sense of 
`current-{input,output,error}-port`, and use the Concurrent ML system to 
monitor its exit status, send it signals, etc. The Racket-level API is 
documented at [4].


I spend as little time as possible with these C-level sorts of issues, 
and I particularly don't know about special considerations for PID 1, 
but I hope there would be some way to meet Shepherd's needs without 
interfering with Fibers.


Second, I'm a little uneasy about `unwind-protect`:

```
+(define-syntax-rule (unwind-protect body ... conclude)
+  "Evaluate BODY... and return its result(s), but always evaluate CONCLUDE
+before leaving, even if an exception is raised.
+
+This is *not* implemented with 'dynamic-wind' in order to play well with
+delimited continuations and fibers."
+  (let ((conclusion (lambda () conclude)))
+(catch #t
+  (lambda ()
+(call-with-values
+(lambda ()
+  body ...)
+  (lambda results
+(conclusion)
+(apply values results
+  (lambda args
+(conclusion)
+(apply throw args)
```

though maybe it's used only internally and doesn't have to deal with the 
general case: I'm not an expert by any means, but my understanding (from 
a Racket mailing list thread at [5], continued at [6]) is that dealing 
with the general case may be something of an open research question.


As an example of the sort of problem, what if the `body`s evaluate 
normally, but an exception is raised in the dynamic extent of 
`(conclusion)`, causing `(conclusion)` to run again? One possible 
mitigation would be to `set!` a local variable before the normal 
`(conclusion)` and check it in the exception handler.


More generally, of course, `body ...` could escape by some mechanism 
other than a raising an exception.


If you were writing Racket, I would recommend 
`(call-with-continuation-barrier (λ () body ...))`—logically, `body ...` 
isn't re-entrant anyway—but I see 

Re: better error messages through assertions

[Philip McGrath]

Hi,

On 3/7/22 05:13, Ludovic Courtès wrote:

Hi Philip,

Philip McGrath  skribis:


Racket's state-of-the-art contract system has many features and nuances. I *do
not* think anyone should try to implement them all in one fell swoop. I'm
hoping there's a way to implement your simple assertions with only a modest
amount of overhead that will provide the right base on which to grow the rest
of a contract system. In the short term, the advantage over:


 (assert-type (listof service?) services
  "SERVICES must be a list of  values.")


is that you don't have to write error messages by hand.

You need two types of values:

  1. Contracts, recognized by `contract?`; and
  2. Blame objects, recognized by `blame?`.


[...]

Thanks for the explanation and references!  I had briefly looked at
Racket’s contract API in the past but your message gave a clearer view
of how this all fits together.



I'm glad this is something Guix people are interested in!


I would love to have contracts in Guix, even very rudimentary contracts. If
it's something the community more generally would be interested in, I'd be
glad to help as much as I can.


It’d be great to benefit from your expertise here.  Like you wrote, I
think we should start with a simple contract system, certainly simpler
than Racket’s, and build from there.

If you’re willing and able to spend time prototyping this, that’s great.
:-)



I'm interested in putting together a prototype.

I've taken my own suggestion and asked the Racket community for more 
advice: 
https://racket.discourse.group/t/advice-on-implementing-a-contract-system/832


To quote the end of my last message there,


The tl;dr of all that is that `(guix records)` seems to ultimately call for 
"indy-dependent" contracts[1].

On the one hand, the distinction between "indy-dependent" `->i`[2] and 
"lax-dependent" `->d`[3] is exactly the sort of hard-learned lesson that I hope the Guix 
community can draw from Racket's decades of experience.

On the other hand, I'm increasingly intrigued by the idea of starting with 
forms along the lines of `invariant-assertion`[4] and `struct-guard/c`[5] and 
truly sticking to flat contracts to start with, leaving all the higher-order 
complexity for another day.


I'm thinking that a reasonable place to start might be to implement a 
`contract->sanitizer` form that would allow using contracts to create 
sanitizers, ideally with no changes to `(guix records)`.


In addition to the questions about contract system design, I realized I 
have a few questions about Guix/Guile that would be relevant when 
starting a prototype.


What is the preferred mechanism for exceptions? I know about:

  * (rnrs exceptions)
  * (ice-9 exceptions)
  * (srfi srfi-34)
  * (srfi srfi-35)

and IIRC I've seen more than one of them used in the Guix codebase.

Likewise, what record system should I use? I think the answer should 
*not* be (guix records): instead, I think (guix records) should 
eventually use (guix contracts). But should I use:


  * (rnrs records syntactic)
  * (rnrs records procedural)
  * (srfi srfi-9)
  * (oop goops)

Of those, I'm most familiar with R6RS records. I know (guix records) is 
implemented on top of (srfi srfi-9), though I vaguely remember some 
discussion about potentially changing that.


Also, I don't know much about how the "abi" aspect of (guix records) 
works and what types of changes there would trigger rebuilds. (Though, 
again, I hope no changes would be needed for the proof-of-concept phase.)


Finally, when I looked again at the example at the top of this thread:

On 2/14/22 17:32, Ricardo Wurmus wrote:

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
In procedure struct-vtable: Wrong type argument in position 1 (expecting 
struct):
--8<---cut here---end--->8---

As you can probably tell easily by looking at this message, the
“service” field of the operating system configuration looked something
like this:

  (services (append (list a b c %desktop-services) #;oops))

instead of this

  (services (append (list a b c) %desktop-services))

This is because INSTANTIATE-MISSING-SERVICES — and FOLD-SERVICES, and
many more — assumes that it is only passed a plain list of services.  It
then proceeds to call SERVICE-KIND on what may or may not be a service.


Another problem here seems to be the fault of (srfi srfi-9). For example:

```
$ guile
GNU Guile 3.0.8
Copyright (C) 1995-2021 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guile-user)> ,use (srfi srfi-9)
scheme@(guile-user)> (define-record-type container (make-container 
contents) container? (contents container-contents))

scheme@(guile-user)> (cont

Re: Semantics of circular imports

[Philip McGrath]

Hi,

(Apparently I wrote this a month ago but left it sitting in "Drafts" ...)

On 2/20/22 12:47, Maxime Devos wrote:
> Philip McGrath schreef op zo 20-02-2022 om 11:47 [-0500]:
>> I was just (or maybe am still?) dealing with some issues caused by cyclic
>> imports of package modules while updating Racket to 8.4: see in particular
>> <https://issues.guix.gnu.org/53878#93>, as well as #66, #112, and #113 in the
>> same thread.
>> [...]
>> I find the semantics of Guile's cyclic module imports very confusing,
>> and I don't know of any documentation for what is and isn't supported
>> other than advice from Ludo’ in <https://issues.guix.gnu.org/48682#7
>> —is there any?
> 
> (The following explanation ignores syntax transformers, #:select and
> #:autoload.)
> 
> Basically, a module consists of a hash table and a thunk
> initialising the hash table.  A few situations to demonstrate:
> 
> ;; Non-cyclic
> (define-module (foo)
>#:export (foo))
> (define foo 0)
> 
> (define-module (bar)
>#:export (bar)
>#:use-module (foo))
> (define bar (+ 1 foo))
> 
> The thunk of 'foo' would be
> 
>(lambda ()
>  (set-module-value! '(foo) 'foo 0))
> 
> and the thunk of 'bar' would be
> 
>(lambda ()
>  ;; This calls the thunk of 'foo' if it hasn't yet been called
>  ;; before.
>  (initialise-module '(foo))
>  (set-module-value! '(bar) 'bar (+ 1 (module-value '(foo) 'foo 0
> 
> ;; Cyclic, non-problematic
> (define-module (foo)
>#:export (foo)
>#:use-module (bar))
> (define foo 0)
> (define (calculate-foobar)
>(+ 1 (calculate-bar))
> 
> (define-module (bar)
>#:export (calculate-bar)
>#:use-module (foo))
> (define (calculate-bar) (+ 1 foo))
> 
> The initialisation thunk of 'foo' would be
> 
>   (lambda ()
> (initialise-module '(bar))  ; L1
> (set-module-value! '(foo) 0) ; L2
> (set-module-value! '(foo) 'calculate-foobar ; L3
>   (lambda () (+ 1 ((module-value '(bar) 'calculate-bar)) ; L4
> 
> and the thunk of 'bar' is:
> 
>(lambda ()
>  (initialise-module '(foo)) ; L6
>  (set-module-value! '(bar) 'calculate-bar ; L7
>(lambda () (+ 1 (module-value '(foo) 'foo) ; L8
> 
> Now let's see what happens if the module (bar) is loaded:
> 
> ; Initialising '(bar)'
> (initialise-module '(foo)) ; L6
> ;; (foo) has not yet begun initialisation, so run the thunk:
> ->  (initialise-module '(bar)) ; L1
> ;; (bar) is being initialised, so don't do anything here
> -> (set-module-value! '(foo) 0) ; L2
> -> (set-module-value! '(foo) 'calculate-foobar ; L3
>  (lambda () (+1 ((module-value '(bar) 'calculate-bar ; L4
> 
>The hash table of '(bar)' does not yet contain 'calculate-bar',
>but that's not a problem because the procedure 'calculate-foobar'
>is not yet run.
> (set-module-value! '(bar) '(calculate-bar) ; L7
>(lambda () (+ 1 (module-value '(foo) 'foo ; L8
> ;; The hash table of '(foo)' contains 'foo', so no problem!
> ;; Alternatively, even if '(foo)' did not contain 'foo',
> ;; the procedure '(calculate-bar)' is not yet run, so no problem!

Oh, wow. I definitely had not realized that, *even inside a declarative 
module*, a reference to a variable with no statically visible definition 
would semantically be a dynamic lookup in a mutable environment at 
runtime (rather than a compile-time error), though I do see now that 
`info guile declarative` does indeed say that marking a module as 
declarative "applies only to the subset of top-level definitions that 
are themselves declarative: those that are defined within the 
compilation unit, and not assigned (‘set!’) or redefined within the 
compilation unit."

Does this mean that Guile treats all imported bindings as non-declarative?
This seems like a big barrier to cross-module inlining, though IIUC Guile
currently doesn't do much of that by default (maybe for this reason).

The use of "top-level" to refer to definitions within a module is 
somewhat confusing to me. I usually understand "top-level" to refer to 
the kind of interactive REPL environment for which R6RS leaves the 
semantics unspecified. Racket uses "module-level variable"  and "module 
context" in contrast to "top-level variable" and "top-level context" to
make this distinction.[1][2][3] (There are also R6RS "top-level 
programs", but I wouldn't think of those unless made very clear from 
context.)

(Also, what is a "compilation unit" in Guile? Is it ever something other 
than a single module corresponding to a single file (potentially using 
`includ

Re: License issue with SRFI 5

[Philip McGrath]

Hi,

On Monday, March 7, 2022 5:41:46 AM EST Ludovic Courtès wrote:
> Hi,
> 
> Philip McGrath  skribis:
> > To start with that, my proposed approach, which I hope satisfies the
> > FSDG, is here: https://github.com/racket/srfi/pull/15
> 
> Good to know; I hope the next Racket release will include it.
> 

Yes! It's been merged upstream, so it will be in 8.5 (and snapshots, 
meanwhile), and we've included it in Guix as of the recent update to Racket 
8.4.

-Philip

signature.asc
Description: This is a digitally signed message part.

Re: better error messages through assertions

[Philip McGrath]

Hi,

On Monday, February 28, 2022 7:59:02 AM EST Ludovic Courtès wrote:
> Hi!
> 
> Ricardo Wurmus  skribis:
> > Philip McGrath  writes:
> >> As a Racketeer, I think you're half way to reinventing contracts.
> > 
> > Yes, I was in fact thinking of contracts, but shied away from mentioning
> > them :)  The reason is that I think we can cover a lot of distance with
> > just a few simple assertions to avoid plowing ahead on bad arguments.
> 
> I’d very much like to have contracts.
> 
> For record types, we have “sanitizers” right now, which we could use to
> insert procedural type checks.  It wouldn’t be as nice as proper
> declarative contracts though, unless we arrange to make them look like
> contracts.
> 

I'm going to try to sketch what I think is a minimal API for contracts.

Racket's state-of-the-art contract system has many features and nuances. I *do
not* think anyone should try to implement them all in one fell swoop. I'm
hoping there's a way to implement your simple assertions with only a modest
amount of overhead that will provide the right base on which to grow the rest
of a contract system. In the short term, the advantage over:

> (assert-type (listof service?) services
>  "SERVICES must be a list of  values.")

is that you don't have to write error messages by hand.

You need two types of values:

 1. Contracts, recognized by `contract?`; and
 2. Blame objects, recognized by `blame?`.

In simplest terms, a blame object holds context information for error
reporting. Ultimately, it is used to raise an exception with
`raise-blame-error`[1]: this can support e.g. substituting "expected"/"given"
with "promised"/"produced" in error messages based on the context. A contract
combinator will typically give its sub-contract a blame object extended using
`blame-add-context`[2].

The primitive operation on a contract is `get/build-late-neg-projection`[3]:
by skipping the decades experimenting with other APIs that turned out to
perform less well, you have the opportunity to give this function a better
name. In pseudo-code, `get/build-late-neg-projection` has the contract:

(-> contract? (-> blame? (-> val neg-party val*)))

That is, given a contract, it returns a function that, when given a blame
object, returns a function that enforces the contract. The enforcer function
accepts the value being checked and the "negative party" and either raises an
exception or returns the checked value, possibly wrapped to enforce higher-
order checks.

(The "negative party" is simultaneously deeply important and something you
could ignore to start with. Simply put, if a module `(server)` provides a
contracted value that is used by a module `(client foo)`, the "negative party"
represents `(client foo)`, e.g. via a source location. It is the only part
that would be different between `(client foo)` and `(client bar)`, so this API
lets everything else be shared.)

The multiple levels of `lambda` let contract combinators do some of their work
ahead of time, improving performance when the contracted value is used.

As a motivating example, here is a very basic definition of the `listof`
contract combinator. Notice in particular:

  * Predicate functions and certain kinds of primitive data are implicitly
promoted to contracts. This wasn't original to Racket's contract system,
but the ergonomics are so compelling you might want it from the beginning.

  * Forms like `define/contract`, `struct/contract`, and `contract-out` (which
is preferred) attach contracts to values. Guix might want
`define-record-type*/contract`. There is no public API for constructing
blame objects.

> #lang racket
> (define (listof elem/c)
>   (let* (;; error if elem/c isn't a contract
>  [elem/c (coerce-contract 'listof elem/c)]
>  [elem-late-neg (get/build-late-neg-projection elem/c)])
> (make-contract
>  #:name (build-compound-type-name 'listof elem/c)
>  #:late-neg-projection
>  (λ (blame)
>(define elem-proj+blame
>  (elem-late-neg (blame-add-context blame "an element of")))
>(λ (val neg-party)
>  (unless (list? val)
>(raise-blame-error
> blame val #:missing-party neg-party
> `(expected: "a list" given: "~e")
> val))
>  (for-each (λ (elem)
>  (elem-proj+blame elem neg-party))
>val)
>  val)
> 
> (define/contract numbers
>   (listof number?)
>   '(1 2 2.5 3))
> 
> (define/contract fruits
>   (listof (or/c 'apple 'banana))
>   (list (λ (x) (x x

The resulting error message:

> fruits: broke its own contract
>   promised: (or/c (quote apple) (quote banana)

Re: Update CoC adapted from upstream 2.1 (instead of 1.4)

[Philip McGrath]

On 2/25/22 11:59, Taylan Kammer wrote:> Transwomen have many challenges 
they have to face, and some of them may be

similar or equivalent to some challenges women have to face, but to claim


I'm concerned that framing "transwomen" and "women" as though they were 
two contrasting groups is itself inconsistent with the current CoC's 
standard of "using welcoming and inclusive language".


While I think the best reading of "Le Deuxième Sexe" affirms that trans 
women are women, I agree with those who have already said that this list 
is not a suitable forum for debating the finer points of feminist theory.


It seems to me that one of the reasons to have a CoC is to communicate 
that the identities and experiences of people who face discrimination 
are not up for debate. Yet here it seems they are, in fact, being called 
into question, even though people have already expressed discomfort and 
asked for it to stop or move off-list.


I've bcc'ed guix-maintain...@gnu.org, since someone requested that earlier.

-Philip

P.S.: Personally, I'm fine with either the current CoC text or with 
Simon's proposed update to bring it into alignment with upstream, if 
that has consensus.

Re: Faster "guix pull" by incremental compilation and non-circular modules?

[Philip McGrath]

Hi,

On Sunday, February 20, 2022 7:19:07 AM EST Maxime Devos wrote:
> Ekaitz Zarraga schreef op zo 20-02-2022 om 11:34 [+]:
> > Making a Guix pull is unpredictable too...
> 
> An idea for making "guix pull" faster:
> 
>   1. Make package imports non-circular, breaking up package modules
>  in parts when necessary.
> 
>   2. Instead of building all of Guix as a single derivation,
>  create a DAG of derivations.  More concretely:
> 
>  First read the *.scm files to determine which module imports
>  which modules. Then to compile, say, (gnu packages acl),
>  a derivation taking gnu/packages/acl.scm and its dependencies
>  gnu/packages/attr.go, gnu/packages/base.go, ... is made
>  compiling gnu/packages/acl.scm to a gnu/packages/acl.go.
> 
>  Then to build all of Guix, 'union-build' or 'file-union' is used.
> 
> The benefit is that if, say, gnu/packages/gnunet.scm is changed, then
> the old gnu/packages/acl.go will be reused because its derivation
> doesn't depend on gnu/packages/gnunet.scm (*).
> 
> The need for non-circular imports can be avoided by computing the
> strongly-connected components and compiling all the modules in a
> component as a single unit.
> 
> Greetings,
> Maxime.
> 
> (*) Actually I didn't verify if acl.scm depends on gnunet.scm or not.

I was just (or maybe am still?) dealing with some issues caused by cyclic
imports of package modules while updating Racket to 8.4: see in particular
, as well as #66, #112, and #113 in the
same thread.

As I mentioned there, I am most familiar with the semantics of Racket
modules,[1][2][3][4][5][6] and secondarily with R6RS libraries.[7]

I find the semantics of Guile's cyclic module imports very confusing, and I
don't know of any documentation for what is and isn't supported other
than advice from Ludo’ in —is there any?

Racket's module system fundamentally requires that module imports for a DAG
(see footnote 1 in § 2.2 of [1]), which is necessary to provide “The Separate
Compilation Guarantee”[1][6]:

> Any effects of the instantiation of the module’s phase 1 due to compilation
> on the Racket runtime system are discarded.

This is an extremely useful property, especially if you care about
reproducibility.

I know that R6RS does not provide The Separate Compilation Guarantee, since
§ 7.2 [8] explicitly says:

> An implementation may distinguish instances/visits of a library for
> different phases or to use an instance/visit at any phase as an
> instance/visit at any other phase. An implementation may further expand
> each library form with distinct visits of libraries in any phase and/or
> instances of libraries in phases above 0. An implementation may create
> instances/visits of more libraries at more phases than required to satisfy
> references. When an identifier appears as an expression in a phase that is
> inconsistent with the identifier's level, then an implementation may raise
> an exception either at expand time or run time, or it may allow the
> reference. Thus, a library whose meaning depends on whether the instances
> of a library are distinguished or shared across phases or library
> expansions may be unportable.

I haven't found anything in R6RS that explicitly addresses cyclic library
imports, but I think this passage from § 7.2 [8]:

> If any of a library's definitions are referenced at phase 0 in the expanded
> form of a program, then an instance of the referenced library is created
> for phase 0 before the program's definitions and expressions are evaluated.
> This rule applies transitively: if the expanded form of one library
> references at phase 0 an identifier from another library, then before the
> referencing library is instantiated at phase n, the referenced library must
> be instantiated at phase n. When an identifier is referenced at any phase n
> greater than 0, in contrast, then the defining library is instantiated at
> phase n at some unspecified time before the reference is evaluated.
> Similarly, when a macro keyword is referenced at phase n during the
> expansion of a library, then the defining library is visited at phase n at
> some unspecified time before the reference is evaluated.

combined with the specification in § 7.1 [9] that:

> The expressions of variable definitions are evaluated from left to right, as
> if in an implicit letrec*, and the body expressions are also evaluated from
> left to right after the expressions of the variable definitions. A fresh
> location is created for each exported variable and initialized to the value
> of its local counterpart. The effect of returning twice to the continuation
> of the last body expression is unspecified.

means that they are prohibited. If library (a) and library (b) were each to
import each other and each to refer to the other's definitions at phase 0,
then library (a) must be instantiated *before* library (b), but library (b)
must also be 

Re: Excessively energy-consuming software considered malware?

[Philip McGrath]

On Sunday, February 20, 2022 7:44:18 AM EST Taylan Kammer wrote:
> On 20.02.2022 13:37, Maxime Devos wrote:
> > The points about slippery slopes, research and niche use-cases seem
> > reasonable to me.  I do see follow-up questions though, should the
> > description in Guix warn about potential issues?  And should the
> > description focus on research uses? ...
> > 
> > More concretely, the p2pool description is:
> > 
> > ‘Monero P2Pool is a peer-to-peer Monero mining pool.  P2Pool
> > combines the advantages of pool and solo mining; you still fully
> > control your Monero node and what it mines, but *you get frequent
> > payouts like on a regular pool.*’
> > 
> > This is quite a bit different from, say, aircrack-ng which seems
> > to be mostly about assessing security, whereas the p2pool description
> > is about gaining money (see ‘payouts’), and without mentioning that
> > mining costs a lot of energy (and hence money, and possibly the money
> > that is gained by mining is smaller than the amount lost due to energy
> > costs!).
> > 
> > My dislike for the description of p2pool might just be my views on
> > money leaking through, though.
> > 
> > Greetings,
> > Maxime.
> 
> I guess a different description would be better.  It sounds almost
> like a sales pitch with the mention of making money. :-)
> 
> I'm not familiar with Monero/P2Pool so I don't know what a better
> description might sound like though.

This description sound like it is promising that, if you run this software, 
you will "get frequent payouts". I don't think that's a claim Guix can or 
should make. I guess there's maybe some tension about to what extent package 
descriptions are speaking on behalf of the package or on behalf of Guix, but I 
don't think it usually causes too much confusion for the description of a 
package like `pypy3` to say what the program's developers think its merits 
are, without Guix as a project appearing to take a position on which is the 
best Python implementation. However, a package description claiming that you 
will get money if you use it seems like quite an extreme case.

(More generally, I share both the concerns about the impacts of crypto mining 
and the concern that for Guix to refuse to package such software might have 
problematic implications.)

-Philip

Re: better error messages through assertions

[Philip McGrath]

Hi,

On 2/14/22 17:32, Ricardo Wurmus wrote:

As you can probably tell easily by looking at this message, the
“service” field of the operating system configuration looked something
like this:

   (services (append (list a b c %desktop-services) #;oops))

instead of this

   (services (append (list a b c) %desktop-services))

This is because INSTANTIATE-MISSING-SERVICES — and FOLD-SERVICES, and
many more — assumes that it is only passed a plain list of services.  It
then proceeds to call SERVICE-KIND on what may or may not be a service.

I think we should add simple type checks, something like this:

   (define (listof pred)
 (lambda (thing)
  (and (list? thing) (every pred thing
   …
   (define (assert-type type-check thing message)
 (or (false-if-exception (type-check thing))
 (report-error (G_ "type error: …\n" message

   ;; Use ASSERT-TYPE in an example procedure.
   (define (do-something-with-services services)
 (assert-type (listof service?) services
  "SERVICES must be a list of  values.")

 ;; Do things…
 (map service-kind services))

What do you think?  There are many different ways of implementing this
(a new variant of DEFINE that also accepts a type declaration, an assert
like above, a fancier assert that composes a helpful error message by
itself, a separate type declaration that is looked up only when the
corresponding procedure is called in a certain context, etc), but I’d
first like to know if there is consensus that we want something like
this.



As a Guix user and contributor, I would love better error messages.

As a Racketeer, I think you're half way to reinventing contracts.

In particular, since the operating system services field is thunked, 
this example already points to the desirability of higher-order contracts.


Using "contracts" need not initially involve all the bells and whistles 
of Racket's contract library. For example, higher-order contracts can be 
implemented with `lambda` in the absence of special runtime support for 
chaperones and impersonators, as illustrated in [1]. But the Racket 
community has accumulated a great deal of both theoretical insight and 
implementation experience in many subtle corners of this problem: I hope 
Guix can build on that experience.


-Philip

[1]: https://docs.racket-lang.org/guide/Building_New_Contracts.html

Re: License issue with SRFI 5

[Philip McGrath]

Hi,

On 10/29/21 10:44, Ludovic Courtès wrote:

Hi,


This reply has taken a while because I did as you suggested and wrote 
some free replacement documentation (which involved paging back in some 
confusing details[1] and shaving several yaks), so now we have something 
concrete to talk about.


To start with that, my proposed approach, which I hope satisfies the 
FSDG, is here: https://github.com/racket/srfi/pull/15


There are some loose ends to tie up (e.g. I have 48 pull requests open 
making tedious changes to the HTML markup of SRFI specification 
documents to bring Racket's version into alignment with upstream, which 
the current SRFI editor, Arthur Gleckler, is stalwartly reviewing), but 
Guix can cherry-pick it on top of Racket 8.4 (which may be released as 
soon as this week).


The question is, do these changes resolve the situation for Guix (as I 
hope they do for Debian and Fedora, too), or is there anything 
more/different that needs to be done?




Philip McGrath  skribis:


Since 2005, SRFIs have used the MIT/Expat license, and all but two
older SRFIs were relicensed: however, the SRFI editors were not able
to contact the author of SRFI 5, Andy Gaynor, so it remains under the
original SRFI license.[1] That license, modeled on that of IETF RFCs,
was intended to be quite permissive while also trying to ensure
derivative works would not be confused with the final, official SRFI
itself. (The many versions of some SRFIs that nonetheless have come up
while hunting down related issues has given me some sympathy for that
goal.) Unfortunately, the restrictions on modifications went to far,
at least in the judgement of Debian and Fedora.

Here is the license text, as it appears at
<https://srfi.schemers.org/srfi-5/srfi-5.html> and
<https://docs.racket-lang.org/srfi-nf/srfi-std/srfi-5.html>:


Oh.

Do people actually use SRFI-5? (Honest question, I didn’t know about it
and don’t feel much appeal.)

Is there code inside Racket that uses it?


In short, no.

When the Debian package maintainer first discovered the problem in late 
2017 [2], the conclusion was that SRFI 5 wasn't used by any other part 
of Racket. The workaround [3] contributed by the Fedora package 
maintainer in early 2018 was to move the offending files into new 
"srfi-lib-nonfree" and "srfi-doc-nonfree" packages and for distribution 
maintainers to patch out the dependencies on them from the "srfi" 
package. This was all to accommodate Racket's exceptionally strong 
commitment to backwards compatibility, because it's a breaking change 
for a Racket package to stop providing a particular module (or 
documentation for a module) without adding an `implies` dependency on 
the new package it has moved to, and Racket packages are never supposed 
to make breaking changes. (The idea is you should make a new package 
under a different name, analogous to how a Debian system could have both 
the "guile-2.2" and the "guile-3.0" packages installed.)


(At the time there were also issues with SRFIs 32 and 29 at the time, 
but those had actually been relicensed to MIT/Expat; it was just a 
matter of resolving lots of confusing different versions of things.)


I then noticed "srfi-lib-nonfree" fly across my terminal, learned of the 
whole mess, and contributed a free reimplementation of SRFI 5 in 2019 
[4], using `syntax-parse` to give vastly better error reporting. At that 
point "srfi-lib-nonfree" became an empty, free package that exists only 
to provide an `implies` dependency on "srfi-lib".


According to [5], though, SRFI 5 is also implemented for Chez, Gauche, 
Larceny, STklos, and Scsh: I'm in the process of adapting my free 
implementation and compatibility test suite, which covers various 
ambiguities and errata [1], to R6RS to send upstream, so hopefully no 
one has to deal with this headache again.





Racketeers have high expectations of their documentation, like being
able to right-click on an identifier in DrRacket (or the equivalent in
Emacs with racket-mode) and jump to the locally-installed
documentation for the relevant binding according to lexical scope and
the module system---even for a binding like `let`, which is defined by
27 different Racket modules, including `srfi/5`.


[...]



This all raises a few questions about Guix policy:

  1. Can Guix distribute the official SRFI 5 standard document under
 the license listed above?


I don’t think so; it looks like a non-free software license to me.


  2. If not, can Guix distribute free documentation that links
 to an online copy of the official SRFI 5 standard document?


I think it would be easy to do a “clean room” section documenting SRFI-5
no?  I mean, once you know the spec, documenting it is trivial, to the
point that it’s even hardly copyrightable (there’s little invention).



This is what I've done.

With these changes, documentation for the `srfi/5` module points to

Re: Public key pinning in guix?

[Philip McGrath]

Hi,

On 1/9/22 06:54, Maxime Devos wrote:

Hi,

Philip McGrath schreef op za 08-01-2022 om 11:37 [-0500]:

This sounds like HTTP Public Key Pinning (HPKP).[1] AIUI, HTTP Public
Key Pinning was deprecated, and support has been removed from major
browser engines by January 2020.[2][3][4] While it seemed like a good
idea for reasons like the ones you list, apparently it not only proved
very difficult for site administrators to configure, with severe
consequences for mistakes, it also enabled potential ransomware attacks
and other bad stuff.[6]

I never followed this feature closely and don't have a strongly-held
opinion on the merits, but, if the "web platform" has deprecated this
feature---more concretely, if it is Considered Harmful by sysadmins and
servers are configured with the expectation that no one does this any
more---I don't think it would improve reliability for Guix to
unilaterally revive HPKP.


It does instead sound like HPKP -- however, what I proposed is in some
sense the inverse of HPKP:

Instead of a webserver telling the client to pin a certain key, the
client has pinned a certain key in advance.  So pinning is Guix'
responsibility, not the web server's.

What I propose is more close to ‘certificate pinning’ (actually
public key pinning), see e.g.
<https://blogs.fsfe.org/jens.lechtenboerger/2014/03/10/certificate-pinning-with-gnutls-in-the-mess-of-ssltls/>.
Even then, it's a bit different: the certificate of the server must
be correct according to both the root CAs in $SSL_CERT_DIR
AND the pin list.


I agree that the specifics of what you proposed are significantly 
different than HPKP as implemented by browsers, and also that public key 
pinning can be quite useful for securing communication with a known server.



That said, let's not use pins when doing "guix pull",
"guix perform-download" or "guix substitute" because "guix pull"
is rather essential and the guix used as the daemon is rarely
updated -- temporarily breaking "guix refresh", "guix download"
or "guix import" is much less a problem.


Yes, it seems much lower-risk to have potential breakage of the 
contributor-oriented commands than the user-facing ones. Also, for many 
of the user-facing commands, we'll ultimately verify the content of what 
we download, whether by hash or channel introductions.



  * Does the fact that web browsers deprecated HPKP matter?

I don't think so. E.g. [5] says that

‘However, this exposes as part of the Open Web Platform considerations
that are external to it: specifically, the choice and selection of CAs
is a product-level security decision made by browsers or by OS vendor,
and the choice and use of sub-CAs, cross-signing, and other aspects of
the PKI hierarchy are made independently by CAs.’

I think that "guix download/refresh/import" qualifies as ‘product level’,
or ‘browser’ here, and that Guix qualifies as OS vendor.

I don't think that the bit about sub-CAs, cross-signing, etc. is relevant
here: we pin public keys, not CAs, and the public key pin can be adjusted
whenever the website decided to use another public key -- albeit with
a (hopefully brief?) period where it is temporarily inaccessible to
"guix download/refresh/import".


The part of the deprecation of HPKP that seems most relevant is that 
some number of servers---I suspect it may be a large number---are 
configured under the assumption that no one relies on their using any 
particular public key. For example, Certbot in its default configuration 
will rotate to a new public key every time it gets a new certificate, 
i.e. every two months (30 days before expiration). There is a 
`--reuse-key` flag, but I don't get the impression that it's widely used 
unless the server knows clients will rely on the key remaining the same. 
In fact, I've heard some people argue against reusing keys, as a way to 
limit the window for damage that could be done by a compromised private 
key. I'm not trying to take a position on which is the best way to 
manage a web server, just to point out that I think some servers will 
change keys very often.


If we have some reason to believe that, say, "hackage.haskell.org" will 
have a stable public key for a reasonably long time, then I'm all for 
this! And I'm not vehemently against it, anyway: there are mitigations 
to the potential downsides for end users. But, if we don't know the 
server's policy, I can imagine even just the seven domains in your 
original email producing more than one break per month, and I don't know 
how we'd distinguish a real attack from a routine failure. It's just a 
hypothesis, though: if anyone has more concrete data, I'd be interested 
to hear it.


-Philip

Re: Public key pinning in guix?

[Philip McGrath]

Hi,

On 1/7/22 16:24, Maxime Devos wrote:

The purpose is to resist a compromise of the CA system. More
concretely, if you now do "guix refresh -u minetest-moreores"
then a MITM that compromised a CA cannot secretly replace
minetest-moreores with a mod that mines bitcoin for the MITM,
or something.

Possibly also useful for "guix download", "guix import", "guix lint",
"guix build --with-latest=...".

A downside is that whenever content.minetest.net changes public keys,
the pinned public key in Guix needs to be updated. How often does this
happen? I wouldn't now. This could be partially automated with
a "./pre-inst-env guix update-the-pinned-keys" script, and there could
be an "GUIX_IGNORE_KEY_PINNING=yes" environment variable as escape
hatch.

WDYT, worth the trouble or not?



This sounds like HTTP Public Key Pinning (HPKP).[1] AIUI, HTTP Public 
Key Pinning was deprecated, and support has been removed from major 
browser engines by January 2020.[2][3][4] While it seemed like a good 
idea for reasons like the ones you list, apparently it not only proved 
very difficult for site administrators to configure, with severe 
consequences for mistakes, it also enabled potential ransomware attacks 
and other bad stuff.[6]


I never followed this feature closely and don't have a strongly-held 
opinion on the merits, but, if the "web platform" has deprecated this 
feature---more concretely, if it is Considered Harmful by sysadmins and 
servers are configured with the expectation that no one does this any 
more---I don't think it would improve reliability for Guix to 
unilaterally revive HPKP.


-Philip

[1]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Public_Key_Pinning
[2]: https://scotthelme.co.uk/hpkp-is-no-more/
[3]: 
http://web.archive.org/web/20200618234723/https://www.fxsitecompat.dev/en-CA/docs/2019/http-public-key-pinning-is-no-longer-supported/

[4]: https://chromestatus.com/feature/5903385005916160
[5]: 
https://groups.google.com/a/chromium.org/g/blink-dev/c/he9tr7p3rZ8/m/eNMwKPmUBAAJ

[6]: https://scotthelme.co.uk/using-security-features-to-do-bad-things/

Re: default tar format for "make dist" and patch file length

[Philip McGrath]

On 11/19/21 09:54, Ludovic Courtès wrote:

Vagrant Cascadian  skribis:

So, I guess I'm leaning towards making the guix lint check a little more
lenient.

Thoughts?


That sounds even better, I’m all for it (changing (guix lint) + fixing
the two remaining issues)!


It might also help to change the warning given by the check.

When a program called "lint" tells me that something is too long, I 
understand that to mean that what I've done is generally considered bad 
style, but there might be a very good reason to do it in some specific 
case. For example, I might exceed a line length guideline to avoid 
inserting linebreaks into a URL.


If instead `guix lint` is telling us about a hard limit that will break 
things, I think it should say so clearly.


-Philip

License issue with SRFI 5

[Philip McGrath]

Hi Guix,

I was recently reminded of a thorny license issue with the SRFI 5 
standard document, which is part of the main Racket distribution. People 
from Racket, the SRFI community, Debian, and Fedora have taken steps to 
help mitigate the problem: I want to make some further improvements, but 
I first need clarity on what Guix's policy requires and permits.


Since 2005, SRFIs have used the MIT/Expat license, and all but two older 
SRFIs were relicensed: however, the SRFI editors were not able to 
contact the author of SRFI 5, Andy Gaynor, so it remains under the 
original SRFI license.[1] That license, modeled on that of IETF RFCs, 
was intended to be quite permissive while also trying to ensure 
derivative works would not be confused with the final, official SRFI 
itself. (The many versions of some SRFIs that nonetheless have come up 
while hunting down related issues has given me some sympathy for that 
goal.) Unfortunately, the restrictions on modifications went to far, at 
least in the judgement of Debian and Fedora.


Here is the license text, as it appears at 
 and 
:



Copyright (C) Andy Gaynor (1999). All Rights Reserved.

This document and translations of it may be copied and furnished to others, and 
derivative works that comment on or otherwise explain it or assist in its 
implementation may be prepared, copied, published and distributed, in whole or 
in part, without restriction of any kind, provided that the above copyright 
notice and this paragraph are included on all such copies and derivative works. 
However, this document itself may not be modified in any way, such as by 
removing the copyright notice or references to the Scheme Request For 
Implementation process or editors, except as needed for the purpose of 
developing SRFIs in which case the procedures for copyrights defined in the 
SRFI process must be followed, or as required to translate it into languages 
other than English.

The limited permissions granted above are perpetual and will not be revoked by 
the authors or their successors or assigns.

This document and the information contained herein is provided on an "AS IS" 
basis and THE AUTHOR AND THE SRFI EDITORS DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, 
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT 
INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A 
PARTICULAR PURPOSE.
Before going any further, let me emphasize that a great deal of progress 
has been made since distro packagers first pointed out the problem.[2] 
First, there was work to separate the problematic SRFIs (plural, at the 
time) into "srfi-lib-nonfree" and "srfi-doc-nonfree" Racket packages.[3] 
Distro packagers could cleanly remove these from their Racket 
distributions, and they also work properly if a user of the 
distro-packaged Racket explicitly decides to install them. Later, I 
wrote a free implementation of SRFI 5, so "srfi-lib-nonfree" is now 
(confusingly) free software, an empty package definition for backwards 
compatibility.[4] SRFI 29 was supposed to have been converted to the 
MIT/Expat license in 2005, but somehow was missed: just today, the 
current SRFI editor, Arthur Gleckler, confirmed the permission of the 
author, Scott Miller, and updated the official document.[5][6][7] So the 
only problem remaining is the SRFI 5 standard document.


Racketeers have high expectations of their documentation, like being 
able to right-click on an identifier in DrRacket (or the equivalent in 
Emacs with racket-mode) and jump to the locally-installed documentation 
for the relevant binding according to lexical scope and the module 
system---even for a binding like `let`, which is defined by 27 different 
Racket modules, including `srfi/5`. My tentative plan is to write free 
replacement documentation for SRFI 5, eliminate everything from 
"srfi-doc-nonfree" but the official SRFI 5 document itself, and program 
the free SRFI 5 documentation (in Racket's Scribble language) to link to 
the SRFI 5 document at racket-lang.org if there isn't a local copy 
installed.


This all raises a few questions about Guix policy:

 1. Can Guix distribute the official SRFI 5 standard document under
the license listed above?

 2. If not, can Guix distribute free documentation that links
to an online copy of the official SRFI 5 standard document?

 3. Would it be permissible for the free documentation to
include instructions for installing the official SRFI 5 standard
document locally, e.g. `raco pkg install srfi-doc-nonfree`?
(Or perhaps `raco pkg install srfi-5-std-doc`, to avoid the
implication of arbitrary non-free materials?)

(Of course, if someone can manage to contact Andy Gaynor, that would be 
even better!)


The first question is fundamental but less immediately important to me: 
since Debian and Fedora, at 

Re: How did you handle making a GNU/Linux distribution?

[Philip McGrath]

On 9/12/21 11:09 PM, Christine Lemmer-Webber wrote:

Philip McGrath  writes:

Christine Lemmer-Webber had floated the idea at some point of trying
to integrate Racket and Guile.

IIRC, I think what she's had in mind was trying to make a Guile
backend for Racket along the lines of the Chez Scheme backend (or the
BC backend, or experimental backends like Pycket).


Yes that's what I had in mind :)


A few stray thoughts, all with the caveat that I haven't actually tried 
any of this ...



there are two things that have struck me
as downsides:

  1. Flatt et al. say in "Rebuilding Racket on Chez Scheme (Experience
 Report)" (§6, p. 13) that, "If our task were to compile Racket to an
 existing target, then we would not have achieved such a high degree
 of compatibility. … we have taken the liberty of modifying Chez
 Scheme to make it an easier target for Racket."

 https://www.cs.utah.edu/plt/publications/icfp19-fddkmstz.pdf

 Presumably a Racket-on-Guile project would face the same trade-off,
 where modifications to Guild, if Racket CS is a guide, could require
 hard work over many years, and lesser compatibility would make the
 result less useful.


At one point when I spoke to Matthew, he was very optimistic that the
work on porting on top of Chez would open Racket to running on top of
many other backends.  But yes... since there have been so many "custom"
modifications to Chez, it's easier to be skeptical about that these
days...


I think there a few possible senses for what "running on top of many 
other backends" could mean. My impression overall is that it has gotten 
easier, but may not necessarily be easy.


The most clearly demonstrated is that it seems to be easier to port Chez 
Scheme to new architectures than to port Racket BC's VM. Racket CS 
supports the Apple M1 chip natively (and hopefully will support the 
Linux kernel on the M1 when that stabilizes), which Racket BC does not. 
Racket CS also fully supports some platforms (ARM in general, IIRC) on 
which Racket BC lacks support for futures, places, or the JIT. The most 
promising route to Racket on WASM also seems to be adding a WASM backend 
to the Chez Scheme compiler. (In fairness, there are also some 
architectures, like ppc64le, to which no one has ported Racket CS yet, 
for which Racket BC can offer at least some support via C.)


More generally, the "linklet" abstraction[1] seems to provide a much 
more clear boundary between the responsibilities of a backend 
implementation and the common frontend than existed before Racket 7.0. 
For example, the backend doesn't have to deal with macro expansion or 
even shadowing of variables: it just need to have some way to 
instantiate linklets (likely with further backend-specific compilation) 
and to provide a whole bunch of primitives. I believe I've heard Sam 
Tobin-Hochstadt say that linklets have made it possible for Pycket (the 
experimental Racket implementation in Reticulated Python) to run most 
Racket code.


[1]: https://docs.racket-lang.org/reference/linklets.html

Another benefit of linklets is that they've defined a clear path for 
implementing parts of Racket in Racket itself, like the regexp 
implementation on Racket CS. This seems like it could help a lot with 
supplying that very large number of primitives.


So I expect it would be entirely possible to implement a linklet-based 
Racket backend in Guile. I do suspect that getting production-quality 
performance and compatibility could be a lot of work. But my bigger 
concern is ...



  2. As you probably know, Racket programs can't generally use
 Chez Scheme implemented libraries, because Chez Scheme effectively
 is the "unsafe" layer of the Racket VM. For example, not all Racket
 procedures are Chez Scheme procedures, and Racket's continuations
 wrap Chez Scheme's to implement delimited and composable control,
 threads, parameters, full continuation marks, etc.

 For Racket CS, this isn't a great loss (there aren't so many
 Chez-specific libraries, and portable libraries can run in Racket's
 R6RS language), but, for a hypothetical Racket-on-Guile,
 bidirectional interoperability would be a big attraction: imagine
 Guix, Shepherd, Mcron, syntax/parse, racket/contract, and more all
 working together.


I don't think a Racket-on-Guile that worked like Racket-on-Chez would 
achieve the most interesting benefit (IMO) of bringing the two languages 
together: safe, bidirectional interoperability.


Probably there are many ways to achieve that goal. In principle, one 
could write a Racket-on-Guile implementation where arbitrary Guile could 
run without breaking Racket's invariants, but I expect that would make 
the task even harder: presumably that's why Racket-on-Chez doesn't work 
that way.


But if linklets are as good as an abstraction as they seem to be for 
compiling code in the presence

Re: How did you handle making a GNU/Linux distribution?

[Philip McGrath]

Hi Simon,

On 8/23/21 11:38 AM, zimoun wrote:

The bootstrap path of Racket in Guix is not clear to me.  I miss if a
Racket interpreter or compiler in its binary format is used (as
Haskell for instance) or if all is compiled from source using tools
already bootstrapped.  Anyway, I miss what you would like bootstrap?
If you mean a trusted seed in order to start a package collection, you
could use the current Guix binaires---as a starting point.


I wrote a long comment at the top of gnu/packages/racket.scm that 
explains the current state of affairs. (For anyone unfamiliar with the 
Guix source tree: 
)


In brief, Racket is bootstrapped from source using only a C 
compiler---including bootstrapping Racket's fork of Chez Scheme without 
an existing Chez Scheme compiler---with one exception: the "expander" 
subsystem (which implements the reader, module system, and macro 
expander) is implemented in Racket, and the older C implementation was 
completely replaced as of Racket 7.0. (One of the motivations was to fix 
many problems with the old implementation.) For bootstrapping, an 
intermediate representation as a "schemified" linklet is checked into 
the Racket source repository.


This impact is mitigated because, by design, the intermediate 
representation is "human-readable and -editable Scheme code". 
Anecdotally, when I read diffs of changes to the Racket repository, I 
sometimes read tens of lines of code before I realize that I'm looking 
at the generated version of the file, not the source. So the expander is 
not bootstrappable in the most rigorous sense, but it is readily auditable.


If someone wanted to tie the knot, the task would be to use an existing 
bootstappable language to write a minimal expander that can expand the 
canonical Racket implementation: if your output is identical to the 
checked-in generated files, you've succeeded. (Using Guile could avoid 
the heavy maintenance burden of the old C expander implementation, and 
indeed you could write just enough of an adapter to load the Racket 
source files into Guile, which is the technique Racket uses to bootstrap 
the Chez Scheme compiler.) I expect Racket would welcome a contribution 
along those lines; it just hasn't been a priority for anyone yet.


-Philip

Re: How did you handle making a GNU/Linux distribution?

[Philip McGrath]

Hi Sage,

On 8/22/21 5:53 PM, Sage Gerard wrote:

Thanks for the detailed answer!

It seems wise to adapt GNU Mes towards Racket or Chez Scheme instead of
Guile to bring GNU's benefits to more Scheme and Racket programmers. Has
someone already tried something like that?


I haven't tried Xiden yet, and I haven't done any concrete work toward 
this (I have been working on managing Racket packages with Guix), but 
Christine Lemmer-Webber had floated the idea at some point of trying to 
integrate Racket and Guile.


IIRC, I think what she's had in mind was trying to make a Guile backend 
for Racket along the lines of the Chez Scheme backend (or the BC 
backend, or experimental backends like Pycket).


As I said, I haven't actually tried any of this, but, as I've thought 
about what might be involved, there are two things that have struck me 
as downsides:


 1. Flatt et al. say in "Rebuilding Racket on Chez Scheme (Experience
Report)" (§6, p. 13) that, "If our task were to compile Racket to an
existing target, then we would not have achieved such a high degree
of compatibility. … we have taken the liberty of modifying Chez
Scheme to make it an easier target for Racket."

https://www.cs.utah.edu/plt/publications/icfp19-fddkmstz.pdf

Presumably a Racket-on-Guile project would face the same trade-off,
where modifications to Guild, if Racket CS is a guide, could require
hard work over many years, and lesser compatibility would make the
result less useful.

 2. As you probably know, Racket programs can't generally use
Chez Scheme implemented libraries, because Chez Scheme effectively
is the "unsafe" layer of the Racket VM. For example, not all Racket
procedures are Chez Scheme procedures, and Racket's continuations
wrap Chez Scheme's to implement delimited and composable control,
threads, parameters, full continuation marks, etc.

For Racket CS, this isn't a great loss (there aren't so many
Chez-specific libraries, and portable libraries can run in Racket's
R6RS language), but, for a hypothetical Racket-on-Guile,
bidirectional interoperability would be a big attraction: imagine
Guix, Shepherd, Mcron, syntax/parse, racket/contract, and more all
working together.

If I were going to work on this, I'd start by looking at having Racket 
and Guile coexist as siblings with interoperability through their FFIs 
level. Even better, eventually you could compile Guile to Racket 
linklets, so the two could coexist in the same primitive module system. 
There would probably always need to be something along the lines of 
require/typed to interoperate between the languages, since Guile has 
mutable pairs and an unusual approach to falsehood and nullity to let 
Scheme's #f and '() coexist with Emacs List's nil. (See 
https://www.gnu.org/software/guile/manual/html_node/Nil.html).



I'm at the point where users are requesting a GNU/Linux distribution for
Xiden, such that Racket is the primary language for day-to-day
operation. I'm ignorant of the scope of work, and am unsure if I can do
it alone.


To me, at least, the scope of the work in creating a new GNU/Linux 
distribution seems daunting. My hope would be that bringing Racket and 
Guile closer together would let most if not all of the effort be shared.


Hope some of this is useful, and best of luck!

-Philip

Re: cowsay could attract copyright and trademark enforcement action

[Philip McGrath]

On 7/16/21 6:26 PM, Leo Prikler wrote:> As others point out, it is 
debatable whether or not such a trademark

can be enforced, but what we should be discussing -- and I'd be more
than happy to be wrong on this -- is whether or not the cowfiles can be
distributed under GPL-compatible terms/in accordance with the FSDG.
These files seem to me to be non-functional data, analogous to game 
graphics, which the FSDG do not require to have free/libre licenses. 
Just as firmware is still code, even if it is represented as an array of 
numbers, I think ASCII art templates are data, even if they are embedded 
in Perl fragments.


(In the Lisp tradition, I think the distinction between code and data is 
a fuzzy one, but this case seems fairly clear to me.)


On the broader issues, I also am not a lawyer, but I would expect all of 
these files to be protected by the rights of fair use, fair dealing, and 
other limitations or exceptions to copyright. Limitations and exceptions 
seem to trademark law also seem to apply.


Fair use and analogous rights are vitally important protections of 
freedom, including software freedom. For example, in the recent case of 
Google v. Oracle, fair use was the legal right which protected the 
freedom of the Java APIs. (The decision also left open the possibility 
that APIs may not be copyrightable at all.)


Apparently these files are over 20 years old and have been widely 
distributed without objections. If Guix were to remove these files, we 
would implicitly be disagreeing with the apparent community consensus 
and saying that users do not have the freedom to create, use, or share 
such cowfiles. That's not a position I think Guix ought to take, 
especially not against the apparent consensus of the broader free 
software community.


-Philip

Re: unifying mcron and shepherd, service woes, improvements

[Philip McGrath]

On 5/15/21 12:59 PM, Ludovic Courtès wrote:

raingloom  skribis:

Or is everyone else happy with the current design and it's just me who
can't use Shepherd properly? 


I think it’s fair to say it’s rough on the edges.  :-)

One thing that’s on the to-do list is switching to a real event loop in
lieu of the current ad-hoc blocking design (this was discussed recently
on this mailing list).  The switch to ‘signalfd’ in the last release in
a step in that direction.  This will unlock “socket activation” and
possibly timers as you mentioned.


From a newcomer's perspective---I haven't used shepherd at all yet---I 
was surprised by the discussion leading up to 
:


On 3/19/21 10:42 PM, raid5atemyhomework wrote:
> Perhaps the point is not so much to be *multi-threaded* but to be 
*concurrent*


From very occasionally reading Andy Wingo's blog, I understand that 
Guile has a Concurrent ML implementation: 
.


I haven't programmed with it in Guile at all, but, from my experience 
using Racket's Concurrent ML system, I think it is the right foundation 
for a supervisor daemon. CML provides a consistent, extensible way to 
express, "when something happens, do some action", where "something 
happens" might be:


  * a timer fires;
  * a service, or a set of them, come up;
  * a subprocess exits (successfully or not);
  * a file descriptor is ready for reading and/or writing;
  * a "green thread" (IIUC, a Guile "fiber") has an uncaught exception;
  * a signal is received from the OS kernel;
  * a user sends a command from the client program;

... etc, etc, etc. The CML implementation handles many low-level details 
that are easy to get wrong.


-Philip

Re: Racket 8 and store references (was [security] Chunked store references in .zo files in Racket 8 #47614)

[Philip McGrath]

Indeed, I expect this is a more precise diagnosis of the same problem. 
My patch in https://issues.guix.gnu.org/47180 solves it by putting the 
store references (search paths for foreign libraries) in a configuration 
data file that isn't compiled, so they don't end up in .zo files in the 
first place.


The .zo format is intentionally undocumented and subject to breaking 
change, including from different compilation options. At a minimum, a 
change to the Racket version number signals a breaking change to 
compiled code (e.g. Git is now at 8.0.0.13, so 13 breaking changes since 
the release). Internally, I don't know all the details, but the normal 
8.0 .zo format has a Racket layer around the Chez Scheme object format, 
which seems to be very complex: it looks like it supports 
user-configurable compression at the granularity of the individual 
object within an object file. So it seems much better to avoid rewriting 
.zo files altogether.


-Philip

On 4/6/21 9:20 PM, Jack Hill wrote:

On Tue, 6 Apr 2021, Mark H Weaver wrote:


Anyway, I doubt that imposing such a limitation would adequately solve
the problem here of chunked references in Racket 8, because I suspect
that Racket 8 could split store references at arbitrary points in the
string.  I doubt that we can safely assume that the hash component of
store references will be stored contiguously in *.zo files.


Mark and everyone,

I wanted to spin off a subthread on guix-devel, to make you aware of 
another problem that we've run into with reference in .zo getting 
mangled: https://issues.guix.gnu.org/47180


Best,
Jack