Re: Profile definition, was Re: bug#53224: Cookbook recipe about profile collisions

[Matt]

  On Tue, 18 Jan 2022 10:36:20 -0500 Ludovic Courtès  wrote 

 > Hi,
 > 
 > Leo Famulari  skribis:
 > 
 > > On Mon, Jan 17, 2022 at 12:56:20PM -0500, Matt wrote:
 > >> Leo is 100% correct that my understanding is still rather weak. I'll do 
 > >> my best despite that to help make the documentation better.
 > >
 > > I hope you will not feel too bad about that. Remember, everyone begins
 > > by not knowing anything. Your situation is not unique at all. Rather,
 > > your energy for improving the documentation for yourself and others is
 > > exemplary, and will improve Guix in the long run.
 > 
 > I agree, and I apologize for throwing my own guess of what you did and
 > did not do, Matt.
 
Hey, thanks.  I accept your apology and really appreciate it.  Recent 
interactions had left me feeling unheard and wondering whether I was wasting my 
time with the Guix community.  I appreciate being seen. I'll continue working 
with Leo and jgart on documentation.

Language menu in the HTML manual

[Ludovic Courtès]

Hello Guix!

I added a language menu in the on-line HTML manual that you can see in
the top-right corner of manual pages:

  https://guix.gnu.org/manual/devel/es/html_node/Empezando.html
  https://guix.gnu.org/manual/devel/ru/guix.ru.html

(CSS needs to be fixed so the menu item colors look right.)

This is done with these two commits:

  
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=7eb883b7c284c78cc17093bfc4ef2d70e0acad83
  
https://git.savannah.gnu.org/cgit/guix/guix-artwork.git/commit/?id=db26102d4149a3a857701add00d23bdf38e21911

I think eventually we could also add explicit links to, say, report
translation errors or other issues.

Let me know what you think!

Ludo’.

Re: Preservation of Guix report 2022-01-16

[Timothy Sample]

Hi,

Vagrant Cascadian  writes:

> On 2022-01-18, Ludovic Courtès wrote:
>> Timothy Sample  skribis:
>>> A really important thing to do at this point is to verify that some
>>> reasonable looking computation is covered by what we are doing already.
>>> For instance, is every source used to build Guile (or Python or R)
>>> preserved?  This will ensure that key sources are not missing, which is
>>> a real possibility given that everything so far has been purely a
>>> numbers game!
>>
>> I wonder if we could have something similar to ‘guix weather -c’, which
>> would highlight missing sources with many dependents.

Definitely.  The simplest way to do that is to use the PoG database
as-is, and just write a script that traverses the derivation graph
checking for coverage.  It’s a bit trickier to integrate into Guix
itself, since we would have to make the data available.  It’s a good job
for the Data Service, but I feel like it’s a long road from here to
there.

>> TeX Live is a big concern: it’s all Subversion, and everything depends
>> on those packages.  IIRC, SWH does not support Subversion yet; and when
>> it does, we’ll have to adjust our code so it can actually fetch
>> Subversion checkouts from SWH.  One issue is partial checkouts: all
>> these ‘texlive-’ packages refer to partial checkouts of the big TeX Live
>> repo.
>
> Maintain a git mirror of texlive SVN using git-svn or similar, and have
> guix packages use that, and make sure SWH ingests it? Is it difficult
> because of the size of TeX Live?
>
> A little ugly, sure, but maybe only include the revisions that guix is
> interested in for starters...

Fortunately, SWH does support Subversion, so we can avoid this.  They
haven’t visited the TeX Live sources yet, but I’m sure they will
eventually.

How to actually match up what we want with what they have is a big
question!  I imagine they do quite a number to make the SVN repos fit
their Git-inspired data model.  It’s not clear to me how it works.  I’ll
have to look over the loader sometime:

https://forge.softwareheritage.org/source/swh-loader-svn/repository/master/

One way to handle things like partial checkouts is to revisit storing
SWHIDs with our origins.  If you have the directory ID, you can just
download the directory from SWH without worrying about SVN at all.  It
would also give us an easy way to handle Bazaar and CVS (which are under
development at SWH).


-- Tim

Re: Preservation of Guix report 2022-01-16

[Vagrant Cascadian]

On 2022-01-18, Ludovic Courtès wrote:
> Timothy Sample  skribis:
>> A really important thing to do at this point is to verify that some
>> reasonable looking computation is covered by what we are doing already.
>> For instance, is every source used to build Guile (or Python or R)
>> preserved?  This will ensure that key sources are not missing, which is
>> a real possibility given that everything so far has been purely a
>> numbers game!
>
> I wonder if we could have something similar to ‘guix weather -c’, which
> would highlight missing sources with many dependents.
>
> TeX Live is a big concern: it’s all Subversion, and everything depends
> on those packages.  IIRC, SWH does not support Subversion yet; and when
> it does, we’ll have to adjust our code so it can actually fetch
> Subversion checkouts from SWH.  One issue is partial checkouts: all
> these ‘texlive-’ packages refer to partial checkouts of the big TeX Live
> repo.

Maintain a git mirror of texlive SVN using git-svn or similar, and have
guix packages use that, and make sure SWH ingests it? Is it difficult
because of the size of TeX Live?

A little ugly, sure, but maybe only include the revisions that guix is
interested in for starters...

It has been some years ago that I used git-svn, but it worked well for a
several year transition for me and a small number of projects...


live well,
  vagrant


signature.asc
Description: PGP signature

Re: using an SRFI that is not available in Guile

[Attila Lendvai]

> There’s no widespread “Maybe” idiom in Scheme; or rather, people use
> TYPE | #f as a way to approximate “Maybe”. It’s not ideal, primarily
> because appropriate handling is not statically checked. Yet, that’s
> what we have and I’m not convinced adding SRFI-189 to the mix would
> bring enough of an improvement to justify it.
>
> Thoughts?


configurations are full of boolean fields, where #f is a valid value.

to represent unset fields, we would essentially need to implement half
of srfi-189 (Maybe and Nothing), and in a potentially buggy way
(e.g. using a symbol like 'disabled to represent an unset field value
(i.e. the current solution) clashes with a field type of symbol?).

i don't think it's worth rolling our own when an srfi covers what we
need.

--
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“Go find yourself first, so you can find me.”
— Rumi (1207–1273)

Re: packages: [PoC] Expand range of objects 'add-input-label' can label

[Ludovic Courtès]

Hi,

elaexuo...@wilsonb.com skribis:

> From 3b8e7fa8fbd58e7e164e3730c708419f612b8549 Mon Sep 17 00:00:00 2001
> From: "B. Wilson" 
> Date: Sun, 16 Jan 2022 23:54:51 +0900
> Subject: [PATCH 1/2] packages: Expand range of objects 'add-input-label' can
>  label
> To: guix-patc...@gnu.org
>
> * guix/packages.scm (%auxiliary-files-subpath-dir): New variable.
> (add-input-label): Support labels from the name property of
> objects that have one.  Also, name auxiliary files from their
> subpath.

[...]

>  (define (add-input-label input)
>"Add an input label to INPUT."
>(match input
> @@ -576,7 +581,24 @@ (define (add-input-label input)
>   (list (package-name package) package))
>  (((? package? package) output);XXX: ugly?
>   (list (package-name package) package output))
> -((? gexp-input?)   ;XXX: misplaced because 'native?' field is 
> ignored?
> +((? local-file? local-file)
> + (list (local-file-name local-file) local-file))
> +((? plain-file? plain-file)
> + (list (plain-file-name plain-file) plain-file))
> +((? computed-file? computed-file)
> + (list (computed-file-name computed-file) computed-file))
> +((? program-file? program-file)
> + (list (program-file-name program-file) program-file))
> +((? scheme-file? scheme-file)
> + (list (scheme-file-name scheme-file) scheme-file))
> +((? string? path)
> + (let* ((regex (string-append %auxiliary-files-subdir-path "/(.*)"))
> +(match (string-match regex input)))
> +   `(,(if match
> +   (match:substring match 1)
> +   "_")
> +  ,input)))
> +((? gexp-input?)  ;XXX: misplaced because 'native?' field is ignored?

I can see why it’s tempting to add more labels, but the medium-term goal
is to remove labels altogether since there are other mechanisms to fill
the same role:

  https://guix.gnu.org/en/blog/2021/the-big-change/

I would keep things as they are and instead encourage you to use gexps
when you need to refer to file-like objects in packages.

We’re still fine-tuning the idioms, so feel free to ask if you don’t see
how to achieve what you want.

HTH!

Ludo’.

Re: Profile definition, was Re: bug#53224: Cookbook recipe about profile collisions

[Ludovic Courtès]

Hi,

Leo Famulari  skribis:

> On Mon, Jan 17, 2022 at 12:56:20PM -0500, Matt wrote:
>> Leo is 100% correct that my understanding is still rather weak. I'll do my 
>> best despite that to help make the documentation better.
>
> I hope you will not feel too bad about that. Remember, everyone begins
> by not knowing anything. Your situation is not unique at all. Rather,
> your energy for improving the documentation for yourself and others is
> exemplary, and will improve Guix in the long run.

I agree, and I apologize for throwing my own guess of what you did and
did not do, Matt.

Ludo’.

Re: Preparing the 1.4.0 release.

[Ludovic Courtès]

Hello!

Mathieu Othacehe  skribis:

> As you may know we have started working actively on the next Guix
> release.
>
> * The release branch is available here:
>   https://git.savannah.gnu.org/cgit/guix.git/log/?h=version-1.4.0
>   
> * It is built by Cuirass here:
>   https://ci.guix.gnu.org/jobset/version-1.4.0
>
> It would be great if we could coordinate and distribute the
> effort. We need support to:
>
> * Fix the bugs in this meta issue:
>   https://issues.guix.gnu.org/53214
>
> * Check the release machinery is still working by following those steps:
>   https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/release.org.
>   
> * Gather the list of changes between 1.3.0 and 1.4.0 in order to
>   complete the NEWS file.
>
> * It is planned to merge the wip-harden-installer branch in the
>   version-1.4.0. Once done, we will need some help to test the
>   installer.
>
> Who's available to help :) ?

I am!

Let’s keep in touch so we can all see how far we are from our
destination and bring more travelers on our ride.  :-)

Ludo’.

Re: version-1.4.0 branch updated

[Ludovic Courtès]

Hi,

Maxim Cournoyer  skribis:

> If nobody has another world rebuilding change deemed necessary in time
> for the release, I suggest we enable substitutes on the branch soon, and
> then get busy trying to get 'make dist' to succeed so that we can issue
> a first RC.

One thing to look at is “make assert-binaries-available”.

Currently, we’re at 83% on ‘master’:

--8<---cut here---start->8---
$ ./pre-inst-env  guix weather -m etc/release-manifest.scm 
--substitute-urls=https://ci.guix.gnu.org
computing 406 package derivations for x86_64-linux...
looking for 519 store items on https://ci.guix.gnu.org...
https://ci.guix.gnu.org
  82.7% substitutes available (429 out of 519)
  at least 3,518.4 MiB of nars (compressed)
  4,433.1 MiB on disk (uncompressed)
  0.483 seconds per request (0.5 seconds in total)
  2.1 requests per second

  0.0% (0 out of 90) of the missing items are queued
  at least 1,000 queued builds
  aarch64-linux: 700 (70.0%)
  powerpc64le-linux: 261 (26.1%)
  x86_64-linux: 38 (3.8%)
  i686-linux: 1 (.1%)
  build rate: 72.54 builds per hour
  i686-linux: 29.81 builds per hour
  x86_64-linux: 37.58 builds per hour
  powerpc64le-linux: 33.64 builds per hour
--8<---cut here---end--->8---

Ludo’.

Re: Preservation of Guix report 2022-01-16

[Ludovic Courtès]

Hi,

Timothy Sample  skribis:

> I’ve published a new preservation of Guix report:
>
> https://ngyro.com/pog-reports/latest/
>
> Actually, the URL is , but I
> thought having a way to reference the latest report would be helpful.

Nice!

[...]

> A really important thing to do at this point is to verify that some
> reasonable looking computation is covered by what we are doing already.
> For instance, is every source used to build Guile (or Python or R)
> preserved?  This will ensure that key sources are not missing, which is
> a real possibility given that everything so far has been purely a
> numbers game!

I wonder if we could have something similar to ‘guix weather -c’, which
would highlight missing sources with many dependents.

TeX Live is a big concern: it’s all Subversion, and everything depends
on those packages.  IIRC, SWH does not support Subversion yet; and when
it does, we’ll have to adjust our code so it can actually fetch
Subversion checkouts from SWH.  One issue is partial checkouts: all
these ‘texlive-’ packages refer to partial checkouts of the big TeX Live
repo.

Thoughts?

Ludo’.

Re: ImageMagick from 2020?

[zimoun]

Hi Ludo,

On Tue, 18 Jan 2022 at 15:39, Ludovic Courtès  wrote:

> It’s now on ci.guix too, though not GC-protected.
>
> Timothy, will your machinery be able to pick it up?

Hum, the issue is the ingestion by SWH, no?  The file 'sources.json'
only contains the last sources of the last revision.  And even, the
file 'sources.json' only contains the upstream URL, not the substitute
ones.


Cheers,
simon

Re: ImageMagick from 2020?

[Ludovic Courtès]

Hi,

Maxime Devos  skribis:

> From repology, I found that CentOS c7 has ImageMagick 6.9.10-68:
>
> https://git.centos.org/rpms/ImageMagick/blob/c7/f/SPECS/ImageMagick.spec
>
> CentOS keeps its copy of the source code at
>
> https://git.centos.org/sources/ImageMagick/c7/
>
> there are two files there, seems like they are named after some kind of
> hash?

It’s now on ci.guix too, though not GC-protected.

Timothy, will your machinery be able to pick it up?

BTW, related to that, could we rsync https://disarchive.ngyro.com to
https://disarchive.guix.gnu.org?  The latter only contains data starting
from ‘master’ revisions ca. November 2021.

Thanks,
Ludo’.

Re: Guix wiki

[Ludovic Courtès]

Hi,

Matt  skribis:

>   On Tue, 11 Jan 2022 18:18:25 -0500 Ricardo Wurmus  
> wrote 
>
>  > This is where we disagree.  I’ve wasted so much time in my life
>  > following outdated or wrong instructions on forums or wikis.  I really
>  > don’t want to see anything half-baked anywhere near this project.  There
>  > are few things that frustrate me more than well-meaning but misleading
>  > advice.
>  
> Do you know that the libreplanet wiki is linked to from the main Guix page?  
> https://guix.gnu.org/ > Help > Wiki

We are aware of this wiki; it has its uses, such as synchronizing for
FOSDEM, GSoC, and other events, but we never intended to use it for
documentation.

It’s unfortunate that your reply to Ricardo dismisses their advice.  It
really boils down to this: read bits of the manual and cookbook that
matter to your use cases, articulate what you think is missing or poorly
structured, and propose changes or new text.

It’s much easier to make progress looking at concrete, focused changes
than in the abstract.

Thanks in advance,
Ludo’.

Re: using an SRFI that is not available in Guile

[Ludovic Courtès]

Hi,

Attila Lendvai  skribis:

> when working on the (gnu services configuration) module, i would like
> to use srfi-189, which is not available in Guile
> (https://srfi.schemers.org/srfi-189/srfi-189.html).

In addition to what Maxime wrote, I’d like to comment on the less
technical side of it.  I think it’s important to keep the code
consistent in terms of API usage.

There are often many different ways to do the same thing in Guile and
Scheme, but within Guix, settling on one (or two) ways for a given
things is preferable: it makes it easier for us all to understand and
maintain the code.

There’s no widespread “Maybe” idiom in Scheme; or rather, people use
TYPE | #f as a way to approximate “Maybe”.  It’s not ideal, primarily
because appropriate handling is not statically checked.  Yet, that’s
what we have and I’m not convinced adding SRFI-189 to the mix would
bring enough of an improvement to justify it.

Thoughts?

Thanks,
Ludo’.

Re: Return back original implementation for text-config serialization

[Ludovic Courtès]

Hi,

Andrew Tropin  skribis:

> During the upstreaming process of Guix Home commit
> fee0bced7fec2f9950957976a28f033edd4f877c slipped into master.  It
> introduces a different serialization approach for text-config from what
> was orginally used:
> https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/services/configuration.scm?h=2c451db39aabc69bdbf186f412ee6e0b4e180ccb#n386
>
> This new approach is inconisistent with the ideas used in the number of
> other home services (not only the ones using text-config):
> https://git.sr.ht/~abcdw/rde/tree/47f6c65c82a4f6761fa1ff5b9405b363cfda6482/item/gnu/home-services
>
> So I had to fork it back to avoid an inconsistency and renamed
> text-config to gexp-text-config to avoid a confusion with upstreamed
> version.
> https://git.sr.ht/~abcdw/rde/tree/47f6c65c82a4f6761fa1ff5b9405b363cfda6482/item/gnu/home-services-utils.scm#L355
>
> Having two serialization approaches stops me from upstreaming the rest
> of home services from rde to Guix and also makes a split to rde home
> services and guix home services, which I would like to avoid.

I’d like to clarify how I think we should work.  Guix development
happens here, on Guix channels, using the project’s peer review
processes.

What rde does certainly is inspirational, and that’s what got us Home in
the first place; in my view, we may sometimes choose to take ideas from
rde in Guix and Guix Home, but rde development alone cannot be used to
justify changes.

> We need to decide, which approach should be used or we will end up with
> two competitive incompatible implementations and unecessary split of
> effort and lose of human force and time.

Making Guix Home part of Guix was and still is a commitment, in
particular the commitment that we’d all be working on one
implementation, that there are no “competitive incompatible
implementations”.  It’s a choice we made, not a phenomenon we’re
passively observing.

[...]

> The new text-config serialization implementation (after fee0bc commit)
> doesn't support gexps and tries to insert the literal content of the
> file in place where file-like object was used, which
>
> 1. Confuses the users.
> People reporting that it's hard to implement something like
>
> source \
> /gnu/store/00fl96dj2aak4i1vqvdqzlhbmmskc7fx-blabla.sh
>
> with the new approach (using file-like objects), and they switch to the
> original implementation of home services for shells (the ones currently
> living in rde repo), which allows to use gexps.

Are there Guix Home issues reporting this?

> 2. Looks strange implementation-wise.
>
> If we want to insert the file path to file-like object for new-style
> text-config internally we do something like
>
> (mixed-text-file ...
>  "source \" "\n"
>  #~(read-everything-from-file
> #$(computed-file "unecessary-file-name"
>#~#$(local-file "blabla.sh"

When would one write such a thing?

A couple of examples from Guix System:  has an
‘include’ field take accepts file-like objects,  has
a ‘config-file’ field.

Guix System users probably never have to use complicated constructs like
the one above.  Why would it be different for Home services?


Are there any new arguments since the already lengthy discussions that
led to fee0bced7fec2f9950957976a28f033edd4f877c?  Is it really what’s
leading to Guix Home being stale, or is there something else?

Thanks,
Ludo’.

Re: Schedule for FOSDEM 2022 available online

[Ludovic Courtès]

Hi Oliver,

Oliver Propst  skribis:

> Just a brief note about that the schedule for FOSDEM 2022 is now
> available online at 
> https://fosdem.org/2022/schedule/track/declarative_and_minimalistic_computing/
> for who is interested in details about the talks.

I had overlooked this message—the program looks exciting, thumbs up!

Ludo’.

Re: Having a package browser on guix.gnu.org

[Ludovic Courtès]

Hi,

Christopher Baines  skribis:

> Ludovic Courtès  writes:

[...]

>> Former Outreachy intern Danjela Lura, together with Chris Baines, had
>> started developing a JS-free package browsing interface:
>>
>>   https://lists.gnu.org/archive/html/guix-devel/2020-07/msg00050.html
>>
>> Chris, could you tell us what the status is and what’s missing before we
>> can use it on the web site?
>
> I think it's unchanged for a while now, I've attached what's probably
> the latest source code (there's not much to it).

Nice.  That’s only the search page though, and not individual package
pages, right?

> I think the remaining work is to settle on a direction in terms of the
> design and how to integrate it in to the website, and then deploy
> something for real. Personally, I'm in favour of a packages.guix.gnu.org
> domain which hosts the search page plus the package pages, and stopping
> generating package pages with haunt.

Sounds like a plan!

> I don't really have the time to try and move this forward myself, but I
> can try and support others.

Yes, that would be nice.  It now takes Haunt ages to generate all these
pages, we have to do something about it.  :-)

So, if anyone feels like picking it up but needs guidance, let us know!

Thanks,
Ludo’.

Re: Celebrating ten years of Guix

[Ludovic Courtès]

Hi Jonathan,

"Jonathan McHugh"  skribis:

> If it helps, feel free to use HSBXL in Belgium as a node.
> => hsbxl.be/
>
> While not a spacious or plush as the Guix Days location, it has ClubMate and 
> better ICT support.
>
> I can be onsite 24/7 whenever you need access.

Thanks for the offer!  I hope we can have an in-person event sometime
this year in Europe (and hopefully others can organize similar events on
other continents).

Ludo’.

Re: Parallel guix builds can trample?

[Ludovic Courtès]

Hi,

Maxime Devos  skribis:

> Phil Beadling schreef op ma 17-01-2022 om 17:23 [+]:
>> For each build that is kicked off in quick succession the local cache
>> of the repo required updated by update-cached-checkout
>>  * https://github.com/guix-
>> mirror/guix/blob/9f526f5dad5f4af69d158c50369e182305147f3b/guix/git.sc
>> m#L476
>>  * https://github.com/guix-
>> mirror/guix/blob/9f526f5dad5f4af69d158c50369e182305147f3b/guix/git.sc
>> m#L279
>
>
> Maybe 'latest-repository-commit' and 'update-cached-checkout' commit
> can be modified to not use 'switch-to-ref', and instead directly ask
> libgit ‘what's the tree structure of commit cabba9e’ and call a
> procedure like 'add-file-tree-to-store'.  That would avoid lock files,
> creating separate directories for concurrent checkouts, ...

That would be ideal.  In Guix ‘write-file-tree’ already supports some
sort of virtual file systems, so it should be possible to use it to feed
the contents of the Git tree to the store.

Ludo’.

Re: 03/06: style: Add support for "newline forms".

[Ludovic Courtès]

Hi!

Tobias Geerinckx-Rice  skribis:

> guix-comm...@gnu.org 写道：
>> +   ('base32 '(sha256 origin))
>
> I know this is your style, but I would *really* prefer this to be:
>
>> +   ('source '(package))
>
> instead.
>
> I know!  But wait!  I think that's defensible: for the same reason
> that we add a newline after arguments, these few extra columns 
> give an (IMO) surprising amount of breathing room down the road.

No strong opinion here, you may well be representing the majority, and
I’m fine with the change you propose.  :-)

There’s a couple of tests that will need to be adjusted but it should be
alright.

(Maybe a few months from now we’ll have additional switches, like
‘-x civodul’, ‘-x nckx’, who knows?  :-))

Ludo’.

Re: Having a package browser on guix.gnu.org

[Ludovic Courtès]

Luis Felipe  skribis:

> I want to help. I'll check the source and start asking questions.

Yay!

Ludo'.

Re: Alternative solution to stat storm problem

[Ludovic Courtès]

Hi,

Tom Scogland  skribis:

> You’re right, the LD_LIBRARY_PATH will not change the loading order,
> but using LD_PRELOAD will by the same mechanism we’re using,
> pre-filling the cache with a library at the same soname.  As part of
> other explorations we’re doing around tweaking or wrapping the loader,
> it may be possible to get semantics like LD_LIBRARY_PATH other ways,
> but at the moment the goal is to make a program that will correctly
> load all the dependencies it would have loaded were it run in the same
> environment it was built in, despite LD_LIBRARY_PATH or RUNPATH in
> dependencies or similar.  Making a little tool that would override the
> same way LD_LIBRARY_PATH would have would be relatively
> straightforward though, would that be worth exploring do you think?

Sure, why not.

My approach was: take the loader and its mechanisms as they exist, and
make the minimal changes needed to adapt it to the
one-directory-per-package layout, when they currently assume FHS.

The approach you describe is more about keeping the loader unchanged and
working around its FHS assumptions “from the outside”.  In that spirit,
a separate mechanism for LD_LIBRARY_PATH-kind of user overrides might
make sense.

Thanks,
Ludo’.

Re: Alternative solution to stat storm problem

[Ludovic Courtès]

Hi Farid,

Farid Zakaria  skribis:

[...]

>> I guess the advantage is that you don’t need the ld.so patch.  The
>> downside is that PatchELF needs to be able to write longer NEEDED
>> strings in the dynamic section, which it may not always be successful at
>> (I think?).
>
> I can't claim to be a ELF specification guru but I have not
> encountered that longer NEEDED strings to be a cause for failure.
> The emacs example is a pretty good test case because the transitive
> closure of all NEEDED libraries is quite large, which all seem to be
> added successfully to the ELF header.

Well, we’d need a closer look, but I think PatchELF may need to enlarge
the relevant string table, and that may not always be possible.

> The benefit to me seems:
> 1 - does not need a glibc patch for functionality (although for other
> libc such as musl it might in this case
> https://www.openwall.com/lists/musl/2021/12/21/1)
> 2 - understanding the dependencies of an application become simpler
> 3 - there are esoteric cases where in fact libraries might link to the
> wrong libraries (although they are correct at build time) given a
> RUNPATH/RPATH since there are subtleties with the inheritance model.
>
> I'm actually researching ways to improve (3) as well through
> mentorship with Tom Scogland by researching alternative ways to do
> linking:
> - RUNPATH per NEEDED
> - the ability to specify whether a RUNPATH should be inherited or not
> to downstream dependencies

OK.

>> Also, I wonder if the absolute file names in NEEDED interfere with uses
>> of $LD_LIBRARY_PATH (making it impossible to force use of another
>> libxyz.so than the one that would be found in RUNPATH.)
>
> Correct. For a system with reproducibility in mind this can perhaps be
> a desired feature.
> It is the current limitation of the proposal.

I think it’s still useful to allow users to bypass normal mechanisms, be
it via LD_LIBRARY_PATH or LD_PRELOAD.

> In fact, Carlos brought up a great philosophical question:
> "Is linking to libraries through a content-addressable value allowed
> for LGPL software?"
> What if the linked address also forced the content-address by having
> it resolve to something on IPFS ?

Oh you mean it could be thought of a static linking, conceptually?
Good question.

Ludo’.

Re: Release v1.4 (or 2.0): process and schedule ?

[Ludovic Courtès]

Hello!

For the record, I created a few days ago an issue to keep track of
progress towards the release by blocking it with issues that we think
must be fixed before we release:

  https://issues.guix.gnu.org/53214

Click on “Details” to see the blocking issues.

Hopefully it’ll allow everyone of us to better understand what remains
to be done and to add (and remove!) blockers.

Ludo’.

Re: Parallel guix builds can trample?

[Phil Beadling]

Hi all,

The issue also exists when using --with-commit - see below for a refined
test that makes it trivial to demonstrate with any package where the source
is retrieved from git.

On Tue, 18 Jan 2022 at 10:10, Phil  wrote:

>
> > Philip Beadling  writes:
>


> >No, and we probably should do, even in spite of this issue.  We're going
> >to repeat the experiment now using --with-commit, to confirm behaviour
> there.
>
>
Let's try and build 2 package variations - for this experiement we don't
even have to run them at the same time:

mkdir checkout-test
cd checkout-test
mkdir cache1 cache2
XDG_CACHE_HOME=`realpath ./cache1` guix build
--with-commit=simm=*3981f60d81d3c9592d78b3e88545259713a93194
*simm
XDG_CACHE_HOME=`realpath ./cache2` guix build
--with-commit=simm=*ec1357c1fdf78a2ac1408d00c0c3e7d025ab9ade
*simm

Now look at the caches:

$ ll cache1/guix/checkouts

total 4.0K
-rw-rw-r-- 1 phil.beadling phil.beadling  10 Jan 18 12:37
last-expiry-cleanup
drwxrwxr-x 6 phil.beadling phil.beadling 162 Jan 18 12:37
lx4ipn27sk6uvu6b33kelt4tik65oianvl7b7yn5bpng7gpt4pyq
drwxrwxr-x 5 phil.beadling phil.beadling 115 Jan 18 12:37
mqqlbc4ksftu2tkbnxydosm4j6mynxsh5nmhpq3cfrofkktkhexq
drwxrwxr-x 5 phil.beadling phil.beadling 103 Jan 18 12:37
qqzuuoqxvqikivkpenqom43kzk2xr7nlntanbm6fpe7bci6kaenq
drwxrwxr-x 4 phil.beadling phil.beadling 202 Jan 18 12:37
smvbhjf46cbaxxju6fsqj7t6ujktxy5i7vhuhzpvgjt2vhid2vqq
drwxrwxr-x 5 phil.beadling phil.beadling 267 Jan 18 12:37
*xulvhb6agjrsf6x7wknutpn7wkt5i4nesk637czazvfxps357eoq*

$ ll cache2/guix/checkouts

total 4.0K
-rw-rw-r-- 1 phil.beadling phil.beadling  10 Jan 18 12:38
last-expiry-cleanup
drwxrwxr-x 6 phil.beadling phil.beadling 162 Jan 18 12:38
lx4ipn27sk6uvu6b33kelt4tik65oianvl7b7yn5bpng7gpt4pyq
drwxrwxr-x 5 phil.beadling phil.beadling 115 Jan 18 12:38
mqqlbc4ksftu2tkbnxydosm4j6mynxsh5nmhpq3cfrofkktkhexq
drwxrwxr-x 5 phil.beadling phil.beadling 103 Jan 18 12:39
qqzuuoqxvqikivkpenqom43kzk2xr7nlntanbm6fpe7bci6kaenq
drwxrwxr-x 4 phil.beadling phil.beadling 202 Jan 18 12:38
smvbhjf46cbaxxju6fsqj7t6ujktxy5i7vhuhzpvgjt2vhid2vqq
drwxrwxr-x 5 phil.beadling phil.beadling 267 Jan 18 12:38
*xulvhb6agjrsf6x7wknutpn7wkt5i4nesk637czazvfxps357eoq*

$

Despite asking for different variants of the simm package, they are both
stored under the same SHA.  If they both used the same cache directory (the
default) - they'd have overwritten each other.  Thus it is not safe to run
--with-commit or any other variation in parallel for the same user.

To prove I've matched the right SHA - I can back-out the original commit
ids provided to --with-commit:

$ cd
cache1/guix/checkouts/xulvhb6agjrsf6x7wknutpn7wkt5i4nesk637czazvfxps357eoq
&& git rev-parse HEAD && cd -

*3981f60d81d3c9592d78b3e88545259713a93194*
~/checkout-test

$ cd
cache2/guix/checkouts/xulvhb6agjrsf6x7wknutpn7wkt5i4nesk637czazvfxps357eoq
&& git rev-parse HEAD && cd -

*ec1357c1fdf78a2ac1408d00c0c3e7d025ab9ade*
~/checkout-test

$

Whilst my original setup was a bit of a edge-case, I think this refined
example demonstrates that the problem can occur with typical/reasonable use
of guix switches?  I can even think of how this could occur without the
user running in parallel - for example if a package required via nth-order
dependencies to build 2 versions of the same repo - it's possible that a
single "guix build" command could cause the issue?

Thoughts?

Re: Parallel guix builds can trample?

[Phil]

Ricardo Wurmus writes:

> Philip Beadling  writes:
>
>> The source in each one will be slightly different and although I only
>> specify commit id in the package
>
> How are you using Guix with this?  Do you generate Guix package
> expressions?  Do you use “guix build --with-commit”?

No, and we probably should do, even in spite of this issue.  We're going
to repeat the experiment now using --with-commit, to confirm behaviour there.

Thanks for reminding me about the switch!

Re: Parallel guix builds can trample?

[Maxime Devos]

Phil schreef op di 18-01-2022 om 09:28 [+]:
> To mitigate performance concerns, I think it's possible to set the
> branches and depth in git such that we only clone the bare-bones only?.

IIUC libgit2 and the Guile bindings guile-git do not currently support
shallow clones (depth, etc.).  Bare-bones should be possible though.

Greetings,
Maxime.


signature.asc
Description: This is a digitally signed message part

Re: Parallel guix builds can trample?

[Phil]

Maxime Devos writes:

> Maybe 'latest-repository-commit' and 'update-cached-checkout' commit
> can be modified to not use 'switch-to-ref', and instead directly ask
> libgit ‘what's the tree structure of commit cabba9e’ and call a
> procedure like 'add-file-tree-to-store'.  That would avoid lock files,
> creating separate directories for concurrent checkouts, ...

Thanks Maxime - I'll take a look at this as an option.

One option I was thinking of is to make this SHA call dependent on the
branch as well as the URL:
https://github.com/guix-mirror/guix/blob/9f526f5dad5f4af69d158c50369e182305147f3b/guix/git.scm#L131-L133

Technically this still leaves a race condition if someone were to do
parallel builds on the same branch however, so to completely solve the
problem I think we'd have make the SHA call on repo+commit-id.

This obviously limit the utility of the cache.

To mitigate performance concerns, I think it's possible to set the
branches and depth in git such that we only clone the bare-bones only?.