RE: How can we decrease the cognitive overhead for contributors?

[phil]

>I really like Guix, I like what it promises, I love the community around
it, and that's what keeps me here. But it's a >deeply frustrating experience
to try to contribute to. I've been a contributor in various forms to a great
many free >and open source software projects over the years, and Guix is
easily one of the worst in that regard.

This aligns with somewhat with my (limited) experience.  I'm a massive fan
of Guix, but I did find the process a bit opaque; even after seeking
clarification from the mailists, and some of my patches eventually fell
through the cracks, or in other cases, someone would kindly submit on my
behalf.  That's really nice of them to do, but it's obviously not the way
things should work.

I've contributed to other Guix channels (some run by people closely
affiliated with Guix proper) and other open source projects and found the
overhead is usually no more than few minutes to read/follow to the how-to
guide.  I don't have a favourite forge nor a specific process I want to
promote - but it should be easy and require little effort to understand and
setup.  Really the typical use-case should be close to push-button - with
the focus being on reviewing the contribution rather than the mechanics of
how this is done.

I (and other colleagues) have a bunch of core enhancements, and quite a few
packages sat in my (now previous) company's Guix channel that would be great
to get into Guix proper (if the wider community deem them appropriate/useful
of course!), but the overhead means we kept avoiding getting this done and
the company runs off a lagged fork of Guix they manually update and rebase
themselves, rather than pushing their work upstream into Guix.

I 100% accept failing to get patches into Guix proper is also lethargy on my
behalf, and perhaps I'm guilty of carping rather than proactively helping,
but in choosing what to contribute to, and when spare-time is sadly limited,
the amount of overhead and chasing required both become inevitable
considerations on where to focus time.

RE: London Guix meetup

[phil]

Excellent - I'll pass this on to various Guix users I know in London!

-Original Message-
From: guix-devel-bounces+phil=beadling.co...@gnu.org 
 On Behalf Of Arun Isaac
Sent: Tuesday, June 13, 2023 11:27 PM
To: guix-devel@gnu.org; guile-u...@gnu.org
Cc: Fabio Natali ; Christopher Baines 
Subject: London Guix meetup


Hi all,

We are organizing a casual Guix meetup in London. There is no specific agenda. 
The intention is for Guix users and would-be Guix users in London to get to 
know each other, and to put faces to names. No prior experience with Guix (or 
Guile) is necessary. Newcomers are very welcome!

Date: Tuesday, June 27 6–9 pm BST
Location: Barbican Centre, Silk Street, EC2Y 8DS, London Mobilizon page:
https://mobilizon.fr/events/f18da336-07f5-489f-ae7c-394458668cea

Looking forward to meeting many of you for a pleasant summer's evening by the 
fountains at the Barbican Centre! We will provide more exact instructions on 
how to find us at the venue as we get closer to the date of the event.

The date and time are almost confirmed, but we can nudge it a little if you 
reply quickly (say, within a couple of days).

Regards,
Arun

Re: RFC: libgit2 is slow/inefficient; switch to git command?

[Phil]

Hi,

Wojtek Kosior via Development of GNU Guix and the GNU System distribution. 
writes:

> Hi,
>
> I just want to add my 2 cents :)

Just to add mine too - libgit2 behaves differently to command-line git
in ways which can make guix do unexpected things when caching clones in
certain cases.  This has resulted in some hard to diagnose issues with
using guix to build PRs for example.

In particular we were forced to make this change to our local guix build
to ensure that guix behaved inline with git:
https://github.com/guix-mirror/guix/commit/473954dd92bbb84693b6fa3f007752eb53c804db

An explanation of why, was raised with libgit2:
https://github.com/libgit2/libgit2/issues/6183

The original guix-devel discussion here:
https://lists.gnu.org/archive/html/guix-devel/2022-03/msg00021.html

This particular issue is somewhat niche - but it demonstrates well the
danger of assuming the libgit2 and git behave in the same way!

This makes me a bit wary of using libgit2 now.

Cheers,
Phil.

Re: Compile skribilo doc containing guix channel references

[Phil]

Hi Simon,

zimoun writes:

> I am missing how the Skribilo file is it compiled?  At “guix pull” time?
> Or manually when running the script compile-docs.scm?
>

At the moment compilation is only manual and from inside the repo clone:
guix environment skribilo guile -- guix repl -- compile-docs.scm

Making it part of guix pull is an interesting idea given this is where
standard package compilation takes place.  For this to work any code
references would have to be of the form:
(use-modules (my-module foo)
(package->code my-package)

rather than direct access to scheme files in the repo clone using:
(source :file "foo.scm" :definiton 'my-package)

The advantage of using pacakge->code is that the documentation
compilation is then decoupled from the repo clone and can be generated
anywhere guix can be run (just like guix pull).

The 2 disadvantages are 1) Files in the repo, but not part of the guix
module system cannot be referenced, 2) The pretty-printing of packages
generated using package->code does not perfectly reproduce the format in
the repo clone - although they are functionally identical.

For example inputs are referenced fully qualified in-situ - eg
(@ (foo bar) baz), rather than just referencing baz in the inputs, and
importing (foo bar) at the top of the module.  Comments etc, are also
lost too.

If there was a nice way of referencing the uncompiled scheme files in
the channel without needing the repo cloned that would be perfect IMHO.
I don't think this is possible tho?

Re: Compile skribilo doc containing guix channel references

[Phil]

Hi,

Phil writes:

> Hi,
>
> Ludovic Courtès writes:
>
>> We toyed with Skribilo/Guix integration in this paper:
>>
>>  https://hal.inria.fr/hal-01161771/en
>>  
>> https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/reppar-2015/reproducible-hpc.skb
>>
>> (Note that it starts with ugly ‘%load-path’ hacks.  :-))

At the risk of cross-posting from skribilo list, whilst our internal
documentation has to remain private, we also have a public fork of guix
which would benefit from some documentation explaining the changes we've made
to guix core code that we hope one day to push upstream - so I've taken
the same technique and applied it here - so people can see a working example:

This generates the docs with guix imports - see compile-command in the header:
https://github.com/quantiletechnologies/qt-guix/blob/feature/EA-133/compile-docs.scm

This is the document - it's less ambitious than my internal version in
terms of generating content from guix - but has a few examples of
generating content from channels:
https://github.com/quantiletechnologies/qt-guix/blob/feature/EA-133/qtdocs/qt-guix.skb

Re: Compile skribilo doc containing guix channel references

[Phil]

Hi,

Ludovic Courtès writes:

> We toyed with Skribilo/Guix integration in this paper:
>
>  https://hal.inria.fr/hal-01161771/en
>  
> https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/reppar-2015/reproducible-hpc.skb
>
> (Note that it starts with ugly ‘%load-path’ hacks.  :-))

Thanks for the reply and the links, good to see another example.

> I would suggest using your own script, executed with ‘guix repl’,
> instead of the ‘skribilo’ executable.  Something like you describe in
> your second message looks good!

I've got the repl approach working without the load path hacks now -
creating an --ad-hoc environment containing both guile and skribilo was
all that was required in the end.  If I can make the approach completely
generic I'll post the code somewhere with an example of use.

What's cool is it is then possible to reference packages without even
having the skribilo document co-located in the same repo as the
packages.  For example if I have a package in guix called "my-package":

(define my-package-str
  (with-output-to-string
(lambda () (pretty-print (package->code my-package)

You can then reference this in skribilo like this:

(prog (source :language scheme my-pacakge-str))

This works great, but I have come-up against one issue I have a proposed
fix for.

package->code uses a function called source->code, to output the
scheme representation of the source.  However source->code assumes that
the source can only be an "origin" record, but it's also valid that
source is a "git-checkout" record.  In particular, if the source repo URL
is an ssh address rather than an https address using git-checkout is the
correct approach - for example:

(source
  (git-checkout
(url "ssh://git@myserver/foo/bar.git")
(commit "abcdef12345")))

We make extensive use of this, combined with an ssh-agent, to automate
authentication on private repositories.

Supporting git-checkout in source->code is easy enough - I've written a
fix in our guix channel and it works well, happy to contribute this to
guix-patches too if it makes sense?

The proposed fix can been see here:
https://github.com/quantiletechnologies/qt-guix/commit/49900405617fdf6a2e4ab6f4004306a2770221e0

Finally, I suspect Guix could also be interestingly combined with Haunt
to produce and serve HTML generated from Guix content too:
https://dthompson.us/manuals/haunt/index.html

Cheers,
Phil.

Re: Compile skribilo doc containing guix channel references

[Phil]

Hi,

Phil writes:

> My guess is there is a better approach - any suggestions?  Perhaps I can
> call the skribilo compiler from inside "guix repl" for example?

To somewhat answer my own question in case it's useful to anyone else or
if anyone has any suggestion on how to do it better - there are 2 ways
I've found to do this.  The most obvious is to abandon the skribilo DSL
and write the document directly as a scheme file which will evaluate and
write itself out as HTML - it can then be called as: guix repl -- foo.scm:

(with-output-to-file "foo.html"
  (lambda ()
(evaluate-document d e

See skribilo docs for full example.

This isn't ideal because you can't use a the skribe format which isn't
pure scheme.  To do that we have to call the scribilo compiler's main entry
point from inside scheme rather than from the command line - the below
example worked for me, but it's quite fiddly to get it to work.  It is
also called from guix repl.

To give a very simple example the document can use guix functions to
reference infomation in the channel I'm documenting:

(p [The ,(package-name my-package) package is set to ,(package-version 
my-package).])

The below script has to be one level above the package directory in my
channel, and the script must be called the directory which houses it.

The output should also go to the same directory.  I then moved the
output HTML afterwards, rather than putting the target path in --output
- this makes sure the links work.

The hardcoding of the load-path is also not ideal, I suspect this can be
improved.

(add-to-load-path 
"/gnu/store/ymvqgkljzykfv4g7vbw36pgghfjbhnps-skribilo-0.9.5/share/guile/site/3.0")
(add-to-load-path 
"/gnu/store/2160nv245ys30vxvhpjxgcbjm9rq67gw-guile-reader-0.6.3/share/guile/site/3.0")

(define local-repo-clone-path
  (dirname (current-filename)))
 
(define local-packages-clone-path
  (string-append local-repo-clone-path "/packages"))

(add-to-load-path local-packages-clone-path)

(use-modules (skribilo condition))

(call-with-skribilo-error-catch/exit
  (λ ()
(apply (module-ref (resolve-interface '(skribilo)) 'skribilo)
   `("--target=html"
 ,(string-append "--source-path=" local-packages-clone-path)
 ,(string-append local-repo-clone-path 
"/docs/my-channel-skribilo-doc.skb")
 "--output=index.html"

Compile skribilo doc containing guix channel references

[Phil]

Hi all,

I'd like to document a private guix channel using Guile-based document
framework: Skribilo - https://www.nongnu.org/skribilo/.

The idea is that the guix channel repo will contain a skb file which can
then reference code in the channel by setting the source-path and then
using the :definition in the source function to reference bits of the
package I want to talk about in my document.

(source :language [:tab 8] [:definition] [:stop] [:start] [:file])

This works really well, but it got me thinking - what would be even
cooler would be if I could import modules in my channel and actually
execute channel functions to generate document output as part of the
document compilation process, rather than just display the package code!

The problem with this is that in order to import modules in my channel,
I'd also have to have the guix core modules on my load-path at skribilo
compilation time, otherwise skribilo complains when my channel makes use
of guix modules.

For example, if I just add the channel packages without guix core
modules at the top of my skribilo document:
(add-to-load-path (string-append (dirname (current-filename)) "/packages"))

I understandably get something like the below:
no code for module (guix git)

Normally when scripting for guix you'd avoid the above problem by using
'guix repl -- some-script.scm' but here I'm using the skribilo
executable, *not* guix as the entry point - it's not entirely clear to me
how to get the required guix modules onto my load path?

As an ill-advised hack I tried the following under guix repl:
(use-modules (gnu packages))
(%package-module-path)

Then I added the returned paths via add-to-load-path at the top of my skb
file.  This resulted in:
$ GUILE_AUTO_COMPILE=0 guix environment --ad-hoc skribilo -- skribilo
--target=html --source-path=.. guix-packages.skb -o index.html
ice-9/boot-9.scm:1685:16: In procedure raise-exception:
/gnu/store/8bab6fga7858whgd11f0wlsi2zsp4yr2-guix-module-union/share/guile/site/3.0/guix/packages.scm:361:5:
Unknown #object: "#~"

My guess is there is a better approach - any suggestions?  Perhaps I can
call the skribilo compiler from inside "guix repl" for example?

Cheers,
Phil.

Re: Types and builds for mypy

[Phil]

Thanks for your reply Maxim,

Maxim Cournoyer writes:


> Is MyPy the only consumer of Python type annotations?  I think so, but
> I'm not sure.  If it's the only one, it'd make sense to move mypy and
> all the annotation types to (gnu packages python-types), I think.

Mypy is certainly a prominent consumer, but you're right - PEP 561 type stub
packages are also used by PyCharm, pytype, and I've heard CPython can
use them to compile, to name a few.

So I think my revised opinion is mypy should stay where it is, but I do
like the idea of keeping all the types in one module?

Co-locating them with the module they provide type for also works fine,
but I kinda see the type files as visual noise if we put them in
packages which are organized into categories they don't directly
contribute to. Perhaps they are better kept in once place?

It also means that when running mypy you can just import your
python-types module, and be sure it's going to work.


> I think if it doesn't negatively impact its use, we should go ahead and
> make this change in the mypy package.
>
> Happy to review patches!

Cool thanks - leave this with me, I have a working package so can submit
a patch shortly!

Re: Pinning package inputs using inferiors?

[Phil]

Thanks Simon - I've given an example below.

zimoun writes:

> For an example, see python-numpy and python-numpy-next in (gnu packages
> python-xyz).

This was my original way of handling this but in what is perhaps a niche
use of Guix by my department - it ultimately doesn't scale well, for our
use-case.

Originally the department was small enough that there was only a handful
of applications sharing one or two common in-house libraries.

As we've scaled-up we now have the situation where 3 or 4 common
libraries are being used by say 10 applications.

We have rapid release schedules - and want to be able to release the
common libraries on a weekly basis.  But the time to sign-off on a
common library takes a few days per application, so it's not practical for
every project to bump version every week - they have other priorities.

In an ideal world automated unit and regression testing would be
comprehensive enough that we could move aggressively each week, but at
least for now that's not practical given the complex nature of signing
off the libraries and the applications which use the libraries.

So, ideally, what we'd like to do is just have each common library
churn-out releases every week, and have the releases available in Guix,
but without having an obligation on dependent applications to adopt
these changes if they don't want to.

Note all libraries and applications share the same channel - one
solution would be to have each library in their own channel, but this
feels ugly to me.

Our solution (somewhat tricky to explain without a whiteboard -
apologies!) is to co-locate the package definition of the common library
in the common library repo itself - we call it something like
.requirements.scm and it is naturally kept in lockstep with the code in
that repo that the definition will build.  This is very different to
traditional Guix where channels contain definitions separately in a
different repo to the code those definitions describe how to build. 

We then have a job in our CI/CD system that allows us to give a tag on the
common library repo, and the name of an application that uses the common
library.

The job will copy the .requirements.scm into our channel inside a
private module specific to the application that uses the common library.

The idea is that you can have many versions of .requirements.scm private
to every application package definition that references it.

You could even read .requirements.scm using a function that clones the
application repo on-the-fly rather than statically storing it in the
channel - we haven't gone this far yet, as it makes the system even more
complex to reason about.

This is basically the same idea as the python-numpy-next but allows for
many versions of python-numpy to co-exist by keeping them all in private
modules so they don't clash.

It's a cool idea and works pretty well, but requires us to augment Guix
with a set of extra tools to lift and shift these private definitions
around, which complicates our setup considerably.

It feels like wanting to make many versions of a library available at
once isn't an unreasonable way to work at-scale.  However, it also feels
like a departure from the philosophy of Guix to decentralise package
definitions and to allow for a potentially large range of versions to
co-exist in the same channel commit.

We could try to further integrate the idea into guix by writing new guix
commands to support it - we're still working out the details ourselves,
but if it works well we'd love to present it at a future Guix Days or
similar!

In the meantime I was wondering if anyone else had a similar use-case
for Guix and if they had tried something similar or different to handle
many versions in an automated way in the same channel commit?

Apologies that's more than I was intending to write - but hopefully that
makes some sense!  If it doesn't I can try to flesh out specific example?

Pinning package inputs using inferiors?

[Phil]

Hi all,

A change in a package ("dependency" in the below example) in a channel I
own has caused a conflict in another package in the same channel that depends
on it ("test-package" in the below).  Whilst fixing the "test-package"
package is the right solution, this is too complicated in do in the
short-term.  I need to pin "dependency" to v1.0 in test-pacakge's
propagated-inputs. Simultaneously, other packages need the new update to
the "dependency" package to use this functionality to deliver new
functionality that can't wait. 

This isn't a one-off situation; this happens frequently and I'm
interested in how other Guixers resolve this with as little friction to
users as possible?

One brainwave I had was to use inferiors - but this doesn't seem to
work.  Continuing from the above example we could define access to a
historical v1.0 of the dependency package for the test-package like so:

(define dependency-inferior
  ;; An inferior containing dependency v1.0.
  (inferior-for-channels dependency-channels))

If we do this then I can get the below manifest to work, just like the
example in the manual:

(packages->manifest
 (list (specification->package "python") ;; useful for testing
   ;; Remove current dependency add old dependency
   (package/inherit test-package
(propagated-inputs
 (modify-inputs (package-propagated-inputs test-package)
(delete "dependency"
   (car (lookup-inferior-packages dependency-inferior "dependency"
 "1.0"

But this isn't a practical approach - knocking "dependency" out of
test-package's inputs means the unit tests would fail, so I would also
have to delete the check phase - I don't want to do this, it's too much
compromise.

Instead I was hoping to replace the dependency *inside* the package
definition with an inferior like this, so it's still available whilst
the inherited package is being built.

(packages->manifest
 (list (specification->package "python") ;; useful for testing
   ;; Remove current dependency, add old dependency
   (package/inherit test-package
(propagated-inputs
 (modify-inputs (package-propagated-inputs test-package)
(replace "dependency" (car 
(lookup-inferior-packages dependency-inferior "dependency" "1.0"

When I try this, depending what operation I perform on the manifest, I
normally get some type mismatch telling me I've used a package-inferior
when Guile was expecting a package.  Nothing works, alas.

Thus I'm assuming the two types are not completely substitutable, and
this won't work?

Given this, the workaround I am employing is to replace a single package
definition of "dependency" with locally scoped definitions for each
package that uses it.  This is duplication feels suboptimal.

FWIW, a much more involved solution is to store the dependency package
inside test-package's project repo (rather than my channel), and then
automate copying this into the channel at build time.  This is a cool
exercise, but means we are decentralizing the channel's package
definitions and co-locating them across many repos - which feels not in
the spirit of Guix, and more like how requirements.txt files are
co-located in (non-Guix) python projects. 

My questions are:

1. Is my above use of inferiors always doomed, or something we code make
work with changes to Guix core?

2. Is there another way of handling the situation elegantly without
using inferiors or duplicating package definitions at module scope. 


Any advice or comments gladly received!

Cheers,
Phil.

Re: Who owns guix-mirror?

[Phil]

Hi Tobias,

Tobias Geerinckx-Rice writes:

> It's Marius Bakke (mbakke) and me (nckx).  It probably wouldn't hurt
> to make that list public.  Marius, what say you?

That would be great if you guys are happy making it public.

>
> So trusted, and not even third, parties.  You're also trusting GitHub,
> and the mirror *is* completely unofficial.  There's no guarantee that
> the synchronisation won't break just when an important security update
> is pushed to Savannah.

All understood - thanks for confirming!

Re: Windows Subsystem for Linux

[Phil]

Maxim Cournoyer writes:

> cookbook.  I have fellow coworkers (not at Savoir-faire Linux, mind you
> :-)) who have yet to escape their Windows jail; giving them a taste of
> what is possible out there with Guix would probably be a real eye
> opener.  At the minimum it would present Guix as a technically viable
> alternative to Docker & friends in these circles.
>

I totally agree with this - at the moment setting on WSL2 (AFAIK) you
have 2 options:

1. Run it on-top of Ubuntu and tweak the various daemons to start on
image boot.

2. Use Busybox to bootstrap a bare-metal install of Guix without a host
Linux.


The main problem I've had with 1 is that the resulting image is produced
is very large and does not reliably install on colleague's machines.
That and Ubuntu of course is just a distraction when trying to showcase
a Guix workflow.

The problem with 2 is that last time I tried (about a year ago), this
requires you to construct your Guix image for scratch - which is an
interesting exercise but will put off the majority of users:
https://gist.github.com/giuliano108/49ec5bd0a9339db98535bc793ceb5ab4

What we need is a WSL2 image like other OSs provided which can just be
import directly into WSL2 in a push-button fashion.  I don't think
this would be particularly difficult to do, and would encourage more
people to try Guix.

It would also be useful for people who already use Guix, but have no
option but to use Windows in some circumstances - WSL2 is already very
popular to escape Windows into Linux in day-to-day workflows, when you
are unable to have a Linux desktop, and has become widely adopted even in
companies that have a Windows-only desktop policy.

Having Guix images to download and install for WSL2 would make it very
easy to showcase Guix without asking people to risk a physical
install (which can be tricky to get hardware working), or even to adopt the
full Graphical Desktop. Users don't need admin rights to install a WSL2
image, and assuming WSL2 is already enabled (for Ubuntu or whatever) it
is trivial to add any other image.  It would be nice complement to the QCOW 
images
already available for QEMU.

Getting ahead of myself :-) I also think providing cloud images in
formats like Amazon's AMI would encourage people to spin-up Guix and
give it whirl.  And even looking at Ubuntu's "multipass" to easily bring
Ubuntu to Windows/Mac/Linux is another way to make it easy to try Ubuntu
on any other OS with a lightweight install.

Re: Help to workaround libgit2 fetch refs issue

[Phil]

Thanks Liliana for the reply,

Liliana Marie Prikler writes:

> Am Freitag, dem 04.03.2022 um 09:34 + schrieb Phil:
> Again I wonder what the introduction of a new record type solves here.
> Wouldn't it be easier to just adapt git-fetch to handle that edge case
> and use normal git-references?  Origin methods have network access, so
> you can pretty much do whatever.
>
> Cheers

The problem with an origin record using git-fetch is that it doesn't
seem to work for key-authenticated repos using SSH.

If I try to change the git-checkout for the standard git-fetch
incantation:

Original:

(git-checkout
(url "ssh://git@bitbucket:7999/foo/bar.git")
(commit commit-production)))

Proposed:

  (source
   (origin
 (method git-fetch)
 (uri (git-reference
   (url "ssh://git@bitbucket:7999/foo/bar.git")
   (commit commit-production)))
 (sha256
  (base32 "0s9dpj0jdkqcg552f00jhd722czji4pffabmpys5pgi6djckq4f4"


This fails because git-fetch is unable to use ssh because it seems to
run inside a container?

Initialized empty Git repository in 
/gnu/store/dbcl57jcrvaavjrj8qwpwskl7sfpzqb4-git-checkout/.git/
error: cannot run ssh: No such file or directory
fatal: unable to fork
Failed to do a shallow fetch; retrying a full fetch...
error: cannot run ssh: No such file or directory
fatal: unable to fork

Even if it did find ssh, it would not have access from the daemon to the SSH 
Agent
environment variables setup by the user to allow access to the SSH Key.

This is covered here I think, and a patch was proposed, but it was
dropped in favour for the original solution above in the original thread:
http://issues.guix.gnu.org/issue/31285

I did think about perhaps resurrecting the proposed "git-fetch/impure"
as shown in the patch of guix-download.scm in the link immediately
above, but I was hoping for a quicker win with duplicating the record -
alas that has it's own significant limitations tho.

If I'm missing another trick with git-fetch I'd glady be proved wrong!
:-)

Cheers

Re: Help to workaround libgit2 fetch refs issue

[Phil]

Just to add

Duplicating of guix-checkout in a private channel has side-effects - if
anyone has any ideas of how to workaround this, I'd be love to
discuss... alas, at this point I think I have 3 temporary options:

1) Patch in my local Guix and roll my own build of Guix (see below)
2) Workaround outside of Guix by cloning the repo for Guix not using libgit2
3) Tell my CI/CD to continue to build all my PRs twice if the first
build fails.

Details below.

Phil writes:

> Thanks for the reply Liliana,
>
> On Wed, 2 Mar 2022 at 21:06, Liliana Marie Prikler 
>  wrote:
>
> I've included what I think is a true minimal implementation, borrowing from 
> (guix git) where possible to minimize duplication - in
> case anyone else ever stumbles into the same issue (although it's pretty 
> niche, I admit!).

Whilst my original post works it restricts the use of --with-branch and
--with-commit options with packages that use my new record type.

It's now obvious why, looking at package-git-url it only handles origin
and git-checkout records.  I need to add my git-checkout-x-ref here too:

(define (package-git-url package)
  "Return the URL of the Git repository for package, or raise an error if
the source of PACKAGE is not fetched from a Git repository."
  (let ((source (package-source package)))
(cond ((and (origin? source)
(git-reference? (origin-uri source)))
   (git-reference-url (origin-uri source)))
  ((git-checkout? source)
   (git-checkout-url source))
  (else
   (raise
(formatted-message (G_ "the source of ~a is not a Git reference")
   (package-full-name package)))

Re: Help to workaround libgit2 fetch refs issue

[Phil]

Thanks for the reply Liliana,

On Wed, 2 Mar 2022 at 21:06, Liliana Marie Prikler <
liliana.prik...@gmail.com> wrote:

>
> >> In Guix this means that the first time I build a PR it fails, and I
> >> have to do something like "guix build  foo | guix build foo" which is
> >> at best a clumsy hack, but it works!
> >Note that you could alternatively just use the requesting repo URL in
> >the git-reference.  Ah, but that would be too boring, wouldn't it?
>

Ha ha - indeed far too boring :-) If you're suggesting I use an HTTP(S)/URL
to access the repository, then annoingly in my specific case the repo is
password protected - this is why I turned to SSH in the first place so I
could use keys to authenticate.  Or I'm missing the point?



> >> I was wondering if anyway could confirm this and/or give me a pointer
> >> of where the compliation occurs (where the record in the source is
> >> handled and compiled into a git clone) - even if my approach is
> >>(quite possibly) unviable, I'd like to understand why!
> >The source field of a package is not thunked, so it is compiled in the
> >package definition itself, rather than handed off.  Note that every
> >hack of doing something fancyful usually invokes at least a delay form,
> >see e.g. the computed-origin-method used by linux-libre.  Hope that
> >helps.
>
>
Yes - thanks for the hints, I got there in the end.  Just to confirm that
although perhaps not the most elegant workaround, I eventually got the idea
to work.

My original effort lacked the exports for the below, and a few other
missing items private to (guix git).
update-cached-checkout-x-ref
latest-repository-commit-x-ref

Having dug a bit deeper into how the hash table for %gexp-compilers is
setup as a mapping of to record type descriptior to gexp-compiler, I've got
a better understanding now of how package compilation works so it was a
worthy if experiment :-)

I've included what I think is a true minimal implementation, borrowing from
(guix git) where possible to minimize duplication - in case anyone else
ever stumbles into the same issue (although it's pretty niche, I admit!).

Now to go and fix it properly in libgit2


;; -*- mode: scheme; eval: (guix-devel-mode 1);
geiser-scheme-implementation: guile -*-


;; This is an alternative to guix-checkout record type, which always does a
fetch
;; even after the initial clone.
;; This is required because of a bug in libgit2 which means that fresh
clones
;; don't pull down any additional refs from the git config:
;; https://github.com/libgit2/libgit2/issues/6183

;; Once this is fixed and Guix builds against the fixed version this should
;; be removed.

(define-module (my-tools git)
  #:use-module (git) ;; openable-repository
  #:use-module (guix git) ;; avoid duplication - use all public members
from here
  #:use-module (guix cache) ;; maybe-remove-expired-cache-entries
  #:use-module (guix store) ;; add-to-store
  #:use-module (guix records)
  #:use-module (guix gexp)
  #:use-module (ice-9 match)
  #:use-module (ice-9 ftw) ;; scandir
  #:use-module (srfi srfi-1) ;; filter-map
  #:use-module (srfi srfi-11) ;; let*-values
  #:export (
update-cached-checkout-x-ref
latest-repository-commit-x-ref

git-checkout-x-refs
git-checkout-x-refs?
git-checkout-x-refs-url
git-checkout-x-refs-branch
git-checkout-x-refs-commit
git-checkout-x-refs-recursive?))


;; Avoid duplicating required private functions
(define make-default-fetch-options (@@ (guix git)
make-default-fetch-options))
(define resolve-reference (@@ (guix git) resolve-reference))
(define clone/swh-fallback (@@ (guix git) clone/swh-fallback))
(define switch-to-ref (@@ (guix git) switch-to-ref))
(define update-submodules (@@ (guix git) update-submodules))
(define reference-available? (@@ (guix git) reference-available?))
(define cached-checkout-expiration (@@ (guix git)
cached-checkout-expiration))
(define %checkout-cache-cleanup-period (@@ (guix git)
%checkout-cache-cleanup-period))
(define delete-checkout (@@ (guix git) delete-checkout))
(define print-git-error (@@ (guix git) print-git-error))


(define* (update-cached-checkout-x-ref url
   #:key
   (ref '())
   recursive?
   (check-out? #t)
   starting-commit
   (log-port (%make-void-port "w"))
   (cache-directory
(url-cache-directory
 url (%repository-cache-directory)
 #:recursive? recursive?)))
  "Update the cached checkout of URL to REF in CACHE-DIRECTORY.  Return
three
values: the cache directory name, and the SHA1 commit (a string)
corresponding
to REF, and the relation of the new commit 

Help to workaround libgit2 fetch refs issue

[Phil]

Hi Guixers,

I raised an issue on libgit2 which effects the use of Guix with SSH git
clones and additional references:
https://github.com/libgit2/libgit2/issues/6183

The issue in summary is that if I want to build off a non-standard git
reference (say, a Pull Request), then even if I update my git config to
include that, eg:
remote.origin.fetch=+refs/pull-requests/*/merge:refs/remotes/origin/pr/*

libgit2's* initial clone* will not pull down the extra refs, despite this
being the behaviour of the git command line tool proper.  After the initial
clone, the git config is adhered to.

In Guix this means that the first time I build a PR it fails, and I have to
do something like "guix build  foo | guix build foo" which is at best a
clumsy hack, but it works!

Whilst the proper solution will be be done in libgit2 I was thinking if I
could quickly improve on my double-build workaround *without* having to
change Guix itself here (by *always *fetching):
https://github.com/guix-mirror/guix/blob/6adce1538d2df6fa2d68abc13ae94e2fa826d124/guix/git.scm#L466

If nothing else I was hoping to learn a bit more about how packages were
compiled as the details are a bit of blindspot for me!  So my plan was to
duplicate the git-checkout record and it's G-Exp compiler such that I could
redirect to a modified version of  update-cached-checkout when the source
is read from the package.

(I should also mention that the far simpler option of providing a new
"fetch" function like url-fetch or git-fetch doesn't work for me as the
repo I'm accessing happens to require SSH authentication and thus I'm using
the method as outlined here:  http://issues.guix.gnu.org/issue/31285#4 )

So I have something like:
(package
  (name "my-test-repo")
  (version production-version)
  (source
   (git-checkout*-x-refs*
(url "ssh://git@bitbucket:7999/ea/my-test-repo.git")
(commit commit-production)))
  (build-system python-build-system)
  and so on

The code I've naively duplicated is at the bottom of this e-mail - it
builds OK, so is syntactically correct, but fails when I reference it in a
package like above.  Having the record and git code as part of my local
channel is wrong, I know, but I'm looking a short term workaround with the
least disruptive footprint whilst I implement and then wait for release of
the strategic solution, probably in libgit2.

I suspect the reason for the failure is that I'd have to import my new
duplicated module somewhere in the Guix core code to make this work - i.e.
having the code in the channel is never going to work?

I was wondering if anyway could confirm this and/or give me a pointer of
where the compliation occurs (where the record in the source is handled and
compiled into a git clone) - even if my approach is (quite possibly)
unviable, I'd like to understand why!


Thanks,
Phil.


$ guix build -L /home/phil/git/guix/guix-packages/packages my-test-repo

guix build: warning: source expression failed to match any pattern
error: git-checkout-x-refs: unbound variable
hint: Did you forget `(use-modules (my-tools git))'?

guix build: error: my-test-repo: unknown package

_____

I put this in /home/phil/git/guix/guix-packages/package/my-tools/git.scm-
I've marked the actual change in *bold *- everything else is just
boilerplate.

;; -*- mode: scheme; eval: (guix-devel-mode 1);
geiser-scheme-implementation: guile -*-

(define-module (my-tools git)
  #:use-module (git) ;; libgit
  #:use-module (guix git) ;; CAREFUL could clash!
  #:use-module (guix records) ;; define-record-type*
  #:use-module (guix gexp)
  #:use-module (ice-9 ftw) ;; scandir
  #:use-module (ice-9 match)
  #:use-module (srfi srfi-11) ;; let*-values
  #:export (git-checkout-x-refs
git-checkout-x-refs?
git-checkout-x-refs-url
git-checkout-x-refs-branch
git-checkout-x-refs-commit
git-checkout-z-refs-recursive?))

;; local functions stolen from guix git
(define clone/swh-fallback (@@ (guix git) clone/swh-fallback))
(define with-libgit2 (@@ (guix git) with-libgit2))

;;;
;;; Checkouts.
;;;

;; Representation of the "latest" checkout of a branch or a specific commit.
;; Shadows git-checkout but uses difference function to get the repo.
(define-record-type* 
  git-checkout-x-refs make-git-checkout-x-refs
  git-checkout-x-refs?
  (url git-checkout-x-refs-url)
  (branch  git-checkout-x-refs-branch (default #f))
  (commit  git-checkout-x-refs-commit (default #f))  ;#f | tag | commit
  (recursive? git-checkout-x-refs-recursive? (default #f)))


(define* (update-cached-checkout-x-ref url
 #:key
 (ref '())
 recursive?
 (check-out? #t)
 starting-commit

Re: Guix Scripts and Unit Testing

[Phil]

This is largely resolved now - turns out my code sucked rather than needing
to do something clever with Guix module importing, which is what I
suspected ;-)

The only compromise I'm making now is that any complicated manifest for
example, I am having to split into a trivial manifest script calling into a
library that I can then unit test - so it's not quite as good as having a
meta switch for manifests or repl scripts, but it's good enough.

On Sun, 13 Feb 2022 at 20:58, Phil  wrote:

> I have a theory about what is causing this - I think it's me not thinking
> about deferred evaluation carefully enough.
>

This is true, but my second post was still missing the point!


>
> What I think is happening is that the propagated-inputs in my newly
> created test-package will have their evaluation delayed until the build
> occurs in the daemon.  I presume at this point my local library I've used
> add-to-load-path to include in the manifest will no longer be accessible
> and hence the eval failure being returned.
>
>
>
The problem was actually much easier to solve than I had originally
over-thought - simply by making my-package- with-my-gurobi a procedure
which returns the package rather than having it defined as a variable that
contains a package all the problems go away apart from a handful of imports
required in the library script.  The my-gurobi-package can also be nested
inside the let statement, and the whole procedure added to the library.

(define (my-package-with-my-gurobi)
  (let* ((test-package-string (get-env-var-or-prompt "GUIX_TEST_PACKAGE"))
 (test-package (specification->package test-package-string))
 (my-gurobi-package ((transform) (specification->package
"gurobipy"
(format #t "~%Setting Test Package: ~a~%" test-package-string)
(package/inherit test-package
 (propagated-inputs
  `(("gurobipy" ,my-gurobi-package) ;; add my new gurobi
,@(alist-delete "gurobipy"
(package-propagated-inputs test-package))) ;; remove the original
gurobi and splice

Re: Guix Scripts and Unit Testing

[Phil]

I have a theory about what is causing this - I think it's me not thinking
about deferred evaluation carefully enough.

On Sat, 12 Feb 2022 at 17:21, Phil  wrote:

>
> To try to work around this, I separated my scripts into a library module
> to be unit tested, and a short runner which will call into the library
> module to generate the manifest, however this approach produces unexpected
> behaviour - for example if I have a simple *gurobi-manifest.scm* like so:
>
>
>
>
>
>
>
> *(add-to-load-path (dirname (current-filename))) ;; put the script's
> location on the module path(use-modules (gurobi-transform-lib)) ;; load the
> script library(newline)(packages->manifest (list my-package-with-my-gurobi*
>
> *  (specification->package "python"))) ;; add
> python for convenience*
>
> The procedure we call to generate the package -
> *my-package-with-my-gurobi*, is in a file called
> *gurobi-transform-lib.scm* in the same directory as the manifest above.
>
>

This is crux of the issue - I think it's an issue with where the code for
my package is evaluated.

If I move just 2 functions back from my library into the manifest itself,
then everything works exactly as I'd expect.











*;; Prompt for Gurobi package if not set in env var(define
my-gurobi-package ((transform) (specification->package "gurobipy")))(define
my-package-with-my-gurobi  (let* ((test-package-string
(get-env-var-or-prompt "GUIX_TEST_PACKAGE")) (test-package
(specification->package test-package-string)))(format #t "~%Setting
Test Package: ~a~%" test-package-string)(package/inherit test-package
   (propagated-inputs  `(("gurobipy"
,my-gurobi-package) ;; add my new gurobi
,@(alist-delete "gurobipy" (package-propagated-inputs test-package)))
;; remove the original gurobi and splice*

What I think is happening is that the propagated-inputs in my newly created
test-package will have their evaluation delayed until the build occurs in
the daemon.  I presume at this point my local library I've used
add-to-load-path to include in the manifest will no longer be accessible
and hence the eval failure being returned.

IIRC there is a key you can add to package definitons to request module
loading on the daemon side as part of the build process - if something like
was to exist then it might be possible to move these 2 functions back to
the library rather than having them in the manifest, but as it stands it
seems the daemon has visibilty of the manifest script itself, but not
load-path and modules as set in the manifest script.

Guix Scripts and Unit Testing

[Phil]

Hi Guix,

I have some Guix scripts which take the form of manifests
(--manifest=file), packages (--load=file), or are called using "guix
repl".  Some of these are complex enough to warrant writing unit tests for
using srfi-64.  However, I'm having some difficulty trying to work out how
to run unit tests over procedures that are also used to descirbe manifests
and repl scripts, and was looking for some advice - details below!

Normally in vanilla Guile I'd structure my code to call a main procedure
using the Guile meta-switch, thus the script could also be imported by a
test script using "use-modules" or executed using the meta-switch:
https://www.gnu.org/software/guile/manual/html_node/The-Meta-Switch.html

However there is no way of doing this (that I can figure out) when the
script is being passed to "guix envrionment" or "guix repl" - meaning the
contents of my "main" procedure cannot be bound by a function in the script.

To try to work around this, I separated my scripts into a library module to
be unit tested, and a short runner which will call into the library module
to generate the manifest, however this approach produces unexpected
behaviour - for example if I have a simple *gurobi-manifest.scm* like so:







*(add-to-load-path (dirname (current-filename))) ;; put the script's
location on the module path(use-modules (gurobi-transform-lib)) ;; load the
script library(newline)(packages->manifest (list my-package-with-my-gurobi*

*  (specification->package "python"))) ;; add
python for convenience*

The procedure we call to generate the package - *my-package-with-my-gurobi*,
is in a file called *gurobi-transform-lib.scm* in the same directory as the
manifest above.

The initial call to the *my-package-with-my-gurobi* succeeds (The stdout
below shows this) but failure quickly follows, with a subsequent call to
the library failing:








*$ guix environment -m gurobi-manifest.scmSetting Gurobi Version: 9.1.2 <-
everything looks OK to start withSetting Test Package: simmguix
environment: error: failed to load
'gurobi-manifest.scm':ice-9/boot-9.scm:3329:6: In procedure
resolve-interface:no code for module (gurobi-transform-lib)*

*$*

If I cut and paste the library and manifest into the same file everything
works as expected.

Thus the best workaround I have at the moment is to have the unit tests run
directly from the manifest script, every time the manifest is used, but
this is only practical for a very small number of unit tests, and even then
it's a bit ugly.

Can anyone suggest a better way of having the unit tests in a separate file
which can import procedures used in repl or manifest scripts in such a way
that they still work or can be used as repl or manifest scripts themselves?

In the above example, I was hoping to have something like:

*guix repl -- gurobi-transform-lib-tests.scm*

Any ideas greatfully received!

Cheers,
Phil.

Re: Maven Build System Dependency Issue

[Phil]

Julien Lepiller writes:

> Hi Phil,
>
> I have already seen the issue previously, and I believe it is now fixed
> on master. As you can see here,
> https://github.com/guix-mirror/guix/blob/master/gnu/packages/java.scm#L7899
> we now propagate the correct parent pom.

Thanks Julien that looks exactly as I'd expect now.  Out of curiosity
was this change only merged to master in the last few days?  When I
looked on Guix mirror I ended up seeing the old 50 version just a few
days ago?  It's always possible I just looked at the wrong branch by
accident!

One last question - I note your change was committed on 21st Dec, which
is bad luck for me as the Python 3.9 upgrade went in on 17th Dec, and
I'm stuck for non-Guix reasons on 3.8.2 (so my Guix is a bit stale for
now).  I'll fix this medium-term on my side, but was wondering if there
is any way around this?

None of my previous e-mail's attempts worked, but using Guix Inferiors
this morning to make "guix" itself the inferior (rather than the maven
package), my pacakge then actually built.  Adopting the example given
in the manual, I was able to create a profile out of the below manifest,
which if I then run "guix build" inside it downloaded and built the older
build system on the fly, including java-commons-codec@1.14 - which is
pretty cool.

I could then continue to add packages from current guix alongside this,
although I'm not sure if this is a particularly good idea!

Given that I want to rewind the build system itself rather than a
specific package is this the right/canonical way to do this?  There
seems no way of using --with-inputs or --with-source to do this,
presumably because you need to make the change *before* guix itself is
called?


(use-modules (guix inferior) (guix channels)
 (srfi srfi-1))   ;for 'first'

(define channels
  ;; This is the commit id just before java-commons-codec was upgraded 1.14 -> 
1.15
  (list (channel
 (name 'guix)
 (url "https://git.savannah.gnu.org/git/guix.git;)
 (commit
  "a3b6e904484e23db65990be70d76ba32c15fd03f" ;; old fix

;;"a348520e2a253bc81fa92566e74f8b3e60fea058" ;; future fix

(define inferior
  ;; An inferior representing the above revision.
  (inferior-for-channels channels))

;; Now create a manifest contain guix itself rewound to where 
java-commons-codec=1.14
(packages->manifest
 (list (first (lookup-inferior-packages inferior "guix"

;;(specification->package "maven"))) ;; add other packages as required

Maven Build System Dependency Issue

[Phil Beadling]

Hi Guixers,

First let me say I'm a Maven novice, so it's possible I'm doing something
dumb on the Maven side of things.

I'm unable to make a bare-bones Maven project build in Guix.

This looks to be a problem with mismatched dependencies in Guix around
java-commons-codec.

I suggest what probably needs to be fixed upstream below (assuming my hunch
is correct!).

However I'm looking for advice on how to workaround this problem today to
get a basic Maven package building.

*My method:*

Following advice here:
https://maven.apache.org/guides/getting-started/maven-in-five-minutes.html

Using an environment like so:
guix environment --ad-hoc maven

I create a new git repo and put this in the root (so pom.xml, etc, sit in
in the root of the repo)

mvn archetype:generate -DgroupId=com.quantile.app
-DartifactId=java-test-repo
-DarchetypeArtifactId=maven-archetype-quickstart -DarchetypeVersion=1.4
-DinteractiveMode=false



I'm able to package and run everything OK (obviously the dependencies are
being pulled from the web in this case):
mvn package
java -cp target/java-test-repo-1.0-SNAPSHOT.jar com.quantile.app.App

Next I try to create a bare-bones Guix package to build this (this time
there will be no web access during the build):

(define-public java-test-repo-integration
  (let ((commit-integration "b9e3894aa3629fc2d60ceadea2f655d4cb6a826b"))
(package
  (name "java-test-repo")
  (version integration-version)
  (source
   (git-checkout
(url "ssh://git@vcs:1234/ea/java-test-repo.git")
(commit commit-integration)))
  (build-system maven-build-system)
  (home-page "https://vcs/projects/EA/repos/java-test-repo/browse;)
  (synopsis "Java Test Repo Production")
  (description "Java/Maven bare-bones framework for testing")
  (license license-quantile

I get the following build failure:

[INFO]

[INFO] BUILD FAILURE
[INFO]

[INFO] Total time:  1.136 s
[INFO] Finished at: 2022-02-08T14:39:36Z
[INFO]

[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M4:test (default-test)
on project java-test-repo: Execution default-test of goal
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M4:test failed: Plugin
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M4 or one of its
dependencies could not be resolved: Failed to collect dependencies at
org.apache.maven.plugins:maven-surefire-plugin:jar:3.0.0-M4 ->
org.apache.maven.surefire:maven-surefire-common:jar:3.0.0-M4 ->
org.apache.maven.surefire:surefire-api:jar:3.0.0-M4 ->
*commons-codec:commons-codec:jar:1.15:
Failed to read artifact descriptor for
commons-codec:commons-codec:jar:1.15: *Cannot access central (
https://repo.maven.apache.org/maven2) in offline mode and the artifact
org.apache.commons:commons-parent:pom:52 has not been downloaded from it
before. -> [Help 1]


After doing some digging I think I know why:

$ guix show java-commons-codec

name: java-commons-codec

version: 1.15

outputs: out

systems: x86_64-linux i686-linux

dependencies: *apache-commons-parent-pom@50* java-commons-lang3@3.9
java-junit@4.12


So it's packaged with a propagated apache-commons-parent-pom at version 50.


See also here for package definition:

https://github.com/guix-mirror/guix/blob/9fe48723c8266ffe8e6a639be2ec2e362cf20cb5/gnu/packages/java.scm#L7671here


But if I look at the POM in the Gnu Store - it is showing version 52 for
the commons-parent which I am assuming (perhaps incorrectly?) should match
the version 50 above?


/gnu/store/7qgi821xidn9nbb07qw35g7wmgv84jln-java-commons-codec-1.15/lib/m2/./commons-codec/commons-codec/1.15/commons-codec-1.15.pom



  

org.apache.commons

commons-parent

*52*

  


Likewise if I look at the source code on Apache's website - it's 52 too:

https://dlcdn.apache.org//commons/codec/source/commons-codec-1.15-src.tar.gz




  

org.apache.commons

commons-parent

*52*

  


What I think has happened is that java-commons-codec has been promoted from
version 1.14 to version 1.15 without changing the propoagated input?

On version 1.14 everything would match:

https://archive.apache.org/dist/commons/codec/source/commons-codec-1.14-src.tar.gz

  

org.apache.commons

commons-parent

50

  

*So assuming this isn't going to cause an avalanche of other issues the
solution proper seems to be either downgrade java-common-codec to version
1.14, or upgrade the dependency on apache-commons-parent-pom to 52?*

What I'm looking for today is any workaround which allows me to use Maven
without having to wait to for upstream fix.

My first attempt was to try - adding an inherited package and then swapping
this in place of 1.15:

Re: Parallel guix builds can trample?

[Phil Beadling]

Hi all,

The issue also exists when using --with-commit - see below for a refined
test that makes it trivial to demonstrate with any package where the source
is retrieved from git.

On Tue, 18 Jan 2022 at 10:10, Phil  wrote:

>
> > Philip Beadling  writes:
>


> >No, and we probably should do, even in spite of this issue.  We're going
> >to repeat the experiment now using --with-commit, to confirm behaviour
> there.
>
>
Let's try and build 2 package variations - for this experiement we don't
even have to run them at the same time:

mkdir checkout-test
cd checkout-test
mkdir cache1 cache2
XDG_CACHE_HOME=`realpath ./cache1` guix build
--with-commit=simm=*3981f60d81d3c9592d78b3e88545259713a93194
*simm
XDG_CACHE_HOME=`realpath ./cache2` guix build
--with-commit=simm=*ec1357c1fdf78a2ac1408d00c0c3e7d025ab9ade
*simm

Now look at the caches:

$ ll cache1/guix/checkouts

total 4.0K
-rw-rw-r-- 1 phil.beadling phil.beadling  10 Jan 18 12:37
last-expiry-cleanup
drwxrwxr-x 6 phil.beadling phil.beadling 162 Jan 18 12:37
lx4ipn27sk6uvu6b33kelt4tik65oianvl7b7yn5bpng7gpt4pyq
drwxrwxr-x 5 phil.beadling phil.beadling 115 Jan 18 12:37
mqqlbc4ksftu2tkbnxydosm4j6mynxsh5nmhpq3cfrofkktkhexq
drwxrwxr-x 5 phil.beadling phil.beadling 103 Jan 18 12:37
qqzuuoqxvqikivkpenqom43kzk2xr7nlntanbm6fpe7bci6kaenq
drwxrwxr-x 4 phil.beadling phil.beadling 202 Jan 18 12:37
smvbhjf46cbaxxju6fsqj7t6ujktxy5i7vhuhzpvgjt2vhid2vqq
drwxrwxr-x 5 phil.beadling phil.beadling 267 Jan 18 12:37
*xulvhb6agjrsf6x7wknutpn7wkt5i4nesk637czazvfxps357eoq*

$ ll cache2/guix/checkouts

total 4.0K
-rw-rw-r-- 1 phil.beadling phil.beadling  10 Jan 18 12:38
last-expiry-cleanup
drwxrwxr-x 6 phil.beadling phil.beadling 162 Jan 18 12:38
lx4ipn27sk6uvu6b33kelt4tik65oianvl7b7yn5bpng7gpt4pyq
drwxrwxr-x 5 phil.beadling phil.beadling 115 Jan 18 12:38
mqqlbc4ksftu2tkbnxydosm4j6mynxsh5nmhpq3cfrofkktkhexq
drwxrwxr-x 5 phil.beadling phil.beadling 103 Jan 18 12:39
qqzuuoqxvqikivkpenqom43kzk2xr7nlntanbm6fpe7bci6kaenq
drwxrwxr-x 4 phil.beadling phil.beadling 202 Jan 18 12:38
smvbhjf46cbaxxju6fsqj7t6ujktxy5i7vhuhzpvgjt2vhid2vqq
drwxrwxr-x 5 phil.beadling phil.beadling 267 Jan 18 12:38
*xulvhb6agjrsf6x7wknutpn7wkt5i4nesk637czazvfxps357eoq*

$

Despite asking for different variants of the simm package, they are both
stored under the same SHA.  If they both used the same cache directory (the
default) - they'd have overwritten each other.  Thus it is not safe to run
--with-commit or any other variation in parallel for the same user.

To prove I've matched the right SHA - I can back-out the original commit
ids provided to --with-commit:

$ cd
cache1/guix/checkouts/xulvhb6agjrsf6x7wknutpn7wkt5i4nesk637czazvfxps357eoq
&& git rev-parse HEAD && cd -

*3981f60d81d3c9592d78b3e88545259713a93194*
~/checkout-test

$ cd
cache2/guix/checkouts/xulvhb6agjrsf6x7wknutpn7wkt5i4nesk637czazvfxps357eoq
&& git rev-parse HEAD && cd -

*ec1357c1fdf78a2ac1408d00c0c3e7d025ab9ade*
~/checkout-test

$

Whilst my original setup was a bit of a edge-case, I think this refined
example demonstrates that the problem can occur with typical/reasonable use
of guix switches?  I can even think of how this could occur without the
user running in parallel - for example if a package required via nth-order
dependencies to build 2 versions of the same repo - it's possible that a
single "guix build" command could cause the issue?

Thoughts?

Re: Parallel guix builds can trample?

[Phil]

Ricardo Wurmus writes:

> Philip Beadling  writes:
>
>> The source in each one will be slightly different and although I only
>> specify commit id in the package
>
> How are you using Guix with this?  Do you generate Guix package
> expressions?  Do you use “guix build --with-commit”?

No, and we probably should do, even in spite of this issue.  We're going
to repeat the experiment now using --with-commit, to confirm behaviour there.

Thanks for reminding me about the switch!

Re: Parallel guix builds can trample?

[Phil]

Maxime Devos writes:

> Maybe 'latest-repository-commit' and 'update-cached-checkout' commit
> can be modified to not use 'switch-to-ref', and instead directly ask
> libgit ‘what's the tree structure of commit cabba9e’ and call a
> procedure like 'add-file-tree-to-store'.  That would avoid lock files,
> creating separate directories for concurrent checkouts, ...

Thanks Maxime - I'll take a look at this as an option.

One option I was thinking of is to make this SHA call dependent on the
branch as well as the URL:
https://github.com/guix-mirror/guix/blob/9f526f5dad5f4af69d158c50369e182305147f3b/guix/git.scm#L131-L133

Technically this still leaves a race condition if someone were to do
parallel builds on the same branch however, so to completely solve the
problem I think we'd have make the SHA call on repo+commit-id.

This obviously limit the utility of the cache.

To mitigate performance concerns, I think it's possible to set the
branches and depth in git such that we only clone the bare-bones only?.

Re: Parallel guix builds can trample?

[Phil Beadling]

Hi Ricardo, all,



I think we’ve worked out what the issue is, and have a proposed workaround,
and perhaps a case for solving the problem in Guix itself (depending on
what you people think!).



The issue is that despite each build being performed in its own isolated
container, these containers are fed by the same per-user cached source
directory.  In the case where *different* versions of the *same* repo are
built at once, this results in a race condition.



In our case we have one Linux account that does a lot of automated Guix
builds for us.



One example is this account watches our source control and automatically
rebuilds all outstanding Pull Requests (PRs) on a repo, after a separate
successful merge to our integration branch.  PRs are uniquely identified as
monotonically increasing PR numbers eg, PR-1, PR-2, PR-3 and so on.  Each
is a different branch on the same Repo with slightly different candidate
changes in it.  They are automatically kept up to date with the integration
branch.



To do this our watcher fires off (near) instantaneously dozens of guix
builds, each with their own local channel customized for the PR it is
building.  Doing them in parallel is important to make the system usably
responsive.



Each fired process does this:

   - Clone the channel containing the package into a local directory
   - Modify the commit id of the package to the new merged head of the PR
   - Modify the package version to some dummy version containing the PR
   number
   - Build the modified package using the local channel
   - Report the result (the build is effectively discarded; it is never
   used for anything)



What we think is happening is the following:



   - For each build that is kicked off in quick succession the local cache
   of the repo required updated by *update-cached-checkou*t
  -
  
https://github.com/guix-mirror/guix/blob/9f526f5dad5f4af69d158c50369e182305147f3b/guix/git.scm#L476
  -
  
https://github.com/guix-mirror/guix/blob/9f526f5dad5f4af69d158c50369e182305147f3b/guix/git.scm#L279
   - The problem with this is because each version is using the same cached
   repo --- before one has a chance to take a copy of the updated checkout,
   that checkout can be changed by a separate build process



Thus there is a race condition in this scenario.  We can provide a longer
test script to demo this if required – it’s quite straightforward to
reproduce just with a bash script, now we know what is causing it.



Our workaround has been to change XDG_CACHE_HOME for each PR build we do.
But this is a bit unsatisfactory as it effects processes beyond Guix – it
casts too wide of a net, but it does resolve the problem for the time being.



Do people think this is enough of an issue to make a switch available in
Guix to prevent sharing of cached clones?  This would be easy enough to
implement – a crude solution would be that each cache directory name would
simply be generated using a SHA of a string which includes the PID or
similar to ensure a unique name, and because it is never going to be reused
it could be deleted immediately after the build.



Whilst this is unlikely to happen at the console, as people script guix
build use-cases to fit their own problems (in particular building lots of
variations of a single piece of software) – I can see this causing a
headache?  I think at least the manual should make it clear that you cannot
build 2 packages referencing the same repo at the same time with the same
user (unless I’ve missed this bit I don’t think it’s made explicitly
clear?).  An even simpler change would be introduce a lock file that
refused the 2nd build and at least preventing the race condition happening,
and ensuring referential transparency, or simpler still just placed a
warning on stderr?



If people are amenable to adding a switch or other config option, we’d be
happy to look writing the patch?


Any thoughts/comments/advice?


Cheers!
Phil.


On Wed, 12 Jan 2022 at 09:37, Phil  wrote:

> Hi - more details below.
>
> Ricardo Wurmus writes:
>
> >
> > How are you using Guix with this?  Do you generate Guix package
> > expressions?  Do you use “guix build --with-commit”?
> >
>
> The situation is like this - if we had a directory of clones of my
> channel:
>
> - pr-1
> - pr-2
> - pr-3
> - pr-4
> ... and so on
>
> Initially all the clones are taken from the master branch of my
> channel and are all identical - but we change the version and commit to
> match the head of each PR branch as per below.
>
> Each clone looks like this:
> - pr-1
>   - my-package.scm
> - pr-2
>   - my-package.scm
> and so on
>
> Each my-package.scm has a package like below - the inital packages are all
> identical, but my system effectively seds the version and commit values
> like the below.  These values are never committed back to master they
> are used only a

Re: Parallel guix builds can trample?

[Phil]

Hi - more details below.

Ricardo Wurmus writes:

>
> How are you using Guix with this?  Do you generate Guix package
> expressions?  Do you use “guix build --with-commit”?
>

The situation is like this - if we had a directory of clones of my
channel:

- pr-1
- pr-2
- pr-3
- pr-4
... and so on

Initially all the clones are taken from the master branch of my
channel and are all identical - but we change the version and commit to
match the head of each PR branch as per below.

Each clone looks like this:
- pr-1
  - my-package.scm
- pr-2
  - my-package.scm
and so on

Each my-package.scm has a package like below - the inital packages are all
identical, but my system effectively seds the version and commit values
like the below.  These values are never committed back to master they
are used only as local channels to build each PR to test each build
still passes.

(define-public my-package
  (package
(name "my-package")
(version "this-is-different-for-each-pr")  ;; replace master version
(source
  (git-checkout
(url "ssh://same@repo:7999/same/repo.git")
(commit "this-is-different-for-each-pr") ;; replace master version 
everything else remains the same in the package


At this point we have lots of local channels referencing different commits, in
the same package, ready to build - so I spawn them all simultaneously -
the equivalent pseudo-shell that I will mock up today would be:

# define some sort of return code array:
RC=[]

for dir in pr-dirs
  RC[${dir}]=`guix build -K -L ${dir} my-package & 2>&1 > /tmp/${dir}.log`  # 
note the ampersand
wait

for rc in $RC
  if $rc.value != 0:
report the failure of build $rc.key

What I'm seeing occasionally is that the logs and return code for say directory 
pr-1
and appearing in the guix build for pr-3 or pr-6 instead.

We know this becuse the code is different enough in pr-1 that it's logs
are unique across all the PRs.  We can also check the source code if the
build fails using --keep-failed to show it doesn't match the commit id
in the package used to build it.

Hopefully that makes sense?  I can post the actual shell script once
I've written the mock.

Parallel guix builds can trample?

[Phil]

Hi all,

I'm seeing strange issues when running many "guix build" commands of the
same package simultaneously on the same linux account/server at the same
time.

Tracing this requires a lot of detective work, so for now I've omitted
the logs - my initial question is - am I doing something which is
obviously not going to work with Guix, or would people expect the below
use-case to work?

In my example I have ~12 clones of my private guix channel, sat in their
own directories under my user account.

I update the same single package on each channel locally with a
different git commit id and package version.  So 12 variations of the
same package, accessing different commit ids, in the same package's
source repo, and having different package versions in Guix.

I then call "guix build -K -L /path/to/each/local/clone package-name" 12
times, one for each clone, these run in parallel shell sessions.

This is all automated so each build is started within a fraction of
second - trying to reproduce this issue by hand is proving difficult,
and the issue is still sporadic in the automated system.

The logs all show that each of the ~12 channels receive a local update
with a unique and valid commit id and version for the changed package.

However what I'm seeing is that some of the builds are failing with the
error messages of other commit ids!

To be clear - the failure is not the surprise here, the builds are
getting mixed up and some are being trampled by what appears to be
either a race condition or stale state.  The surprise is that the
failures are being tied to the wrong commit ids which do not contain the
failures reported.

Use of "--keep-failed" means I can prove this unequivocally, where the
saved down /tmp/package-name.verson.drv.0 source code does not match the
source of the commit id stated within the updated package - which I can
see in the logs.  I can also show that the source code matches exactly
one of the other commit ids for one of the other clones, which was
expected to fail.

This happens sporadically, but we can reproduce the issue several times
a day.

I don't yet have 100% proof that the issue is happening inside Guix, but
I've ruled out most (but not all) other causes so far, and the
--keep-failed evidence tied with logs showing correct inputs is quite telling.

Have any other bugs ever been reported like this that people are aware
of?  Have other people ran multiple builds in Guix under one account at
the same time without issue?  Any advice on how trap the issue, given
it's hard to reproduce?

The problem as never been seen when we do each guix build in serial.

Apologies for the long and rather circumstantial e-mail!

Cheers,
Phil.

Re: Guix Packaging Meetup Tomorrow (Saturday) Oct 30 18:00 UTC (14:00 ET)

[Phil]

Hi jgart,

Is there a recording of this available? - I couldn't make it yesterday.

jgart writes:

> Hi Guixers!
>
> I'd like to invite you to a guix packaging meetup tomorrow (Sat) Oct 30 2021 
> at
>
> 14:00 ET (18:00 PM UTC).
>

Issues with v910 of mypy

[Phil Beadling]

Hi all,

Was wondering if anyone had come across the issue with mypy using modular
stubs in v910.  As far as I can see installing for example
python-types-requests and python-mypy in Guix leads to mypy ignoring the
supplemented request stubs provided by the types-requests package.  I've
raised this on mypy, but it not really a mypy bug as there solution does
work with pip (although other package managers like PDM have reported
similar issues):

https://github.com/python/mypy/issues/11395

Was wondering if anyone has
a) experienced the same issue?
b) had a workaround?

I could consider downgrading mypy to the last non-modular stub version if
this issue is widely expreienced and there is no immediate workaround from
guix or mypy message boards?

As it standards the version in guix is pretty unusable to anyone developing
outside of python core library sets and using mypy - we'd be better on a
slightly older version that works for now, I think?

Thanks,
Phil.

Re: --with-source version not honored?

[Phil Beadling]

I was able to write a short manifest that avoided the overwriting of the
original "foobarpy" package in my case, and instead cleanly replace it with
a newer version specified using "with-source".

By setting, for example, GUIX_FOOBAR_VERSION=1.23 and
GUIX_TEST_PACKAGE=my-package - I can now create an environment which builds
the version of foobarpy I want from an https link generated from
GUIX_FOOBAR_VERSION, and then create a modified version of "my-package"
with foobarpy replaced, rather than overwritten.

This is much better than my original hack, but it's still not recursive so
whilst it does rebuild my-package with the new version of foobarpy as a
propagated input, it isn't rebuilding the whole dependency tree - other
inputs to my-package could also depend on foobarpy, and should be rebuilt
too - so what I've done is more of a graft, buthappens to be good enough
for my specific use case (the foobarpy is not used anywhere else in my
case).

Anyway thought I'd share this here as a template for simple non-recursive
cases.  I suspect it's not a massive effort to make it recursive, when I
have a moment I'll have a think about that too.  Any thoughts on how to
improve are welcome!

 (use-modules (guix transformations) ;; options->transformation
  (guix packages) ;; package/inherit
  (srfi srfi-1) ;; fold
  (srfi srfi-98)) ;; get-environment-variable


 (define (generate-foobar-link version)
   (string-append "https://packages.foobar.com/;
  (substring version 0 3) "/foobar"
  version "_linux64.tar.gz"))

 (define transform
   ;; The package transformation procedure.
   (let* ((foobar-version (get-environment-variable "GUIX_FOOBAR_VERSION"))
  (foobar-link (string-append
"foobarpy@" foobar-version
"=" (generate-foobar-link foobar-version
 (format #t "~%Setting Foobar Version: ~a~%" foobar-version)
 (options->transformation
  `((with-source . ,foobar-link)

 (define my-foobar-package (transform (specification->package "foobarpy")))

 (define-public my-package-with-my-foobar
   (let* ((test-package-string (get-environment-variable
"GUIX_TEST_PACKAGE"))
  (test-package (specification->package test-package-string)))
 (package/inherit test-package
   (propagated-inputs
`(("foobarpy" ,my-foobar-package)
  ,@(fold alist-delete (package-propagated-inputs test-package)
'("foobarpy")))

 (packages->manifest (list my-package-with-my-foobar))



On Sat, 23 Oct 2021 at 11:04, Phil  wrote:

> Hi,
>
> Ludovic Courtès writes:
>
> > For historical reasons, ‘--with-source’ only applies to leaf packages,
> > unlike most (all?) other transformation inputs.  Concretely, this works:
>
> Yes I think this is what I'm hitting.  I want to reference a package
> using --with-source, but then immediately build an environment which
> feeds this source as an input into a 2nd package - forcing it to
> rebuild.
>
> My hack works but for propagated dependencies in python - but it's not a
> proper solution, and is brittle.  I was going to try to improve
> it by writing a Guile script to avoid the overwriting - I can post any
> solution I find if I get something working.
>
> Currently I can only do 2 things at the command line it seems:
>
> 1. I can try package X without writing a package but only in isolation
> or as a leaf.
>
> 2. I can replace package X as a dependency of package Y, if I write a
> package
> for it.
>
> Being able to combine 1 and 2 at the command line - would be very
> useful.
>
> Cheers,
> Phil.
>

Re: --with-source version not honored?

[Phil]

Hi,

Ludovic Courtès writes:

> For historical reasons, ‘--with-source’ only applies to leaf packages,
> unlike most (all?) other transformation inputs.  Concretely, this works:

Yes I think this is what I'm hitting.  I want to reference a package
using --with-source, but then immediately build an environment which
feeds this source as an input into a 2nd package - forcing it to
rebuild.

My hack works but for propagated dependencies in python - but it's not a
proper solution, and is brittle.  I was going to try to improve
it by writing a Guile script to avoid the overwriting - I can post any
solution I find if I get something working.

Currently I can only do 2 things at the command line it seems:

1. I can try package X without writing a package but only in isolation
or as a leaf.

2. I can replace package X as a dependency of package Y, if I write a package
for it.

Being able to combine 1 and 2 at the command line - would be very
useful.

Cheers,
Phil.

Re: --with-source version not honored?

[Phil]

Hi

zimoun writes:

> It reminds me this thread:
>
> 

Thanks this is an interesting thread - I stumbled upon a quirk trying to
find the right combination of switches.

I found that if I do this (which I think is bad):

guix environment --with-source=foobar@9.5.0=/path/to/package
some-package-that-depends-on-foobar --ad-hoc foobar

This gives me the warning that with-source will have no effect on
some-package-that-depends-on-foobar, but that's not strictly true here.

In this case the depending package's dependencies are installed into the
environment including the original version of foobar (9.0.1) first, but this
is then overwritten by foobar@9.0.5 - I can see files from
both packages under the site-packages for foobar.

Weirdly this is almost what I was trying to do:
"Install some-package-that-depends-on-foobar, but rebuilt with the version of
foobar from with-source."

However the reason it works is probably not intended behavior.

I couldn't work out an incantation that provided this facility properly
- my best guess was:

guix environment --with-source=foobar@9.5.0=/path/to/package
--with-inputs=foobar=foobar@9.5.0 some-package-that-depends-on-foobar

Which returns that it can't find foobar@9.5.0.

I figure at this point I might have more luck if I drop into Guile to do
this, but it's always a win when I can do it as a one-liner in shell.

Any ideas if I can create a new package with --with-source and then
substitute it in the same command for an input of another package? 

Re: --with-source version not honored?

[Phil]

Thanks for the reply.

Julien Lepiller writes:

> evaluated before you can import the package. Maybe (package-version 
> this-package) would work?

Yes! (package-version this-package) worked perfectly - thanks for your help.

--with-source version not honored?

[Phil Beadling]

Hi all,

I'm using the following incantation:

guix build
--with-source=foobar@9.5.0=/opt/thirdparty/foobar/foobar950_beta/linux64
<--with-source=gurobipy@9.5.0=/opt/thirdparty/gurobi/gurobi950_beta/linux64>
foobar


However the package build is failing with:

(copy-file "lib/libfoobar.so.9.0.1" "/gnu/store/gkawzac…")

In procedure copy-file: No such file or directory


That is the new version number 9.5.0 is not written to every place when
transforming the original package (version 9.0.1).  I think only the
package-version is updated, but the other package components are not then
regenerated, meaning that if they use the package-version as an input we
get a disjoint package.

In the example above I use version like so:






*(add-after 'install 'install-foobar-library
  (lambda* (#:key outputs #:allow-other-keys)
  (let* ((dir (string-append (assoc-ref outputs "out")

"/lib/python3.8/site-packages/foobar/"))
 (lib-to-install (string-append
"libfoobar.so." ,version))*

But ",version" is not updated, nor is it updated if I change this to call
(package-version foobar).


If I drop into Guile I can see this a bit more clearly by writing a
manifest - the code below gives exactly the same error however when the
package-version is displayed it correctly responds with 9.5.0.

Not sure if this should be considered a bug, or if there is a lazy way of
evaluating version so avoid the problem - I think it's unexpected from a
practicioners point of view as packages end up inconsistent.

I presume I can manually replace the arguments section of the package in
the manifest to workaround this - is there a standard way of doing this?

Any ideas or clarifications welcome!

Cheers,
Phil.
















*(use-modules (guix transformations) (guix packages))(define
transform  ;; The package transformation procedure.
(options->transformation   '((with-source .
"gurobipy@9.5.0=/opt/thirdparty/foobar/foobar950_beta/linux64"(define
my-package (transform (specification->package "foobar")))(display
(package-version my-package)) ;; this will display version
9.5.0(newline)(packages->manifest (list my-package)) ;; building this will
fail because copy-file still looks for 9.0.1*

Guix+Jenkins slides/video

[Phil]

Hi all,

A few of you might know me from guix-help or the Guix reddit channel.

I did a talk last week and there's been some interest so thought I'd
post the link here:
https://www.devopsworld.com/agenda/session/617842

Slides are available at the bottom without registration.  The video
requires registration (and needs full screen).

This is still a work in progress, so any suggestions, comments,
questions very welcome.

Current work is focusing on generating AWS Lambda functions in Jenkins
using Guix profiles, and making Guix-Juptyer available on our Jenkins
deployments - depending on progress we could offer to do talks at Guix
Days 2021 on these.

Cheers,
Phil.