Re: Debugging missing architecture support

[Konrad Hinsen]

Hi Ludo,

> This shows an attempt to compile libwebp natively for aarch64-linux,
> which fails on your machine.
>
> To cross compile, you would run:
>
>   guix build python-jupyterlab --target=aarch64-linux-gnu

That's already what I did, by giving the same option to "guix pack".

I had someone check on a real ARM machine, and there libwebp compiles
just fine. There seems to be some cross-compilation issues, for whatever
reasons - this isn't my priority for now.

I ended up tracking down all the issues with the container I was trying
to produce, and in the end it's always ghc that prevents builds for
aarch64-linux. Most frequently by having pandoc as a dependency.

I'll have to take a closer look at Efraim's diff (thanks Efraim!),
but I fear it's above my paygrade.

Cheers,
  Konrad.

Re: Debugging missing architecture support

[Konrad Hinsen]

Hi Saku,

> Maybe someone else can give more general or Guix specific advice on
> finding out the cause of such problems, but I believe that in this case
> the fix would just be packaging GHC for aarch64-linux. It should[1] be
> possible but it will require some work.
>
> [1]: https://gitlab.haskell.org/ghc/ghc/-/wikis/platforms

Thanks for the pointer! It looks indeed like GHC already support
aarch64, so it's just a matter of integrating that support into Guix.
I'll see if I can find someone motivated and competent to do that.

Cheers,
  Konrad

Re: Debugging missing architecture support

[Konrad Hinsen]

Ricardo Wurmus  writes:

> You can try this in guix repl:
>
> --8<---cut here---start->8---
> (import (srfi srfi-1)
> (guix packages)
> (gnu packages))
>
> (define p (specification->package "git-annex"))
> (define deps (package-development-inputs p))
> (find (lambda (pkg)
> (not (member "aarch64-linux" (package-supported-systems pkg
>   (map cadr deps))
> --8<---cut here---end--->8---

Thanks, that's very useful!

I changed the last command to

--8<---cut here---start->8---
(find (lambda (pkg)
(and (package? pkg)
 (not (member "aarch64-linux" (package-supported-systems pkg)
  (map cadr deps))
--8<---cut here---end--->8---

because some packages have local files among the development inputs.

This search doesn't work for all cases though. For "python-jupyterlab"
(from the guix-science channel) it returns no problematic dependencies,
and yet I cannot build the package for aarch64-linux.

The problem in that case seems to be cross-compilation. The dependency
of python-jupyterlab that fails to build is libwebp, whose build log
says:

   @ unsupported-platform 
/gnu/store/7fj9ckgxw27r196vkisc9cm3n8v9072x-libwebp-1.3.2.drv aarch64-linux
   while setting up the build environment: a `aarch64-linux' is required to
   build `/gnu/store/7fj9ckgxw27r196vkisc9cm3n8v9072x-libwebp-1.3.2.drv',
   but I am a `x86_64-linux'

Maybe this would build on an actual ARM64 machine.

Cheers,
  Konrad.

Debugging missing architecture support

[Konrad Hinsen]

Hi Guix experts,

in trying to build a Docker image for ARM64 (aarch64-linux), I am
hitting errors of the kind

  "package git-annex@10.20230926 does not support aarch64-linux"

There doesn't seem to be anything specifically in the package
definitions that precludes building for aarch64-linux, so I suspect it's
either a dependency or the build system that lacks the required support.

How would I go about debugging such issues? Maybe the causes are simple
and I can fix them. But I don't even know where to start.

Cheers,
  Konrad.

Re: Symlinks in "guix pack" and "guix shell"

[Konrad Hinsen]

Konrad Hinsen  writes:

> I found a simpler one, using a more recent Guix commit:
>
>guix time-machine --commit=7b0863f07a113caef26fea13909bd97d250b629e \
>-- pack -S /etc/ssl=etc/ssl --format=squashfs bash nss-certs
>
> Unfortunately, I have no idea how to debug this, as the image is
> constructed by the daemon.

I think I found the bug, by playing with variations on my command line:

   https://issues.guix.gnu.org/68841

Cheers,
  Konrad.

Re: Symlinks in "guix pack" and "guix shell"

[Konrad Hinsen]

Konrad Hinsen  writes:

> Here is my most recent experiment:

I found a simpler one, using a more recent Guix commit:

   guix time-machine --commit=7b0863f07a113caef26fea13909bd97d250b629e \
   -- pack -S /etc/ssl=etc/ssl --format=squashfs bash nss-certs

Unfortunately, I have no idea how to debug this, as the image is
constructed by the daemon.

Cheers,
  Konrad

Re: Symlinks in "guix pack" and "guix shell"

[Konrad Hinsen]

Hi Tomas,

Thanks for your reply!

> I think the relevant part is whether some certificate package is installed 
> (via
> propagated inputs) or not.  If I explicitly add nss-certs to the command 
> above,
> it starts to work:

Sorry, I forgot to add that I always have nss-certs among my packages
when I try this symlink. Otherwise I get an error message as well.

> Could you please share some examples where you get no error, but it has no
> effect?  Maybe they really are bugs (and should be reported).

Here is my most recent experiment:

$ guix time-machine --url=https://codeberg.org/khinsen/guix.git 
--commit=615023ddc9407f27ec7c04060fb520a4cbb3b03f  --disable-authentication -- 
pack -S /etc/ssl=etc/ssl -S /etc/protocols=etc/protocols --format=squashfs 
--save-provenance snakemake bash git git-annex coreutils wget python nss-certs 
net-base
guix time-machine: warning: channel authentication disabled
guix pack: warning: ambiguous package specification `python'
guix pack: warning: choosing python@3.10.7 from gnu/packages/python.scm:610:2
/gnu/store/psq0bqcsnbk7p5w1m2771mlilyikicp6-snakemake-bash-git-git-annex-coreutils-squashfs-pack.gz.squashfs

$ singularity exec 
/gnu/store/psq0bqcsnbk7p5w1m2771mlilyikicp6-snakemake-bash-git-git-annex-coreutils-squashfs-pack.gz.squashfs
 bash
WARNING: passwd file doesn't exist in container, not updating
WARNING: group file doesn't exist in container, not updating
Singularity> ls /etc
hosts  localtime  resolv.conf
Singularity> exit

Neither /etc/ssl nor /etc/protocols are present in the container.

On the other hand, the same packages exported for Docker:

$ guix time-machine --url=https://codeberg.org/khinsen/guix.git 
--commit=615023ddc9407f27ec7c04060fb520a4cbb3b03f  --disable-authentication -- 
pack -S /etc/ssl=etc/ssl -S /etc/protocols=etc/protocols --format=docker 
--save-provenance snakemake bash git git-annex coreutils wget python nss-certs 
net-base
guix time-machine: warning: channel authentication disabled
guix pack: warning: ambiguous package specification `python'
guix pack: warning: choosing python@3.10.7 from gnu/packages/python.scm:432:2
/gnu/store/90ylyh60vvi2inqg9i8wlcfbv8m2px89-snakemake-bash-git-git-annex-coreutils-docker-pack.tar.gz

$ docker load -i 
/gnu/store/90ylyh60vvi2inqg9i8wlcfbv8m2px89-snakemake-bash-git-git-annex-coreutils-docker-pack.tar.gz
Loaded image: snakemake-bash-git-git-annex-coreutils:latest

$ docker run -it snakemake-bash-git-git-annex-coreutils:latest bash
bash-5.1# ls /etc
ls /etc
hostname  hosts  mtab  protocols  resolv.conf  ssl

Both links are present.

Finally, "guix shell" now works (I hadn't tried for a few months):

$ guix time-machine --url=https://codeberg.org/khinsen/guix.git 
--commit=615023ddc9407f27ec7c04060fb520a4cbb3b03f  --disable-authentication -- 
shell -C -N -S /etc/ssl=etc/ssl -S /etc/protocols=etc/protocols snakemake bash 
git git-annex coreutils wget python nss-certs net-base
guix time-machine: warning: channel authentication disabled
guix shell: warning: ambiguous package specification `python'
guix shell: warning: choosing python@3.10.7 from gnu/packages/python.scm:432:2
[env]$ ls /etc
group  hosts  nsswitch.conf  passwd  protocols  resolv.conf  services  ssl

For this example, the only failure is for "pack –format=squashfs".
But the very same command line applied to a different commit (a few
weeks earlier) worked fine.

In case you wonder why I use a personal fork of Guix: For Singularity
export to be usable, I need a patch that is on core-updates
(https://git.savannah.gnu.org/cgit/guix.git/commit/?h=core-updates=d8875d76555faf1a11f6838312bfe4b7f6121d2c).
 In
the meantime, I added a graft version of this patch to my personal fork.

Finally, I cleaned up the output of the above commands by removing the
usual messages from the build process, keeping only the messages that
are specific to my example.

Cheers,
  Konrad.

Symlinks in "guix pack" and "guix shell"

[Konrad Hinsen]

Dear Guix experts,

I am seeing inconsistent behavior with the creation of symbolic links
in containers and exports, and I wonder whether this is a bug or missing
documentation.

According to the documentation, there are no constraints on what I can
link to. In practice, some symbolic link requests have no effect, but
there is no error message either.

Example: I am trying to tell programs in a container to use the
certificates from package nss-certs, instead of requiring the user
to share /etc/ssl from the host machine. So I add 

   -S /etc/ssl=etc/ssl

to my command lines for "guix shell -C" and "guix pack" (for creating
Docker or Singularity images.

Sometimes this works, sometimes it doesn't. With "guix shell", I never
got it to work. With "guix pack", the observed behavior is random.
Adding a completely unrelated package to my container can make the
difference.

Can someone shed some light on this?

Cheers,
  Konrad

Re: What's the difference between a shell environment and a profile?

[Konrad Hinsen]

Hi Simon and Ludo,

>> It does not differ.  What differs is that “guix shell” raises nothing
>> for the collision – maybe “guix shell” does not check the collision, I
>> do not remember – when “guix package” raises an error for the same
>> collision.
>
> Indeed, ‘guix shell’ does not check collisions, for “historical
> reasons”.  Perhaps something worth fixing at some point:
>
>   https://issues.guix.gnu.org/54350

Thanks for pointing out this behavior, which I find surprising but
understandable for the use case of development environments (although I
didn't expect collisions to be so frequent even in that context).

The problem I have now is how to update my recommendations for
computational scientists, which can be summarized as "run all research
computation in "guix shell" (see
e.g. 
https://10years.guix.gnu.org/program/#guix-as-a-tool-for-computational-science).
 As
I just found out, I am recommending people to switch off an important
level of error checking.

I'll think about this a bit and then perhaps comment on the old issue
thread!

Cheers,
  Konrad.

What's the difference between a shell environment and a profile?

[Konrad Hinsen]

Hi Guix experts,

While doing some software archeology, I ran into a behavior that I don't
quit understand.

First, I tested some Python 2 scripts in a shell environment:

guix time-machine -C ./channels.scm \
-- shell --container \
   python2 python2-mmtk python2-matplotlib \
   --with-input=python2-numpy=python2-numpy@1.8.2 \
   -- python

Works fine. Next, I wanted to create a Guix profile with the same
software, mainly to keep it in the store for long-term future use,
surviving garbage collection:

guix time-machine -C ./channels.scm \
-- package -p ./python2-profile \
   -i python2 python2-mmtk python2-matplotlib \
   --with-input=python2-numpy=python2-numpy@1.8.2

Guix wasn't happy at all with this:

  guix package: warning: Consider running 'guix pull' followed by
  'guix package -u' to get up-to-date packages and security updates.

  The following packages will be installed:
 python22.7.18
 python2-matplotlib 2.2.5
 python2-mmtk   2.7.13

  guix package: error: profile contains conflicting entries for glib
  guix package: error:   first entry: glib@2.70.2 
/gnu/store/64bdjb3nwdkadmy5z2wph9cgqr0bwijm-glib-2.70.2
  guix package: error:... propagated from cairo@1.16.0
  guix package: error:... propagated from python2-pycairo@1.18.2
  guix package: error:... propagated from python2-matplotlib@2.2.5
  guix package: error:   second entry: glib@2.73.3 
/gnu/store/kf488k7v0lc48ylbs4xxpam0dbl3r4jl-glib-2.73.3
  guix package: error:... propagated from gobject-introspection@1.73.1
  guix package: error:... propagated from python2-matplotlib@2.2.5
  hint: You cannot have two different versions or variants of 
`python2-matplotlib' in
the same profile.

In the end I did:

guix time-machine -C ./channels.scm \
-- shell --container -r ./python2-profile \
   python2 python2-mmtk python2-matplotlib \
   --with-input=python2-numpy=python2-numpy@1.8.2 \
   -- python

which should have the effect I am looking for, but I don't understand
how this differs from the "guix package" attempt that failed. The
difference seems to be related to the –with-input transform, because
I don't see what else could cause the "two different versions of
python2-matplotlib" to be around.

Any ideas?

Cheers,
  Konrad

Re: Time travel accident

[Konrad Hinsen]

Simon Tournier  writes:

> I have,
>
> --8<---cut here---start->8---
> $ guix gc --list-dead | grep 9gfmn1yra7rzavxb9wppqi4lpdvqid8c
> finding garbage collector roots...
> determining live/dead paths...
> /gnu/store/9gfmn1yra7rzavxb9wppqi4lpdvqid8c-inferior-script.scm

For me it is (well, was, see below) under "–list-live". But it has no
referrers. And...

> $ guix gc -D /gnu/store/9gfmn1yra7rzavxb9wppqi4lpdvqid8c-inferior-script.scm

fails for me (as it should for a live item).

Yesterday, I did "rm -rf ~/.cache/guix/inferiors" followed by "guix gc".
Right now, after the above tests, I did another "guix gc". And now...
the two items we are testing here are "dead" and I could remove them.

So it seems that two garbage collections are better than one, even
though nothing much happened in between (just a few "guix shell"
executions that added a few items to the store).

Cheers,
  Konrad

Re: Time travel accident

[Konrad Hinsen]

Hi Simon,

> I get the same thing without doing anything special.

Interesting. Before my clock-changing experiment, the same command line
got me much further: the old Guix started to work and only failed when
building OpenSSL.

> Well, v1.0 appears to me as the zero for time-travel – soft limit.  And
> the introduction of inferiors is the hard limit, so I guess you are
> hitting that limit, no?

Another good question. I have hit it often enough in the past, but the
error message I got looked different. But then, I guess the error
message for illicit time travel is not part of the documented behavior
of Guix.

Cheers,
  Konrad.

Re: Time travel accident

[Konrad Hinsen]

Konrad Hinsen  writes:

>building profile with 1 package...
>;;; WARNING: loading compiled file 
> /gnu/store/41zsnwsk02549kqb5njd3fadgnmkzww8-guix-module-union/lib/guile/3.0/site-ccache/guix/ui.go
>  failed:
>;;; In procedure load-thunk-from-memory: incompatible bytecode kind

I tried to track down why this file remains in the store even after
clearing the "inferiors" cache and doing a gc. Tracking the chain of
referrers, I find:

  /gnu/store/41zsnwsk02549kqb5njd3fadgnmkzww8-guix-module-union
  –> /gnu/store/7r72vknkpnpgp143ckh5kfbg5zan3xsp-guix-command
  –> /gnu/store/cwqaas3mwlx2rhc0ckzmqvygmy7n2s7k-guix-ce35dc8
  –> /gnu/store/2h03nwj2r2jywjk5sk7sxn2hgm979v2m-profile

This last item has two referrers:

 1. /gnu/store/9gfmn1yra7rzavxb9wppqi4lpdvqid8c-inferior-script.scm
 2. itself!

The only items I have encountered so far that have themselves as
referrers are gc roots. But this profile item is not in the list of gc
roots.

The inferior-script item has no referrers at all. It's not a gc root
either. Nevertheless, it is on the output of "guix gc --list-live".
According to my understanding of the store, that shouldn't be possible!

Cheers,
  Konrad.

Re: Time travel accident

[Konrad Hinsen]

Julien Lepiller  writes:

> If you're able to find the derivation, you could set your system time
> and guix build /gnu/store/that.drv. this should not require network at
> all, so guix shouldn't complain.

Sounds good. The error message contains the path to the derivation,
so... let's just do it!

A few minutes later: I can't do anything at all with time-machine on
that commit any more. Maybe my attempt to build with a modified system
time broke something in the store?

   $ guix time-machine --commit=ce35dc84a10b05dc891bfae03f613b907337945e --
shell --pure python  -- python3
...
   building 
/gnu/store/6126jqfx0rk9cccm518f53li2nj1q1h0-inferior-script.scm.drv...
   building package cache...
   building profile with 1 package...
   ;;; WARNING: loading compiled file 
/gnu/store/41zsnwsk02549kqb5njd3fadgnmkzww8-guix-module-union/lib/guile/3.0/site-ccache/guix/ui.go
 failed:
   ;;; In procedure load-thunk-from-memory: incompatible bytecode kind
...
   ;;; WARNING: loading compiled file 
/gnu/store/41zsnwsk02549kqb5njd3fadgnmkzww8-guix-module-union/lib/guile/3.0/site-ccache/guix/licenses.go
 failed:
   ;;; In procedure load-thunk-from-memory: incompatible bytecode kind
   WARNING: Use of `load' in declarative module (guix ui).  Add #:declarative? 
#f to your define-module invocation.
   Backtrace:
  5 (primitive-load 
"/home/hinsen/.cache/guix/inferiors/wisrwo5p2aq7o6r…")
   In ice-9/eval.scm:
   619:8  4 (_ #(#(#) 
"/home/hinsen/.cache…" …))
   619:8  3 (_ #(#(#)))
   159:9  2 (_ #(#(#)))
  223:20  1 (proc #(#(#)))
   In unknown file:
  0 (%resolve-variable (7 . _IOLBF) #)

   ERROR: In procedure %resolve-variable:
   _IOLBF: unbound variable

In an attempt to clean up, I deleted ~/.cache/guix/inferiors, and then
did a "guix gc". But that makes no difference.

Time travelers live dangerously!

Cheers,
  Konrad.

Re: Time travel accident

[Konrad Hinsen]

Hi Josselin and Julien,

Thanks to both of you for your suggestions!

Josselin Poiret  writes:

> We have --without-tests=package already, see --help-transform for all
> available package transformations.  The one annoying thing is that
> disabling tests will change the derivation and you thus will not recover
> the same store item (it might be bit-for-bit equivalent, but its path
> will not be the same), preventing you from using substitutes either.

For my case, that sounds OK. There are no substitutes for that
three-year old commit any more, so I am building everything. And I don't
care about bit-for-bit equivalence, I just want to run Python 3.8.2.

> Though, I'm not sure it will help you here because openssl is built as
> part of the `guix time-machine`'s build process, which afaik cannot be
> transformed.

Ahhh... there's the bad news.

> By the way, we can also "fix the past" by using guix/quirks.scm.  Since

Oohhh... There's always one more surprise in Guix!


Julien Lepiller  writes:

> Changing your system date should let it build.

Interesting idea! I tried, but it doesn't work: Guix itself complains
about a certificate failure if I set the clock three years back.

Maybe "guix time-machine" should have an option for setting the clock to
the commit timestamp just for the build process...

Cheers,
  Konrad.

Time travel accident

[Konrad Hinsen]

Hi everyone,

For doing some experiments with Python 3.8.2, I tried to use the commit
that introduced that version into Guix;

  guix time-machine --commit=ce35dc84a10b05dc891bfae03f613b907337945e \
   -- shell --pure python \
   -- python3

That fails, due to a build failure in OpenSSL:

   building /gnu/store/nh63q12x95irxyqzls0sfalf8ih5qass-openssl-1.1.1d.drv...
   | 'check' phasebuilder for 
`/gnu/store/nh63q12x95irxyqzls0sfalf8ih5qass-openssl-1.1.1d.drv' failed with 
exit code 1
   build of /gnu/store/nh63q12x95irxyqzls0sfalf8ih5qass-openssl-1.1.1d.drv 
failed
   View build log at 
'/var/log/guix/drvs/nh/63q12x95irxyqzls0sfalf8ih5qass-openssl-1.1.1d.drv.bz2'.

Inspecting the build log shows that the cause is a failing test:

   Test Summary Report
   ---
   ../test/recipes/80-test_ssl_new.t(Wstat: 256 Tests: 29 
Failed: 1)

That problem has been reported elsewhere and identified as caused by an
expired certificate:

   https://github.com/openssl/openssl/issues/18441


I guess there is nothing we can do retroactively to fix this, but can we
do something to prevent such issues in the future?

One idea is to allow disabling tests at the command line. I'd then run
"guix build" for that specific package with tests disabled, and
continue. That should be doable with a suitable package transformation.

Cheers,
  Konrad

Re: Using Guix inside a Guix container

[Konrad Hinsen]

Ludovic Courtès  writes:

> That’s an interesting use case!  I guess we have a hard-enough time
> getting the message through regarding the environment of tasks that we
> didn’t really consider the environment of the “driver”.

It takes something messy such as Snakemake to illustrate the importance
of this.

> (Well, in a way, GWL and Guix-Jupyter sidestep the issue by integrating
> the mechanism to declare task environments.)

OK, let's pass a law that workflows must be written in Guile or in a
language implemented in Guile ;-)

> I’d be inclined to add a new ‘-W’ (say) option to (1) share the whole
> store, and (2) share the daemon socket.  That would be the documented
> way to create a container with support for nested containers.

Sounds good to me!

> That one’s interesting.  Reported here:
>
>   https://issues.guix.gnu.org/61690
>
> At least there’s a workaround: using ‘-CN’ in the nested container.

That's already good to know.

Thanks,
  Konrad.

Re: Using Guix inside a Guix container

[Konrad Hinsen]

Hi Simon,

> Which part of Guix do you need inside the containerized shell that you
> cannot do outside?

That's not the right question. There's always a way to do what I want to
do outside. But that may be very inconvenient.

> Considering your use-case with Snakemake, what I am doing is to wrap
> each rule with one containerized Guix shell which controls the
> permissions, rule by rule; or a big containerized shell:
>
> guix shell -C -m manifest.scm --expose=…

Nice example. I do the same: "guix shell" in every rule. Then I add
stuff to my Snakefile, which is a Python script after all. For example,
I import pandas to read a data frame from which I construct my workflow.
Now I am at the point where I'd like to run snakemake itself in a
container, to manage the dependencies of my Snakefile. In fact, given
that I have workflows that depend on specific Snakemake versions, I'd
really like to run Snakemake in a container all the time, even without
additional dependencies.

Without nested containers, I have to go through all the rules, collect
the packages from their manifest files (or command line), and add them
to the container in which I run the whole workflow. Possible, but not
convenient.

Another example: I run command-line programs from my Pharo image, and I
have developed the habit of doing this always through Guix. The
advantage is that my Pharo code becomes portable: it depends on Guix,
but not on my profile.

But if I want, one day, to move on to a full Guix system, I have to run
Pharo in a container with LFS simulation. And then all my command line
shell-outs will break.

Both examples are about composing tools freely, without worrying if they
use Guix internally or now.

Cheers,
  Konrad

Re: Using Guix inside a Guix container

[Konrad Hinsen]

James Thomas  writes:

> Makes sense to me, because the guix package definition was only updated
> with the new release (see 'guix edit guix'). Maybe you can try:

Indeed. That's at least an explanation, though I still don't see *why*
it is done this way. In fact, the role of the package "guix" isn't
really clear to me.\

>   guix shell -C guix --with-version=guix=VERSION -- guix describe
>
> Or --with-commit etc.

That's an interesting idea! I could also define my own package (in a
file) called "guix-current" or something like that, which inherits from
"guix" and updates the commit.


Jim  writes:

> Or even:
>
>   guix shell -C -p /var/guix/profiles/per-user/.../current-guix -- guix
> describe

That would create a container from the Guix profile. Not sure if this
works at all, but it's very probably not what I want because I have to
add other packages to my container, to get some real work done :-)

Thanks to both of you for your suggestions!

Cheers,
  Konrad

Re: License of “10 years of stories behind Guix”

[Konrad Hinsen]

Dear lawyers of the world,

I agree for my contribution to the blog post

https://guix.gnu.org/en/blog/2022/10-years-of-stories-behind-guix/

to be published under CC-BY-SA 4.0 and GFDL version 1.3 or later (or
whatever other open license the maintainers of the blog might prefer in
the future).

Konrad.

Using Guix inside a Guix container

[Konrad Hinsen]

Hi Guix,

I have been playing with nested Guix containers recently, with some
suprising findings, and I am wondering if what I am doing is considered
officially supported or not.

First: why? My use case is scientific workflows, for example using
snakemake. I want to run my workflows in Guix containers, for
reproducibility plus other reasons. But my workflows run other programs
in their tasks (basically just "shelling out"), and those tasks may use
their own Guix containers.

Superficially, this works fine if I add the "guix" package to my "outer"
container and expose the store plus the daemon's socket:

guix shell -C guix \
  --expose=/var/guix/daemon-socket/socket \
  --expose=/gnu/store \
  -- \
  guix shell -C coreutils -- ls /

But now for the first surprise:

   $ guix describe
   Generation 35janv. 19 2023 12:34:57  (current)
 guix 8221cb6
   repository URL: https://git.savannah.gnu.org/git/guix.git
   branch: master
   commit: 8221cb6d2ae5624829bf514d25ae234c073e35d5

   $ guix shell -C guix -- guix describe
 guix 9fe5b49
   repository URL: https://git.savannah.gnu.org/git/guix.git
   branch: master
   commit: 9fe5b490df83ff32e2e0a604bf636eca48b9e240

The Guix in my container is an older one, apparently the 1.4.0 release.
Why? Can I change this?

My first attempt was time-machine:

guix shell -C -N guix nss-certs \
  --expose=/var/guix/daemon-socket/socket \
  --expose=/gnu/store \
  -- \
  guix time-machine -C channels.scm -- describe

   Updating channel 'guix' from Git repository at 
'https://git.savannah.gnu.org/git/guix.git'...
   Authenticating channel 'guix', commits 9edb3f6 to 8221cb6 (331 new 
commits)...
   Computing Guix derivation for 'x86_64-linux'... |
 guix 8221cb6
   repository URL: https://git.savannah.gnu.org/git/guix.git
   branch: master
   commit: 8221cb6d2ae5624829bf514d25ae234c073e35d5

Great! Except that every time I run this command, it does the channel
update from scratch, so it's prohibitively slow. Sharing
${HOME}/.cache/guix seems to fix that. So... finally...

guix shell -C -N guix nss-certs \
  --expose=/var/guix/daemon-socket/socket \
  --expose=/gnu/store \
  --share=${HOME}/.cache/guix \
  -- \
  guix time-machine -C channels.scm \
  -- \
  shell -C coreutils \
  -- \
  ls /

 guix shell: error: mount: mount "none" on
 "/tmp/guix-directory.vpOEDC/sys": Operation not permitted

Now I am lost. It doesn't matter which command I put on the last line,
it's creating a container via time-machine running in another container
that leads to the error.

Any ideas?

Cheers,
  Konrad.

Re: Creating a container from a container

[Konrad Hinsen]

Konrad Hinsen  writes:

> In case it matters, I use -N for the outer container (in which I
> use wget to fetch files), but not for the inner containers (the ones
> whose creation causes the error).

It does seem to matter... if I add -N for the inner container as well,
the error disappears. Which provides a workaround, but I'd really like
to understand what is going on here!

Cheers,
  Konrad

Creating a container from a container

[Konrad Hinsen]

Hi everyone,

I am trying to run "guix shell -C" inside another container created with
"guix shell -C". I figured out that I need to take care of three
aspects:

 - Include the package "guix" in the outer container.

 - --expose=/gnu/store such that guix-in-the-container can access the
   store

 - --expose=/var/guix/daemon-socket/socket such that
guix-in-the-container can connect to the daemon

With those precautions, my setup works some of the time but in other
conditions (which I haven't yet identified) I get the error message

  guix shell: error: mount: mount "none" on "/tmp/guix-directory.K4MoFR/sys": 
Operation not permitted

when I try to run "guix shell" inside the outer container. Browsing the
Guix source code, I found the probable location where this happens:
in (gnu build linux-container), the procedure run-container calls

   (mount-file-systems root mounts
  #:mount-/proc? (memq 'pid namespaces)
  #:mount-/sys?  (memq 'net namespaces))

which in turn does

   ;; A sysfs mount requires the user to have the CAP_SYS_ADMIN capability in
   ;; the current network namespace.
   (when mount-/sys?
 (mount* "none" (scope "/sys") "sysfs"
 (logior MS_NOEXEC MS_NOSUID MS_NODEV MS_RDONLY)))

and that looks suspect in view of my error message. But I have no idea
why this happens, nor what that "network namespace" is or why it is
needed. In case it matters, I use -N for the outer container (in which I
use wget to fetch files), but not for the inner containers (the ones
whose creation causes the error).

Does anybody have an idea of the cause of the error, and ideally a
solution or workaround?

Finally, some background as to why I am trying to do this: my outer
container runs a workflow (using good old "make"), which in turn
contains rules that create containers as well. I cannot run the workflow
plus all of its rules in a single container because different rules require
conflicting packages.

Thanks in advance,
  Konrad

Re: A real-life test of long-term reproducibility

[Konrad Hinsen]

Hi Ludo and Simon,

> Sorry I replied to quickly; it should have been
> ‘--disable-authentication’.
>
> Why?  Because here you’re trying to travel to a commit that’s not part
> of the “authenticated history”—that is, it’s not a descendant of the
> “introductory commit”, which is the first commit in history starting
> from which the “authentication invariant” holds.
>
> Hopefully this is clearer; maybe?  :-)

It's perfectly clear. I just wonder how I would find the solution
starting from the error message.

zimoun  writes:

>> And explain what it's about. I don't consider myself an absolute
>> beginner with Guix, but I don't understand what's going on here!
>
> Well, it is documented. ;-) The entry of ’time-machine’ [1] is the
> manual says:
...

Same remark: How would I get to that section of the manual from the
error message?

> I agree it could be improved.  Any suggestion? :-)

My impression is that the error message I quoted is very specific to
this situation, so it could actually say "Retry with
--disable-authentication if you are sure that there is no security
issue". Right?

Cheers,
  Konrad.

Re: A real-life test of long-term reproducibility

[Konrad Hinsen]

Hi Ludo and Simon,

> On Fri, 02 Sep 2022 at 15:17, Ludovic Courtès  wrote:
>
>> Here you would need ‘--allow-downgrades’.
>
> Maybe time-machine could advertise of this option?

And explain what it's about. I don't consider myself an absolute
beginner with Guix, but I don't understand what's going on here!

> Well aside, it can be indeed confusing that the version-1.x branch does
> not match the tag v1.x; I understand why.

OK, now I do a well... but should users need to understand the
developers' git workflow?

Ultimately, the issue here is that guix works on the implementation
level of git commit references, whereas the development process uses an
abstraction layer on top of it. Maybe "guix pull" and "guix
time-machine" should acquire some built-in intelligence when given a tag
instead of a hash? Issue a warning when an inappropriate tag is given,
and suggest a correction?

Cheers,
  Konrad.

Re: A real-life test of long-term reproducibility

[Konrad Hinsen]

Hi Simon,

> However, many things can be out of rail.  This claim about
> reproducibility over the time assumes:
>
>  1. compatibility of the Linux kernel
>  2. availability of all the source code
>  3. compatibility of the hardware
>
> Well, until now, nothing had been reported about #1.  But, we have
> examples of issues about #2 and #3.

Thanks for pointing this out. I have had #3 issues myself, with a
package (OpenBLAS or something related) failing to build on processors
later than the software release date. But I guess #2 is the main issue
for reviving old code. Software Heritage should settle that for the
future.

> All that said, Guix is the best and most advanced solution on the market
> for reproducible time-traveling. :-)  For most of the cases, it is
> awesome to just type “guix time-machine” and rebuild a complete
> computational environment exactly as it was 2 or 3 years ago.

Indeed!

> That’s because version-1.0.0 (48aa30ce73) is a branch and indeed not a
> descendant.
...
> What you want is tag v1.0.0 (6298c3ffd9).  Otherwise, you need the
> option ’--disable-authentication’. 

OK, thanks, I will try with that!

Cheers,
  Konrad.

Re: A real-life test of long-term reproducibility

[Konrad Hinsen]

Hi Ludo,

> We can go back to 1.0.0, and presumably to 0.15.0, but anything older
> than this is unknown territory.

Even 1.0.0 isn't obvious:

  $ guix time-machine --commit=version-1.0.0 -- environment guix
  guix time-machine: error: Git error: unable to parse OID - contains invalid 
characters

OK, so let's try the commit hash:

  $ guix time-machine --commit=48aa30ce73d45dc5f126f42f01e65f1be4a9b578 -- 
environment guix
  Updating channel 'guix' from Git repository at 
'https://git.savannah.gnu.org/git/guix.git'...
  Authenticating channel 'guix', commits 9edb3f6 to 48aa30c (6 new commits)...
  guix time-machine: error: commit
  48aa30ce73d45dc5f126f42f01e65f1be4a9b578 is not a descendant of
  introductory commit 9edb3f66fd807b096b48283debdcddccfea34bad

Cheers,
  Konrad.

Re: A real-life test of long-term reproducibility

[Konrad Hinsen]

Hi Ludo and Tim,

Thanks for your comments and experiments!


Ludovic Courtès  writes:

> That’s a commit from January 2018, which is a few months before the
> release of 0.15.0, the first release with proper ‘guix pull’ support:

Ouch. I wasn't aware that even "guix pull" happened later!

>> I don't understand what is going wrong here, but it may be related to
>> the fact that the commit I am trying to go back to is older than "guix
>> time-machine". If that's the explanation, it would help if Guix showed
>> some clear error message instead of crashing.
>
> It could check whether the target commit is in the closure of the
> v0.15.0 commit, but then that would give special treatment to the
> existing commit history.  Maybe that’s OK though?

I'd say yes, because the only problem is the existing commit history.
At least until we extend time-machine to actually change the past ;-)

> IIRC that was a bug in a Gnulib test (bundled in several GNU packages)
> that would hang on machines with maybe more than 4 cores.  (See commit
> acc2dab7f2f50c9169d6388007c770878eae4a9c for example.)  There might be
> hints on how to work around it in the mailing list archive…

Hardware dependencies...

> That said, it may be possible to build that Jan. 2018 Guix using an
> environment created from Guix 0.15.0 or 1.0.0; it’s likely to have the
> right versions of dependencies, and it should be reachable with
> time-machine.

I'll try with 1.0.0, that looks nicer than 0.15.0 ;-)

> Besides, I recently added ‘etc/time-travel-manifest.scm’ and added a
> corresponding jobset at 

Nice! Guix will be certified for time travel!


Timothy Sample  writes:

> It turns out it’s pretty easy, apart from having to boil the ocean.
> Here’s what I did.

Very interesting, thanks!

> I’m honestly shocked that it worked so well.  I wish I had a better way
> to keep track of where the sources came from.  For example, I’m curious
> how many came from the build farm or other fallback options.
>
> Overall, I give Guix two thumbs up!  Other than the Python 3 optimizer
> bit (which might be solvable), nothing substantive had to be changed to
> make this happen.

Indeed. And yet, from the point of view of a non-expert user, even the
slightest fix required is a show stopper.

> For best practices, I do have one suggestion.  The Guix package
> collection is not uniformly reproducible or archived.  The best thing
> you can do to ensure the long-term prospects of your projects is to
> actually check how much of the source code is archived and how many of
> the builds are reproducible.  There is no turn-key solution for this

Yes, that's a good idea, and I have done it for my most recent packages.
Time will tell if this is enough.

> Thanks Konrad for the interesting experiment.  While testing this out, I
> came to really appreciate how hackable Guix is.  Even if I couldn’t find
> Mesa 17.2.1, say, I could proceed with a similar version or try to build
> it with Git.  It’s not ideal to have to make changes, but it’s nice to
> know that Guix fails gracefully.

Definitely!

Cheers,
  Konrad.

A real-life test of long-term reproducibility

[Konrad Hinsen]

Hi everyone,

One of our claims is that Guix can rebuild code identically as long as
we have a machine with a Linux kernel and a POSIX filesystem. This week
I had an occasion to put this to a real-life test. So far it's a
failure. I can guess reasons for my failed attempts, but I don't think
they were unreasonable to try. So I'd like to document something that
works, to avoid others falling into the same traps. I just don't know
yet what the Right Way To Do It is!

The package I want to rebuild and use is "nmoldyn" from Guix commit
f250a868d8c687df08559682fa68fb4ea2a1ea69. That's the commit referenced
in my notes, obtained via "guix describe" in early 2018. I am pretty
sure it worked fine back then.

First attempt:

   $ guix time-machine --commit=f250a868d8c687df08559682fa68fb4ea2a1ea69 -- 
build nmoldyn
   Updating channel 'guix' from Git repository at 
'https://git.savannah.gnu.org/git/guix.git'...
   Backtrace:
   In guix/store.scm:
   672:3 19 (_)
   In ice-9/boot-9.scm:
 1752:10 18 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
   In guix/store.scm:
  659:37 17 (thunk)
   In guix/status.scm:
   815:4 16 (call-with-status-report _ _)
   In guix/store.scm:
  1298:8 15 (call-with-build-handler # …)
 2168:25 14 (run-with-store # _ # _ # 
_ # _)
   In guix/inferior.scm:
   903:8 13 (_ _)
   In guix/channels.scm:
   944:2 12 (_ _)
   891:2 11 (_ _)
   In ./guix/monads.scm:
   487:9 10 (_ _)
   In guix/store.scm:
  1996:8  9 (_ _)
   In guix/channels.scm:
  642:36  8 (_ #)
  703:11  7 (_)
   In ice-9/eval.scm:
   619:8  6 (_ #(#(#(#) "/gnu/store/…" 
…) …))
  626:19  5 (_ #(#(#(#) "/gnu/store/…" 
…) …))
   155:9  4 (_ #(#(#(#) "/gnu/store/…" 
…) …))
  223:20  3 (proc #(#(#(#) "/gnu/sto…" 
…) …))
   In unknown file:
  2 (%resolve-variable (7 . %guix-register-program) #)
   In ice-9/boot-9.scm:
 1685:16  1 (raise-exception _ #:continuable? _)
 1685:16  0 (raise-exception _ #:continuable? _)

   ice-9/boot-9.scm:1685:16: In procedure raise-exception:
   error: %guix-register-program: unbound variable

I don't understand what is going wrong here, but it may be related to
the fact that the commit I am trying to go back to is older than "guix
time-machine". If that's the explanation, it would help if Guix showed
some clear error message instead of crashing.

Next I tried checking out the source code for commit
f250a868d8c687df08559682fa68fb4ea2a1ea69, and building it from
source. This is a bit tricky because 2018 Guix cannot be built in
today's Guix build environment. For example, today we have Guile 3, but
back then we had Guile 2.2. So I need to do "guix environment guix" in
an older Guix, before the Guile 3 transition, but later than the
introduction of time-machine. I picked one somewhat at random:

   $ guix time-machine --commit=e2293cbbe0cd20ddeb932e6f5616565ab468c087
   -- environment –pure guix

Then I did "bootstrap", "configure –localstatedir=/var", "make
check". The latter shows 15 failures, some of which look important:

   FAIL: tests/builders.scm
   FAIL: tests/derivations.scm
   FAIL: tests/packages.scm
   FAIL: tests/guix-environment.sh
   FAIL: tests/guix-daemon.sh

And indeed I cannot build much with my compiled guix:

   $ ./pre-inst-env guix build nmoldyn

hangs after a while, running a binary called "test-lock" for hours.

Given the time lapse, I suppose there have been incompatible changes in
the build daemon, making the old Guix incompatible with the rather
recent build daemon running on my machine. But is there a way around
this, other than installing an old Guix in a fully isolated VM?

And if installing the old Guix in a VM is the only solution, what would
be the best way to do that? I can't think of much else than starting
from another distribution (e.g. Debian) and following the installation
instructions. That's already a lot of work, but it's made worse by the
installation instructions being hidden inside the manual of that old
commit, which I cannot easily consult.

I'd be grateful for any suggestions!

Cheers,
  Konrad

Re: Investigating a reproducibility failure

[Konrad Hinsen]

Hi Simon,

> We are far from OpenBLAS. :-)

That's fine with me. The more distance between me and OpenBLAS, the
happier I am ;-)

> On Wed, 16 Feb 2022 at 14:04, Konrad Hinsen  
> wrote:
>
>> Making scientific computations bit-for-bit reproducible is the moral
>> equivalent of keeping a detailed lab notebook: doing your best to tell
>> others exactly what you did.
>
> A detailed lab notebook implies transparency and full control of
> variability, not bit-for-bit reproducibility.

That's why I said "moral" equivalent. Computations are different from
experiments. Typical mistakes are different, and technical possibilities
are different.

1. You can't have the equivalent of bit-for-bit reproducibility with
   experiment. You can with computers, and with good tool support
   (Guix!)  it can become a routine task that takes little
   effort. So... why *not* do it?

2. A computation involves many more details than any typical experiment.
   Just writing down what you did is *not* enough for documenting a
   computation, as experience has shown. So you need more than the
   lab notebook. If your computation is bit-for-bit reproducible, you
   know that you have documented every last detail. Inversely, if you
   cannot reproduce to the bit level, you know that *something* is out
   of your control.

In the end, my argument is more pragmatic than philosophical. If
bit-for-bit reproducibility is (1) useful for resolving issues in the
future, and (2) cheap to get with good tool support, then we should
go for it.

The main reason why people argue against it is lack of tool support in
their work environments. They conclude that it's a difficult goal to
achieve, and then start to reason that it's not strictly necessary for
the scientific method. Which is true. But... it's still very useful.

>> And that's the role of bit-for-bit reproducibility.
>
> From my understanding, the validation of a reproduction depends on
> trust: what is the confidence about this or that?  Well, bit-for-bit
> reproducibility is one criteria for establishing such trust.  However,
> IMHO, such criteria is not the unique one, and defeating it can be
> compensated by other criteria used by many experimental sciences.

Definitely. But in many cases, bit-for-bit reproducibility is the
cheapest way to build trust, given good tool support. In other cases,
e.g. HPC or exotic hardware, it's expensive, and then you look for
something else.

Cheers,
  Konrad.

Re: Investigating a reproducibility failure

[Konrad Hinsen]

Hi Bengt and Simon,

zimoun  writes:

> Note that some people are calling for bit-to-bit scientific
> reproduction.  I am not.  Because the meaning of “same” or “equal”

I am. Not as a goal in itself, because in the larger scientific context
it's robust replicability that matters, not bit-for-bit re-execution.
And yet, the latter matters for two reasons :

 - It's verifiable automatically, making it cheap and fast to check.
   No need to bother an expert for a qualified opinion.

 - If you hit a case of non-replicability (scientifically relevant
   differences in two computations that everybody expects to yield
   equivalent results), then it is nearly impossible to investigate
   if the individual computations are not bit-for-bit reproducible.

Making scientific computations bit-for-bit reproducible is the moral
equivalent of keeping a detailed lab notebook: doing your best to tell
others exactly what you did.

> conclusions hold.  Again, transparency and full control of the
> variability are fundamental here.  How to argue if they are not
> satisfied?

Exactly, that's very similar to my second point.

Or, in Bengt's formulation:

> The details of Fortran version or Julia/Clang or guile
> pedigree only really come into play for forensics looking
> for where the abstract was implemented differently.

When the forensics are called in, then...

> Thus far, "show me the code" is the usual way to ask someone
> what they did, and guix makes is possible to answer in great
> detail.

... "show me the code" is not sufficient. You must also be sure that the
code you look at is really the code that was run. And that's the role of
bit-for-bit reproducibility.

Cheers,
  Konrad.

Re: Investigating a reproducibility failure

[Konrad Hinsen]

Hi Ludo,

> Konrad Hinsen  skribis:
>
>> To see the failure, do
>>
>>guix time-machine \
>> --commit=7357b3d7a52eb5db1674012c50d308d792741c48 \
>> -- build openblas
>
> For the record, there’s still a substitute available for this one:

...

> That doesn’t solve the fact that OpenBLAS compilation is not
> reproducible, as zimoun noted¹, and we need to fix it, but at least this
> colleague of yours should have been able to fetch substitutes, no?

Good point. If I try to use it now, it works, fetching the substitute.
Back when I started investigating this, on the same machine, Guix tried
to build locally.  So I guess there was some ephemeral problem with
accessing the substitute server.

Cheers,
  Konrad

Re: License of your contributions to the blog at guix.gnu.org

[Konrad Hinsen]

Ludovic Courtès  writes:

> With a few exceptions, these articles do not have a clear license, which
> we would like to fix.  We propose to dual-license all the articles under
> CC-BY-SA 4.0 and GFDL version 1.3 or later, with no Invariant Sections,
> no Front-Cover Texts, and no Back-Cover Texts.
>
> Do you agree with the proposed licensing terms for your contributions to
> the blog?

I agree.

Konrad

Re: Investigating a reproducibility failure

[Konrad Hinsen]

Hi Ricardo and Simon,

Ricardo Wurmus  writes:

> The case of OpenBLAS is an anomaly in that this mechanism seems to
> produce different binaries dependent on where it is built.  When I first

Thanks a lot for those explanations, I hadn't realized how peculiar
OpenBLAS is!

> Your problem is that the OpenBLAS build system doesn’t recognize your
> modern CPU.  Ideally, it wouldn’t need to know anything about the
> build-time CPU to build all the different code paths for different CPU
> features.  The only way around this — retroactively — is to pretend to
> have an older CPU, e.g. by using qemu.

So all we need is a "QEMU build system" in Guix, just for OpenBLAS ;-)

> The new “--tune” feature is supposed to take care of cases like this.

Right, I remember Ludo's blog post about this.


zimoun  writes:

> Somehow, “recent” processors cannot build old versions.

That's a whole new level of planned obsolescence!

Cheers,
  Konrad.

Re: Investigating a reproducibility failure

[Konrad Hinsen]

Hi Ricardo and Simon,

Thanks for your insight! I didn't even know about lscpu. The output for
my laptop is shown below. I tried building on a virtual machine, and
that works fine.

> CPU detection is a bottomless can of worms.

That sounds very credible. But what can we do about this?

There is obviously a trade-off between reproducibility and performance
here. Can we support both, in a way that users can understand and manage?

The OpenBlas package in Guix is (or at least was, back then) written for
performance. Can I, as a user, ask for a reproducible version? That
could either be a generic version for any x86 architecture (preferably),
or one that always builds for a given sub-architecture and then fails at
runtime if the CPU doesn't match.

Next: can I, as a user of dependent code, ask for reproducible versions
of all my dependencies? In my case, I was packaging Python code that
calls OpenBlas via NumPy. Many people in that situation don't even know
what OpenBlas is. I did know, but wasn't aware of the build-time CPU
detection.

There is of course the issue that we can never be sure if a build will
be reproducible in the future. But we can at least take care of the
cases where the packager is aware of non-reproducibility issues, and
make them transparent and manageable.

Cheers,
  Konrad

--8<---cut here---start->8---
$ lscpu
Architecture:x86_64
CPU op-mode(s):  32-bit, 64-bit
Byte Order:  Little Endian
Address sizes:   39 bits physical, 48 bits virtual
CPU(s):  8
On-line CPU(s) list: 0-7
Thread(s) per core:  2
Core(s) per socket:  4
Socket(s):   1
NUMA node(s):1
Vendor ID:   GenuineIntel
CPU family:  6
Model:   140
Model name:  11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz
Stepping:1
CPU MHz: 1800.000
CPU max MHz: 4800,
CPU min MHz: 400,
BogoMIPS:3609.60
Virtualization:  VT-x
L1d cache:   192 KiB
L1i cache:   128 KiB
L2 cache:5 MiB
L3 cache:12 MiB
NUMA node0 CPU(s):   0-7
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf:  Not affected
Vulnerability Mds:   Not affected
Vulnerability Meltdown:  Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled 
via p
 rctl and seccomp
Vulnerability Spectre v1:Mitigation; usercopy/swapgs barriers and 
__user poi
 nter sanitization
Vulnerability Spectre v2:Mitigation; Enhanced IBRS, IBPB conditional, 
RSB fi
 lling
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort:   Not affected
Flags:   fpu vme de pse tsc msr pae mce cx8 apic sep 
mtrr pg
 e mca cmov pat pse36 clflush dts acpi mmx fxsr 
sse 
 sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm 
cons
 tant_tsc art arch_perfmon pebs bts rep_good 
nopl xt
 opology nonstop_tsc cpuid aperfmperf 
tsc_known_freq
  pni pclmulqdq dtes64 monitor ds_cpl vmx smx 
est tm
 2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 
sse4_2 
 x2apic movbe popcnt tsc_deadline_timer aes 
xsave av
 x f16c rdrand lahf_lm abm 3dnowprefetch 
cpuid_fault
  epb cat_l2 invpcid_single cdp_l2 ssbd ibrs 
ibpb st
 ibp ibrs_enhanced tpr_shadow vnmi flexpriority 
ept 
 vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep 
bmi2
  erms invpcid rdt_a avx512f avx512dq rdseed 
adx sma
 p avx512ifma clflushopt clwb intel_pt avx512cd 
sha_
 ni avx512bw avx512vl xsaveopt xsavec xgetbv1 
xsaves
  split_lock_detect dtherm ida arat pln pts hwp 
hwp_
 notify hwp_act_window hwp_epp hwp_pkg_req 
avx512vbm
 i umip pku ospke avx512_vbmi2 gfni vaes 
vpclmulqdq 
 avx512_vnni avx512_bitalg tme avx512_vpopcntdq 
rdpi
 d movdiri movdir64b fsrm avx512_vp2intersect 
md_cle
 ar flush_l1d arch_capabilities
--8<---cut here---end--->8---

Investigating a reproducibility failure

[Konrad Hinsen]

Hi everyone,

Two years ago, I published a supposedly reproducible computation,
explaining how to re-run it at any time using Guix (it's at
https://github.com/khinsen/rescience-ten-year-challenge-paper-3/). Yesterday,
I got an e-mail from someone who tried, and failed. I tried myself, and
failed as well. But I don't understand what's going on.

To see the failure, do

   guix time-machine \
--commit=7357b3d7a52eb5db1674012c50d308d792741c48 \
-- build openblas

The build log is attached, the first error is

   getarch_2nd.c: In function ‘main’:
   getarch_2nd.c:12:35: error: ‘SGEMM_DEFAULT_UNROLL_M’ undeclared (first use 
in this function); did you mean ‘XGEMM_DEFAULT_UNROLL_M’?
printf("SGEMM_UNROLL_M=%d\n", SGEMM_DEFAULT_UNROLL_M);
  ^~
  XGEMM_DEFAULT_UNROLL_M

What makes this complicated is the DYNAMIC_ARCH feature of openblas that
Guix uses on X86 architectures. I don't know the details of who this
should work and why it could fail. In particular, I don't know if the
source code file getarch₂nd is supposed to be compiled at all if all
goes well.

I doubt we can do anything to fix the past, but I would like to
understand what exactly went wrong here so we can make sure we do better
in the future.

Cheers,
  Konrad.

Re: Investigating a reproducibility failure

[Konrad Hinsen]

Konrad Hinsen  writes:

> To see the failure, do
>
>guix time-machine \
> --commit=7357b3d7a52eb5db1674012c50d308d792741c48 \
> -- build openblas
>
> The build log is attached, the first error is

Oops... Two mistakes ! First, I forgot the attachment, so here it comes,
Second, I didn't quote the right commit. The failure happens with

guix time-machine \
 --commit=87e7faa2ae641d8302efc8b90f1e45f43f67f6da \
 -- build openblas

Cheers,
  Konrad.



6df92lhfz4vccgn5v2z0rc092bhz89-openblas-0.3.a.drv.bz2
Description: Binary data

"guix package" failed with "integer expected from stream"

[Konrad Hinsen]

Hi Guix,

Updating my packages after today's "guix pull" fails with the error
message

guix package: error: integer expected from stream

There are two similar-looking bugs, both closed, suggesting that the
problem has been fixed. I don't understand all the details, but I can
confirm that it's related to substitutes. Disabling substitutes makes
the process go on fine, but I can't really build all my packages myself
in a reasonable length of time.

Should I open a new bug report for this? Or is this problem really fixed
and I just need to do something to profit from the fix?

Cheers,
  Konrad.

Re: Any go expert willing to help with updating IPFS?

[Konrad Hinsen]

"Leo Famulari"  writes:

> It's likely that you need to use a newer version of Go.

Thanks, that did it! With go-1.17 it compiles fine.

Cheers,
  Konrad

Re: Any go expert willing to help with updating IPFS?

[Konrad Hinsen]

Konrad Hinsen  writes:

> My current package definition is attached.

Well, now it is:

(define-public go-ipfs
  (package
(name "go-ipfs")
(version "0.11.0")
(source
 (origin
   (method url-fetch/tarbomb)
   (uri (string-append
 "https://dist.ipfs.io/go-ipfs/v; version
 "/go-ipfs-source.tar.gz"))
   (sha256
(base32 "13pmj83hwpz6mk7x52qn0cjnfqxqw2qri3r0k4b270w3bafcccwm"))
   (file-name (string-append name "-" version "-source"
(build-system go-build-system)
(arguments
 '(#:unpack-path "github.com/ipfs/go-ipfs"
   #:import-path "github.com/ipfs/go-ipfs/cmd/ipfs"
   #:phases (modify-phases %standard-phases
  (add-before 'reset-gzip-timestamps 'make-files-writable
(lambda* (#:key outputs #:allow-other-keys)
  ;; Make sure .gz files are writable so that the
  ;; 'reset-gzip-timestamps' phase can do its work.
  (let ((out (assoc-ref outputs "out")))
(for-each make-file-writable
  (find-files out "\\.gz$"))
#t))
(native-inputs
 `(("python" ,python-minimal-wrapper)
   ("zsh" ,zsh)))
(home-page "https://ipfs.io;)
(synopsis "Go implementation of IPFS, a peer-to-peer hypermedia protocol")
(description "IPFS is a global, versioned, peer-to-peer file system.  It
combines good ideas from Git, BitTorrent, Kademlia, SFS, and the Web.  It is
like a single bittorrent swarm, exchanging git objects.  IPFS provides an
interface as simple as the HTTP web, but with permanence built in.  You can
also mount the world at @code{/ipfs}.")
(license license:expat)))

Any go expert willing to help with updating IPFS?

[Konrad Hinsen]

Hi Guix,

the version of IPFS in Guix is 0.8, and in view of the important changes
introduced in 0.10, that's obsolete by now. Which is why I am trying to
update to 0.11.

My current package definition is attached. It fails, but provides no
clear hint (to the Go ignorant that I am) about what is actually going
wrong:

   starting phase `build'
   
src/github.com/ipfs/go-ipfs/vendor/github.com/lucas-clemente/quic-go/internal/qerr/error_codes.go:6:2:
 build constraints exclude all Go files in 
/tmp/guix-build-go-ipfs-0.11.0.drv-0/src/github.com/ipfs/go-ipfs/vendor/github.com/lucas-clemente/quic-go/internal/qtls
   Building 'github.com/ipfs/go-ipfs/cmd/ipfs' failed.

The build log then shows the results of `go env`, where nothing looks
suspicious to me. Then another cryptic line:

   command "go" "install" "-v" "-x" "-ldflags=-s -w" 
"github.com/ipfs/go-ipfs/cmd/ipfs" failed with status 1

Does anyone have an idea for debugging this issue?

Cheers,
  Konrad.

Re: [Spam:]Re: “What’s in a package”

[Konrad Hinsen]

Katherine Cox-Buday  writes:

> As we've seen these past years with COVID-19 and the world's supply
> chains, efficiency has some kind of inverse relationship with
> robustness. If you go too far down the path of efficiency, you are not
> very flexible, and you're building sand castles.

That's exactly what I have seen happening in scientific software for a
while :

  https://hal.archives-ouvertes.fr/hal-02117588

> It's for this reason I appreciate having "robust" software underneath
> my sand castle. At least I know only so much can crumble :)

100 % agreement!

> I want to be careful here in what I suggest. I think it is very
> important that Guix remain a bastion of robust software with very high
> standards. I don't want to see the PyPi PyTorch packages of the world

Me neither. My suggestion was for support in Guix the tool, not Guix the
software distribution. People can/should package their sand castles in
their private channels.

> So with your example: make it really easy to transform that PyPi
> package into a terrible Guix primitive of some kind, but don't let me
> commit it to Guix proper.

I trust our maintainer team to not let this happen.

> Maybe interactive software that introspects how a package
> is written and behaves at runtime (in a container?) and utilizes the
> homoiconicity of scheme to suggest modifications of the package, or
> next steps. E.g. expand the linter to suggest things like

That sounds interesting!

> Speaking of industry, I don't think we leverage software to build software 
> enough.

Definitely not.

> And by the way, none of those ideas would be possible if Guix weren't
> such a robust and sane ecosystem.

Exactly. We can discuss (and more) adding sloppy stuff on top of Guix,
but it wouldn't work the other way round.


"Jonathan McHugh"  writes:

> Your focus regarding a transition from exploratory to robust is
> important (though may have equal significance in the other
> direction?).

Not equal as I see it, but yes, it matters as well, for dragging a
stable package out int the open again for significant improvements.

> Would security experts have (understandable) criteria to prioritise
> choices for 'robust corridors' within an ecosystem of sourcefiles and
> encapsulated blobs?

I'd love to hear from security experts too!

Konrad.

Re: [Spam:]Re: “What’s in a package”

[Konrad Hinsen]

Hi Katherine and Ludo,

> I appreciate this post very much. Setting aside questions of freedom,

+1

> This is perhaps a rehash of the "worse is better"[2] conversation, but
> I often struggle with deciding whether to do things the "fast" way, or
> the "correct" way. I think when your path is clear, the correct way
> will get you farther, faster. But when you're doing experiments, or
> exploratory programming, being bogged down with the "correct" way of
> doing things (i.e. Guix packages) might take a lot of time for no

Exactly. Most software engineering tools situate themselves somewhere on
the "fast" vs. "robust" scale, and defend their position as the one and
only Good Thing. Guix is at the "robust" end of the scale in the
software management category. And that's what I want for most of the
software I use, i.e. everything I don't hack on myself. Which is why I
like Guix :-)

What is so far insufficiently supported by computing technology is the
necessary transition from "fast" to "robust". There are a few
exceptions, such as programming language with gradual typing. In most
situations, moving software from exploratory to robust involves a lot of
rewriting, often manually, with no tooling support.

> Bringing this back to Guix, and maybe the GNU philosophy, it has been
> very helpful for me to be able to leverage the flexibility of Guix to
> occasionally do things the "fast" way, perhaps by packaging a
> binary. Paradoxically, it has allowed me to stay within the Guix and
> free software ecosystem. In my opinion, flexibility is key to growing
> the ecosystem and community, and I would encourage Guix as a project
> to take every opportunity to give the user options.

+100 :-)

There is a lot we can improve here. Tutorials would be a good start.
Example: How do you package a binary in Guix? In particular, how do you
deal with binaries that have binary dependencies that they expect in
/lib etc.? A next step would be tool support: Grab whatever PyPI offers,
even if it's only binary wheels, and turn that into a Guix package.

Another aspect would be supporting software development moving from fast
to robust. Suppose I have software I compile by hand, or via a simple
Makefile, somewhere in my home directory. How do I go from there to (1)
a quick-and-dirty Guix package, then (2) a very basic publishable Guix
package and finally (3) a Guix package with tests and documentation?
The path should be supported by various tools, from automatic rewriting
to debugging. As an example, something I have wished for more than once
is the possibility to run the individual build steps of a Guix package
under my own account in my home directory, for debugging purposes.

Konrad
-- 
-
Konrad Hinsen
Centre de Biophysique Moléculaire, CNRS Orléans
Synchrotron Soleil - Division Expériences
Saint Aubin - BP 48
91192 Gif sur Yvette Cedex, France
Tel. +33-1 69 35 97 15
E-Mail: konrad DOT hinsen AT cnrs DOT fr
http://dirac.cnrs-orleans.fr/~hinsen/
ORCID: https://orcid.org/-0003-0330-9428
Twitter: @khinsen
-

Re: Removal of Python 2?

[Konrad Hinsen]

Hi Ryan,

> Python 2 is no longer supported, but the Tauthon project [1] is
> continuing support and backporting features from Python 3. We should

Didn't know about Tauthon, thanks for the pointer!

> consider packaging Tauthon in Guix and updating packages to depend on

Yes, that sounds like an interesting strategy to explore. One potential
obstacle is that most of the Python2-dependent package are out of
maintenance as well. But then, many of them don't need any maintenance.

If we introduce Tauthon, we should probably untie its dependents from
the Python (3) version, i.e. stop deriving them from Python 3 package
definitions. The latter will ever more evolve in an incompatible way.

Cheers,
  Konrad.

Re: Removal of Python 2?

[Konrad Hinsen]

Hi Ludo and Simon,

Ludovic Courtès  writes:

> What we could do is start removing ‘python2-’ packages, especially those
> with a non-negligible maintenance cost (numpy, scipy, matplotlib, etc.).
> We can move them to the Guix-Past channel if there’s interest.

Yes, that sounds like a good plan. With the long-term goal of moving
python2 itself to guix-past, if it ever ceases to be an important
dependency (which I am not yet convinced of).

> The attached script lists unused ‘python2-’ leaf packages:
...
> Are these good candidates for removal?

These and more: once they are removed, there will be a new set of leaf
packages.

It might be worth checking if there is a Python 3 equivalent before
removing a library. But in the majority of cases there is.


zimoun  writes:

> Starting with e13ab46c2052609c4ad792aeb304a752b50d034e, I remove the
> broken leaf Python2 packages.  I find them using "guix weather" and
> "guix refresh -l".  More are coming. ;-)

It certainly makes sense to remove broken packages, but for the rest I'd
say guix-past is a better destination than the trashcan.

Cheers,
  Konrad

Re: Removal of Python 2?

[Konrad Hinsen]

Hi Hartmut,

> Python 2 is dead, dead, dead like the parrot and end-of-prolonged life 
> as of more than 1 1/2 years. Anyhow, there might still be quite some 
> software not ported to Python 3 after 10 years. So I'm afraid we need to 
> keep Python 2.

At this time, more than 2500 packages still depend on Python 2, and
that's not just Python libraries. The list includes packages such as
TeXlive, LLVM, Rust, OpenJDK, Calibre, OCaml, qemu, etc.

For many of these packages, Python 2 is used only as part of the build
procedure. Which is run in a highly protective sandbox. So I don't think
we need to worry much about having Python 2 in Guix. To discourage
people from installing it by accident, we could make it a hidden
package.

Cheers,
  Konrad.

Re: Idea: a meta language for (language) build systems - npm, Racket, Rust cargo

[Konrad Hinsen]

Hi Pjotr,

> I have a feeling they won't be that interested ;).

Depends on who and when. My idea is not to send them a mail saying,
"look, we have a great idea, you should us it!". It's rather to open
issues on their build systems to suggest features and actually help with
the implementations. Good old proselytism ;-) It's a long-term strategy,
not a quick fix project.

> Industry appears to go the other way. There are several initiatives to
> speed up Rust through caching and storing compiled items in the
> cloud(?!).

Then it's time to intervene before it's too late ;-)

There is a clear momentum building up for reproducibility. Few people
argue nowadays that it doesn't matter. It's more like "would be nice,
but it's too difficult/complicated". That's where Guix can help: as a
demonstration of possibility. And once your builds are reproducible,
you can safely cache them in the cloud!

The next message to get across is the need for cross-language build
systems. The idea of one language, one platform doesn't work any more.
Real software systems need to mix languages, even runtimes. And it's not
just X + C any more either. Again, Guix serves as the proof of concept,
even for people who dislike some specific implementation aspect of Guix.

Cheers,
  Konrad.

Re: Idea: a meta language for (language) build systems - npm, Racket, Rust cargo

[Konrad Hinsen]

Hi Pjotr,

> Maybe this is a crazy idea:

Not crazy enough for me ;-)

Today's mess with language and platform specific build systems is indeed
a problem, and I have been thinking as well about how Guix could help
with this beyond what it already does. Guix' big achievement, beyond
being a practically useful tool, is demonstrating what it takes to have
a full system build system. Everyone's life would be easier if software
packaging were made more compatible with how Guix (and Nix of course)
work today. And one way to get the rest of the world move towards this
goal would be to help them: propose and implement changes to their build
systems that would make integration with Guix easier.

So...

> My idea is this: rather than generating 'imports' from native build
> packages for every target - which is kinda laborious in its way - we
> could translate an existing Rust/Node/Racket package into an
> intermediate simple package tar ball - call it a sexp-pack ;). This

How about pushing all the other package manager towards producing
sexp-packs, and helping them to get there?

Cheers,
  Konrad

Re: Finding the store path of a package

[Konrad Hinsen]

Hi Ludo,

> Why #:graft? #f?  Because if you enable graft, you’ll potentially have
> to build/download the thing, and that wouldn’t buy you anything because
> the set of file names is the same in the grafted package.

Four weeks later: this mostly works, but sometimes fails (by
downloading/building package) and it rarely fails completely
(error message). And I have no idea what is going on.

As an experiment, I ran the attached script via "guix repl". It requests
and displays the store paths for all non-hidden packages. Using
"guix gc" before and after, I found that it downloads/builds
more than 5 store entries with a total size of 7.5 GB.

One small but interesting example: ABCL

##
Store paths for abcl@1.8.0:
fetching path `/gnu/store/rz42ba0my9vrgbkjpkzr2drmnjk5ah50-python-3.8.2'...
Downloading 
https://ci.guix.gnu.org/nar/lzip/rz42ba0my9vrgbkjpkzr2drmnjk5ah50-python-3.8.2...
 python-3.8.2  12.0MiB1.8MiB/s 00:07 [##] 100.0%

fetching path 
`/gnu/store/bvd09gb8ka642jzgxd2lpqlpdp160gn0-python-wrapper-3.8.2'...
Downloading 
https://ci.guix.gnu.org/nar/lzip/bvd09gb8ka642jzgxd2lpqlpdp160gn0-python-wrapper-3.8.2...
 python-wrapper-3.8.2  347B   177KiB/s 00:00 [##] 100.0%

fetching path `/gnu/store/h8z924ip7ialjhd3sc2id5yh0jy7cj20-python-nose-1.3.7'...
Downloading 
https://ci.guix.gnu.org/nar/lzip/h8z924ip7ialjhd3sc2id5yh0jy7cj20-python-nose-1.3.7...
 python-nose-1.3.7  197KiB1.4MiB/s 00:00 [##] 100.0%

fetching path `/gnu/store/cshy0265w5ifh12v9dcmlrdd3cwb61db-mercurial-5.6.1'...
Downloading 
https://ci.guix.gnu.org/nar/lzip/cshy0265w5ifh12v9dcmlrdd3cwb61db-mercurial-5.6.1...
 mercurial-5.6.1  3.3MiB  3.1MiB/s 00:01 [##] 100.0%

((out . "/gnu/store/nz8xi3x7n3v2k4s4piiw7r8plvxam9qz-abcl-1.8.0"))
##

Why does computing the store path for a Java package require fetching
Mercurial (and thus Python)? The only link I see is icedtea-6, whose
build procedure apparently uses Mercurial to download the OpenJDK
sources. Probably this gets inherited by icedtea-8, which is used for
building ABCL. But... why does it take Mercurial to just compute the
store path??? If my understanding of Guix hashes is correct (which I am
beginning to doubt), it shouldn't require more than the hash of the
Mercurial package.

As an example for a hard failure, see sunxi-tools@1.4.2. Asking for
its store paths downloads plenty of stuff, before failing with:

  while setting up the build environment: a `armhf-linux' is required to build 
`/gnu/store/4blcfrrzc3yx6xjvw31mhnr1r9lpw4mj-Python-3.8.2.tar.xz.drv', but I am 
a `x86_64-linux'

That makes sense, of course, and I don't mind getting an error in such a
case, but I'd prefer it to fail without first downloading tons of stuff,
including the heavyweight gcc-7.5.0.

Cheers,
  Konrad.



display-store-paths.scm
Description: Binary data

Re: Finding the channel in which a package is defined

[Konrad Hinsen]

Hi everyone,

thanks for your suggestions. My award for the most useful one (to me)
goes to Mathieu :

> You can run something like:
>
> --8<---cut here---start->8---
> ,use (guix describe) (gnu packages) (gnu packages linux)
> (%package-module-path)
> (package-channels strace)
> --8<---cut here---end--->8---

Since the information is stored with all packages, it would be
reasonable for "guix show" to display it as well. I'll look at this and
propose a patch if I succeed.

It's a bit surprising at first sight that there are multiple channels. I
suspect that the list contains the channels of all inputs as well,
recursively, meaning that channel "guix" is present almost everywhere.

Cheers,
  Konrad.

Re: Finding the channel in which a package is defined

[Konrad Hinsen]

Tobias,

Thanks for your reply!

> Does the ‘location’ field of ‘guix show PACKAGE’ do what you want?

Taking coreutils as a test case, it displays:

   gnu/packages/base.scm:328:2

as a link pointing to:

   
/gnu/store/3qykwxq1mqlin3lrb93s3rzi1ah5xia8-guix-module-union/share/guile/site/3.0/gnu/packages/base.scm

and that is the same file that is opened with "guix edit". But it's a
copy of the input source file that is part of some channel, so the
provenance is lost.

Cheers,
  Konrad

Finding the channel in which a package is defined

[Konrad Hinsen]

Dear Guix experts,

Is there a simple way to find out in which channel a given package was
defined? I tried "guix edit" to see the source code, but it shows a file
from a "module-union" directory in the store.

Cheers,
  Konrad

Re: Finding the store path of a package

[Konrad Hinsen]

Hi Ludo,

> Here’s an example of how to do that:

Works fine, thanks!

> Why #:graft? #f?  Because if you enable graft, you’ll potentially have
> to build/download the thing, and that wouldn’t buy you anything because
> the set of file names is the same in the grafted package.

OK, so that's the secret, because that's the only difference with what
I tried before.

One day I'll figure out how grafts work, but that day is not today ;-)

> For #2, there have been discussions about building a service that would
> create such a database—a mapping from file names to packages.  It’s not
> possible to do with purely local knowledge because, by definition, you’d
> have to build/download every package.  I don’t think it has materialized
> yet, though.

That would certainly be the best solution, but in the meantime, I'll go
ahead with what is possible today. In practice, I expect most such
queries to succeed because there are only a few packages that contain
popular commands, and those are usually in the store.

Cheers,
  Konrad.

Re: Finding the store path of a package

[Konrad Hinsen]

Hi Simon,

> On a side note, Ricardo did recently some stuff as UI for packages,
>
>   
>   

Looks good! My project has a lot of overlap, except that it is very
intentionally not based on Web technology, but on a malleable platform
(http://gtoolkit.com/) in which you can also do data analysis and other
computation. It's more like Emacs than like a Web app.

> And on another side note, I would like to have the abilities to join or
> intersect graphs; be able to visualize “guix graph foo bar” using D3.js,
> or the intersection or the complementary of the intersection, etc.
> Something similar for packages as the Ludo’s proof of concept for
> services,
>
>

Nice! That kind of visualization is not my focus for now, but the people
who build the Glamorous Toolkit platform are very much into that and
there's a lot of support code for visually exploring software systems:

   https://gtoolkit.com/usecases/software-assessment/

Cheers,
  Konrad.

Re: Finding the store path of a package

[Konrad Hinsen]

Hi Ludo,

> Yes.  In the presence of grafts, run “guix build PKG”.  That always
> gives you the store file name of PKG, 100% reliable!

At the cost of a few hours of CPU time, in the worst case.

> I regularly do things like:
>
>   ls $(guix build PKG)/bin
>   find $(guix build PKG) -name …

What I am looking for is the equivalent of

   ls $(guix build PKG)

that fails in whatever way for packages that are not in the store, but
guarantees (1) not adding anything to the store and (2) response times
short enough for interactive user interfaces.

> If you want a variant that does that without building/downloading it,
> it’s also possible, though not as easily from the command line.

Guile is fine, no problem. But so far, I haven't found anything even at
the Guile level that respects my two conditions.

Background: I am working on a interactive UI for running reproducible
computations via Guix:

  https://github.com/khinsen/guix-gtoolkit/

I'd like to implement (1) browsing package contents ("what exactly do I
get by adding "core-utils" to my environment?") and (2) searching
packages by the files they contain ("which package do I have to add to
my environment to get the ls command?"). There will be a button for
explicitly building a package, but I don't want it to happen as a side
effect when doing operations that need to be fast.

Cheers,
  Konrad.

Re: Finding the store path of a package

[Konrad Hinsen]

Hi Simon,

>> does “guix build  -n” fit your use-case?
>
> Checking... yes! I hadn't even considered using "build" when building is
> exactly what I do not want to happen. But yes, it works just fine.

Not quite:

$ guix build -n zziplib
substitute: 
/gnu/store/pwcp239kjf7lnj5i4lkdzcfcxwcfyk72-bash-minimal-5.0.16/bin/bash: 
warning: setlocale: LC_ALL: cannot change locale (en_US.utf8)
24,6 MB would be downloaded:
   /gnu/store/9h9153akbd6g2520mcl2d0vgbvyfi5qm-openssl-1.1.1i
   /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31
   /gnu/store/s3dcqzwqaakv1yx37by9chksdbkgih17-glibc-2.31-static
   /gnu/store/fbn395nfpbp4d4fr6jsbmwcx6n10kg16-python-minimal-3.8.2
   /gnu/store/7hizrpdsqf6q3pjgzmi51r5vbzlijkw0-python-minimal-wrapper-3.8.2

My first impression is that "guix build -n" shows the store path of the
package only if all dependencies are in the store. Otherwise, it shows
the dependencies.

>> Well, ’package-output’ in (guix packages) is what you need, I guess.
>
> And that works just fine as well.

That one is worse because it even fetches/builds the dependencies:

$guix repl
GNU Guile 3.0.5
Copyright (C) 1995-2021 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guix-user)> (use-modules (guix packages) (guix store) (gnu packages 
compression))
scheme@(guix-user)> (with-store store (package-output store zziplib))
substitute: 
/gnu/store/pwcp239kjf7lnj5i4lkdzcfcxwcfyk72-bash-minimal-5.0.16/bin/bash: 
warning: setlocale: LC_ALL: cannot change locale (en_US.utf8)
fetching path `/gnu/store/9h9153akbd6g2520mcl2d0vgbvyfi5qm-openssl-1.1.1i'...
/gnu/store/pwcp239kjf7lnj5i4lkdzcfcxwcfyk72-bash-minimal-5.0.16/bin/bash: 
warning: setlocale: LC_ALL: cannot change locale (en_US.utf8)
Downloading 
https://ci.guix.gnu.org/nar/lzip/9h9153akbd6g2520mcl2d0vgbvyfi5qm-openssl-1.1.1i...
 openssl-1.1.1i  2.1MiB   3.2MiB/s 00:01 [##] 100.0%

fetching path 
`/gnu/store/fbn395nfpbp4d4fr6jsbmwcx6n10kg16-python-minimal-3.8.2'...
/gnu/store/pwcp239kjf7lnj5i4lkdzcfcxwcfyk72-bash-minimal-5.0.16/bin/bash: 
warning: setlocale: LC_ALL: cannot change locale (en_US.utf8)
Downloading 
https://ci.guix.gnu.org/nar/lzip/fbn395nfpbp4d4fr6jsbmwcx6n10kg16-python-minimal-3.8.2...
 python-minimal-3.8.2  11.9MiB2.5MiB/s 00:05 [##] 100.0%
...


After that, I can do

$ guix build -n zziplib
/gnu/store/fx0cdzzppd8jc09sianbq6gl1h7mxx3x-zziplib-0.13.72


Cheers,
  Konrad.

Re: Finding the store path of a package

[Konrad Hinsen]

Hi Simon,

> does “guix build  -n” fit your use-case?

Checking... yes! I hadn't even considered using "build" when building is
exactly what I do not want to happen. But yes, it works just fine.

> Well, ’package-output’ in (guix packages) is what you need, I guess.

And that works just fine as well.

Thanks,
  Konrad.

Finding the store path of a package

[Konrad Hinsen]

Dear Guix experts,

I wonder if there is a straightforward way to find the store path
corresponding to a package, assuming that the package actually is in the
store. I don't care if it's done via the CLI or via Guile code.

Use case: Looking at the files inside a package. What I do now is "ls
/gnu/store/**", but that usually lists many variants of
the package, and I don't know which of them actually is the current one.

I came up with some Guile code that does the job:

   (define (store-path specification)
 (let*-values (((package output)
(specification->package+output specification))
   ((entry)
(package->manifest-entry package output))
   ((l-entry)
(with-store store
  (run-with-store store
(lower-manifest-entry entry (%current-system))
   (manifest-entry-item l-entry)))

but it also downloads/builds the package if it's not yet in the store,
which is not what I want. In fact, I don't care what happens then the
package is not in the store. Returning a non-existing path is fine,
as is raising an error or returning #f.

Another attempt is "guix package –-list-installed", but this works only
for packages installed in a profile. I am shifting more and more to
on-the-fly environments, meaning that many packages in my store belong
to no profile.

Cheers,
  Konrad.

Re: Guix in Debian!

[Konrad Hinsen]

On 24/01/2021 05:04, Vagrant Cascadian wrote:

Now on Debian you should be able to:

   apt install guix
   guix install dpkg
   guix environment --ad-hoc dpkg -- dpkg -i ./guix_1.2.0-3_amd64.deb

It is almost like symmetry!


Wow, that's excellent news. Probably the biggest to improvement to 
onboarding new users since... forever, as far as I am concerned ;-)



Thanks,

  Konrad

Re: etc/news copyright (was Re: Translating the web site)

[Konrad Hinsen]

Ludovic Courtès  writes:

>> I attach a patch adding copyright notices, but I guess the authors
>> should agree.  I put them in Cc.
>
> If the authors agree, fine with me!

Since I am in CC, I must be one of the authors, though I don't really
remember what I could have contributed. Anyway, this is fine with me as
well!

Konrad.

Re: [OUTREACHY] Proposal: substitutes over IPFS

[Konrad Hinsen]

Hi Ludo,

> I prefer not to volunteer to mentor it, but I’m happy to contribute to
> discussions and code review.
>
> Who’d be willing to mentor it?

I don't know what exactly that implies, so all I say for now is that I'd
be happy to participate in this project. I know the IPFS side rather
well, but have only a user-level knowledge of how Guix handles
substitutes.

Does Outreachy call for a single mentor, or can it be a team?

Konrad.

Re: Improve ASDF build system for Common Lisp libraries

[Konrad Hinsen]

Am 12.09.20 um 11:36 schrieb Guillaume Le Vaillant:


I've been working on some changes to the asdf-build-system for Common
Lisp libraries and programs:


That sounds very promising, thanks for this substantial effort! I can't 
test this right now, but will do so as soon as possible.


Cheers,
  Konrad

Re: Reproducible Research Hackathon: Friday, July 3rd

[Konrad Hinsen]

Hi Bonface and Ludo,

> That's strange. To get the right results, you'd have to do a `2L ** 64`.

Exactly. The long integer arithmetic works fine, it's the detection
of the overflow of 64-bit ints that fails.

> When I tried `2 ** 63` I got `-9223372036854775808`. There's also an

Good catch, I hadn't tried that one!

> Oooh, thank you!  It looks like an “interesting” bug, one of those
> that can help make the case for precise software environment control.
> :-)

Exactly. Most people have understood by now that Python and libraries
higher up on the stack suffer from breaking changes, but C and
libc/libm, that's still the unshakable ground on which software can
safely be built.

> Uh, weird!  We could check whether building Python with ‘-fwrapv’ helps.
> See also .

Interesting. I am finding out that I don't know most of the possible
suspects in this crime story ;-)

Cheers,
  Konrad

Re: Reproducible Research Hackathon: Friday, July 3rd

[Konrad Hinsen]

Hi Ludo,

> Apologies for the delay.  What was this bug exactly?
>
> I know Bonface addressed an issue related to how the Python 2.4 build
> system would capture the kernel version via ‘uname’ a build time:
>
>   
> https://gitlab.inria.fr/guix-hpc/guix-past/-/commit/d1977f5dccd73341f363cfa8d58ae3f2b2700ad7
>
> But presumably you’re referring to something else, right?

Indeed. The details are here:

  https://gitlab.inria.fr/guix-hpc/guix-past/-/issues/1

Since I won't be able to look into this before my summer vacations,
I opened an issue as a reminder.

Cheers,
  Konrad

Re: Reproducible Research Hackathon: Friday, July 3rd

[Konrad Hinsen]

Hi Simon et al.,

> We are interested to hear your feedback.  Especially about what pass,
> what fail and what you have learnt, if you enjoyed the experience, or on
> the contrary if you not, what could be improved for the next round.

For me this was the occasion to finally start playing with a channel
(guix-past), which has been on my agenda for a while. It was much easier
with people around to answer practical questions that would take a long
time to resolve by going through the documentation.

It was also a good occasion for thinking about how to package artifacts
of historical interest (i.e. recording release dates as metadata).

In terms of packaging techniques, there wasn't much new in it for me.
Python 2.4 is handled very much like Python 2.7, and since I installed
all this stuff by hand many times back in 2008, it felt almost familiar.

There is an interesting issue left for me to explore, which is why
Python 2.4 compiled with today's gcc has a bug that it definitely didn't
have back then. It's probably related to the many intentional
ambiguities in the C language standard, check out John Regehr's blog
(https://blog.regehr.org/) for interesting examples. That could become a
selling point for Guix because we have the option of compiling old
software with old compilers, which is hard with other infrastructures.

Cheers,
  Konrad

Re: “Reproducible research articles, from source code to PDF”

[Konrad Hinsen]

Konrad Hinsen  writes:

> Sounds fine. I am not much of a hackathon expert, so I don't propose
> myself for organizing this, but I can make a preselection of suitable
> submissions to the ReScience challenge (no proprietary software etc.)
> with comments about the specific challenges.

Here is my list of candidate projects. There are three general
categories:

1) Package old software that is of sufficiently wide interest
   (i.e. add to guix-past)
- g77 (used in https://github.com/ReScience/submissions/issues/41)
- SciPy ecosystem from 2007 (at least Python, NumPy, matplotlib)
  (used in https://github.com/ReScience/submissions/issues/14)

2) Package highly specialized research software

   These programs are too specialized for the Guix distribution, so
   "packaging" means writing a guix.scm. The long-term goal is to learn how
   to make this kind of packaging easier, to the point that scientists are
   willing to do it themselves. This means it must be doable with minimal
   Guile competence, ideally by modifying templates provided by experts.

   I have picked four cases, listed by increasing level of difficulty:

   a) https://github.com/ReScience/submissions/issues/42

   A rather standard Fortran code, with only the popular BLAS and LAPACK
   libraries as dependencies. Instructions are given for manual
   compilation.

   b) https://github.com/ReScience/submissions/issues/36

   A medium-sized Fortran program with a Makefile.

   c) https://github.com/ReScience/submissions/issues/41

   A mixed C-Fortran code from 2008, built with autotools. Looks simple,
   but the author did not succeed in compiling it on a modern machine
   because it requires the abandoned g77 compiler.

   d) https://github.com/ReScience/submissions/issues/20

   A medium-sized Fortran library with a Makefile. Tricky because it adds
   its own wrappers around the Fortran compiler.

3) Fully automated reproductions of results (typically figures)

   There is only one case (other than Ludo's which already uses Guix):

   - https://github.com/ReScience/submissions/issues/39

   A fully reproducible reproduction of two Open Source simulation software
   packages (C/C++), based on Debian and its debuerreotype system. The
   challenge is to demonstrate how Guix can do it better!

Cheers,
  Konrad
-- 
-----
Konrad Hinsen
Centre de Biophysique Moléculaire, CNRS Orléans
Synchrotron Soleil - Division Expériences
Saint Aubin - BP 48
91192 Gif sur Yvette Cedex, France
Tel. +33-1 69 35 97 15
E-Mail: konrad DOT hinsen AT cnrs DOT fr
http://dirac.cnrs-orleans.fr/~hinsen/
ORCID: https://orcid.org/-0003-0330-9428
Twitter: @khinsen
-

Re: “Reproducible research articles, from source code to PDF”

[Konrad Hinsen]

Ludovic Courtès  writes:

>> Maybe we could organize a Virtual Hackathon?  Over 1 day?  Or 2 days?
>> The power of collective motivation. :-)
>
> Sounds like a good idea!  Perhaps a one-day hackathon to begin with?
> Early July maybe?

Sounds fine. I am not much of a hackathon expert, so I don't propose
myself for organizing this, but I can make a preselection of suitable
submissions to the ReScience challenge (no proprietary software etc.)
with comments about the specific challenges.

Cheers,
  Konrad
-- 
---------
Konrad Hinsen
Centre de Biophysique Moléculaire, CNRS Orléans
Synchrotron Soleil - Division Expériences
Saint Aubin - BP 48
91192 Gif sur Yvette Cedex, France
Tel. +33-1 69 35 97 15
E-Mail: konrad DOT hinsen AT cnrs DOT fr
http://dirac.cnrs-orleans.fr/~hinsen/
ORCID: https://orcid.org/-0003-0330-9428
Twitter: @khinsen
-

Re: “Reproducible research articles, from source code to PDF”

[Konrad Hinsen]

Hi Ludo and Simon,

> I don’t like the phrase “average scientist”, and we’re talking about
> people with a PhD who definitely know how to learn.

I didn't take that phrase as a reference to ability, but to prior
knowledge. I am pretty sure that anyone who uses Python can also learn
to use Guile, but a computer scientist having experience with ten
languages will have less effort to do so than an archaeologist or a
wetlab biologist who has never used anything else but Python.

> Apart from that, I agree with the comments above: putting it in the
> hands of scientists will be the real challenge.  I think providing
> modules and ready-to-use “templates” for people who use R+RMarkdown, or
> LaTeX, or Jupyter, etc. is a necessary step.

I'd start somewhat differently: generate diverse use case examples.  The
contributions to the ReScience reproducibility challenge could be a nice
starting point: go through them, one by one, and try to re-implement the
authors' various approaches with Guix.  Then, in a second step, try to
identify additional tooling support in Guix that would make the recipes
simpler to implement. That might well lead to the development of ready
to use templates, but I prefer starting from a use case analysis to see
what is needed in real life.

Another obstacle to adoption is the difficulty of deployment. Right now,
if I use Guix to make my work reproducible, I require my readers to
install Guix on their computers, which is a lot of work for Linux users
and a major headache for Windows/macOS users. We really need to reduce
that barrier to deployment.

Some ideas: Simple deployment of a VM running Guix System on a major
cloud provider would be nice to have. Or a service like mybinder.org,
but based on Guix rather than Docker. Or, for local execution, a Docker
image containing Guix plus some tooling to do the equivalent of "guix
time-machine –commit=xxx – build -f guix.scm" plus copying the contents
of the generated package into the user's directory.

Cheers,
  Konrad
-- 
---------
Konrad Hinsen
Centre de Biophysique Moléculaire, CNRS Orléans
Synchrotron Soleil - Division Expériences
Saint Aubin - BP 48
91192 Gif sur Yvette Cedex, France
Tel. +33-1 69 35 97 15
E-Mail: konrad DOT hinsen AT cnrs DOT fr
http://dirac.cnrs-orleans.fr/~hinsen/
ORCID: https://orcid.org/-0003-0330-9428
Twitter: @khinsen
-

Re: unexpected reproducibility of reproducible blog post?

[Konrad Hinsen]

Hi Ludo,

> Grafts are normal derivations, and they’re deterministic: it’s just
> about replacing a set of strings by another set of strings.
>
> On the implementation, see also
> .

Thanks for the clarification. That post makes it clear that grafts only
modify the build process of grafted packages.

Cheers,
  Konrad

Re: unexpected reproducibility of reproducible blog post?

[Konrad Hinsen]

Hi Simon,

> I will add something overthere for tracking reproduciblity infos in
> the future.

It would actually be nice to have some external Guix reproducibility
surveillance. A few benchmark packages that will be rebuilt regularly,
using frozen commits via time-machine, and checked for bit-by-bit
identity explicitly, not relying on Guix' hash mechanism. Trust but
verify.

My example is perhaps not such a bad start. Building a Docker container
containing gcc exercises a lot of code in Guix.

I looked a bit at grafts. The documentation at

  https://guix.gnu.org/manual/en/html_node/Security-Updates.html

isn't very explicit about the reproducibility of grafts. In particular,
it doesn't say if a package containing patched binaries retains its
original hash, or receives a new unique one. With a unique hash, grafts
would just be a tweak in the build system, and no less reproducible than
standard builds. It looks like I have to dive into the source code to
find out!

Cheers,
  Konrad

Re: unexpected reproducibility of reproducible blog post?

[Konrad Hinsen]

zimoun  writes:

> Argh! The author should watch the Fun MOOC about computational
> reproducibility. ;-)

That would probably help. I'll pass on the message ;-)

I have also opened an issue for this:

  https://github.com/khinsen/reproducibility-with-guix/issues/2

> Grafts or maybe Guile 2 -> 3?

With time-machine, you run the full Guix from back then, so you run
Guile 2 if that's what it takes. What I am not so sure about is how the
old Guix release is built. If the build uses the equivalent of "guix
environment guix", it would start using Guile 3.

Time travelling is not as simple as it looks, but then we should have
expected that!

Cheers,
  Konrad

Re: unexpected reproducibility of reproducible blog post?

[Konrad Hinsen]

Hi Simon,

> Based on the nice blog post [1], instead of really travelling I just
> travel in time. :-)
> If I read correctly and if I did not do any mistake, the final hash is
> not the same now than before. It is not what I was expecting.
>
> Expected output (blog post):
> /gnu/store/iqn9yyvi8im18g7y9f064lw9s9knxp0w-docker-pack.tar
>
> Returned output:
> /gnu/store/klisfr3a4wxb9dc5sgibb45kky72kg65-docker-pack.tar
>
> Has the file 'guix-version-for-reproduction.txt' been tracked?

Unfortunately not. The repository for the preparation of the post
is at 

  https://github.com/khinsen/reproducibility-with-guix/

but it doesn't contain the file 'guix-version-for-reproduction.txt'.

> Is really the commit 769b96b62e8c09b078f73adc09fb860505920f8f used to
> produce the Docker image listed in the blog post?

Hard to say... I can't play with that right now because I am running
jobs on my computer that eat all the memory.

One question I have been wondering about is the possibility of grafts
being an obstacle to reproducibility. Grafts are something I don't
really understand yet, so I cannot answer this question. In particular,
does a grafted package get a different hash from a package built with
grafting disabled?

Cheers,
  Konrad.

Re: TLS certificates for web browsers in guix environment --container

[Konrad Hinsen]

On 21/04/2020 22:50, Pierre Neidhardt wrote:

Makes sense, thank you for the details.

What about adding the above example to the manual page of `guix environment'??



Good idea. I suppose it's not just Web browsers that will need this. I 
remember occasional certificate problems when running scripts in 
containerized environments. Perhaps they were just using wget or curl. I 
was too lazy to explore this so I switched to --pure.



Cheers,

  Konrad

Re: GNU Guix 1.1.0 released

[Konrad Hinsen]

On 15/04/2020 15:17, Ludovic Courtès wrote:

We are pleased to announce the release of GNU Guix 1.1.0.


The news has made it to a widely read IT news site in Germany (which I 
didn't expect):



https://www.heise.de/developer/meldung/Linux-GNU-Guix-1-1-ist-auf-dem-Weg-zu-minimalistischem-Bootstrapping-4703451.html


Thanks to all who made this happen!


Cheers,

  Konrad

Re: Hyperlinks!

[Konrad Hinsen]

On 13/04/2020 13:13, Pierre Neidhardt wrote:

Ludovic Courtès  writes:


I’d like to extend it to include references to the Guile manual, so that
one could click on, say, ‘append’, but there might be too many false
positives at that point.  And then we would need DrRacket fanciness to
be able to determine what an identifier really refers to…

Perfect time for Rix, Guix in Racket!  Haha! :D


Easy. "Just" implement #lang guile in Racket ;-)


Cheers,

  Konrad.

RE: good practices in science

[Konrad Hinsen]

bijan ghavami-kia  writes:

> It’s an interesting prospect, shouldn’t we be working towards this
> fantastical goal?

We (Guix) are already working towards the abstract goal of this project,
because what Guix does is effectively provenance tracking for
computations. Guix' package dependency graph is part of the public
knowledge graph.

If and how we could collaborate with the Underlay in a more
institutional and technical sense is an interesting question to think
about. One ingredient mentioned by Underlay is IPFS, with which Guix has
already made contact. And IPFS is looking into including git
repositories by providing CIDs for Git commits (though this doesn't seem
high-priority task). Guix is already connected to the Software Heritage
archive (two ways: Guix is archived there, and it can work with source
code from the archive), which is basically Git scaled up massively.

All that means that hooking up Guix with the Underlay is a pretty
straightforward endeavor from a technical point of view. It's mainly a
question of all interested parties agreeing on some form of
collaboration. So we are back to politics! Personally I'd love to see
this happen, and I'd definitely be willing to participate actively.

Cheers,
  Konrad

Re: good practices in science

[Konrad Hinsen]

Hi Bengt,

> (I guess I get excited reading prose that shows attention
> to the distinction between abstractions and their representations.
> Sort of like reading quotes from Plato, and thinking, "Hey, wow,
> I've had some of those thoughts." :)

There are plenty of good ideas in that project, I am not claiming the
opposite. Unfortunately, that has never been a guarantee for success.

> I hope they are not just preparing themselves for spinning off
> and becoming rich as a monopolist takeover target ;-/

With people like Danny Hillis on board, they should be aware of that
trap!

> So what makes you hopeful about guix? :)

It's so technical that politics-minded people won't even look at it.

Cheers,
  Konrad.

Re: good practices in science

[Konrad Hinsen]

Hi Pierre,

> I had never heard about this project, looks like it's a most critical
> venture these days! :)
>
> https://underlay.mit.edu/
>
> Any idea if there is a public project page?

My understanding is that the project just started and hasn't much to
show for now. It's on my "have-a-look-every-three-months" list.

The real question with this type of infrastructure project is if it will
produce something convincing enough for many players to adhere to. It's
as much politics as technology.

Cheers,
  Konrad

Re: good practices in science

[Konrad Hinsen]

Hi Marco,

> Are there any natural scientists here?

I have no idea how numerous we are, but yes, there are. As for myself,
I am in computational biophysics.

> I am sending this to this list because Guix is an obvious tool for
> scientific (and other) computing.  None of my collegues anywhere in
> the world have heard of it and they are not interested when I mention
> it.  (Furthermore, brendyyn on #guix suggested this list.)

Don't worry, that will change.

> In my mind, this must mean that one writes plain text everywhere.
> This is plain/text for e-mail, LaTeX for papers, code is code,
> Markdown or similar for most other documents.  All this is in version
> control.  You can push, share, collaborate quite easily.  Anyone is
> free to make a pretty PDF of it or do whatever else.  Because, of
> course it is all free as in speech.  You know all this.

That is a workflow which is being advocated increasingly. You could
point your doubting colleagues to this MOOC, for example:

  https://www.fun-mooc.fr/courses/course-v1:inria+41016+self-paced/about

(disclaimer: I am one of its authors). Guix is not covered there, but it
will in a more advanced sequel currently under preparation.

> I would like to find a community where I can do science in a good way.
> I want to use free software and would like to collaborate through
> version control, IRC, Jitsi, well formatted e-mails.  Does such a
> community exist?

It is growing. I can't say about your field or your neigbourhood, but
check out communities such as The Carpentries
(https://carpentries.org/), which is organizing tutorials all around the
globe to teach the tools that you like.

Cheers,
  Konrad

Re: Thoughts on making Guix even better

[Konrad Hinsen]

Hi everyone,


 > The guix system transactions are NON-MODULAR. That is, you cannot
selectively reconfigure certain parts of the system. For example,
you either reconfigure the system as a whole (or) you do not
reconfigure the system at all.


Today's software systems are inherently non-modular. Guix is the most 
advanced technology we have to work around this problem, but it doesn't 
solve it. The problem is so hard to solve because it requires most of 
all a change in everyone's attitude towards software development, and 
thus very probably different organizational structures (because of 
Conway's Law).


Cheers,
  Konrad.

Re: time-machine broken from 2019-04-07 to 2019-05-04

[Konrad Hinsen]

zimoun  writes:

> What should be the fix?
> Because it is annoying to be dependent of upstream bad practise.
>
> Well, is it fixable?

The root cause of the problem is the mutability of URLs. The solution is
to move on to immutable references. Such as git commits.

My impression is that tarballs are the preferred source code reference
used in Guix today, with git commits used only when there is no release
tarball. Is that an official policy, or just historical accident?
If a policy, what's the reason for it?

Cheers,
  Konrad.

Re: Store channel specification in profile

[Konrad Hinsen]

Pierre Neidhardt  writes:

>>> Cleaning up just means updating the profile.
>>
>> You mean with
>>
>> guix upgrade -p /path/to/profile?

Right.

>> OK, but what about a system where you don't have the profile, only the
>> specification file?

OK, that's at the other end, for the user of a specification file. That
must indeed be handled by some option to "guix package".

> Instead, I suggest we use a --use-default-channels boolean flag and then we
> could leverage guix time-machine to force a specific channel
> specification on a manifest:
>
> --8<---cut here---start->8---
> guix time-machine --channels=foo.scm -- package -m specs.scm 
> --use-default-channels
> --8<---cut here---end--->8---

LGTM!

Cheers,
  Konrad.

Re: Store channel specification in profile

[Konrad Hinsen]

Hi Pierre,

> We've made some progress on the topic during the Guix Days.

Thanks for the summary!

> The Plan©:
>
> On every profile installation, we generate a "specifications.scm" file 
> alongside
> the internal "manifest".

Sounds good.

> Problems:
>
> - There may be too many provenances, we need to add a CLI flag to
>   ignore provenance.

Too many in what sense?

I'd just print a warning if the specification refers to more than, say,
five inferiors. Users can then clean up their profile or live with the
mess. Cleaning up just means updating the profile. That would re-create
specifications.scm, in a better way than a special CLI flag because
specifications.scm is always the same as the current profile.

> Proposed format for "specifications.scm": we can reuse
> `specifications->manifest`.  Each entry is either or string, in which case it
> acts as before, or a list, with the following self-explanatory elements:

Looks good.

> A somewhat unrelated propostion: To avoid further confusion between the 
> internal
> "manifest" and the user-facing "--manifest", we could rename the internal
> manifest to $profile/internal-maifest.

Fine, but not so critical. I doubt there are many users who ever look at
that file.

Cheers,
  Konrad.

Re: Looking for help with packaging a Common Lisp library

[Konrad Hinsen]

Pierre Neidhardt  writes:

> Great, I've let upstream know:
> https://github.com/HDFGroup/hdf5-cffi/issues/70

And I just submitted a fix as a pull request:

  https://github.com/HDFGroup/hdf5-cffi/issues/71

Thanks to both of you for getting this sorted out quickly! If upstream
doesn't merge my patch in the next days, I'll submit my current Guix package
which does the patch itself.

Cheers,
  Konrad.

Re: Looking for help with packaging a Common Lisp library

[Konrad Hinsen]

Pierre Neidhardt  writes:

> Indeed.  What I meant is that if the Common Lisp package (the code
> inside the src folder) depends on cffi-grovel, then I believe it
> should be added to :depends-on as well.  We can ask ASDF and
> hdf5-cffi.

I'd be happy to ask hdf5-cffi, but I excpect their answer to be
"we just copied the example from the cffi-grovel manual", which
indeed they did. So maybe cffi-grovel is the better place to ask.
Although ultimately it's an ASDF question in my opinion.
And ASDF looks seriously underdocumented compared to the complexity of
all it does.

Cheers,
  Konrad.

Re: Looking for help with packaging a Common Lisp library

[Konrad Hinsen]

Pierre Neidhardt  writes:

> The Guix package recreates the .asd file (without the
> defsystem-depends-on), which is why it worked in Quicklisp but not in
> Guix.

OK, that makes sense.

> We could merge Guillaume's patch, but first I'd like to make sure that
> this is not an upstream bug.  Isn't cffi-grovel supposed to be added to
> the :depends-on list instead?

I am certainly not the right person to answer that question. But here's
what the ASDF manual says:

   The :defsystem-depends-on option to defsystem allows the programmer
   to specify another ASDF-defined system or set of systems that must be
   loaded before the system definition is processed. Typically this is
   used to load an ASDF extension that is used in the system definition.

This looks like what the cffi-grovel manual suggests, with an example:

   
https://common-lisp.net/project/cffi/manual/html_node/Groveller-ASDF-Integration.html#Groveller-ASDF-Integration

In hdf5-cffi.asd, some components are labelled :cffi-grovel-file,
which looks like an ASDF extension provided by cffi-grovel. If that is
true, then the :defsystem-depends-on is necessary. A quick test supports
that hypothesis: if you remove the line, the system fails to load.

Cheers,
  Konrad.

Re: Looking for help with packaging a Common Lisp library

[Konrad Hinsen]

Hi Pierre and Guillaume,

thanks for your quick help!

Guillaume Le Vaillant  writes:

> I think the generated '...-sbcl-hdf5-cffi-1.8.18/lib/sbcl/hdf5-cffi.asd'
> file is missing a dependency. It contains ':depends-on ("cffi")', but
> I think it should be ':depends-on ("cffi" "cffi-grovel")', because the

That was the problem indeed. I patched hdf5-cffi.asd to add :cffi-grovel
to the dependencies, and with that patch everything works fine.

> We could add all the dependencies declared in the original asd files (by
> 'defsystem-depends-on' and 'depends-on') to the generated asd files with
> the following patch:

I don't know ASDF and the ASDF build system well enough to comment on
that. All I can say is that in a "standard" SBCL + QuickLisp
environment, it is not necessary to patch  hdf5-cffi.asd, so it seems
to be correct as far as I can tell, meaning that it would be nice if it
worked out of the box as well in Guix.


Pierre Neidhardt  writes:

> It seems unrelated, but first there seems to be an issue during the
> build regarding the missing pkg-config package.

I am so used to pkg-config related errors that I hardly see them any
more. So yes, this ought to be fixed, but it's not related to the
problem I had.


Thanks again,
  Konrad.

Looking for help with packaging a Common Lisp library

[Konrad Hinsen]

Hi Guix,

I hope that some Common Lisp expert here can help me with a strange
error message.

I am trying to package the library hdf5-cffi, my current package
definition is attached. It builds without any error message, but
when I try to load the library with

  (asdf:load-system "hdf5-cffi")

I get the error message

 The name "CFFI-GROVEL" does not designate any package.

when loading

  
/gnu/store/3lcw9qvx76770wpsbgdvma154y84v54i-sbcl-hdf5-cffi-1.8.18/lib/sbcl/hdf5-cffi--system.fasl

It doesn't make a difference if I put sbcl-cffi-grovel under inputs or
propagated-inputs.

Does anyone have an idea what could be going wrong here?

Cheers,
  Konrad.



cl-hdf5-cffi.scm
Description: Binary data

Re: Proposal for a blog contribution on reproducible computations

[Konrad Hinsen]

Hi Ludo,

> Perfect, thanks!  I had to move files where Haunt expects them, and to
> make the table pure ASCII because guile-commonmark doesn’t support HTML
> nor tables, but here we are:
>
>   https://guix.gnu.org/blog/2020/reproducible-computations-with-guix/
>   https://hpc.guix.info/blog/2020/01/reproducible-computations-with-guix/

Looks good, thanks!

One problem though: the link to the script is broken:

  
https://guix.gnu.org/blog/2020/reproducible-computations-with-guix/show-dependencies.scm

yield 404.

I had put it into the same directory as the Markdown file and the
images, apparently Haunt didn't like that either.

Cheers,
  Konrad.

Re: Proposal for a blog contribution on reproducible computations

[Konrad Hinsen]

Ludovic Courtès  writes:

> Another thing that comes to mind: would it make sense to mention ‘guix
> graph’ in the part where you pipe the output of ‘guix show’ to ‘recsel’,
> etc.?

Forgot that one, sorry. Yes, it would make sense, though I'd place it a
bit later in the text. But I'd have to figure out first how how the
various options of "guix graph" relate exactly to what I am writing.

  ‘package’
   This is the default type used in the example above.  It shows the
   DAG of package objects, excluding implicit dependencies.  It is
   concise, but filters out many details.

Are "implicit dependencies" those added by the build system? If yes,
this edges in this graph would correspond to package-direct-inputs.

  ‘bag’
   Similar to ‘bag-emerged’, but this time including all the bootstrap
   dependencies.

And that is package-closure with arrows defined by bag-direct-inputs, right?

Cheers,
  Konrad.

Re: Proposal for a blog contribution on reproducible computations

[Konrad Hinsen]

Hi Ludo, Simon, and GIovanni,

Thanks for your feedback !

> Minor comments:
>
>   • You write “Build systems are packages as well”.  This could be
> slightly misleading: build systems are (1) a set of packages, and
> (2) a build procedure.  Dunno if it makes sense to clarify that.

Maybe I got something wrong, but I think I described this as you say
(please check!). Quote:

  Build systems are pieces of Guile code that are part of Guix. But this
  Guile code is only a shallow layer orchestrating invocations of other
  software, such as =gcc= or =make=. And that software is defined by
  packages.

The build procedure is that "shallow layer orchestrating invocations".
Does this sound right?

>   • In the ‘guix pack’ example, you could perhaps omit all the -S flags
> except for /bin, and mention ‘--save-provenance’.

I'll have to look up ‘--save-provenance’ first. I don't use "guix pack"
that much, though I should probably use it more, if only to expose more
people indirectly to Guix.

>   • Would it make sense to mention MPFR in the paragraph about IEEE 754?

I considered it, but left it out because it would probably create
confusion. And people who are aware of MPFR probably don't need my
explanation of floats.

>   • Regarding ‘--container’, you write that namespaces “may not be
> present on your system, or may be disabled by default”, which is a
> bit strong; “may be present on your system, but perhaps disabled by
> default” would be more accurate.  :-)

Fixed. I don't know anything about the implementation techniques of
–container, so I'll blindly write what you say :-)

> The format we use is Markdown fed to Haunt:

OK, pandoc should get me there.

> You can post a patch against the guix-artwork.git repo here when you’re
> ready.

OK.

> If you want we can publish it next Tuesday or Thursday.  We could have
> it on both hpc.guix.info and guix.gnu.org, with one saying that it’s a
> re-post of the other.

Fine with me!


zimoun  writes:

> That said, I also find interesting the command-line and hashes comparisons:
>
> --8<---cut here---start->8---
> /usr/bin/gcc pi.c -o pi-debian-gcc8
> docker run -v `pwd`:`pwd` -w `pwd` -ti gcc-toolchain gcc pi.c -o pi-docker
> guix environment --container --ad-hoc gcc-toolchain -- gcc pi.c -o pi-guix
>
> md5sum pi-*
>
> b268af34d62763a2a707944403bf7b0b  pi-debian-gcc8
> 1be3c1b5d1e065017e4c56f725b1a692  pi-docker
> 1be3c1b5d1e065017e4c56f725b1a692  pi-guix
> --8<---cut here---end--->8---
>
> Anyway! :-)

Nice! Not sure I want to go into that because it requires adding another
system (Debian), which I think is mainly a source of confusion.

>>   • Would it make sense to mention MPFR in the paragraph about IEEE 754?
>
> And MPFI? ;-)

OK, I see another blog post coming ;-) But there are people more
competent than myself for that.


Giovanni Biscuolo  writes:

>> (which is sad because your Org file with Babel sessions is much nicer…).
>> I think Pierre had something to convert Org to Markdown.
>
> you could try pandoc or emacs-ox-hugo, both in Guix
> I can help convert/adapt if needed

My plan for now is pandoc, but if that doesn't work as expected, I'll
come back to your offer for help!

Thanks everyone,
  Konrad.

Proposal for a blog contribution on reproducible computations

[Konrad Hinsen]

Hi Guix,

about a month ago, I proposed:

> I am working on an article (ultimately for "Computing in Science and
> Engineering") that takes up some of those subjects and illustrates them
> using Guix. It would be nice to get feedback from the Guix community
> early, to make sure I don't do something stupid. So I could do a first
> version for the Guix blog, and thus propose it here for review,
> and then make it a bit less technical for a wider audience.

Here is a first complete draft:

  
https://github.com/khinsen/reproducibility-with-guix/blob/master/reproducibility-with-guix.org

Feedback welcome, be it by mail or as issues on GitHub.

Also, what is the procedure for submitting blog posts? What are the
right formats for text and graphics?

Cheers,
  Konrad.

Re: bug#38529: Make --ad-hoc the default for guix environment proposed deprecation mechanism

[Konrad Hinsen]

Hi Ricardo,

> I wonder if we should simply bump the version number to indicate that
> this is a breaking change?

That's a possibility, but who ever looks at Guix version numbers?

> Another more difficult option would be to do what responsible API
> developers on the web do: to version their API and to make the API
> version selectable.  I don’t know *how* to do this elegantly, and

That's an interesting idea which would also take care of similar
situations in the future.

One way to implement this is to have executables "guix1", "guix2"
etc. Most users would then define an alias "guix" for interactive use,
but hopefully script authors would use the versioned executables.

Cheers,
  Konrad.

Re: bug#38529: Make --ad-hoc the default for guix environment proposed deprecation mechanism

[Konrad Hinsen]

Hi Simon,

> Assuming "guix environment" would stay and following the proposed
> plan, you would need to add GUIX_ENVIRONMENT_DEPRECATED=1 on the top
> of your script. In this would not be a problem for travelling back in
> time.

The problem is not how I update my scripts - I can manage that, no
matter what it takes.

The problem is scripts circulating in public repositories, tutorials,
etc. New users will find them and use them for inspiration. It's very
discouraging to see examples from tutorials fail or do something weird.

The main precedent is the Python 2->3 transition. There are tons of
GitHub repositories with Python code but no indication if it's 2, 3, or
both. I even had to use one that executed with either 2 or 3, but gave
different results. It takes a lot of motivation to persist.

For guix, there's the additional issue that we use the reproducibility
of builds as an argument. Non-reproducible examples are then a bit of a
credibility problem.

Cheers,
  Konrad.

Re: Deprecating ‘guix environment’?

[Konrad Hinsen]

Hi Ludo,

> Clearly there’s a tension between that and keeping Guix open to changes.

That's indeed the main problem and here as elsewhere, it is often a
topic of heated arguments.

My point of view (long form:
https://hal.archives-ouvertes.fr/hal-02117588)
is that software projects should adopt a backwards compatibility policy
early on, state it clearly in their documentation, and stick to it. That
prevents misunderstandings, bad surprises, and heated debates.

As for what that policy should be for Guix, that's a more difficult
story. For projects with versioned releases, I like the principles
of semantic versioning, but Guix is more of a rolling-release
project. (Test question: does anyone know what the current Guix version
number is? Does anyone care?) I am not aware of any good precedents
in terms of policy for such projects.

> The hard question then becomes: what do we call it?  I vote against
> abbreviations.  :-)
>
> Also, what other goals would we set for that command?  How would we
> frame it in the set of commands?

I vote for discussing the second point before the first one. Names
should reflect the functionality behind them.

How about a unified command for constructing environments and profiles
declaratively? In other words, combine "guix environment" and the
declarative parts of "guix package". We could probably get rid of
the somewhat obscure "guix environment -r" in the process.

Cheers,
  Konrad.

Re: bug#38529: Make --ad-hoc the default for guix environment proposed deprecation mechanism

[Konrad Hinsen]

Hi Simon,

> Maybe I miss a point. It is not: "watch out, this will do something
> else in the future" but "watch out, this was doing something else in
> the past and the change happened the  in ".

Concrete example: I am writing a tutorial about using Guix for
reproducible research. It shows several uses of "guix environment", some
of them without '–add-hoc' or '–inputs-of'. I know my examples will
cease to work in a few months. What am I supposed to do about this?

> First, I am not convinced that there is not so much scripts that will
> be broken. And second, I am not convinced neither that these very
> scripts need time-traveling.

Perhaps it's just me, but I use "guix environment" quite a lot in
scripts, in order to make them more reproducible. Here's a simple
example:

   #!/usr/bin/env bash
   guix environment --container --ad-hoc gcc-toolchain <> The first rule of backwards-compatibility is: never change the meaning
>> of an existing valid command/API. Add new valid syntax, deprecate old
>> valid syntax, but don't change the meaning of something that was and
>> will be valid.
>
> I agree on the rule.
> But it is mitigated but the number of users and the popularity of the tool. 
> ;-)

Indeed!

> Yes, it is probably the most adequate to do. But it is sad to loose
> the good name "guix environment"... and we know that naming is hard.
> ;-)

I definitely agree. As a lesson for the future, maybe we should use
not-so-nice names for new commands during a kind of beta-testing phase.

Cheers,
  Konrad

Re: bug#38529: Make --ad-hoc the default for guix environment proposed deprecation mechanism

[Konrad Hinsen]

On 16/12/2019 23:09, Ludovic Courtès wrote:

So in a more algorithmic manner:

1. if ad-hoc and inputs-of is present at the same invocation: fail
hard. (With an error like incompatible options present)
2. if only ad-hoc is present, then print a deprecation warning (yes,
we could make this suspendable with an environment variable, like you
described)
3. if only inputs-of present, then do the new behaviour.
4. if neither ad-hoc nor inputs-of present then
   a. if GUIX_ENVIRONMENT_DEPRECATED is 1: do the current behaviour,
   b. if GUIX_ENVIRONMENT_DEPRECATED is undefined, or is not 1: do the
new behaviour.

That sounds like a good plan to me.

#4 is the trickiest, and I think it’d be good to give users a bit of
time so they can start adjusting before deprecation is in effect.


#4 is trickiest for another reason: there is no future-proof use of 
"guix environment" that works right now and will continue to work. Nor 
is there any way to see, when looking at a command line, whether it's 
old-style or new-style, if neither --ad-hoc nor --inputs-of are present. 
This means that all existing documentation (tutorials etc.) will become 
misleading in the future. Worse, even documentation written today, in 
full awareness of a coming change, can't do better than saying "watch 
out, this will do something else in the future".


The first rule of backwards-compatibility is: never change the meaning 
of an existing valid command/API. Add new valid syntax, deprecate old 
valid syntax, but don't change the meaning of something that was and 
will be valid.


How about a more drastic measure: deprecate "guix environment" and 
introduce a new subcommand with the desired new behaviour?



Cheers,

  Konrad.

Re: Feedback from JRES in Dijon

[Konrad Hinsen]

Hi Ludo,

>> I am working on an article (ultimately for "Computing in Science and
>> Engineering") that takes up some of those subjects and illustrates them
>> using Guix. It would be nice to get feedback from the Guix community
>> early, to make sure I don't do something stupid. So I could do a first
>> version for the Guix blog, and thus propose it here for review,
>> and then make it a bit less technical for a wider audience.
>
> That would be great!  Also, a “how to do reproducible science with Guix”
> hands-on would be nice (we could also have a section in the cookbook.)

Since I am working for something quite similar for the Reproducible
Research MOOC, I can also make a variant for the cookbook. I just need
to find the time to make progress on all that!

Cheers,
  Konrad.

Re: Feedback from JRES in Dijon

[Konrad Hinsen]

Hi Bengt,

> BTW3, Konrad,
> That was a nice presentation -- are the tools you used to prepare it and 
> present it
> available as libre packages? (I'm not insisting you answer ;-)

That's LaTeX with the Beamer package, plus Inkscape for the graphics.
It's all in Guix.

A bit of history: for many years I used Apple's Keynote software for my
presentations. Until I discovered one day that I couldn't open my old
presentations any more. Apple had changed the format a few times, and
then quietly dropped support for the older formats. And the older
software versions that could read the old presentations were no longer
available.

So now it's LibreOffice for short quick-and-dirty slide hacking, and
LaTeX with Beamer for everything else.

Cheers,
  Konrad.

Re: On DSLs

[Konrad Hinsen]

Hi Simon,

> Out of curiosity, do you have examples about YAML or JSON with an
> interpreter running it?

I just encountered an example: GitHub actions.

Cheers,
  Konrad.