Re: guix environment --load vs. --file inconsistency

[Ludovic Courtès]

Hi Attila,

Attila Lendvai  skribis:

> i was writing the documentation of a guix.scm file, and i realized that 
> there's an inconsistency among the three most used commands in this context:
>
> so, there's:
> - guix build --file
> - guix package --file
>
> and then there's:
> - guix environment --load
> - guix pack # has neither
>
> i'd propose to change guix environment to also use --file, but maybe i'm 
> overlooking something, so, please speak up if you think it's a bad idea!
>
> i never used guix pack, but maybe that also deserves a --file argument?

Good point.  ‘--file’ predates ‘--manifest’; we could perhaps deprecate
‘--file’ in favor of ‘--manifest’, though I think there’s one special
case not handled elsewhere:

  guix build -f foo.json

We need to do something about it.

‘guix environment --load’ could be similarly deprecated, either to be
eventually removed or to be renamed to ‘--file’.

Thoughts?

Ludo’.

Re: Merging wip-guix-home to master

[Ludovic Courtès]

Hi,

Andrew Tropin  skribis:

> I'm about a week on wip-guix-home branch completely and Guix Home works
> fine.  There are no any major issues on rde-devel and guix-devel mailing
> lists and it seems that branch is ready to be merged.

Yay!  I’d like to take another look (I know I’ve been terribly MIA,
apologies!), and I hope other folks familiar with Guix System can
comment as well.

> There is a discussion[fn:2] on moving home services to (gnu services
> ...)  modules, which is likely to happen, but it's possible to do the
> migration relatively painless by re-exporting necessary symbols in
> (gnu home-services ...) at first and removing them completely later.

I know it can be annoying to existing Guix Home users, but I’d prefer
not to carry pre-merge baggage; that is, we’d just rename and not
provide those modules under their former names at all.

> Another important part of the work related to Guix Home project is
> covering related modules and cli with tests, but it can be done in
> parallel and is not a blocker for merging.

Do you have ideas of a possible testing strategy?

We should be able to test at least the CLI, either arranging to avoid
large builds (as in tests/guix-build.sh) or talking to the “real”
guix-daemon (as in tests/guix-pack-relocatable.sh) if we’re going to
need packages.

It’d be great to have this part ready soonish.

The way I see it, in 1.4 (2.0?), we’d mark Guix Home as a “technology
preview” in the manual with a prominent note.  That will allow us to get
feedback from new users and to fine-tune code correspondingly, and
that’ll make it clear to users that things are still subject to change.

Thoughts?

Thanks,
Ludo’.

Re: guix weather -m etc/sources-manifest.scm and CI?

[zimoun]

Hi,

On Thu, 23 Sep 2021 at 22:18, Ludovic Courtès  wrote:

> BTW, I meant to send a message here about these two manifests

Sorry if I shot your surprise effect announcement. :-)

> I set up a Cuirass job the other day¹, but due to my aforementioned
> disorganization, I haven’t yet taken the time to investigate its
> failure.
>
> ¹ https://ci.guix.gnu.org/jobset/source

Cool!

(Find my stuff with Cuirass always requires magic for me :-))


>> Does it make sense to duplicate the storage of all these origins?
>
> It surely does.

Do you mean having all twice: on Berlin and Bordeaux.  In addition to SWH
(although it is not ready yet).


>> PS: about etc/disarchive-manifest.scm, I guess 'all-origins' is missing:

> No, there’s:
>
>   (include "source-manifest.scm")
>
> at the top.

Oops!


> So the goal here was to move forward on this front by (1) setting up a
> CI jobset, and (2) having an mcron job or something that would
> periodically copy the “disarchives” in a directory served by nginx.  We
> have had all the code and most of the infrastructure for several months,
> so I thought it’s high time to get our act together.  Help welcome!

Thanks for pushing forward. :-)

Cheers,
simon

Re: Help for packaging LibreAdventure (for Guix Days :-))

[Ludovic Courtès]

Hi,

zimoun  skribis:

> In order to ease the interactions at onlinel event such as Guix Days
> or others hackathons, it could be nice to have something like
> WorkAdventure [1].  It makes a friendly space for online social
> interactions.  Here [2] is a free fork:
>
> 2: 

That’d be sweet!  Because of all the JS in there, doing it following
“Guix standards” may be… hard.  So I think the challenger(s) are advised
to take shortcuts (like keeping bundled JS), hosting the result in a
“staging” channel—the goal being to have something rather than nothing.

Ludo’.

Re: data.guix.gnu.org problem with revision 33d2574

[Ludovic Courtès]

Hi Chris,

Looking at that log you pasted:

Christopher Baines  skribis:

> guix-data-service: computing the derivation-file-name for mips64el-linux
> Computing Guix derivation for 'mips64el-linux'...

[...]

> @ build-started 
> /gnu/store/l89jsjd4881jv5pgc06z76vfw02hr7xs-guile-bootstrap-2.0.drv - 
> mips64el-linux 
> /var/log/guix/drvs/l8//9jsjd4881jv5pgc06z76vfw02hr7xs-guile-bootstrap-2.0.drv.bz2
>  28674
> @ build-started 
> /gnu/store/ibk4hi6nxsh2j0v69i5fc55lw923s98z-gcc-4.8.2.tar.xz.drv - 
> mips64el-linux 
> /var/log/guix/drvs/ib//k4hi6nxsh2j0v69i5fc55lw923s98z-gcc-4.8.2.tar.xz.drv.bz2
>  28680
> Downloading 
> https://ci.guix.gnu.org/nar/f2j6pi0d18pbz35ypflp61wzhbfcr8dp-linux-libre-4.14.67-gnu.tar.xz...
> . linux-libre-4.14.67-gnu.tar.xz  94.7MiB  
>   
>  0B/s 00:00 [ 
>  ]   0.0%. linux-libre-4.14.67-gnu.tar.xz  94.7MiB 
>   
>   
>  62KiB/s 00:00 [  ]   0.0%@ unsupported-platform 
> /gnu/store/l89jsjd4881jv5pgc06z76vfw02hr7xs-guile-bootstrap-2.0.drv 
> mips64el-linux
> while setting up the build environment: a `mips64el-linux' is required to 
> build `/gnu/store/l89jsjd4881jv5pgc06z76vfw02hr7xs-guile-bootstrap-2.0.drv', 
> but I am a `x86_64-linux'
> builder for 
> `/gnu/store/l89jsjd4881jv5pgc06z76vfw02hr7xs-guile-bootstrap-2.0.drv' failed 
> with exit code 1

It seems to be working as expected, no?  Computing the derivation for
mips64el-linux entails building mips64el-linux derivations (due to
grafts), and that fails.

HTH!

Ludo’.

Re: guix weather -m etc/sources-manifest.scm and CI?

[Ludovic Courtès]

Hello!

zimoun  skribis:

> Playing with the new 'etc/sources-manifest.scm', using fb32a38, I get:
>
> $ guix weather -m ~/src/guix/guix/etc/source-manifest.scm
> computing 16,831 package derivations for x86_64-linux...
> looking for 16,831 store items on https://ci.guix.gnu.org...
> https://ci.guix.gnu.org
>   74.6% substitutes available (12,556 out of 16,831)
>   at least 65,367.1 MiB of nars (compressed)
>   81,988.1 MiB on disk (uncompressed)
>   0.095 seconds per request (1,606.8 seconds in total)
>   10.5 requests per second
>
>   0.0% (0 out of 4,275) of the missing items are queued
>   5 queued builds
>   aarch64-linux: 4 (80.0%)
>   powerpc64le-linux: 1 (20.0%)
>   build rate: .00 builds per hour
>   powerpc64le-linux: 0.00 builds per hour
>   aarch64-linux: 0.00 builds per hour
>   i686-linux: 0.00 builds per hour
>   x86_64-linux: 0.00 builds per hour
> looking for 16,831 store items on https://bordeaux.guix.gnu.org...
> https://bordeaux.guix.gnu.org
>   99.8% substitutes available (16,804 out of 16,831)
>   62,195.0 MiB of nars (compressed)
>   108,212.7 MiB on disk (uncompressed)
>   0.049 seconds per request (829.2 seconds in total)
>   20.3 requests per second
>   (continuous integration information unavailable)
>
>
> The questions are:
>
> Why ci.guix.gnu.org contains only 75%?  And bordeaux almost everything?
> (I guess the missing ones on bordeaux are corner cases as icecat, 
> linux-libre).

I wonder too!

BTW, I meant to send a message here about these two manifests, and then
got caught in a crazy patch-review/bug-fix/work loop.  The goal of this
manifest is to run ‘weather’ and ‘build’ so we can get all the source
and check the status at once, as you found out.

I set up a Cuirass job the other day¹, but due to my aforementioned
disorganization, I haven’t yet taken the time to investigate its
failure.

¹ https://ci.guix.gnu.org/jobset/source

> Does it make sense to duplicate the storage of all these origins?

It surely does.

> PS: about etc/disarchive-manifest.scm, I guess 'all-origins' is missing:
>
> (let ((origins (all-origins)))
>   (manifest
>(list (manifest-entry
>(name "disarchive-collection")
>(version (length origins))
>(item (disarchive-collection origins))

No, there’s:

  (include "source-manifest.scm")

at the top.

So the goal here was to move forward on this front by (1) setting up a
CI jobset, and (2) having an mcron job or something that would
periodically copy the “disarchives” in a directory served by nginx.  We
have had all the code and most of the infrastructure for several months,
so I thought it’s high time to get our act together.  Help welcome!

Thanks,
Ludo’.

Re: On the naming of System and Home services modules.

[Ludovic Courtès]

Hi,

Andrew Tropin  skribis:

> However, ~(gnu home services ...)~ also looks cool, but it would be a
> little inconsistent with system services, which will have one level of
> nestiness less: ~(gnu services)~.
>
> IMO, ~(gnu home services ...)~ would be a good choice if we use ~(gnu
> system services)~ for system services.

Regarding naming, I agree that what you propose would be more
consistent, but we’re not going to rename Guix System modules just now.
:-)

So, from a purely cosmetic standpoint, I’d still prefer (gnu home
services …) rather than (gnu home-services …).  It’s more consistent
with the rest IMO.

Thanks,
Ludo’.

Re: On the naming of System and Home services modules.

[Ludovic Courtès]

Hi,

Xinglu Chen  skribis:

> Some services might be useful to have in both Guix System and Guix Home;
> for instance, Guix System currently has a service for configuring
> Syncthing, and I think it makes sense to also have one for Guix Home,
> this would mean that people not using Guix System (me :-)) could also
> have Guix manage Syncthing.  With the current approach, we would have to
> copy and paste quite a bit of code, and if the Syncthing service for
> Guix System changes, then the one for Guix Home might have to change as
> well.

Silly question, but why do we need to have two different configuration
record types in the first place?

Sharing configuration between Home and System sounds important to me: it
means users can easily move services from one to the other, which is
pretty big deal.  It also means we’d have much less code to maintain.

Would that be feasible?  (Apologies if this has already been discussed!)

Also, I proposed earlier a possible way to generate a Home service type
from the corresponding System service type—or, IOW, to generate a Home
service type graph from the System graph.  Does that sound feasible?

Thanks,
Ludo’.

guix environment --load vs. --file inconsistency

[Attila Lendvai]

dear all,

i was writing the documentation of a guix.scm file, and i realized that there's 
an inconsistency among the three most used commands in this context:

so, there's:
- guix build --file
- guix package --file

and then there's:
- guix environment --load
- guix pack # has neither

i'd propose to change guix environment to also use --file, but maybe i'm 
overlooking something, so, please speak up if you think it's a bad idea!

i never used guix pack, but maybe that also deserves a --file argument?

- attila
PGP: 5D5F 45C7 DFCD 0A39

Re: Merging wip-guix-home to master

[Katherine Cox-Buday]

Andrew Tropin  writes:

> The core part of Guix Home project has been moved from rde
> repository[fn:1] to wip-guix-home branch of guix repository.
>
> I'm about a week on wip-guix-home branch completely and Guix Home works
> fine.  There are no any major issues on rde-devel and guix-devel mailing
> lists and it seems that branch is ready to be merged.

I just want to thank you for the work. I don't think I'll use this everywhere, 
but it is definitely going to be helpful in some environments. Thank you!

-- 
Katherine

Re: “What’s in a package”

[Katherine Cox-Buday]

Ludovic Courtès  writes:

> I agree.  Like Konrad and you wrote, it’s good that we can all have our
> quick-and-dirty packages in personal channels, and it’s good that these
> are separate channels.

Definitely! I would also encourage Guix to adopt some of the tools that make 
creating quick, but bad, packages easier. E.g. builds that patch elfs with the 
library paths of their dependencies.

> I’m not sure how tooling could help in the way of making quick packages,
> but it’s worth exploring.  Examples that come to mind are: merging the
> npm importer that currently lives in a branch, and providing a generic
> “guix import upstream” importer that would figure out as much as
> possible so that one doesn’t have to start from a blank page.

That sounds like a great start. I tossed out some other ideas elsewhere in the 
thread. Most of them involve meta-inspection of the package, Guix ecosystem, 
runtime environment and logs. It would be nice in general to have a kind of 
"agent" that you could run repeatedly over the course of packaging that would 
suggest next steps on ~stderr~ and next logical packaging definition on 
~stdout~. Kind of like pair-programming with Guix :)

It would perform different operations dependent on what stage in the life-cycle 
the package is at, i.e. ~import~ when no package definition exists, build when 
one does, and possibly running the result in a container when the package build 
succeeds.

E.g. your PyTorch example, starting from scratch (note: ~guix import~ may not 
always feel like the right command to invoke in this example. This may be some 
larger concept than import; also, the example always redirects to package.scm 
for brevity, but the user would probably want to look at it first):

#+begin_example
  $ guix import upstream pytorch

  stderr: This looks like it might be python package (heuristics.scm:123 - 
package name starts with py), try this instead:
  stdout: guix import upstream pypi pytorch

  $ guix import upstream pypi pytorch | tee package.scm

  $ guix import upstream package.scm | tee package.scm

  stderr: downloading...
  stderr: It looks like this fails to build because it's missing autoconf 
(heuristics.scm:133 - grepping build output found a missing autoconf error). 
Try adding it as a native-input.
  stdout: (package definition with imports defined and native-input modified)

  $ guix import upstream package.scm

  stderr: downloading...
  stderr: It looks like this package comes with binaries that are available as 
Guix packages (heuristics.scm:143 - unpacking source includes binary or object 
files, heuristics.scm:153 - bundled files match output of known packages). Try 
this package definition instead:
  stdout: (package definition with suggested inputs and overridden phases to 
remove the binaries from the download)

  $ guix import upstream package.scm | tee package.scm

  stderr: It looks like this package vendors libraries that are available as 
Guix packages (heuristics.scm:163 - unpacking source includes vendored 
libraries, heuristics.scm:153 - bundled files match output of known packages). 
Try this package definition instead:
  stdout: (package definition with suggested inputs and overridden phases to 
remove the vendored libraries from the download)

  $ guix import upstream package.scm | tee package.scm
  
  stderr: It looks like this package searches XDG_DATA_DIRS for some files 
(heuristics.scm:163 - grep an strace of a containerized run of the output). Try 
this package definition instead:
  stdout: (package definition with ~native-search-paths~ defined)
#+end_example

etc., etc. Typing that out, it feels dangerously close to Microsoft's Clippy, 
but hopefully more helpful :)

Heuristics, by definition, wouldn't be correct all the time, but this kind of 
thing could help new contributors (or experienced contributors with bad 
memories like me!), and in some cases actually do some of the programming.

And every time someone comes to the mailing list or IRC with a question, we can 
ask ourselves if this is a common question, and maybe create a new heuristic.

-- 
Katherine

Merging wip-guix-home to master

[Andrew Tropin]

The core part of Guix Home project has been moved from rde
repository[fn:1] to wip-guix-home branch of guix repository.

I'm about a week on wip-guix-home branch completely and Guix Home works
fine.  There are no any major issues on rde-devel and guix-devel mailing
lists and it seems that branch is ready to be merged.

My guix describe looks like:
--8<---cut here---start->8---
Generation 114  Sep 17 2021 13:33:55(current)
  rde 31f8003
repository URL: https://git.sr.ht/~abcdw/rde
branch: without-guix-home
commit: 31f800353a781cef25fc80c05ad824a068a049c8
  guix a2324d8
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: wip-guix-home
commit: a2324d8b56eabf8117bca220a507cc791edffd2e
--8<---cut here---end--->8---


There is a discussion[fn:2] on moving home services to (gnu services
...)  modules, which is likely to happen, but it's possible to do the
migration relatively painless by re-exporting necessary symbols in
(gnu home-services ...) at first and removing them completely later.

Another important part of the work related to Guix Home project is
covering related modules and cli with tests, but it can be done in
parallel and is not a blocker for merging.

* Footnotes

[fn:1] https://git.sr.ht/~abcdw/rde

[fn:2] https://lists.gnu.org/archive/html/guix-devel/2021-09/msg00169.html


signature.asc
Description: PGP signature

Re: “What’s in a package”

[Ludovic Courtès]

Hi Katherine,

Katherine Cox-Buday  skribis:

> This is perhaps a rehash of the "worse is better"[2] conversation, but
> I often struggle with deciding whether to do things the "fast" way, or
> the "correct" way. I think when your path is clear, the correct way
> will get you farther, faster. But when you're doing experiments, or
> exploratory programming, being bogged down with the "correct" way of
> doing things (i.e. Guix packages) might take a lot of time for no
> benefit. E.g. maybe you end up packaging a cluster of things that you
> find out don't work out for you. Of course the challenge is: if you
> choose the fast way, and it works out, do you got back to do it the
> correct way so that you're on sound footing?

I can very much relate to this.  PyTorch was one case where I hesitated
between the “fast way” and the “correct way”; I chose the latter
thinking that it would probably be beneficial to others, otherwise you
wouldn’t have heard about it.  ;-)

> Bringing this back to Guix, and maybe the GNU philosophy, it has been
> very helpful for me to be able to leverage the flexibility of Guix to
> occasionally do things the "fast" way, perhaps by packaging a
> binary. Paradoxically, it has allowed me to stay within the Guix and
> free software ecosystem. In my opinion, flexibility is key to growing
> the ecosystem and community, and I would encourage Guix as a project
> to take every opportunity to give the user options.

I agree.  Like Konrad and you wrote, it’s good that we can all have our
quick-and-dirty packages in personal channels, and it’s good that these
are separate channels.

I’m not sure how tooling could help in the way of making quick packages,
but it’s worth exploring.  Examples that come to mind are: merging the
npm importer that currently lives in a branch, and providing a generic
“guix import upstream” importer that would figure out as much as
possible so that one doesn’t have to start from a blank page.

Thanks,
Ludo’.