bug#22629: Towards a new 'guix pull'

[Nils Gillmann]

Ludovic Courtès transcribed 2.0K bytes:
> Hello Guix!
> 
> l...@gnu.org (Ludovic Courtès) skribis:
> 
> > Here’s a series of improvements that I think we should make in ‘guix
> > pull’:
> >
> >   • Use Git instead of downloading a whole snapshot every time.  The Git
> > checkout would be kept in ~/.cache/guix/pull/checkouts, say.
> 
> That was done a while back.   :-)
> 
> >   • Build & install not only Scheme code, but also locales and the Info
> > manual.
> 
> … and also the C++ bits.
> 
> With the work done in , we can build all of
> Guix and its dependencies when doing ‘guix pull’, which brings us closer
> to what the ‘guix’ package provides.
> 
> Thus we can at least partly address the issue described in
> , i.e., have
> ‘guix pull’ install not just a bunch of .go files but also guile-git,
> guile-ssh, gnutls, and Guile itself.
> 
> To do that, we need a “paradigm shift” (sounds cute, no? :-)).
> Currently, ‘scripts/guix’ fiddles with its %load-path and adds
> ~/.config/guix/latest if it exists.  The new ‘guix pull’ should provide
> not just .go files but also a ‘guix’ command referring to the latest
> Guile, Guile-Git, etc.
> 
> To achieve this, I think the new approach would be to have:
> 
>   export PATH=$HOME/.config/guix/current/bin:$PATH
>   export INFOPATH=$HOME/.config/guix/current/share/info:$INFOPATH
>   …
> 
> ‘guix pull’ would update ~/.config/guix/current, which would contain the
> ‘guix’ command, all the .go files, locales, the Info manual, and ideally
> ‘guix-daemon’ as well.
> 
> ~/.config/guix/current could be a profile, or something that resembles a
> profile.
> 
> For a while ‘guix pull’ will have to keep updating ~/.config/guix/latest
> as well, for users running an old ‘guix’ command.
> 
> Thoughts?

First thought: Woo! Good progress!
On your propossed steps, sounds good to me.

> Ludo’.
> 
> 
> 

bug#22629: Towards a new 'guix pull'

[Ludovic Courtès]

Hello Guix!

l...@gnu.org (Ludovic Courtès) skribis:

> Here’s a series of improvements that I think we should make in ‘guix
> pull’:
>
>   • Use Git instead of downloading a whole snapshot every time.  The Git
> checkout would be kept in ~/.cache/guix/pull/checkouts, say.

That was done a while back.   :-)

>   • Build & install not only Scheme code, but also locales and the Info
> manual.

… and also the C++ bits.

With the work done in , we can build all of
Guix and its dependencies when doing ‘guix pull’, which brings us closer
to what the ‘guix’ package provides.

Thus we can at least partly address the issue described in
, i.e., have
‘guix pull’ install not just a bunch of .go files but also guile-git,
guile-ssh, gnutls, and Guile itself.

To do that, we need a “paradigm shift” (sounds cute, no? :-)).
Currently, ‘scripts/guix’ fiddles with its %load-path and adds
~/.config/guix/latest if it exists.  The new ‘guix pull’ should provide
not just .go files but also a ‘guix’ command referring to the latest
Guile, Guile-Git, etc.

To achieve this, I think the new approach would be to have:

  export PATH=$HOME/.config/guix/current/bin:$PATH
  export INFOPATH=$HOME/.config/guix/current/share/info:$INFOPATH
  …

‘guix pull’ would update ~/.config/guix/current, which would contain the
‘guix’ command, all the .go files, locales, the Info manual, and ideally
‘guix-daemon’ as well.

~/.config/guix/current could be a profile, or something that resembles a
profile.

For a while ‘guix pull’ will have to keep updating ~/.config/guix/latest
as well, for users running an old ‘guix’ command.

Thoughts?

Ludo’.

bug#22629: Towards a new 'guix pull'

[Leo Famulari]

On Wed, Feb 22, 2017 at 07:57:06AM +, Pjotr Prins wrote:
> To keep things simple I propose a channel simply reflects a fixed
> version of a git tree of a Guix package list. I.e., the exact *state*
> of the combined list of packages is what we support in a channel. The
> git checkout can be stored in something like

You shared *a lot* of ideas following this paragraph, but I think that
from here we can start thinking about how to incrementally evolve from
the current `guix pull` to something better.

Currently, you can `guix pull` any commit of GNU Guix with `guix pull
--url` [0]. This feature depends on the Git server implementation's HTTP
interface, but I bet it's a common feature, and it can already be used
with other Git repositories besides our Savannah repo. This achieves the
goal of channels, "easy specialization of deployments". Well, "easy" is
a matter of degrees ;)

I was thinking we could start by keeping `guix pull` the same from the
user's point of view, but change it to support Git in addition to
downloading a tarball. We already have code for this in (guix
git-download) which uses the Git command-line tool. [1]

The next step would be to keep some metadata about which channel the
last pull came from, so that subsequent pulls would point to it.

What do people think?

I haven't wrapped my head around all the proposed changes. I think it
will be best to keep the goal of "easy specialization of deployments" in
mind and work towards it incrementally.

[0]
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=8a9cffb202414b20081910115ba76402924bdcdd

[1] In the long run, we should ditch the Git CLI and use something like
guile-git:
https://gitlab.com/amirouche/guile-git


signature.asc
Description: PGP signature

bug#22629: Towards a new 'guix pull'

[Pjotr Prins]

I wrote up a 'white paper' on a new implementation of Guix channels.
The easy on the eyes version you can read here:

https://github.com/pjotrp/guix-notes/blob/master/proposals/channel.org

* On Channels

Related to debbugs https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22629 and 
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22883#103

** Updating package definitions through channels

*** Introduction

The main purpose of creating Guix channels is to allow users to use
different versions of the Guix package tree that differ from the
master branch, i.e., have people share packages with each other that
are not in the GNU Guix distribution itself.

Within Guix there are good reasons to handle such cases. For example,
to support an older package, such as, ruby 1.8.7 (currently in the
master tree, but could be removed), and to have different editions of
combinations of packages. This versioning we currently achieve by
explicitely working from git checkouts of trees - not trivial for
everyone. Channels will just make versioning easier and accessible for
everyone.

Outside Guix, channels can support also different combinations of
packages, e.g., such as for easily deploying different versions of the
GeneNetwork webservice (development/staging/testing/production
branches). Channels will also make it possible to switch testing off
in certain packages, for example, or to distribute data packages that
do not belong or are not wanted in GNU Guix.

In short, channels allow for easy specialization of deployments.

Also, channels and 'guix publish' distributed services can go hand in
hand in providing such specializations.

*** Usage

Proposed syntax for adding a channel

:  guix channel add channel-name URI branch

Examples

:  guix channel add guix-stable git://git.sv.gnu.org/guix.git stable
:  guix channel add guix-master git://git.sv.gnu.org/guix.git master
:  guix channel add ruby-1.8.7 git:URI/my-ruby-repo.git ruby-1.8.7
:  guix channel add genenetwork-development git:URI/genenetwork-staging.git 
development
:  guix channel add genenetwork-staging git:URI/genenetwork-staging.git staging

To keep things simple I propose a channel simply reflects a fixed
version of a git tree of a Guix package list. I.e., the exact *state*
of the combined list of packages is what we support in a channel. The
git checkout can be stored in something like

:  ~/.guix-channels/ruby-1.8.7/...

There will only be one directory for one channel. Channels are not
shared between users so they can go somewhere in $HOME.

To get to a certain state, or update a channel, we should allow for
tree updating based on a git SHA value, a tag or just the HEAD. Many
channels will go by HEAD, but it is unwanted because a tree may be in
an unstable and even unusable state. Especially for Guix itself we
should not be using HEAD - the current default with a 'guix pull'.
Ideally 'guix pull' should be tried by core developers before giving
it to everyone. I think we can figure out the last tagged release and
have that as a default rather than HEAD. When someone wants HEAD it
should be explicit. So to get the actual HEAD the syntax can be:

:  guix channel update guix-master origin/HEAD

To get to a specific SHA value on origin

:  guix channel update guix-master 9cf265449f88f89c900e510fe29baba26f30159b

To get a named tagged release

:  guix channel update guix-master v0.12.0-8

To get the latest tagged release (default behaviour)

:  guix channel update guix-master

To switch channels for use

:  guix channel set guix-master

When a channel gets updated I think we ought to check it out fully
from scratch by default. Reason being that we can't deal with
conflicts and maybe changed upstream trees - git also allows rewriting
history and that can be nasty. If some person feels confident, we
could still add a switch

:  guix channel update guix-master --pull

or

:  guix channel update guix-master --checkout v0.12.0-8

which would do what you expect it to do.

Obviously all this behaviour reflects the way git behaves. I think it
is fine because if we start adding other transports (say https,
mercurial or darcs) it would be clear from the channel URI what it was
and we can change the commands.

** Channel metadata

The channel should describe its purpose and set certain prerequisites,
such as a description and the version of guile it requires. This can
be added as a channel-meta.scm file in the root of the source tree in
git. It could also point to one or more 'guix publish' servers.

** Compiling the package tree

So far, we have only discussed the package tree. To compile it we need
guile. One thing the current 'guix pull' does is compile guile to
compile the tree. This behaviour does not make use of substitute
servers, takes time and is somewhat error prone. I think what we
should do is reuse a guile already on the system or install it as a
binary using a substitute server and then compile the packages.

** Updating guix and guix-daemon

Now we come to updati

bug#22629: Towards a new 'guix pull'

[Ludovic Courtès]

Hello!

Here’s a series of improvements that I think we should make in ‘guix
pull’:

  • Use Git instead of downloading a whole snapshot every time.  The Git
checkout would be kept in ~/.cache/guix/pull/checkouts, say.

A related question is whether to use Git itself, which is pretty big
per ‘guix size’, or to use some libgit2 bindings such as
 (the closure of libgit2 is
435 MiB; that of Git is 761 MiB.)

  • Build & install not only Scheme code, but also locales and the Info
manual.

  • Have a “channel” mechanism, similar to ‘nix-channel’, that would
allow users to have several Guix variants available in parallel
instead of just “latest”.  Could work like this:

  guix channel add latest git://git.sv.gnu.org/guix.git master
  guix channel add stable git://git.sv.gnu.org/guix.git stable
  guix channel pull latest
  guix channel set latest
  # here i see the latest versions of everything
  guix channel set stable
  # and here everything is old but super stable ;-)

All 3 items can be done separately, I think.

Any takers?  :-)

Ludo’.

PS: I do not mention the issue of authenticating code here, which is
obviously very important and deserves to be treated separately.
Related to that is the question of making sure that what you think
is the latest version really is the latest version.  We need someone
to sign certificates saying what the latest commit ID of a repo is.
See the “The Update Framework” paper!