Re: "guix potluck", a moveable feast

[Chris Marusich]

l...@gnu.org (Ludovic Courtès) writes:

>>> To work around that, I think the server should resolve package
>>> specifications when the potluck.scm file is submitted, and insert each
>>> package in the Guix package graph of the moment.  Does that make sense?
>>> Maybe that’s what you were describing when you talk about rewriting
>>> potluck.scm files so?
>>
>> When you say "insert each package in the Guix package graph," do you
>> mean, "add the package definition to the Guix source tree"?
>
> No no, it’s a separate source tree.  I mean add the potluck packages to
> the graph as per GUIX_PACKAGE_PATH.

OK, that makes sense.

>> What if "the potluck" maintained a pointer to the version (i.e., the
>> commit) of the Guix package definitions that it uses as its "base"?
>> From time to time, the potluck could update its pointer to point to a
>> more recent version of Guix's package definitions.  In this way, every
>> version of the potluck would precisely specify the dependencies of all
>> the packages in that version of the potluck, including any transitive
>> dependencies that ultimately come from the official Guix package
>> definitions (as defined in the "base" version); there would be no
>> surprising version drift.  I wonder if that would work?
>
> Then there’s the problem that Mark pointed out earlier, which is that it
> would force users to use a specific set of dependencies, possibly not
> current, when using the potluck.
>
> I think it’s nicer if both repos are decoupled, although that means we
> have to pay attention to version issues when the potluck is referring to
> packages provided by Guix.

I've reconsidered, and I think what I suggested is probably not a good
idea.  The result of pinning a package's dependencies to specific
versions tends to be that it becomes harder to update those
dependencies.  And when finally somebody does manually update those
dependencies (which is not often), it causes breakage which is difficult
to fix, but which would have been easier to fix if it had been noticed
sooner.  So, decoupling in the way you suggest is probably better.
Reproducible builds can still be achieved by specifying the precise
version (i.e., commit) of the potluck and also of the Guix package
definitions.

>> What if someone wants to add a package definition to the Guix source
>> tree which depends on a package that is defined in the potluck?
>
> I guess we wouldn’t allow that in the Guix repo proper.

That makes sense.  If a package can be imported wholesale into the
collection of Guix package definitions (instead of the potluck), then
that seems like the right thing to do.  Situations where I can imagine
that not being possible are, for example, where the software defined by
a potluck package is not available under a free license.

-- 
Chris


signature.asc
Description: PGP signature

Re: "guix potluck", a moveable feast

[Christopher Allan Webber]

Andy Wingo writes:

> Christopher Webber asks about breakage due to version skew between peer
> channels and channels and Guix itself.  I think I would like to just
> ignore this problem for now: if you add channels and things break
> somehow due to an update in Guix or an update in some channel, then the
> workaround is to disable channels until developers fix things.

It isn't an impossible problem to get around anyway, at least from a
channel to channel perspective... since we're building channels around
git repositories, *eventually* we could allow a channel to include
another channel as a dependency with a specific hash.  But we can work
on that once we get things working.

Re: "guix potluck", a moveable feast

[Andy Wingo]

Hi :)

Thanks all for review; comments and suggestions very welcome.  Choosing
this message to reply to.

On Sun 02 Apr 2017 01:05, l...@gnu.org (Ludovic Courtès) writes:

> Andy Wingo  skribis:
>
>>   (1) Install Guix as a user.  (This needs to be easier.)
>>   (2) guix channel add potluck https://gitlab.com/potluck/potluck master
>>   (3) guix channel enable potluck
>
> So users would see the union of independent potluck “dishes”, right?

Yes I think so: a union of all potluck "dishes" with the Guix package
set as well.

Christopher Webber asks about breakage due to version skew between peer
channels and channels and Guix itself.  I think I would like to just
ignore this problem for now: if you add channels and things break
somehow due to an update in Guix or an update in some channel, then the
workaround is to disable channels until developers fix things.

> The sandbox would have transitive access to a lot of modules; I wonder
> if this might somehow make it easier to escape the sandbox, by
> increasing the attack surface.  For instance,
>
>   (source-module-closure '((guix packages)) #:select? (const #t))

I think the strategy here would be to avoid making a sandbox binding set
that is "unsafe".  Having source-module-closure in that binding set
would seem to make it unsafe.

> I think the server should resolve package specifications when the
> potluck.scm file is submitted, and insert each package in the Guix
> package graph of the moment.  Does that make sense?  Maybe that’s what
> you were describing when you talk about rewriting potluck.scm files
> so?

Yes I think this is a good idea.

Incidentally I am now thinking that all the potluck stuff should be in a
potluck dir; you run "guix potluck init" and it makes

  potluck/README.md
  potluck/mypackage.scm

and the .scm files should evaluate to a single package, like:

  (import-packages ...)
  (package
   ...)

The rewrite would create files like:

  gnu/packages/potluck/gitlab-com-wingo-foo-master/mypackage.scm
  gnu/packages/potluck/gitlab-com-wingo-foo-master/mypackage2.scm

These files would look like:

  (define-module (gnu packages potluck gitlab-com-wingo-foo-master mypackage)
#:pure
;; The sandbox.  We've already verified that the user code works in
;; this sandbox when we rewrite the package, so this allows us to
;; provide a stable language for sandbox packages
#:use-module (guix potluck environment)
;; The individual module imports, resolved by channel manager.
#:use-module ((gnu packages guile) #:select (guile))
...
#:export (mypackage))

  (define mypackage
(package ))

You can compile files from the channel, so guix startup time will be
only minimally affected.

>> There is a particular concern about staging: there is staged Scheme code
>> in these modules that runs inside build processes in guix-daemon.  I
>> don't have any nice solution here.
>
> What’s the problem anyway?  The build environment is a “sandbox” so it’s
> not a problem if staged code attempts to do nasty things.

I guess so, yeah.

Andy

Re: "guix potluck", a moveable feast

[Ludovic Courtès]

Chris Marusich  skribis:

> l...@gnu.org (Ludovic Courtès) writes:
>
>> Beside, related to Chris’ comment, I’m a bit concerned about versioning
>> in such a widely distributed repo.  The package graph in Guix has zero
>> degrees of liberty: every package is connected to other packages; every
>> Guix user sees the exact same graph.
>>
>> Here, we’d have to be more flexible and allow potluck.scm files to just
>> say “import guile” or “import guile@2.0”; “import guile” might provide
>> 2.0 on a machine running an older Guix, and it might give 2.2.9 on an
>> up-to-date machine.
>>
>> IOW, we’re no longer describing one specific graph, but instead
>> describing a family of graphs with some constraints.  The benefits are
>> decentralization, but the main drawback is non-reproducibility: the
>> result would depend on the user machine’s initial state.
>>
>> To work around that, I think the server should resolve package
>> specifications when the potluck.scm file is submitted, and insert each
>> package in the Guix package graph of the moment.  Does that make sense?
>> Maybe that’s what you were describing when you talk about rewriting
>> potluck.scm files so?
>
> When you say "insert each package in the Guix package graph," do you
> mean, "add the package definition to the Guix source tree"?

No no, it’s a separate source tree.  I mean add the potluck packages to
the graph as per GUIX_PACKAGE_PATH.

> What if "the potluck" maintained a pointer to the version (i.e., the
> commit) of the Guix package definitions that it uses as its "base"?
> From time to time, the potluck could update its pointer to point to a
> more recent version of Guix's package definitions.  In this way, every
> version of the potluck would precisely specify the dependencies of all
> the packages in that version of the potluck, including any transitive
> dependencies that ultimately come from the official Guix package
> definitions (as defined in the "base" version); there would be no
> surprising version drift.  I wonder if that would work?

Then there’s the problem that Mark pointed out earlier, which is that it
would force users to use a specific set of dependencies, possibly not
current, when using the potluck.

I think it’s nicer if both repos are decoupled, although that means we
have to pay attention to version issues when the potluck is referring to
packages provided by Guix.

> What if someone wants to add a package definition to the Guix source
> tree which depends on a package that is defined in the potluck?

I guess we wouldn’t allow that in the Guix repo proper.

Ludo’.

Re: "guix potluck", a moveable feast

[Chris Marusich]

l...@gnu.org (Ludovic Courtès) writes:

> Beside, related to Chris’ comment, I’m a bit concerned about versioning
> in such a widely distributed repo.  The package graph in Guix has zero
> degrees of liberty: every package is connected to other packages; every
> Guix user sees the exact same graph.
>
> Here, we’d have to be more flexible and allow potluck.scm files to just
> say “import guile” or “import guile@2.0”; “import guile” might provide
> 2.0 on a machine running an older Guix, and it might give 2.2.9 on an
> up-to-date machine.
>
> IOW, we’re no longer describing one specific graph, but instead
> describing a family of graphs with some constraints.  The benefits are
> decentralization, but the main drawback is non-reproducibility: the
> result would depend on the user machine’s initial state.
>
> To work around that, I think the server should resolve package
> specifications when the potluck.scm file is submitted, and insert each
> package in the Guix package graph of the moment.  Does that make sense?
> Maybe that’s what you were describing when you talk about rewriting
> potluck.scm files so?

When you say "insert each package in the Guix package graph," do you
mean, "add the package definition to the Guix source tree"?

What if "the potluck" maintained a pointer to the version (i.e., the
commit) of the Guix package definitions that it uses as its "base"?
From time to time, the potluck could update its pointer to point to a
more recent version of Guix's package definitions.  In this way, every
version of the potluck would precisely specify the dependencies of all
the packages in that version of the potluck, including any transitive
dependencies that ultimately come from the official Guix package
definitions (as defined in the "base" version); there would be no
surprising version drift.  I wonder if that would work?

What if someone wants to add a package definition to the Guix source
tree which depends on a package that is defined in the potluck?

-- 
Chris


signature.asc
Description: PGP signature

Re: "guix potluck", a moveable feast

[Ludovic Courtès]

Hello!

Andy Wingo  skribis:

> As an interlude, here is how a user would enter an environment that has
> a potluck package "foo" using Guix (using a pack is also possible).  We
> start with setup steps:
>
>   (1) Install Guix as a user.  (This needs to be easier.)
>   (2) guix channel add potluck https://gitlab.com/potluck/potluck master
>   (3) guix channel enable potluck

So users would see the union of independent potluck “dishes”, right?

> A packaging language for stability and security
> ---
>
> So how do packages enter the potluck channel?  Good question, fictional
> reader!  This is the tricky bit.  There are some concerns here:
>
>   (1) The Guix API is not stable and has no plans to be stable.  This
>   works great for now because all packages are in one atomic
>   repository and people work on making the whole thing make sense
>   together.  One of the goals of the potluck effort is to
>   decentralize things a bit, so we have an impedance mismatch
>   between potluck packages and Guix itself.
>
>   (2) Potluck package definitions will live in many different git
>   repositories across the internet, and anyone should be able to
>   make a potluck package.  Some potluck package authors will be
>   malicious.  They could:
>
>1. Damage the server that manages the potluck channel
>
>2. Damage the users that run Guix commands with the potluck
>   channel enabled
>
>3. Damage the users that install potluck packages
>
>   I think we need to forget about 3, for now at least.  (Flatpak
>   solves this, more or less; Guix has ongoing work to do here I
>   think.)
>
> Both of these large issues point to the need for careful design of the
> language that potluck packages are written in.  The language that Guix
> packages are written in is inappropriate because of (1).  In particular
> we should not depend on which module a package comes from, and what
> identifier binds any given package.  For (2), packages are currently
> written in full Scheme, staged between the Guix command itself and the
> sandbox that runs inside guix-daemon.  Full Scheme might be OK in the
> daemon but it's not OK in the Guix command itself.
>
> Concretely I would propose that the language that potluck files are
> written in is like this:
>
>   (1) It's code, not inert data.
>
>   (2) It's a subset of Scheme, like core Guix packages.
>
>   (3) The general structure looks like this:
>
>   (import-guix-packages ((guile "guile@2.0")
>  (glibc "glibc")))
>   (import-potluck-packages ((foo "foo")))
>
>   (define bar
> (package
>   (name "guile-bar")
>   (version "1.0.0")
>   (build-system gnu-build-system)
>   (inputs `(("guile" ,guile))
>   )))

That makes sense to me.

The sandbox would have transitive access to a lot of modules; I wonder
if this might somehow make it easier to escape the sandbox, by
increasing the attack surface.  For instance,

  (source-module-closure '((guix packages)) #:select? (const #t))

contains (system foreign).  That’s probably more of a topic for
guile-devel though.

Beside, related to Chris’ comment, I’m a bit concerned about versioning
in such a widely distributed repo.  The package graph in Guix has zero
degrees of liberty: every package is connected to other packages; every
Guix user sees the exact same graph.

Here, we’d have to be more flexible and allow potluck.scm files to just
say “import guile” or “import guile@2.0”; “import guile” might provide
2.0 on a machine running an older Guix, and it might give 2.2.9 on an
up-to-date machine.

IOW, we’re no longer describing one specific graph, but instead
describing a family of graphs with some constraints.  The benefits are
decentralization, but the main drawback is non-reproducibility: the
result would depend on the user machine’s initial state.

To work around that, I think the server should resolve package
specifications when the potluck.scm file is submitted, and insert each
package in the Guix package graph of the moment.  Does that make sense?
Maybe that’s what you were describing when you talk about rewriting
potluck.scm files so?

> There is a particular concern about staging: there is staged Scheme code
> in these modules that runs inside build processes in guix-daemon.  I
> don't have any nice solution here.

What’s the problem anyway?  The build environment is a “sandbox” so it’s
not a problem if staged code attempts to do nasty things.

> A potluck channel manager
> -
>
> The "guix potluck manage-channel" command manages a registry of sources
> of potluck definitions and turns them into a git branch of package
> files.  This is the web service.
>
> The idea is that as a developer, you should be able to do:
>
>   guix potluck add https://gitlab.com/wingo/foo master
>
> This causes the client 

Re: "guix potluck", a moveable feast

[ng0]

Christopher Allan Webber transcribed 1.8K bytes:
> Andy Wingo writes:
> 
> > Hi!
> 
> Hi!
> 
> > potluck.guixsd.org needs to be isolated from other hosts because it will
> > load potluck.scm files from untrusted sources; we hope the sandbox works
> > but we need a bit of defense-in-depth.
> 
> Well now I see the motivation behind (ice-9 sandbox) ... :)
> 
> > As I mentioned, I think it would be nice to be able to install some
> > potluck packages directly from git, without requiring those packages to
> > make releases and update the potluck.scm.  But until then, we can make
> > it so that the source is fixed in the potluck.scm as it is with other
> > Guix packages, and therefore that any update to potluck.scm in the
> > source git branch registered with potluck.guixsd.org constitutes a new
> > release which replaces the old one.  A developer should signal
> > potluck.guixsd.org about the update via a re-invocation of "guix potluck
> > add".  Maybe "guix potluck add" could remember the branch, dunno.
> >
> > Anyway!  The result of the "guix potluck channel-manager" is a stream of
> > guix modules as a continually updated git tree -- a guix channel.  I am
> > thinking that we need to rewrite these files to be more "normal" -- like
> > starting with a (define-module), but a #:pure module and an appropriate
> > set of imports to enforce the sandbox.  We should be able to compile
> > this module, to prevent the potluck channel from slowing things down.
> > So basically the channel-manager rewrites the potluck.scm files.
> 
> It sounds nice!
> 
> One challenge though... what do we do about multiple channels
> introducing version skew?  (Maybe I'm abusing that term?)  This isn't
> something we've dealt with before in Guix... if my channel adds
> something that depends on your channel's package definition, do I
> explicitly set a revision for your channel?  Otherwise else, your
> channel could change as you upgrade your software version, and that
> might unexpectedly break my channel...
> 
I think there's something we can learn from Gentoo here.
You might or might not know their 'overlays' (I don't know the exact
gentoo rfc when they introduced them but it's been very long ago).
They do this kind of thing. They have no opinion other than that
'portage', in Guix terms 'master branch at savannah', takes the highest
priority by default. You can explicitly change this.
If you start using a specific overlay and use a software recipe from it
which does exists in multiple overlays, it's like this:
- if you don't edit the specific file which tells portage about this
  recipe, it picks the highest stable version.
- if you get rid of stable and allow everything, it picks the highest
  version and has no opinion from where it comes.
- if you specifically point out the overlay for it, it picks the version
  from there, no questions asked. well actually it asks questions if you
  tell it to do so ;)

So I think we could have some way to define the priority of the channel,
a value to define stable / unstable (similar to Gentoo's "experimental"
and "official" classification of overlays).

No warranty that this is accurate, I tried to explain Gentoo overlays
without assuming too much or explaining too much of it.
In case I misunderstood the question, enjoy your 2 minutes of
'Things Gentoo did excellent'.

Re: "guix potluck", a moveable feast

[Christopher Allan Webber]

Andy Wingo writes:

> Hi!

Hi!

> potluck.guixsd.org needs to be isolated from other hosts because it will
> load potluck.scm files from untrusted sources; we hope the sandbox works
> but we need a bit of defense-in-depth.

Well now I see the motivation behind (ice-9 sandbox) ... :)

> As I mentioned, I think it would be nice to be able to install some
> potluck packages directly from git, without requiring those packages to
> make releases and update the potluck.scm.  But until then, we can make
> it so that the source is fixed in the potluck.scm as it is with other
> Guix packages, and therefore that any update to potluck.scm in the
> source git branch registered with potluck.guixsd.org constitutes a new
> release which replaces the old one.  A developer should signal
> potluck.guixsd.org about the update via a re-invocation of "guix potluck
> add".  Maybe "guix potluck add" could remember the branch, dunno.
>
> Anyway!  The result of the "guix potluck channel-manager" is a stream of
> guix modules as a continually updated git tree -- a guix channel.  I am
> thinking that we need to rewrite these files to be more "normal" -- like
> starting with a (define-module), but a #:pure module and an appropriate
> set of imports to enforce the sandbox.  We should be able to compile
> this module, to prevent the potluck channel from slowing things down.
> So basically the channel-manager rewrites the potluck.scm files.

It sounds nice!

One challenge though... what do we do about multiple channels
introducing version skew?  (Maybe I'm abusing that term?)  This isn't
something we've dealt with before in Guix... if my channel adds
something that depends on your channel's package definition, do I
explicitly set a revision for your channel?  Otherwise else, your
channel could change as you upgrade your software version, and that
might unexpectedly break my channel...

"guix potluck", a moveable feast

[Andy Wingo]

Hi!

Following on thoughts from earlier this month that Guix is the guildhall
that we always wanted, I think I have more of a plan.


s/guildhall/potluck/


Firstly, a name change: the guidhall was a somewhat unified, curated
thing that was Guile-specific, whereas I think I think we need to take
this feast outdoors and move it around a bit.  I would like to use the
term "potluck" to refer to the offering of tasty home-hacked goods, the
composition of which is but loosely planned.  This is inspired by the
"potluck" dishes that we would hack up to celebrate the Guile 2.0
release anniversary.

I don't want to take the metaphor too far of course; I think there was a
forced aspect around the guildhall name that never really became
natural.

Also, the potluck facility is just one part of the system.  Firstly we
use Guix itself as the package manager.  Secondly we build on the
upcoming "guix channel" facility.  (In its most basic form, a channel is
just a git branch somewhere that contains package files.)  The job of
the potluck facility is just cultivating the git branch that is the
"potluck channel".

As an interlude, here is how a user would enter an environment that has
a potluck package "foo" using Guix (using a pack is also possible).  We
start with setup steps:

  (1) Install Guix as a user.  (This needs to be easier.)
  (2) guix channel add potluck https://gitlab.com/potluck/potluck master
  (3) guix channel enable potluck

Then:

  (4) guix environment --ad-hoc foo

Assuming "foo" is a package that's in the potluck channel.  Basically
the potluck channel augments the set of available packages.  This is
handled by the channel mechanism.


A packaging language for stability and security
---

So how do packages enter the potluck channel?  Good question, fictional
reader!  This is the tricky bit.  There are some concerns here:

  (1) The Guix API is not stable and has no plans to be stable.  This
  works great for now because all packages are in one atomic
  repository and people work on making the whole thing make sense
  together.  One of the goals of the potluck effort is to
  decentralize things a bit, so we have an impedance mismatch
  between potluck packages and Guix itself.

  (2) Potluck package definitions will live in many different git
  repositories across the internet, and anyone should be able to
  make a potluck package.  Some potluck package authors will be
  malicious.  They could:

   1. Damage the server that manages the potluck channel

   2. Damage the users that run Guix commands with the potluck
  channel enabled

   3. Damage the users that install potluck packages

  I think we need to forget about 3, for now at least.  (Flatpak
  solves this, more or less; Guix has ongoing work to do here I
  think.)

Both of these large issues point to the need for careful design of the
language that potluck packages are written in.  The language that Guix
packages are written in is inappropriate because of (1).  In particular
we should not depend on which module a package comes from, and what
identifier binds any given package.  For (2), packages are currently
written in full Scheme, staged between the Guix command itself and the
sandbox that runs inside guix-daemon.  Full Scheme might be OK in the
daemon but it's not OK in the Guix command itself.

Concretely I would propose that the language that potluck files are
written in is like this:

  (1) It's code, not inert data.

  (2) It's a subset of Scheme, like core Guix packages.

  (3) The general structure looks like this:

  (import-guix-packages ((guile "guile@2.0")
 (glibc "glibc")))
  (import-potluck-packages ((foo "foo")))

  (define bar
(package
  (name "guile-bar")
  (version "1.0.0")
  (build-system gnu-build-system)
  (inputs `(("guile" ,guile))
  )))

I.e. we have a special mechanism to import packages by name using
specification->package.  We can define packages using Guix's "package"
form, and a number of standard Guix bindings are available to the code
(license:gplv3+, gnu-build-system, etc).

There is a particular concern about staging: there is staged Scheme code
in these modules that runs inside build processes in guix-daemon.  I
don't have any nice solution here.  Simply sandboxing the host Scheme
will be fine enough for me, using (ice-9 sandbox) if that lands soon
(see
https://lists.gnu.org/archive/html/guile-devel/2017-03/msg00111.html).
This requires Guile 2.2.1 (when it is released).

I can see an argument to support more simple package definition,
specifically that it should be easier to build Guile-only projects from
git -- skipping autotools, skipping tarballs.  But we should be able to
support tarball builds containing C or anything else, so the full
package capabilities are needed.  I think a first