Re: Thoughts on stateful services in Guix

[Ludovic Courtès]

Hi Alex,

Thanks for sharing your thoughts and this tricky but important use case!

Alex Sassmannshausen  skribis:

> 3 A generalisation: the Stateful-Service Service
> 
>
>   State dumping and restoration *should* be generalisable.
>
>   It should normally consist of one or more operations of:
>   • running a special program to dump data, encrypting the dump, and
> storing it, together with the service revision hash, in a known
> location.
>   • tarring up a directory tree, encrypting the archive and storing it,
> together wtih the service revision hash, in a known location.
>
>   These operations should be able to be provided by a daemon, managed by
>   shepherd, which can be configured with a gpg key for encryption, a
>   mechanism for decryption (interactive or programmatic), a location for
>   storing/retrieving dumps (local or remote?), and a DSL for mapping
>   data dump / data restoration program invocations or file-system
>   locations to data dumps it already knows of.

Back in my Nix days, Wouter den Breejen worked on “S-Nix”, whose goal
was to explore how Nix could be extended to support state¹.  Back then,
I think the experiment was not considered fruitful, in part due to its
complexity, the implementation’s reliance on ext3cow, things like that.

There may still be good ideas to take from that, although it’s probably
more reasonable to start with something simple and practical, trying to
address a couple of concrete use cases before trying to generalize.
What you propose above sounds like it could be a good start!

Ludo’.

¹ “Managing state in a purely functional deployment model”, master
  thesis,
  .
  I couldn’t find a copy on-line so here’s one from my attic:
  .

Re: Thoughts on stateful services in Guix

[Giovanni Biscuolo]

Hi Alex,

I'm very interested in deploying "as stateless as possible" services
with Guix, thank you for sharing your thougts on this!

I'm still non able to contribute code fot this task... I'm working on it

Alex Sassmannshausen  writes:

> As a result of FOSDEM conversations today I felt inspired to put some
> thoughts on paper about an area where I think we currently run into
> complications.

I also think that deploying services in a functional way (ala Guix
packages to be clear) is _complicated_ and... it's a problem

to be clear: this complication it's *absolutely not* Guix specific, but
it's something that becomes very clear when you want to use the
declarative approach to "everithing" once you've got Guix :-O

(and after you were deluded by the promises from tools like Puppet and
alike)

> Not sure if it is appropriate to write this blog-post-ish contribution
> here, but as I don't have a blog, and it's about Guix development, I
> figured it might be OK.

maybe with a little help by someone more qualified than me this should
become a Guix blog post... or better be generalized ("how to write
staletess services") and included in the cookbook

anyway, I think your analisys is very helpful in discussing how Guix can
help fighting the "state dragon"

I've some comment and questions

> Best wishes,
>
> Alex
>
> 1 Introduction
> ══

[...]

>   Disciplined developers can implement many services in such a way that
>   their statefulness is delegated to other dedicated services.  In this
>   way the problem of statefulness can be isolated.  However, many
>   existing useful services /have not/ been implemented in such a way.

do you mean upstream?

[...]

from now on you used Drupal as an example: I do not know how Drupal
works, but I know other problematic web services :-)

> 2.3.1 Enter the state dragon
> 
>
>   But hang-on… if Mysql service is a dependency, and we store state in a
>   Mysql DB,

in this regard some time ago I found
http://gfxmonk.net/2015/01/03/nixos-and-stateless-deployment.html (see
"Lumps of impurity") a good explanation on what should be state and what
not; this is the relevant part:

--8<---cut here---start->8---
NixOS keeps the “OS” parts of the machine stateless, meaning that the
only state you need to manage is that which you create yourself. From
the top of my head (not an exhaustive list):

- /var, /tmp, /run: stateful, unmanaged

- each user’s $HOME directory: stateful, unmanaged (rarely used on a server, 
though)

- Users & Groups: stateless

- Installed software: stateless

- Installed services (using systemd): stateless

- All program configuration (i.e all of /etc): stateless

- Kernel & kernel modules, grub configuration: stateless (but requires a reboot 
to activate)

- Disk mounts: stateless

By “stateless”, I mean that the given item is generated entirely from
the pure nix expression of the system, and isn’t affected by any
previous state.
--8<---cut here---end--->8---

I was not able to find a similar description in some Guix
document... but it's OT now :-)

(...and software config stored in $HOME should be stateless, also;
guix-home-manager [1] is a _very_ interesting work on this topic)

storing state it's obviously the job of many services, what's bad it's
some software (Drupal?  Discourse for sure, Wordpress for sure) store
(part of?) configuration as status; some software provides a CLI tool
(usually as a afterthougt, e.g. wp-cli) to imperatively configure it,
but it's always treated as status

in other words: some (usually web) services "embeds" config with status
and provide one or more interface to configure, others clearly separates
config from status leaving the "burden" of writing configuration
(usually in /etc) to sysadmins... and that software is "easy" to
integrate as a stateless (and functional) Guix service

my question is: can we work around this very bad design choiche?  can we
patch the package and have stateless config for, e.g., Drupal?

I see this someway similar to "de-blobbing" our packages: how hard it's
to do depends on upstream... unfortunately not all developers agrees
that stateful config is a problem (in a similar way not all agrees that
distributing binary-blobs it's a problem)

>   what happens when we upgrade to a newer version of Drupal,
>   with a different schema?  Would this service instantiation change the
>   state of our system?  Could we still roll-back?

oh yes, schema migration of status is also a problem to manage when we
upgrade software... and usually we do it "by hand" (making backups of
status, ecc.)

schema migration is usual when status is stored in a database,
theoretically schema migration could be applyed also when storing state
in XML (YAML?) in files, but I've never seen this use case

your proposal (below) is interesting!

>   And remember that broken symlink we introduced in the package?  That
>   

Thoughts on stateful services in Guix

[Alex Sassmannshausen]

Hello,

As a result of FOSDEM conversations today I felt inspired to put some
thoughts on paper about an area where I think we currently run into
complications.

Not sure if it is appropriate to write this blog-post-ish contribution
here, but as I don't have a blog, and it's about Guix development, I
figured it might be OK.

Best wishes,

Alex

1 Introduction
══

  Guix is amazing.  A large part of why it continues to be amazing is
  because it provides strong guarantees to the end-user and developer.
  As such it makes reasoning about packages and deployments relatively
  straight-forward.

  The functional paradigm cleary works fantastically for packages.
  Unfortunately it is not quite clear that it works just as well for
  services.  The reason for this is that too many useful services are
  inherently stateful.  Their statefulness means they have side-effects,
  which in turn cause issues when relying on Guix features such as
  roll-back or automated deployment & guaranteed reproducability.

  Disciplined developers can implement many services in such a way that
  their statefulness is delegated to other dedicated services.  In this
  way the problem of statefulness can be isolated.  However, many
  existing useful services /have not/ been implemented in such a way.

  These notes are an attempt to think through ways of formalising how we
  mitigate stateful services in Guix.

  The notes below are organised around the example of a popular PHP
  content management system, Drupal.  Similar problems will apply to
  many other end-user services.

  My intent here is to communicate my thinking in the hopes that others
  can point to obvious flaws in my reasoning — or to stimulate
  conversation about the topic.  I hope at the very least that it is an
  interesting read!


2 Learning by Doing: Packaging & Deploying Drupal
═

2.1 Problem (1): the Drupal tarball is a binary blob!
─

  Released Drupal tarballs are shipped with a bunch of PHP dependencies,
  as well as compiled JS files.  A fully source-distributed installation
  of Drupal would:
  1) delete all shipped PHP dependencies
  2) independently build all PHP dependencies and make them available to
 Drupal (through it's vendor directory as symlinks?)
  3) delete all compiled JS files & libraries
  4) independently build the JS dependencies and make them avalable to
 Drupal.


2.2 Packages must always be stateless!
──

  What does this mean in practice?  Let's look at Drupal again.  When
  you download Drupal, the resulting tarball contains the source code,
  and an empty sites/ folder.  The sites folder is intended to contain
  /state/ files.  The normal installation procedure is to simply drop
  the drupal distribution in your web root directory, and for state
  files to live underneath sites/ within your webroot.

  In Guix, Drupal is packaged so that it is installed in the store.

  The store is read-only and hence no /state/ files can live under the
  sites/ directory in the store.

  How do we get around this?
  1) either we patch drupal to expect the sites/ directory outside of
 its own folder tree.
  2) or we symlink /gnu/store/…drupal…/sites/ to a different location on
 the filesystem (e.g. /var/lib/drupal/sites/)

  The latter solution, while easy, means that a *successful*
  installation of the package Drupal in Guix results in an installation
  in the store with a *broken symlink* pointing outside the store.

  But the package itself has been rendered stateless!


2.3 Drupal: a stateful service
──

  Services in Guix are rich and multidimensional entities.  At core they
  are promises of things that will have happened when a system is up and
  running.  These promises can be arbitrary, like generating a
  configuration file every boot; or they can be an extension of other,
  already existing services.

  A particularly popular service to extend is the shepherd service,
  which ensures that particular daemons are started as soon as possible
  after the system has started.

  What would a Drupal service look like?  Essentially the software is
  just a bunch of files in a webroot — so at it's heart it simply
  extends a web service (e.g. nginx) with new location directives.

  On the other hand it also requires that tha web server, a sql backend
  and php-fpm are running, so that drupal can actually function.

  These two requirements are easily met with the usual service
  infrastructure: simply extend nginx with a location definition
  pointing to the Drupal folder in the store as the webroot and extend
  shepherd to require mysql, nginx and php-fpm.


2.3.1 Enter the state dragon


  But hang-on… if Mysql service is a dependency, and we store state in a
  Mysql DB, what happens when we upgrade to a newer version of Drupal,