Re: Building Docker images of GuixSD
Christopher Baineswrites: > Unfortunately, while I could get a shell using "docker exec ...", I had > to start the guix-daemon manually as the shepherd service didn't seem to > work, at least initially. Also, when I had started it, I tried > installing a package, and there was some promising output to start off > with, but then it failed with: > > guix package: error: build failed: cloning builder process: Operation > not permitted > > Anyway, this is all pretty great! Awesome work getting this far. I'm > very excited to see what services will run this way, as Docker could > provide, albeit with some overhead, a layer of interoperability between > software that can handle Docker containers, and Guix. I tried making the changes you suggested. I launched a container without using docker's --privileged option. However, the "boot" script failed because something couldn't mount something in the container. I am not excited about the idea of trying to figure out which esoteric combination of capabilities [1] are needed to run without the --privileged option, but I suppose that is necessary if I want to minimize the container's access to the host system. To be honest, I find it puzzling that Docker requires me to think so much about the capabilities in the first place. Perhaps I'm naive, but I had hoped that within a Docker container, everything would be "private" in the sense that, as root in the container, I can do anything and everything, including mounting, including creating device nodes, and no changes will be visible outside of the container. The fact that that is not the case (at least by default) comes as quite a surprise to me. I also noticed that some services, like nscd, failed to start. No error messages anywhere except Shepherd saying something like "could not start service nscd". How do I begin to debug something like that? All I can think of is to inspect the Guix code that runs the service, and the Shepherd code, commit time to learning about it, and then hopefully find a way to insert debug statements of some kind that give me a hint about what's going wrong. Is there a faster, better way? [1] https://docs.docker.com/engine/security/security/#linux-kernel-capabilities -- Chris signature.asc Description: PGP signature
Re: Building Docker images of GuixSD
Hi, Christopher Baineswrites: > Awesome stuff Chris, I've tried this myself, on a Debian machine with > Docker installed. Great! It's heartening to know that I'm not the only one tinkering with this. :-) > I changed the %base-file-systems in the very-bare-bones system with > %container-file-systems, and then things started working. I see. I'll have to try this, too. > I tried without privileged mode, and got a error related to the firmware > service. This isn't included when you build call > operating-system-derivation with the #:container? #t argument, and sure > enough I was able to get the system up without the Docker --privileged > flag. I think Ludo mentioned this in his reply. Excellent! I'll have to do that, too. > Unfortunately, while I could get a shell using "docker exec ...", I had > to start the guix-daemon manually as the shepherd service didn't seem to > work, at least initially. Also, when I had started it, I tried > installing a package, and there was some promising output to start off > with, but then it failed with: > > guix package: error: build failed: cloning builder process: Operation > not permitted Huh. I'll look into this. > Anyway, this is all pretty great! Awesome work getting this far. I'm > very excited to see what services will run this way, as Docker could > provide, albeit with some overhead, a layer of interoperability between > software that can handle Docker containers, and Guix. I agree! Thank you for taking the time to test this out. It's extremely helpful to get a second pair of eyes on it. -- Chris signature.asc Description: PGP signature
Re: Building Docker images of GuixSD
Hello, Christopher Bainesskribis: > Unfortunately, while I could get a shell using "docker exec ...", I had > to start the guix-daemon manually as the shepherd service didn't seem to > work, at least initially. Also, when I had started it, I tried > installing a package, and there was some promising output to start off > with, but then it failed with: > > guix package: error: build failed: cloning builder process: Operation > not permitted Presumably this is about clone(2) creating a child process with separate namespaces. At first sight I can’t think of an obvious reason why it wouldn’t work. Anyway, that’s great progress already, I think GuixSD containers are useful even without guix-daemon, for the (hopefully common) case of stateless containers. Cheers, Ludo’.
Re: Building Docker images of GuixSD
Chris Marusich writes: > Hi Ludo and others following along, ... > Thanks for reading this far. I look forward to hearing your thoughts! Awesome stuff Chris, I've tried this myself, on a Debian machine with Docker installed. I struggled getting root, as su and sudo didn't seem to work, until I realised I could just replace alice with root in the "docker exec" command... Anyway, when I got root, I could tell that the system hadn't come up correctly. The problem seemed to be related to cgroups. root@komputilo /# mount -t cgroup /sys/fs/cgroup/cpu mount: /sys/fs/cgroup/cpu: cgroup already mounted on /sys/fs/cgroup/systemd. root@komputilo /# herd status Started: + file-system-/sys/fs/cgroup/perf_event + file-system-/dev/shm + host-name + root + file-system-/sys/fs/cgroup + file-system-/sys/fs/cgroup/cpuset + file-system-/dev/pts + user-file-systems + root-file-system + file-system-/gnu/store + file-system-/sys/fs/cgroup/freezer + file-system-/sys/fs/cgroup/memory + file-system-/sys/fs/cgroup/devices + file-system-/sys/fs/cgroup/blkio Stopped: - file-system-/sys/fs/cgroup/hugetlb - file-system-/sys/fs/cgroup/cpuacct - file-system-/sys/fs/cgroup/cpu - guix-daemon - file-systems - syslogd - urandom-seed - nscd - user-homes - user-processes root@komputilo /# herd start guix-daemon herd: exception caught while executing 'start' on service 'file-system-/sys/fs/cgroup/cpu': ERROR: In procedure mount: mount "cgroup" on "///sys/fs/cgroup/cpu": Device or resource busy I changed the %base-file-systems in the very-bare-bones system with %container-file-systems, and then things started working. I tried without privileged mode, and got a error related to the firmware service. This isn't included when you build call operating-system-derivation with the #:container? #t argument, and sure enough I was able to get the system up without the Docker --privileged flag. I think Ludo mentioned this in his reply. Unfortunately, while I could get a shell using "docker exec ...", I had to start the guix-daemon manually as the shepherd service didn't seem to work, at least initially. Also, when I had started it, I tried installing a package, and there was some promising output to start off with, but then it failed with: guix package: error: build failed: cloning builder process: Operation not permitted Anyway, this is all pretty great! Awesome work getting this far. I'm very excited to see what services will run this way, as Docker could provide, albeit with some overhead, a layer of interoperability between software that can handle Docker containers, and Guix. Thanks again, Chris signature.asc Description: PGP signature
Re: Building Docker images of GuixSD
Hi Chris, Chris Marusichskribis: > Run GuixSD in Docker > > > The attached patch makes it possible to build a GuixSD Docker image from > an operating system configuration file. For some reason, I had overlooked this message, but it’s awesome! > Pretty neat! Yup! > Problems I Noticed > == [...] > Second, I noticed the following error in the Guix daemon's logs. It > might be benign, since package installation worked fine, but I'm not > sure what it means or how to debug it: > > error in finalization thread: Bad file descriptor I’ve noticed this since we use Shepherd on Guile 2.2, but I haven’t checked where that comes from; it doesn’t seem to be a serious issue. ;-) Anyway, it’s not related to your experiment. > Third, I noticed that the shepherd failed to start syslogd and nscd (and > user-homes, although I wasn't as concerned about that because the home > directory for alice did in fact get created). [...] > I thought maybe syslogd wasn't working because /dev/log hadn't been > created in the Docker image, so I tried creating it manually. However, > that didn't help; the Shepherd still couldn't start syslogd. Hmm, I would have thought /dev/log was the issue. Any other hints? > Fourth, I wasn't able to run GuixSD in a Docker container without > supplying the "--privileged" option. GuixSD writes to sysfs during boot > (I don't know why, but the details are apparently in > guix/gnu/build/activation.scm), so the only way to get GuixSD to start > is to run the container in privileged mode. This is unfortunate, > because privileged mode sounds quite dangerous for a lot of reasons. I don’t think so: there’s a special case for when one creates a container with ‘guix system container’ that disables this kind of thing. I guess we should use it here. It’s mostly about passing #:container? #f somewhere. > From 25d5527b14302fc835af5c338bf37cf621c63a4e Mon Sep 17 00:00:00 2001 > From: Chris Marusich > Date: Sat, 21 Oct 2017 14:40:58 -0700 > Subject: [PATCH] Make it possible to build GuixSD docker images > > --- > gnu/build/linux-boot.scm| 5 +- > gnu/build/vm.scm| 14 ++-- > gnu/system/linux-initrd.scm | 12 ++-- > gnu/system/vm.scm | 169 > ++-- > guix/docker.scm | 23 -- > guix/scripts/pack.scm | 5 +- > guix/scripts/system.scm | 3 +- > 7 files changed, 191 insertions(+), 40 deletions(-) [...] > + (cond ((string=? "iso9660" file-system-type) > + (iso9660-image #:name name > +#:file-system-label root-label > +#:file-system-uuid root-uuid > +#:os-drv os-drv > +#:register-closures? #t > +#:bootcfg-drv bootcfg > +#:bootloader (bootloader-configuration-bootloader > + (operating-system-bootloader os)) > +#:inputs `(("system" ,os-drv) > + ("bootcfg" ,bootcfg > +((string=? "docker" file-system-type) > + (display "made it to docker image part\n") > + (os-docker-image #:name name > + #:os-drv os-drv > + #:register-closures? #t)) I’m not sure this is the right place for it since “docker” is not a file system type. Perhaps we need a separate procedure instead? > @@ -106,7 +107,9 @@ return \"a\"." > #:key closure compressor > (symlinks '()) > (system (utsname:machine (uname))) > - (creation-time (current-time time-utc))) > + (creation-time (current-time time-utc)) > + (tmpdir "/tmp") > + extra-items-dir) >"Write to IMAGE a Docker image archive from the given store PATH. The > image > contains the closure of PATH, as specified in CLOSURE (a file produced by > #:references-graphs). SYMLINKS must be a list of (SOURCE -> TARGET) tuples > @@ -116,7 +119,7 @@ binaries at PATH are for; it is used to produce metadata > in the image. > > Use COMPRESSOR, a command such as '(\"gzip\" \"-9n\"), to compress IMAGE. > Use > CREATION-TIME, a SRFI-19 time-utc object, as the creation time in metadata." > - (let ((directory "/tmp/docker-image") ;temporary working > directory > + (let ((directory (string-append tmpdir "/docker-image")) ;temporary > working directory Why do we need that? Would it be enough to honor $TMPDIR? > --- a/guix/scripts/system.scm > +++ b/guix/scripts/system.scm > @@ -638,8 +638,9 @@ any, are available. Raise an error if they're not." > #:mappings
Re: Building Docker images of GuixSD
Hi Chris, I've run GuixSD in a Docker container and returned to tell the tale! Congratulations! And thanks for exploring all this. > Is this helpful? Is it worth polishing up and maintaining? I'm not > entirely sure, and I'd like to know what you think. I think it is useful, mainly for reason 3: * If you want to run Guix on a system to which Guix hasn't been ported (like macOS) but your system does run Docker, now you can run Guix on that system by running it from a GuixSD Docker container. To which I might add a less obvious one: with GuixSD nicely integrated into the Docker universe, it has a better chance of adoption by people committed to Docker, and thus a better chance of becoming a/the preferred way of constructing Docker images. In other words, an attempt to take over the (Docker) world from inside. For the third bullet point, I don't know of any other reasonable way to get Guix working in Docker (although one could certainly run Guix in a VM using a technology other than Docker, such as QEMU). To run Guix, I have been trying this approach for a while, but I am still much further away from running Guix on my Mac than you are. First, QEMU: forget it on the Mac. It's not nearly as advanced/stable as it is under Linux. You can run Guix with QEMU under macOS, but it's slow and crashes a bit too often to rely on it. I moved on to VirtualBox, which runs a basic GuixSD without any major problem. But what you get is an isolated virtual machine. I haven't yet found a usable strategy for accessing the macOS file system from GuixSD. VirtualBox relies on its proprietary guest OS add-ins. I suspect they could be ported to GuixSD from a technical point of view, but it's not a trivial job and you'd have to remove the term "free software" from your brain for a while to do it. Currently I am trying NFS, exporting my Mac home directory via an NFS server on the Mac (easy) and mounting it from GuixSD (no success so far, because of the very incomplete NFS support in GuixSD). I expect this will work eventually, but in terms of performance it will probably never get to what you can achieve with Docker. Konrad.
Re: Building Docker images of GuixSD
Chris, this is very interesting! Even with privileged mode it makes it much easier to experiment. Pj. On Wed, Nov 08, 2017 at 10:15:38PM -0800, Chris Marusich wrote: > Hi Ludo and others following along, > > I've run GuixSD in a Docker container and returned to tell the tale! > The attached patch requires a lot of cleaning up (e.g., proper ChangeLog > entry, update documentation, remove some unnecessary imports and debug > messages that are probably still in there), so I'm taking a moment to > share my results and ask for feedback before committing to spending more > time on this. > > Run GuixSD in Docker > > > The attached patch makes it possible to build a GuixSD Docker image from > an operating system configuration file. > > You can build your own like this: > > 1) Apply this patch to 3b2fa4787938a408fab27ef7b3bc1302b6b6a805. > > 2) Build an image (I used the attached file "very-bare-bones.scm"): > > ./pre-inst-env guix system disk-image -t docker very-bare-bones.scm > > 3) Copy the resulting image onto a host that has Docker installed. > > 4) On the host with Docker, load the image and note the image ID: > > docker load < pw3d4r4m1x9yc3d1kg9x3y6abdzq9z7g-docker-image.tar.gz > > 5) Run a Docker container from the image, and note the container ID: > > docker run --privileged -d -e GUIX_NEW_SYSTEM=/var/guix/profiles/system > --net host --entrypoint /var/guix/profiles/system/profile/bin/guile > dcaa8fb677c7 /var/guix/profiles/system/boot > > 6) Run a shell in the container, install a package, and use it: > > docker exec -it -e USER=alice -u alice fb06fdcd3a0d > /run/current-system/profile/bin/bash --login > > 7) Install a package and use it: > > alice@komputilo /$ guix package -i hello > ... > Creating manual page database for 1 packages... done in 0.110 s > 1 package in profile > alice@komputilo /$ guix package --list-installed > hello 2.10out > /gnu/store/wf65hjwqwpz4wllasn63zysi5irql2sx-hello-2.10 > alice@komputilo /$ hello > Hello, world! > > Pretty neat! > > How Useful Is This? > === > > Using Guix, it was already possible to generate Docker images using > "guix pack". For example, I could have just generated a Docker image > from the GNU Hello package, created a container from that, and then run > "hello" from that container. What does running GuixSD in Docker give us > that we don't have already? At a minimum, it gives us the following: > > * The ability to define what service(s) should run in the resulting > Docker container, including their configs and start/stop scripts. > > * Since the Docker image is generated from a GuixSD operating system > configuration file, the rules for defining and configuring services > are the same as always. You don't have to learn anything new. > > * If you want to run Guix on a system to which Guix hasn't been ported > (like macOS) but your system does run Docker, now you can run Guix on > that system by running it from a GuixSD Docker container. > > Is this helpful? Is it worth polishing up and maintaining? I'm not > entirely sure, and I'd like to know what you think. > > For the first two bullet points, that's nice, but instead of using a > full-blown OS and relying on the Shepherd for process management in this > case, would it be simpler to just provide a way to easily bundle > start/stop scripts inside of the packs produced by "guix pack"? An > enterprising user can probably do this today by simply defining a > package that builds start/stop scripts for a given service; the user > would then just need to include that package in the pack. The downside, > I guess, is that you can't re-use the service-specific stuff that you > can normally use in a GuixSD operating system configuration file. > > For the third bullet point, I don't know of any other reasonable way to > get Guix working in Docker (although one could certainly run Guix in a > VM using a technology other than Docker, such as QEMU). To run Guix, > you need the Guix daemon running somewhere, right? And the Guix daemon > requires that certain build users exist. It might require other things > from its environment, too. In any case, you can't just run "guix pack > -t docker guix" and expect the "guix" command to work in the container > (I tried, and it doesn't work). You have to take additional measures, > like create build users, at which point it seems easier to just put all > of GuixSD into a Docker image. That's what my patch lets you do. > > What do you think? Is this worth polishing up and maintaining? > > Problems I Noticed > == > > Now I'll mention some specific problems I've noticed while running > GuixSD in a Docker container. First, I saw this while the Docker image > was being generated: > > tar: Removing leading `/' from member names > tar: Removing leading `/' from hard link targets > tar: ./dev/log:
Re: Building Docker images of GuixSD
Hi Ludo and others following along, I've run GuixSD in a Docker container and returned to tell the tale! The attached patch requires a lot of cleaning up (e.g., proper ChangeLog entry, update documentation, remove some unnecessary imports and debug messages that are probably still in there), so I'm taking a moment to share my results and ask for feedback before committing to spending more time on this. Run GuixSD in Docker The attached patch makes it possible to build a GuixSD Docker image from an operating system configuration file. You can build your own like this: 1) Apply this patch to 3b2fa4787938a408fab27ef7b3bc1302b6b6a805. 2) Build an image (I used the attached file "very-bare-bones.scm"): ./pre-inst-env guix system disk-image -t docker very-bare-bones.scm 3) Copy the resulting image onto a host that has Docker installed. 4) On the host with Docker, load the image and note the image ID: docker load < pw3d4r4m1x9yc3d1kg9x3y6abdzq9z7g-docker-image.tar.gz 5) Run a Docker container from the image, and note the container ID: docker run --privileged -d -e GUIX_NEW_SYSTEM=/var/guix/profiles/system --net host --entrypoint /var/guix/profiles/system/profile/bin/guile dcaa8fb677c7 /var/guix/profiles/system/boot 6) Run a shell in the container, install a package, and use it: docker exec -it -e USER=alice -u alice fb06fdcd3a0d /run/current-system/profile/bin/bash --login 7) Install a package and use it: alice@komputilo /$ guix package -i hello ... Creating manual page database for 1 packages... done in 0.110 s 1 package in profile alice@komputilo /$ guix package --list-installed hello 2.10out /gnu/store/wf65hjwqwpz4wllasn63zysi5irql2sx-hello-2.10 alice@komputilo /$ hello Hello, world! Pretty neat! How Useful Is This? === Using Guix, it was already possible to generate Docker images using "guix pack". For example, I could have just generated a Docker image from the GNU Hello package, created a container from that, and then run "hello" from that container. What does running GuixSD in Docker give us that we don't have already? At a minimum, it gives us the following: * The ability to define what service(s) should run in the resulting Docker container, including their configs and start/stop scripts. * Since the Docker image is generated from a GuixSD operating system configuration file, the rules for defining and configuring services are the same as always. You don't have to learn anything new. * If you want to run Guix on a system to which Guix hasn't been ported (like macOS) but your system does run Docker, now you can run Guix on that system by running it from a GuixSD Docker container. Is this helpful? Is it worth polishing up and maintaining? I'm not entirely sure, and I'd like to know what you think. For the first two bullet points, that's nice, but instead of using a full-blown OS and relying on the Shepherd for process management in this case, would it be simpler to just provide a way to easily bundle start/stop scripts inside of the packs produced by "guix pack"? An enterprising user can probably do this today by simply defining a package that builds start/stop scripts for a given service; the user would then just need to include that package in the pack. The downside, I guess, is that you can't re-use the service-specific stuff that you can normally use in a GuixSD operating system configuration file. For the third bullet point, I don't know of any other reasonable way to get Guix working in Docker (although one could certainly run Guix in a VM using a technology other than Docker, such as QEMU). To run Guix, you need the Guix daemon running somewhere, right? And the Guix daemon requires that certain build users exist. It might require other things from its environment, too. In any case, you can't just run "guix pack -t docker guix" and expect the "guix" command to work in the container (I tried, and it doesn't work). You have to take additional measures, like create build users, at which point it seems easier to just put all of GuixSD into a Docker image. That's what my patch lets you do. What do you think? Is this worth polishing up and maintaining? Problems I Noticed == Now I'll mention some specific problems I've noticed while running GuixSD in a Docker container. First, I saw this while the Docker image was being generated: tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: ./dev/log: socket ignored It's fine that we remove the leading '/' from member names, since it looks like the tarball will be extracted relative to '/'. I think the same is true for the hard link targets. However, because tar ignored '/dev/log', that socket is missing in the Docker image. I don't know if that will interfere with syslogd, but it sure doesn't sound good. Second, I noticed the
Re: Building Docker images of GuixSD
Hi! Chris Marusichskribis: > l...@gnu.org (Ludovic Courtès) writes: [...] >> Or we mount the host store over 9p and exec a dynamically-linked Guile >> from there. > > I understand hat by "9p" you mean to share the file system with the VM, > but I don't quite understand what this option entails. If the initrd > doesn't use the right Guile, why will sharing the store via 9p work? Do > you mean to exec directly somehow, instead of using an initrd...? 9p support is in the kernel, so we can always mount a 9p file system, provided the relevant 9p kernel modules are loaded. HTH! Ludo’.
Re: Building Docker images of GuixSD
l...@gnu.org (Ludovic Courtès) writes: > Chris Marusichskribis: > >> [1.345843] FS-Cache: Loaded >> [1.362140] 9pnet: Installing 9P2000 support >> [1.366118] 9p: Installing v9fs 9p2000 file system support >> [1.368730] FS-Cache: Netfs '9p' registered for caching >> configuring QEMU networking... >> loading '/gnu/store/jy509dgcsz82y13fmizv2sqaj90s1vfg-linux-vm-loader'... >> ERROR: In procedure dynamic-link: >> ERROR: In procedure dynamic-link: file: >> "/gnu/store/hwygv5jwd47amhp1m67iy3bkvxqjlbhm-libgcrypt-1.8.1/lib/libgcrypt", >> message: "file not found" > > The code above is running in the initrd, which means that it’s executed > by ‘guile-static-stripped’, which does not support dlopening: Glad I asked! It would have required much more fumbling around before I had even though about a possibility like that. It makes sense now. > So we should either make a big initrd with the dynamically-linked >Guile, > but then we may need to pass “-m 512” or similar to qemu… I will try this. I'm running the gexp in a linux vm is only because I need permission to create some of the device files. If you try to run this as a normal builder (e.g., using gexp->derivation), the build will fail because the builder tries to create device nodes and lacks permission to do so. I might also look into building the image in a container, since we also have some build-side container logic available (e.g., call-with-container). I'll let you know how it goes. > Or we mount the host store over 9p and exec a dynamically-linked Guile > from there. I understand hat by "9p" you mean to share the file system with the VM, but I don't quite understand what this option entails. If the initrd doesn't use the right Guile, why will sharing the store via 9p work? Do you mean to exec directly somehow, instead of using an initrd...? -- Chris signature.asc Description: PGP signature
Building Docker images of GuixSD
Hi Chris, Nice work on building Docker images of GuixSD! Chris Marusich <cmmarus...@gmail.com> skribis: > [1.345843] FS-Cache: Loaded > [1.362140] 9pnet: Installing 9P2000 support > [1.366118] 9p: Installing v9fs 9p2000 file system support > [1.368730] FS-Cache: Netfs '9p' registered for caching > configuring QEMU networking... > loading '/gnu/store/jy509dgcsz82y13fmizv2sqaj90s1vfg-linux-vm-loader'... > ERROR: In procedure dynamic-link: > ERROR: In procedure dynamic-link: file: > "/gnu/store/hwygv5jwd47amhp1m67iy3bkvxqjlbhm-libgcrypt-1.8.1/lib/libgcrypt", > message: "file not found" The code above is running in the initrd, which means that it’s executed by ‘guile-static-stripped’, which does not support dlopening: --8<---cut here---start->8--- $ guix environment -C --ad-hoc guile-static-stripped libgcrypt -- guile [...] GNU Guile 2.2.2 Copyright (C) 1995-2017 Free Software Foundation, Inc. Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'. This program is free software, and you are welcome to redistribute it under certain conditions; type `,show c' for details. Enter `,help' for help. scheme@(guile-user)> (file-exists? (string-append (getenv "GUIX_ENVIRONMENT") "/lib/libgcrypt.so")) $1 = #t scheme@(guile-user)> (dynamic-link (string-append (getenv "GUIX_ENVIRONMENT") "/lib/libgcrypt")) ERROR: In procedure dynamic-link: ERROR: In procedure dynamic-link: file: "/gnu/store/wplxvw0mxxy35j7019j6mkjvpgl0hs1g-profile/lib/libgcrypt", message: "file not found" Entering a new prompt. Type `,bt' for a backtrace or `,q' to continue. scheme@(guile-user) [1]> --8<---cut here---end--->8--- (The “file not found” message is misleading.) So we should either make a big initrd with the dynamically-linked Guile, but then we may need to pass “-m 512” or similar to qemu… Or we mount the host store over 9p and exec a dynamically-linked Guile from there. I realize it’s a bit sketchy, but I hope it makes sense. Thanks, Ludo’.