Network services in ‘guix system docker-image’?

[Ludovic Courtès]

Hello!

Does it make any sense at all to have things like a DHCP client and nscd
inside a Docker image produced by ‘guix system docker-image’?

I was going to apply the patch below to allow users to get rid of these
services by running ‘guix system docker-image --network’.  But now, I’m
wondering if there’s a use case to have them at all.

Thoughts?

Ludo’.

diff --git a/doc/guix.texi b/doc/guix.texi
index d0592220a7..fd3d8b1e52 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -26934,6 +26934,10 @@ example, if you intend to build software using Guix inside of the Docker
 container, you may need to pass the @option{--privileged} option to
 @code{docker create}.
 
+Last, the @code{--network} option applies to @command{guix system
+docker-image}: it produces an image where network is supposedly shared
+with the host, and thus without services like nscd or NetworkManager.
+
 @item container
 Return a script to run the operating system declared in @var{file}
 within a container.  Containers are a set of lightweight isolation
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 6f81ac16ff..082133407a 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -508,13 +508,17 @@ system."
 (define* (system-docker-image os
   #:key
   (name "guix-docker-image")
-  (register-closures? (has-guix-service-type? os)))
+  (register-closures? (has-guix-service-type? os))
+  shared-network?)
   "Build a docker image.  OS is the desired .  NAME is the
-base name to use for the output file.  When REGISTER-CLOSURES? is true,
-register the closure of OS with Guix in the resulting Docker image.  By
-default, REGISTER-CLOSURES? is set to true only if a service of type
-GUIX-SERVICE-TYPE is present in the services definition of the operating
-system."
+base name to use for the output file.  When SHARED-NETWORK? is true, assume
+that the container will share network with the host and thus doesn't need a
+DHCP client, nscd, and so on.
+
+When REGISTER-CLOSURES? is true, register the closure of OS with Guix in the
+resulting Docker image.  By default, REGISTER-CLOSURES? is set to true only if
+a service of type GUIX-SERVICE-TYPE is present in the services definition of
+the operating system."
   (define schema
 (and register-closures?
  (local-file (search-path %load-path
@@ -531,7 +535,9 @@ system."
 
 
   (let ((os(operating-system-with-gc-roots
-(containerized-operating-system os '())
+(containerized-operating-system os '()
+#:shared-network?
+shared-network?)
 (list boot-program)))
 (name  (string-append name ".tar.gz"))
 (graph "system-graph"))
diff --git a/guix/scripts/system.scm b/guix/scripts/system.scm
index 2664c66a30..664ecac9e6 100644
--- a/guix/scripts/system.scm
+++ b/guix/scripts/system.scm
@@ -699,7 +699,7 @@ checking this by themselves in their 'check' procedure."
 #:disk-image-size image-size
 #:file-system-type file-system-type))
 ((docker-image)
- (system-docker-image os
+ (system-docker-image os #:shared-network? container-shared-network?
 
 (define (maybe-suggest-running-guix-pull)
   "Suggest running 'guix pull' if this has never been done before."

Re: 20/36: system: hurd: Add the Shepherd.

[Ludovic Courtès]

guix-comm...@gnu.org skribis:

> commit bb17242a511014e1691d494b17152865db0580e0
> Author: Jan (janneke) Nieuwenhuizen 
> AuthorDate: Tue Apr 7 08:03:03 2020 +0200
>
> system: hurd: Add the Shepherd.
> 
> This starts console and ttys using the Shepherd.  Shepherd is not running 
> as
> PID 1 yet, its started from `rc'.
> 
> * gnu/system/hurd.scm (%base-packages/hurd): Add "shepherd".
> (%base-services/hurd): New variable.
> (%hurd-os): New variable.
> (hurd-shepherd-services): New function.
> (cross-hurd-image): Use them to generate an (unused and incomplete) ...
> (shepherd.conf): Generate from services defined in
> %hurd-os.
> * gnu/packages/hurd.scm (hurd-rc-script): Do not start console, start the
> shepherd instead.
> (hurd)[arguments]: Create pty devices.
> * gnu/system/hurd.scm (shepherd.conf): New file.

[...]

> +;; XXX: We will replace this by addding (gnu services shepherd).
> +(define shepherd-configuration-file
> +  (@@ (gnu services shepherd) shepherd-configuration-file))
> +
>  (define %base-packages/hurd
>(list hurd bash coreutils file findutils grep sed
>  guile-3.0 guile-colorized guile-readline
> -net-base inetutils less which))
> +net-base inetutils less shepherd which))
> +
> +(define %base-services/hurd
> +  (list (service user-processes-service-type)
> +(service hurd-console-service-type
> + (hurd-console-configuration (hurd hurd)))
> +(service hurd-ttys-service-type
> + (hurd-ttys-configuration (hurd hurd)
> +
> +(define %hurd-os
> +  (operating-system
> +(host-name "guixygnu")
> +(bootloader #f)
> +(file-systems '())
> +(timezone "GNUrope")
> +(services %base-services/hurd)))
> +
> +(define (hurd-shepherd-services os)
> +  (append-map hurd-service->shepherd-service (operating-system-services os)))
>  
>  (define* (cross-hurd-image #:key (hurd hurd) (gnumach gnumach))
>"Return a cross-built GNU/Hurd image."
> @@ -136,6 +163,10 @@ if [ -f \"$GUIX_PROFILE/etc/profile\" ]; then
>. \"$GUIX_PROFILE/etc/profile\"
>  fi\n"))
>  
> +  (define shepherd.conf
> +(with-parameters ((%current-target-system "i586-pc-gnu"))
> +  (shepherd-configuration-file (hurd-shepherd-services %hurd-os

This looks like an intermediate before fully using the service/OS
framework, but I wonder how much it helps.

The way I see it, now that you’ve defined services for Hurd-specific
things like the console client, you could very much write:

  (operating-system
;; …
(kernel gnumach)
(hurd hurd)  ;<- we probably need this new field
(essential-services (hurd-essential-services this-operating-system))
(services %base-hurd-services))

(Eventually we can even arrange for ‘essential-services’ and ‘services’
to pick the right default as a function of the chosen kernel.)

and then do:

  guix system build --target=i586-pc-gnu the-config-above.scm

Assuming system cross-compilation works as advertised :-), it should
cross-compile the whole system.

At that stage, you don’t need ‘cross-hurd-image’ any longer and you can
instead run:

  guix system vm-image --target=i586-pc-gnu the-config-above.scm

to achieve the same result.

But!  There’s one missing bit: a Hurdish grub.cfg.  I think that can be
hacked in (gnu bootloader grub), by inheriting from ‘grub-bootloader’
and providing a custom ‘configuration-file-generator’ field and maybe
‘installer’ as well.

Thus, (gnu system hurd) would be left with nothing but the definitions
of ‘%base-services/hurd’, ‘%base-packages/hurd’, and so on.
‘hurd-essential-services’ could maybe go to (gnu services hurd).

But I wonder if I’m missing something, WDYT?

Thanks for the exciting developments again!  :-)

Ludo’.

Re: 18/36: services: hurd: Add dummy loopback.

[Ludovic Courtès]

guix-comm...@gnu.org skribis:

> +(define (hurd-loopback-shepherd-service _)
> +  "Return the 'loopback' Shepherd service."
> +
> +  (list (shepherd-service
> + (documentation "Dummy for bootstrapping (gnu services) on the 
> Hurd.")
> + (provision '(loopback))
> + (requirement '())
> + (start #~(const #t))
> + (stop #~(const #t))
> + (respawn? #f
> +
> +(define hurd-loopback-service-type
> +  (service-type
> +   (name 'loopback)
> +   (extensions (list (service-extension shepherd-root-service-type
> +hurd-loopback-shepherd-service)))
> +   (compose concatenate)
> +   (extend first-of-two)
> +   (default-value '(loopback)) ;canary for hurd-service->shepherd-service
> +   (description "Dummy service to bootstrap (gnu services) on the
> +Hurd.")))

I believe the code currently used in ‘static-networking-service’ to
setup the loopback device on GNU/Linux should also work on GNU/Hurd: it
uses the “traditional” ioctls provided by (guix build syscalls).

So hopefully we don’t need a Hurd-specific variant.

Ludo’.

Re: 13/36: services: Add hurd-console-service-type.

[Ludovic Courtès]

guix-comm...@gnu.org skribis:

> commit d7a217e67dcd3b14402d746f4428db8545ce83f2
> Author: Jan (janneke) Nieuwenhuizen 
> AuthorDate: Mon Apr 13 10:13:05 2020 +0200
>
> services: Add hurd-console-service-type.
> 
> * gnu/services/hurd.scm (): New variable.
> (hurd-console-shepherd-service, hurd-console-service-type): New function.
> (hurd-service->shepherd-service): Add entry.

Now for the exciting stuff, neat!

> -  #:export (hurd-service->shepherd-service))
> +  #:export (hurd-console-service-type
> +hurd-service->shepherd-service))
>  
>  ;;; Commentary:
>  ;;;
> @@ -38,9 +39,51 @@
>  (define (hurd-service->shepherd-service service)
>(let ((config (service-value service)))
>  (match config
> +  (($ ) (hurd-console-shepherd-service 
> config))
>(_ '()

Hmm do we really need this ‘hurd-service->shepherd-service’ procedure?
I haven’t read the rest, but it seems surprising at first sight.

> +  (list (shepherd-service
> + (documentation "Hurd console.")

“Run the Hurd’s VGA console client.”

> + (provision '(console))
> + (requirement '())

Should be '(user-processes) I think.

> + (start #~(lambda _ (fork+exec-command #$console-command) #t))

Rather #~(make-forkexec-constructor #$console-command).

> +(define hurd-console-service-type
> +  (service-type
> +   (name 'console)
> +   (description
> +"Run a hurd console, @command{console}.")

Maybe “Run the Hurd console client.”

> +   (compose concatenate)
> +   (extend first-of-two)

I don’t think we need these two fields since this service is not meant
to be extensible.

> +   (default-value (hurd-console-configuration

Please add a ‘description’ too!

Also, normally we document every new service type in the manual, but
maybe we can delay it here so that you can unleash your hack powers in
the meantime.  :-)

Thanks,
Ludo’.

Re: branch wip-hurd-vm created (now fdb35e3)

[Ludovic Courtès]

guix-comm...@gnu.org skribis:

>  new 330f3ab  gnu: guix: Cross-build fixup for wrap-program.
>  new af9b4d0  gnu: guix: Use gnutls-3.6.13 when cross-compiling.
>  new de41617  gnu: guix: Apply courage for the Hurd.
>  new 309963e  gnu: guix: Update to de41617661.
>  new ddb44df  vm: Make the device node procedure a parameter.
>  new f3b1d64  gnu: hurd: Fix references to /bin/w.
>  new 695e2b4  gnu: hurd: Use default Qemu guest ip: 10.0.2.15.
>  new 2f5d2dc  gnu: hurd: Add NFS support.
>  new 29ea4dc  gnu: Add libtirpc/hurd.
>  new d01f38a  system: hurd: Add less, which to %base-packages/hurd.
>  new df00a4a  system: hurd: Add /etc/group.

With the feedback I gave, these 11 patches can go to ‘core-updates’ IMO!

Thank you!

Ludo’.

Re: 09/36: gnu: Add libtirpc/hurd.

[Ludovic Courtès]

guix-comm...@gnu.org skribis:

> commit 29ea4dc2c1df0d4562f1add01501726a3f963b91
> Author: Ricardo Wurmus 
> AuthorDate: Sun Apr 12 23:04:49 2020 +0200
>
> gnu: Add libtirpc/hurd.
> 
> * gnu/packages/onc-rpc.scm (libtirpc/hurd): New variable.
> * gnu/packages/patches/libtirpc-hurd-client.patch,
> gnu/packages/patches/libtirpc-hurd.patch: New files.
> * gnu/local.mk (dist_patch_DATA): Add them.

LGTM!

On the next ‘staging’ cycle, we should merge the two libtirpc’s.

Ludo’.

Re: 07/36: gnu: hurd: Use default Qemu guest ip: 10.0.2.15.

[Ludovic Courtès]

guix-comm...@gnu.org skribis:

> commit 695e2b4b5691910e6c82c56ec3fd31e6acdb9373
> Author: Jan (janneke) Nieuwenhuizen 
> AuthorDate: Sun Apr 12 13:53:04 2020 +0200
>
> gnu: hurd: Use default Qemu guest ip: 10.0.2.15.
> 
> Specifying 10.0.2.77 also works
> 
> -device rtl8139,netdev=net0 -netdev 
> user,id=net0,hostfwd=tcp:127.0.0.1:2228-10.0.2.77:22
> 
> but using this default allows us to use the widely advertised
> 
> -device rtl8139,netdev=net0 -netdev 
> user,id=net0,hostfwd=tcp:127.0.0.1:2228-:22

Oh!

> * gnu/packages/hurd.scm (hurd-rc-script): Use 10.0.2.15.
  ^
You can also mention [hurd-directives] here.

> -"--address" "10.0.2.77"
> +"--address" "10.0.2.15"

Perhaps add a margin comment saying this is the default QEMU guest IP.

LGTM!

Ludo’.

Re: 08/36: gnu: hurd: Add NFS support.

[Ludovic Courtès]

guix-comm...@gnu.org skribis:

> commit 2f5d2dc8240c22ea109c52c8ca957be9b155ff36
> Author: Ricardo Wurmus 
> AuthorDate: Sun Apr 12 23:06:30 2020 +0200
>
> gnu: hurd: Add NFS support.
> 
> * gnu/packages/hurd.scm (hurd)[inputs]: Add libtirpc/hurd.
> [arguments]: Add configure flags to use libtirpc; add phase "find-tirpc" 
> to
> find the RPC headers; add phase "fix-rpc-headers" to include missing 
> headers.

LGTM!

Re: 05/36: vm: Make the device node procedure a parameter.

[Ludovic Courtès]

Hi,

guix-comm...@gnu.org skribis:

> commit ddb44df650592a29528ef3f4a913336230b1d3a1
> Author: Ludovic Courtès 
> AuthorDate: Tue Apr 7 10:21:48 2020 +0200
>
> vm: Make the device node procedure a parameter.
> 
> * gnu/build/vm.scm (root-partition-initializer): Add #:make-device-nodes
> parameter and use it.
> * gnu/system/vm.scm (qemu-image): Add #:device-node parameter.  Pass
>  #:make-device-nodes to 'root-partition-initializer'.
> * gnu/system/hurd.scm (cross-hurd-image): Pass #:device-nodes 'hurd to
> 'qemu-image'.
> ---
>  gnu/system/hurd.scm | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/gnu/system/hurd.scm b/gnu/system/hurd.scm
> index 2583ff0..059bfaf 100644
> --- a/gnu/system/hurd.scm
> +++ b/gnu/system/hurd.scm
> @@ -209,6 +209,7 @@ fi\n"))
>#:bootcfg-drv grub.cfg
>#:bootloader grub-bootloader
>#:register-closures? #f
> +  #:device-nodes 'hurd
>#:extra-directives hurd-directives))

On ‘core-updates’, there’s already #:device-nodes 'hurd, though not on
the same line.

I think we can drop this patch, no?

Ludo’.

Re: 06/36: gnu: hurd: Fix references to /bin/w.

[Ludovic Courtès]

guix-comm...@gnu.org skribis:

> janneke pushed a commit to branch wip-hurd-vm
> in repository guix.
>
> commit f3b1d64cecdd964699a4e22a2b4b5b7fc53c
> Author: Rene Saavedra 
> AuthorDate: Sun Apr 12 22:33:08 2020 -0500
>
> gnu: hurd: Fix references to /bin/w.
> 
> * gnu/packages/hurd.scm (hurd): Use '/bin/w' from hurd package.

OK!

Re: 02/36: gnu: guix: Use gnutls-3.6.13 when cross-compiling.

[Ludovic Courtès]

Hello!

Time for random review comments on ‘wip-hurd-vm’.  :-)

guix-comm...@gnu.org skribis:

> commit af9b4d0ba3d55ef89e956f408853548451243e95
> Author: Jan (janneke) Nieuwenhuizen 
> AuthorDate: Fri Apr 10 09:54:02 2020 +0200
>
> gnu: guix: Use gnutls-3.6.13 when cross-compiling.
> 
> * gnu/packages/package-management.scm (guix)[propagated-inputs]: When
> cross-compiling, use patched gnutls-3.6.13.

[...]

>(propagated-inputs
> -   `(("gnutls" ,guile3.0-gnutls)
> +   `(("gnutls" ,(if (%current-target-system)
> +(@@ (gnu packages tls) gnutls-3.6.13)

Please avoid @@ by making ‘gnutls-3.6.13’ public (and making the package
itself hidden if necessary).

Otherwise LGTM!

Ludo’.

Forgive brevity, sent from my Emacs under lockdown.
:-)

Re: Medium-term road map

[Joshua Branson]

Andy Wingo  writes:

>
> I think I have a solution here, as discussed on IRC.  Basic idea is to
> make a direct compiler from Tree-IL to bytecode, skipping the CPS step.
> The result won't be optimal in terms of generated code quality (I
> estimate on average 2x slowdown) but it will be very cheap to produce (I
> estimate 10x improvement in time, and going from O(n log n) space to
> O(n), with better constant factors.  We'd use this compiler for -O0 or
> maybe even -O1 code.
>
> Got to spend a few days working it up tho!
>
> Andy

I love how Andy can say something like "yeah I've thought of this
awesome idea that will result in a 10x performance gain"...and "oh yeah,
it'll only take me a few days to do it."  What's awesome is he is such a
genius, that he'll probably pull it off.  :)

--
Joshua Branson
Sent from Emacs and Gnus

Re: Bump stackage LTS

[Timothy Sample]

Hi John,

I’m just catching up on the list after a long hiatus, so sorry for the
wait.

John Soo  writes:

> Hi Guix,
>
> Stackage and ghc have moved quite a bit since our stackage and ghc
> versions.  Would it be ok to start work on bumping our package set to
> a newer set of working packages and compilers?  What is the state of
> the wip-haskell branch?

Yes!  That would be great.  The way I see it, there are three things we
should do:

1. Make a change for 
2. Update GHC to 8.8.3
3. Update all of the “ghc-*” packages to their LTS 15 versions

I’ve been meaning to get started on all of this, and this is nice
reminder to do so!  :)  I have something ready for [1], and I have a
couple of scripts for getting [3] off the ground quickly.  It would be
great if you could take a look at [2].  (Although, if you’ve started
working on [3] already, we can make different arrangements – let me
know.)

AFAIK, we are OK to use the wip-haskell branch.  I will reset it to
master and get started today or tomorrow.


-- Tim

Re: unexpected reproducibility of reproducible blog post?

[zimoun]

On Wed, 29 Apr 2020 at 18:00, Konrad Hinsen  wrote:

> I have also opened an issue for this:
>
>   https://github.com/khinsen/reproducibility-with-guix/issues/2

I will add something overthere for tracking reproduciblity infos in the future.


> > Grafts or maybe Guile 2 -> 3?
>
> With time-machine, you run the full Guix from back then, so you run
> Guile 2 if that's what it takes. What I am not so sure about is how the
> old Guix release is built. If the build uses the equivalent of "guix
> environment guix", it would start using Guile 3.

>From [1] and assuming that the commit was the same, i.e.,
769b96b62e8c09b078f73adc09fb860505920f8f, there is also a mismatch
about the resulting binary.

Expected:
1be3c1b5d1e065017e4c56f725b1a692

Now:
2805a33e2e48f648307c6b913b69e41c

--8<---cut here---start->8---
guix describe # f03e5ca
guix time-machine \
 --commit=769b96b62e8c09b078f73adc09fb860505920f8f \
 -- environment --container --ad-hoc gcc-toolchain \
 -- gcc pi.c -o pi-guix
--8<---cut here---end--->8---

[1] https://lists.gnu.org/archive/html/guix-devel/2020-01/msg00192.html

>From f03e5ca, the time machine downloads the substitute:
   
https://ci.guix.gnu.org/nar/lzip/ij38zh495f81xpzmp4qzqz4fprczwck2-gcc-toolchain-9.2.0


> Time travelling is not as simple as it looks, but then we should have
> expected that!

I agree but it is annoying.
Because `in fine` the computations are not more reproducible than say
Debian if 3 months later we are not able to reproduce them bit-to-bit.
I do not know. Maybe it is about 'time-machine', maybe about the exact
commit used (most probable! :-)), maybe about the Guix build toolchain
(seed) used to travel back and restore the previous build toolchain.
Who knows? :-)

Well, I will try later with my desktop machine when I will be back at
the office; hoping that I did not garbage collected. :-)


Cheers,
simon

Re: core-updates call for testing

[zimoun]

On Wed, 29 Apr 2020 at 15:07, Marius Bakke  wrote:
>
> Ricardo Wurmus  writes:
>
> > Leo Famulari  writes:
> >
> >> I'm doing `guix pull --branch=core-updates`, with a `guix describe` of
> >> commit a533c5a183 (core-updates from 2 weeks ago), on Debian, in tmux,
> >> and I see this weird thing:
> >>
> >> --
> >> substitute: updating substitutes from 'https://private.mirror'... 100.0%
> >> @ substituter-started 
> >> /gnu/store/zfskbazynmdgmrbhnkbvar8qn5rpqgh2-git-minimal-2.26.2 substitute
> >> /@ download-started 
> >> /gnu/store/zfskbazynmdgmrbhnkbvar8qn5rpqgh2-git-minimal-2.26.2 
> >> https://private.mirror/nar/lzip/zfskbazynmdgmrbhnkbvar8qn5rpqgh2-git-minimal-2.26.2
> >>  4856299
> >> \@ download-progress 
> >> /gnu/store/zfskbazynmdgmrbhnkbvar8qn5rpqgh2-git-minimal-2.26.2 
> >> https://private.mirror/nar
> >
> > These tags are status information that Guix should hide and use to build
> > progress bars and the like.  They have been around for quite some time,
> > but they should not ever be visible.
>
> Could it be because locales are not yet available for the new Guix?

I had the same issue '@ substituter-started' when `guix pull
--branch=core-updates -p /tmp/core-updates`.
And today too, when simply `guix pull` from 619f918 to f03e5ca; on Debian.
I have noted that it happens time to time when pulling; so it does not
seems related to 'core-updates'.

--8<---cut here---start->8---
@ substituter-succeeded /gnu/store/mp8nkjfpba6fsrg05l06278gvbk2j147-graphviz-2.4
@ substituter-started /gnu/store/98lzwzblb79x8hqsnc1i7gls2sfg6b8l-apr-1.6.5 subs
-@ download-started /gnu/store/98lzwzblb79x8hqsnc1i7gls2sfg6b8l-apr-1.6.5 https:
@ download-progress /gnu/store/98lzwzblb79x8hqsnc1i7gls2sfg6b8l-apr-1.6.5 https:
@ download-progress /gnu/store/98lzwzblb79x8hqsnc1i7gls2sfg6b8l-apr-1.6.5 https:
@ download-progress /gnu/store/98lzwzblb79x8hqsnc1i7gls2sfg6b8l-apr-1.6.5 https:
@ download-succeeded /gnu/store/98lzwzblb79x8hqsnc1i7gls2sfg6b8l-apr-1.6.5 https
--8<---cut here---end--->8---

All the best,
simon