Using a swapfile on btrfs for hibernation

[Brice Waegeneire]

Hello Guix!

Here is a feedback on enabling hibernation using a swapfile stored on btrfs
filesystem. It wasn 't straight forward as it's not documented and I had to
read the code do make it work.

Note that even tho swpafile on btrfs filesystem are supported, therea are
some limitations⁵:
- the filesystem must be an a single device, ie. no btrfs RAID,
- the swapfile shouldn't be copy-on-write,
- the swapfile shouldn't be compressed.

Let's start with the btrfs specific part first. The following instructions
was compiled from tutorials¹ ² ³ for other Linux distributions. First we
gather the tools we'll need for the job, especially btrfs_map_physical⁴to
find the offset from the start of the btrfs partition where the swapfile is
located, this tool isn't packaged (I don't know if it should) so we'll
build it in the old fashion way.

--8<---cut here---start->8---
guix environment --ad-hoc gcc-toolchain curl gawk e2fsprogs btrfs-progs
curl -OL 
https://github.com/osandov/osandov-linux/raw/master/scripts/btrfs_map_physical.c
gcc -O2 -o ./btrfs_map_physical ./btrfs_map_physical.c
--8<---cut here---end--->8---

Next we create our swapfile in its own subvolume named “swap” on the btrfs
filesystem of device “/dev/sda2” (mounted at “/mnt/rootfs”) and we make
sure our new swapfile isn't COWable nor compressed. To be able to hibernate
in all cironstancese the swapfile need to be at least as big as the total
RAM available.

--8<---cut here---start->8---
swappath=/mnt/rootfs/swap
swapfile="$swappath/swapfile"
cd "$path"
btrfs subvolume create swap
mount -o subvol=swap /dev/sda2 ./swap
truncate -s 0 "$swapfile"
chattr +C "$swapfile"
btrfs property set "$swapfile" compression none
swap_size="$(grep MemTotal /proc/meminfo | awk '{print $2 * 1024}')"
fallocate --length "$swap_size" "$swapfile"
chmod 600 "$swapfile"
mkswap "$swapfile"
--8<---cut here---end--->8---

As we have a working swapfile, we'll genereate the kernel arguments to pass
to the kernel. The less obvious being the offset of the swapfile from the
start of the filesystem, to do so we are using the previously compiled
“btrfs_map_physical” binary and few shell utilities.

--8<---cut here---start->8---
swapon "$swapfile"
swap_physical_offset=$(btrfs_map_physical "$swapfile" | sed -n "2p" | awk 
"{print \$NF}")
swap_offset=$(echo "${swap_physical_offset} / $(getconf PAGESIZE)" | bc)
swap_uuid=$(findmnt -no UUID -T "$swapfile")
resume_args="resume=${swap_uuid} resume_offset=${swap_offset}"
echo "${resume_args}"
--8<---cut here---end--->8---

What's left is just to tweak our “operating-system” to make use of ourr new
swapfile.

--8<---cut here---start->8---
(operating-system
   (file-systems
(cons* ...
   (file-system
(mount-point "/")
(device
 (uuid "12345678-1234-1234-1234-123456789abc" 'btrfs))
(type "btrfs")
(options "compress=zstd,subvol=guix-system"))
   (file-system
(mount-point "/swap")
(device
 (uuid "12345678-1234-1234-1234-123456789abc" 'btrfs))
(type "btrfs")
;; Needed to access the swapfile? (not sure)
(needed-for-boot? #t)
(options "subvol=swap"))
   %base-file-systems))

(swap-devices (list "/swap/swapfile"))

(kernel-arguments
 (cons* "resume="12345678-1234-1234-1234-123456789abc"
"resume_offset=16400"
 %default-kernel-arguments))
...)
--8<---cut here---end--->8---

Now we can reconfigure our system and reboot it. At that point we are able
to use “loginctl hibernate” or “loginctl hybrid-sleep” to save our current
system state to the swapfile and recover it at the next reboot.

As a keen reader may have noticed, enabling hibernation in Guix in this
case has a quirk.  The most the most inconvenient is the Guix's “resume”
kernel argument format being different from Linux's⁶ « {/dev/ |
PARTUUID= | : | } » and other distributions allowing «
UUID= »; in Guix the uuid isn't prefixed with “UUID=” or “PARTUUID=”
and this isn't documented anywhere, you are forced to read the code.

Also, having swap related configuration spread between “swap-devices” and
“kernel-arguments” fields make me think having specific records for that
first field would make sense. And that would make the “resume“ format issue
irrelevant. WDYT about adding the following records to be used in
“swap-devices” record in addition to the current types?

--8<---cut here---start->8---
(swap-file
 (file "/swap/swapfile")
 (offset 16400)
 (device (uuid "12345678-1234-1234-1234-123456789abc"))
 (resume? #t)   ; default #f
 (priority 21)) ; swapflags argument passed to swapo

Nginx and certbot cervices don't play well togther

[Brice Waegeneire]

Hello Guix,

After an suggestion from Tobias to give a try at forcing HTTPS for
Guix's websites on berlin, I had a go at it but it was more complex that
what I was expecting.  Looking deeper at nginx and certbot services it
appear both services don't play that well together, requering a inital
dance when deploying a new HTTPS virtual server. As explained in #36389¹
you need to:
« - run system configuration with just the certbot service
- use certbot to generate your initial certificates
- reconfigure with additional nginx server configuration, pointing to
  the SSL certificates created by certbot »

Indeed, with an operating-system continaing the following services it's
impossible to sart Nginx and Certbot at once as one would expect:

--8<---cut here---start->8---
(service nginx-service-type)
(service php-fpm-service-type)

(service certbot-service-type
 (certbot-configuration
   (certificates
 (list (certificate-configuration
 (domains '("test.sama.re"))
 (deploy-hook
   (program-file
 "nginx-deploy-hook"
 #~(let ((pid (call-with-input-file "/var/run/nginx.pid"
read)))
 (kill pid SIGHUP)

(cat-avatar-generator-service
  #:configuration
  (nginx-server-configuration
(listen '("443 ssl"))
(server-name '("test.sama.re"))
(ssl-certificate
  "/etc/letsencrypt/live/test.sama.re/fullchain.pem")
(ssl-certificate-key
  "/etc/letsencrypt/live/test.sama.re/privkey.pem")))
--8<---cut here---end--->8---

Here is the error from reconfiguring the system:

--8<---cut here---start->8---
# guix system reconfigure /etc/config.sm
[...]
building /gnu/store/55cq2ja4i5489s55viv9fh50032d1ziy-switch-to-system.scm.drv...
making '/gnu/store/p2rkcmrnpls5py7x2iappf2qcbxwlb95-system' the current 
system...
setting up setuid programs in '/run/setuid-programs'...
populating /etc from /gnu/store/k2kb8hsq3q0dhhad4a9pjh4kx32mn4g0-etc...
/var/lib/certbot/renew-certificates may need to be run
creating nginx log directory '/var/log/nginx'
creating nginx run directory '/var/run/nginx'
creating nginx temp directories 
'/var/run/nginx/{client_body,proxy,fastcgi,uwsgi,scgi}_temp'
nginx: [emerg] cannot load certificate 
"/etc/letsencrypt/live/test.sama.re/fullchain.pem": BIO_new_file() failed (SSL: 
error:02001002:system library:fopen:No such file or 
directory:fopen('/etc/letsencrypt/live/test.sama.re/fullchain.pem','r') 
error:2006D080:BIO routines:BIO_new_file:no such file)
nginx: configuration file 
/gnu/store/chpw631djay2w39x7agg8zz53iayy4zy-nginx.conf test failed
`/gnu/store/jyxc290q7jyhhpalski0h13h8z9zvnka-openssh-authorized-keys/bricewge' 
-> `/etc/ssh/authorized_keys.d/bricewge'
The following derivation will be built:
   /gnu/store/qlzbrmpx6wnhzqcpqi9yrbb6xva82kvr-install-bootloader.scm.drv

building 
/gnu/store/qlzbrmpx6wnhzqcpqi9yrbb6xva82kvr-install-bootloader.scm.drv...
guix system: bootloader successfully installed on '/dev/sda'
The following derivation will be built:
   /gnu/store/ikak44inrnz3b3dx8j8csdakgqafbijn-upgrade-shepherd-services.scm.drv

building 
/gnu/store/ikak44inrnz3b3dx8j8csdakgqafbijn-upgrade-shepherd-services.scm.drv...
shepherd: Removing service 'dbus-system'...
shepherd: Service dbus-system has been stopped.
shepherd: Done.
shepherd: Service host-name has been started.
shepherd: Service user-homes has been started.
shepherd: Service host-name has been started.
shepherd: Service term-auto could not be started.
shepherd: Service php-fpm has been started.
guix system: warning: exception caught while executing 'start' on service 
'nginx':
Throw to key `%exception' with args `("#<&invoke-error program: 
\"/gnu/store/hn1mvgafkpf5knrnzvwpgpdlzmq553al-nginx-1.19.6/sbin/nginx\" 
arguments: (\"-c\" \"/gnu/store/chpw631djay2w39x7agg8zz53iayy4zy-nginx.conf\" 
\"-p\" \"/var/run/nginx\") exit-status: 1 term-signal: #f stop-signal: #f>")'.
guix system: warning: some services could not be upgraded
hint: To allow changes to all the system services to take effect, you will need 
to reboot.
--8<---cut here---end--->8---

What happen is Nginx won't start because the certficate related files
present in it's configuration doesn't exist and we can't get a Let's
Encrypt certificate from a HTTP-01 challenge without that web server
running.  NixOS broke that chicken and egg problem by generating a
self-signed certificate first, after that starting nginx, then
requesting a valid Lets' Encrypt certificate and finally reloading
Nginx.  That way we end up with a Nginx server using Let's Encrypt
certificate with no more that a simple system reconfiguration.  Note
that, the initial self-signed certificate will need to be at the path
were certbot will

Re: Getting rid of the mandb profile hook?

[Brice Waegeneire]

Hello Ludovic,

On 2021-03-03 15:13, Ludovic Courtès wrote:

I’m thinking we could get rid of the mandb hook.  However, the
functionality matters IMO (we need good tools so users can browse 
local

documentation; mandb is not that good but better than no search
mechanism.)  Here are several options that come to mind:

  1. Provide a ‘man’ wrapper or modify the ‘man-db’ package such that
 the database gets built on the first use of ‘man -k’, unless 
it’s

 already up-to-date.


That would mean the database would live in some user-specific writable
area of the file system correct (where?), right?  And could use the
common 'update' mechanism of man-db to make it as fast as possible.

This sounds good from a performance perpective, but could introduce
cache issues every now and then (if man-db changes a lot).  I wouldn't
expect much problem given how mature man-db is, but that's one thing 
to

consider.


I looked a bit at man-db, thinking it must have that already done more
or less.  Indeed, one can run “mandb -uc” to create the database.

The problem is that it insists on writing databases and ‘CACHEDIR.TAG’
files in the same directory as man pages.  In our case, these are all
read-only, so just prints a warning for each directory and keeps going.

It looks like man-db is not written with a situation like ours in mind.


What about using mandoc¹, the manpage compiler from OpenBSD, instead of
man-db? As from it's manual it support specifying the database location:

“makewhatis -d dir [file ...]”²

It isn't packaged in Guix yet, but other Linux distros have done it, 
some

 are even using it as their default.


[...]

One option I contemplated at one point is to simply have fewer man 
pages

in the first place.  :-)  There were packages that install man pages
when they shouldn’t.  This led to commits like
305eefc0627eb1d047e6fc4320d7e56897719ab8 and
4b797193d7508ddc53bb1ff7a267a0d50c1fe298 (and parent commits).


More outputs would be great tho having a way to force the installation 
of

specifics outputs for every installed package would improve quality of
live.  For a specific example in that case, when installing ncurses from
 the cli it would install it's man output too if you always want man 
page

 to be installed.

¹ https://mandoc.bsd.lv/
² https://www.mankier.com/8/makewhatis.mandoc

Cheers,
- Brice

[WIP 0/6] Add kernel-module-configuration service

[Brice Waegeneire]

Hello Guix,

Here is a work-in-progress to create a service centralizing the configuration
for built-in and loadable kernel modules.  The kernel module configuration
service has been implemented by making some fields from 'operating-system'
custom from the 'services' field.

Here are are some example of how to use it.
--8<---cut here---start->8---
;; Loadable kernel module
(simple-service 'ddcci-module kernel-module-configuration-service-type
   (list (kernel-module
  (name "ddcci")
  (package ddcci-driver-linux)
  (load? #t)
  (options '("dyndbg" "delay=600")))
 (kernel-module
  (name "ddcci-backlight")
  (blacklist? #t
;; Built-in kernel module
(simple-service 'intel-module
 kernel-module-configuration-service-type
 (list (kernel-module
(name "i915")
(options '("fastboot=1")
--8<---cut here---end--->8---

This WIP version is mainly missing the ability to configure built-in modules
trough the kernel arguments. It's the continuation of the work on
kernel-module-loader-service[1] and is needed to help fixing #41082[2].

I will try to go back to it after the GSOC network boot project.

[1]: https://issues.guix.info/40274
[2]: https://issues.guix.info/41082

- Brice

Brice Waegeneire (6):
  services: simulated-wifi: Use 'kernel-module-loader'.
  services: Add 'kernel-profile-service-type'.
  services: Add 'modprobe-service-type'.
  services: kernel-module-loader: Return a single 'shepherd-service'.
  WIP services: Add kernel-arguments-service-type.
  WIP services: Add kernel-module-configuration service.

 gnu/services.scm   | 102 +---
 gnu/services/linux.scm | 276 +
 gnu/services/networking.scm|  25 +--
 gnu/system.scm |  55 +--
 gnu/system/linux-container.scm |   2 +
 gnu/tests/linux-modules.scm|  65 +---
 6 files changed, 420 insertions(+), 105 deletions(-)

-- 
2.26.2

Re: [GSOC 2020] network-boot-service

[Brice Waegeneire]

Hello Vagrant,

On 2020-07-02 16:34, Vagrant Cascadian wrote:

On 2020-07-02, Brice Waegeneire wrote:

To support the widest hardware and boot options possible I went with
iPXE
as a chainloader. Meaning that any machine doing a PXE boot (or with
builtin iPXE with restricted feature set) will load the iPXE 
bootloader

first, which will then properly load the initrd.


iPXE is really cool! The main downside is I don't think the support for
non-x86 architectures is there, but would be happy to find out
otherwise.



It looks like it does support arm 32 and 64 bits EFI:
https://ipxe.org/appnote/buildtargets. Anyway dnsmasq can be configured 
to
provide PXE entries by on the client's architecture or feature it 
support.
For example using network-boot-service, a iPXE client without http 
support

will chainload Guix's iPXE which has http support.

Currently I'm working on the initrd part to add NFS mounting 
capability

to
it. At this point I'm blocked by building a lightweight staticly built
'nfs-utils' package to be included in the initrd. It's current total
size
219.2 MiB, I manage to reduced it to 162.2 MiB which is still one 
order

of
magnitude larger than my initrd at 19 MiB.

My issue building a static 'nfs-utils' is that it can't find
'getrpcbynumber{,_r}' “configure: error: Neither getrpcbynumber_r nor
getrpcbynumber are available”. This function should be provided by the
libc
or by libtirpc if it's not that first one. The problem is that
'libtirpc'
don't build it's own 'getrpcbynumber' because it find one in libc but
nfs-utils can't find it...


Have you looked into using klibc (or maybe busybox) instead? They might
be smaller and have enough capability to mount NFS.


Non I did not, thanks for the tip. Busybox is smaller that was I
expecting, 1.0 MiB by itself, but it doesn't seems it would follow the
Guix way of doing most of the work in Guile (and using boot-to-Guile™)
and it's unique binary forbid us to extract only the NFS part. As for
klibc, it seems to be what NixOS[1] is using and TIL that it also the
the original source of Arch's mkinitcpio-nfs-utils which is 
unmaintained.

So klibc may work here, I'll try packaging it.


Some other distros are using the kernel parameter 'nfsroot'[2], but we
probably don't want to use it since it can't be used together with an
initrd and it also mean we need built-in modules for NFS and network
card
driver in the kernel.


Debian at least implements support for the nfsroot kernel parameter by
emulating the behavior in the initrd, using kernel modules instead of
built-ins. Seems like Guix could do the same?


Do you happen to know which package implement it in Debian? Yes, there
will be a need to pass parameters to the initrd so we better use the
common syntax.


If you're trying to use user-space NFS, I suspect you'll run into a lot
of issues... Debian at least dropped support for parts of the 
user-space

NFS stack years ago as it had huge numbers of bugs and little activity
upstream.


What do you mean by “user-space NFS”, be it nfs-utils, busybox or klibc,
they all are user-space tools to used to mount NFS shares?
Do you have some reference about this, I would like to learn more about
it. It's true that I did not find anyone trying to build a static
nfs-utils...


Great to hear about progress on this work!


live well,
  vagrant


[1]: 
https://github.com/NixOS/nixpkgs/commit/d25c1a8fdc383b8997f6e7b4e1479875df1f06b2
[2]: 
https://github.com/NixOS/nixpkgs/blob/84cf00f98031e93f389f1eb93c4a7374a33cc0a9/pkgs/os-specific/linux/mkinitcpio-nfs-utils/default.nix


- Brice

[GSOC 2020] network-boot-service

[Brice Waegeneire]

Hello Danny, Gábor,

Sorry for the very late update on the status of this GSOC about network
booting. At the moment I have working network boot service which I'm 
using

to boot various baremetal machines, I'm currently working on adding NFS
support to the initrd.


To support the widest hardware and boot options possible I went with 
iPXE

as a chainloader. Meaning that any machine doing a PXE boot (or with
builtin iPXE with restricted feature set) will load the iPXE bootloader
first, which will then properly load the initrd.

For the DHCP/TFTP servers I choose dnsmasq instead of isc-dhcp-server 
with

a separate TFTP server mainly because it support ProxyDHCP mode which is
required for one of the most used case.

Those technical choice where instructed from the LTSP[1] project and 
some

Guixers advice, Vagrant and Giovani to name a few.


Speaking of uses cases for the network boot service, I see three of 
them.
Configuration wise, the most straight-forward is as an authoritative 
DHCP

server where all of the interfaces of the server provide the only DHCP
server. I'm guessing it won't be used a lot yet since it imply that the
machine running Guix is a router which quite rare ATM since our 
networking
configuration are limited ATM. Probably the most popular use case will 
be
as a ProxyDHCP, in a network where there is already a non-PXE 
authoritative

DHCP server, the authoritative server provides IP addresses where our
dnsmasq server only send PXE entries and provides images through TFTP. 
The

last one is as a interface specific DHCP server, where dnsmasq attach to
one interface to avoid messing around on an already configured network, 
a
NAT can be put in place to allow client to access Internet. That's the 
one

I'm currently using to develop and test the network boot features.

The new network-boot-service allows all of those use cases for clients
doing PXE boot or UEFI HTTP Boot, arbitrary images to boot from can be
specified or extended from an other service.


Currently I'm working on the initrd part to add NFS mounting capability 
to

it. At this point I'm blocked by building a lightweight staticly built
'nfs-utils' package to be included in the initrd. It's current total 
size
219.2 MiB, I manage to reduced it to 162.2 MiB which is still one order 
of

magnitude larger than my initrd at 19 MiB.

My issue building a static 'nfs-utils' is that it can't find
'getrpcbynumber{,_r}' “configure: error: Neither getrpcbynumber_r nor
getrpcbynumber are available”. This function should be provided by the 
libc
or by libtirpc if it's not that first one. The problem is that 
'libtirpc'

don't build it's own 'getrpcbynumber' because it find one in libc but
nfs-utils can't find it...

Some other distros are using the kernel parameter 'nfsroot'[2], but we
probably don't want to use it since it can't be used together with an
initrd and it also mean we need built-in modules for NFS and network 
card

driver in the kernel.

I tried to workaround NFS mounting by copying the image from HTTPS using
(web client) but it's not really elegant since the image has to be 
loaded

in RAM and even simple Guix images can be relatively large.


After that the intrd work is done I will need to add support for PXE to
qemu so that network boot functionality and the network-boot-service can 
be

tested in Guix; I'll try doing so with OVMF.

You can find the part of my work which is committed in the
'wip-network-boot' at https://git.sr.ht/~bricewge/guix.

[1]: https://ltsp.org/
[2]: 
https://www.kernel.org/doc/Documentation/filesystems/nfs/nfsroot.txt


Have a good day,
- Brice

Re: [PATCH] doc: cookbook: Update entry about getting substitutes through Tor.

[Brice Waegeneire]

Hello André,

On 2020-06-18 14:06, André Batista wrote:

[...]


qua 17 jun 2020 às 08:37:59 (1592393879), br...@waegenei.re enviou:

I would like to keep the warnings at the beginning of the section
to be sure that readers don't miss it when skimming trough it.
Any rewording of that part to make the scope of the section or
the warnings more clear is welcome.


It follows attached a new version of the previous patch which
changes the comment to the warning quote. I had previously thought
that it would be worse to inflate the warning with this comment even
more so as the section's title already mentions it's related to
substitutes.


I tought I already had applied your patch, but I forgot to do it.
It's now applied as f8945734a5abff69644284231cc47fb67456657b, sorry
for the delay.

[...]


On a wider front I would prefer to have a foolproof configuration
that route *all* guix related traffic through Tor, instead of that
half-way setup.  Providing a way to 'torify' any service with
something like 'make-forkexec-constructor/trosocks', as
'make-forkexec-constructor/container' does for containerizing a
service, would be great[0].  A less engaged option would be to
make 'guix-daemon' compatible with 'torsocks' since doing it so
makes guix unusable[1].


I too would prefer it, but a half-way setup is what we have for now.
So a three-quarters-way would be an improvement though not the fix
we're in need. I'll dig deeper and will come back to you if I make
any progress.


I would love to know when you manage to advance on that front.

Have a good day,
- Brice

Re: [bug#41694] [PATCH] doc: cookbook: Add entry about getting substitutes through Tor.

[Brice Waegeneire]

Hello André,

Thank you for the patch and your feedback!

On 2020-06-17 02:19, André Batista wrote:

Hello Brice,

I think it would be useful to warn users that when pulling there is
a direct connection to guix git repos, so to route it through Tor,
one needs to use torsocks. It wont make the configuration foolproof,
but it will reduce the leaks to clearnet.


When writing this section of the cookbook I was worried that some
readers will misunderstood it so I added a big warning at the
front but it doesn't seems to be enough since you sent this mail.

--8<---cut here---start->8---
@section Getting substitutes from Tor

Guix daemon can use a HTTP proxy to get substitutes, here we are
configuring it to get them via Tor.

@quotation Warning
@emph{Not all} Guix daemon's traffic will go through Tor!  Only
HTTP/HTTPS will get proxied; FTP, Git protocol, SSH, etc connections
will still go through the clearnet.  Again, this configuration isn't
foolproof some of your traffic won't get routed by Tor at all.  Use it
at your own risk.
@end quotation
--8<---cut here---end--->8---

+Note that the procedure described above applies only to package 
substitution.
+When you update your guix distribution with @command{guix pull}, you 
should

+use @command{torsocks} if you want to route the connection to guix git
+repository servers through Tor.
+
 @c 
*

 @node Advanced package management
 @chapter Advanced package management


I would like to keep the warnings at the beginning of the section
to be sure that readers don't miss it when skimming trough it.
Any rewording of that part to make the scope of the section or
the warnings more clear is welcome.

Note that this section is only about getting *substitutes* through
tor and it should probably be kept that way to avoid confusing the
user in regard to what (narrow) security benefit this configuration
offer.

On a wider front I would prefer to have a foolproof configuration
that route *all* guix related traffic through Tor, instead of that
half-way setup.  Providing a way to 'torify' any service with
something like 'make-forkexec-constructor/trosocks', as
'make-forkexec-constructor/container' does for containerizing a
service, would be great[0].  A less engaged option would be to
make 'guix-daemon' compatible with 'torsocks' since doing it so
makes guix unusable[1].

[0]: http://logs.guix.gnu.org/guix/2020-06-03.log#142909
[1]: https://lists.gnu.org/archive/html/guix-devel/2020-05/msg00214.html

- Brice

Re: [bug#41694] [PATCH] doc: cookbook: Add entry about getting substitutes through Tor.

[Brice Waegeneire]

Hello,

On 2020-06-04 12:29, Ludovic Courtès wrote:

Hi,

Brice Waegeneire  skribis:


* doc/guix-cookbook.texi (Getting substitutes from Tor): New section.


Yay!


+@node Getting substitutes from Tor
+@section Getting substitutes from Tor
+
+@quotation Warning
+@emph{Not all} Guix daemon's traffic will go through Tor!  Only
+HTTP/HTTPS will get proxied; FTP, Git protocol, SSH, etc connections
+will still go through the clearnet.  Again, this configuration isn't
+foolproof some of your traffic won't get routed by Tor at all.  Use 
it

+at your own risk.
+@end quotation


I would suggest adding a line of intro before the warning, otherwise we
see the warning before even knowing what the section is about.  :-)

+Guix's substitute server is available as a hidden service, if you 
want


I think official terminology these days is “Onion service”.


+to use it to get your substitutes from Tor configure your system as
+follow:
+
+@lisp
+(use-modules (gnu))
+(use-service-module base networking)
+
+(operating-system
+  …
+  (services
+(cons
+  (service tor-service-type
+  (tor-configuration
+(config-file (plain-file "tor-config"
+ "HTTPTunnelPort 
127.0.0.1:9250"

+  (modify-services %base-services
+   (guix-service-type

 ^
Too many spaces here.


+@example
+# herd set-http-proxy guix-daemon http://localhost:9250
+$ guix build --substitute-urls=https://bp7o7ckwlewr4slm.onion hello
+@end example


To make it copy/pastable, you can remove the prompt and write it as:

  sudo herd set-http-proxy …
  guix build …

Something along these lines LGTM.

Thank you!

Ludo’.


Thank you for the review Ludovic.

Pushed as c987b72382e739bf887849b02c533eda317ea52b with the 3 
modifications you

were requesting.

- Brice

[PATCH] doc: cookbook: Add entry about getting substitutes through Tor.

[Brice Waegeneire]

* doc/guix-cookbook.texi (Getting substitutes from Tor): New section.
---
 doc/guix-cookbook.texi | 55 ++
 1 file changed, 55 insertions(+)

diff --git a/doc/guix-cookbook.texi b/doc/guix-cookbook.texi
index 5574a60857..83abc704ca 100644
--- a/doc/guix-cookbook.texi
+++ b/doc/guix-cookbook.texi
@@ -14,6 +14,7 @@ Copyright @copyright{} 2019 Pierre Neidhardt@*
 Copyright @copyright{} 2020 Oleg Pykhalov@*
 Copyright @copyright{} 2020 Matthew Brooks@*
 Copyright @copyright{} 2020 Marcin Karpezo@*
+Copyright @copyright{} 2020 Brice Waegeneire@*
 
 Permission is granted to copy, distribute and/or modify this document
 under the terms of the GNU Free Documentation License, Version 1.3 or
@@ -1326,6 +1327,7 @@ reference.
 * Connecting to Wireguard VPN::  Connecting to a Wireguard VPN.
 * Customizing a Window Manager:: Handle customization of a Window manager on 
Guix System.
 * Setting up a bind mount:: Setting up a bind mount in the file-systems 
definition.
+* Getting substitutes from Tor:: Configuring Guix daemon to get substitutes 
through Tor.
 @end menu
 
 @node Customizing the Kernel
@@ -1785,6 +1787,59 @@ mount itself.
 ))
 @end lisp
 
+@node Getting substitutes from Tor
+@section Getting substitutes from Tor
+
+@quotation Warning
+@emph{Not all} Guix daemon's traffic will go through Tor!  Only
+HTTP/HTTPS will get proxied; FTP, Git protocol, SSH, etc connections
+will still go through the clearnet.  Again, this configuration isn't
+foolproof some of your traffic won't get routed by Tor at all.  Use it
+at your own risk.
+@end quotation
+
+Guix's substitute server is available as a hidden service, if you want
+to use it to get your substitutes from Tor configure your system as
+follow:
+
+@lisp
+(use-modules (gnu))
+(use-service-module base networking)
+
+(operating-system
+  …
+  (services
+(cons
+  (service tor-service-type
+  (tor-configuration
+(config-file (plain-file "tor-config"
+ "HTTPTunnelPort 127.0.0.1:9250"
+  (modify-services %base-services
+   (guix-service-type
+ config => (guix-configuration
+ (inherit config)
+ ;; ci.guix.gnu.org's hidden service
+ (substitute-urls 
"https://bp7o7ckwlewr4slm.onion";)
+ (http-proxy "http://localhost:9250";)))
+@end lisp
+
+This will keep a tor process running that provides a HTTP CONNECT tunnel
+which will be used by @command{guix-daemon}.  The daemon can use other
+protocols than HTTP(S) to get remote resources, request using those
+protocols won't go through Tor since we are only setting a HTTP tunnel
+here.  Note that @code{substitutes-urls} is using HTTPS and not HTTP or
+it won't work, that's a limitation of Tor's tunnel; you may want to use
+@command{privoxy} instead to avoid such limitations.
+
+If you don't want to always get substitutes through Tor but using it just
+some of the times, then skip the @code{guix-configuration}.  When you
+want to get a substitute from the Tor tunnel run:
+
+@example
+# herd set-http-proxy guix-daemon http://localhost:9250
+$ guix build --substitute-urls=https://bp7o7ckwlewr4slm.onion hello
+@end example
+
 @c *
 @node Advanced package management
 @chapter Advanced package management
-- 
2.26.2

Re: Routing Guix services traffic trough Tor

[Brice Waegeneire]

On 2020-05-17 22:33, Ludovic Courtès wrote:

Hi Brice,

Brice Waegeneire  skribis:


Today I played a bit with Tor and Guix, trying to fetch substitutes
trough
the Tor network as blaze_cornbread asked on IRC[0] how to do this.  I
managed to get it working but in the end I don't think we should
encourage
people doing it this way, that's why I haven't submitted a patch to 
the
cookbook for it.  Currently the only supported way to proxy traffic 
for

'guix-daemon' is by setting a HTTP proxy[1] the drawback is that DNS
query
will still be in clear and wont go trough the proxy in contrast to a
SOCKS5
proxy where the query will happen on the other side of the proxy.


I don’t think that’s the case: when an HTTP proxy is in use, clients
make a CONNECT or GET HTTP request to the proxy, which resolves the 
host

name on their behalf.  That’s why you can pass
‘--substitute-urls=http://bp7o7ckwlewr4slm.onion’ and it Just Works.

So I think you message could make a great section in the cookbook.  :-)

Thanks,
Ludo’.

Kernel module configuration service

[Brice Waegeneire]

Hello Guix,

I'm working on the future 'kernel-module-configuration-service-type'[0]
(KMCS) and I need some guidance. This service will be a one stop shop 
for

all kernel module related configuration by doing the following 4 things:
1. Generate a config directory for modrope to use
2. Load loadable kernel module by extending 
'kernel-module-loader-service'

3. Configure built-in module by being a new source for
   'operating-system-kernel-arguments'
4. Add loadable kernel modules to the kernel profile by being a source 
for

   'operating-system-kernel-loadable-modules'

ATM I need help with point number 1 in regard to putting in place the
config directory for modprobe. To do so I need to change
'%modprobe-wrapper' into a procedure to take the configuration directory 
as
an argument, since the directory will now come from '/gnu/store/' 
instead

of '/etc/'. The problem is that the wrapper is currently put in place by
'%linux-bare-metal-service' which is an essential service, so it seems 
that
KMCS will have to become one too. Looking around at some essential 
service

most of them are straight forward and don't extend services like
'kernel-module-loader-service-type' (which itself extend
'shepherd-root-service-type').

What are the guidelines to add a service to the essential-services? Can
KMCS become an essential-service? Any other remarks/idea?

[0]: https://issues.guix.info/40274#18

Have a good day,
- Brice

Re: best practise between git-fetch vs url-fetch?

[Brice Waegeneire]

Hello,

On 2020-05-13 01:08, zimoun wrote:

Based on these 2 messages [1,2], what is the consensus between
git-fetch and url-fetch?


I was hoping that some one bring this up, thanks.


Pushing to SWH when linting appears to me winning the pros/cons.  Even
if SWH should eventually fetch http://guix.gnu.org/sources.json soon.
And the other big pros from my point of view is the content-addressed
Git references.


Git references being content-addressed is important because it
make it more difficult for a lazy upstream developer to replace a
tarball in place -because it was somehow broken and they didn't
wanted to bump the version number- and broke a package.  Instead
with git, to broke a package in similar way (with a hash mismatch)
that developer would have to do “git push --force” which is much
more frowned upon since it will affect all the users of that git repo.

An other argument in recommending the 'git-fetch' method is that it
facilitate the use of “guix build --with-commit”:
1. You don't need to find out the git-url of the package,
2. Nor finding which dependencies are missing compared to a tarball
   build (often it's autoconf, libtool, & co.),
3. You don't need to manually tweak phases relating to autoconf and
   the like.
All in all recommending 'git-fetch' make “--with-commit” really more
useful.  This feature is one of my favorite from Guix since it make
testing a patch from an upstream developer really easy, eg.:
“guix build picom --with-commit=picom=vNext”.


Well, does it make sense to state on a recommended method?


It does, it avoid having to discuss it with each new contributor
and avoid noise in patches about changing the source's method based
on each developer preference (I'm personally guilty of that).


[1] https://lists.gnu.org/archive/html/guix-devel/2020-03/msg00091.html
On Fri, 6 Mar 2020 at 18:41, Marius Bakke  wrote:


url-fetch requires less bandwidth, and does not depend on 'git'.

Though the most important distinction is that uploaded releases
sometimes contain pre-processed sources (e.g. documentation) that need
additional dependencies or scripts when building from the raw 
repository
(this is why you often need to add autoconf, libtool & friends as 
inputs

when building Autotools projects from git).

I don't know whether there is a difference between the uploaded fmt
zipball and the git repository.



[2] https://lists.gnu.org/archive/html/guix-devel/2020-03/msg00189.html
On Wed, 11 Mar 2020 at 15:39, Ludovic Courtès  wrote:


Other considerations:

  - Bandwidth requirement for source code downloads has never been a
criterion so far.

  - Git references are nice because they’re (roughly) 
content-addressed.


  - ‘guix lint -c archival’ archives Git references on Software
Heritage; it does not archive tarballs (though SWH will do it
for us eventually.)


Cheers,
- Brice

Routing Guix services traffic trough Tor

[Brice Waegeneire]

Hello Guix,

Today I played a bit with Tor and Guix, trying to fetch substitutes 
trough

the Tor network as blaze_cornbread asked on IRC[0] how to do this.  I
managed to get it working but in the end I don't think we should 
encourage

people doing it this way, that's why I haven't submitted a patch to the
cookbook for it.  Currently the only supported way to proxy traffic for
'guix-daemon' is by setting a HTTP proxy[1] the drawback is that DNS 
query
will still be in clear and wont go trough the proxy in contrast to a 
SOCKS5

proxy where the query will happen on the other side of the proxy.  So
setting guix-daemon to use tor by this mean can put people at risk when
they think that all their guix traffic go trough tor™.

A better approach would be to have a mean to "torify" services with
torsocks, it would proxy the service's traffic (DNS included) trough tor 
via

a SOCKS5 proxy. I don't know how to implement such feature tho. But a
generic method to modify a shepherd service from the configuration could
also be helpful to start service in containers based on the user need
instead of being tied to

The two following examples are **insecure** since the DNS traffic won't 
go

trough tor.  Here is a example of a system configuration:

--8<---cut here---start->8---
(use-modules (gnu))
(use-service-module base networking)

(operating-system
  …
  (services
(append
  (list ((service tor-service-type
  (tor-configuration
(config-file (plain-file "tor-config"
 "HTTPTunnelPort 
127.0.0.1:9052"))

  (modify-services %base-services
  (guix-service-type
   config => (guix-configuration
  (http-proxy 
"http://localhost:9052";)))

--8<---cut here---end--->8---

Following is an example on how to do it, in a less Guixy way, by using
privoxy; it assume a default configured tor service is already present 
on

your system..

--8<---cut here---start->8---
$ sudo herd start tor
Service tor has been started.
$ cat privoxy-tor.conf
forward-socks5 / localhost:9050 .
$ privoxy privoxy-tor.conf
$ sudo herd set-http-proxy guix-daemon http://localhost:8118
changing HTTP/HTTPS proxy of 'guix-daemon' to "http://localhost:8118";...
Service guix-daemon has been stopped.
Service guix-daemon has been started.
$ LANGUAGE=C guix build audacity
substitute: mise à jour des substituts depuis « https://ci.guix.gnu.org 
»... 100.0 %

The following derivation will be built:
   /gnu/store/lz209608z1lw3zbw33hyp3rsx1az2khi-audacity-2.3.3.drv
38,1 MB will be downloaded:
   /gnu/store/ssc6x6dsxz3f5b26p84d02z42lcj8p3h-lv2-1.18.0
   /gnu/store/przpq26zaj858zmyayns6i4y13hr3d32-suil-0.10.6
   /gnu/store/y74d9xvxl33vra8aq9p3ywsvc8yaz04w-portmidi-217
   /gnu/store/2xmhv8ra20bhj73d3qirqbskdpq3lsim-vamp-2.6
   /gnu/store/1j3nhsacnqilyr4gqccfh9bzb33xvqak-audacity-2.3.3.tar.xz
   /gnu/store/bpp52ds6g1709s2h1ln1i81hz4v7gw6h-serd-0.30.4
   /gnu/store/vwx0zf02r9vxja8rmy6vs8w81907w3bz-sord-0.16.4
   /gnu/store/0ci33f2s2bm9rwply6b47sj6vn10ybaw-sratom-0.6.4
   /gnu/store/b5liczxlxxdhf9p8s61mx21v9x7rbsbi-lilv-0.24.6
substituting 
/gnu/store/1j3nhsacnqilyr4gqccfh9bzb33xvqak-audacity-2.3.3.tar.xz...
downloading from 
https://ci.guix.gnu.org/nar/1j3nhsacnqilyr4gqccfh9bzb33xvqak-audacity-2.3.3.tar.xz 
...
 audacity-2.3.3.tar.xz  35.7MiB  
548KiB/s 00:02 [  ]   3.1

--8<---cut here---end--->8---

If during the download of the substitutes the tor service is stopped 
with

“sudo herd stop tor” guix will stop too and complains about a network
error, as expected.  The above setup can be tweaked to proxy trough SSH
instead by doing port forwarding trough SOCKS “ssh -D 8008 my-host” 
(don't

forget to adjust the privoxy config for the port you are forwarding).

PS: Do not try to modify the shepherd guix-daemon service to use 
torsocks
or you'll wont be able to reconfigure, switch-generation or rollback: 
“guix

system: error: while setting up the build environment: cannot open IP
socket: Operation not permitted”.

PPS: The substitutes server are available trough tor
“--substitute-urls=http://bp7o7ckwlewr4slm.onion”.

[0]: http://logs.guix.gnu.org/guix/2020-05-12.log#093952
[1]: 
https://guix.gnu.org/manual/devel/en/html_node/Proxy-Settings.html#Proxy-Settings


- Brice

New commiter

[Brice Waegeneire]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Hello Guix,

You may have seen me around in the mailing lists or on IRC as bricewge. 
For
some months now I have been contributing to Guix on a regular basis. And 
it
seems I have been helpful in some ways since yesterday I was granted 
commit

access to the git repo. It's an honor to be on board, thank you all.

I'll sign my commits with the GPG key
8929 BBC5 73CD 9206 3DDD  979D 3D36 CAA0 116F 0F99
which you can find on my Savannah profile[0]. I hope I'll be able to 
serve

well the GNU Guix project!

[0]: https://savannah.gnu.org/users/bricewge

See you around,
- - Brice
-BEGIN PGP SIGNATURE-

iQIzBAEBCAAdFiEEHJvwBRoaakRSV1maqUkDoWahj64FAl6sPpAACgkQqUkDoWah
j646QBAAtVmF0TiWAR8PlMa4WepCat9B1ks5/TRMPNKs1Pp6cuqUHNK/vhVrWXgf
mGMCVM73GsOYLScXz4j7nvdaCP3SldnxrvpIzvzma67ehcCc46vvJEf2KVl5pI3+
pc5BTqiJcyjc1CSFj+SvzE0oMT8q/MLyzS8eTGw3/KxROeHVjU43TVfVbqzG5w/r
+UQAfLrAE4/I0VKz5hMzpZdliHcKy/pRNAeEZVOxxP8FWFMwp/hS0nETs2WajWwi
BkQXTXRdxyEMx5oB4L7eJ5w9jbZFA8ANlTHduOtaTv2WX2feiP8Yxx0bXbfWbO6e
Pv/MMC5ARfU1zdy5V+RhHHVgDIFH9o6KpYFC/hlJXmvYp/KzauLvBCf0uJ1b4O92
E8uJ1u1IXq2u7BvQ9xawU5Qkkca/jc0DXdfFQoVDLP89VhWmmzqXzbtyj6L6jLoT
uiPmXVhjruOy2p9qehYYH0+F+FqfAifTDY+tlheBkekVDk2P1iwR3vjf52EnT1YY
siCr3w79lrS0b2bsQrbprQv2P773aVUUlrEU7xyt7soLYcvJOQoloPFJHdu3MSyK
UT188Es7VdpFNJbwxovmgul/lri/53QwqZ/3OD5Om1Tp2PEekI7xXjMXvlKsCxMg
i+/IgxsfPzl3Aph2k1gBh/be6/RvOvVV7gBaziLQ8HMktB83Bts=
=d2eX
-END PGP SIGNATURE-

Re: iPXE network booting (was Re: [GSOC 2020] Booting via network)

[Brice Waegeneire]

Hello Giovanni,

On 2020-04-10 13:44, Giovanni Biscuolo wrote:


[...]

I never used iPXE but... please consider using iPXE (if possible) for
Guix network booting and consider that this feature is a prerequisite
for seamless remote desktop with Guix (using x2go or xrdp like the new
LTSP is doing [1]) in addition to "diskless fat clients"; a very cool
feature, I think :-D


Since the proposal has already been submitted I can't promise to add 
iPXE

support in the context of this GSoC, however I will keep it in mind when
implementing PXE booting. If I'm able to finish it early I'll look into
adding iPXE booting as a bonus.


In addition to LAN booting, iPXE supports booting from:

* a web server via HTTP/HTTPS
* an iSCSI SAN
* a Fibre Channel SAN via FCoE
* an AoE SAN
* a wireless network
* a wide-area network
* an Infiniband network

inlcuding "code signing" to verify the authenticity and integrity of
files downloaded by iPXE.

Users will have many interesting, configurable [2] and secure ways to
boot Guix with iPXE :-D (imagine booting from a remote host connected
via a wireguard network connection... could it be possible?!?)


Wow, that sounds cool. I would like to find out if it's possible too.



[...]

HTH! Thanks, Gio'


Yes it is helping, really, that was dense with information.

Cheers,
- Brice

Re: [GSOC 2020] Booting via network

[Brice Waegeneire]

Hello Vagrant,

On 2020-03-30 23:16, Vagrant Cascadian wrote:
I was just thinking Guix needed network boot support yesterday! Happy 
to

hear you're looking into it...

I've been the LTSP maintainer in Debian for almost 15 years, so have a
good amount of experience with network booting. LTSP was rewritten from
scratch last year and there might be elements of it useful to you:

  https://ltsp.org

None of it is scheme code, but there are possibly some useful ideas in
there you could make use of. One of the big changes is making extensive
use of iPXE, though that might need some further auditing to meet the
FSDG (Free Software Distribution Guidelines?) for inclusion into Guix.


I see the rewrite has been done during a GSoC too; it's interesting that 
it

has mostly been done in shell. It looks like a interesting project, I'll
keep reading...

- Brice

Re: [GSOC 2020] Booting via network

[Brice Waegeneire]

Hello Vincent,

On 2020-03-30 22:10, Vincent Legoll wrote:

Hello,

that's a great project, I hope to be able to lend a hand,
here and there...


Looks like you already started by packaging iPXE. :)
Thanks!

- Brice

Re: 01/02: services: Allow modprobe to use "/etc/modprobe.d".

[Brice Waegeneire]

Hello,

On 2020-04-07 09:35, Ludovic Courtès wrote:

Brice Waegeneire  skribis:

On 2020-04-05 21:15, Ludovic Courtès wrote:

guix-comm...@gnu.org skribis:
Looking at this, I was wondering if it would be possible to not use
/etc/modprobe.d and instead have a way to tell the modprobe wrapper 
to
pass “-C /gnu/store/…-modprobe.d”, which would contain the right 
thing.


Thoughts?


What's the issue with using /etc/modrpobe.d?


In general, use of the global file system name space isn’t great: it’s
ambiguous (compare to a /gnu/store reference) and doesn’t work well 
upon

rollback or reconfigure (things that refer to /etc end up referring to
the “new” /etc after reconfigure, even though that might not actually
work.)

Conversely, “-C /gnu/store/…-modprobe.d” unambiguously refers to the
intended directory.

WDYT?


It's good with me, I'll do as suggested -using the environment variable-
when implementing “kernel-module-configuratuion-service-type”.

- Brice

Re: 01/02: services: Allow modprobe to use "/etc/modprobe.d".

[Brice Waegeneire]

Hello Bengt,

On 2020-04-06 09:29, Bengt Richter wrote:

On +2020-04-06 07:54:47 +, Brice Waegeneire wrote:

What's the issue with using /etc/modrpobe.d?


I would think the fundamental issue is pure vs impure dependencies:
i.e., /gnu/... vs /var/guix vs /elsewhere/...

IIUC, the consequence of using /etc/... or ~/... or other non-/gnu/...
is that if you want to run something in a container with chrooted root,
you have to cow-fake /etc and all the rest of non-/gnu/... environment,
so your executable is not as generally usable as possible if
nuisance adjustments were not necessary.

People who might want to use it anyway have to think about a bunch
of stuff not relevant to what they actually want to do -- they
will wind up debugging functionally-irrelevant implementation stuff.

Maybe I misunderstand, but are you and Ludo on the same page
re the fundamental concept of guix and how it plays in various 
contexts?
(allowing for "practicality beats purity"[1] when absolutely necessary 
;-)


Yes, only the reason why to do it was eluding me. I'll keep in mind your 
rule

thumb.

- Brice

Re: [Help] Adding file-system utils packages to system profile

[Brice Waegeneire]

On 2020-04-06 17:07, Brice Waegeneire wrote:
I tried implementing this in the attached patch but I'm currently stuck 
and
need some help. I've probably overlooked something basic but I can;t 
put a

finger on it... Guix compile successfully with the patch and testing
“file-system-utils” in “guix repl” returns a list of packages as 
expected.

But when I try building a system/vm the packages aren't added to the
system's profile as I expected. From what I understand, that function 
as

used in “file-system-service-type” should get as arguments the list of
file-systems defined in the operating-system and should return a list 
of

packages, themselves appended to the system's profile list of packages.


It turns out “file-system-service-type” doesn't get *all* the 
file-systems,

just the one not need for boot.

With this issue fixed I have submitted it as patch #40480[0].

[0]: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=40480

- Brice

[Help] Adding file-system utils packages to system profile

[Brice Waegeneire]

Hello Guix,

There was several discussions about adding file-system utils packages to
the system profile based on the types in the “file-systems” field.

I tried implementing this in the attached patch but I'm currently stuck 
and
need some help. I've probably overlooked something basic but I can;t put 
a

finger on it... Guix compile successfully with the patch and testing
“file-system-utils” in “guix repl” returns a list of packages as 
expected.

But when I try building a system/vm the packages aren't added to the
system's profile as I expected. From what I understand, that function as
used in “file-system-service-type” should get as arguments the list of
file-systems defined in the operating-system and should return a list of
packages, themselves appended to the system's profile list of packages.

Halp! What am I missing?

I also welcome any suggestion UI wise, currently the only change is the
additions of the “utils?” filed in the “file-system” record.

- BriceFrom 57a6dc8c6ba2fb2b5ce97bffa26d61a430d2c16b Mon Sep 17 00:00:00 2001
From: Brice Waegeneire 
Date: Mon, 6 Apr 2020 18:00:11 +0200
Subject: [PATCH] services: Add file-system utils to profile.

---
 gnu/services/base.scm   | 42 +++--
 gnu/system/file-systems.scm |  6 +-
 2 files changed, 45 insertions(+), 3 deletions(-)

diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index 070765ab83..16519b5a7b 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -44,13 +44,20 @@
 #:select (file-system-packages))
   #:use-module (gnu packages admin)
   #:use-module ((gnu packages linux)
-#:select (alsa-utils crda eudev e2fsprogs fuse gpm kbd lvm2 rng-tools))
+#:select (alsa-utils btrfs-progs crda eudev eudev/btrfs-fix
+  e2fsprogs f2fs-tools fuse gpm kbd lvm2 rng-tools
+  util-linux xfsprogs))
   #:use-module (gnu packages bash)
   #:use-module ((gnu packages base)
 #:select (canonical-package coreutils glibc glibc-utf8-locales))
   #:use-module (gnu packages package-management)
   #:use-module ((gnu packages gnupg) #:select (guile-gcrypt))
-  #:use-module (gnu packages linux)
+  #:use-module ((gnu packages disk)
+#:select (dosfstools))
+  #:use-module ((gnu packages file-systems)
+#:select (bcachefs-tools jfsutils zfs))
+  #:use-module ((gnu packages mtools)
+#:select (exfat-utils))
   #:use-module (gnu packages terminals)
   #:use-module ((gnu build file-systems)
 #:select (mount-flags->bit-mask))
@@ -59,8 +66,10 @@
   #:use-module (guix modules)
   #:use-module ((guix self) #:select (make-config.scm))
   #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-2)
   #:use-module (srfi srfi-26)
   #:use-module (ice-9 match)
+  #:use-module (ice-9 regex)
   #:use-module (ice-9 format)
   #:export (fstab-service-type
 root-file-system-service
@@ -535,6 +544,33 @@ FILE-SYSTEM."
 (memq 'bind-mount (file-system-flags file-system
   file-systems))
 
+(define (file-system-type->utils type)
+  "Return a utils package for file system TYPE."
+  (define pattern->utils
+`(("ext[234]" . ,e2fsprogs)
+  ("btrfs" . ,btrfs-progs)
+  ("jfs" . ,jfsutils)
+  ("exfat" . ,exfat-utils)
+  ("bachefs" . ,bcachefs-tools)
+  ("xfs" . ,xfsprogs)
+  ("fat" . ,dosfstools)
+  ("f2fs" . ,f2fs-tools)
+  ("zfs" . ,zfs)))
+  (and-let* ((utils
+  (find (lambda (a) (string-match (car a) type)) pattern->utils)))
+(cdr utils)))
+
+(define (file-system-utils file-systems)
+  "Return the list of file-system utils packages for FILE-SYSTEMS"
+  (fold (lambda (file-system prev)
+  (let ((utils? (file-system-utils? file-system))
+(utils (file-system-type->utils (file-system-type file-system
+(if (and utils? utils
+ (not (member utils prev)))
+(cons* utils prev)
+prev)))
+'() file-systems))
+
 (define file-system-service-type
   (service-type (name 'file-systems)
 (extensions
@@ -542,6 +578,8 @@ FILE-SYSTEM."
   file-system-shepherd-services)
(service-extension fstab-service-type
   file-system-fstab-entries)
+   (service-extension profile-service-type
+  file-system-utils)
 
;; Have 'user-processes' depend on 'file-systems'.
(service-extension user-processes-service-type
diff --git a/gnu/system/file-systems.scm b/gnu/system/file-systems.scm
index 3b599efa8e..9bc

Re: 01/02: services: Allow modprobe to use "/etc/modprobe.d".

[Brice Waegeneire]

Hello Ludo',

On 2020-04-05 21:15, Ludovic Courtès wrote:

guix-comm...@gnu.org skribis:

   #~(begin
   (setenv "LINUX_MODULE_DIRECTORY"
   
"/run/booted-system/kernel/lib/modules")
+  ;; FIXME: Remove this crutch when the patch 
#40422,

+  ;; updating to kmod 27 is merged.
+  (setenv "MODPROBE_OPTIONS"
+  "-C /etc/modprobe.d")


[...]


+  (services (cons* (service kernel-module-loader-service-type
+'("ddcci" "ddcci_backlight"))
+   (simple-service 'ddcci-config etc-service-type
+   (list `("modprobe.d/ddcci.conf"
+   ,ddcci-config)))
+   %base-services))


Looking at this, I was wondering if it would be possible to not use
/etc/modprobe.d and instead have a way to tell the modprobe wrapper to
pass “-C /gnu/store/…-modprobe.d”, which would contain the right thing.

Thoughts?


What's the issue with using /etc/modrpobe.d?

As noted in the comments I thought setting MODPROBE_OPTIONS was kinda of 
a
hack; #40422[0] was there to fix it. But if you think it's appropriate 
to

use this environment variable it can be done in a future
“kernel-module-configuration-service-type” we discussed with Danny[1].
Instead of extending “etc-service-type” we would use
“activation-service-type”, as “%modprobe-wrapper” is currently put
in place by a simple activation service.

[0]: https://issues.guix.info/issue/40422
[1]: https://issues.guix.info/issue/40274#29

- Brice

Re: Adding a %desktop-packages

[Brice Waegeneire]

Hello John,

On 2020-04-04 00:22, John Soo wrote:

I raised the idea of providing smaller lists of packages that might go
well together instead of one large %desktop-packages. One reason to do
this, for instance, might be to not make someone who wants to use btrfs
always import the ext4 packages. Or not lock someone into using 
nettools

if they are using iproute2, etc.


Regarding the file-system utils packages I'm working on a patch adding 
them
to the system profile based on the type of file-system defined. So, in 
the
end, `e2fsprogs`, `btrfs-progs` and such will be removed from 
%base-packages.


Also I've just submitted a patch[0] to split %base-packages as we talked
about on IRC a couple of weeks ago.

[0]: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=40423

- Brice

Re: Issues and improvement for `kernel-loadable-modules'

[Brice Waegeneire]

Hello Danny,

Sorry for the empty email; cancel and send buttons were too close for
me...

On 2020-03-26 15:13, Danny Milosavljevic wrote:

Hi Brice,

On Thu, 26 Mar 2020 14:34:03 +
Brice Waegeneire  wrote:


First I was expecting the packages in `kernel-loadable-modules' to use
the
`kernel' field as their kernel input or to have a simple procedure to 
do
so. Otherwise you get a “Specified Linux kernel and Linux kernel 
modules

are not all of the same version”. It makes it more difficult that it
needs
to be to write composable configurations; IOW why would I want to use 
a

module built with a different kernel that the one I'm specifying in my
`operating-system'.


Because packages are modular, changing the build system can only be 
done
retroactively by a procedure.  We could totally do that but it would 
make
the kernel-loadable-modules field more magical and more difficult to 
debug.

Also, I guess it would hard-code linux-module-build-system for those
(right now you can use whatever build system you want).  Do we want it 
anyway?


Magic isn't the way to go, for sure. I tried writting a procedure to 
abstract
it, but I wasn't successful: see my second issue with inferiors, the 
fact that
it only handle `linux-build-system' and that I need to use a direct 
reference to

a package. On that last point I would like to use something like
“(operating-system-kernel this-operating-system)” to reference the 
kernel
but my guile foo isn't good enough: “this-operating-system” has to be in 
scope

 of “operating-system”. Following is the procedure I'm talking about:

--8<---cut here---start->8---
  (kernel-loadable-modules
   (map (lambda (module)
  (package/inherit module
   (arguments
(ensure-keyword-arguments
 (package-arguments module)
 `(#:linux ,my-linux)
 
--8<---cut here---end--->8---



And last issue, we are missing a service to load module manually when
they
can't be auto-loaded as it's the case with `ddcci`[1]. I have managed 
to

solve this one by writing my first service
`load-kernel-modules-service'.
What can I improve before submitting it as a patch -- except the 
missing

documentation?


I think that things should be described by nouns and actions should be
described by verbs.

So "load-kernel-modules-service" sounds really wrong to me.
Maybe "kernel-module-loader-service"?

Others don't care so much because Scheme kinda erodes the boundary 
anyway.


That's definitely a better name.


Otherwise looks good to me.

I guess it could be nice to be able to extend this service from other 
services

using (service-extension load-kernel-modules-service-type '("module1"
"module2")) or something.


I think it's already the case, there are `compose' and `extend' fields 
in the

`service-type' and I tried it with
“(simple-service 'ddcci-module kernel-module-loader-service-type 
'("ddcci"))”.


Brice.

Re: Issues and improvement for `kernel-loadable-modules'

[Brice Waegeneire]

On 2020-03-26 15:13, Danny Milosavljevic wrote:

Hi Brice,

On Thu, 26 Mar 2020 14:34:03 +
Brice Waegeneire  wrote:


First I was expecting the packages in `kernel-loadable-modules' to use
the
`kernel' field as their kernel input or to have a simple procedure to 
do
so. Otherwise you get a “Specified Linux kernel and Linux kernel 
modules

are not all of the same version”. It makes it more difficult that it
needs
to be to write composable configurations; IOW why would I want to use 
a

module built with a different kernel that the one I'm specifying in my
`operating-system'.


Because packages are modular, changing the build system can only be 
done
retroactively by a procedure.  We could totally do that but it would 
make
the kernel-loadable-modules field more magical and more difficult to 
debug.

Also, I guess it would hard-code linux-module-build-system for those
(right now you can use whatever build system you want).  Do we want it 
anyway?



And last issue, we are missing a service to load module manually when
they
can't be auto-loaded as it's the case with `ddcci`[1]. I have managed 
to

solve this one by writing my first service
`load-kernel-modules-service'.
What can I improve before submitting it as a patch -- except the 
missing

documentation?


I think that things should be described by nouns and actions should be
described by verbs.

So "load-kernel-modules-service" sounds really wrong to me.
Maybe "kernel-module-loader-service"?

Others don't care so much because Scheme kinda erodes the boundary 
anyway.


Otherwise looks good to me.

I guess it could be nice to be able to extend this service from other 
services

using (service-extension load-kernel-modules-service-type '("module1"
"module2")) or something.

Enhancing `modify-services' to support name in addition to type

[Brice Waegeneire]

Hello,

I was thinking of improving `modify-services' by adding the ability to
specify a service to modify based on it's name and not just it's type. 
This

would allow us to modify singleton services like the ones returned by
`simple-service'. I'm not sure if that's a good idea, that's why I 
prefer
to ask before starting to write a patch. Following is an example of how 
it

would like when modifying `set-xorg-configuration':

--8<---cut here---start->8---
(define t430-xorg-extra-config "
Section \"InputClass\"
  Identifier \"touchpad\"
  Driver \"synaptics\"
  Option \"HorizTwoFingerScroll\" \"on\"
EndSection")

(define t430-services
  (modify-services workstation-services
('set-xorg-configuration
 config => (xorg-configuration
(inherit config)
(extra-config (list t430-xorg-extra-config))
--8<---cut here---end--->8---

Brice.

Issues and improvement for `kernel-loadable-modules'

[Brice Waegeneire]

Hello Guix,

Thanks to Danny's work in[0] we have, since a few days, a way for 
packages

to provide Linux modules in the system profile. I have been waiting for
such a feature since I packaged `ddcci-driver-linux', which was kind of
useless without it. Using the new field `kernel-loadable-modules' I
stumbled upon three issues.

First I was expecting the packages in `kernel-loadable-modules' to use 
the

`kernel' field as their kernel input or to have a simple procedure to do
so. Otherwise you get a “Specified Linux kernel and Linux kernel modules
are not all of the same version”. It makes it more difficult that it 
needs

to be to write composable configurations; IOW why would I want to use a
module built with a different kernel that the one I'm specifying in my
`operating-system'.

Second issue, I can't manage to use an inferior as the kernel input for 
the

packages in `kernel-loadable-modules'. See the two following
snippets:

--8<---cut here---start->8---
LANGUAGE=C guix system vm kernel-loadable-modules-barbones.scm
Updating channel 'guix' from Git repository at 
'https://git.savannah.gnu.org/git/guix.git'...

Backtrace:
   1 (primitive-load "/home/bricewge/.config/guix/current/bi…")
In guix/ui.scm:
  1894:12  0 (run-guix-command _ . _)

guix/ui.scm:1894:12: In procedure run-guix-command:
In procedure %package-native-inputs-real: Wrong type argument: 
#

--8<---cut here---end--->8---

Following is `kernel-loadable-modules-barbones.scm':

--8<---cut here---start->8---
(use-modules (gnu)
 (gnu system)
 (srfi srfi-1)
 (guix inferior)
 (guix utils)
 (guix packages)
 (guix channels))
(use-package-modules linux)

(define channels
  (list (channel
 (name 'guix)
 (url "https://git.savannah.gnu.org/git/guix.git";)
 (commit
  "591faabd8c93bfb6879910d8a424f0db835066c2"

(define my-linux
  (let ((inferior (inferior-for-channels channels)))
(first (lookup-inferior-packages inferior "linux-libre" 
"4.4.217"


;; ;; NOTE It is already in master
;; (define operating-system-kernel-loadable-modules
;; (@@ (gnu system) operating-system-kernel-loadable-modules))

(define os
  (operating-system
(host-name "komputilo")
(timezone "Europe/Berlin")
(locale "en_US.utf8")
(bootloader (bootloader-configuration
 (bootloader grub-bootloader)
 (target "/dev/sdX")))
(file-systems (cons (file-system
  (device (file-system-label "my-root"))
  (mount-point "/")
  (type "ext4"))
%base-file-systems))

(kernel-loadable-modules (list ddcci-driver-linux

(operating-system
  (inherit os)

  (kernel my-linux)
  (kernel-loadable-modules
   (map (lambda (module)
  (package/inherit module
   (arguments
(ensure-keyword-arguments
 (package-arguments module)
 ;; `(#:linux ,(specification->package 
"linux@4.4.217")) ; NOTE It should works

 `(#:linux ,my-linux) ; NOTE That's issue #2
 
(operating-system-kernel-loadable-modules os
--8<---cut here---end--->8---

And last issue, we are missing a service to load module manually when 
they

can't be auto-loaded as it's the case with `ddcci`[1]. I have managed to
solve this one by writing my first service 
`load-kernel-modules-service'.

What can I improve before submitting it as a patch -- except the missing
documentation?

--8<---cut here---start->8---
(define-record-type* 
load-kernel-modules-configuration make-load-kernel-modules-configuration
load-kernel-modules-configuration? (modules
load-kernel-modules-configuration-modules ; list of strings (default 
'(


(define load-kernel-modules-shepherd-service
  (match-lambda
(($  modules)
 (list
  (shepherd-service
   (documentation "Load kernel modules.")
   (provision '(load-kernel-modules))
   (respawn? #f)
   (one-shot? #t)
   (start
#~(lambda _
(zero? (system* #$(file-append kmod "/bin/modprobe")
#$@modules)

(define load-kernel-modules-service-type
  (service-type
   (name 'load-kernel-modules)
   (description "Load kernel modules.")
   (extensions
(list
 (service-extension shepherd-root-service-type
load-kernel-modules-shepherd-service)))
   (compose concatenate)
   (extend (lambda (config modules)
 (load-kernel-modules-configuration
  (inherit config)
  (modules (append
(load-kernel-modules-configuration-modules 
config)

Re: native-search-paths and shepherd services (help wanted)

[Brice Waegeneire]

On 2020-02-08 19:10, Jonathan Frederickson wrote:

I'm still not quite sure I have a good understanding of when
'native-search-paths' applies and when it doesn't, but... at least
setting the env var directly seems to work in this case.


I just learned about native-search-paths when looking into your issue so 
I may be off about how it works...
What I understand is that native-search-paths add an entry to your 
/etc/profile when installed - for your default user profile it's 
$HOME/.guix-profile/etc/profile. In our case, when installed, minetest 
sets the environment variable MINETEST_SUBGAME_PATH so the binary know 
where to looks for the games. But when called directly, as it's the case 
in a shepherd service, the /etc/profile file isn't sourced so the 
environment variable isn't set. That's why we need to set it manually in 
the service.

Re: native-search-paths and shepherd services (help wanted)

[Brice Waegeneire]

On 2020-02-08 09:06, Jonathan Frederickson wrote:

Hi - I'm working on a Guix service for Minetest, and I'm running into
some issues. The Guix package for Minetest is divided into two
variables: "minetest" and "minetest-data", with only the former being
an installable package. The Minetest package uses native-search-paths
to allow Minetest to find additional available games
(MINETEST_SUBGAME_PATH), *including* minetest_game which is
effectively the default.

This is all well and good when you're installing Minetest as a package
and running it manually, but I'd like to do this with a Guix service
instead. Problem is... the native-search-paths don't seem to affect
the environment variables set when I run Minetest through Shepherd.
I'm checking my env vars by running Minetest through strace in the
shepherd service at the moment:

  (start #~(make-forkexec-constructor
(list #$(file-append strace "/bin/strace") "-s1024"
"-vfe" "trace=execve" #$(file-append package "/bin/minetest")
  "--server" "testworld"
  "--map-dir" #$map-dir)
#:user "minetest"
#:group "minetest"))

(I've attached my current services/games.scm for reference.)

Is there a way for me to use the native-search-paths from the
installed Minetest package when starting it through Shepherd?


I think I got it working, I managed to log and "play" on the server.

You were really close to have a working service for minetest though. You 
were only missing the environment variable MINETEST_SUBGAME_PATH and 
*maybe* your profile-service-type extension was wrong (I just blandly 
copied how it was done for SDDM). I have attached your games.scm with 
the added fixes.
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2018 Arun Isaac 
;;;
;;; This file is part of GNU Guix.
;;;
;;; GNU Guix is free software; you can redistribute it and/or modify it
;;; under the terms of the GNU General Public License as published by
;;; the Free Software Foundation; either version 3 of the License, or (at
;;; your option) any later version.
;;;
;;; GNU Guix is distributed in the hope that it will be useful, but
;;; WITHOUT ANY WARRANTY; without even the implied warranty of
;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;;; GNU General Public License for more details.
;;;
;;; You should have received a copy of the GNU General Public License
;;; along with GNU Guix.  If not, see .

(define-module (gnu services games)
  #:use-module (gnu services)
  #:use-module (gnu services shepherd)
  #:use-module (gnu packages admin)
  #:use-module (gnu packages linux) ;; for strace - remove this
  #:use-module (gnu packages games)
  #:use-module (gnu system shadow)
  #:use-module (guix gexp)
  #:use-module (guix modules)
  #:use-module (guix records)
  #:use-module (ice-9 match)
  #:export (wesnothd-configuration
wesnothd-configuration?
wesnothd-service-type
minetest-configuration
minetest-configuration?
minetest-service-type))

;;;
;;; The Battle for Wesnoth server
;;;

(define-record-type* 
  wesnothd-configuration make-wesnothd-configuration wesnothd-configuration?
  (package wesnothd-configuration-package
   (default wesnoth-server))
  (port wesnothd-configuration-port
(default 15000)))

(define %wesnothd-accounts
  (list
   (user-account
(name "wesnothd")
(group "wesnothd")
(system? #t)
(comment "Wesnoth daemon user")
(home-directory "/var/empty")
(shell
 (file-append shadow "/sbin/nologin")))
   (user-group
(name "wesnothd")
(system? #t

(define wesnothd-shepherd-service
  (match-lambda
(($  package port)
 (with-imported-modules
 (source-module-closure
  '((gnu build shepherd)))
   (shepherd-service
(documentation "The Battle for Wesnoth server")
(provision
 '(wesnoth-daemon))
(requirement
 '(networking))
(modules
 '((gnu build shepherd)))
(start #~(make-forkexec-constructor/container
  (list #$(file-append package "/bin/wesnothd")
"-p" #$(number->string port))
  #:user "wesnothd" #:group "wesnothd"))
(stop #~(make-kill-destructor)))

(define wesnothd-service-type
  (service-type
   (name 'wesnothd)
   (description
"Run The Battle for Wesnoth server @command{wesnothd}.")
   (extensions
(list
 (service-extension account-service-type
(const %wesnothd-accounts))
 (service-extension shepherd-root-service-type
(compose list wesnothd-shepherd-service
   (default-value
 (wesnothd-configuration

(define-record-type* 
  minetest-configuration make-minetest-configuration minetest-configuration?
  (package minetest-configuration-package
   (default minetest))
  (config-file minetest-

Re: Documenting Yubikey setup

[Brice Waegeneire]

On 2020-01-23 14:22, Martin Becze wrote:

Did you write something for the cookbook?


The only thing that I know to put in the cookbook is the below
snippet. I think it should be expanded a bit. But I haven't had a
chance to futher explore using Yubikey. I still have some problems
using it with icecat that I need to figure out.


I also had no luck with u2f tokens in icecat, it's probably related with 
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=38341.
Though it works well with chromium when the udev rules are in place, the 
plugdev group exists and you are member of it.

Re: GRUB and installer invisible on serial console

[Brice Waegeneire]

On 2019-12-21 18:31, Ricardo Wurmus wrote:

Hi Guix,

I just wanted to install Guix System on a server where I currently only
have remote access via serial console — and which happens to have a 
Guix
USB stick stuck in its USB port.  I can see the system’s UEFI output 
and

see that it boots from USB, but “Welcome to GRUB!” is the very last
thing I see.  There is no further output.

Our GRUB is using fancy graphics, so I wonder if that means that it has
switched to a mode that I simply can’t see over my serial connection.

Is there a way to make it start GRUB in text mode — and stay in text
mode throughout the installation?

--
Ricardo


Hello Ricardo,

I got the same setup as you working some weeks ago with the following 
config.
I also had to tweak my BIOS settings to have a reliable serial 
connection.


--8<---cut here---start->8---
   (kernel-arguments
 '("quiet" "console=tty0" "console=ttyS0,115200"))

  (bootloader
(bootloader-configuration
  (bootloader grub-efi-bootloader)
  (target "/boot/efi")
  (keyboard-layout keyboard-layout)
  (terminal-outputs '(gfxterm serial))
  (terminal-inputs '(console serial))
  (serial-unit 0)
  (serial-speed 115200)))
--8<---cut here---end--->8---

The order of the kernel argument "console=" is important because
« The last device will be used when you open /dev/console » [1].
And note that my baudrate is not 9600, the default value, because it's 
too slow
when displaying files. If you have a low quality serial link use the 
default.


I was wondering about writing a patch to make this kind of setup the 
default for
the image installer to allow installing Guix through a serial connection 
only.

Unfortunately such a setup will remove the current eye-candy GRUB theme.

#38580 may also interest you, it adds access to the BIOS settings from 
GRUB when

using UEFI.

[1]: 
https://www.kernel.org/doc/html/latest/admin-guide/serial-console.html

[2]: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=38580