Re: ZFS on Guix
Hi guix-developers,
Another issue here is that ZFS prefers to not be managed via
`/etc/fstab`/`mount`/`umount`. Instead, at startup ZFS magically imports ZFS
pools and mounts ZFS datasets in the correct place, as configured in the
`mountpoint` properties of the dataset. This magic is actually implemented by
executing `zpool import -a` at startup after ZFS module loading. Note that
`zpool import -a` has to be executed always so that ZFS can detect its pools,
though the automatic mounting can be suppressed with `zpool import -a -N`.
(`systemd`-based distros have a number of `systemd` services that handle ZFS
importation by use of a `/etc/zfs/zpool.cache` file that ZFS maintains, and
then does `zfs mount -a` to do automounting, but I'll leave that for later.)
Now, properly speaking, the shepherd `file-systems` service should not be
started until we have actually performed ZFS automounting. This means that
`file-systems` has to have as a requirement the ZFS shepherd service that
implements the automounting. And of course it should *not* require that if ZFS
isn't installed in the system.
So, I made another patch that makes the `file-systems` shepherd service have an
extensible set of requirements, like `user-processes` does. It's below.
Actual `file-system`s declared in the operating system then make themselves
requirements of `file-systems`. Then, a ZFS automounting shepherd service can
be installed by the `zfs-service-type` and added as a requirement of the
`file-systems` shepherd service.
This is important since one possible use of ZFS is to have it provide the
`/home` filesystem. And the `/home` filesystem has to be mounted before
`user-homes` shepherd service starts. So the ZFS automounting shepherd service
has to be a requirement of `file-systems`, which is enabled by the below.
Please review!
>From 792a8f8efc95e4fe9a94d42f839ddcfb034b8540 Mon Sep 17 00:00:00 2001
From: raid5atemyhomework
Date: Wed, 6 Jan 2021 08:15:54 +0800
Subject: [PATCH] gnu: Make file-systems target extensible by services.
* gnu/services/base.scm (file-system-shepherd-services): Move file-systems
shepherd service to ...
(file-systems-target-shepherd-services): ... new procedure here.
(file-systems-target-service-type): New variable.
(file-system-service-type): Extend file-systems-target service to add each
file-system as a requirement.
* gnu/system.scm (operating-system-default-essential-services): Instantiate
file-systems-target-service-type.
(hurd-default-essential-services): Instantiate file-systems-target-service-type.
---
gnu/services/base.scm | 37 +++--
gnu/system.scm| 2 ++
2 files changed, 29 insertions(+), 10 deletions(-)
diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index 945b546607..13cfb6a8a2 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -13,6 +13,7 @@
;;; Copyright © 2019 Jan (janneke) Nieuwenhuizen
;;; Copyright © 2020 Florian Pelz
;;; Copyright © 2020 Brice Waegeneire
+;;; Copyright © 2021 raid5atemyhomework
;;;
;;; This file is part of GNU Guix.
;;;
@@ -67,6 +68,7 @@
#:export (fstab-service-type
root-file-system-service
file-system-service-type
+file-systems-target-service-type
swap-service
host-name-service
console-keymap-service
@@ -362,18 +364,29 @@ FILE-SYSTEM."
(gnu system file-systems)
,@%default-modules)))
+(define (file-systems-target-shepherd-services requirements)
+ (list
+(shepherd-service
+ (provision '(file-systems))
+ (requirement (cons* 'root-file-system 'user-file-systems requirements))
+ (documentation "Target for all the initially-mounted file systems")
+ (start #~(const #t))
+ (stop #~(const #t)
+(define file-systems-target-service-type
+ (service-type
+(name 'file-systems)
+(extensions (list (service-extension shepherd-root-service-type
+
file-systems-target-shepherd-services)))
+(compose concatenate)
+(extend append)
+;; Extensions can add new values to this list.
+(default-value '())
+(description "The @code{file-systems} service is the target that is started
+when all file systems have been mounted.")))
+
(define (file-system-shepherd-services file-systems)
"Return the list of Shepherd services for FILE-SYSTEMS."
(let* ((file-systems (filter file-system-mount? file-systems)))
-(define sink
- (shepherd-service
- (provision '(file-systems))
- (requirement (cons* 'root-file-system 'user-file-systems
- (map file-system->shepherd-service-name
-file-systems)))
- (documentation "Target for all the initially-mounted file systems")
- (start #~(const #t))
- (stop #~(const #f
(define known-mount-points
(map
Re: ZFS on Guix
Thank you Carlo! I've now tested the new tests I added in `gnu/tests/linux-modules.scm`, and the existing tests as well, and they all pass. Hope to get some review on that patch! Thanks raid5atemyhomework
[Outreachy] Strategy to implement guix git log --pretty=
Hello Guix, As you might know, as part of my Outreachy internship I'm currently working on implementing the subcommand 'guix git log', for browsing the history of all packages. So far, it works with '--oneline' and '--format=', and FORMAT can be 'oneline', 'medium' or 'full'. If you want to see it, the code can be found at https://gitlab.com/magalilemes/guix On the road to adding another option to the subcommand, '--pretty=' arose as an idea. With git log, you can do something like git log pretty= And this string can have placeholders, such as %h for showing the short hash of a commit, and %s for showing the commit subject. For instance, you could have git log --pretty="%h %s" and this would display the commit history log with the short hash and subject of commits. So, in order to implement 'guix git log --pretty=', I'd like help with a strategy to parse the string. Any examples, ideas and tips would be really appreciated. Cheers, Magali
Re: ZFS on Guix
Apologies for the short reply, but that permission issue can be solved in the short term by changing the permissions on /dev/kvm (I set it to 777 to force it when necessary, because I don't know what the right permissions are). The proper solution is probably to add your user to a kvm group on your distribution, which should give you the appropriate access.
Re: ZFS on Guix
Hi Carlo,
Thanks, I modified patch as below.
I also can't get `guix system vm` to run in my hacking env, so I couldn't make
a VM for testing like you did. I got this:
```
Formatting '/gnu/store/imdlq0cay61d2cw3199g04z9bp16qx8j-qemu-image', fmt=qcow2
cluster_size=65536 compression_type=zlib size=73400320 lazy_refcounts=off
refcount_bits=16
Could not access KVM kernel module: Permission denied
qemu-system-x86_64: failed to initialize kvm: Permission denied
```
Any tips on getting `guix system vm` working? I've never used VMs before. My
dev env is a foreign distro with Guix installed on top.
The above issue also means I haven't actually run the tests I made in
`gnu/tests/linux-modules.scm`, because it requires building a VM as well.
However, I *did* do a bunch of `guix system build` tests:
* With the same `configuration.scm` that uses the legacy
`(kernel-loadable-modules ...)` declaration in `operating-system`, I tried both
with and without the patch.
* Without patch and with patch created builds with a different hash, however,
they linked to the same directories. `diff -r` did not find any differences.
* With the patch, I made two variants of `configuration.scm`, one that used the
legacy `(kernel-loadable-modules...)` and another that used a service that
extended `kernel-laodable-module-service-type`.
* Builds resulted in the exact same system with exact same hash.
>From 4beb73c62995cf236b402dad8e1c36016027c781 Mon Sep 17 00:00:00 2001
From: raid5atemyhomework
Date: Tue, 5 Jan 2021 22:27:56 +0800
Subject: [PATCH] gnu: Allow services to install kernel-loadable modules.
* gnu/system.scm (operating-system-directory-base-entries): Remove code
to handle generation of "kernel" and "hurd".
(operating-system-default-essential-services): Instantiate
kernel-loadable-module-service.
(hurd-default-essential-services): Instantiate
kernel-loadable-module-service.
(package-for-kernel): Move ...
* gnu/services.scm: ... to here.
(kernel-loadable-module-service-type): New variable.
(kernel-loadable-module-service): New procedure.
* gnu/tests/linux-modules.scm (run-loadable-kernel-modules-test): Move
code to ...
(run-loadable-kernel-modules-test-base): ... new procedure here.
(run-loadable-kernel-modules-service-test): New procedure.
(%test-loadable-kernel-modules-service-0): New variable.
(%test-loadable-kernel-modules-service-1): New variable.
(%test-loadable-kernel-modules-service-2): New variable.
* doc/guix.texi: Document kernel-loadable-module-service-type.
---
doc/guix.texi | 6 +++
gnu/services.scm| 70
gnu/system.scm | 37 +
gnu/tests/linux-modules.scm | 81 -
4 files changed, 157 insertions(+), 37 deletions(-)
diff --git a/doc/guix.texi b/doc/guix.texi
index 0f6e95a65a..78770151e3 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -32409,6 +32409,12 @@ configuration when you use @command{guix system
reconfigure},
@command{guix system init}, or @command{guix deploy}.
@end defvr
+@defvr {Scheme Variable} kernel-loadable-module-service-type
+Type of the service that collects lists of packages containing
+kernel-loadable modules, and adds them to the set of kernel-loadable
+modules.
+@end defvr
+
@node Shepherd Services
@subsection Shepherd Services
diff --git a/gnu/services.scm b/gnu/services.scm
index 13259dfaee..d7332a46b2 100644
--- a/gnu/services.scm
+++ b/gnu/services.scm
@@ -2,6 +2,7 @@
;;; Copyright © 2015, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès
;;; Copyright © 2016 Chris Marusich
;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen
+;;; Copyright © 2021 raid5atemyhomework
;;;
;;; This file is part of GNU Guix.
;;;
@@ -33,6 +34,8 @@
#:use-module (guix diagnostics)
#:autoload (guix openpgp) (openpgp-format-fingerprint)
#:use-module (guix modules)
+ #:use-module (guix packages)
+ #:use-module (guix utils)
#:use-module (gnu packages base)
#:use-module (gnu packages bash)
#:use-module (gnu packages hurd)
@@ -75,6 +78,7 @@
service-back-edges
instantiate-missing-services
fold-services
+kernel-loadable-module-service
service-error?
missing-value-service-error?
@@ -106,6 +110,7 @@
profile-service-type
firmware-service-type
gc-root-service-type
+kernel-loadable-module-service-type
%boot-service
%activation-service
@@ -864,6 +869,71 @@ as Wifi cards.")))
will not be reclaimed by the garbage collector.")
(default-value '(
+;; Configuration for the kernel builder.
+(define-record-type*
kernel-builder-configuration
+ make-kernel-builder-configuration
+ kernel-builder-configuration?
+ this-kernel-builder-configuration
+
+ (kernel kernel-builder-configuration-kernel (default #f))
+ (hurd kernel-builder-configuration-hurd (default #f))
+
Re: ZFS on Guix
Hi raid5atemyhomework,
On Wed, Jan 06 2021, raid5atemyhomework wrote:
I have this patch below for creating a new
`kernel-loadable-module-service-type`, which can be extended by
another service to add kernel-loadable modules provided by
packages, hope for a review.
I tried out building a VM using this patch and it seemed to work
properly. I was able to add a kernel module by extending
kernel-loadable-module-service-type, and then load it in the
running VM with modprobe.
I only have superficial comments about the patch itself. Hopefully
someone else will provide a better review than I can.
From 984602faba1e18b9eb64e62970147aab653d997f Mon Sep 17
00:00:00 2001
From: raid5atemyhomework
Date: Tue, 5 Jan 2021 22:27:56 +0800
Subject: [PATCH] gnu: Add 'kernel-loadable-module-service-type'
for services
to extend with kernel-loadable modules.
This will need a proper commit message before being committed.
Guix generally follows the GNU Change Log style[1]. You can see
examples of what commit messages look like by looking at other
commits in the repository.
+;; Only used by the KERNEL-LOADABLE-MODULE-SERVICE-TYPE.
I would remove this comment. This is a description of the current
state, rather than a restriction on how it should be used. As a
description, it might get out of date (if we use
values elsewhere), and it doesn't
add much value (because it can be easily verified, and the record
type isn't exported). If you indent this to be normative, the
comment should explain why this type should not be used elsewhere.
+ (return `(("kernel" ,kernel)
+,@(if hurd `(("hurd" ,hurd)) '()))
+(define (kernel-builder-configuration-add-modules config
modules)
+ "Constructs a kernel builder configuration that has its
modules extended."
Put empty lines between definitions here ...
+ "Register packages containing kernel-loadable
modules and adds them
+to the system.")))
+(define (kernel-loadable-module-service kernel hurd modules)
+ "Constructs the service that sets up kernel loadable
modules."
... and here.
It would also be worth adding a test to
gnu/tests/linux-modules.scm to test loading modules through this
service.
Other than that, it looks good to me and it seems to work in my
limited testing. Good job!
Carlo
[1]: https://www.gnu.org/prep/standards/html_node/Change-Logs.html
Re: Staging branch [aarch64 failures]
On Tue, Jan 05, 2021 at 02:01:35PM +0200, Efraim Flashner wrote: > qtbase built for me using qemu-binfmt emulation. Send it through again? I had Cuirass retry the build, and it failed immediately: https://ci.guix.gnu.org/build/163857/details The log file reads: "while setting up the build environment: executing `/gnu/store/x3gq648qnfnla7nppyfjvj62s2i8y7rl-guile-3.0.2/bin/guile': No such file or directory" Does anybody have advice?
Re: Staging branch [aarch64 failures]
On Tue, Jan 05, 2021 at 02:01:35PM +0200, Efraim Flashner wrote: > On Mon, Jan 04, 2021 at 08:37:44PM -0500, Leo Famulari wrote: > > The branch is building again! > > > > http://ci.guix.gnu.org/eval/10974 > > > > Qtbase is failing on aarch64: > > > > https://ci.guix.gnu.org/build/166439/details > > > > There errors like this: > > > > -- > > g++ -c -pipe -O2 -w -fPIC -I. > > -I/gnu/store/48i8mxxb1v4x632dff3i1dbdhsazm8bw-mariadb-10.5.8-dev/include/mysql > > > > -I/gnu/store/48i8mxxb1v4x632dff3i1dbdhsazm8bw-mariadb-10.5.8-dev/include/mysql/mysql > > > > -I/tmp/guix-build-qtbase-5.15.2.drv-0/qtbase-everywhere-src-5.15.2/mkspecs/linux-g++ > > -o main.o main.cpp > > g++ -Wl,-O1 -o mysql main.o > > -L/gnu/store/c9id1jf4r3cfys6289p92wpdhk1ryns0-mariadb-10.5.8-lib/lib/ > > ld: main.o: in function `main': > > main.cpp:(.text.startup+0x8): undefined reference to > > `mysql_get_client_version' > > collect2: error: ld returned 1 exit status > > make: *** [Makefile:69: mysql] Error 1 > > -- > > qtbase built for me using qemu-binfmt emulation. Send it through again? > Also built for me just fine on my pine64. -- Efraim Flashner אפרים פלשנר GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted signature.asc Description: PGP signature
Re: Mailman From header rewrite (was: [bug#45644] closed (Re: [bug#45644] [PATCH] gnu: esbuild: Update to 0.8.29.))
On Tuesday, January 5th, 2021 at 5:01 AM, Kyle Meyer wrote: > add the appropriate "From:" header to the body of each patch. `git am` will > take the in-body header over the actual header. Thank you Kyle! I will give that a try.
Re: ZFS on Guix
Hi Carlo and guix-developers,
I have this patch below for creating a new
`kernel-loadable-module-service-type`, which can be extended by another service
to add kernel-loadable modules provided by packages, hope for a review.
>From 984602faba1e18b9eb64e62970147aab653d997f Mon Sep 17 00:00:00 2001
From: raid5atemyhomework
Date: Tue, 5 Jan 2021 22:27:56 +0800
Subject: [PATCH] gnu: Add 'kernel-loadable-module-service-type' for services
to extend with kernel-loadable modules.
---
doc/guix.texi| 6 +
gnu/services.scm | 68
gnu/system.scm | 36 +++--
3 files changed, 83 insertions(+), 27 deletions(-)
diff --git a/doc/guix.texi b/doc/guix.texi
index 0f6e95a65a..78770151e3 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -32409,6 +32409,12 @@ configuration when you use @command{guix system
reconfigure},
@command{guix system init}, or @command{guix deploy}.
@end defvr
+@defvr {Scheme Variable} kernel-loadable-module-service-type
+Type of the service that collects lists of packages containing
+kernel-loadable modules, and adds them to the set of kernel-loadable
+modules.
+@end defvr
+
@node Shepherd Services
@subsection Shepherd Services
diff --git a/gnu/services.scm b/gnu/services.scm
index 13259dfaee..2c0bbf9725 100644
--- a/gnu/services.scm
+++ b/gnu/services.scm
@@ -33,6 +33,8 @@
#:use-module (guix diagnostics)
#:autoload (guix openpgp) (openpgp-format-fingerprint)
#:use-module (guix modules)
+ #:use-module (guix packages)
+ #:use-module (guix utils)
#:use-module (gnu packages base)
#:use-module (gnu packages bash)
#:use-module (gnu packages hurd)
@@ -75,6 +77,7 @@
service-back-edges
instantiate-missing-services
fold-services
+kernel-loadable-module-service
service-error?
missing-value-service-error?
@@ -106,6 +109,7 @@
profile-service-type
firmware-service-type
gc-root-service-type
+kernel-loadable-module-service-type
%boot-service
%activation-service
@@ -864,6 +868,70 @@ as Wifi cards.")))
will not be reclaimed by the garbage collector.")
(default-value '(
+;; Configuration for the kernel builder.
+;; Only used by the KERNEL-LOADABLE-MODULE-SERVICE-TYPE.
+(define-record-type*
kernel-builder-configuration
+ make-kernel-builder-configuration
+ kernel-builder-configuration?
+ this-kernel-builder-configuration
+
+ (kernel kernel-builder-configuration-kernel (default #f))
+ (hurd kernel-builder-configuration-hurd (default #f))
+ (modules kernel-builder-configuration-modules (default '(
+
+(define (package-for-kernel target-kernel module-package)
+ "Return a package like MODULE-PACKAGE, adapted for TARGET-KERNEL, if
+possible (that is if there's a LINUX keyword argument in the build system)."
+ (package
+(inherit module-package)
+(arguments
+ (substitute-keyword-arguments (package-arguments module-package)
+ ((#:linux kernel #f)
+target-kernel)
+
+(define (kernel-builder-configuration->system-entry config)
+ "Return the kernel and hurd entries of the 'system' directory of OS."
+ (mbegin %store-monad
+(let* ((kernel (kernel-builder-configuration-kernel config))
+ (hurd(kernel-builder-configuration-hurd config))
+ (modules (kernel-builder-configuration-modules config))
+ (kernel (if hurd
+kernel
+(profile
+ (content (packages->manifest
+ (cons kernel
+ (map (lambda (module)
+(if (package? module)
+(package-for-kernel kernel
module)
+module))
+ modules
+ (hooks (list linux-module-database))
+ (return `(("kernel" ,kernel)
+,@(if hurd `(("hurd" ,hurd)) '()))
+(define (kernel-builder-configuration-add-modules config modules)
+ "Constructs a kernel builder configuration that has its modules extended."
+ (kernel-builder-configuration
+(inherit config)
+(modules (append (kernel-builder-configuration-modules config) modules
+
+(define kernel-loadable-module-service-type
+ (service-type (name 'kernel-loadable-modules)
+(extensions
+ (list (service-extension system-service-type
+
kernel-builder-configuration->system-entry)))
+(compose concatenate)
+(extend kernel-builder-configuration-add-modules)
+(description
+ "Register packages containing kernel-loadable modules and
adds them
+to the
Re: Staging branch [aarch64 failures]
On Mon, Jan 04, 2021 at 08:37:44PM -0500, Leo Famulari wrote: > The branch is building again! > > http://ci.guix.gnu.org/eval/10974 > > Qtbase is failing on aarch64: > > https://ci.guix.gnu.org/build/166439/details > > There errors like this: > > -- > g++ -c -pipe -O2 -w -fPIC -I. > -I/gnu/store/48i8mxxb1v4x632dff3i1dbdhsazm8bw-mariadb-10.5.8-dev/include/mysql > > -I/gnu/store/48i8mxxb1v4x632dff3i1dbdhsazm8bw-mariadb-10.5.8-dev/include/mysql/mysql > > -I/tmp/guix-build-qtbase-5.15.2.drv-0/qtbase-everywhere-src-5.15.2/mkspecs/linux-g++ > -o main.o main.cpp > g++ -Wl,-O1 -o mysql main.o > -L/gnu/store/c9id1jf4r3cfys6289p92wpdhk1ryns0-mariadb-10.5.8-lib/lib/ > ld: main.o: in function `main': > main.cpp:(.text.startup+0x8): undefined reference to > `mysql_get_client_version' > collect2: error: ld returned 1 exit status > make: *** [Makefile:69: mysql] Error 1 > -- qtbase built for me using qemu-binfmt emulation. Send it through again? -- Efraim Flashner אפרים פלשנר GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted signature.asc Description: PGP signature
Re: ZFS on Guix
Hi guix-developers, In https://lists.gnu.org/archive/html/guix-devel/2021-01/msg00053.html Carlo mentioned to instead use the `service` system to modify all parts needed in the operating system to get ZFS installed, and noted as well that the only thing missing in the `service` system is the ability to push kernel modules into the operating system. Here's a sketch of an attempt to add that feature so we can conveniently add ZFS on Guix by just adding a `(service zfs-service-type...)`. First, we need a new `kernel-loadable-module-service-type` in `gnu/services.scm`: ```scheme (define kernel-loadable-module-service-type ;; A service to add the kernel modules of a package. (service-type (name 'kernel-loadable-module) (extensions '()) (compose concatenate) (extend append) (description "Register kernel modules that will be placed in a loadable place where the kernel can find them.") (default-value '( ``` Absolutely no idea if a `'()` extensions makes sense but *shrug*. Then, in `gnu/system.scm`, we create this new function: ```scheme (define (operating-system-all-kernel-loadable-modules os) ;; Gathers the kernel-loadable modules in the KERNEL-LOADABLE-FIELD ;; field of the OS, as well as those registered by services. (let ((service (fold-services (cons (service kernel-loadable-module-service-type '()) (operating-system-user-services os)) #:target-type kernel-loadable-module-service-type))) (append (service-value service) (operating-system-kernel-loadable-modules ``` Does that make sense? Am I misusing `fold-services` and `service-value` here? Onward, we modify as well the function `operating-system-directory-base`: ```scheme (define* (operating-system-directory-base-entries os) "Return the basic entries of the 'system' directory of OS for use as the value of the SYSTEM-SERVICE-TYPE service." (let* ((locale (operating-system-locale-directory os)) (kernel (operating-system-kernel os)) (hurd(operating-system-hurd os)) (modules (operating-system-all-kernel-loadable-modules os)) ;... ``` The above is the reason why we need to `cons` a synthetic `kernel-loadable-module-service-type` service, the function `operating-system-directory-base-entries` is used do create the default for `operating-system-essential-services`, and the `operating-sstem-services` is just the concatenation of the user and essential services. I think. Would this work? Thanks raid5atemyhomework
Re: [RFC] Improve Python package quality
On Tue, Jan 5, 2021 at 11:28 AM Lars-Dominik Braun wrote: > > Like in a separate pure-python file. > I don’t know how unfortunately. Any ideas? No sorry, I'm still a newbie > I moved it into a separate top-level variable now and turned it into a > single multi-line Scheme string. That makes it easier to read. That is better, but the separate file would allow to have proper syntax highlighting, allow linting/pep8'ing, etc. Cheers -- Vincent Legoll
Re: [RFC] Improve Python package quality
Hi Vincent, > I like the idea of better testing for our python packages, but would it be > possible to avoid embedding the python code as scheme strings ? > Like in a separate pure-python file. I don’t know how unfortunately. Any ideas? I moved it into a separate top-level variable now and turned it into a single multi-line Scheme string. That makes it easier to read. Cheers, Lars
Re: Questions regarding Python packaging
Hi Tanguy, > So, I've tried packaging `python-keyring` with those two… > > `pep517` keeps on trying to download dependencies, which won't work. > > `build` crashes with "ZIP does not support timestamps before 1980", > which, I guess is related to the fact that everything in the store is > timestamped to January 1st 1970. have you been looking into a python-build-system using `build`[1]? I’ve had the same issue with egg versions set to 0.0.0 and think in the long run moving to a PEP 517-style build is the way forward. Cheers, Lars [1] https://github.com/pypa/build
[PATCH] Discover extensions via GUIX_EXTENSIONS_PATH.
* guix/scripts.scm (%command-categories): Add extension category.
* guix/ui.scm (command-files): Accept an optional directory argument.
(extension-directories): New procedure.
(commands): Use it.
(show-guix-help): Hide empty categories.
(run-guix-command): Try loading an extension if there is no Guix command.
---
guix/scripts.scm | 4 +++-
guix/ui.scm | 60 +++-
2 files changed, 47 insertions(+), 17 deletions(-)
diff --git a/guix/scripts.scm b/guix/scripts.scm
index 9792aaebe9..34cba35401 100644
--- a/guix/scripts.scm
+++ b/guix/scripts.scm
@@ -3,6 +3,7 @@
;;; Copyright © 2014 Deck Pickard
;;; Copyright © 2015, 2016 Alex Kost
;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen
+;;; Copyright © 2021 Ricardo Wurmus
;;;
;;; This file is part of GNU Guix.
;;;
@@ -86,7 +87,8 @@
(development (G_ "software development commands"))
(packaging (G_ "packaging commands"))
(plumbing(G_ "plumbing commands"))
- (internal(G_ "internal commands")))
+ (internal(G_ "internal commands"))
+ (extension (G_ "extension commands")))
(define-syntax define-command
(syntax-rules (category synopsis)
diff --git a/guix/ui.scm b/guix/ui.scm
index 0a1c9bd615..2ecfb53c7b 100644
--- a/guix/ui.scm
+++ b/guix/ui.scm
@@ -2046,24 +2046,36 @@ contain a 'define-command' form."
(_
(loop)))
-(define (command-files)
+(define* (command-files #:optional directory)
"Return the list of source files that define Guix sub-commands."
- (define directory
-(and=> (search-path %load-path "guix.scm")
- (compose (cut string-append <> "/guix/scripts")
-dirname)))
+ (define directory*
+(or directory
+(and=> (search-path %load-path "guix.scm")
+ (compose (cut string-append <> "/guix/scripts")
+dirname
(define dot-scm?
(cut string-suffix? ".scm" <>))
- (if directory
- (map (cut string-append directory "/" <>)
- (scandir directory dot-scm?))
+ (if directory*
+ (map (cut string-append directory* "/" <>)
+ (scandir directory* dot-scm?))
'()))
+(define (extension-directories)
+ "Return the list of directories containing Guix extensions."
+ (filter-map (lambda (directory)
+(let ((scripts (string-append directory "/guix/scripts")))
+ (and (file-exists? scripts) scripts)))
+ (parse-path
+ (getenv "GUIX_EXTENSIONS_PATH"
+
(define (commands)
"Return the list of commands, alphabetically sorted."
- (filter-map source-file-command (command-files)))
+ (filter-map source-file-command
+ (append (command-files)
+ (append-map command-files
+ (extension-directories)
(define (show-guix-help)
(define (internal? command)
@@ -2098,9 +2110,14 @@ Run COMMAND with ARGS.\n"))
(('internal . _)
#t) ;hide internal commands
((category . synopsis)
- (format #t "~% ~a~%" (G_ synopsis))
- (display-commands (filter (category-predicate category)
- commands
+ (let ((relevant-commands (filter (category-predicate category)
+ commands)))
+ ;; Only print categories that contain commands.
+ (match relevant-commands
+ ((one . more)
+ (format #t "~% ~a~%" (G_ synopsis))
+ (display-commands relevant-commands))
+ (_ #f)
categories))
(show-bug-report-information))
@@ -2111,10 +2128,21 @@ found."
(catch 'misc-error
(lambda ()
(resolve-interface `(guix scripts ,command)))
- (lambda -
-(format (current-error-port)
-(G_ "guix: ~a: command not found~%") command)
-(show-guix-usage
+ (lambda _
+;; Check if there is a matching extension.
+(catch 'misc-error
+ (lambda ()
+(match (search-path (extension-directories)
+(format #f "~a.scm" command))
+ (file
+ (load file)
+ (resolve-interface `(guix scripts ,command)))
+ (_
+ (throw 'misc-error
+ (lambda _
+(format (current-error-port)
+(G_ "guix: ~a: command not found~%") command)
+(show-guix-usage))
(let ((command-main (module-ref module
(symbol-append 'guix- command
--
2.29.2
Re: A new paradigm for modifying operating system declarations
Hi Carlo,
> In principle, I think this should all be handled by a service.
> Services have a number of extension points where they can impact
> the operating system being declared, by extending other services.
> For example, adding a package into the global profile is done by
> extending profile-service-type (which you can find in
> gnu/services.scm). Adding a shepherd service to manage a process
> is done by extending shepherd-root-service-type (in
> gnu/services/shepherd.scm).
>
> This is how many services work. As an example, sddm-service-type
> extends services to: (a) start a process on the running system,
> (b) put files in /etc, (c) install some pam services, (d) add an
> account on the system, and (e) install packages in the global
> profile.
>
> As far as I can tell, the only thing missing for a
> zfs-service-type to do what you want is that services can't
> currently add new kernel modules (although they can load them via
> kernel-module-loader-service-type). I may have missed a mechanism
> for this, though. If we added the ability to do this, then it
> should be possible to add zfs support by adding a single (service
> zfs-service-type) to your services list.
>
> The approach of using services in this way has some advantages
> which are outlined in a blog post from 2015[1]. For me the most
> compelling advantage is that an zfs-service-type is more
> restricted in what it can do, and must be more explicit. An
> install-zfs procedure has free-reign over the entire
> operating-system definition that it gets passed, which makes it
> harder to reason about the composition of such procedures.
Thank you for this, this is certainly something I would prefer to simplify
installing ZFS onto the operating system! I wasn't aware of the
`profile-service-type`.
I already have a `zfs-loader-service-type` here:
https://issues.guix.gnu.org/45643#1
These are the things that the service has to do:
* Get a kernel module compiled and added to the set of kernel modules that can
be loaded.
* From what I understand of what you describe, this is non-existent for now.
How difficult would this be? Where do I start in getting this implemented?
* Load the kernel module.
* Execute `zpool import -a -l` to import all ZFS pools, mount those that are
marked automount, and request for passwords if encrypted.
* This has to occur after kernel module loading `(requirements
'(kernel-module-loader))`.
* This has to occur before `file-systems`. In
https://issues.guix.gnu.org/45643#2 I made `file-systems` into a target similar
to `user-processes` (i.e. it has a `file-systems-service-type` that can be
extended with a list of `requirements`, just like `user-procesess`) so that the
`zpool import` service is a dependency of `file-systems`.
* This is needed in order to get `/home` on ZFS, since `user-homes` can
race with this and populate `/home` before the ZFS module is loaded and the
pools are imported, and ZFS will refuse to mount on top of a non-empty
directory. We need to make sure that ZFS gets to mount before `file-systems`
is started, since `file-systems` will also trigger `user-homes`.
* Add the compiled ZFS on the profile.
* Add ZFS module configuration into an `/etc/modprobe.d/zfs.conf`.
The above would not get us `/` on ZFS, since we would need to have the ZFS
kernel module inside the `initrd`, have the kernel load the module before
launch, probably import pools early, and then look for root pool and so on.
The problem is getting the ZFS configuration inside the `initrd` as well, in
addition to the userspace tools.
So, here's a sketch:
```scheme
(operating-system
(kernel linux-libre-5.4)
;...
(services
(cons* (service zfs-service-type
(zfs-configuration
; we need to compile for a specific kernel, the alternative here
; would be (operating-system this-operating-system) but the below
; is a good bit shorter, despite the DRY violation...
(kernel linux-libre-5.4)
(options
'(("zfs_arc_max" 50)
;...
%desktop-services))
#;...)
```
Thanks
raid5atemyhomework
Re: [RFC] Improve Python package quality
Hello, I like the idea of better testing for our python packages, but would it be possible to avoid embedding the python code as scheme strings ? Like in a separate pure-python file. WDYT ? -- Vincent Legoll
Re: [RFC] Improve Python package quality
Hi Hartmut, > this is a good idea. (Since you where mentioning setuptools, I first was > afraid your solution would be tightened to setuptools, but it is not. > Well done!) afaik pkg_resources is technically a part of setuptools, although it is distributed with Python. > This comment should go behind the line of code, as it only related to > that single line. > […] > I suggest putting the comments into the python source. This would allow > to indent them according the the python code, which would make it easier > to understand. This would also allow to use a single multi-line > guile-string, which allows to easiyl copy the script out and in from the > guile-source for testing it. > […] > Please follow PEP8 (no space before opening parentheses) - also at other > places. > […] > Add `end=""`, thus the "result" can be printed on the same line. > Print result terse, on same line, without repeating the name: You’re right, all fixed. I’ll send a non-hacky patch (with test-cases!) to guix-patches@ for review once we’ve figured out a path to merge it. I guess it would be best to fix packages directly on master and merge this new phase to core-updates? Shall I apply for commit access or can you (or Tobias?) review and “proxy” required changes? Right now I have fixes for about 10 packges, but there will be more. > Would is be better to use mkdtemp here to ge a fresh, empty directory? I tried that, but mkdtemp! is not available and I’m not confident enough to add that module to the closure. Any ideas? Cheers, Lars
Re: guix build -d with a target causes many builds
Chris Marusich writes: > Hi, > > I've noticed that Guix builds many things when I ask it to instantiate a > derivation in the following way: > > [0] [env] marusich@garuda-lan:~/guix/repos/guix-worktrees/wip-ppc64 > $ guix build -d --target=powerpc64-linux-gnu -e '(@@ (gnu packages > make-bootstrap) %gcc-static)' > The following derivations will be built: >/gnu/store/i5wn3xl6p0zw1vglscgk0bs9dwc6hdh6-gcc-static-5.5.0.drv > > /gnu/store/3h2sk37iim53fh7g9r3sd1q0xzhqwa51-gcc-cross-powerpc64-linux-gnu-7.5.0.drv > > /gnu/store/84k0j5jm316cwf7h66vrw1vmvkd4kbck-glibc-cross-powerpc64-linux-gnu-2.31.drv > > /gnu/store/d36n7qy9xbgwpaw3nw8k9dj51hzmdnr4-gcc-cross-sans-libc-powerpc64-linux-gnu-7.5.0.drv > > /gnu/store/mqar9bnapfcfkna3rvy28awhlpd3q65q-binutils-cross-powerpc64-linux-gnu-2.34.drv > > /gnu/store/pzp93dw3rr6sp2ybi3dzs6kd7gvigfsk-ld-wrapper-powerpc64-linux-gnu-0.drv > > /gnu/store/n7dhpsq41q4kdbqgniljbwrlawvmmlp6-linux-libre-headers-cross-powerpc64-linux-gnu-5.4.20.drv > > /gnu/store/9p5anrji5wgkf66k09jhbsr3fqwwi7cn-gcc-cross-powerpc64-linux-gnu-7.5.0.drv > > /gnu/store/r4ac80znwlrnh4jmj2sbczc4mn66mqdg-glibc-cross-powerpc64-linux-gnu-2.31.drv >/gnu/store/ap8ri9ddka13vyrsl72pzqslagi4v7vj-gmp-6.2.0.drv > > /gnu/store/arxf2alzwf9rmz5hz8h11j4j12drxm3i-glibc-cross-powerpc64-linux-gnu-2.31.drv > > /gnu/store/d127w5flv12s4bfmpf4nwrvg3sibvfya-linux-libre-headers-cross-powerpc64-linux-gnu-5.4.20.drv >/gnu/store/j3d5kr7qlr6g3lq0dwc8z8jh6w814z9v-isl-0.18.drv >/gnu/store/j90wwahzd5ldw7ai11zf5lnp3kbbrmkh-mpfr-4.0.2.drv >/gnu/store/mz9fdir4avdda5cw1snyf8vhpq70c9na-libelf-0.8.13.drv >/gnu/store/q9x04y75mq2nfp2a6gwa0pvrgv60aah9-mpc-1.1.0.drv >/gnu/store/xk4yv7xj15qnl3zv2m8nnzrw0bdgjsx3-zlib-1.2.11.drv > 171.3 MB will be downloaded: >/gnu/store/ir3092v7657h6g4g2vlsw3zrli3rndb3-zlib-1.2.11.tar.gz >/gnu/store/amc0nizxsdcj212nk9a3ivr946hzhl6c-mpc-1.1.0 >/gnu/store/j4npmpn7dxmfknyfnhj4q4jmdwmk3klg-mpc-1.1.0.tar.gz >/gnu/store/0z3z3lhig0xyy817nv70p2hp1n1wqawa-libelf-0.8.13.tar.gz >/gnu/store/bkyiyc4hrjcd4ljx6jqf7z05hm4qxcwd-mpfr-4.0.2.tar.xz >/gnu/store/2jj3il6p5xrc4gkncj9303an81x2csc9-perl-5.30.2 >/gnu/store/n1yvkd7jk50qg1vv9cca6ywynkqvaqgq-ncurses-6.2 >/gnu/store/j709qpwy790bcra6w8kvyz1v5zcsw8df-texinfo-6.7 >/gnu/store/jk5k0sgqpj0sj4ymgq7m8g8617i0xji2-m4-1.4.18 >/gnu/store/57i37x74wz7ar703smykildzvhpdds1g-gmp-6.2.0 >/gnu/store/f2r1w8y7l3lpwh4i47nq2s1vqlqxq0jb-glibc-2.31 >/gnu/store/rgi1k6kx4v9m8449w00i6jfxvpgaz73g-glibc-2.31-static >/gnu/store/df1gdl0vwwbzv04snfha0g88rj02pni9-gcc-5.5.0 >/gnu/store/waz3iz17vlbpfc2fm9yiym6bgbsajghf-mpfr-4.0.2 >/gnu/store/hnsi8iaimgss3v81h7h1r8ck55c0968h-popt-1.18 >/gnu/store/vpy0bcjw0yzaj7j7qx8rfc88c7r357k3-rsync-3.1.3 >/gnu/store/0zcl1i3rbjc356138hx86b7yaz29g6fj-linux-libre-5.4.20-gnu.tar.xz >/gnu/store/l788x07ska5vffayz0gayv4hsx5flxal-module-import-compiled >/gnu/store/lqz1pygx3x5dd6ad2l3n8ixm1vh6czj4-python-minimal-3.8.2 >/gnu/store/ba6s3g925nggb57b1gpj2jkhqsq24s4q-libstdc++-7.5.0 >/gnu/store/xaclbfx6rvnbsq5qmry0251r7y82rgnv-libstdc++-headers-7.5.0 >/gnu/store/j8b9i4czpzb298zwa15wpyr42471qfbm-module-import-compiled > > The Guix documentation ((guix) Additional Build Options) says the "-d" > option should just give me the derivation paths, not the output paths: > > ‘--derivations’ > ‘-d’ > Return the derivation paths, not the output paths, of the given > packages. > > So, I'm confused about why all these builds need to happen. Didn't I > only ask Guix to instantiate the derivation - not realize it? In other > words, I expected Guix to calculate the transitive closure of the > requested derivation's inputs (mainly other derivations, I think?) and > write them to the store, without actually executing any significant > builds. > > I suppose that Guix is building these things in order to do just that, > but I don't quite understand why this happens. Can someone explain it? Grafts. → guix build --no-grafts -d --target=powerpc64-linux-gnu -e '(@@ (gnu packages make-bootstrap) %gcc-static)' /gnu/store/i5wn3xl6p0zw1vglscgk0bs9dwc6hdh6-gcc-static-5.5.0.drv They're on by default, so when you use -d without --no-grafts, I believe you're actually getting the derivation that does the grafting, rather than the derivation for the package build. signature.asc Description: PGP signature
