Re: Losing signing keys for custom Guix channel
Ludovic Courtès wrote: > elaexuo...@wilsonb.com skribis: > > > Well, the catch 22 is that I've lost the original key and so can only sign > > .guix-authorizations with the new one. > > Ah sorry, I misread the thing I quoted. :-) > > So, you have your new key. You add it to ‘.guix-authorizations’ in a > commit signed with that new key. And then, you make this commit the new > introduction of your channel. > > Does that make sense? > > Ludo’. Makes perfect sense! It's also exactly what I tried and what ends up failing authorization on guix pull.
Re: Losing signing keys for custom Guix channel
Jake wrote: > If commit i adds a new signing key to the channel’s authorisations file and > commit i+1 is signed with that signing key, then commit i+1 can be used in > channel intro. > > You can’t add a signing key to the authorisations in a commit and sign that > same commit with the new key. Is that issue here? I don't think that's completely accurate. My original channel introduction commit is precisely the one creating a .guix-authorizations file with my old key info. I can certainly add an extra signing key to the authorizations; I just can't sign that commit with the old key, since the old key has been lost.
Re: Losing signing keys for custom Guix channel
> > from reading about guix authentication I think the new signing key > > must be first added to the .guix-authoriations file and that commit > > must signed with the current signing keys before the new signing > > key can be used. > > Yes, it’s likely the problem; the rest of the description you gave > elaexuotee looks fine to me. > > (No need to rewrite the history; changing the introduction is enough.) > > Ludo’. Well, the catch 22 is that I've lost the original key and so can only sign .guix-authorizations with the new one. > (No need to rewrite the history; changing the introduction is enough.) Without the old key, I'm gathering that a history rewrite is the only way right now. Seems like a fresh channel introduction should be enough, but our current authorization check appears to look at earlier commits even in that case, IIUC. Maybe forcing history rewrites on key loss is the desired behavior? I'm not sure. From a client perspective, the only difference is whether or not you have to specify --allow-downgrades on the next pull. In either case a channel intro update is necessary.
Losing signing keys for custom Guix channel
Hey devs, So I lost the PGP key that I was using to sign commits on a private Guix channel of mine. Is there a way to introduce a hard break in my channel authentication? Despite updating authorization settings, pulls complain that my latest commit isn't signed by an authorized key. Here are the changes I've made: - New public key added to keyring branch - Appended new key fingerprint to .guix-authorizations (at commit X) - Update introduction in .config/guix/channels.scm - Point to commit X - Update openpgp-fingerprint As a sanity check, I've confirmed that the fingerprint on commit X, the fingerprint in .guix-authorizations, and the openpgp-fingerprint in my channels.scm are all the same. What am I missing?
Re: Bug in strip phase of gnu-build-system?
Ludovic Courtès wrote: > Hi, > > elaexuo...@wilsonb.com skribis: > > > When `strip-binaries?` is `#f` and a "debug" output is defined, said output > > remains empty. Instead, I expect "debug" to get populated with separated > > debug > > files. > > Hmm I see. The logic was different: #:strip-binaries? #f disables > stripping, but it doesn’t populate the “debug” output either. > > If you want to populate the “debug” output, you need to declare a > “debug” output in the package and keep #:strip-binaries? #t. > > (I can see arguments for and against both interpretations, but that’s > how it is currently.) Oh. Blah. It was something obivous. Hehe. Thank you for cleaing up my confusion. I apparently misread `lower` from (guix build-system gnu), thinking it stripped the "debug" output if #:strip-binaries? is true. > [...] > > > If the above is correct, then would it make sense for --with-debug-info to > > also > > inject a "debug" output as needed? It looks like this would be pretty easy > > under guix/transformations.scm:transforma-package-with-debug-info. > > ‘--with-debug-info’ is meant to be used when you want to keep debug info > (and there’s no “debug” output). It seemed to me there’s no point in > having that debug info separate in that case, especially since you > wouldn’t be able to refer to it. > > HTH! Very! Much obliged.
Re: Bug in strip phase of gnu-build-system?
Mark H Weaver wrote: > The conditional was originally introduced by Ludovic, here: > > > https://git.sv.gnu.org/cgit/guix.git/commit/?id=877217b85ac318a0a702578f183dd638c9704112 > > My role was merely to reformulate the conditional as a 'when' instead of > an 'or', while making an unrelated change in a later commit, here: > > > https://git.sv.gnu.org/cgit/guix.git/commit/?id=9a87649c863e1ff8b073b356875eb05eecedbcf7 > > I've never studied this corner of Guix, and I don't have time to do so, > but perhaps Ludovic can respond further. Whoops. Thank you for the correction.
Bug in strip phase of gnu-build-system?
Hello, Sanity check, please. When `strip-binaries?` is `#f` and a "debug" output is defined, said output remains empty. Instead, I expect "debug" to get populated with separated debug files. The logic for creating debug files exists in (guix build gnu-build-system) under the definition of `strip`. In particular `strip-dir` extracts the debug info into "debug" if and only fi that output exists. Note, that (guix build-system gnu) already removes "debug" from the outputs when strip-binaries? is true. However, for some reason the actual `strip-dir1 invocation is wrapped in a (if strip-binaries? ...) check, meaning that the entire phase becomes a no-op. Shouldn't that if check be removed? I have CC'ed Mark Weaver, since it looks like he is the author of this particular if block. Cheers, B. Wilson P.S. If the above is correct, then would it make sense for --with-debug-info to also inject a "debug" output as needed? It looks like this would be pretty easy under guix/transformations.scm:transforma-package-with-debug-info.
Adding custom X Keyboard layout definitions
Hey Guix, Is there a mechanism for adding custom X Keyboard definitions that will get picked up by setxkbmap and friends? I have a package that offers a layout, but simply throwing it under share/X11/xkb/symbols/ is mostly useless since setxkbmap is built with the xkeyboard-config path hard-coded in. Cheers, B. Wilson
packages: [PoC] Expand range of objects 'add-input-label' can label
Hey Guix! This is me grubbing around with internals I know nothing about. Here is a quick and dirty proof-of-concept patch that extends 'add-input-label' to use the 'name' property of plain-file, program-file, etc. In addition, it also labels auxiliary files, using the same path string one provides to search-auxiliary-file. This part is the dirtiest, especially since we cannot make use of %auxiliary-files-path from (gnu packages) since that would introduce a circular dependency. The motivation for this arose while recently munging a package to use new-style inputs. The package's inputs include an auxiliary file and custom program-file gexp; however, with new-style inputs there is no obvious way to get get access to these input paths within the build phases: - Both get the default "_" label; and - search-input-file only finds files inside store path *directories*. There potentially a workaround using file-union, but it's clearly friction. Naive package-writing me wanted one of two things: 1) Ability to mix old, explicit labels with the new labelless inputs, or 2) Labels that come from the gexp name. The first option is much more general, but somehow I stumbled across the add-input-label procedure, so went with 2. What are your thoughts? Clear unknows to me: - Is add-input-label in a fast path? This patch significantly increases the cases supplied to 'match', which feels like it has the potential to hit performance in a bad way. - Using the 'name' property is kind of obvious, but add-input-label looks deliberatly small. Was using 'name' intentionally avoided for some reason? - Are there other implicit concerns/constraints within guix/packages.scm that I am oblivious to? - etc? Anyway, by no means do I expect this to get merged, mostly just hoping to learn something. Cheers, B. Wilson From 3b8e7fa8fbd58e7e164e3730c708419f612b8549 Mon Sep 17 00:00:00 2001 From: "B. Wilson" Date: Sun, 16 Jan 2022 23:54:51 +0900 Subject: [PATCH 1/2] packages: Expand range of objects 'add-input-label' can label To: guix-patc...@gnu.org * guix/packages.scm (%auxiliary-files-subpath-dir): New variable. (add-input-label): Support labels from the name property of objects that have one. Also, name auxiliary files from their subpath. --- guix/packages.scm | 24 +++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/guix/packages.scm b/guix/packages.scm index 9d5b23eb8a..4feea8ad5f 100644 --- a/guix/packages.scm +++ b/guix/packages.scm @@ -7,6 +7,7 @@ ;;; Copyright © 2019 Marius Bakke ;;; Copyright © 2020, 2021 Maxim Cournoyer ;;; Copyright © 2021 Chris Marusich +;;; Copyright © 2022 B. Wilson ;;; ;;; This file is part of GNU Guix. ;;; @@ -569,6 +570,10 @@ (define-record-type* (default (current-definition-location)) (innate))) +;; Note: This morally imports from gnu/packages.scm, but since they import us, +;; we define here instead. +(define %auxiliary-files-subdir-path "/gnu/packages/aux-files") + (define (add-input-label input) "Add an input label to INPUT." (match input @@ -576,7 +581,24 @@ (define (add-input-label input) (list (package-name package) package)) (((? package? package) output);XXX: ugly? (list (package-name package) package output)) -((? gexp-input?) ;XXX: misplaced because 'native?' field is ignored? +((? local-file? local-file) + (list (local-file-name local-file) local-file)) +((? plain-file? plain-file) + (list (plain-file-name plain-file) plain-file)) +((? computed-file? computed-file) + (list (computed-file-name computed-file) computed-file)) +((? program-file? program-file) + (list (program-file-name program-file) program-file)) +((? scheme-file? scheme-file) + (list (scheme-file-name scheme-file) scheme-file)) +((? string? path) + (let* ((regex (string-append %auxiliary-files-subdir-path "/(.*)")) +(match (string-match regex input))) + `(,(if match + (match:substring match 1) + "_") + ,input))) +((? gexp-input?) ;XXX: misplaced because 'native?' field is ignored? (let ((obj(gexp-input-thing input)) (output (gexp-input-output input))) `(,(if (package? obj) -- 2.34.0
qemu-i486 no longer exists, but qemu-binfmt thinks it does
Hey Guix, Just for fun, I tried putting the following in my services: (service qemu-bitfmt-service-type (qemu-bitfmt-configuration (platforms %qemu-platforms))) However, after reconfiguration, shepherd fails to start qemu-binfmt: $ sudo herd start qemu-binfmt herd: exception caught while executing 'start' on service 'qemu-binfmt' fport_write: No such file or directory That message is not particularly helpful, but $ grep binfmt_misc /var/log/messages Apr 25 19:35:14 localhost vmunix: [ 2052.220124] binfmt_misc: register: failed to install interpreter file /gnu/store/3l74cx3dgr61kj0q8padb9ayn0dai45b-qemu-5.2.0-static/bin/qemu-i486 Indeed, it looks like the qemu-i486 binary isn't in the qemu output at all. Any idea what happened to it?
Re: Installing a wrapper guile script in /bin
Thank you for the thoughtful input. Maxime Devos wrote: > * forego representing `wrapper-script` as a gexp (using #~), instead represent > `wrapper-script` as something quasiquoted. Then write this expression to > a file (with 'write', and include an appropriate shebang line) > * The procedure ‘program-file’ turns a gexp into a representation of a store > item. After spending most of the day figuring out how to manually create and build derivations. I hit a fundamental problem including that code in a package def which made me realize that manually embedding scripts is a flawed approach. The issue is that a script itself has dependencies, e.g. guile, bash, whatever. Those dependencies also need to show up in the transitive closure of whatever package embeds this script. > * Define a wrapper package. See wrap-python3 in packages/python.scm > for an example. (This option probably has to be combined with one of the > first two options.) Your suggestion here eventually set me on the right track. Say we want the final executable to end up at /bin/X. The basic idea is this: 1) Creat original package with a name like X-lib, and leave /bin/X empty, 2) Write a wrapper script as a gexp argument to program-file. In this gexp we reference the X-lib package, and finally 3) Create an trivial-build-system package X that includes the program-file wrapper as an input. The build step will copy this input script to /bin/X! This way dependencies are correctly tracked, and there's no need for manual derivation building hackery. Anyway, thanks for the pointers! They motivated me to keep pushing until something worked.
Re: Installing a wrapper guile script in /bin
Ludovic Courtès wrote: > I wrote about this topic in the past: > > https://hpc.guix.info/blog/2018/01/pre-built-binaries-vs-performance/ Very nice overview! Thanks for sharing (and writing!) that article. Definitely feeling the urge to jump down this rabbit hole and patch upstream now. Hehe. > I you’re the upstream author, I recommend using one of the techniques > given above to provide so-called “fat binaries” that contain several > implementations of the performance-sensitive functions; the loader > will pick the right implementation when the program starts. > > If you’re downstream… it depends on the specifics. The loader is also > able to pick a .so from the right lib/ sub-directory depending on the > micro-architecture. You can try: I'm downstream, unfortunately. However, the final executable actually provides a flag to explicitly specify a path to the lib, so that's not really a hurdle in this case. Given the small size of the build products, I feel like it would be nice to fake a fat binary at the filesystem level. Mind if we just entertain this idea for a second? Say I have a script that reads /proc/cpuinfo and runs my executable with the correct flags to load the library with the best CPU features possible. How can I embed such a script in the package definition (as a gexp?) and install it under /bin/?
Installing a wrapper guile script in /bin
Hello Guix, Writing package definition, I have need of a non-trivial wrapper script that decides how to execute the installed binary. How do I accomplish this? With my vague understanding, I am envisioning writing a gexp directly in the install phase and would like to somehow reify this into a guile script and install that file under /bin. Is this correct, at the high level? More specifically, the package I have builds separate libraries for CPUs with AVX, AVX2, and no AVX support. Since build-type isn't sufficiently specific to distinguish such CPU features, I have, so far, opted to just build all three libs and stuff them under /lib/. My idea is to have the linker script check CPU features at runtime (by parsing /proc/cpuinfo or something) and executing the binary with the parameters to load the correct binary. Perhaps there is a better overall approach? Appreciate your thoughts!
Re: how to "guix pack" a profile?
At this point, I am probably firmly in kOOk territory. :p For now I will stay silent on the issue until I have some concrete code to share. Thank you for sincerely engaging with me. Cheers, zimoun wrote: > On Fri, 19 Jun 2020 at 14:34, wrote: > > > > 1. Composable profiles, > > > > > > This is already possible. > > > > Would you kindly point me to the documentation? I am failing to find it. > > guix install hello -p foo > guix install emacs -p bar > eval $(guix package -p foo -p bar --search-paths=exact) > > So it composes. However maybe not what you had in mind. ;-) > > > By "composable profiles" what I have in mind is the sum of two existing > > profiles and extending them as necessary > > > > $ guix environment --with-profile saved-profiles/profile-A \ > >--with-profile saved-profiles/profile-B \ > > > > Yes, because "guix environment" cannot support "profile". As said, > IMHO you raised 2 interesting use cases: > > 1. create an environment based on a profile > 2. pack an existing profile > > Obviously "a profile" means the composition of profiles if any. > > > > I agree. If I'm to make any points, I should just submit some concrete > > patches. > > Patches are always welcome. :-) > > > Cheers, > simon > > ps: > Well, maybe we are falling in the "take it to private email" thread pattern. > ;-) > https://joeyh.name/blog/entry/thread_patterns/
Re: how to "guix pack" a profile?
Ludovic Courtès wrote: > Hi, > > zimoun skribis: > > > Ludo, WDYT about "guix pack -p profile" to generate a (relocatable) > > tarball or Docker image? I mean if it is not already possible. :-) > > That could work, but honestly I’d rather encourage -m. I’m not > convinced there’s a need for what you suggest here. > > Ludo’. The original proposal for this was to enable packing a development environment, as opposed to an --ad-hoc collection of packages. Am I simply missing how to do this? signature.asc Description: PGP signature
Re: how to "guix pack" a profile?
> > 1. Composable profiles, > > This is already possible. Would you kindly point me to the documentation? I am failing to find it. By "composable profiles" what I have in mind is the sum of two existing profiles and extending them as necessary $ guix environment --with-profile saved-profiles/profile-A \ --with-profile saved-profiles/profile-B \ > This snippet of quotation shows that it is not "easy" for the general > case. And the transformation using "--load-path" has not been evoked. > For example: > > --8<---cut here---start->8--- > (define-module (foo) > #:use-module (guix packages) > #:use-module (guix profiles) ;; imagine it is used > #:use-module (gnu packages base)) > > (define (crazy-transformation pkg) > "Could do really complicated things" > (package > (inherit pkg) > (name (string-append "ex-" (package-name pkg) > > (define bonjour > (package > (inherit hello) > (name "bonjour"))) > > (define-public crazy-bonjour > (crazy-transformation bonjour)) > --8<---cut here---end--->8--- > > then "guix install -L /path/to/foo ex-bonjour". Thank you for the concrete example. I guess it wasn't clear in my previous emails, but I proposed *simply copying /path/to/foo* into the generated profile and letting "manifest.scm" refer to these copies. Is this obviously broken. > Thank you for the ideas. Especially the two ones: > > 1. create an environment from a profile > 2. pack a profile > > Well, I do not know if they will happen but it should be cool to have. I agree. If I'm to make any points, I should just submit some concrete patches. Again, thank you for your time and patience. Cheers, signature.asc Description: PGP signature
Re: how to "guix pack" a profile?
> I am not sure we are talking about the same channels.scm file. I guess, > you have in mind ~/.config/guix/channels.scm but it is not the one of > interest for reproducing. You need the real one, i.e., "guix describe > -f channels". > > AFAIU, manifest.scm+$(guix describe -f channel) is enough to have > "deterministic profile (re)generation". If not, it means there is a > "bug" in Inferiors or I miss something. Bingo! This is exactly what I was missing. Thank you for sifting this out. Given the stated "goal of ... [allowing] users to move towards declarative 'manifest.scm' style..." I completely mis-interpreted the sense in which you were using "channels.scm". signature.asc Description: PGP signature
Re: how to "guix pack" a profile?
zimoun, Thank you for you time. Apparently, I am indeed missing some important conceptual details. I will come back later when I know more. Cheers and happy Guixing, zimoun wrote: > Dear, > > On Fri, 19 Jun 2020 at 11:52, elaexuo...@wilsonb.com wrote: > > > What Pierre (and others?) initially proposed---what I am > > re-proposing---is that we put a blob of Guile into the profiles that > > is capable of uniquely and completely generating the profile where it > > resides. > > Others? For example me. With Pierre, we spent some time at Guix Days > to propose a new format. > > > 1. Composable profiles, > > This is already possible. > > > 2. Sharing "light" profiles buy sending only the "recipe.scm" instead > > of an entire container. > > I am not sure to get what is a "light" profile but from my understanding > what you want here already exist and it is manifest.scm+channel.scm. > > > 3. guix archive --manifest > > 4. guix profile --manifest-from-recipe /recipe.scm > > > > The last one there is intended to be the tool for "migrate from > > imperative to declarative" user profile management, starting from a > > given profile. > > See below. > > >> What you describe here is exactly what Pierre and other have > >> proposed. And the work-to-do is to prototype the format of what you > >> called "recipe.scm", which means equivalently in the previous emails > >> change the format of /manifest. > > > > I agree. However, in previous emails I have tried to make a rebuttal > > to Ludo's argument than the best we can do is *approximate* a > > manifest.scm. > > > > See https://lists.gnu.org/archive/html/guix-devel/2020-02/msg00098.html: > > I have the impression that the discussion is going nowhere. > > Make what you called "recipe.scm" or modify " become an acceptable "--manifest" is doable. Note that the idea is not > new either. As I said, all the technical material is here. But Ludo > and others (me) are not convinced that it will pay off because the > general case is hard. Well, to be concrete, there are 2 possible next > actions: > > a) prototype the new format and implement it > b) make an approximation exposed with "--export" > > Frankly, I doubt that a) happens because no one will do the tough work. > Therefore, b) is pragmatic. > > > (Ludo) > As far as faithfulness is concerned, we should also keep in mind > that we don’t always have all the information needed to reconstruct > what’s in a profile. For example, we lack information about the > package transformation options that were used (if any), > >>> > >>> (Pierre) > >>> Like `--with-input'? I didn't think of this. Can't we extend the > >>> format then to include all transformation options, with future-proof > >>> provisions? > >> > >> (Ludo) > >> We couldstore package transformations as manifest entry properties. > >> > >> However, that’ll be an approximation: the exact implementation of > >> ‘--with-input’, for instance, can vary over time. > > This snippet of quotation shows that it is not "easy" for the general > case. And the transformation using "--load-path" has not been evoked. > For example: > > --8<---cut here---start->8--- > (define-module (foo) > #:use-module (guix packages) > #:use-module (guix profiles) ;; imagine it is used > #:use-module (gnu packages base)) > > (define (crazy-transformation pkg) > "Could do really complicated things" > (package > (inherit pkg) > (name (string-append "ex-" (package-name pkg) > > (define bonjour > (package > (inherit hello) > (name "bonjour"))) > > (define-public crazy-bonjour > (crazy-transformation bonjour)) > --8<---cut here---end--->8--- > > then "guix install -L /path/to/foo ex-bonjour". > > > > However, with `time-machine' and a given `guix environment' or `guix > > profile' invocation, we are able to deterministically resolve a > > /gnu/store/-profile, no? Better yet, this is in a future-proof > > way, no? If that is so, then why not canonify this profile recipe in > > guile code instead of what is needed now: guile + bash? What am I > > missing here? > > You are missing the difficulty to make it concretely, I guess. :-) > > > > Just to be clear, I would be more than thrilled with a --from-profile > > option to `guix pack'. However, I am trying to make a case that "first > > class profiles" is both feasible and may pay back more in maintenance > > cost than it consumes. > > Thank you for the ideas. Especially the two ones: > > 1. create an environment from a profile > 2. pack a profile > > Well, I do not know if they will happen but it should be cool to have. > > > I am going to move on since we already raised all the points, I guess. :-) > > > All the best, > simon signature.asc Description: PGP signature
Re: how to "guix pack" a profile?
Ludo, Pierre, would you mind commenting? Starting to doubt my sanity here. > Well, from my understanding, any profile point to /gnu/store/-profile. Exactly. This is something we both know. I feel like we are not communicating well with each other. You emphasize that declarative package management with "manifest.scm" is the way to go. I agree but wanted to point out that $ guix package -m manifest.scm produces a different /gnu/store/-profile, depending on when it's run (or more precisely, depending on which channel commits guix resolves for the invocation). You know that, of course, but the point I try to make is that "manifest.scm" and "channels.scm" are *not enough* to uniquely determine a specific /gnu/store/-profile. We need to separate out the concepts of "declarative profile management" and "deterministic profile regeneration." What Pierre (and others?) initially proposed---what I am re-proposing---is that we put a blob of Guile into the profiles that is capable of uniquely and completely generating the profile where it resides. > For which use case? The examples I gave in my previous email... I am not sure what's getting miscommunicated here. > Well, let say there is a profile which could be a mess and I only see > two usages currently uncovered: > > 1. create an environment based on this profile; either to temporarily >extend it (which is AFAIK already more or less covered), either to >isolate it, > 2. pack it for sharing with your colleague. Sure, about the only new feature now would be to guix pack an existing profile. However, it's not too hard to think of potential uses or features users may want in the future: 1. Composable profiles, 2. Sharing "light" profiles buy sending only the "recipe.scm" instead of an entire container. 3. guix archive --manifest 4. guix profile --manifest-from-recipe /recipe.scm The last one there is intended to be the tool for "migrate from imperative to declarative" user profile management, starting from a given profile. > What you describe here is exactly what Pierre and other have proposed. > And the work-to-do is to prototype the format of what you called > "recipe.scm", which means equivalently in the previous emails change the > format of /manifest. I agree. However, in previous emails I have tried to make a rebuttal to Ludo's argument than the best we can do is *approximate* a manifest.scm. See https://lists.gnu.org/archive/html/guix-devel/2020-02/msg00098.html: >>> (Ludo) >>> As far as faithfulness is concerned, we should also keep in mind that we >>> don’t always have all the information needed to reconstruct what’s in a >>> profile. For example, we lack information about the package >>> transformation options that were used (if any), >> >> (Pierre) >> Like `--with-input'? >> I didn't think of this. Can't we extend the format then to include all >> transformation options, with future-proof provisions? > > (Ludo) > We could store package transformations as manifest entry properties. > > However, that’ll be an approximation: the exact implementation of > ‘--with-input’, for instance, can vary over time. However, with `time-machine' and a given `guix environment' or `guix profile' invocation, we are able to deterministically resolve a /gnu/store/-profile, no? Better yet, this is in a future-proof way, no? If that is so, then why not canonify this profile recipe in guile code instead of what is needed now: guile + bash? What am I missing here? Just to be clear, I would be more than thrilled with a --from-profile option to `guix pack'. However, I am trying to make a case that "first class profiles" is both feasible and may pay back more in maintenance cost than it consumes. signature.asc Description: PGP signature
Re: how to "guix pack" a profile?
Zimoun, thank you for taking the time to closely read my long emails. > Well, if I re-read correctly the emails and proposal, they are 2 points: > > 1. Easily share a profile via "guix pack" > 2. A mean via recreating "manifest.scm" files Hrm. I agree that point 1 is the immediate goal I have and what prompted me to create this issue. However, my proposal is more general and orthogonal to point 2. > From my understanding, the way to go is the declarative via manifest.scm Perhaps I am using terminology incorrectly. I completely agree that declarative management of a `user profile' is the way to go. But before I go any further, let me distinguish some terms: By `profile', I mean any store output named like /gnu/store/-profile. These outputs are created by `guix package' invocations as well as `guix environment' ones. By `user profile', I mean the (non-static) contents pointed to by ~/.guix-profile. These contents get updated on `guix package' invocations (by updating the ~/.guix-profile symlink). (Do the above two concepts have established nomenclature?) Using this terminology, we both agree than managing your `user profile' via "manifest.scm" (and "channels.scm") is the way to go. Those two files provide a specification which guix *resolves* into a concrete `profile'. The imminently practical issue that you, Ludo, and Pierre seem to be working on is a way of easing the transition from imperative to declarative `user profile' management. In contrast, I am proposing making `profiles' first class. By this I mean a file under /gnu/store/-profile that contains the code capable of completely and deterministically rebuilding that exact profile, and that profile *only*. Just for the sake of argument, let's call this file "recipe.scm", and let's choose a profile, /gnu/store/123abc-profile. Then the following should be true: $ guix package --manifest /gnu/store/123abc-profile/recipe.scm $ readlink -f ~/guix-profile /gnu/store/123abc-profile I.e. that `guix package' invocation would have the effect of setting the current user generation to that exact profile. Of course, if we had such a "recipe.scm" then the following would Just Work: # Containerize a profile. $ guix pack --manifest /gnu/store/123abc-profile/recipe.scm # Share profiles with your Guix friends. $ guix environment --manifest ~/downloads/friends-awesome-profile.scm And it plays well with potential future features: # Copy whole profiles between Guix stores $ guix archive --manifest /gnu/store/123abc-profile/recipe.scm Hopefully my generalized idea is clearer now. So, what do you say? Are "first class profiles" a coherent and interesting concept? What about practicality? signature.asc Description: PGP signature
Re: Using --manfistest with /manifest files
Thank you for the direct reply, zimoun. > [The problem] is a pragmatic one. As any good Zen says: "Now is better than > never. Although never is often better than *right* now." Okay. If that is the case, then I very much empathize with the problem. > Going from imperative/sequential "install, pull, remove" way to the > declarative manifest.scm way, in the general case, needs to change the > format of /manifest (or add another file). Which means > transition plan, etc.. Otherwise, on the technical level, all the > material is there. That said. This makes me wonder that we may be thinking of different things altogether. The discussion seems to have congealed around smoothing the transition to declarative profile management for users. However, the proposal I have in mind is *first class profiles*. I am thinking of a file that can be fed to the --manifest (or some potential future equivalent) option of various guix commands. This hypothetical file would let users operate directly on profiles as needed. My current specific use case for this is packing the *development environments* produced by `guix environment'. > I do not see why it is straightforward for some cases. guix environment --ad-hoc can be indirectly packed via guix pack unless I am missing something. Otherwise, users are at an impasse. > Because your use case -- pack an existing profile for sharing -- is not > really related to transform /manifest to a valid manifest.scm, > if I understand correctly. And I agree with you that it should be > possible to pack an existing profile (created by any mean). > > Does "pack --profile-name" fit your needs? I am not quite sure this works. From the manual: > ‘--localstatedir’ > ‘--profile-name=NAME’ > Include the “local state directory”, ‘/var/guix’, in the resulting > pack, and notably the ‘/var/guix/profiles/per-user/root/NAME’ > profile—by default NAME is ‘guix-profile’, which corresponds to > ‘~root/.guix-profile’. > > ‘/var/guix’ contains the store database (*note The Store::) as well > as garbage-collector roots (*note Invoking guix gc::). Providing > it in the pack means that the store is “complete” and manageable by > Guix; not providing it pack means that the store is “dead”: items > cannot be added to it or removed from it after extraction of the > pack. I do not see how to use this with the transitory /gnu/store/-profile produced by a `guix environment' invocation. Also, my intention is simply to provide the profile's environment, not include the local state directory. Put more simply, I want to be able to produce a tarball/container capable of reproducing `guix environment --container '. I think this would be very useful. Am I just failing to grok something fundamental? Thoughts? More generally, I think first class profiles could be both a powerful feature and an important future-proofing against extra maintenance burden. Profiles are a central concept to guix usage. They form the atomic unit with which users interact. Wanting to tarball a profile is just one use case, but future guix commands (guix merge, anyone?) or future --manifest options (guix archive, anyone?) seem likely to directly benefit from an existing infrastructure that supports store profiles being created, recreated and munged. zimoun wrote: > Dear, > > On Tue, 16 Jun 2020 at 20:33, elaexuo...@wilsonb.com wrote: > >> 1. We can only approximate that actual profile content; storing > >>an approximate ‘manifest.scm’ along with the profile would IMO be > >>deceptive. > > > > Is this a technical barrier or a pragmatic one? > > [...] > > > If the problem is of pragmatics, then at the very least I would be > > interested > > in hearing a delineation of the challenges. I think this could be helpful > > for > > the discussion though. > > It is a pragmatic one. As any good Zen says: "Now is better than never. > Although never is often better than *right* now." > > Going from imperative/sequential "install, pull, remove" way to the > declarative manifest.scm way, in the general case, needs to change the > format of /manifest (or add another file). Which means > transition plan, etc.. Otherwise, on the technical level, all the > material is there. > > So it is some work and it is not clear that it will pay off. > > > >> Yeah, I think our goal is just to provide a tool to migrate from the > >> “imperative” way to the declarative way. Once people have gotten > >> started with manifests, they no longer need that migration tool. > > > > Would you mind commenting on the use case that I started this thread with? > > Specifically, I was trying to `guix pack' a `guix environment'. The > > equivalent > > is straightforward for purely --ad-hoc environments but not otherwise. > > I do not see why it is straightforward for some cases. > > > > Personally, I have already encountered several instances where this would > > have > > been useful. I
Re: Using --manfistest with /manifest files
> 1. We can only approximate that actual profile content; storing >an approximate ‘manifest.scm’ along with the profile would IMO be >deceptive. Is this a technical barrier or a pragmatic one? If it is the former, then I don't quite grok why. I explain my reasoning in great detail in a previous reply, but the gist is this: generating profiles is deterministic, no? That is, given the inputs of channel revision, command line invocation, and any referred-to external files, then `guix time-machine' is enough to recreate a profile, no? If the problem is of pragmatics, then at the very least I would be interested in hearing a delineation of the challenges. I think this could be helpful for the discussion though. > Yeah, I think our goal is just to provide a tool to migrate from the > “imperative” way to the declarative way. Once people have gotten > started with manifests, they no longer need that migration tool. Would you mind commenting on the use case that I started this thread with? Specifically, I was trying to `guix pack' a `guix environment'. The equivalent is straightforward for purely --ad-hoc environments but not otherwise. Personally, I have already encountered several instances where this would have been useful. I also think it would be just plain cool to have the ability to pack up, containerize, and share arbitrary profiles with non-guix users. Thoughts? signature.asc Description: PGP signature
Re: Using --manfistest with /manifest files
This is a good point. The naming of `/manifest' does invite confusion when first encountering it. That said, I am pretty sure there is a place for `/manifest.scm'. Given the `--manifest' option to several commands, it makes profiles first class. In particular, it would let users easily `guix pack' or `guix archive' arbitrary profiles as needed. At the moment, it's not at all obvious how to `guix pack' the equivalent of `guix environment --container '. FWIW, I never expected `/manifest' to encode "this is what the user ordered," so much as "this is the recipe for (deterministically) reproducing this exact profile." For the former we have `packages->manifest', `specifications->manifest' etc. The latter is what I understand this discussion to be about. George Clemmer wrote: > > Ludovic Courtès writes: > > > elaexuo...@wilsonb.com skribis: > > >> First, am I missing something? Is there a better/preferred way to make use > >> of > >> the `manifest' files in profiles? > > > You’re not missing anything: it’s a longstanding source of confusion > > that these ‘manifest’ files are not like the ‘manifest.scm’ files. > > These ‘manifest’ files are meant for internal consumption. > > This hurt my head for a while a few years ago until I realized that > 'manifest.scm' is the guix "order" and ‘.guix-profile/manifest’ is the > guix "packing list". > > But actually a guix' 'manifest' packing list goes well beyond what we > normally find in a packing list by containing detailed info about how > the specific products were made, down to the specific design for the > specific version shipped. > > Thought of this way it is easy to understand why a receiver of a > 'manifest' can only estimate the set of 'manifest.scm' that might > produce it. A simple-minded example: did the manifest.scm specify the > version of the package shipped or is this an artifact of a) when > 'manifest.scm' was processed or b) of the requirements of the other > packages that were received? > > In any event, once I saw it this way it no longer troubled me that guix > doesn't have a pushbutton way to "reverse" 'manifest' into > 'manifest.scm'. > > ISTM we set ourselves up for confused users and a lot of explaining by > labeling two very different things with same name :-0 > > Yes, only 'manifest.scm' is in the doc, but '.guix-profile/manifest' > smacks a user in the face pretty quickly which leads to these messy > questions. > > IMO we could dramatically simplify the situation, and simplify our > lives, by simply renaming the .guix-profile/manifest file ;-) > > George signature.asc Description: PGP signature
Re: Using --manfistest with /manifest files
zimoun, In response to your previous email, I gave a long-form reply to the general discussion. However, I just want to note that the personal issue I am encountering isn't with user profiles; instead it is with those generated by `guix environment'. In particular, I was trying to use `guix pack' to share the *development* environment of a package with a non-guix user. As I shared in my original email, the (imperfect) solution I used was by wrapping `/manifest' with `read-manifest' and feeding that to `guix pack --manifest': (call-with-input-file "/manifest" (@@ (guix packages) read-manifest) zimoun wrote: > Hi, > > On Sun, 14 Jun 2020 at 17:24, Ludovic Courtès wrote: > > > I think there were several issues we discussed: > > > > 1. We can only approximate that actual profile content; storing > > an approximate ‘manifest.scm’ along with the profile would IMO be > > deceptive. > > > > 2. It’s easy to maintain compatibility over a data format, but it’s > > much harder to maintain compatibility for code. > > > > I think we discussed these issues the best we could in the megathread, > > so I’m personally in favor of moving forward in a pragmatic way. > > By pragmatic way, you mean: > > - let the format of /manifest as it is, > - write '--export-manifest' as an approximation > > right? > > Well, I personally changed my workflow and now I always use manifest > files. And the situation that I described in the manifest about the > "Working Scientific" doing install, pull, install, pull, remove, > install, etc. is rooted in bad practises, so it should be avoided. > > Therefore, I agree that '--export-manifest' is the right approach, as an > helping tool; too bad for some corner cases. :-) > > > All the best, > simon signature.asc Description: PGP signature
Re: Using --manfistest with /manifest files
> It is more "complicated" than that. The detailed explanations are in > the mega thread. :-) In short and from my understanding, going from > "/manifest" to "-m manifest.scm" cannot be done in the general > case because two concepts -- imperative vs declarative -- are not well > aligned. Pragmatically, it means that the result could be more than > often unpractical with too much inferiors. Well, only an > "approximation" could be exported. I went ahead and read through the threads that Pierre shared in a different reply. For posterity and to collect my own thoughts, let me see if I can distill the discussion so far: The goal is to enable a profile to generate itself by reifying it into some collection of code and data. Given such a tool, two places of use are proposed: 1) Implicitly, upon profile generation, outputting new files under /etc, or 2) Explicitly via a command like `guix package --export'. It turns out that `guix system reconfigure' does some (most?) of this by generating `/gnu/store/-provenance' and `/gnu/store/-channels.scm' along with the profile itself. Naively, a profile is just a sum of outputs; however, there are subtleties: a) Provenance and inferiors For reproducibility, channel and revision information must be stored. This is alreading in profile/manifest, however, so we already have the necessary infrastracture in this regard. b) The family of `--with-*' options to `guix package' means that provenance data itself is insufficient. Worse, `--expression' means that the collection of outputs might be specified with *arbitrary code*. That said, the sum of provenance and command-line invocation should be sufficient, no? If so, an extreme proof-of-concept reification could simply be a bash script, something akin to the following: #!/bin/env bash guix time-machine -C /channels.scm -- \ guix package -p [ ...] c) Profile reifications need to be forward-compatible, meaning that future revisions of guix should produce the same profile as current/older ones given a reification. This, of course, shows why the bash script idea is untenable, but with a more reasonable Guile implementation, storing version information in the way of `/manifest' neatly solves this issue. Am I missing any major points? Almost surely there are minor ones I am overlooking. Here are some of my questions: Theorem: Given provenance data (e.g. a `channels.scm' file) and a command line invocation, profiles generation is deterministic. Note that by "command line invocation" I am including any external files its options reference as well. Is this true at least mostly? Ludo mentions something about the "possibility of multiple provenances" which I fail to grok. What is going on here? Does this introduce a source of non-determinism for users building profiles? I.e. given the right bash script, can a user reliably reproduce a given profile? If the answer to the final question above is no, then that seems like a much larger problem. However, if the answer is yes, then I would naively expect profile reification to be mostly a matter of collecting together all the sources of input that define a profile. Does forward-compatibility make this less straightforward than I am thinking? What else am I missing? > Sorry I am too lazy to search, but I think I remember that at the time > Pierre sent -- probably in the mega thread :-) -- a small script to > extract relevant information from /manifest; the preliminary > for '--export-manifest'. :-) Perhaps you are thinking of Pierre's script here? https://lists.gnu.org/archive/html/guix-devel/2020-02/msg00154.html Cheers, signature.asc Description: PGP signature
Using --manfistest with /manifest files
In an attempt to tar up the *build* environment for a package to share with a colleague, I encountered this: [env]$ guix pack -m $GUIX_ENVIRONMENT/manifest (manifest ...): Wrong number of arguments From playing around a bit, my guess is that the `/manifest' files are just human-readable serializations of objects and don't deserialize back: $ guix environment -m ~/.guix-profile/manifest (manifest ...): Wrong number of arguments Asking around on #guix, user leoprikler suggested a workaround by putting this in a file: (call-with-input-file "/path/to/manifest" (@@ (guix profiles) read-manifest)) and feeding that file to --manifest. This works as intended. First, am I missing something? Is there a better/preferred way to make use of the `manifest' files in profiles? If not, does it seem reasonable to make the `manifest' files automatically deserialize into the objects they describe? I think this could be a fun, simple project for a Guix newbie like me, but before I start figuring out how to do this, I just want to sanity check the idea. Cheers! signature.asc Description: PGP signature
[no subject]
Attempting to define and build my own package for the first time, I am havi= ng trouble simply setting up the basic environment. In particular, make is failing wit= h a plethora of "@ref reference to nonexistent node" errors on doc/*.de.texi fi= les. It looks like issues #35913 and #35786 may be related; however, at the mome= nt issues.guix.gnu.org is returning 500 when I attempt to open them. Attached is the output of make, and below is how I am getting to that point= : $ guix environment --pure guix --ad-hoc coreutils findutils which $ ./bootstrap $ ./configure --localstatedir=3D/var $ make Am I just flubbing up something simple? For clarity, this is happening on a= freshly pulled master: $ git rev-parse HEAD f6145358c7d33e73c0708bbe400e6e8e65512ef3 make all-recursive make[1]: Entering directory '/home/x/devel/guix' Making all in po/guix make[2]: Entering directory '/home/x/devel/guix/po/guix' make[2]: Leaving directory '/home/x/devel/guix/po/guix' Making all in po/packages make[2]: Entering directory '/home/x/devel/guix/po/packages' make[2]: Leaving directory '/home/x/devel/guix/po/packages' make[2]: Entering directory '/home/x/devel/guix' MAKEINFO doc/guix.de.info ./doc/guix.de.texi:419: @ref reference to nonexistent node `GNU Distribution' ./doc/guix.de.texi:431: @ref reference to nonexistent node `Invoking guix package' ./doc/guix.de.texi:432: @ref reference to nonexistent node `Development' ./doc/guix.de.texi:433: @ref reference to nonexistent node `Utilities' ./doc/guix.de.texi:434: @ref reference to nonexistent node `Programming Interface' ./doc/guix.de.texi:437: @ref reference to nonexistent node `Setting Up the Daemon' ./doc/guix.de.texi:439: @ref reference to nonexistent node `Substitutes' ./doc/guix.de.texi:446: @ref reference to nonexistent node `Defining Packages' ./doc/guix.de.texi:448: @ref reference to nonexistent node `Package Modules' ./doc/guix.de.texi:450: @ref reference to nonexistent node `Package Transformation Options' ./doc/guix.de.texi:456: @ref reference to nonexistent node `Acknowledgments' ./doc/guix.de.texi:473: @ref reference to nonexistent node `The Store' ./doc/guix.de.texi:482: @ref reference to nonexistent node `Features' ./doc/guix.de.texi:493: @ref reference to nonexistent node `System Installation' ./doc/guix.de.texi:504: @ref reference to nonexistent node `Invoking guix package' ./doc/guix.de.texi:544: @ref reference to nonexistent node `System Configuration' ./doc/guix.de.texi:555: @ref reference to nonexistent node `Porting' ./doc/guix.de.texi:558: @ref reference to nonexistent node `Contributing' ./doc/guix.de.texi:576: @ref reference to nonexistent node `System Installation' ./doc/guix.de.texi:591: @ref reference to nonexistent node `Invoking guix pull' ./doc/guix.de.texi:683: @ref reference to nonexistent node `The Store' ./doc/guix.de.texi:700: @ref reference to nonexistent node `Invoking guix pull' ./doc/guix.de.texi:718: @ref reference to nonexistent node `Build Environment Setup' ./doc/guix.de.texi:785: @ref reference to nonexistent node `Substitutes' ./doc/guix.de.texi:795: @ref reference to nonexistent node `Application Setup' ./doc/guix.de.texi:823: @ref reference to nonexistent node `Invoking guix pack' ./doc/guix.de.texi:867: @pxref reference to nonexistent node `Daemon Offload Setup' ./doc/guix.de.texi:868: @pxref reference to nonexistent node `Invoking guix copy' ./doc/guix.de.texi:901: @ref reference to nonexistent node `The Store' ./doc/guix.de.texi:945: @ref reference to nonexistent node `Requirements' ./doc/guix.de.texi:969: @ref reference to nonexistent node `Substitutes' ./doc/guix.de.texi:989: @ref reference to nonexistent node `Substitutes' ./doc/guix.de.texi:1028: @ref reference to nonexistent node `Introduction' ./doc/guix.de.texi:1049: @ref reference to nonexistent node `Invoking guix-daemon' ./doc/guix.de.texi:1054: @ref reference to nonexistent node `Invoking guix system' ./doc/guix.de.texi:1116: @ref reference to nonexistent node `Derivations' ./doc/guix.de.texi:1117: @ref reference to nonexistent node `Substitutes' ./doc/guix.de.texi:1289: @ref reference to nonexistent node `Invoking guix archive' ./doc/guix.de.texi:1460: @ref reference to nonexistent node `Setting Up the Daemon' ./doc/guix.de.texi:1472: @ref reference to nonexistent node `Programming Interface' ./doc/guix.de.texi:1478: @ref reference to nonexistent node `Features' ./doc/guix.de.texi:1489: @ref reference to nonexistent node `Invoking guix build' ./doc/guix.de.texi:1497: @ref reference to nonexistent node `Invoking guix processes' ./doc/guix.de.texi:1504: @ref reference to nonexistent node `Setting Up the Daemon' ./doc/guix.de.texi:1510: @ref reference to nonexistent node `Substitutes' ./doc/guix.de.texi:1514: @ref reference to nonexistent node `The Store' ./doc/guix.de.texi:1524: @ref reference to nonexistent node `Substitutes' ./doc/guix.de.texi:1529: @ref reference to nonexistent node
Build fails on 'doc/guix.de.texi': @ref reference to nonexistent node...
Attempting to define and build my own package for the first time, I am havi= ng trouble simply setting up the basic environment. In particular, make is failing wit= h a plethora of "@ref reference to nonexistent node" errors on doc/*.de.texi fi= les. It looks like issues #35913 and #35786 may be related; however, at the mome= nt issues.guix.gnu.org is returning 500 when I attempt to open them. Attached is the output of make, and below is how I am getting to that point= : $ guix environment --pure guix --ad-hoc coreutils findutils which $ ./bootstrap $ ./configure --localstatedir=3D/var $ make Am I just flubbing up something simple? For clarity, this is happening on a= freshly pulled master: $ git rev-parse HEAD f6145358c7d33e73c0708bbe400e6e8e65512ef3 make.log Description: plain/text signature.asc Description: PGP signature
Re: Guix System on a GCE VM -- Success
elaexuotee--- via "Development of GNU Guix and the GNU System distribution."
wrote:
> > This all looks pretty nifty! Just to be clear, under which license did
> > you make this script available?
>
> Ouch. That was a major oversight. Thanks for bringing this up!
> I went ahead and put it under the BSD-3-CLAUSE. Attached is the same source
> with a copyright notice and license header.
It looks like I messed up the attachment. Trying again.
#!/usr/bin/env sh
set -o errexit -o noclobber -o nounset
# Copyright (c) 2020 elaexuo...@wilsonb.com. All rights reserved.
#
# Licensed under <https://spdx.org/licenses/BSD-3-Clause.html>
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#
# 1. Redistributions of source code must retain the above copyright notice,
#this list of conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright notice,
#this list of conditions and the following disclaimer in the documentation
#and/or other materials provided with the distribution.
#
# 3. Neither the name of the copyright holder nor the names of its contributors
#may be used to endorse or promote products derived from this software
#without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
## GCE Custom Image
#
# This script intends to be "executable documention" for the process of setting
# up a Google Compute Engine (GCE) virtual machine from a custom image.
usage() {
cat <<-USAGE
Usage: $(basename "${0}") [options] [ [..]]
This script aids in setting up Google Compute Engine with an instance
running a custom image.
Execute commands in order provided at the command line. The procedure
for a fresh setup follows the order listed below in Status Commands.
Setup Commands:
mkconf Configure files on raw image partition
mkraw Convert image to raw format
mksize Grow image to integer multiple of GBs
mktar Archive raw image
mkgsb Create GS bucket
togsa Upload archive to GS bucket
togci Create GCE image from GS archive
togvm Create GCE instance from GCE image
Status Commands:
ckconf Check contents of qcow2 image
ckraw Check integrity of raw image
cksize Check size of raw image
cktar Check contents of tar archive
ckgsb Check existence of GS bucket
ckgsa Check existence of tar archive in GS bucket
ckgci Check existence of GCE image
ckgvm Check status of GCE instance
ttysro View GCE instance's serial output
ttysrw Connect to instance serial port
Removal Commands:
rmgvm Remove GCE instance
rmgci Remove GCE image
rmgsa Remove archive from GS bucket
rmgsb Remove GS bucket
Metainfo Commands:
helpShow this help message
varsPrint value of configuration variables
Options:
-f Family name for GCE image
-v Version string of GCE image
-nName of GCE image
-NName of GCE instance
-s Subnet name of GCE instance
-mMachine type of GCE instance
-B URI of GS bucket
-A URI of GS tar file
-p Partition of image on which / resides
-S Serial port of GCE instance
-bCommon path base for files and names
-gPath on image of grub.cfg
-iLocal path of source image file
-rLocal path of raw image file
-aLocal path of tar file
-h Show this help message
USAGE
}
fail() {
msg=${1}; errno=${2-1}
>&2 p
Guix System on a GCE VM -- Success
Attached is a minimal system configuration I have used to successfully
create a virtual machine instance on Google's Compute Engine (GCE).
The stock QEMU image [1] fails to boot under GCE, and since there is some past
interest [0], I figure it may be helpful to share the salient issues and how I
deal with them in the config.scm.
The boot problem fix boils down to this:
(initrd-modules (cons "virtio_scsi" %base-initrd-modules))
Referencing Google's documentation on operating system requirements [2], we see
that their VMs run with Virtio-SCSI controllers; however, it just so happens
that the corresponding kernel module, virtio_scsi, is not included in the
initrd by default. Would it make sense to change this default?
That should be enough to get a working GCE instance running, provided you
follow Google's requirements for importing custom images [3]. It's not
challenging, but it takes time to familiarize oneself with GCE and the
associated toolchain.
To simplify the process I created a script capable of setting up a GCE instance
from a local qcow2 image, which I am also attaching. The script is intentially
not a simple turnkey solution, since you should be somewhat familiar with the
GCE infrastructure before using it. However, feel free to ask questions if
anything is unclear.
Returning back to config.scm, the only other non-obvious pieces deal with the
bootloader setup. On GCE at boot time, our only choice of interface is a serial
console, so the following set that up in case it becomes necessary:
(bootloader (bootloader-configuration
...
(serial-speed 115200)
(terminal-outputs '(serial))
(terminal-inputs '(serial
(kernel-arguments '("console=ttyS0,115200"))
We set the baud rate to 115200 instead of the default 9600, for two reasons.
One it is essential that grub.cfg get populated with the following:
serial [options]
terminal_output serial
terminal_input serial
and for some reason, bootloader-configuration seems to elide these lines
without serial-speed set to something other than 9600. Secondly, baud 115200
ends up providing a better shell experience anyway.
Finally, the attached config.scm sets a bootloader timeout of 0:
(bootloader (bootloader-configuration
...
(timeout 0)))
Since the vm is completely headless during normal bootup, anything else just
slows down boot times.
Finally, GCE allows community supported images [4]. If there is enough
interest, it might make sense to invest the few cents per month to maintain a
Guix System one. Thoughts?
Hopefully, the above is clear. I am happy to answer any questions.
[0]:https://lists.gnu.org/archive/html/guix-devel/2017-01/msg01815.html
[1]:https://ftp.gnu.org/gnu/guix/guix-system-vm-image-1.0.1.x86_64-linux.xz
[2]:https://cloud.google.com/compute/docs/images/building-custom-os
[3]:https://cloud.google.com/compute/docs/import/import-existing-image
[4]:https://cloud.google.com/compute/docs/images#community_supported_images
(use-modules (gnu))
(use-service-modules networking ssh)
(operating-system
(host-name "guixsd")
(timezone "Europe/Berlin")
(locale "en_US.utf8")
(bootloader (bootloader-configuration
(bootloader grub-bootloader)
(target "/dev/sda")
(theme (grub-theme))
(timeout 0)
(serial-speed 115200)
(terminal-outputs '(serial))
(terminal-inputs '(serial
(kernel-arguments '("console=ttyS0,115200"))
(initrd-modules (cons "virtio_scsi" %base-initrd-modules))
(file-systems (cons (file-system
(device (uuid "b6a32092-b004-43ba-bab7-e4c8d5480026"))
(mount-point "/")
(type "ext4"))
%base-file-systems))
(users (cons (user-account
(name "guest")
(group "users")
(supplementary-groups '("wheel")))
%base-user-accounts))
(services (append (list (service dhcp-client-service-type)
(service openssh-service-type))
%base-services)))
#!/usr/bin/env sh
set -o errexit -o noclobber -o nounset
## GCE Custom Image
#
# This script intends to be "executable documention" for the process of setting
# up a Google Compute Engine (GCE) virtual machine from a custom image.
usage() {
cat <<-USAGE
Usage: $(basename "${0}") [options] [ [..]]
This script aids in setting up Google Compute Engine with an instance
running a custom image.
Execute commands in order provided at the command line. The procedure
for a fresh setup follows the order listed below in Status Commands.
Setup Commands:
mkconf Configure files on raw image partition
mkraw Convert image to raw format
