Re: Expensive builds when computing channel instance derivations
Christopher Baines writes: >> It would be tempting to turn off grafts in this case. But then, ‘guix >> pull’ would always give us an ungrafted Guix, which is undesirable. For >> the purposes of the Data Service though, you could consider turning off >> grafts globally. > > I get that not applying grafts to guix could be undesirable, but I'm > still don't see why there aren't other options? > > What would be the implications of treating it like package grafting. As > in, you'd always build guix without grafts, then you'd apply the > grafting process to the ungrafted output? That would be more consistent > with the way it works for packages, and it would probably go all or some > of the way to resolve the issues for the Guix Data Service. I've actually surprised myself by making some quick progress on this. I think I see now why the current situation is not really anything more than the normal package grafting code applying to the computing of the guix derivation. I've sent some patches which seem to superficially work to [1]. 1: https://issues.guix.gnu.org/61363 Thanks, Chris signature.asc Description: PGP signature
Re: Expensive builds when computing channel instance derivations
Ludovic Courtès writes: > Christopher Baines skribis: > >> I think I figured out that it's related to grafts. I've attached a test >> script which computes the channel instance derivation for mips64el-linux >> which I'm not set up to build things for (if you can build for >> mips64el-linux, replace this with a system you can't build for). You'll >> need to use the attached patch (also present on [2]) when running this >> script. >> >> 2: >> https://git.cbaines.net/guix/log/?h=channel-instances-manifest-graft-control >> >> When I run this script locally, it first succeeds in computing the >> channel instance derivation when grafts are disabled, but then fails >> when grafts are enabled: >> >> while setting up the build environment: a `mips64el-linux' is required >> to build >> `/gnu/store/g40shyhsd00r5dq3mm76c2b1krnr1njh-guile-bootstrap-2.0.drv', >> but I am a `x86_64-linux' >> >> Even though I think this shows that grafts are involved, I'm not sure >> what this means? I'm guessing that the effects of grafts aren't as clear >> cut as for packages here, since the grafts are happening here as part of >> computing a derivation I'm guessing that the derivation is actually >> built with the grafted outputs, which differs from how grafts affect >> packages. Given this, I guess computing the derivation without grafts >> means that the replacements that would be grafted in are just not used >> at all. >> >> To recap on my aim here, I really just want to be able to compute >> channel instance derivations without performing any expensive builds, or >> if that's not possible, it would be good to understand why? For some reason I didn't follow up on this thread at the time, I can't remember now it's been so long. This issue has remained at the back of my mind though since within the Guix Data Service, I think it's becoming more impactful. > In general, computing the derivation of a package may incur > builds/downloads due to the way grafts work: depending on the *build > result* of a package, ‘package->derivation’ might return a grafting > derivation or it might return the original derivation. > > The “Computing Guix derivation” bit of ‘guix pull’ is no exception: it > computes the derivation of objects that depend on packages that may need > to be grafted. Thus, it ends up triggering builds as part of this > process. While that's a way of thinking about it to highlight the similarities, it's the differences I'm interested in here. For package derivations, I think about grafts as a layer on top. First you build the packages normally, then you apply any grafts including building any replacements as required. I don't know if that's an entirely correct way of thinking about it, but it seems to work as a mental model. For bordeaux.guix.gnu.org to provide substitues, it's sufficient to build the packages (including replacements) without any grafting, because that's a separate concern. I don't think the same can be said for the channel instance derivations though, probably because of the difference you highlight above regarding grafts actually affecting how the derivations involved are constructed, rather than acting as a layer on top. > It would be tempting to turn off grafts in this case. But then, ‘guix > pull’ would always give us an ungrafted Guix, which is undesirable. For > the purposes of the Data Service though, you could consider turning off > grafts globally. I get that not applying grafts to guix could be undesirable, but I'm still don't see why there aren't other options? What would be the implications of treating it like package grafting. As in, you'd always build guix without grafts, then you'd apply the grafting process to the ungrafted output? That would be more consistent with the way it works for packages, and it would probably go all or some of the way to resolve the issues for the Guix Data Service. There's maybe other options as well, although as I raised in my initial email, it's hard for me to think about this since I haven't worked out why the grafting here doesn't work like packages grafting for packages in the first place. Thanks, Chris signature.asc Description: PGP signature
Re: Expensive builds when computing channel instance derivations
Hi, Christopher Baines skribis: > I think I figured out that it's related to grafts. I've attached a test > script which computes the channel instance derivation for mips64el-linux > which I'm not set up to build things for (if you can build for > mips64el-linux, replace this with a system you can't build for). You'll > need to use the attached patch (also present on [2]) when running this > script. > > 2: > https://git.cbaines.net/guix/log/?h=channel-instances-manifest-graft-control > > When I run this script locally, it first succeeds in computing the > channel instance derivation when grafts are disabled, but then fails > when grafts are enabled: > > while setting up the build environment: a `mips64el-linux' is required > to build > `/gnu/store/g40shyhsd00r5dq3mm76c2b1krnr1njh-guile-bootstrap-2.0.drv', > but I am a `x86_64-linux' > > Even though I think this shows that grafts are involved, I'm not sure > what this means? I'm guessing that the effects of grafts aren't as clear > cut as for packages here, since the grafts are happening here as part of > computing a derivation I'm guessing that the derivation is actually > built with the grafted outputs, which differs from how grafts affect > packages. Given this, I guess computing the derivation without grafts > means that the replacements that would be grafted in are just not used > at all. > > To recap on my aim here, I really just want to be able to compute > channel instance derivations without performing any expensive builds, or > if that's not possible, it would be good to understand why? In general, computing the derivation of a package may incur builds/downloads due to the way grafts work: depending on the *build result* of a package, ‘package->derivation’ might return a grafting derivation or it might return the original derivation. The “Computing Guix derivation” bit of ‘guix pull’ is no exception: it computes the derivation of objects that depend on packages that may need to be grafted. Thus, it ends up triggering builds as part of this process. It would be tempting to turn off grafts in this case. But then, ‘guix pull’ would always give us an ungrafted Guix, which is undesirable. For the purposes of the Data Service though, you could consider turning off grafts globally. HTH! Ludo’.
Expensive builds when computing channel instance derivations
Hey, Back in early 2021, I was trying to address the issues the Guix Data Service has when trying to compute channel instance derivations for various systems [1]. 1: https://issues.guix.gnu.org/47989 Some changes did some out of that, and I believe the helped, but even with those changes, being able to build things for the system you want to compute the channel instance derivation for seemed to remain necessary. This poses an operational issue for things like data.guix.gnu.org, since it can only compute channel instance derivations for systems it can build for, which in practice means that it's limited by the systems which QEMU emulation is enabled for. Even for those systems it can build for, because builds can happen when attempting to compute these derivations, it means that data.guix.gnu.org spends a lot of time waiting for builds to complete, just so it can store these derivations. I have a suspicion that this issue is a big contributor to the data.guix.gnu.org slowness when processing new revisions, so I tried to dig in to it more recently as I didn't know why these builds were happening. I think I figured out that it's related to grafts. I've attached a test script which computes the channel instance derivation for mips64el-linux which I'm not set up to build things for (if you can build for mips64el-linux, replace this with a system you can't build for). You'll need to use the attached patch (also present on [2]) when running this script. 2: https://git.cbaines.net/guix/log/?h=channel-instances-manifest-graft-control When I run this script locally, it first succeeds in computing the channel instance derivation when grafts are disabled, but then fails when grafts are enabled: while setting up the build environment: a `mips64el-linux' is required to build `/gnu/store/g40shyhsd00r5dq3mm76c2b1krnr1njh-guile-bootstrap-2.0.drv', but I am a `x86_64-linux' Even though I think this shows that grafts are involved, I'm not sure what this means? I'm guessing that the effects of grafts aren't as clear cut as for packages here, since the grafts are happening here as part of computing a derivation I'm guessing that the derivation is actually built with the grafted outputs, which differs from how grafts affect packages. Given this, I guess computing the derivation without grafts means that the replacements that would be grafted in are just not used at all. To recap on my aim here, I really just want to be able to compute channel instance derivations without performing any expensive builds, or if that's not possible, it would be good to understand why? Thanks, Chris From 38a12f0f22841b76050a0cf5163cdc65b7f92193 Mon Sep 17 00:00:00 2001 From: Christopher Baines Date: Fri, 18 Feb 2022 23:06:57 + Subject: [PATCH] channels: Allow disabling grafts when computing derivations. --- build-aux/build-self.scm | 23 +++ guix/channels.scm| 19 +++ 2 files changed, 26 insertions(+), 16 deletions(-) diff --git a/build-aux/build-self.scm b/build-aux/build-self.scm index 02822a2ee8..0e7fc2907d 100644 --- a/build-aux/build-self.scm +++ b/build-aux/build-self.scm @@ -241,7 +241,8 @@ (define guile-gcrypt (define* (build-program source version #:optional (guile-version (effective-version)) -#:key (pull-version 0) (channel-metadata #f)) +#:key (pull-version 0) (channel-metadata #f) +(graft? #t)) "Return a program that computes the derivation to build Guix from SOURCE." (define select? ;; Select every module but (guix config) and non-Guix modules. @@ -316,6 +317,8 @@ (define fake-git (read-disable 'positions)) (use-modules (guix store) + (guix grafts) + (guix monads) (guix self) (guix derivations) (srfi srfi-1)) @@ -348,12 +351,14 @@ (define fake-git (%make-void-port "w")) (current-build-output-port sock)) (run-with-store store - (guix-derivation source version - #$guile-version - #:channel-metadata - '#$channel-metadata - #:pull-version - #$pull-version) + (mbegin %store-monad + (set-grafting #$graft?) + (guix-derivation source version +#$guile-version +
