Hello Guix! I’ve just pushed a pretty cool hack to ‘wip-build-accumulator’. :-)
Grafts are a function of the references of a package. Suppose Bash has a ‘replacement’ field. If a package has a build-time dependency on Bash, it’s a candidate for being grafted the Bash replacement. However, if that package has no reference on Bash (i.e., “guix gc --references $(guix build the-package --no-grafts) | grep bash” is empty), then we don’t graft it. This reduces the number of grafts needed, and thus saves CPU time (fewer grafts derivations built) and space (fewer grafted variants in the store). Because grafting depends on build results, they constitute “dynamic dependencies”, using the terminology of “Build Systems à la Carte”. Currently, grafts are handled in an ad-hoc fashion: we use substitute info to try and determine ahead of time the references of a build output if it’s unavailable locally. When substitute info is available, that allows us to display the build plan upfront. When substitute info is unavailable, we cannot display the build plan upfront anyway, hence the need for ‘with-build-handler’¹. Additionally, when substitute info is available, we can end up looking up substitutes one by one, leading to repeated “updating the list of substitutes” messages and related slowness¹ (this is particularly visible when /var/guix/substitute/cache is empty.) To address that, I think we need a better way to handle “dynamic dependencies”. The ‘wip-build-accumulator’ branch does that by taking advantage of ‘with-build-handler’. On that branch, grafts no longer use substitute info. Instead, they just build missing store items and get their references. To avoid building things one by one, we install a build handler that “accumulates” the list of ‘build-things’ requests; eventually, we build all these things at once and resume the continuations of the ‘build-things’ calls. The goal here is to improve efficiency and to allow the UI to shows these stages in a meaningful way. Here’s a sample session (slightly edited for clarity) --8<---------------cut here---------------start------------->8--- ludo@ribbon ~/src/guix$ rm /tmp/foo* ludo@ribbon ~/src/guix$ guix gc -D $(./pre-inst-env guix build vim-full zile-on-guile --no-grafts) $(./pre-inst-env guix build vim-full zile-on-guile) ludo@ribbon ~/src/guix$ ./pre-inst-env guix install zile-on-guile vim-full -p /tmp/foo -n The following packages would be installed: zile-on-guile 2.4.14-0.fd09781 vim-full 8.2.0411 9.4 MB would be downloaded: /gnu/store/vf7w4yiax38ra7x8aqqvbnc38c0ldgpm-zile-on-guile-2.4.14-0.fd09781 /gnu/store/dnj9wljcck9vdwgp7dwxk00qnnk1g3c5-vim-full-8.2.0411 ludo@ribbon ~/src/guix$ ./pre-inst-env guix install zile-on-guile vim-full -p /tmp/foo The following packages will be installed: zile-on-guile 2.4.14-0.fd09781 vim-full 8.2.0411 9.4 MB will be downloaded: /gnu/store/vf7w4yiax38ra7x8aqqvbnc38c0ldgpm-zile-on-guile-2.4.14-0.fd09781 /gnu/store/dnj9wljcck9vdwgp7dwxk00qnnk1g3c5-vim-full-8.2.0411 downloading from https://ci.guix.gnu.org/nar/lzip/dnj9wljcck9vdwgp7dwxk00qnnk1g3c5-vim-full-8.2.0411... vim-full-8.2.0411 8.9MiB 7.6MiB/s 00:01 [##################] 100.0% downloading from https://ci.guix.gnu.org/nar/lzip/vf7w4yiax38ra7x8aqqvbnc38c0ldgpm-zile-on-guile-2.4.14-0.fd09781... zile-on-guile-2.4.14-0.fd09781 140KiB 1.8MiB/s 00:00 [##################] 100.0% The following derivation will be built: /gnu/store/d9xms78917w67xq71pqsx5x9s6dmq6d7-profile.drv The following graft will be made: /gnu/store/4n6dmg6iwjg0adpcvqygr9wgsnclswss-vim-full-8.2.0411.drv applying 8 grafts for /gnu/store/4n6dmg6iwjg0adpcvqygr9wgsnclswss-vim-full-8.2.0411.drv... building /gnu/store/d9xms78917w67xq71pqsx5x9s6dmq6d7-profile.drv... --8<---------------cut here---------------end--------------->8--- Here, with ‘-n’, we see the main tasks (downloads). Without ‘-n’, we see again these two downloads first, and then a second stage showing the graft and profile derivations. What if we remove ‘git-minimal’ from the store, which is itself required to build the source of some nodes on the graph? --8<---------------cut here---------------start------------->8--- ludo@ribbon ~/src/guix$ rm /tmp/foo* ludo@ribbon ~/src/guix$ guix gc -D $(./pre-inst-env guix build git-minimal vim-full zile-on-guile --no-grafts) $(./pre-inst-env guix build git-minimal vim-full zile-on-guile) ludo@ribbon ~/src/guix$ ./pre-inst-env guix install zile-on-guile vim-full -p /tmp/foo -n The following packages would be installed: zile-on-guile 2.4.14-0.fd09781 vim-full 8.2.0411 4.9 MB would be downloaded: /gnu/store/wwjhip4wvhjyfj6fs7936bbsgd4yax6g-git-minimal-2.26.0 ludo@ribbon ~/src/guix$ ./pre-inst-env guix install zile-on-guile vim-full -p /tmp/foo The following packages will be installed: zile-on-guile 2.4.14-0.fd09781 vim-full 8.2.0411 4.9 MB will be downloaded: /gnu/store/wwjhip4wvhjyfj6fs7936bbsgd4yax6g-git-minimal-2.26.0 downloading from https://ci.guix.gnu.org/nar/lzip/wwjhip4wvhjyfj6fs7936bbsgd4yax6g-git-minimal-2.26.0... git-minimal-2.26.0 4.6MiB 7.0MiB/s 00:01 [##################] 100.0% 9.4 MB will be downloaded: /gnu/store/vf7w4yiax38ra7x8aqqvbnc38c0ldgpm-zile-on-guile-2.4.14-0.fd09781 /gnu/store/dnj9wljcck9vdwgp7dwxk00qnnk1g3c5-vim-full-8.2.0411 downloading from https://ci.guix.gnu.org/nar/lzip/dnj9wljcck9vdwgp7dwxk00qnnk1g3c5-vim-full-8.2.0411... vim-full-8.2.0411 8.9MiB 7.6MiB/s 00:01 [##################] 100.0% downloading from https://ci.guix.gnu.org/nar/lzip/vf7w4yiax38ra7x8aqqvbnc38c0ldgpm-zile-on-guile-2.4.14-0.fd09781... zile-on-guile-2.4.14-0.fd09781 140KiB 1.8MiB/s 00:00 [##################] 100.0% The following derivation will be built: /gnu/store/d9xms78917w67xq71pqsx5x9s6dmq6d7-profile.drv The following graft will be made: /gnu/store/4n6dmg6iwjg0adpcvqygr9wgsnclswss-vim-full-8.2.0411.drv applying 8 grafts for /gnu/store/4n6dmg6iwjg0adpcvqygr9wgsnclswss-vim-full-8.2.0411.drv... building /gnu/store/d9xms78917w67xq71pqsx5x9s6dmq6d7-profile.drv... --8<---------------cut here---------------end--------------->8--- This time we see three phases. The dry run is a bit confusing because it only shows the first of these phases (on the branch ‘--dry-run’ no longer implies ‘--no-grafts’, but that’s an optional change.) You can give it a spin and report on the UX by doing: guix pull --branch=wip-build-accumulator -p /tmp/test /tmp/test/bin/guix … Thoughts? The insight here is that the call graph mirrors the dependency graph. Capturing the continuation when we encounter a ‘build-things’ call is akin to modeling dynamic dependencies in the graph and implementing backtracking. I haven’t measured the cost of all this but it’s not noticeable so far (and it very much improves the situation by avoiding repeated substitute fetches!). Thanks, Ludo’. ¹ https://issues.guix.gnu.org/issue/40130 ² https://issues.guix.gnu.org/issue/22990
signature.asc
Description: PGP signature
