On Wed, Jan 28, 2026 at 11:57:20AM +0000, Christopher Baines wrote: > Steve George <[email protected]> writes: > > > One of our current challenges is that we don't want to commit changes > > to the master branch that cause large rebuilds. As I understand it the > > main issue is that the build farm takes a while to build all the > > dependent packages and in the meantime uses that do guix pull won't > > have substitutes so will have to build locally. > > > > Nix seems to solve this by having developers push to a different > > branch from the one that users pull from [0]. This seems like a good > > solution, has Guix tried this previously, am I missing something? > > The way I see it at least is a balance between reducing churn, and > trying to do continuous integration (working on master). > > Large rebuilds often don't lead to packages changing in behaviour, but > the store items changing does mean that users need to build or download > more things, as well as storing more items locally (deduplication does > help a lot with the storage). This is what I think of as churn. > > Part of addressing this comes from improving the package definitions, > and avoiding unnecessary changes to unrelated packages. For example, if > I were making a whole bunch of Ruby related changes, and that happeend > to cause a rebuild of the whole of the texlive packages, that would be > something to avoid by changing the packagee definitions. > > Both the bordeaux build farm and ci.guix.gnu.org provide substitutes for > non-master branches, although this functionality for the bordeaux build > farm is relying on the guix-patches bugs, which should have been > switched off at the end of last year. With QA we previously had > substitutes immediately for both small patches and large branches. The > automated patch testing required building things, and those built things > were made available for people to download. (...)
Right, so I think your first point is that our package definitions are not tight enough leading to an overly complicated dependency tree which in turn causes large numbers of rebuilds. So we could reduce 'churn' by improving our package definitions. Fair enough, maybe that's a separate topic because I guess we need tooling to help developers reduce dependencies and/or we have to split packages more so that they tree can be more atomic? > > A rough proposal for how this would work: > > > > 1. Developers push to a new branch (e.g. 'main') > > 2. CI builds each commit > > 3a. If the build is successful the commit is updated on master > > 3b. The substitute from the build is available to users > > 4. User pull down from master receiving the new commit and any associated > > substitute > > As I say, we already had this with QA, and I think that was better since > the build information could be considered before merging things, and the > setup was simpler since things were merged directly to master (or > another branch). > > > One potential problem is how long it takes the build-farm to do a > > world rebuild or a large rebuild. My impression is that a single world > > rebuild could take two days or more, so if two commits caused that > > then the whole flow of changes would fall behind. Do we agree that > > build resources is a bottle-neck? > > I don't see it this way, as I see it as a balance between churn and > trying to get things merged in quickly. Even if the build farm was > infinitely fast, you'd still want to limit the churn as this would cause > users to have to build or download more. (...) Exclude 'improving packages to reduce the dependency tree' (churn) for a moment. Do you think our velocity is being limited by the 'build farms' speed? What I see devs do is limit making changes to packages 'deep' in the dependency tree because it will cause a 'world rebuild'. Second, there's the issue of the user impact of substitutes not being available immediately on master. AFAIK every 'commit' (update to a package) means the 'build farm' has to build it and all dependencies. When it's done the build the binary it built (substitute) is available to users. At the moment whenever someone commits to 'master' there's a period of time where the binary substitute is not available. So we continually put users in the position where a 'guix pull' may mean they have to build something locally. The larger the number of dependents the longer it takes and the larger the problem for end-users. All things being equal, this idea means that users won't have that problem since 'guix pull' only has commmits which have substitutes that are built. It also has 'quality' improvements since we would only provide the commit to the user if the build was successful. > Speed is relevant when trying to get quick feedback, but this is more > when the build farms are used for testing changes. Things could be > improved, but from my perspective at least, it's not the most important > issue. So I guess I'm seeing three potential benefits: 1. speed of development 2. users only receiving updates with substitutes 3. quality improvements as we only ship things to users that build (bit handwavy I reckon with our tooling) Steve / Futurile
signature.asc
Description: PGP signature
