Hello! Christopher Baines <[email protected]> skribis:
> In summary, the space issue I mentioned in the previous update has > effectively been addressed. All the paused agents are now unpaused and > builds are happening again. Yay! > However, due to the time spent not building things, the backlog is > longer than usual, and the substitute availability (especially for > x86_64-linux and i686-linux) is lower than usual. Yeah, ‘guix weather coreutils’ finds nothing on bordeaux.guix right now. > ** Space issues and the nar-herder > > bordeaux.guix.gnu.org wasn't building things for 2 weeks as the space on > bayfront was getting a little scarce. This week I started rolling out > the nar-herder [2], a utility I've been thinking about for a while. This > has enabled moving nars off of bayfront on to another machine which I've > confusingly named lakefront. Woow, neat! Regarding nar-herder, I think it’d be nice to have a solution to mirroring in Guix proper, developed similarly to other components, because it could be a fairly central tool. ‘guix publish’ is probably not extensible enough to support that, but we could make it a new ‘guix mirror’ or ‘guix sync’ or whatever command. > The lakefront machine is hosted by Hetzner in Germany, and has 6TB of > storage across 2 hard drives. When a nar is requested from bayfront, it > will check it's local storage, and serve it from there if it exists, > otherwise it will forward the request to lakefront. There might be a > slight drop in the speed you can download nars, but apart from that this > change shouldn't be visible. Excellent, thanks for acting this promptly as problems crop up! I think we should make sure this is funded by the project and that credentials are shared. As discussed before, I think it’s best to keep things tidy in that respect, in the spirit of building and maintaining this collectively. > The nar-herder is now busy deleting nars on bayfront which are available > on lakefront. Once it's got through the backlog, I'll enable NGinx > caching for the nars on bayfront, which should help improve > performance. I've tested downloading the largest nars (~2GB) though, and > it seems to work fine. > > In addition to lakefront, I've also added a 6TB hard drive to hatysa, > the HoneyComb LX2 machine that I host. Like lakefront, it's busy > downloading the nars from bayfront. This will act as a backup in case > lakefront is lost. > > In general this is an important step in being more flexible where the > nars are stored. There's still a reliance on storing pretty much all the > nars on a single machine, but which machine has this role is more > flexible now. I think this architecture also makes it easier to break > the "all nars on a single machine" restriction in the future as well. That’s really cool. > Going forward, it would be good to have an additional full backup of the > nars that bayfront can serve things from, to provide more > redundancy. I'm hoping the nar-herder will also enable setting up > geographically distributed mirrors, which will hopefully improve > redundancy further, and maybe performance of fetching nars too. > > Once I've had a chance to neaten up the code a little, I'll get a Guix > package and service written, plus I'll draft a Guix blog post about the > nar-herder. Usually I’m the one asking for blog posts :-), but I’d really like us as a project to collectively engage on the topic before we publicize this specific approach. [...] > That means there's the following currently running build agents (by > architecture): > > - x86_64-linux + i686-linux (3 machines): > - 4 core Intel NUC (giedi) > - Max 16 cores for 1 concurrent build on bayfront > - 32 cores on milano-guix-1 (slow storage though) > - aarch64-linux + armhf-linux (2 machines) > - 16 core HoneyComb LX2 (hatysa) > - 4 core Overdrive (monokuma) > - powerpc64le-linux (1 machine) > - 64 core machine (polaris) This is looking pretty nice! I’m also all for streamlining machine handling, both administratively (in maintenance.git) and financially. > Ironically, I think that the most under-resourced area is x86_64-linux + > i686-linux. bayfront is restricted in what it can do since it also runs > the coordinator, and things go badly if the machine gets > overloaded. bayfront and milano-guix-1 both have hard drive storage, > which also can slow them down when building things (especially > milano-guix-1). > > If we (as a project) want bordeaux.guix.gnu.org to have the capacity to > keep up, it would be good to make a plan to add capacity. I think even a > single high core count x86_64-linux machine with performant storage > would make a big difference. Yes, we should do that, we still have funds for it. > ** IPv6 not supported (yet) > > I was slow to notice, but bordeaux.guix.gnu.org isn't available over > IPv6 yet, since bayfront doesn't seem to have IPv6 connectivity. I want > to address this, but I haven't worked out quite how to yet. Should be almost done now, as discussed on IRC today. \o/ Thanks! Ludo’.
