Ciao Giovanni, Giovanni Biscuolo <g...@xelera.eu> skribis:
> recently users and developers are facing hard to manage problems due to > the maintainance of hydra.gnu.org and its proxy mirror.hydra.gnu.org [1] > since 23 Oct 2018 We Guix developers don’t have control over the physical hardware behind hydra.gnu.org; for this machine, we rely on the work of the FSF sysadmins for all things hardware/networking. Unfortunately in this case, this maintenance period was rather unprepared: it wasn’t supposed to last a whole week, rather a few hours or a day at most. Most of the time it took was about copying data to a new disk (!). Had this been prepared, we could have arranged to keep hydra.gnu.org up until the replacement was ready. We Guix developers didn’t have much visibility over what was going on though, and we just didn’t anticipate this. It is clear that this prolonged downtime was harmful to many users and to the project’s reputation. What to do from here? Our main focus is on making berlin.guixsd.org the primary build farm of the project. It has the advantage that one Guix dev has physical access to it (Ricardo); it’s also much more powerful than hydra.gnu.org and the associated build machines. Yet, there’s more work to do: berlin has just 1T of disk space. Ricardo started looking on growing it but was stuck on software issues IIRC. I think fixing this should be a priority, so I think we should help Ricardo fix the software issues as much as we can. That alone doesn’t fix the resilience issue: berlin.guixsd.org could go down at some point for some time. To address that, a possibility that was discussed recently on guix-sysadmin is use bayfront.guixsd.org has a separate build farm and/or mirror of berlin. On top of that we could have a service like httpredir.debian.org, or maybe even a CDN where we’d replicate substitutes, or torrents (looking at you, Julien ;-)). At this point, all these options are still on the table, and anyone with expertise in these areas is very much welcome! > given the prolonged issue, please also consider writing an *official* > blog post explaining the current situation and steps adopted to prevent > similar issues in the future We set up the info-guix mailing list with that in mind (but too late for this incident). Posting blog posts is also a good idea; we should have done that, with instructions on how to switch to berlin.guixsd.org. > 1. is there a method to "replicate the whole store of an official server > (e.g. hydra.gnu.org once healed)" so we can just "guix publish" a > *complete* mirror? In this case a ready to use official > mirror-config.scm could be useful mirror.hydra.gnu.org is a simple nginx proxy to hydra.gnu.org. You can find its config here: https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/nginx/mirror.conf In the past a few people set up their own mirrors using a similar configuration. > 2. is there an official mirrors directory users can look at when needed? No. > 3. is there a plan to build a service similar to > http://httpredir.debian.org/? (I looked on the web but did not find any > reference to such plan) Like I wrote, there’s no concrete plan at this point, which means it’s an opportunity for you and anyone else to chime in and give a hand! Thanks, Ludo’.
