Hi! sorry for my late reply
I confess I haven't still read the whole Guix/GuixSD Reference Maulal, so my apologies if I'm asking something already documented :-S l...@gnu.org (Ludovic Courtès) writes: [...] > We Guix developers don’t have control over the physical hardware behind > hydra.gnu.org; for this machine, we rely on the work of the FSF > sysadmins for all things hardware/networking. OK, thanks for this info > Unfortunately in this case, this maintenance period was rather > unprepared: it wasn’t supposed to last a whole week, rather a few hours > or a day at most. Most of the time it took was about copying data to a > new disk (!). is it published somewhere what are the minimum hardware and disk needs for a complete GuixSD distribution build server? > Had this been prepared, we could have arranged to keep > hydra.gnu.org up until the replacement was ready. We Guix developers > didn’t have much visibility over what was going on though, and we just > didn’t anticipate this. sorry about that, I'm a sysadmin and I know how much my work is impacting others :-) > It is clear that this prolonged downtime was harmful to many users and > to the project’s reputation. GuixSD does not deserve this kind of harm :-( > What to do from here? I once saw the existance of https://git.savannah.gnu.org/cgit/guix/maintenance.git [1] you pointed me (below), but did not read the entire tree now I see we have https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/1.0.org should we add a new "super" task named "resilience of subsitutes network"? looking at https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/machines.scm it seems that some deggree of resilience for hydra.gnu.org is already in place but this does not seem to work as a distributed source of substitute servers, but "just" to offload build jobs to the defined list of build servers could servers in "machines.scm" also be used as substitutes servers? > Our main focus is on making berlin.guixsd.org the primary build farm of > the project. It has the advantage that one Guix dev has physical access > to it (Ricardo); it’s also much more powerful than hydra.gnu.org and the > associated build machines. OK, I see it https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/1.0.org#n30 more details could help fix related issues IMHO a public guixsd.org Sysadmins Manual should be in the roadmap (as MAYBE): that could help the core team job, show the community how the job is done *and* help others to build on our best practices Guix/GuixSD is *the* perfect tool for IaC (infrastructure as code), could be *very* interesting to develop a "Literate GuixSD IaC package" as a meta-project :-) maybe we could (slowly) build a reproducible IaC literate devops document, based on org-mode babel, so we'd have both tangled code and exported documentation > Yet, there’s more work to do: berlin has just 1T of disk space. Ricardo > started looking on growing it but was stuck on software issues IIRC. I > think fixing this should be a priority, so I think we should help > Ricardo fix the software issues as much as we can. I realize I'm pretty new in this community and you can't trust me since we do non even know each other... but I could help if needed, just tell me (in private if more appropriate) what's the hardware issue > That alone doesn’t fix the resilience issue: berlin.guixsd.org could go > down at some point for some time. > > To address that, a possibility that was discussed recently on > guix-sysadmin is use bayfront.guixsd.org has a separate build farm guess you meant "use bayfront.guixsd.org *as* a separate build farm" > and/or mirror of berlin. [...] >> given the prolonged issue, please also consider writing an *official* >> blog post explaining the current situation and steps adopted to prevent >> similar issues in the future > > We set up the info-guix mailing list with that in mind (but too late for > this incident). Posting blog posts is also a good idea; we should have > done that, with instructions on how to switch to berlin.guixsd.org. given the impact on project reputation, please consider a "post-mortem" blog post on what happened: something in line with Ludo's reply to me not all interested users and observers read this (and others) mailing list archives >> 1. is there a method to "replicate the whole store of an official server >> (e.g. hydra.gnu.org once healed)" so we can just "guix publish" a >> *complete* mirror? In this case a ready to use official >> mirror-config.scm could be useful > > mirror.hydra.gnu.org is a simple nginx proxy to hydra.gnu.org. You can > find its config here: > > > https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/nginx/mirror.conf OK, so it's caching proxy I'll see if and how I can build a similar one sorry but I still don't understand why mirror.hydra.gnu.org failed serving substitutes during a 0.15 installation started from the install CD: it was a cache size problem? > In the past a few people set up their own mirrors using a similar > configuration. we shold build a network of organizations and individuals for this >> 2. is there an official mirrors directory users can look at when needed? > > No. I volunteer to keep such a list and coordinate the "volunteers network", if you want >> 3. is there a plan to build a service similar to >> http://httpredir.debian.org/? (I looked on the web but did not find any >> reference to such plan) > > Like I wrote, there’s no concrete plan at this point, which means it’s > an opportunity for you and anyone else to chime in and give a hand! I have no experience in building such a service but it definitely fits in my professional enhancement plan, so I'm still not able to lead such a project but I can help ciao Giovanni -- Giovanni Biscuolo Xelera IT Infrastructures
signature.asc
Description: PGP signature
