Re: GitLab is down: urgent

2021-03-19 Thread Moritz Angermann
I can try to step up and be backup on the other side of the planet. Ben and I are almost 12hs apart exactly. On Sat, 20 Mar 2021 at 1:32 AM, Richard Eisenberg wrote: > > > On Mar 19, 2021, at 12:44 PM, howard.b.gol...@gmail.com wrote: > > I would like to help however I can. I already maintain

Re: Configuration documentation (Was Re: GitLab is down: urgent)

2021-03-19 Thread Gershom B
> How would I learn of that address? Who is ad...@haskell.org? In fact, I know you knew this address at one point, because you emailed it asking for help with the haskell symposium website in 2019 :-) But yes, we all learn things and forget them, so documenting them accessibly is important. The

Re: Configuration documentation (Was Re: GitLab is down: urgent)

2021-03-19 Thread Richard Eisenberg
Thanks, Gershom, for the quick and very illuminating response. > On Mar 19, 2021, at 2:21 PM, Gershom B wrote: > > Cc: ad...@haskell.org which remains (since it was set up over five? > years ago) the contact address for haskell infra admin stuff. How would I learn of that address? Who is

Re: Configuration documentation (Was Re: GitLab is down: urgent)

2021-03-19 Thread howard . b . golden
I suggest continuing this discussion/effort on the Haskell Discourse at https://discourse.haskell.org/t/documentation-of-haskell-websites-servers-and-their-configurations/2153 . Howard On Fri, 2021-03-19 at 14:21 -0400, Gershom B wrote: > Cc: ad...@haskell.org which remains (since it was set

Re: Configuration documentation (Was Re: GitLab is down: urgent)

2021-03-19 Thread Richard Eisenberg
Thanks, Howard! > On Mar 19, 2021, at 2:13 PM, howard.b.gol...@gmail.com wrote: > > I also > will abide by the Haskell Foundation and the haskell.org committee in > arranging this documentation according to their needs and preferences. Thanks for this explicit comment, but I don't think there

Configuration documentation (Was Re: GitLab is down: urgent)

2021-03-19 Thread howard . b . golden
Hi Richard, Gershom and Ben, I have access to the server that runs the Haskell wiki. There are other websites on that server as well. I can document them as well. I know that Gershom B. has done most (all?) of the work on that server. I ask him and the haskell.org committee to send me or point me

Re: GitLab is down: urgent

2021-03-19 Thread Richard Eisenberg
> On Mar 19, 2021, at 12:44 PM, howard.b.gol...@gmail.com wrote: > > I would like to help however I can. I already maintain the Haskell > wiki, and I would like to improve and document its configuration using > devops techniques, preferably consistent with gitlab.haskell.org >

Re: GitLab is down: urgent

2021-03-19 Thread howard . b . golden
Hi Ben, Richard and Davean, I would like to help however I can. I already maintain the Haskell wiki, and I would like to improve and document its configuration using devops techniques, preferably consistent with gitlab.haskell.org. Regards, Howard On Fri, 2021-03-19 at 12:14 -0400, davean

Re: GitLab is down: urgent

2021-03-19 Thread davean
There is the haskell-infrastructure group, who has access. Its knowledge of the peculiarities of each of the very different services that's harder to keep up to date between people. If Ben was hit by a bus it is prepared to be handled. With this diversity and level of manpower though stepping in

Re: GitLab is down: urgent

2021-03-19 Thread Giorgio Marinelli
Hi Everyone, Didn't there exist the ghc-devops group to talk about and manage those problems/topics? Best, Giorgio On Fri, 19 Mar 2021 at 15:00, Richard Eisenberg wrote: > > Hi Ben, > > Thanks for getting the fix in so quickly. > > However: suppose you were unavailable to do this, due to a

Re: GitLab is down: urgent

2021-03-19 Thread Richard Eisenberg
Hi Ben, Thanks for getting the fix in so quickly. However: suppose you were unavailable to do this, due to a well-deserved holiday perhaps. Who else has the credentials and know-how to step in? I feel as if we should have a resource somewhere with a definitive list of critical services and

Re: GitLab is down: urgent

2021-03-19 Thread Ben Gamari
Simon Peyton Jones via ghc-devs writes: > GHC's GitLab seems to be down. Ben? > (I just get 502's) Everything should now be back to normal. I have sent a post-mortem describing the failure mode and fix. Apologies for the inconvenience! Cheers, - Ben signature.asc Description: PGP signature

Post-mortem for last-night's GitLab outage

2021-03-19 Thread Ben Gamari
Hi everyone, It appears that gitlab.haskell.org's GitLab services went down around nine hours ago (around midnight EST). Surprisingly, the outage appears to be entirely unrelated to yesterday's upgrade. Rather, the problem was merely that the docker repository had grown to fill the entirety of

Re: GitLab is down: urgent

2021-03-19 Thread Ben Gamari
Simon Peyton Jones via ghc-devs writes: > GHC's GitLab seems to be down. Ben? > (I just get 502's) I am currently working on this. Unfortunately it looks like we yet again ran out of disk space (in part as a result of the upgrade). Will be up within the hour. Cheers, - Ben signature.asc

Re: GitLab is down: urgent

2021-03-19 Thread Spiwack, Arnaud
>From IRC: it appears that some disks on the gitlab machine are full. I don't know who can fix this issue when Ben is away. On Fri, Mar 19, 2021 at 9:35 AM Simon Peyton Jones via ghc-devs < ghc-devs@haskell.org> wrote: > GHC’s GitLab seems to be down. Ben? > > (I just get 502’s) > > Simon >

GitLab is down: urgent

2021-03-19 Thread Simon Peyton Jones via ghc-devs
GHC's GitLab seems to be down. Ben? (I just get 502's) Simon ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: GitLab upgrade soon

2021-03-19 Thread Giorgio Marinelli
502 On Fri, 19 Mar 2021 at 00:05, Ben Gamari wrote: > > Ben Gamari writes: > > > Hi all, > > > > I will be performing a GitLab upgrade starting in approximately one > > hour. While I generally try to do this with more notice and out of > > working hours, in this case there is an exploitable