Re: GitLab partial outage - attempting to mitigate

2023-03-20 Thread Ben Gamari
Brandon Allbery writes: > Isn't it just "move /nix out of the way, bind mount a new one from a > larger drive, use rsync to move the data"? > Something like that, yes [1]. Cheers, - Ben [1] https://nixos.wiki/wiki/Storage_optimization#Moving_the_store signature.asc Description: PGP

Re: GitLab partial outage - attempting to mitigate

2023-03-20 Thread Ben Gamari
Ben Gamari writes: > Bryan Richter via ghc-devs writes: > >> I eventually resorted to a server reboot, which cleared up all the problems >> I was seeing. I think we're back in business. >> > The root partition was close to running out of disk space yesterday. The > problem appears to be that

Re: GitLab partial outage - attempting to mitigate

2023-03-20 Thread Brandon Allbery
Isn't it just "move /nix out of the way, bind mount a new one from a larger drive, use rsync to move the data"? On Mon, Mar 20, 2023 at 9:25 AM Ben Gamari wrote: > > Bryan Richter via ghc-devs writes: > > > I eventually resorted to a server reboot, which cleared up all the problems > > I was

Re: GitLab partial outage - attempting to mitigate

2023-03-20 Thread Ben Gamari
Bryan Richter via ghc-devs writes: > I eventually resorted to a server reboot, which cleared up all the problems > I was seeing. I think we're back in business. > The root partition was close to running out of disk space yesterday. The problem appears to be that /nix is located on the small

Re: GitLab partial outage - attempting to mitigate

2023-03-20 Thread Bryan Richter via ghc-devs
I eventually resorted to a server reboot, which cleared up all the problems I was seeing. I think we're back in business. Symptoms were: * No new data coming into https://grafana.gitlab.haskell.org/d/iiCppweMz/marge-bot?orgId=2=now-7d=now=30m * High-frequency repetition of the system log message

GitLab partial outage - attempting to mitigate

2023-03-20 Thread Bryan Richter via ghc-devs
I am seeing a few different problems with the GitLab server right now. I am gonna try to mitigate the issues, so the server might be unavailable for a few short periods. ___ ghc-devs mailing list ghc-devs@haskell.org