turns out that minikube/k8s and friends were being oom-killed and this was causing all sorts of weirdnesses.
i've upped the ram limits on all of the k8s jobs to 8G (from 6G), and we'll keep an eye on things and see how they go. On Mon, Aug 9, 2021 at 12:02 PM shane knapp ☠ <skn...@berkeley.edu> wrote: > as workers are continuing to fail, i've stopped jenkins from accepting new > builds for the time being. > > more updates as they come. > > On Mon, Aug 9, 2021 at 9:17 AM shane knapp ☠ <skn...@berkeley.edu> wrote: > >> happy monday! >> >> the server gods did not smile upon us this weekend, and 4 of the workers >> are down. we'll most likely need to head to our colo some time today and >> give them an in-person kick and see what's going on. >> >> i'll send an update when they're back up. >> >> shane >> -- >> Shane Knapp >> Computer Guy / Voice of Reason >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu >> > > > -- > Shane Knapp > Computer Guy / Voice of Reason > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > -- Shane Knapp Computer Guy / Voice of Reason UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
