Re: [build system] colo maintenance & outage tomorrow, 10am-2pm PDT
a couple of workers needed a bit more time to finish booting up, so no need for my excursion tomorrow. :) builds be building, things look happy. On Thu, Aug 15, 2019 at 6:46 PM Shane Knapp wrote: > it's back up! some of the workers didn't come back cleanly, so i'll have > to hit up the colo tomorrow and persuade them in person. > > On Thu, Aug 15, 2019 at 6:45 PM Wenchen Fan wrote: > >> Thanks for tracking it Shane! >> >> On Fri, Aug 16, 2019 at 7:41 AM Shane Knapp wrote: >> >>> just got an update: >>> >>> there was a problem w/the replacement part, and they're trying to fix >>> it. if that's successful, the expect to have power restored within the >>> hour. >>> >>> if that doesn't work, a new (new) replacement part is scheduled to >>> arrive at 8am tomorrow. >>> >>> shane >>> >>> On Thu, Aug 15, 2019 at 2:07 PM Shane Knapp wrote: >>> quick update: it's been 4 hours, the colo is still down, and i haven't gotten any news yet as to when they're planning on getting power restored. once i hear something i will let everyone know what's up. On Wed, Aug 14, 2019 at 10:22 AM Shane Knapp wrote: > the berkeley colo had a major power distribution breaker fail, and > they've scheduled an emergency repair for tomorrow (thursday) @ 10am. > > they expect this to take ~4 hours. > > i will be shutting down the machines (again) ~9am, and bringing them > all back up once i get the all-clear. > > sorry for the inconvenience... > > shane > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu >>> >>> >>> -- >>> Shane Knapp >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>> https://rise.cs.berkeley.edu >>> >> > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
Re: [build system] colo maintenance & outage tomorrow, 10am-2pm PDT
it's back up! some of the workers didn't come back cleanly, so i'll have to hit up the colo tomorrow and persuade them in person. On Thu, Aug 15, 2019 at 6:45 PM Wenchen Fan wrote: > Thanks for tracking it Shane! > > On Fri, Aug 16, 2019 at 7:41 AM Shane Knapp wrote: > >> just got an update: >> >> there was a problem w/the replacement part, and they're trying to fix >> it. if that's successful, the expect to have power restored within the >> hour. >> >> if that doesn't work, a new (new) replacement part is scheduled to arrive >> at 8am tomorrow. >> >> shane >> >> On Thu, Aug 15, 2019 at 2:07 PM Shane Knapp wrote: >> >>> quick update: >>> >>> it's been 4 hours, the colo is still down, and i haven't gotten any news >>> yet as to when they're planning on getting power restored. >>> >>> once i hear something i will let everyone know what's up. >>> >>> On Wed, Aug 14, 2019 at 10:22 AM Shane Knapp >>> wrote: >>> the berkeley colo had a major power distribution breaker fail, and they've scheduled an emergency repair for tomorrow (thursday) @ 10am. they expect this to take ~4 hours. i will be shutting down the machines (again) ~9am, and bringing them all back up once i get the all-clear. sorry for the inconvenience... shane -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu >>> >>> >>> -- >>> Shane Knapp >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>> https://rise.cs.berkeley.edu >>> >> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu >> > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
Re: [build system] colo maintenance & outage tomorrow, 10am-2pm PDT
Thanks for tracking it Shane! On Fri, Aug 16, 2019 at 7:41 AM Shane Knapp wrote: > just got an update: > > there was a problem w/the replacement part, and they're trying to fix it. > if that's successful, the expect to have power restored within the hour. > > if that doesn't work, a new (new) replacement part is scheduled to arrive > at 8am tomorrow. > > shane > > On Thu, Aug 15, 2019 at 2:07 PM Shane Knapp wrote: > >> quick update: >> >> it's been 4 hours, the colo is still down, and i haven't gotten any news >> yet as to when they're planning on getting power restored. >> >> once i hear something i will let everyone know what's up. >> >> On Wed, Aug 14, 2019 at 10:22 AM Shane Knapp wrote: >> >>> the berkeley colo had a major power distribution breaker fail, and >>> they've scheduled an emergency repair for tomorrow (thursday) @ 10am. >>> >>> they expect this to take ~4 hours. >>> >>> i will be shutting down the machines (again) ~9am, and bringing them all >>> back up once i get the all-clear. >>> >>> sorry for the inconvenience... >>> >>> shane >>> -- >>> Shane Knapp >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>> https://rise.cs.berkeley.edu >>> >> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu >> > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu >
Re: [build system] colo maintenance & outage tomorrow, 10am-2pm PDT
just got an update: there was a problem w/the replacement part, and they're trying to fix it. if that's successful, the expect to have power restored within the hour. if that doesn't work, a new (new) replacement part is scheduled to arrive at 8am tomorrow. shane On Thu, Aug 15, 2019 at 2:07 PM Shane Knapp wrote: > quick update: > > it's been 4 hours, the colo is still down, and i haven't gotten any news > yet as to when they're planning on getting power restored. > > once i hear something i will let everyone know what's up. > > On Wed, Aug 14, 2019 at 10:22 AM Shane Knapp wrote: > >> the berkeley colo had a major power distribution breaker fail, and >> they've scheduled an emergency repair for tomorrow (thursday) @ 10am. >> >> they expect this to take ~4 hours. >> >> i will be shutting down the machines (again) ~9am, and bringing them all >> back up once i get the all-clear. >> >> sorry for the inconvenience... >> >> shane >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu >> > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
Re: [build system] colo maintenance & outage tomorrow, 10am-2pm PDT
quick update: it's been 4 hours, the colo is still down, and i haven't gotten any news yet as to when they're planning on getting power restored. once i hear something i will let everyone know what's up. On Wed, Aug 14, 2019 at 10:22 AM Shane Knapp wrote: > the berkeley colo had a major power distribution breaker fail, and they've > scheduled an emergency repair for tomorrow (thursday) @ 10am. > > they expect this to take ~4 hours. > > i will be shutting down the machines (again) ~9am, and bringing them all > back up once i get the all-clear. > > sorry for the inconvenience... > > shane > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
