Re: [build system] our colo is having power issues again. there will be a few 'events' this week
alright, they switched over from generator power back to the grid about an hour ago... all the workers are back up and building! shane On Tue, Sep 24, 2019 at 3:23 PM Takeshi Yamamuro wrote: > > Shane, thanks for the hard work! > > Bests, > Takeshi > > On Wed, Sep 25, 2019 at 6:07 AM Jungtaek Lim wrote: >> >> Awesome, thanks for the quick update! >> >> On Wed, Sep 25, 2019 at 6:04 AM Shane Knapp wrote: >>> >>> no worries. since we deprecated the packaging builds i put that >>> worker back in to the rotation... there was a slight env issue but >>> that's fixed and we should be g2g w/python. >>> >>> On Tue, Sep 24, 2019 at 1:50 PM Jungtaek Lim wrote: >>> > >>> > Hi Shane, >>> > >>> > Thanks for the update, and take care of build system! >>> > >>> > Looks like some of build just got failed without test failures, looks >>> > like env. issue. >>> > https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111308/ >>> > >>> > + ./dev/run-tests-jenkins >>> > Python versions prior to 2.7 are not supported. >>> > >>> > >>> > Could you please check on this? >>> > >>> > Thanks, >>> > Jungtaek Lim (HeartSaVioR) >>> > >>> > >>> > On Wed, Sep 25, 2019 at 2:48 AM Shane Knapp wrote: >>> >> >>> >> quick update from our colo admin: they are going to keep the colo on >>> >> generator power until monday morning and not switch back and forth. >>> >> this is great as we'll have solid uptime until monday morning when we >>> >> go back to grid power. >>> >> >>> >> if this changes i will be certain to update everyone here. >>> >> >>> >> thanks, >>> >> >>> >> shane >>> >> >>> >> On Tue, Sep 24, 2019 at 9:50 AM Shane Knapp wrote: >>> >> > >>> >> > aand that was quick! everything is back up and building >>> >> > >>> >> > On Tue, Sep 24, 2019 at 9:39 AM Shane Knapp >>> >> > wrote: >>> >> > > >>> >> > > power switchover is happening now. more updates to come once >>> >> > > machines >>> >> > > come back up. >>> >> > > >>> >> > > On Mon, Sep 23, 2019 at 3:16 PM Shane Knapp >>> >> > > wrote: >>> >> > > > >>> >> > > > the main transformer for our colo is experiencing major issues, and >>> >> > > > campus be performing emergency work on it starting tomorrow morning >>> >> > > > (tuesday sept 24, 9am PDT). it's pretty dire. :( >>> >> > > > >>> >> > > > there's a lot going on, but please expect some sporadic jenkins >>> >> > > > downtime until monday. here's the abbreviated list of when we can >>> >> > > > expect things to happen: >>> >> > > > >>> >> > > > * tomorrow @ 9am we are switching to generator power. there will >>> >> > > > be >>> >> > > > temporary loss of power to the jenkins workers, and if any don't >>> >> > > > come >>> >> > > > back after the switch i will need to head down there and manually >>> >> > > > power them back on. >>> >> > > > >>> >> > > > * thursday @ 9am, switch back to campus power. again, i will deal >>> >> > > > w/any workers that don't come back manually. >>> >> > > > >>> >> > > > * saturday @ 9am, back to generator power. same deal as above. >>> >> > > > >>> >> > > > * saturday or sunday the repairs will be completed. i will monitor >>> >> > > > the situation, and deal w/any problem servers on my way in to work >>> >> > > > on >>> >> > > > monday. >>> >> > > > >>> >> > > > i apologize for any inconvenience that this will cause... and i >>> >> > > > hope >>> >> > > > that this is the final piece of the power puzzle that our colo team >>> >> > > > has to deal with. >>> >> > > > >>> >> > > > thanks in advance, >>> >> > > > >>> >> > > > shane >>> >> > > > -- >>> >> > > > Shane Knapp >>> >> > > > UC Berkeley EECS Research / RISELab Staff Technical Lead >>> >> > > > https://rise.cs.berkeley.edu >>> >> > > >>> >> > > >>> >> > > >>> >> > > -- >>> >> > > Shane Knapp >>> >> > > UC Berkeley EECS Research / RISELab Staff Technical Lead >>> >> > > https://rise.cs.berkeley.edu >>> >> > >>> >> > >>> >> > >>> >> > -- >>> >> > Shane Knapp >>> >> > UC Berkeley EECS Research / RISELab Staff Technical Lead >>> >> > https://rise.cs.berkeley.edu >>> >> >>> >> >>> >> >>> >> -- >>> >> Shane Knapp >>> >> UC Berkeley EECS Research / RISELab Staff Technical Lead >>> >> https://rise.cs.berkeley.edu >>> >> >>> >> - >>> >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>> >> >>> > >>> > >>> > -- >>> > Name : Jungtaek Lim >>> > Blog : http://medium.com/@heartsavior >>> > Twitter : http://twitter.com/heartsavior >>> > LinkedIn : http://www.linkedin.com/in/heartsavior >>> >>> >>> >>> -- >>> Shane Knapp >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>> https://rise.cs.berkeley.edu >> >> >> >> -- >> Name : Jungtaek Lim >> Blog : http://medium.com/@heartsavior >> Twitter : http://twitter.com/heartsavior >> LinkedIn : http://www.linkedin.com/in/heartsavior > > > > -- > --- > Takeshi Yamamuro -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
Re: [build system] our colo is having power issues again. there will be a few 'events' this week
Shane, thanks for the hard work! Bests, Takeshi On Wed, Sep 25, 2019 at 6:07 AM Jungtaek Lim wrote: > Awesome, thanks for the quick update! > > On Wed, Sep 25, 2019 at 6:04 AM Shane Knapp wrote: > >> no worries. since we deprecated the packaging builds i put that >> worker back in to the rotation... there was a slight env issue but >> that's fixed and we should be g2g w/python. >> >> On Tue, Sep 24, 2019 at 1:50 PM Jungtaek Lim wrote: >> > >> > Hi Shane, >> > >> > Thanks for the update, and take care of build system! >> > >> > Looks like some of build just got failed without test failures, looks >> like env. issue. >> > >> https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111308/ >> > >> > + ./dev/run-tests-jenkins >> > Python versions prior to 2.7 are not supported. >> > >> > >> > Could you please check on this? >> > >> > Thanks, >> > Jungtaek Lim (HeartSaVioR) >> > >> > >> > On Wed, Sep 25, 2019 at 2:48 AM Shane Knapp >> wrote: >> >> >> >> quick update from our colo admin: they are going to keep the colo on >> >> generator power until monday morning and not switch back and forth. >> >> this is great as we'll have solid uptime until monday morning when we >> >> go back to grid power. >> >> >> >> if this changes i will be certain to update everyone here. >> >> >> >> thanks, >> >> >> >> shane >> >> >> >> On Tue, Sep 24, 2019 at 9:50 AM Shane Knapp >> wrote: >> >> > >> >> > aand that was quick! everything is back up and building >> >> > >> >> > On Tue, Sep 24, 2019 at 9:39 AM Shane Knapp >> wrote: >> >> > > >> >> > > power switchover is happening now. more updates to come once >> machines >> >> > > come back up. >> >> > > >> >> > > On Mon, Sep 23, 2019 at 3:16 PM Shane Knapp >> wrote: >> >> > > > >> >> > > > the main transformer for our colo is experiencing major issues, >> and >> >> > > > campus be performing emergency work on it starting tomorrow >> morning >> >> > > > (tuesday sept 24, 9am PDT). it's pretty dire. :( >> >> > > > >> >> > > > there's a lot going on, but please expect some sporadic jenkins >> >> > > > downtime until monday. here's the abbreviated list of when we >> can >> >> > > > expect things to happen: >> >> > > > >> >> > > > * tomorrow @ 9am we are switching to generator power. there >> will be >> >> > > > temporary loss of power to the jenkins workers, and if any don't >> come >> >> > > > back after the switch i will need to head down there and manually >> >> > > > power them back on. >> >> > > > >> >> > > > * thursday @ 9am, switch back to campus power. again, i will >> deal >> >> > > > w/any workers that don't come back manually. >> >> > > > >> >> > > > * saturday @ 9am, back to generator power. same deal as above. >> >> > > > >> >> > > > * saturday or sunday the repairs will be completed. i will >> monitor >> >> > > > the situation, and deal w/any problem servers on my way in to >> work on >> >> > > > monday. >> >> > > > >> >> > > > i apologize for any inconvenience that this will cause... and i >> hope >> >> > > > that this is the final piece of the power puzzle that our colo >> team >> >> > > > has to deal with. >> >> > > > >> >> > > > thanks in advance, >> >> > > > >> >> > > > shane >> >> > > > -- >> >> > > > Shane Knapp >> >> > > > UC Berkeley EECS Research / RISELab Staff Technical Lead >> >> > > > https://rise.cs.berkeley.edu >> >> > > >> >> > > >> >> > > >> >> > > -- >> >> > > Shane Knapp >> >> > > UC Berkeley EECS Research / RISELab Staff Technical Lead >> >> > > https://rise.cs.berkeley.edu >> >> > >> >> > >> >> > >> >> > -- >> >> > Shane Knapp >> >> > UC Berkeley EECS Research / RISELab Staff Technical Lead >> >> > https://rise.cs.berkeley.edu >> >> >> >> >> >> >> >> -- >> >> Shane Knapp >> >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> >> https://rise.cs.berkeley.edu >> >> >> >> - >> >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >> >> > >> > >> > -- >> > Name : Jungtaek Lim >> > Blog : http://medium.com/@heartsavior >> > Twitter : http://twitter.com/heartsavior >> > LinkedIn : http://www.linkedin.com/in/heartsavior >> >> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu >> > > > -- > Name : Jungtaek Lim > Blog : http://medium.com/@heartsavior > Twitter : http://twitter.com/heartsavior > LinkedIn : http://www.linkedin.com/in/heartsavior > -- --- Takeshi Yamamuro
Re: [build system] our colo is having power issues again. there will be a few 'events' this week
Awesome, thanks for the quick update! On Wed, Sep 25, 2019 at 6:04 AM Shane Knapp wrote: > no worries. since we deprecated the packaging builds i put that > worker back in to the rotation... there was a slight env issue but > that's fixed and we should be g2g w/python. > > On Tue, Sep 24, 2019 at 1:50 PM Jungtaek Lim wrote: > > > > Hi Shane, > > > > Thanks for the update, and take care of build system! > > > > Looks like some of build just got failed without test failures, looks > like env. issue. > > > https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111308/ > > > > + ./dev/run-tests-jenkins > > Python versions prior to 2.7 are not supported. > > > > > > Could you please check on this? > > > > Thanks, > > Jungtaek Lim (HeartSaVioR) > > > > > > On Wed, Sep 25, 2019 at 2:48 AM Shane Knapp wrote: > >> > >> quick update from our colo admin: they are going to keep the colo on > >> generator power until monday morning and not switch back and forth. > >> this is great as we'll have solid uptime until monday morning when we > >> go back to grid power. > >> > >> if this changes i will be certain to update everyone here. > >> > >> thanks, > >> > >> shane > >> > >> On Tue, Sep 24, 2019 at 9:50 AM Shane Knapp > wrote: > >> > > >> > aand that was quick! everything is back up and building > >> > > >> > On Tue, Sep 24, 2019 at 9:39 AM Shane Knapp > wrote: > >> > > > >> > > power switchover is happening now. more updates to come once > machines > >> > > come back up. > >> > > > >> > > On Mon, Sep 23, 2019 at 3:16 PM Shane Knapp > wrote: > >> > > > > >> > > > the main transformer for our colo is experiencing major issues, > and > >> > > > campus be performing emergency work on it starting tomorrow > morning > >> > > > (tuesday sept 24, 9am PDT). it's pretty dire. :( > >> > > > > >> > > > there's a lot going on, but please expect some sporadic jenkins > >> > > > downtime until monday. here's the abbreviated list of when we can > >> > > > expect things to happen: > >> > > > > >> > > > * tomorrow @ 9am we are switching to generator power. there will > be > >> > > > temporary loss of power to the jenkins workers, and if any don't > come > >> > > > back after the switch i will need to head down there and manually > >> > > > power them back on. > >> > > > > >> > > > * thursday @ 9am, switch back to campus power. again, i will deal > >> > > > w/any workers that don't come back manually. > >> > > > > >> > > > * saturday @ 9am, back to generator power. same deal as above. > >> > > > > >> > > > * saturday or sunday the repairs will be completed. i will > monitor > >> > > > the situation, and deal w/any problem servers on my way in to > work on > >> > > > monday. > >> > > > > >> > > > i apologize for any inconvenience that this will cause... and i > hope > >> > > > that this is the final piece of the power puzzle that our colo > team > >> > > > has to deal with. > >> > > > > >> > > > thanks in advance, > >> > > > > >> > > > shane > >> > > > -- > >> > > > Shane Knapp > >> > > > UC Berkeley EECS Research / RISELab Staff Technical Lead > >> > > > https://rise.cs.berkeley.edu > >> > > > >> > > > >> > > > >> > > -- > >> > > Shane Knapp > >> > > UC Berkeley EECS Research / RISELab Staff Technical Lead > >> > > https://rise.cs.berkeley.edu > >> > > >> > > >> > > >> > -- > >> > Shane Knapp > >> > UC Berkeley EECS Research / RISELab Staff Technical Lead > >> > https://rise.cs.berkeley.edu > >> > >> > >> > >> -- > >> Shane Knapp > >> UC Berkeley EECS Research / RISELab Staff Technical Lead > >> https://rise.cs.berkeley.edu > >> > >> - > >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >> > > > > > > -- > > Name : Jungtaek Lim > > Blog : http://medium.com/@heartsavior > > Twitter : http://twitter.com/heartsavior > > LinkedIn : http://www.linkedin.com/in/heartsavior > > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > -- Name : Jungtaek Lim Blog : http://medium.com/@heartsavior Twitter : http://twitter.com/heartsavior LinkedIn : http://www.linkedin.com/in/heartsavior
Re: [build system] our colo is having power issues again. there will be a few 'events' this week
no worries. since we deprecated the packaging builds i put that worker back in to the rotation... there was a slight env issue but that's fixed and we should be g2g w/python. On Tue, Sep 24, 2019 at 1:50 PM Jungtaek Lim wrote: > > Hi Shane, > > Thanks for the update, and take care of build system! > > Looks like some of build just got failed without test failures, looks like > env. issue. > https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111308/ > > + ./dev/run-tests-jenkins > Python versions prior to 2.7 are not supported. > > > Could you please check on this? > > Thanks, > Jungtaek Lim (HeartSaVioR) > > > On Wed, Sep 25, 2019 at 2:48 AM Shane Knapp wrote: >> >> quick update from our colo admin: they are going to keep the colo on >> generator power until monday morning and not switch back and forth. >> this is great as we'll have solid uptime until monday morning when we >> go back to grid power. >> >> if this changes i will be certain to update everyone here. >> >> thanks, >> >> shane >> >> On Tue, Sep 24, 2019 at 9:50 AM Shane Knapp wrote: >> > >> > aand that was quick! everything is back up and building >> > >> > On Tue, Sep 24, 2019 at 9:39 AM Shane Knapp wrote: >> > > >> > > power switchover is happening now. more updates to come once machines >> > > come back up. >> > > >> > > On Mon, Sep 23, 2019 at 3:16 PM Shane Knapp wrote: >> > > > >> > > > the main transformer for our colo is experiencing major issues, and >> > > > campus be performing emergency work on it starting tomorrow morning >> > > > (tuesday sept 24, 9am PDT). it's pretty dire. :( >> > > > >> > > > there's a lot going on, but please expect some sporadic jenkins >> > > > downtime until monday. here's the abbreviated list of when we can >> > > > expect things to happen: >> > > > >> > > > * tomorrow @ 9am we are switching to generator power. there will be >> > > > temporary loss of power to the jenkins workers, and if any don't come >> > > > back after the switch i will need to head down there and manually >> > > > power them back on. >> > > > >> > > > * thursday @ 9am, switch back to campus power. again, i will deal >> > > > w/any workers that don't come back manually. >> > > > >> > > > * saturday @ 9am, back to generator power. same deal as above. >> > > > >> > > > * saturday or sunday the repairs will be completed. i will monitor >> > > > the situation, and deal w/any problem servers on my way in to work on >> > > > monday. >> > > > >> > > > i apologize for any inconvenience that this will cause... and i hope >> > > > that this is the final piece of the power puzzle that our colo team >> > > > has to deal with. >> > > > >> > > > thanks in advance, >> > > > >> > > > shane >> > > > -- >> > > > Shane Knapp >> > > > UC Berkeley EECS Research / RISELab Staff Technical Lead >> > > > https://rise.cs.berkeley.edu >> > > >> > > >> > > >> > > -- >> > > Shane Knapp >> > > UC Berkeley EECS Research / RISELab Staff Technical Lead >> > > https://rise.cs.berkeley.edu >> > >> > >> > >> > -- >> > Shane Knapp >> > UC Berkeley EECS Research / RISELab Staff Technical Lead >> > https://rise.cs.berkeley.edu >> >> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu >> >> - >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> > > > -- > Name : Jungtaek Lim > Blog : http://medium.com/@heartsavior > Twitter : http://twitter.com/heartsavior > LinkedIn : http://www.linkedin.com/in/heartsavior -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: [build system] our colo is having power issues again. there will be a few 'events' this week
Hi Shane, Thanks for the update, and take care of build system! Looks like some of build just got failed without test failures, looks like env. issue. https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111308/ + ./dev/run-tests-jenkins Python versions prior to 2.7 are not supported. Could you please check on this? Thanks, Jungtaek Lim (HeartSaVioR) On Wed, Sep 25, 2019 at 2:48 AM Shane Knapp wrote: > quick update from our colo admin: they are going to keep the colo on > generator power until monday morning and not switch back and forth. > this is great as we'll have solid uptime until monday morning when we > go back to grid power. > > if this changes i will be certain to update everyone here. > > thanks, > > shane > > On Tue, Sep 24, 2019 at 9:50 AM Shane Knapp wrote: > > > > aand that was quick! everything is back up and building > > > > On Tue, Sep 24, 2019 at 9:39 AM Shane Knapp wrote: > > > > > > power switchover is happening now. more updates to come once machines > > > come back up. > > > > > > On Mon, Sep 23, 2019 at 3:16 PM Shane Knapp > wrote: > > > > > > > > the main transformer for our colo is experiencing major issues, and > > > > campus be performing emergency work on it starting tomorrow morning > > > > (tuesday sept 24, 9am PDT). it's pretty dire. :( > > > > > > > > there's a lot going on, but please expect some sporadic jenkins > > > > downtime until monday. here's the abbreviated list of when we can > > > > expect things to happen: > > > > > > > > * tomorrow @ 9am we are switching to generator power. there will be > > > > temporary loss of power to the jenkins workers, and if any don't come > > > > back after the switch i will need to head down there and manually > > > > power them back on. > > > > > > > > * thursday @ 9am, switch back to campus power. again, i will deal > > > > w/any workers that don't come back manually. > > > > > > > > * saturday @ 9am, back to generator power. same deal as above. > > > > > > > > * saturday or sunday the repairs will be completed. i will monitor > > > > the situation, and deal w/any problem servers on my way in to work on > > > > monday. > > > > > > > > i apologize for any inconvenience that this will cause... and i hope > > > > that this is the final piece of the power puzzle that our colo team > > > > has to deal with. > > > > > > > > thanks in advance, > > > > > > > > shane > > > > -- > > > > Shane Knapp > > > > UC Berkeley EECS Research / RISELab Staff Technical Lead > > > > https://rise.cs.berkeley.edu > > > > > > > > > > > > -- > > > Shane Knapp > > > UC Berkeley EECS Research / RISELab Staff Technical Lead > > > https://rise.cs.berkeley.edu > > > > > > > > -- > > Shane Knapp > > UC Berkeley EECS Research / RISELab Staff Technical Lead > > https://rise.cs.berkeley.edu > > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > > -- Name : Jungtaek Lim Blog : http://medium.com/@heartsavior Twitter : http://twitter.com/heartsavior LinkedIn : http://www.linkedin.com/in/heartsavior
Re: [build system] our colo is having power issues again. there will be a few 'events' this week
quick update from our colo admin: they are going to keep the colo on generator power until monday morning and not switch back and forth. this is great as we'll have solid uptime until monday morning when we go back to grid power. if this changes i will be certain to update everyone here. thanks, shane On Tue, Sep 24, 2019 at 9:50 AM Shane Knapp wrote: > > aand that was quick! everything is back up and building > > On Tue, Sep 24, 2019 at 9:39 AM Shane Knapp wrote: > > > > power switchover is happening now. more updates to come once machines > > come back up. > > > > On Mon, Sep 23, 2019 at 3:16 PM Shane Knapp wrote: > > > > > > the main transformer for our colo is experiencing major issues, and > > > campus be performing emergency work on it starting tomorrow morning > > > (tuesday sept 24, 9am PDT). it's pretty dire. :( > > > > > > there's a lot going on, but please expect some sporadic jenkins > > > downtime until monday. here's the abbreviated list of when we can > > > expect things to happen: > > > > > > * tomorrow @ 9am we are switching to generator power. there will be > > > temporary loss of power to the jenkins workers, and if any don't come > > > back after the switch i will need to head down there and manually > > > power them back on. > > > > > > * thursday @ 9am, switch back to campus power. again, i will deal > > > w/any workers that don't come back manually. > > > > > > * saturday @ 9am, back to generator power. same deal as above. > > > > > > * saturday or sunday the repairs will be completed. i will monitor > > > the situation, and deal w/any problem servers on my way in to work on > > > monday. > > > > > > i apologize for any inconvenience that this will cause... and i hope > > > that this is the final piece of the power puzzle that our colo team > > > has to deal with. > > > > > > thanks in advance, > > > > > > shane > > > -- > > > Shane Knapp > > > UC Berkeley EECS Research / RISELab Staff Technical Lead > > > https://rise.cs.berkeley.edu > > > > > > > > -- > > Shane Knapp > > UC Berkeley EECS Research / RISELab Staff Technical Lead > > https://rise.cs.berkeley.edu > > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: [build system] our colo is having power issues again. there will be a few 'events' this week
aand that was quick! everything is back up and building On Tue, Sep 24, 2019 at 9:39 AM Shane Knapp wrote: > > power switchover is happening now. more updates to come once machines > come back up. > > On Mon, Sep 23, 2019 at 3:16 PM Shane Knapp wrote: > > > > the main transformer for our colo is experiencing major issues, and > > campus be performing emergency work on it starting tomorrow morning > > (tuesday sept 24, 9am PDT). it's pretty dire. :( > > > > there's a lot going on, but please expect some sporadic jenkins > > downtime until monday. here's the abbreviated list of when we can > > expect things to happen: > > > > * tomorrow @ 9am we are switching to generator power. there will be > > temporary loss of power to the jenkins workers, and if any don't come > > back after the switch i will need to head down there and manually > > power them back on. > > > > * thursday @ 9am, switch back to campus power. again, i will deal > > w/any workers that don't come back manually. > > > > * saturday @ 9am, back to generator power. same deal as above. > > > > * saturday or sunday the repairs will be completed. i will monitor > > the situation, and deal w/any problem servers on my way in to work on > > monday. > > > > i apologize for any inconvenience that this will cause... and i hope > > that this is the final piece of the power puzzle that our colo team > > has to deal with. > > > > thanks in advance, > > > > shane > > -- > > Shane Knapp > > UC Berkeley EECS Research / RISELab Staff Technical Lead > > https://rise.cs.berkeley.edu > > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: [build system] our colo is having power issues again. there will be a few 'events' this week
power switchover is happening now. more updates to come once machines come back up. On Mon, Sep 23, 2019 at 3:16 PM Shane Knapp wrote: > > the main transformer for our colo is experiencing major issues, and > campus be performing emergency work on it starting tomorrow morning > (tuesday sept 24, 9am PDT). it's pretty dire. :( > > there's a lot going on, but please expect some sporadic jenkins > downtime until monday. here's the abbreviated list of when we can > expect things to happen: > > * tomorrow @ 9am we are switching to generator power. there will be > temporary loss of power to the jenkins workers, and if any don't come > back after the switch i will need to head down there and manually > power them back on. > > * thursday @ 9am, switch back to campus power. again, i will deal > w/any workers that don't come back manually. > > * saturday @ 9am, back to generator power. same deal as above. > > * saturday or sunday the repairs will be completed. i will monitor > the situation, and deal w/any problem servers on my way in to work on > monday. > > i apologize for any inconvenience that this will cause... and i hope > that this is the final piece of the power puzzle that our colo team > has to deal with. > > thanks in advance, > > shane > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
[build system] our colo is having power issues again. there will be a few 'events' this week
the main transformer for our colo is experiencing major issues, and campus be performing emergency work on it starting tomorrow morning (tuesday sept 24, 9am PDT). it's pretty dire. :( there's a lot going on, but please expect some sporadic jenkins downtime until monday. here's the abbreviated list of when we can expect things to happen: * tomorrow @ 9am we are switching to generator power. there will be temporary loss of power to the jenkins workers, and if any don't come back after the switch i will need to head down there and manually power them back on. * thursday @ 9am, switch back to campus power. again, i will deal w/any workers that don't come back manually. * saturday @ 9am, back to generator power. same deal as above. * saturday or sunday the repairs will be completed. i will monitor the situation, and deal w/any problem servers on my way in to work on monday. i apologize for any inconvenience that this will cause... and i hope that this is the final piece of the power puzzle that our colo team has to deal with. thanks in advance, shane -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
