Re: 365 Datacenters Tampa AC Failure
James, Any word on how redundant systems failed at the same time? Or why the secondary was untested and not known-working? It seems like HVAC maintenance was done by the night crew which was quietly eliminated at some point. On Tue, Jun 13, 2023 at 12:02 AM James Ashton wrote: > Hello all, > From our side (365 Data Centers) We saw the issue start at a little after > 6:30 PM yesterday. We had a component failure and we saw a rise in > temperature. We brought additional capacity online while we were working > with our vendors on getting replacement hardware. > The hardware has now been replaced and the system is back online. > This unit and others in this suite are on the block for replacement this > year as we add cooling capacity to our various spaces within the Franklin > Exchange building. > > On a side note, this was not related to the issue mentioned from earlier > in the month. This was a straight component failure and replacement. > > Any customers impacted can reach out to the 365 Customer Service Center > for details and documentation. > > Thank you, > James Ashton > VP Network Engineering > 365 Data Centers > > > > -- > *From: *"NANOG mailing list" > *To: *"George Herbert" > *Cc: *"NANOG mailing list" > *Sent: *Monday, June 12, 2023 8:03:10 PM > *Subject: *Re: 365 Datacenters Tampa AC Failure > > Issue started a little after 2am, I was hard down till about 11:30am > (servers did a high temp shutdown) > > On Jun 12, 2023 8:01 PM, Michael Spears wrote: > > Yep there's issues over there. They had some compressors go down. Should > be getting back to normal now... Hasn't been a good month for them in > regards to cooling.. > > On Jun 12, 2023 7:15 PM, George Herbert wrote: > > Oof. Get ready to replace all spinning media you may have there. > > -George > > Sent from my iPhone > > > On Jun 12, 2023, at 4:06 PM, Nick Olsen wrote: > > > > > > Just a heads up to anyone else colo'd at 365 TPA1/TAMSFLDE. Currently > seeing floor temps of ~105F as reported by equipment. Started yesterday at > ~5:30PM eastern. 2nd AC failure in the last 30 days. They have not sent any > advisory notices as of yet. > > > >
Re: 365 Datacenters Tampa AC Failure
Hello all, >From our side (365 Data Centers) We saw the issue start at a little after 6:30 >PM yesterday. We had a component failure and we saw a rise in temperature. We >brought additional capacity online while we were working with our vendors on >getting replacement hardware. The hardware has now been replaced and the system is back online. This unit and others in this suite are on the block for replacement this year as we add cooling capacity to our various spaces within the Franklin Exchange building. On a side note, this was not related to the issue mentioned from earlier in the month. This was a straight component failure and replacement. Any customers impacted can reach out to the 365 Customer Service Center for details and documentation. Thank you, James Ashton VP Network Engineering 365 Data Centers From: "NANOG mailing list" To: "George Herbert" Cc: "NANOG mailing list" Sent: Monday, June 12, 2023 8:03:10 PM Subject: Re: 365 Datacenters Tampa AC Failure Issue started a little after 2am, I was hard down till about 11:30am (servers did a high temp shutdown) On Jun 12, 2023 8:01 PM, Michael Spears wrote: Yep there's issues over there. They had some compressors go down. Should be getting back to normal now... Hasn't been a good month for them in regards to cooling.. On Jun 12, 2023 7:15 PM, George Herbert wrote: BQ_BEGIN Oof. Get ready to replace all spinning media you may have there. -George Sent from my iPhone > On Jun 12, 2023, at 4:06 PM, Nick Olsen wrote: > > > Just a heads up to anyone else colo'd at 365 TPA1/TAMSFLDE. Currently seeing > floor temps of ~105F as reported by equipment. Started yesterday at ~5:30PM > eastern. 2nd AC failure in the last 30 days. They have not sent any advisory > notices as of yet. BQ_END
Re: 365 Datacenters Tampa AC Failure
Is this the building with the lizard mural? On Mon, Jun 12, 2023 at 4:03 PM Nick Olsen wrote: > Just a heads up to anyone else colo'd at 365 TPA1/TAMSFLDE. Currently > seeing floor temps of ~105F as reported by equipment. Started yesterday at > ~5:30PM eastern. 2nd AC failure in the last 30 days. They have not sent any > advisory notices as of yet. > -- -- Joe Hamelin, W7COM, Tulalip, WA, 360-474-7474
Re: 365 Datacenters Tampa AC Failure
Issue started a little after 2am, I was hard down till about 11:30am (servers did a high temp shutdown)On Jun 12, 2023 8:01 PM, Michael Spears wrote:Yep there's issues over there. They had some compressors go down. Should be getting back to normal now... Hasn't been a good month for them in regards to cooling..On Jun 12, 2023 7:15 PM, George Herbert wrote:Oof. Get ready to replace all spinning media you may have there. -George Sent from my iPhone > On Jun 12, 2023, at 4:06 PM, Nick Olsen wrote: > > > Just a heads up to anyone else colo'd at 365 TPA1/TAMSFLDE. Currently seeing floor temps of ~105F as reported by equipment. Started yesterday at ~5:30PM eastern. 2nd AC failure in the last 30 days. They have not sent any advisory notices as of yet.
Re: 365 Datacenters Tampa AC Failure
Yep there's issues over there. They had some compressors go down. Should be getting back to normal now... Hasn't been a good month for them in regards to cooling..On Jun 12, 2023 7:15 PM, George Herbert wrote:Oof. Get ready to replace all spinning media you may have there. -George Sent from my iPhone > On Jun 12, 2023, at 4:06 PM, Nick Olsen wrote: > > > Just a heads up to anyone else colo'd at 365 TPA1/TAMSFLDE. Currently seeing floor temps of ~105F as reported by equipment. Started yesterday at ~5:30PM eastern. 2nd AC failure in the last 30 days. They have not sent any advisory notices as of yet.
Re: 365 Datacenters Tampa AC Failure
Oof. Get ready to replace all spinning media you may have there. -George Sent from my iPhone > On Jun 12, 2023, at 4:06 PM, Nick Olsen wrote: > > > Just a heads up to anyone else colo'd at 365 TPA1/TAMSFLDE. Currently seeing > floor temps of ~105F as reported by equipment. Started yesterday at ~5:30PM > eastern. 2nd AC failure in the last 30 days. They have not sent any advisory > notices as of yet.
365 Datacenters Tampa AC Failure
Just a heads up to anyone else colo'd at 365 TPA1/TAMSFLDE. Currently seeing floor temps of ~105F as reported by equipment. Started yesterday at ~5:30PM eastern. 2nd AC failure in the last 30 days. They have not sent any advisory notices as of yet.
