our service provide took an outage, impacted linux.org,
velocitysoftware.com as well as multiple user groups. And our disk
subsystem (6800) did not come back up, so will let you know when I know.
-------- Original Message --------
Subject: Incident at Layer42 Santa Clara Datacenter
Date: Tue, 22 Sep 2009 12:59:09 -0700
From: Layer42 NOC <[email protected]>
To: undisclosed-recipients:;
Affected Customers: Multiple
Issue Severity: Critical
Date of incident: 09/22/2009
Start of Incident: 05:58AM PDT
End of Incident: 06:42AM PDT
Total Outage Time: 44 Minutes
Ticket number: Multiple
Services Affected: Colocated Services at 3080 Raymond
Incident Description:
At approximately 05:58AM on September 22nd, 2009, there was a power
event which caused a fault in one of our UPS modules and System Bypass
Module. The failed UPS caused a cascading effect on the remaining UPS
systems as well as upstream toward utility power and generator system,
resulting in critical power drop.
Impact Description:
AC Power to data center floor was lost
Root Cause:
Currently under investigation.
Actions Taken:
1. The UPS manufacturer is currently on site investigating the issue.
All UPS systems and related power systems will be inspected.
2. The power system is currently running on bypass, pending a thorough
inspection of the UPS system. We will send a status update before
moving customer power back to UPS.
3. A full RFO will be issued pending the completion of our ongoing
investigation.
If you need help with restoring your servers, please let us know. Thank
you for your understanding, and continued patience.
