> As a reminder for the bitrot in the last 16 years... > > http://www.slimey.org/bbc_ticket_10083.txt
Our version of the same is appended (newest updates at top). Rob > Ticket Number: 20010912-2 Ticket Status: UPDATE > Ticket Type : Unscheduled Ticket Source: TEN-US NOC > Ticket Scope : Site Site/Line : New York > Ticket Owner : TEN-US NOC Problem Fixer: Telehouse NY > Ticket Opened: 20010912 12:08 UTC Problem Start: 20010911 20:35 UTC > Ticket Update: 20010918 05:13 UTC > Ticket Closed: Problem Ends : > > > Ticket Summary: Status of 25 Broadway > > Problem Description: > > This ticket is being issued to track the status of the infrastructure > at the DANTE World Services PoP in New York. Individual circuits > will be dealt with separately. Updates will be in the "actions" section > below. > > Affected: > > DANTE World Service > > Actions: > > cziarhe 20010918 05:13 UTC > Load was switched from the Con Ed generator to the critical generator > as planned. Unfortunately, after about 45 minutes the generator > again began to overheat and the load was switched back. This was > achieved without interruption to service. It is now believed this > could be due to limescale build-up in the radiator. > > This is to be rectified by using an acid wash through the cooling > system and again running the generator under load at 22:30 UTC > tonight (Tuesday). > > A plan is also being developed to bring in a new generator should > the situation with the critical generator persist. > > > > Time to Fix: (Hours:Mins) > > > > Fix: > > > > History: > > cziarhe 20010917 21:53 UTC > Telehouse NY believe the problems with their "critical" generator > have been caused by faulty thermostats. These have now been replaced > and the generator has been spun up for a basic test. The next > step is to try a load test, which will begin at 22:30 UTC. If > this is successful, then load from the "essential" generator, which > supplies the air conditioning will be transferred to the Con Ed > generator for maintainence to take place on this generator, which > has also been running overtemperature. > > Assuming all this works successfully, Telehouse plan to run the > Con Ed generator in conjunction with either the essential (Air > Con) or critical (equipment) generator to make the best use of > available fuel. Further fuel supply problems are not thought to > be likely as there are 17 tankers in the city, and a further 30 > generators are available. > > The IP NOC will monitor the initial switch to the critical generator > to ensure equipment remains in operation, and reinstall the workaround > with KPN should it be needed. > > cziarhe 20010916 14:59 UTC > The latest update from Telehouse reports that an extra 2000 gallon > fuel tank is expected to be connected to the Con Ed generator soon > to allow for longer refuelling intervals. in addition, Con Ed's > refuelling process to all buildings in the area has now settled > down, so it should be as reliable as possible in the circumstances. > > An engineer is currently onsite investigating the problems with > Telehouse's own generator. > > cziarhe 20010916 12:05 UTC > Regular fuel deliveries appear to have been secured for the Con > Ed generator. There were deliveries at 02:00 UTC and 11:30 UTC, > and another is scheduled for 21:30 UTC. > > The status of the Telehouse generator is currently unknown. > > cziajom 20010915 20:16 UTC > Contacted Telehouse NY. Informed that Con Ed generator is now up, > with enough fuel until Tuesday (18 Sep). Engineers are still working > on the Telehouse generator. NY Routers are starting to come up. > > cziarhe 20010915 18:25 UTC > Despite what we were informed of earlier, the Con Ed generator ran > out of fuel at 16:45 UTC. Telehouse's generator then started up, > but overheated again. There is a fuel truck just outside the cordoned > area, but it is not being allowed through, and unfortunately we > have no ETA for the fuel, or time-to-repair for the Telehouse generator. > > cziarhe 20010915 18:15 UTC > Power was lost again at 17:45 UTC, we are contacting Telehouse for > information. > > cziarhe 20010915 15:52 UTC > The current status relayed to us by Telehouse is as follows: > > The Con Ed generator is monitored around the clock by Con Ed personnel, > who are responsible for refuelling it. As of 15:00 UTC there was > approximately 15 hours worth of fuel remaining. > > The water pump on Telehouse's own generator has been replaced and > the unit has been run for about 30 minutes to test. Telehouse > are taking this opportunity to perform some routine maintenance > (changing oil and filters, pressure washing the radiator) to ensure > it is ready should it be needed for another extended run. > > Fuel currently available onsite should last until Thursday at current > consumption rates. Another delivery is being planned for Monday. > > Engineer access is possible, but the procedure is slightly different > due to movement restrictions still in effect. > > cziarhe 20010915 08:55 UTC > Power was returned via the Con Edison generator at approximately > 02:45 UTC this morning (Saturday). All DANTE World Service equipment > is functioning without any failed power supplies. Telehouse also > expect their own generator to be fixed soon, so there will be two > sources of power again, as the Con Edison generator will remain > until grid power is restored. > > cziarhe 20010914 18:06 UTC > Con Edison have delivered the transformer and connected it to be > building's power supply, they expect to have the generator itself > online by approximately 21:00 UTC this evening (Friday). > > Unfortunately the water pump for Telehouse's own generator has > been difficult to source and is currently being shipped from Pennsylvania > with an expected arrival time at 25 Broadway of midnight tonight > UTC. Installation is planned for tomorrow. > > Hopefully the Con Edison generator should allow the DANTE World > Service to resume at some point this evening. > > cziarhe 20010914 08:11 UTC > To clarify the previous entry, the generator from Con Edison was > expected onsite yesterday evening US time for connection this morning > their time, i.e. early to mid-afternoon European time. > > cziaseo 20010914 06:16 UTC > The generator failed because of the water pump. Estimated repair > time > is late tomorrow (Friday, 9/13). In the meantime, Con Ed has committed > > to provide generator from the street and we expect delivery tonight. > > Estimated connection is sometime tomorrow morning. > > > cziajom 20010914 04:57 UTC > An engineer is on-site to try and repair generator. However, still > not able to give any indication of a resolution time. Further information > will be given, as it becomes known to us. > > cziarhe 20010914 01:50 UTC > Power failed again at 01:40 UTC. Apparently there is a generator > technician onsite, and Telehouse are talking to Con Edison about > borrowing a generator in the event the current one cannot be fixed. > > cziarhe 20010914 01:39 UTC > Power has returned at 01:30 UTC. More information will be supplied > as we have it. > > cziarhe 20010914 00:20 UTC > Have spoken to Telehouse Ops, they are waiting for an engineer for > the diesel generator and will let us know via email as soon as > they have more information. This ticket will be updated when the > information reaches us. > > > cziajom 20010913 23:11 UTC > Received msg from M+ (23:06 BST) to contact WorldCom. Contacted > WorldCom. Engineer stated that generator had to be powered off > due to overheating .... and that if generator cannot be fixed, > then they will have to await the arrival of a new generator. Therefore > not possible to give any restoration time for power. > > Ticket # 447111 issued by WC. > > Asked WorldCom to contact us, as they get further information. > > cziarhe 20010913 21:45 UTC > As of 21:40 UTC, Telehouse have been required to shut down the generator > due to a rapid increase in temperature, this has meant a loss of > power to all DANTE World Service equipment. We currently have > no time for restoration of power. > > cziarhe 20010913 16:24 UTC > Fuel has arrived on-site and is currently being transferred into > Telehouse's storage tanks. Once this is complete there will be > sufficient diesel for another four days of generator power. > > cziarhe 20010913 14:29 UTC > The fuel delivery truck is currently preparing to leave Staten Island > for 25 Broadway, however, progress is expected to be slow. > > Should this fail, Con Edison are prepared to deliver fuel to 25 > Broadway. They are still unable to give an estimate as to when > utility power will be restored. > > cziarhe 20010912 21:35 UTC > Commercial power to 25 Broadway was interrupted when debris from > WTC 7 (The Salomon building) fell on the substation. It is not > expected to be restored before the weekend, so delivery of the > fuel mentioned in a previous update is being investigated. > > Telehouse believe they will be able to allow personnel onto the > site at some point tomorrow (Thursday). > > cziarhe 20010912 18:23 UTC > Telehouse have secured an additional 4500 gallons of diesel fuel > and are attempting to get permission from the Port Authority of > New York to transport it to 25 Broadway. > > cziarhe 20010912 13:31 UTC > Telehouse have informed us that if commercial power is not restored, > it is unlikely that they will be able to replenish the fuel supply > when it becomes depleted. > > In addition, the area in which 25 Broadway is located has been > evacuated. Essential staff are remaining on-site, but should they > be forced to vacate the building, Telehouse say the ability to > continue to provide emergency power would be impacted. > > > cziarhe 20010912 12:17 UTC > Sorry, "18:35 UTC" in the previous entry should read "18:35 UTC > on Friday September 14th". > > cziarhe 20010912 12:13 UTC > As of 20:35 UTC on September 11th, 25 Broadway is running on the > emergency diesel generator. Initial information was that they > had 24 hours worth of fuel, but drain is currently much lower and > we have been informed that they could last up to 70 hours, which > takes service up to 18:35 UTC, although this is a maximum. > > It is not clear whether refuelling will currently be possible, > or when commercial grid power will be restored.
signature.asc
Description: Message signed with OpenPGP
