We have equipment in that PoP. One of the generators has now failed, the DC batteries serving us drained, and we've lost power. Things are not quite peachy there yet.
I just hope everyone there in the region is safe. Dale Thus spake Tim Burke via Outages ([email protected]) on Tue, Jul 09, 2024 at 05:04:02PM +0000: > Latest update from Lumen outage case - they appear to be turning gear up now: > > 2024-07-09 16:41:39 GMT - Field personnel have successfully connected > additional air handlers at the site, resulting in a steady decrease in > temperature. As conditions return to normal, technicians have commenced the > process of powering up the equipment. An additional generator is being > dispatched from Tulsa and is anticipated to arrive at the site around 21:00 > GMT. Upon its arrival, technicians will initiate the integration of the > generator to supply extra voltage to the site. > > > On Jul 9, 2024, at 9:23 AM, Tim Burke via Outages <[email protected]> wrote: > > One of my Lumen transit circuits in Houston came back up at 2:30am but seeing > triple-digit latency across it. > > Another Lumen circuit (same facility in Houston) off of another router still > remains hard down. > > On Jul 9, 2024, at 9:14 AM, Dustin Ortiz via Outages <[email protected]> > wrote: > > From lumen regarding the Generator outages at the Houston Gateway site: > > 2024-07-09 12:58:07 GMT - Upon evaluation, technicians have identified that > the chillers are not effectively cooling the equipment as intended. Efforts > are underway to diagnose and resolve the issue. In support of these efforts, > Lumen Management has reached out to the equipment vendors for additional > assistance as technicians continue to work towards restoring the nodes. > > 2024-07-09 11:36:09 GMT - The chillers have begun to cool the equipment and > Field Operations continues in efforts to begin powering up priority network > elements. > > 2024-07-09 10:20:14 GMT - The 2.5 Mega Watt generator has been brought online > and field teams are diligently working to activate the chillers, which will > facilitate the cooling of our equipment. As the cooling process commences, > technicians are sequentially restoring critical core equipment. Additionally, > we have procured several replacement cards in anticipation of potential > equipment damage caused by elevated temperatures > > 2024-07-09 08:57:26 GMT - Crews have advised that a 2-megawatt generator has > arrived at the gateway site in addition to the initial generator - rated at > 25-kilowatts, and wiring efforts are preparing to commence at this time. > Teams have advised that wiring is underway and anticipated to be completed on > the 2-megawatt generator in approximately 1 hour, and the completion of > wiring on the 25-kilowatt generator will follow. Once the installation of the > 2-megawatt generator is completed, crews have advised that the expanded load > capacity will be used to power 2 chillers which will be used to begin cooling > site temperatures and power priority equipment as additional installation > efforts are completed. Additionally, an existing generator onsite has been > recovered, and DC power has been stabilized. The next update will be provided > upon the installation of the 2-megawatt generator, or as new information > becomes available. > > 2024-07-09 08:40:13 GMT - The first generator has arrived onsite and Lumen > teams have advised that preparations are underway to commence with the wiring > process to install the generator. While the single generator is not > anticipated to power the site singlehandedly, it is anticipated that the > addition will power some equipment necessary to cool the site and assist with > balancing the power load. Lumen teams are closely monitoring conditions and > completing assessments on a regular basis. The next update will be provided > upon the completed wiring of the first generator, or as new information > becomes available. > > 2024-07-09 07:41:25 GMT - Investigations by the Critical Infrastructure > Technical Support (CITS) Team revealed the two on-site generators at a Lumen > Gateway Facility in Houston, TX were experiencing power loading issues. As a > result, the site was running on one on-site generator for most of the day and > the generator was able to support the load. At approximately 01:00 GMT, the > second on-site generator began experiencing issues as well. The generators > will turn on and begin to run which begins to power the three on-site Direct > Current (DC) power plants; however, after about 30 minutes, the generator is > unable to regulate itself and then shuts off. As a result of this, teams were > having to toggle between charging the DC plant via the generators while > running, and cooling the site via the cooling system that is powered by the > on-site DC batteries. At 02:24 GMT, teams began to gracefully shut down > shelves within an Infinera ring to help conserve power to the affected site > and to protect any equipment that may be bouncing as a result of the > generator failures. Other equipment at the site is being evaluated as the > outage progresses. > > There are currently two replacement generators en route to the site. One > generator is coming from Dallas, TX and is expected to be on site and wired > up within 2 hours. The second generator is en route from Midland, TX and > should be arriving on site within the next 6-12 hours. Both generators are > needed to power the site due to the size of the gateway site. The next update > will be provided by 16:00 GMT or as new information becomes available. > > 2024-07-09 03:53:10 GMT - On July 09, 2024, at 03:40 GMT, Lumen was alerted > to service impact in Houston, TX caused by severe area weather conditions > resulting from Hurricane Beryl. As this network fault is impacting multiple > clients, the event has increased visibility with Lumen leadership; however, > as local authorities have issued a state of emergency requiring civilian > parties to shelter in place, Lumen has postponed further action until the > order has been lifted to ensure the safety of personnel. In the interim, > Lumen is actively monitoring Hurricane Beryl's path and will remain in close > contact with local, state, and federal offices of emergency management to > provide close coordination and timely information. Each outage situation will > be evaluated, and a determination made if a remote restoration action is > viable or if on-site support is necessary. For the latter, and if evacuations > occur, Field Operations personnel will dispatch once government authorization > is received. The safety of our employees is of utmost concern; we are taking > every step to ensure their safety, which may delay our network repair > response times. Updates will continue to be provided every 4 hours or as new > information is received. > > > > Dustin Ortiz | VP Systems Engineering > 214-945-2201 | [email protected]<mailto:[email protected]> > > From: Outages > <[email protected]<mailto:[email protected]>> On Behalf > Of Anurag Bhatia via Outages > Sent: Tuesday, July 9, 2024 4:29 AM > To: Outage Mailing list <[email protected]<mailto:[email protected]>> > Subject: Re: [outages] Lumen issues between East Coast and West Coast? > > CAUTION: This email originated from outside of the organization. Do not click > links or open attachments unless you recognize the sender and know the > content is safe. > > (Seems like my last message is on hold for moderation) > > I see the issue is resolved just now. It was there from 1:30 am to 9:16 am > UTC. > > > > > On Tue, Jul 9, 2024 at 2:11 PM Anurag Bhatia > <[email protected]<mailto:[email protected]>> wrote: > Since 1:30am UTC - I am getting massive 800ms latency on EU -> US East > US > West routing over Lumen AS3356. > > Wonder if it's just me (my endpoints) or anyone else getting these issues as > well? > > > mtr from my server in Germany to Lumen in California: > > 1. AS51167 > ip-1-75-136-213.static.contabo.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=ipht-%2F%2Fpt%3A651-213-7-is13.atc.t.tcoebonnat&s=78n42c16MVZCatawSzOAbXiuLzg> > (213.136.75.1) 1.0% 100 6.3 2.0 0.3 12.5 2.7 > 2. AS??? 10.0.50.3 1.0% > 100 6.9 2.4 0.3 17.1 3.0 > 3. AS3356 > lag-104-951.ear3.Frankfurt1.Level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=lphtg%2F%2Fat%3A14-1e95.0-n.arfrak3fv1urlleet.ten.3&s=sn-UEMQHVUU9XYarXpFr600Vth8> > (62.67.36.137) 0.0% 100 8.8 10.2 8.1 29.9 4.0 > 4. AS3356 > ae43.3221.ebr2.frf1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ae23.r1.b32nr2.o1.efften.tloc.&s=PoNcXIFn9faAV4OJRwJzBwaqQpI> > (171.75.10.150) 0.0% 100 8.9 10.2 8.3 29.1 3.8 > 5. AS3356 > ae3.3.ebr4.frf1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht3%2F%2Fet%3A.e.3rr4f.bcnf1lo.o.eten.t&s=SVDSBsti6BgCQHm2LSCIS7M5wxY> > (171.75.0.43) 0.0% 100 10.7 10.9 8.3 28.7 4.5 > 6. AS3356 > ae49.49.ebr1.dus1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar99..eb14.o1ducne.s.ten.tlo&s=lLj74I1dI3EAbec-FjCS60M3eaM> > (171.75.0.79) 0.0% 100 12.1 13.5 11.7 33.5 3.9 > 7. AS3356 > ae10.10.ebr3.dus1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar00..eb31.o1ducne.s.ten.tlo&s=aeogCMeSk7LOs3CzvlygqcT_D2Q> > (171.75.0.29) 0.0% 100 11.7 11.7 11.5 11.9 0.1 > 8. AS3356 > ae1.100.ebr4.dus1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar0.1.eb40.o1ducne.s.ten.tlo&s=YoKtpI5IOo16ZEqIZUKOjdBaCWg> > (171.75.0.45) 0.0% 100 11.7 11.7 11.6 12.1 0.1 > 9. AS3356 > ae49.49.ebr3.ams1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar99..eb34.o1amcne.s.ten.tlo&s=SoLhdJw3zMtgWdDnr1UcZsLdRFo> > (171.75.0.121) 0.0% 100 14.7 14.6 14.5 14.9 0.1 > 10. AS3356 > ae1.100.ebr4.ams1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar0.1.eb40.o1amcne.s.ten.tlo&s=GDTwRVBe8SplBHqrbPoU3b6-0ho> > (171.75.0.15) 0.0% 100 14.7 14.6 14.6 15.0 0.1 > 11. AS3356 > ae47.47.ebr4.lon1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar77..eb44.o1locne.n.ten.tlo&s=FeULy0zOgR7KfT3DPHvgNUUs53E> > (171.75.0.54) 0.0% 100 21.3 21.2 21.1 21.4 0.1 > 12. AS3356 > ae1.100.ebr3.lon1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar0.1.eb30.o1locne.n.ten.tlo&s=dFakOg6Eq_qDjEorYrjLELsps8o> > (171.75.0.46) 0.0% 100 21.2 21.2 21.1 27.0 0.6 > 13. AS3356 > ae48.48.ebr3.lon2.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar88..eb34.o2locne.n.ten.tlo&s=FgBVNcouxNeefID11Pvo4CVCh7I> > (171.75.0.59) 0.0% 100 21.3 21.3 21.1 23.5 0.2 > 14. AS3356 > ae43.43.ebr1.nyc6.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar33..eb14.v6nylleec.ten.3&s=cg9sqzTnTbYH55OWNzYDnL5s66s> > (171.75.16.1) 0.0% 100 90.3 90.3 90.1 90.6 0.1 > 15. AS3356 > ae9.9.ebr3.NewYork6.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht9%2F%2Fet%3A.e.9er3n.blrwyv6.eokten.3le&s=iJ2Hho1oUUIT-OarVl_c5E8pDlw> > (4.69.226.222) 0.0% 100 90.7 91.7 90.3 104.3 2.5 > 16. AS3356 > ae1.11.ebr4.NewYork6.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3A4..1nbr.1e.oewek6lyrten.3lev&s=AdrzHMpY4tnoB9EYO0QT3ExzOfU> > (4.69.209.38) 0.0% 100 105.4 91.7 90.3 105.5 2.6 > 17. AS3356 > ae10.10.ebr1.SanJose1.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar00..eb11.1jsalse.noten.3leve&s=yYgFmKz5WsmUhsGu6IBKFW1RJqs> > (4.69.160.78) 1.0% 100 804.5 800.9 669.2 871.6 46.6 > 18. AS3356 > ae1.0.ebr2.SanJose1.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3A.e.0ar2s.blsnjv1.eoeten.3le&s=zM84PLGELMdmUMx6E7ZUUQtVGoc> > (4.69.142.70) 0.0% 100 819.3 802.1 667.4 879.3 47.5 > 19. AS3356 > ae2.30.bear2.Sacramento1.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht2%2F%2Fet%3Ar..3.ea20bnrsaometca3e1.nel.lvte&s=gzqPYSqc17FtpgaKe5VbgiNNmSY> > (4.69.141.77) 11.0% 100 825.1 801.5 668.0 865.0 45.1 > 20. AS3356 > system.0.bear1.Sacramento1.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=sphts%2F%2Fyt%3Ae.ter.bam0ma1.nraesce.to3evl1lten.&s=HgYEJnrRdawfz5Wi45acR276RGw> > (4.69.183.132) 0.0% 100 821.3 800.5 662.7 865.9 47.1 > > > > > Thanks. > > > > -- > Anurag Bhatia > anuragbhatia.com<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=%2Fphtn%3A%2Fatstgurahaiabmoc.&s=S-51H0xHJ-Z8oewcounpVrWuXp0> > > > -- > Anurag Bhatia > anuragbhatia.com<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=%2Fphtn%3A%2Fatstgurahaiabmoc.&s=S-51H0xHJ-Z8oewcounpVrWuXp0> > ________________________________ > > This email has been scanned for spam & viruses. If you believe this email > should have been stopped by our filters, click > here<https://portal.mailanyone.net/index.html#/outer/reportspam?token=dXNlcj1kb3J0aXpAenRoZXJuZXQuY29tO3RzPTE3MjA1MTc0NTM7dXVpZD02NjhEMDM0NkJDNDY3MDY0QzBERTkxRjUzNzU3RjY2RTt0b2tlbj04MjAwYzBmMTJmZTU4YzY5YzkzNDZlZDIxODU4YmI3NDVlYjEzMWExOw%3D%3D> > to report it. > > _______________________________________________ > Outages mailing list > [email protected]<mailto:[email protected]> > https://puck.nether.net/mailman/listinfo/outages > > _______________________________________________ > Outages mailing list > [email protected] > https://puck.nether.net/mailman/listinfo/outages > > _______________________________________________ > Outages mailing list > [email protected] > https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list [email protected] https://puck.nether.net/mailman/listinfo/outages
