One of my Lumen transit circuits in Houston came back up at 2:30am but seeing triple-digit latency across it.
Another Lumen circuit (same facility in Houston) off of another router still remains hard down. On Jul 9, 2024, at 9:14 AM, Dustin Ortiz via Outages <[email protected]> wrote: From lumen regarding the Generator outages at the Houston Gateway site: 2024-07-09 12:58:07 GMT - Upon evaluation, technicians have identified that the chillers are not effectively cooling the equipment as intended. Efforts are underway to diagnose and resolve the issue. In support of these efforts, Lumen Management has reached out to the equipment vendors for additional assistance as technicians continue to work towards restoring the nodes. 2024-07-09 11:36:09 GMT - The chillers have begun to cool the equipment and Field Operations continues in efforts to begin powering up priority network elements. 2024-07-09 10:20:14 GMT - The 2.5 Mega Watt generator has been brought online and field teams are diligently working to activate the chillers, which will facilitate the cooling of our equipment. As the cooling process commences, technicians are sequentially restoring critical core equipment. Additionally, we have procured several replacement cards in anticipation of potential equipment damage caused by elevated temperatures 2024-07-09 08:57:26 GMT - Crews have advised that a 2-megawatt generator has arrived at the gateway site in addition to the initial generator - rated at 25-kilowatts, and wiring efforts are preparing to commence at this time. Teams have advised that wiring is underway and anticipated to be completed on the 2-megawatt generator in approximately 1 hour, and the completion of wiring on the 25-kilowatt generator will follow. Once the installation of the 2-megawatt generator is completed, crews have advised that the expanded load capacity will be used to power 2 chillers which will be used to begin cooling site temperatures and power priority equipment as additional installation efforts are completed. Additionally, an existing generator onsite has been recovered, and DC power has been stabilized. The next update will be provided upon the installation of the 2-megawatt generator, or as new information becomes available. 2024-07-09 08:40:13 GMT - The first generator has arrived onsite and Lumen teams have advised that preparations are underway to commence with the wiring process to install the generator. While the single generator is not anticipated to power the site singlehandedly, it is anticipated that the addition will power some equipment necessary to cool the site and assist with balancing the power load. Lumen teams are closely monitoring conditions and completing assessments on a regular basis. The next update will be provided upon the completed wiring of the first generator, or as new information becomes available. 2024-07-09 07:41:25 GMT - Investigations by the Critical Infrastructure Technical Support (CITS) Team revealed the two on-site generators at a Lumen Gateway Facility in Houston, TX were experiencing power loading issues. As a result, the site was running on one on-site generator for most of the day and the generator was able to support the load. At approximately 01:00 GMT, the second on-site generator began experiencing issues as well. The generators will turn on and begin to run which begins to power the three on-site Direct Current (DC) power plants; however, after about 30 minutes, the generator is unable to regulate itself and then shuts off. As a result of this, teams were having to toggle between charging the DC plant via the generators while running, and cooling the site via the cooling system that is powered by the on-site DC batteries. At 02:24 GMT, teams began to gracefully shut down shelves within an Infinera ring to help conserve power to the affected site and to protect any equipment that may be bouncing as a result of the generator failures. Other equipment at the site is being evaluated as the outage progresses. There are currently two replacement generators en route to the site. One generator is coming from Dallas, TX and is expected to be on site and wired up within 2 hours. The second generator is en route from Midland, TX and should be arriving on site within the next 6-12 hours. Both generators are needed to power the site due to the size of the gateway site. The next update will be provided by 16:00 GMT or as new information becomes available. 2024-07-09 03:53:10 GMT - On July 09, 2024, at 03:40 GMT, Lumen was alerted to service impact in Houston, TX caused by severe area weather conditions resulting from Hurricane Beryl. As this network fault is impacting multiple clients, the event has increased visibility with Lumen leadership; however, as local authorities have issued a state of emergency requiring civilian parties to shelter in place, Lumen has postponed further action until the order has been lifted to ensure the safety of personnel. In the interim, Lumen is actively monitoring Hurricane Beryl's path and will remain in close contact with local, state, and federal offices of emergency management to provide close coordination and timely information. Each outage situation will be evaluated, and a determination made if a remote restoration action is viable or if on-site support is necessary. For the latter, and if evacuations occur, Field Operations personnel will dispatch once government authorization is received. The safety of our employees is of utmost concern; we are taking every step to ensure their safety, which may delay our network repair response times. Updates will continue to be provided every 4 hours or as new information is received. Dustin Ortiz | VP Systems Engineering 214-945-2201 | [email protected]<mailto:[email protected]> From: Outages <[email protected]<mailto:[email protected]>> On Behalf Of Anurag Bhatia via Outages Sent: Tuesday, July 9, 2024 4:29 AM To: Outage Mailing list <[email protected]<mailto:[email protected]>> Subject: Re: [outages] Lumen issues between East Coast and West Coast? CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. (Seems like my last message is on hold for moderation) I see the issue is resolved just now. It was there from 1:30 am to 9:16 am UTC. On Tue, Jul 9, 2024 at 2:11 PM Anurag Bhatia <[email protected]<mailto:[email protected]>> wrote: Since 1:30am UTC - I am getting massive 800ms latency on EU -> US East > US West routing over Lumen AS3356. Wonder if it's just me (my endpoints) or anyone else getting these issues as well? mtr from my server in Germany to Lumen in California: 1. AS51167 ip-1-75-136-213.static.contabo.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=ipht-%2F%2Fpt%3A651-213-7-is13.atc.t.tcoebonnat&s=78n42c16MVZCatawSzOAbXiuLzg> (213.136.75.1) 1.0% 100 6.3 2.0 0.3 12.5 2.7 2. AS??? 10.0.50.3 1.0% 100 6.9 2.4 0.3 17.1 3.0 3. AS3356 lag-104-951.ear3.Frankfurt1.Level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=lphtg%2F%2Fat%3A14-1e95.0-n.arfrak3fv1urlleet.ten.3&s=sn-UEMQHVUU9XYarXpFr600Vth8> (62.67.36.137) 0.0% 100 8.8 10.2 8.1 29.9 4.0 4. AS3356 ae43.3221.ebr2.frf1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ae23.r1.b32nr2.o1.efften.tloc.&s=PoNcXIFn9faAV4OJRwJzBwaqQpI> (171.75.10.150) 0.0% 100 8.9 10.2 8.3 29.1 3.8 5. AS3356 ae3.3.ebr4.frf1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht3%2F%2Fet%3A.e.3rr4f.bcnf1lo.o.eten.t&s=SVDSBsti6BgCQHm2LSCIS7M5wxY> (171.75.0.43) 0.0% 100 10.7 10.9 8.3 28.7 4.5 6. AS3356 ae49.49.ebr1.dus1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar99..eb14.o1ducne.s.ten.tlo&s=lLj74I1dI3EAbec-FjCS60M3eaM> (171.75.0.79) 0.0% 100 12.1 13.5 11.7 33.5 3.9 7. AS3356 ae10.10.ebr3.dus1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar00..eb31.o1ducne.s.ten.tlo&s=aeogCMeSk7LOs3CzvlygqcT_D2Q> (171.75.0.29) 0.0% 100 11.7 11.7 11.5 11.9 0.1 8. AS3356 ae1.100.ebr4.dus1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar0.1.eb40.o1ducne.s.ten.tlo&s=YoKtpI5IOo16ZEqIZUKOjdBaCWg> (171.75.0.45) 0.0% 100 11.7 11.7 11.6 12.1 0.1 9. AS3356 ae49.49.ebr3.ams1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar99..eb34.o1amcne.s.ten.tlo&s=SoLhdJw3zMtgWdDnr1UcZsLdRFo> (171.75.0.121) 0.0% 100 14.7 14.6 14.5 14.9 0.1 10. AS3356 ae1.100.ebr4.ams1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar0.1.eb40.o1amcne.s.ten.tlo&s=GDTwRVBe8SplBHqrbPoU3b6-0ho> (171.75.0.15) 0.0% 100 14.7 14.6 14.6 15.0 0.1 11. AS3356 ae47.47.ebr4.lon1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar77..eb44.o1locne.n.ten.tlo&s=FeULy0zOgR7KfT3DPHvgNUUs53E> (171.75.0.54) 0.0% 100 21.3 21.2 21.1 21.4 0.1 12. AS3356 ae1.100.ebr3.lon1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar0.1.eb30.o1locne.n.ten.tlo&s=dFakOg6Eq_qDjEorYrjLELsps8o> (171.75.0.46) 0.0% 100 21.2 21.2 21.1 27.0 0.6 13. AS3356 ae48.48.ebr3.lon2.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar88..eb34.o2locne.n.ten.tlo&s=FgBVNcouxNeefID11Pvo4CVCh7I> (171.75.0.59) 0.0% 100 21.3 21.3 21.1 23.5 0.2 14. AS3356 ae43.43.ebr1.nyc6.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar33..eb14.v6nylleec.ten.3&s=cg9sqzTnTbYH55OWNzYDnL5s66s> (171.75.16.1) 0.0% 100 90.3 90.3 90.1 90.6 0.1 15. AS3356 ae9.9.ebr3.NewYork6.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht9%2F%2Fet%3A.e.9er3n.blrwyv6.eokten.3le&s=iJ2Hho1oUUIT-OarVl_c5E8pDlw> (4.69.226.222) 0.0% 100 90.7 91.7 90.3 104.3 2.5 16. AS3356 ae1.11.ebr4.NewYork6.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3A4..1nbr.1e.oewek6lyrten.3lev&s=AdrzHMpY4tnoB9EYO0QT3ExzOfU> (4.69.209.38) 0.0% 100 105.4 91.7 90.3 105.5 2.6 17. AS3356 ae10.10.ebr1.SanJose1.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar00..eb11.1jsalse.noten.3leve&s=yYgFmKz5WsmUhsGu6IBKFW1RJqs> (4.69.160.78) 1.0% 100 804.5 800.9 669.2 871.6 46.6 18. AS3356 ae1.0.ebr2.SanJose1.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3A.e.0ar2s.blsnjv1.eoeten.3le&s=zM84PLGELMdmUMx6E7ZUUQtVGoc> (4.69.142.70) 0.0% 100 819.3 802.1 667.4 879.3 47.5 19. AS3356 ae2.30.bear2.Sacramento1.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht2%2F%2Fet%3Ar..3.ea20bnrsaometca3e1.nel.lvte&s=gzqPYSqc17FtpgaKe5VbgiNNmSY> (4.69.141.77) 11.0% 100 825.1 801.5 668.0 865.0 45.1 20. AS3356 system.0.bear1.Sacramento1.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=sphts%2F%2Fyt%3Ae.ter.bam0ma1.nraesce.to3evl1lten.&s=HgYEJnrRdawfz5Wi45acR276RGw> (4.69.183.132) 0.0% 100 821.3 800.5 662.7 865.9 47.1 Thanks. -- Anurag Bhatia anuragbhatia.com<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=%2Fphtn%3A%2Fatstgurahaiabmoc.&s=S-51H0xHJ-Z8oewcounpVrWuXp0> -- Anurag Bhatia anuragbhatia.com<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=%2Fphtn%3A%2Fatstgurahaiabmoc.&s=S-51H0xHJ-Z8oewcounpVrWuXp0> ________________________________ This email has been scanned for spam & viruses. If you believe this email should have been stopped by our filters, click here<https://portal.mailanyone.net/index.html#/outer/reportspam?token=dXNlcj1kb3J0aXpAenRoZXJuZXQuY29tO3RzPTE3MjA1MTc0NTM7dXVpZD02NjhEMDM0NkJDNDY3MDY0QzBERTkxRjUzNzU3RjY2RTt0b2tlbj04MjAwYzBmMTJmZTU4YzY5YzkzNDZlZDIxODU4YmI3NDVlYjEzMWExOw%3D%3D> to report it. _______________________________________________ Outages mailing list [email protected]<mailto:[email protected]> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list [email protected] https://puck.nether.net/mailman/listinfo/outages
