We observed roughly 50% of Lumen paths going east<>west in the USA encountering substantial congestion on the order of Anurag's MTR results.
The problem has come and gone a couple of times. Given that 50% of their paths looked totally un-congested, there was probably a lot of room for traffic engineering to reduce the impact here. See: https://imgur.com/a/IBjvecN (36 monitoring flows from NYC<>LA. Green lines are 50th percentile RTT and little multicoloured lines on the bottom are loss. All times UTC). Cheers, Tyler Automattic Network Operations On Tue, Jul 9, 2024 at 2:16 PM Dustin Ortiz via Outages <[email protected]> wrote: > From lumen regarding the Generator outages at the Houston Gateway site: > > > > 2024-07-09 12:58:07 GMT - Upon evaluation, technicians have identified > that the chillers are not effectively cooling the equipment as intended. > Efforts are underway to diagnose and resolve the issue. In support of these > efforts, Lumen Management has reached out to the equipment vendors for > additional assistance as technicians continue to work towards restoring the > nodes. > > 2024-07-09 11:36:09 GMT - The chillers have begun to cool the equipment > and Field Operations continues in efforts to begin powering up priority > network elements. > > 2024-07-09 10:20:14 GMT - The 2.5 Mega Watt generator has been brought > online and field teams are diligently working to activate the chillers, > which will facilitate the cooling of our equipment. As the cooling process > commences, technicians are sequentially restoring critical core equipment. > Additionally, we have procured several replacement cards in anticipation of > potential equipment damage caused by elevated temperatures > > 2024-07-09 08:57:26 GMT - Crews have advised that a 2-megawatt generator > has arrived at the gateway site in addition to the initial generator - > rated at 25-kilowatts, and wiring efforts are preparing to commence at this > time. Teams have advised that wiring is underway and anticipated to be > completed on the 2-megawatt generator in approximately 1 hour, and the > completion of wiring on the 25-kilowatt generator will follow. Once the > installation of the 2-megawatt generator is completed, crews have advised > that the expanded load capacity will be used to power 2 chillers which will > be used to begin cooling site temperatures and power priority equipment as > additional installation efforts are completed. Additionally, an existing > generator onsite has been recovered, and DC power has been stabilized. The > next update will be provided upon the installation of the 2-megawatt > generator, or as new information becomes available. > > 2024-07-09 08:40:13 GMT - The first generator has arrived onsite and Lumen > teams have advised that preparations are underway to commence with the > wiring process to install the generator. While the single generator is not > anticipated to power the site singlehandedly, it is anticipated that the > addition will power some equipment necessary to cool the site and assist > with balancing the power load. Lumen teams are closely monitoring > conditions and completing assessments on a regular basis. The next update > will be provided upon the completed wiring of the first generator, or as > new information becomes available. > > 2024-07-09 07:41:25 GMT - Investigations by the Critical Infrastructure > Technical Support (CITS) Team revealed the two on-site generators at a > Lumen Gateway Facility in Houston, TX were experiencing power loading > issues. As a result, the site was running on one on-site generator for most > of the day and the generator was able to support the load. At approximately > 01:00 GMT, the second on-site generator began experiencing issues as well. > The generators will turn on and begin to run which begins to power the > three on-site Direct Current (DC) power plants; however, after about 30 > minutes, the generator is unable to regulate itself and then shuts off. As > a result of this, teams were having to toggle between charging the DC plant > via the generators while running, and cooling the site via the cooling > system that is powered by the on-site DC batteries. At 02:24 GMT, teams > began to gracefully shut down shelves within an Infinera ring to help > conserve power to the affected site and to protect any equipment that may > be bouncing as a result of the generator failures. Other equipment at the > site is being evaluated as the outage progresses. > > There are currently two replacement generators en route to the site. One > generator is coming from Dallas, TX and is expected to be on site and wired > up within 2 hours. The second generator is en route from Midland, TX and > should be arriving on site within the next 6-12 hours. Both generators are > needed to power the site due to the size of the gateway site. The next > update will be provided by 16:00 GMT or as new information becomes > available. > > 2024-07-09 03:53:10 GMT - On July 09, 2024, at 03:40 GMT, Lumen was > alerted to service impact in Houston, TX caused by severe area weather > conditions resulting from Hurricane Beryl. As this network fault is > impacting multiple clients, the event has increased visibility with Lumen > leadership; however, as local authorities have issued a state of emergency > requiring civilian parties to shelter in place, Lumen has postponed further > action until the order has been lifted to ensure the safety of personnel. > In the interim, Lumen is actively monitoring Hurricane Beryl's path and > will remain in close contact with local, state, and federal offices of > emergency management to provide close coordination and timely information. > Each outage situation will be evaluated, and a determination made if a > remote restoration action is viable or if on-site support is necessary. For > the latter, and if evacuations occur, Field Operations personnel will > dispatch once government authorization is received. The safety of our > employees is of utmost concern; we are taking every step to ensure their > safety, which may delay our network repair response times. Updates will > continue to be provided every 4 hours or as new information is received. > > > > > > > > *Dustin Ortiz | *VP Systems Engineering > *214-945-2201 | [email protected] <[email protected]>* > > > > *From:* Outages <[email protected]> *On Behalf Of *Anurag > Bhatia via Outages > *Sent:* Tuesday, July 9, 2024 4:29 AM > *To:* Outage Mailing list <[email protected]> > *Subject:* Re: [outages] Lumen issues between East Coast and West Coast? > > > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you recognize the sender and know > the content is safe. > > > > (Seems like my last message is on hold for moderation) > > I see the issue is resolved just now. It was there from 1:30 am to 9:16 am > UTC. > > > > > > > > > > On Tue, Jul 9, 2024 at 2:11 PM Anurag Bhatia <[email protected]> wrote: > > Since 1:30am UTC - I am getting massive 800ms latency on EU -> US East > > US West routing over Lumen AS3356. > > > > Wonder if it's just me (my endpoints) or anyone else getting these issues > as well? > > > > > mtr from my server in Germany to Lumen in California: > > > > 1. AS51167 ip-1-75-136-213.static.contabo.net > <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=ipht-%2F%2Fpt%3A651-213-7-is13.atc.t.tcoebonnat&s=78n42c16MVZCatawSzOAbXiuLzg> > (213.136.75.1) 1.0% 100 6.3 2.0 0.3 12.5 2.7 > 2. AS??? 10.0.50.3 1.0% > 100 6.9 2.4 0.3 17.1 3.0 > 3. AS3356 lag-104-951.ear3.Frankfurt1.Level3.net > <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=lphtg%2F%2Fat%3A14-1e95.0-n.arfrak3fv1urlleet.ten.3&s=sn-UEMQHVUU9XYarXpFr600Vth8> > (62.67.36.137) 0.0% 100 8.8 10.2 8.1 29.9 4.0 > 4. AS3356 ae43.3221.ebr2.frf1.neo.colt.net > <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ae23.r1.b32nr2.o1.efften.tloc.&s=PoNcXIFn9faAV4OJRwJzBwaqQpI> > (171.75.10.150) 0.0% 100 8.9 10.2 8.3 29.1 3.8 > 5. AS3356 ae3.3.ebr4.frf1.neo.colt.net > <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht3%2F%2Fet%3A.e.3rr4f.bcnf1lo.o.eten.t&s=SVDSBsti6BgCQHm2LSCIS7M5wxY> > (171.75.0.43) 0.0% 100 10.7 10.9 8.3 28.7 4.5 > 6. AS3356 ae49.49.ebr1.dus1.neo.colt.net > <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar99..eb14.o1ducne.s.ten.tlo&s=lLj74I1dI3EAbec-FjCS60M3eaM> > (171.75.0.79) 0.0% 100 12.1 13.5 11.7 33.5 3.9 > 7. AS3356 ae10.10.ebr3.dus1.neo.colt.net > <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar00..eb31.o1ducne.s.ten.tlo&s=aeogCMeSk7LOs3CzvlygqcT_D2Q> > (171.75.0.29) 0.0% 100 11.7 11.7 11.5 11.9 0.1 > 8. AS3356 ae1.100.ebr4.dus1.neo.colt.net > <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar0.1.eb40.o1ducne.s.ten.tlo&s=YoKtpI5IOo16ZEqIZUKOjdBaCWg> > (171.75.0.45) 0.0% 100 11.7 11.7 11.6 12.1 0.1 > 9. AS3356 ae49.49.ebr3.ams1.neo.colt.net > <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar99..eb34.o1amcne.s.ten.tlo&s=SoLhdJw3zMtgWdDnr1UcZsLdRFo> > (171.75.0.121) 0.0% 100 14.7 14.6 14.5 14.9 0.1 > 10. AS3356 ae1.100.ebr4.ams1.neo.colt.net > <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar0.1.eb40.o1amcne.s.ten.tlo&s=GDTwRVBe8SplBHqrbPoU3b6-0ho> > (171.75.0.15) 0.0% 100 14.7 14.6 14.6 15.0 0.1 > 11. AS3356 ae47.47.ebr4.lon1.neo.colt.net > <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar77..eb44.o1locne.n.ten.tlo&s=FeULy0zOgR7KfT3DPHvgNUUs53E> > (171.75.0.54) 0.0% 100 21.3 21.2 21.1 21.4 0.1 > 12. AS3356 ae1.100.ebr3.lon1.neo.colt.net > <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar0.1.eb30.o1locne.n.ten.tlo&s=dFakOg6Eq_qDjEorYrjLELsps8o> > (171.75.0.46) 0.0% 100 21.2 21.2 21.1 27.0 0.6 > 13. AS3356 ae48.48.ebr3.lon2.neo.colt.net > <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar88..eb34.o2locne.n.ten.tlo&s=FgBVNcouxNeefID11Pvo4CVCh7I> > (171.75.0.59) 0.0% 100 21.3 21.3 21.1 23.5 0.2 > 14. AS3356 ae43.43.ebr1.nyc6.level3.net > <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar33..eb14.v6nylleec.ten.3&s=cg9sqzTnTbYH55OWNzYDnL5s66s> > (171.75.16.1) 0.0% 100 90.3 90.3 90.1 90.6 0.1 > 15. AS3356 ae9.9.ebr3.NewYork6.level3.net > <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht9%2F%2Fet%3A.e.9er3n.blrwyv6.eokten.3le&s=iJ2Hho1oUUIT-OarVl_c5E8pDlw> > (4.69.226.222) 0.0% 100 90.7 91.7 90.3 104.3 2.5 > 16. AS3356 ae1.11.ebr4.NewYork6.level3.net > <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3A4..1nbr.1e.oewek6lyrten.3lev&s=AdrzHMpY4tnoB9EYO0QT3ExzOfU> > (4.69.209.38) 0.0% 100 105.4 91.7 90.3 105.5 2.6 > 17. AS3356 ae10.10.ebr1.SanJose1.level3.net > <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar00..eb11.1jsalse.noten.3leve&s=yYgFmKz5WsmUhsGu6IBKFW1RJqs> > (4.69.160.78) 1.0% 100 804.5 800.9 669.2 871.6 46.6 > 18. AS3356 ae1.0.ebr2.SanJose1.level3.net > <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3A.e.0ar2s.blsnjv1.eoeten.3le&s=zM84PLGELMdmUMx6E7ZUUQtVGoc> > (4.69.142.70) 0.0% 100 819.3 802.1 667.4 879.3 47.5 > 19. AS3356 ae2.30.bear2.Sacramento1.level3.net > <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht2%2F%2Fet%3Ar..3.ea20bnrsaometca3e1.nel.lvte&s=gzqPYSqc17FtpgaKe5VbgiNNmSY> > (4.69.141.77) 11.0% 100 825.1 801.5 668.0 865.0 45.1 > 20. AS3356 system.0.bear1.Sacramento1.level3.net > <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=sphts%2F%2Fyt%3Ae.ter.bam0ma1.nraesce.to3evl1lten.&s=HgYEJnrRdawfz5Wi45acR276RGw> > (4.69.183.132) 0.0% 100 821.3 800.5 662.7 865.9 47.1 > > > > > > > > > > Thanks. > > > > > > > -- > > Anurag Bhatia > > anuragbhatia.com > <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=%2Fphtn%3A%2Fatstgurahaiabmoc.&s=S-51H0xHJ-Z8oewcounpVrWuXp0> > > > > > -- > > Anurag Bhatia > > anuragbhatia.com > <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=%2Fphtn%3A%2Fatstgurahaiabmoc.&s=S-51H0xHJ-Z8oewcounpVrWuXp0> > ------------------------------ > > This email has been scanned for spam & viruses. If you believe this email > should have been stopped by our filters, click here > <https://portal.mailanyone.net/index.html#/outer/reportspam?token=dXNlcj1kb3J0aXpAenRoZXJuZXQuY29tO3RzPTE3MjA1MTc0NTM7dXVpZD02NjhEMDM0NkJDNDY3MDY0QzBERTkxRjUzNzU3RjY2RTt0b2tlbj04MjAwYzBmMTJmZTU4YzY5YzkzNDZlZDIxODU4YmI3NDVlYjEzMWExOw%3D%3D> > to report it. > _______________________________________________ > Outages mailing list > [email protected] > https://puck.nether.net/mailman/listinfo/outages >
_______________________________________________ Outages mailing list [email protected] https://puck.nether.net/mailman/listinfo/outages
