Apparently it was due to Verizon doing maintenance on their gateway router. It might not have been a Telia problem exactly. It could have been the handoff from Telia to Verizon.
It knocked some Verizon circuits completely offline for 1 1/2 hours. From the Verizon side, it started at around 12:40 AM PST and was resolved around 2:15 AM PST. On Fri, Oct 2, 2020 at 12:02 PM Béla Törös via Outages <[email protected]> wrote: > > Our ticket got updated that it is resolved and stable since 9:00 UTC. > > On Fri, 2 Oct 2020 at 18:17, Total Uptime Netops via Outages > <[email protected]> wrote: > > > > We saw it clear at 4:30 AM ET / 8:30 AM UTC. > > > > No word yet from Telia. > > > > Sincerely, > > > > Charles Henry > > Network Operations Team > > Total Uptime Technologies, LLC > > Support Portal: https://support.totaluptime.com > > > > +1 800-584-1514 - TEL (US & Canada) > > +44 330 808 0228 - TEL (United Kingdom) > > +1 888-584-1194 - FAX > > > > Privileged/Confidential Information may be contained in this message. If > > you are not the addressee indicated in this message (or responsible for > > delivery of the message to such person), you may not copy or deliver this > > message to anyone. In such case, you should destroy this message and kindly > > notify the sender by reply email. Please advise immediately if you or your > > employer does not consent to email or messages of this kind. Opinions, > > conclusions and other information in this message that do not relate to the > > official business of the firm or its subsidiaries or business units shall > > be understood as neither given nor endorsed by it. > > > > ________________________________ > > From: Tomas Lynch <[email protected]> > > Sent: Fri, 02 Oct 2020 12:45:08 +0000 > > To: Total Uptime Netops <[email protected]> > > Subject: Re: [outages] [18B-2709942B-0003] Telia issues in LAX > > Any idea if this is solved by now? > > > > On Fri, Oct 2, 2020 at 3:41 AM Total Uptime Netops via Outages > > <[email protected]> wrote: > >> > >> FYI, Telia has a major network outage in Los Angeles. Their case 01222734 > >> > >> We noticed some time ago when traffic between Los Angles and San Jose got > >> there via Tokyo and Hong Kong. Lovey architecture that is probably > >> affecting others. > >> > >> For your amusement, I paste a trace: > >> > >> Tracing route to te10-1.lb14.lax1.totaluptime.net [199.38.148.127] > >> over a maximum of 30 hops: > >> > >> 1 1 ms 1 ms 1 ms 10.232.22.1 > >> 2 3 ms 3 ms 3 ms (redacted) > >> 3 2 ms 2 ms 2 ms 10.220.204.1 > >> 4 3 ms 2 ms 2 ms eth.15.1.bb1.phx0.phoenixnap.com > >> [108.170.0.10] > >> 5 3 ms 2 ms 2 ms 10.140.2.42 > >> 6 12 ms 11 ms 12 ms unknown.telstraglobal.net [210.176.40.33] > >> 7 12 ms 13 ms 12 ms i-93.1wlt-core02.telstraglobal.net > >> [202.84.253.85] > >> 8 11 ms 11 ms 13 ms i-95.eqla02.telstraglobal.net > >> [202.84.143.30] > >> 9 187 ms 187 ms 187 ms las-b21-link.telia.net [213.248.92.106] > >> 10 * 191 ms 191 ms tky-b1-link.telia.net [62.115.142.129] > >> 11 191 ms * * hnk-b2-link.telia.net [62.115.143.60] > >> 12 * * 192 ms sjo-b21-link.telia.net [62.115.121.194] > >> 13 191 ms * 191 ms ntt-ic-323771-sjo-b21.c.telia.net > >> [62.115.12.53] > >> 14 190 ms * * ae-10.r24.snjsca04.us.bb.gin.ntt.net > >> [129.250.3.26] > >> 15 * 190 ms * ae-3.r25.lsanca07.us.bb.gin.ntt.net > >> [129.250.4.151] > >> 16 190 ms * * ae-2.r00.lsanca07.us.bb.gin.ntt.net > >> [129.250.3.238] > >> 17 189 ms * * ae-0.a00.lsanca07.us.bb.gin.ntt.net > >> [129.250.2.104] > >> 18 189 ms * 191 ms xe-0-0-15-0.a00.lsanca07.us.ce.gin.ntt.net > >> [130.94.195.50] > >> 19 * * * Request timed out. > >> 20 * * * Request timed out. > >> 21 189 ms 190 ms * te10-1.lb14.lax1.totaluptime.net > >> [199.38.148.127] > >> 22 189 ms 189 ms 189 ms te10-1.lb14.lax1.totaluptime.net > >> [199.38.148.127] > >> > >> Sincerely, > >> > >> Charles Henry > >> Network Operations Team > >> Total Uptime Technologies, LLC > >> Support Portal: https://support.totaluptime.com > >> > >> +1 800-584-1514 - TEL (US & Canada) > >> +44 330 808 0228 - TEL (United Kingdom) > >> +1 888-584-1194 - FAX > >> > >> Privileged/Confidential Information may be contained in this message. If > >> you are not the addressee indicated in this message (or responsible for > >> delivery of the message to such person), you may not copy or deliver this > >> message to anyone. In such case, you should destroy this message and > >> kindly notify the sender by reply email. Please advise immediately if you > >> or your employer does not consent to email or messages of this kind. > >> Opinions, conclusions and other information in this message that do not > >> relate to the official business of the firm or its subsidiaries or > >> business units shall be understood as neither given nor endorsed by it. > >> _______________________________________________ > >> Outages mailing list > >> [email protected] > >> https://puck.nether.net/mailman/listinfo/outages > > > > ________________________________ > > From: Total Uptime Netops <[email protected]> > > Sent: Fri, 02 Oct 2020 07:40:37 +0000 > > To: [email protected] > > Subject: [outages] [18B-2709942B-0003] Telia issues in LAX > > FYI, Telia has a major network outage in Los Angeles. Their case 01222734 > > > > We noticed some time ago when traffic between Los Angles and San Jose got > > there via Tokyo and Hong Kong. Lovey architecture that is probably > > affecting others. > > > > For your amusement, I paste a trace: > > > > Tracing route to te10-1.lb14.lax1.totaluptime.net [199.38.148.127] > > over a maximum of 30 hops: > > > > 1 1 ms 1 ms 1 ms 10.232.22.1 > > 2 3 ms 3 ms 3 ms (redacted) > > 3 2 ms 2 ms 2 ms 10.220.204.1 > > 4 3 ms 2 ms 2 ms eth.15.1.bb1.phx0.phoenixnap.com > > [108.170.0.10] > > 5 3 ms 2 ms 2 ms 10.140.2.42 > > 6 12 ms 11 ms 12 ms unknown.telstraglobal.net [210.176.40.33] > > 7 12 ms 13 ms 12 ms i-93.1wlt-core02.telstraglobal.net > > [202.84.253.85] > > 8 11 ms 11 ms 13 ms i-95.eqla02.telstraglobal.net > > [202.84.143.30] > > 9 187 ms 187 ms 187 ms las-b21-link.telia.net [213.248.92.106] > > 10 * 191 ms 191 ms tky-b1-link.telia.net [62.115.142.129] > > 11 191 ms * * hnk-b2-link.telia.net [62.115.143.60] > > 12 * * 192 ms sjo-b21-link.telia.net [62.115.121.194] > > 13 191 ms * 191 ms ntt-ic-323771-sjo-b21.c.telia.net > > [62.115.12.53] > > 14 190 ms * * ae-10.r24.snjsca04.us.bb.gin.ntt.net > > [129.250.3.26] > > 15 * 190 ms * ae-3.r25.lsanca07.us.bb.gin.ntt.net > > [129.250.4.151] > > 16 190 ms * * ae-2.r00.lsanca07.us.bb.gin.ntt.net > > [129.250.3.238] > > 17 189 ms * * ae-0.a00.lsanca07.us.bb.gin.ntt.net > > [129.250.2.104] > > 18 189 ms * 191 ms xe-0-0-15-0.a00.lsanca07.us.ce.gin.ntt.net > > [130.94.195.50] > > 19 * * * Request timed out. > > 20 * * * Request timed out. > > 21 189 ms 190 ms * te10-1.lb14.lax1.totaluptime.net > > [199.38.148.127] > > 22 189 ms 189 ms 189 ms te10-1.lb14.lax1.totaluptime.net > > [199.38.148.127] > > > > Sincerely, > > > > Charles Henry > > Network Operations Team > > Total Uptime Technologies, LLC > > Support Portal: https://support.totaluptime.com > > > > +1 800-584-1514 - TEL (US & Canada) > > +44 330 808 0228 - TEL (United Kingdom) > > +1 888-584-1194 - FAX > > > > Privileged/Confidential Information may be contained in this message. If > > you are not the addressee indicated in this message (or responsible for > > delivery of the message to such person), you may not copy or deliver this > > message to anyone. In such case, you should destroy this message and kindly > > notify the sender by reply email. Please advise immediately if you or your > > employer does not consent to email or messages of this kind. Opinions, > > conclusions and other information in this message that do not relate to the > > official business of the firm or its subsidiaries or business units shall > > be understood as neither given nor endorsed by it. > > ________________________________ > > From: Netops > > Sent: Fri, 02 Oct 2020 07:32:59 +0000 > > To: [email protected] > > Subject: Telia issues in LAX > > FYI, Telia has a major network outage in Los Angeles. Their case 01222734 > > > > We noticed some time ago when traffic between Los Angles and San Jose got > > there via Tokyo and Hong Kong. Lovey architecture that is probably > > affecting others. > > > > For your amusement, I paste a trace: > > > > Tracing route to te10-1.lb14.lax1.totaluptime.net [199.38.148.127] > > over a maximum of 30 hops: > > > > 1 1 ms 1 ms 1 ms 10.232.22.1 > > 2 3 ms 3 ms 3 ms (redacted) > > 3 2 ms 2 ms 2 ms 10.220.204.1 > > 4 3 ms 2 ms 2 ms eth.15.1.bb1.phx0.phoenixnap.com > > [108.170.0.10] > > 5 3 ms 2 ms 2 ms 10.140.2.42 > > 6 12 ms 11 ms 12 ms unknown.telstraglobal.net [210.176.40.33] > > 7 12 ms 13 ms 12 ms i-93.1wlt-core02.telstraglobal.net > > [202.84.253.85] > > 8 11 ms 11 ms 13 ms i-95.eqla02.telstraglobal.net > > [202.84.143.30] > > 9 187 ms 187 ms 187 ms las-b21-link.telia.net [213.248.92.106] > > 10 * 191 ms 191 ms tky-b1-link.telia.net [62.115.142.129] > > 11 191 ms * * hnk-b2-link.telia.net [62.115.143.60] > > 12 * * 192 ms sjo-b21-link.telia.net [62.115.121.194] > > 13 191 ms * 191 ms ntt-ic-323771-sjo-b21.c.telia.net > > [62.115.12.53] > > 14 190 ms * * ae-10.r24.snjsca04.us.bb.gin.ntt.net > > [129.250.3.26] > > 15 * 190 ms * ae-3.r25.lsanca07.us.bb.gin.ntt.net > > [129.250.4.151] > > 16 190 ms * * ae-2.r00.lsanca07.us.bb.gin.ntt.net > > [129.250.3.238] > > 17 189 ms * * ae-0.a00.lsanca07.us.bb.gin.ntt.net > > [129.250.2.104] > > 18 189 ms * 191 ms xe-0-0-15-0.a00.lsanca07.us.ce.gin.ntt.net > > [130.94.195.50] > > 19 * * * Request timed out. > > 20 * * * Request timed out. > > 21 189 ms 190 ms * te10-1.lb14.lax1.totaluptime.net > > [199.38.148.127] > > 22 189 ms 189 ms 189 ms te10-1.lb14.lax1.totaluptime.net > > [199.38.148.127] > > > > Sincerely, > > > > Charles Henry > > Network Operations Team > > Total Uptime Technologies, LLC > > Support Portal: https://support.totaluptime.com > > > > +1 800-584-1514 - TEL (US & Canada) > > +44 330 808 0228 - TEL (United Kingdom) > > +1 888-584-1194 - FAX > > > > Privileged/Confidential Information may be contained in this message. If > > you are not the addressee indicated in this message (or responsible for > > delivery of the message to such person), you may not copy or deliver this > > message to anyone. In such case, you should destroy this message and kindly > > notify the sender by reply email. Please advise immediately if you or your > > employer does not consent to email or messages of this kind. Opinions, > > conclusions and other information in this message that do not relate to the > > official business of the firm or its subsidiaries or business units shall > > be understood as neither given nor endorsed by it. > > _______________________________________________ > > Outages mailing list > > [email protected] > > https://puck.nether.net/mailman/listinfo/outages > _______________________________________________ > Outages mailing list > [email protected] > https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list [email protected] https://puck.nether.net/mailman/listinfo/outages
