Stable,
latest statement (Flowspec):
Summary: On August 30, 2020 10:04 GMT, CenturyLink identified an issue
to be affecting users across multiple markets. The IP Network Operations
Center (NOC) was engaged, and initial research identified that an
offending flowspec announcement prevented Border Gateway Protocol (BGP)
from establishing across multiple elements throughout the CenturyLink
Network. The IP NOC deployed a global configuration change to block the
offending flowspec announcement, which allowed BGP to begin to correctly
establish. As the change propagated through the network, the IP NOC
observed all associated service affecting alarms clearing and services
returning to a stable state.
On 30 Aug 2020, at 18:33, Tim Choe via Outages wrote:
According to Centurylink tweet, their service has been restored. Can
someone help me confirm that this is true before I start shifting the
traffic back?
From: Outages [mailto:[email protected]] On Behalf Of
Stephen Flynn via Outages
Sent: Sunday, August 30, 2020 8:41 AM
To: [email protected]
Subject: Re: [outages] 3556 problems
Latest update from CenturyLink
Notes History:
08/30/2020 15:31:25 GMT - The IP NOC continues to observe service
affecting alarms clearing as the configuration that was deployed
continues to propagate through the affected devices. Once affected
services have been confirmed restored and stable, a final notification
will be provided.
08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the
Operations Engineering team confirmed a routing issue to be preventing
BGP sessions from establishing correctly. A configuration adjustment
was deployed at a high level, and sessions began to re-establish with
stability. As the change propagates through the affected devices,
service affecting alarms continue to clear.
Due to the nature of this outage, it may be necessary to reset your
services locally at your equipment, or manually reset your BGP
session. If after that action has been performed a service issue
prevails, please contact the CenturyLink Repair Center for
troubleshooting assistance.
08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border
Gateway Protocol (BGP) issues causing service impacts across multiple
markets. Cooperative escalated investigation efforts with CenturyLink
leadership are underway to isolate and troubleshoot the issue and
expedite service restoral.
08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated
investigation and troubleshooting efforts remain ongoing this time.
08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative
escalated investigations to isolate and troubleshoot the fault at this
time.
08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink
identified a market wide. As this network fault is impacting multiple
clients, the event has increased visibility with CenturyLink
leadership. As such, client trouble tickets associated to this fault
have been automatically escalated to higher priority.
The NOC is engaged and investigating in order to isolate the cause.
Please be advised that updates for this event will be relayed at a
minimum of hourly unless otherwise noted. The information conveyed
hereafter is associated to live troubleshooting effort and as the
discovery process evolves through to service resolution, ticket
closure, or post incident review, details may evolve.
Regards,
Stephen Flynn
Atlantic.Net
Direct: (321) 206-1390
[email protected] <mailto:[email protected]>
www.atlantic.net <http://www.atlantic.net/>
From: Outages <[email protected]
<mailto:[email protected]> > On Behalf Of Dylan Ebner via
Outages
Sent: Sunday, August 30, 2020 11:00 AM
To: Michael McDonald <[email protected]
<mailto:[email protected]> >
Cc: [email protected] <mailto:[email protected]>
Subject: Re: [outages] 3556 problems
CAUTION: This email originated from outside of the organization. Do
not click links or open attachments unless you recognize the sender
and know the content is safe.
Here is their latest update:
The IP NOC with the assistance of the Operations Engineering team
confirmed a routing issue to be preventing BGP sessions from
establishing correctly. A configuration adjustment was deployed at a
high level, and sessions began to re-establish with stability. As the
change propagates through the affected devices, service affecting
alarms continue to clear.
Due to the nature of this outage, it may be necessary to reset your
services locally at your equipment, or manually reset your BGP
session. If after that action has been performed a service issue
prevails, please contact the CenturyLink Repair Center for
troubleshooting assistance.
Next update by: 2020-08-30 15:50 GMT
This notification is sent from an unmonitored email account. Please
click the link below to reply:
mailto:[email protected]?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,
<mailto:[email protected]?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A>
&body=Customer%20Reply%3A%0D%0A
Notes History:
08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the
Operations Engineering team confirmed a routing issue to be preventing
BGP sessions from establishing correctly. A configuration adjustment
was deployed at a high level, and sessions began to re-establish with
stability. As the change propagates through the affected devices,
service affecting alarms continue to clear.
Due to the nature of this outage, it may be necessary to reset your
services locally at your equipment, or manually reset your BGP
session. If after that action has been performed a service issue
prevails, please contact the CenturyLink Repair Center for
troubleshooting assistance.
08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border
Gateway Protocol (BGP) issues causing service impacts across multiple
markets. Cooperative escalated investigation efforts with CenturyLink
leadership are underway to isolate and troubleshoot the issue and
expedite service restoral.
08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated
investigation and troubleshooting efforts remain ongoing this time.
08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative
escalated investigations to isolate and troubleshoot the fault at this
time.
08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink
identified a market wide. As this network fault is impacting multiple
clients, the event has increased visibility with CenturyLink
leadership. As such, client trouble tickets associated to this fault
have been automatically escalated to higher priority.
The NOC is engaged and investigating in order to isolate the cause.
Please be advised that updates for this event will be relayed at a
minimum of hourly unless otherwise noted. The information conveyed
hereafter is associated to live troubleshooting effort and as the
discovery process evolves through to service resolution, ticket
closure, or post incident review, details may evolve.
From: Michael McDonald <[email protected]
<mailto:[email protected]> >
Sent: Sunday, August 30, 2020 9:56 AM
To: Dylan Ebner <[email protected]
<mailto:[email protected]> >
Cc: Jay Ashworth <[email protected] <mailto:[email protected]> >;
[email protected] <mailto:[email protected]>
Subject: Re: [outages] 3556 problems
Not on that ticket, so thanks a ton for the update! Execs are asking
questions about what's going on and it's nice to give them feedback!
On Sun, Aug 30, 2020 at 10:52 AM Dylan Ebner via Outages
<[email protected] <mailto:[email protected]> > wrote:
For those who aren't on the master CL ticket, here is the latest from
CL. Ticket number: 19545403
08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border
Gateway Protocol (BGP) issues causing service impacts across multiple
markets. Cooperative escalated investigation efforts with CenturyLink
leadership are underway to isolate and troubleshoot the issue and
expedite service restoral.
-------- Original message --------
From: Jay Ashworth via Outages <[email protected]
<mailto:[email protected]> >
Date: 8/30/20 9:36 AM (GMT-06:00)
To: [email protected] <mailto:[email protected]>
Subject: [outages] 3556 problems
Okay, it's pretty clear that they're having trouble. :-) Let's limit
it now to actual tickets and reports of clearance.
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
_______________________________________________
Outages mailing list
[email protected] <mailto:[email protected]>
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
[email protected]
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
[email protected]
https://puck.nether.net/mailman/listinfo/outages
