We were effected by this in reaching the Google SAS. It's been working without problems for 100 minutes.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Sun, Aug 30, 2020, 12:38 PM Tim Choe via Outages <[email protected]> wrote: > According to Centurylink tweet, their service has been restored. Can > someone help me confirm that this is true before I start shifting the > traffic back? > > > > *From:* Outages [mailto:[email protected]] *On Behalf Of *Stephen > Flynn via Outages > *Sent:* Sunday, August 30, 2020 8:41 AM > *To:* [email protected] > *Subject:* Re: [outages] 3556 problems > > > > Latest update from CenturyLink > > > > Notes History: > > > > 08/30/2020 15:31:25 GMT - The IP NOC continues to observe service > affecting alarms clearing as the configuration that was deployed continues > to propagate through the affected devices. Once affected services have been > confirmed restored and stable, a final notification will be provided. > > > > 08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations > Engineering team confirmed a routing issue to be preventing BGP sessions > from establishing correctly. A configuration adjustment was deployed at a > high level, and sessions began to re-establish with stability. As the > change propagates through the affected devices, service affecting alarms > continue to clear. > > > > Due to the nature of this outage, it may be necessary to reset your > services locally at your equipment, or manually reset your BGP session. If > after that action has been performed a service issue prevails, please > contact the CenturyLink Repair Center for troubleshooting assistance. > > > > 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway > Protocol (BGP) issues causing service impacts across multiple markets. > Cooperative escalated investigation efforts with CenturyLink leadership > are underway to isolate and troubleshoot the issue and expedite service > restoral. > > > > 08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated > investigation and troubleshooting efforts remain ongoing this time. > > > > 08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated > investigations to isolate and troubleshoot the fault at this time. > > > > 08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink > identified a market wide. As this network fault is impacting multiple > clients, the event has increased visibility with CenturyLink leadership. As > such, client trouble tickets associated to this fault have been > automatically escalated to higher priority. > > > > The NOC is engaged and investigating in order to isolate the cause. Please > be advised that updates for this event will be relayed at a minimum of > hourly unless otherwise noted. The information conveyed hereafter is > associated to live troubleshooting effort and as the discovery process > evolves through to service resolution, ticket closure, or post incident > review, details may evolve. > > > > > > > > Regards, > > Stephen Flynn > Atlantic.Net > Direct: (321) 206-1390 > [email protected] > www.atlantic.net > > > > *From:* Outages <[email protected]> *On Behalf Of *Dylan Ebner > via Outages > *Sent:* Sunday, August 30, 2020 11:00 AM > *To:* Michael McDonald <[email protected]> > *Cc:* [email protected] > *Subject:* Re: [outages] 3556 problems > > > > > > *CAUTION:* This email originated from outside of the organization. Do not > click links or open attachments unless you recognize the sender and know > the content is safe. > > > > Here is their latest update: > > > > > > The IP NOC with the assistance of the Operations Engineering team > confirmed a routing issue to be preventing BGP sessions from establishing > correctly. A configuration adjustment was deployed at a high level, and > sessions began to re-establish with stability. As the change propagates > through the affected devices, service affecting alarms continue to clear. > > > > Due to the nature of this outage, it may be necessary to reset your > services locally at your equipment, or manually reset your BGP session. If > after that action has been performed a service issue prevails, please > contact the CenturyLink Repair Center for troubleshooting assistance. > > > > Next update by: 2020-08-30 15:50 GMT > > > > This notification is sent from an unmonitored email account. Please click > the link below to reply: > > > > > mailto:[email protected]?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A > <[email protected]?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A> > > > > Notes History: > > > > 08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations > Engineering team confirmed a routing issue to be preventing BGP sessions > from establishing correctly. A configuration adjustment was deployed at a > high level, and sessions began to re-establish with stability. As the > change propagates through the affected devices, service affecting alarms > continue to clear. > > > > Due to the nature of this outage, it may be necessary to reset your > services locally at your equipment, or manually reset your BGP session. If > after that action has been performed a service issue prevails, please > contact the CenturyLink Repair Center for troubleshooting assistance. > > > > 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway > Protocol (BGP) issues causing service impacts across multiple markets. > Cooperative escalated investigation efforts with CenturyLink leadership > are underway to isolate and troubleshoot the issue and expedite service > restoral. > > > > 08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated > investigation and troubleshooting efforts remain ongoing this time. > > > > 08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated > investigations to isolate and troubleshoot the fault at this time. > > > > 08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink > identified a market wide. As this network fault is impacting multiple > clients, the event has increased visibility with CenturyLink leadership. As > such, client trouble tickets associated to this fault have been > automatically escalated to higher priority. > > > > The NOC is engaged and investigating in order to isolate the cause. Please > be advised that updates for this event will be relayed at a minimum of > hourly unless otherwise noted. The information conveyed hereafter is > associated to live troubleshooting effort and as the discovery process > evolves through to service resolution, ticket closure, or post incident > review, details may evolve. > > > > > > *From:* Michael McDonald <[email protected]> > *Sent:* Sunday, August 30, 2020 9:56 AM > *To:* Dylan Ebner <[email protected]> > *Cc:* Jay Ashworth <[email protected]>; [email protected] > *Subject:* Re: [outages] 3556 problems > > > > Not on that ticket, so thanks a ton for the update! Execs are asking > questions about what's going on and it's nice to give them feedback! > > > > On Sun, Aug 30, 2020 at 10:52 AM Dylan Ebner via Outages < > [email protected]> wrote: > > For those who aren't on the master CL ticket, here is the latest from CL. > Ticket number: 19545403 > > > > > > > 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway > Protocol (BGP) issues causing service impacts across multiple markets. > Cooperative escalated investigation efforts with CenturyLink leadership are > underway to isolate and troubleshoot the issue and expedite service > restoral. > > > > > > > > -------- Original message -------- > > From: Jay Ashworth via Outages <[email protected]> > > Date: 8/30/20 9:36 AM (GMT-06:00) > > To: [email protected] > > Subject: [outages] 3556 problems > > > > Okay, it's pretty clear that they're having trouble. :-) Let's limit it > now to actual tickets and reports of clearance. > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. > > _______________________________________________ > Outages mailing list > [email protected] > https://puck.nether.net/mailman/listinfo/outages > > _______________________________________________ > Outages mailing list > [email protected] > https://puck.nether.net/mailman/listinfo/outages >
_______________________________________________ Outages mailing list [email protected] https://puck.nether.net/mailman/listinfo/outages
