I just shifted back and everything looks normal. I'm in Waltham MA. On Sun, Aug 30, 2020 at 12:44 PM Josh Luthman via Outages < [email protected]> wrote:
> We were effected by this in reaching the Google SAS. It's been working > without problems for 100 minutes. > > Josh Luthman > Office: 937-552-2340 > Direct: 937-552-2343 > 1100 Wayne St > Suite 1337 > Troy, OH 45373 > > On Sun, Aug 30, 2020, 12:38 PM Tim Choe via Outages <[email protected]> > wrote: > >> According to Centurylink tweet, their service has been restored. Can >> someone help me confirm that this is true before I start shifting the >> traffic back? >> >> >> >> *From:* Outages [mailto:[email protected]] *On Behalf Of *Stephen >> Flynn via Outages >> *Sent:* Sunday, August 30, 2020 8:41 AM >> *To:* [email protected] >> *Subject:* Re: [outages] 3556 problems >> >> >> >> Latest update from CenturyLink >> >> >> >> Notes History: >> >> >> >> 08/30/2020 15:31:25 GMT - The IP NOC continues to observe service >> affecting alarms clearing as the configuration that was deployed continues >> to propagate through the affected devices. Once affected services have been >> confirmed restored and stable, a final notification will be provided. >> >> >> >> 08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the >> Operations Engineering team confirmed a routing issue to be preventing BGP >> sessions from establishing correctly. A configuration adjustment was >> deployed at a high level, and sessions began to re-establish with >> stability. As the change propagates through the affected devices, service >> affecting alarms continue to clear. >> >> >> >> Due to the nature of this outage, it may be necessary to reset your >> services locally at your equipment, or manually reset your BGP session. If >> after that action has been performed a service issue prevails, please >> contact the CenturyLink Repair Center for troubleshooting assistance. >> >> >> >> 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway >> Protocol (BGP) issues causing service impacts across multiple markets. >> Cooperative escalated investigation efforts with CenturyLink leadership >> are underway to isolate and troubleshoot the issue and expedite service >> restoral. >> >> >> >> 08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated >> investigation and troubleshooting efforts remain ongoing this time. >> >> >> >> 08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated >> investigations to isolate and troubleshoot the fault at this time. >> >> >> >> 08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink >> identified a market wide. As this network fault is impacting multiple >> clients, the event has increased visibility with CenturyLink leadership. As >> such, client trouble tickets associated to this fault have been >> automatically escalated to higher priority. >> >> >> >> The NOC is engaged and investigating in order to isolate the cause. >> Please be advised that updates for this event will be relayed at a minimum >> of hourly unless otherwise noted. The information conveyed hereafter is >> associated to live troubleshooting effort and as the discovery process >> evolves through to service resolution, ticket closure, or post incident >> review, details may evolve. >> >> >> >> >> >> >> >> Regards, >> >> Stephen Flynn >> Atlantic.Net >> Direct: (321) 206-1390 >> [email protected] >> www.atlantic.net >> >> >> >> *From:* Outages <[email protected]> *On Behalf Of *Dylan Ebner >> via Outages >> *Sent:* Sunday, August 30, 2020 11:00 AM >> *To:* Michael McDonald <[email protected]> >> *Cc:* [email protected] >> *Subject:* Re: [outages] 3556 problems >> >> >> >> >> >> *CAUTION:* This email originated from outside of the organization. Do >> not click links or open attachments unless you recognize the sender and >> know the content is safe. >> >> >> >> Here is their latest update: >> >> >> >> >> >> The IP NOC with the assistance of the Operations Engineering team >> confirmed a routing issue to be preventing BGP sessions from establishing >> correctly. A configuration adjustment was deployed at a high level, and >> sessions began to re-establish with stability. As the change propagates >> through the affected devices, service affecting alarms continue to clear. >> >> >> >> Due to the nature of this outage, it may be necessary to reset your >> services locally at your equipment, or manually reset your BGP session. If >> after that action has been performed a service issue prevails, please >> contact the CenturyLink Repair Center for troubleshooting assistance. >> >> >> >> Next update by: 2020-08-30 15:50 GMT >> >> >> >> This notification is sent from an unmonitored email account. Please >> click the link below to reply: >> >> >> >> >> mailto:[email protected]?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A >> <[email protected]?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A> >> >> >> >> Notes History: >> >> >> >> 08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the >> Operations Engineering team confirmed a routing issue to be preventing BGP >> sessions from establishing correctly. A configuration adjustment was >> deployed at a high level, and sessions began to re-establish with >> stability. As the change propagates through the affected devices, service >> affecting alarms continue to clear. >> >> >> >> Due to the nature of this outage, it may be necessary to reset your >> services locally at your equipment, or manually reset your BGP session. If >> after that action has been performed a service issue prevails, please >> contact the CenturyLink Repair Center for troubleshooting assistance. >> >> >> >> 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway >> Protocol (BGP) issues causing service impacts across multiple markets. >> Cooperative escalated investigation efforts with CenturyLink leadership >> are underway to isolate and troubleshoot the issue and expedite service >> restoral. >> >> >> >> 08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated >> investigation and troubleshooting efforts remain ongoing this time. >> >> >> >> 08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated >> investigations to isolate and troubleshoot the fault at this time. >> >> >> >> 08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink >> identified a market wide. As this network fault is impacting multiple >> clients, the event has increased visibility with CenturyLink leadership. As >> such, client trouble tickets associated to this fault have been >> automatically escalated to higher priority. >> >> >> >> The NOC is engaged and investigating in order to isolate the cause. >> Please be advised that updates for this event will be relayed at a minimum >> of hourly unless otherwise noted. The information conveyed hereafter is >> associated to live troubleshooting effort and as the discovery process >> evolves through to service resolution, ticket closure, or post incident >> review, details may evolve. >> >> >> >> >> >> *From:* Michael McDonald <[email protected]> >> *Sent:* Sunday, August 30, 2020 9:56 AM >> *To:* Dylan Ebner <[email protected]> >> *Cc:* Jay Ashworth <[email protected]>; [email protected] >> *Subject:* Re: [outages] 3556 problems >> >> >> >> Not on that ticket, so thanks a ton for the update! Execs are asking >> questions about what's going on and it's nice to give them feedback! >> >> >> >> On Sun, Aug 30, 2020 at 10:52 AM Dylan Ebner via Outages < >> [email protected]> wrote: >> >> For those who aren't on the master CL ticket, here is the latest from CL. >> Ticket number: 19545403 >> >> >> >> >> >> >> 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway >> Protocol (BGP) issues causing service impacts across multiple markets. >> Cooperative escalated investigation efforts with CenturyLink leadership are >> underway to isolate and troubleshoot the issue and expedite service >> restoral. >> >> >> >> >> >> >> >> -------- Original message -------- >> >> From: Jay Ashworth via Outages <[email protected]> >> >> Date: 8/30/20 9:36 AM (GMT-06:00) >> >> To: [email protected] >> >> Subject: [outages] 3556 problems >> >> >> >> Okay, it's pretty clear that they're having trouble. :-) Let's limit it >> now to actual tickets and reports of clearance. >> -- >> Sent from my Android device with K-9 Mail. Please excuse my brevity. >> >> _______________________________________________ >> Outages mailing list >> [email protected] >> https://puck.nether.net/mailman/listinfo/outages >> >> _______________________________________________ >> Outages mailing list >> [email protected] >> https://puck.nether.net/mailman/listinfo/outages >> > _______________________________________________ > Outages mailing list > [email protected] > https://puck.nether.net/mailman/listinfo/outages > -- Sincerely, Jason W Kuehl Cell 920-419-8983 [email protected]
_______________________________________________ Outages mailing list [email protected] https://puck.nether.net/mailman/listinfo/outages
