Facebook is blowing up with 911 outages and claiming it's CenturyLink to blame.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Fri, Dec 28, 2018, 2:09 AM Erik Sundberg via Outages <[email protected] wrote: > Maybe light at the end of the tunnel... > > > > latest we got from Centurylink was at 10:45pm CST. > > Our engineers and technicians have identified the network element that has > affected our customer services. Services are restoring, and the current > estimated time for full recovery is four hours. > > CenturyLink will be conducting an extensive post-incident investigation > and root cause analysis to provide follow-up information to our customers > impacted by this event. We will provide one more Sales Alert from this > address once services are fully restored. > > > > > > > > *From:* Rich Breton <[email protected]> > *Sent:* Thursday, December 27, 2018 2:59 PM > *To:* Christopher Conley <[email protected]> > *Cc:* Erik Sundberg <[email protected]>; [email protected] > *Subject:* Re: [outages] CenturyLink Outages this morning > > > > I understand that the transport waves went out first, and then they try to > reroute to something in Arizona that begin introducing all of the broadcast > traffic. How legit that is I’m not sure > > Sent from my iPhone > > > On Dec 27, 2018, at 12:58 PM, Christopher Conley via Outages < > [email protected]> wrote: > > > Finally received something useful from centurylink… I have more > questions now like how broadcast traffic effect 4x Transport waves. > > > > Ineptitude. > > > > > > Christopher Conley > > Systems Administrator | Fors Marsh Group > > Arlington, VA > > > > *From:* Outages <[email protected]> *On Behalf Of *Erik > Sundberg via Outages > *Sent:* Thursday, December 27, 2018 2:46 PM > *To:* [email protected] > *Subject:* Re: [outages] CenturyLink Outages this morning > > > > Finally received something useful from centurylink… I have more questions > now like how broadcast traffic effect 4x Transport waves. > > > > > > Update: > > On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact > in New Orleans, LA. The NOC is engaged and investigating in order to > isolate the cause. Field Operations were engaged and dispatched for > additional investigations. Tier IV Equipment Vendor Support was later > engaged. During cooperative troubleshooting a device in San Antonio, TX was > isolated from the network as it was seeming to broadcast traffic consuming > capacity, which seemed to alleviate some impact. Investigations remained > ongoing. Following the isolation of the San Antonio, TX device > troubleshooting efforts focused on additional sites that teams were > remotely unable to troubleshoot. Field Operations were dispatched to sites > in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL. Tier IV > Equipment Vendor Support continued to investigate the equipment logs to > further assist with isolation. Once visibility was restored to the site in > Kansas City, MO a filter was applied to the equipment to further alleviate > the impact observed. All of the necessary troubleshooting teams in > cooperation with Tier IV Equipment Vendor Support are working to restore > remote visibility to the remaining sites at this time. We understand how > important these services are to our clients and the issue has been > escalated to the highest levels within CenturyLink Service Assurance > Leadership. > > > > > > *From:* Outages <[email protected]> *On Behalf Of *Bruce > Freshwater via Outages > *Sent:* Thursday, December 27, 2018 1:24 PM > *To:* [email protected] > *Subject:* Re: [outages] CenturyLink Outages this morning > > > > CenturyLink finally acknowledged: > > Great Lakes Region CenturyLink Partners/Agents: > > > > FYI. Our network is experiencing a disruption affecting customer services. > Customers are currently unable to generate tickets through the Control > Center portal. > > > > We know how important these services are to our customers, and we are > working to restore services as quickly as possible. > > > > Our technical teams are working to resolve the issue, and we will provide > updates as the situation progresses. > > > > > > -bruce > > > > > > On Thu, Dec 27, 2018 at 10:16 AM -0500, "Erik Sundberg via Outages" < > [email protected]> wrote: > > So it's a bad day for centurylink. > > > > The lady who answered the phone for there NOC said that we are having a > nation wide outage and we are unable to open tickets at this time. Please > call be in 60-90 mins. > > > > She had no info on what was going on > > > > > ------------------------------ > > *From:* Michael Bolton <[email protected]> > *Sent:* Thursday, December 27, 2018 8:55:03 AM > *To:* Erik Sundberg; [email protected] > *Subject:* RE: CenturyLink Outages this morning > > > > Seeing the same in NC. Circuits are up but the routing between POPs has > changed and we are seeing higher than normal latency. > > > > VoIP portal is down also. Allows you to login but you cannot make any > changes or view any details. > > > > > > > > *From:* Outages <[email protected]> *On Behalf Of *Erik > Sundberg via Outages > *Sent:* Thursday, December 27, 2018 9:45 AM > *To:* [email protected] > *Subject:* [outages] CenturyLink Outages this morning > > > > > > *CAUTION:* This email originated from outside of the organization. Do not > click links or open attachments unless you recognize the sender and know > the content is safe. > > We are seeing a lot of centurylink circuits and services down this morning. > > > > Centurylink Control Center portal comes up but auth failing unable to login > > > > Waves down or bouncing > > Denver - Seattle (Bouncing) > > Denver - Chicago (Bouncing) > > New York - Los Angeles (Down) > > Chicago - Atlanta (Down) > > > ENNI Down in Denver (Down) > > > > Still tying to open tickets with them there portal is not working and > stuck on hold right now. > > > > > ------------------------------ > > > CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files > or previous e-mail messages attached to it may contain confidential > information that is legally privileged. If you are not the intended > recipient, or a person responsible for delivering it to the intended > recipient, you are hereby notified that any disclosure, copying, > distribution or use of any of the information contained in or attached to > this transmission is STRICTLY PROHIBITED. If you have received this > transmission in error please notify the sender immediately by replying to > this e-mail. You must destroy the original transmission and its attachments > without reading or saving in any manner. Thank you. > > IMPORTANT NOTICE: This e-mail message is intended to be received only by > persons entitled to receive the confidential information it may contain. > E-mail messages to clients of Holmes Security Systems may contain > information that is confidential and legally privileged. Please do not > read, copy, forward, or store this message unless you are an intended > recipient of it. If you have received this message in error, please forward > it to the sender and delete it completely from your computer system. > > > ------------------------------ > > > CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files > or previous e-mail messages attached to it may contain confidential > information that is legally privileged. If you are not the intended > recipient, or a person responsible for delivering it to the intended > recipient, you are hereby notified that any disclosure, copying, > distribution or use of any of the information contained in or attached to > this transmission is STRICTLY PROHIBITED. If you have received this > transmission in error please notify the sender immediately by replying to > this e-mail. You must destroy the original transmission and its attachments > without reading or saving in any manner. Thank you. > > > ------------------------------ > > > CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files > or previous e-mail messages attached to it may contain confidential > information that is legally privileged. If you are not the intended > recipient, or a person responsible for delivering it to the intended > recipient, you are hereby notified that any disclosure, copying, > distribution or use of any of the information contained in or attached to > this transmission is STRICTLY PROHIBITED. If you have received this > transmission in error please notify the sender immediately by replying to > this e-mail. You must destroy the original transmission and its attachments > without reading or saving in any manner. Thank you. > > _______________________________________________ > Outages mailing list > [email protected] > https://puck.nether.net/mailman/listinfo/outages > > > ------------------------------ > > CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files > or previous e-mail messages attached to it may contain confidential > information that is legally privileged. If you are not the intended > recipient, or a person responsible for delivering it to the intended > recipient, you are hereby notified that any disclosure, copying, > distribution or use of any of the information contained in or attached to > this transmission is STRICTLY PROHIBITED. If you have received this > transmission in error please notify the sender immediately by replying to > this e-mail. You must destroy the original transmission and its attachments > without reading or saving in any manner. Thank you. > _______________________________________________ > Outages mailing list > [email protected] > https://puck.nether.net/mailman/listinfo/outages >
_______________________________________________ Outages mailing list [email protected] https://puck.nether.net/mailman/listinfo/outages
