Most public safety agencies in Massachusetts posted on Facebook and Twitter between 6 AM and 8 AM EST about wireless 911 outages caused by CenturyLink. It was affecting wireless routing, so some calls were being misrouted to other towns, or even other states; some callers were getting reorder tones or other error messages. The problem was reported to be resolved prior to 8 AM EST.
Seeing reports from PSAP's that the issue started again in the last 15 minutes. This is wireless only. Landline 911 calls were / are not affected. On Fri, Dec 28, 2018 at 9:32 AM Mel Beckman via Outages <[email protected]> wrote: > Don't be a fear monger. I searched Facebook and found *three* posts about > 911 outages, all of them simply reposts of local news stories. > > Facebook is not "blowing up". > > Sheesh! > > [image: image1.PNG] > > -mel via cell > > -mel via cell > > On Dec 28, 2018, at 5:23 AM, Josh Luthman via Outages <[email protected]> > wrote: > > Facebook is blowing up with 911 outages and claiming it's CenturyLink to > blame. > > Josh Luthman > Office: 937-552-2340 > Direct: 937-552-2343 > 1100 Wayne St > Suite 1337 > Troy, OH 45373 > > On Fri, Dec 28, 2018, 2:09 AM Erik Sundberg via Outages < > [email protected] wrote: > >> Maybe light at the end of the tunnel... >> >> >> >> latest we got from Centurylink was at 10:45pm CST. >> >> Our engineers and technicians have identified the network element that >> has affected our customer services. Services are restoring, and the current >> estimated time for full recovery is four hours. >> >> CenturyLink will be conducting an extensive post-incident investigation >> and root cause analysis to provide follow-up information to our customers >> impacted by this event. We will provide one more Sales Alert from this >> address once services are fully restored. >> >> >> >> >> >> >> >> *From:* Rich Breton <[email protected]> >> *Sent:* Thursday, December 27, 2018 2:59 PM >> *To:* Christopher Conley <[email protected]> >> *Cc:* Erik Sundberg <[email protected]>; [email protected] >> *Subject:* Re: [outages] CenturyLink Outages this morning >> >> >> >> I understand that the transport waves went out first, and then they try >> to reroute to something in Arizona that begin introducing all of the >> broadcast traffic. How legit that is I’m not sure >> >> Sent from my iPhone >> >> >> On Dec 27, 2018, at 12:58 PM, Christopher Conley via Outages < >> [email protected]> wrote: >> >> > Finally received something useful from centurylink… I have more >> questions now like how broadcast traffic effect 4x Transport waves. >> >> >> >> Ineptitude. >> >> >> >> >> >> Christopher Conley >> >> Systems Administrator | Fors Marsh Group >> >> Arlington, VA >> >> >> >> *From:* Outages <[email protected]> *On Behalf Of *Erik >> Sundberg via Outages >> *Sent:* Thursday, December 27, 2018 2:46 PM >> *To:* [email protected] >> *Subject:* Re: [outages] CenturyLink Outages this morning >> >> >> >> Finally received something useful from centurylink… I have more questions >> now like how broadcast traffic effect 4x Transport waves. >> >> >> >> >> >> Update: >> >> On December 27, 2018 at 02:40 GMT, CenturyLink identified a service >> impact in New Orleans, LA. The NOC is engaged and investigating in order to >> isolate the cause. Field Operations were engaged and dispatched for >> additional investigations. Tier IV Equipment Vendor Support was later >> engaged. During cooperative troubleshooting a device in San Antonio, TX was >> isolated from the network as it was seeming to broadcast traffic consuming >> capacity, which seemed to alleviate some impact. Investigations remained >> ongoing. Following the isolation of the San Antonio, TX device >> troubleshooting efforts focused on additional sites that teams were >> remotely unable to troubleshoot. Field Operations were dispatched to sites >> in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL. Tier IV >> Equipment Vendor Support continued to investigate the equipment logs to >> further assist with isolation. Once visibility was restored to the site in >> Kansas City, MO a filter was applied to the equipment to further alleviate >> the impact observed. All of the necessary troubleshooting teams in >> cooperation with Tier IV Equipment Vendor Support are working to restore >> remote visibility to the remaining sites at this time. We understand how >> important these services are to our clients and the issue has been >> escalated to the highest levels within CenturyLink Service Assurance >> Leadership. >> >> >> >> >> >> *From:* Outages <[email protected]> *On Behalf Of *Bruce >> Freshwater via Outages >> *Sent:* Thursday, December 27, 2018 1:24 PM >> *To:* [email protected] >> *Subject:* Re: [outages] CenturyLink Outages this morning >> >> >> >> CenturyLink finally acknowledged: >> >> Great Lakes Region CenturyLink Partners/Agents: >> >> >> >> FYI. Our network is experiencing a disruption affecting customer >> services. Customers are currently unable to generate tickets through the >> Control Center portal. >> >> >> >> We know how important these services are to our customers, and we are >> working to restore services as quickly as possible. >> >> >> >> Our technical teams are working to resolve the issue, and we will provide >> updates as the situation progresses. >> >> >> >> >> >> -bruce >> >> >> >> >> >> On Thu, Dec 27, 2018 at 10:16 AM -0500, "Erik Sundberg via Outages" < >> [email protected]> wrote: >> >> So it's a bad day for centurylink. >> >> >> >> The lady who answered the phone for there NOC said that we are having a >> nation wide outage and we are unable to open tickets at this time. Please >> call be in 60-90 mins. >> >> >> >> She had no info on what was going on >> >> >> >> >> ------------------------------ >> >> *From:* Michael Bolton <[email protected]> >> *Sent:* Thursday, December 27, 2018 8:55:03 AM >> *To:* Erik Sundberg; [email protected] >> *Subject:* RE: CenturyLink Outages this morning >> >> >> >> Seeing the same in NC. Circuits are up but the routing between POPs has >> changed and we are seeing higher than normal latency. >> >> >> >> VoIP portal is down also. Allows you to login but you cannot make any >> changes or view any details. >> >> >> >> >> >> >> >> *From:* Outages <[email protected]> *On Behalf Of *Erik >> Sundberg via Outages >> *Sent:* Thursday, December 27, 2018 9:45 AM >> *To:* [email protected] >> *Subject:* [outages] CenturyLink Outages this morning >> >> >> >> >> >> *CAUTION:* This email originated from outside of the organization. Do >> not click links or open attachments unless you recognize the sender and >> know the content is safe. >> >> We are seeing a lot of centurylink circuits and services down this >> morning. >> >> >> >> Centurylink Control Center portal comes up but auth failing unable to >> login >> >> >> >> Waves down or bouncing >> >> Denver - Seattle (Bouncing) >> >> Denver - Chicago (Bouncing) >> >> New York - Los Angeles (Down) >> >> Chicago - Atlanta (Down) >> >> >> ENNI Down in Denver (Down) >> >> >> >> Still tying to open tickets with them there portal is not working and >> stuck on hold right now. >> >> >> >> >> ------------------------------ >> >> >> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, >> files or previous e-mail messages attached to it may contain confidential >> information that is legally privileged. If you are not the intended >> recipient, or a person responsible for delivering it to the intended >> recipient, you are hereby notified that any disclosure, copying, >> distribution or use of any of the information contained in or attached to >> this transmission is STRICTLY PROHIBITED. If you have received this >> transmission in error please notify the sender immediately by replying to >> this e-mail. You must destroy the original transmission and its attachments >> without reading or saving in any manner. Thank you. >> >> IMPORTANT NOTICE: This e-mail message is intended to be received only by >> persons entitled to receive the confidential information it may contain. >> E-mail messages to clients of Holmes Security Systems may contain >> information that is confidential and legally privileged. Please do not >> read, copy, forward, or store this message unless you are an intended >> recipient of it. If you have received this message in error, please forward >> it to the sender and delete it completely from your computer system. >> >> >> ------------------------------ >> >> >> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, >> files or previous e-mail messages attached to it may contain confidential >> information that is legally privileged. If you are not the intended >> recipient, or a person responsible for delivering it to the intended >> recipient, you are hereby notified that any disclosure, copying, >> distribution or use of any of the information contained in or attached to >> this transmission is STRICTLY PROHIBITED. If you have received this >> transmission in error please notify the sender immediately by replying to >> this e-mail. You must destroy the original transmission and its attachments >> without reading or saving in any manner. Thank you. >> >> >> ------------------------------ >> >> >> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, >> files or previous e-mail messages attached to it may contain confidential >> information that is legally privileged. If you are not the intended >> recipient, or a person responsible for delivering it to the intended >> recipient, you are hereby notified that any disclosure, copying, >> distribution or use of any of the information contained in or attached to >> this transmission is STRICTLY PROHIBITED. If you have received this >> transmission in error please notify the sender immediately by replying to >> this e-mail. You must destroy the original transmission and its attachments >> without reading or saving in any manner. Thank you. >> >> _______________________________________________ >> Outages mailing list >> [email protected] >> https://puck.nether.net/mailman/listinfo/outages >> >> >> ------------------------------ >> >> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, >> files or previous e-mail messages attached to it may contain confidential >> information that is legally privileged. If you are not the intended >> recipient, or a person responsible for delivering it to the intended >> recipient, you are hereby notified that any disclosure, copying, >> distribution or use of any of the information contained in or attached to >> this transmission is STRICTLY PROHIBITED. If you have received this >> transmission in error please notify the sender immediately by replying to >> this e-mail. You must destroy the original transmission and its attachments >> without reading or saving in any manner. Thank you. >> _______________________________________________ >> Outages mailing list >> [email protected] >> https://puck.nether.net/mailman/listinfo/outages >> > _______________________________________________ > Outages mailing list > [email protected] > https://puck.nether.net/mailman/listinfo/outages > > _______________________________________________ > Outages mailing list > [email protected] > https://puck.nether.net/mailman/listinfo/outages >
_______________________________________________ Outages mailing list [email protected] https://puck.nether.net/mailman/listinfo/outages
