I don't think this was TODAYS outage. It was from over a day ago.
On Thu, Feb 13, 2014 at 12:21 PM, Luke Rockwell < [email protected]> wrote: > Offical L3 log > > Network Event Detail > Network Event Summary: A suspected Denial of Service (DoS) attack on > several links was impacting Voice and IP services in multiple markets. Event > Ticket ID: > 7531588Market Area Affected: Multiple Markets Ticket Create Date: 2/12/14 > 1:26:49 AM GMT Impacted For: *2 hours 32 minutes* Event Status:Restored > Resolve Date: 2/13/14 5:02:13 PM GMT Time Since Last Update: *1 day 12 > hours 39 minutes* *There are currently no reported impacts to your > services caused by this network event. If you are experiencing a service > disruption, please report the issue to Level 3 by submitting a ticket via > the MyLevel3 Portal or calling 877-4LEVEL3.* > Recent Notes > DateNotes 2/12/14 4:41:37 AM GMTThe IP NOC and Operations Engineering > have confirmed that the issue has resolved. It has been advised that > monitoring will continue to ensure the links remain stable. Should further > issues be experienced, please contact the Technical Service Center at > 877-4LEVEL3. 2/12/14 4:01:32 AM GMT The IP NOC has advised that the issue > is believed to have been mitigated. Operations Engineers are currently > working to confirm that no further impact is being experienced at this time. > 2/12/14 > 2:57:57 AM GMTOperations Engineers continue investigating the issue with > the IP NOC and Tier III Technical Support. 2/12/14 2:25:58 AM GMTOperations > Engineering has advised that Tier III Technical Support is being > engaged at this time to assist with investigations. 2/12/14 2:03:18 AM GMTThe > IP NOC continues working diligently with Operations Engineering to identify > the source of the attack. 2/12/14 1:41:56 AM GMTIt has been confirmed > that IP services are also being impacted. The IP NOC has advised that that > the issue is suspected to be the result of a Denial of Service (DoS) attack > on several links in the United States. Operations Engineering are being > engaged at this time to assist with investigations. 2/12/14 1:30:01 AM GMTThe > Voice NOC has reported that the Border Gateway Protocol bounced on an IP > link, impacting voice services in multiple markets. Investigations are > currently ongoing to determine the full impact of the issue. > > Here is another event I dont know if it is related > Network Event Detail > Network Event Summary: Packet loss across multiple markets in North > America was impacting IP services. Event Ticket ID: 7524423Market Area > Affected: Multiple Markets Ticket Create Date: 2/10/14 5:06:48 PM GMT > Impacted > For: *17 hours 8 minutes* Event Status: Restored Resolve Date: 2/13/14 > 5:05:39 PM GMT Time Since Last Update: *2 days 13 hours 45 minutes* *There > are currently no reported impacts to your services caused by this network > event. If you are experiencing a service disruption, please report the > issue to Level 3 by submitting a ticket via the MyLevel3 Portal or calling > 877-4LEVEL3.* > Recent Notes > DateNotes 2/11/14 3:34:25 AM GMTThe Operations Engineering team reported > that the script that was developed was successful in resolving the packet > loss issue. It has been informed that the issue has resolved and services > have restored. If additional issues are experienced, please contact the > Level 3 Technical Service Center. 2/11/14 2:12:21 AM GMT The Operations > Engineering team advised that the scripts are currently running at this > time in an effort to restore services. An estimated restoral time remains > unknown and the next update will be sent as new information is received. > 2/10/14 > 11:30:05 PM GMTAt this time, the Technical Service Center is working with > the IP NOC to develop a script that can push the filter to a larger group > of customers to expedite restoration. After the script is run, teams will > validate deployment success to ensure stability. The Operations Engineering > team reported that approximately fifty percent of the impact has been > mitigated and efforts continue to increase that percentage as quickly as > possible. The next update will be provided at approximately 01:00 GMT or as > new information is made available. 2/10/14 11:10:30 PM GMT The Operation > Engineering team is diligently working to restore impacted services. An > estimated restoration time is unavailable, but updates will be provided as > soon as more information is made available. 2/10/14 9:00:58 PM GMTThe > Operation Engineering team has advised that they are investigating a mass > deployment to expedite service restoral. Unfortunately, an estimated > restoral time is not available as it may vary for impacted customers. 2/10/14 > 7:51:53 PM GMT The Operations Engineering team continues to work to > isolate a solution for the packet loss. The investigating group is unable > to provide an ETTR for this case at this time. 2/10/14 7:10:14 PM GMTThe > Operations Engineering team continues to troubleshoot the issue in an > effort to restore services. 2/10/14 5:28:31 PM GMT The IP Technical > Service Center reported multiple customers that are experiencing an outage > to IP services due to heavy packet loss across multiple markets in North > America. Numerous managed devices are unresponsive at this time, and the > Network Operations Engineering teams are currently working to mitigate the > packet loss and restore stability to individual customers. An estimated > restoral time is not available as it may vary for impacted customers. > > On Thu, Feb 13, 2014 at 4:59 AM, Chris Stone <[email protected]> wrote: > >> Got that from L3 as well. Looks like my sessions have been up and >> stable since 02:23:32 MST. >> >> >> >> On 02/13/2014 03:41 AM, Mitch Patterson wrote: >> >> Final notes from them. Any one still seeing issues? My direct level 3 >> circuits look good on a spot check. Still having issues with a ISP that has >> Level 3 as their only major upstream >> >> *** CASCADED EXTERNAL NOTES 13-Feb-2014 10:32:49 GMT From CASE: >> 7538219 - Event A packet loss issue impacted IP Services across multiple >> markets in the North America. The Technical Service Center worked with the >> IP NOC to develop a configuration change to a larger group of customers to >> expedite restoration. After the configuration change was run, teams >> validated the deployment success to ensure stability. The Operations >> Engineering team reported that the configuration change that was developed >> was successful in resolving the packet loss issue. It was communicated that >> the issue has resolved and services have restored. If additional issues are >> experienced, please contact the Level 3 Technical Service Center. >> >> >> On Thu, Feb 13, 2014 at 4:10 AM, Mitch Patterson >> <[email protected]>wrote: >> >>> Latest update >>> >>> *** CASCADED EXTERNAL NOTES 13-Feb-2014 09:07:11 GMT From CASE: >>> 7538219 - Event Security Ops attempted to apply filters on the network >>> routers to reject the IP traffic of the DDoS attacks. However, the filters >>> were unsuccessful due to the multiple sources of the attack. Security is >>> attempting another plan to limit the effect of the attack on the targeted >>> destination of the DDoS attack. >>> >>> >>> On Thu, Feb 13, 2014 at 3:28 AM, Mitch Patterson <[email protected] >>> > wrote: >>> >>>> Same reply they gave me. >>>> On Feb 13, 2014 3:27 AM, "Petter Bruland" < >>>> [email protected]> wrote: >>>> >>>>> We saw similar issues with Level3 at Switch SuperNAP in Las Vegas. >>>>> >>>>> Last we heard: >>>>> Level-3 IP NOC Team and Security Ops are applying filters on affected >>>>> routers to re-establish routing sessions and restore normal operations but >>>>> they have not provided any ETR as of now. >>>>> >>>>> -P >>>>> >>>>> >>>>> *Petter Bruland *| Network Engineer >>>>> Allegiant Travel Company >>>>> 8360 S. Durango Drive, Las Vegas, NV 89113 >>>>> Phone: (702) 874-3332 <%28702%29%20874-3332> | Cell: (702) >>>>> 286-6549<%28702%29%20286-6549> >>>>> [email protected] >>>>> http://www.allegiantair.com >>>>> ------------------------------ >>>>> *From:* Outages [[email protected]] on behalf of Mitch >>>>> Patterson [[email protected]] >>>>> *Sent:* Wednesday, February 12, 2014 11:38 PM >>>>> *To:* [email protected] >>>>> *Subject:* [outages] Level 3 issues....again >>>>> >>>>> Massive Level 3 issues again. Got this from them: >>>>> >>>>> >>>>> >>>>> The IP NOC has detected Denial of Service (DoS) attacks on various >>>>> network routers in Houston, TX. The IP NOC is analyzing the routing >>>>> protocol session failures to determine the next course of action. >>>>> >>>>> >>>>> >>>>> IP NOC and Security Ops are working on resolving the BGP sessions >>>>> failures on network routers affected by the DDoS attack. >>>>> >>>> >>> >> >> >> _______________________________________________ >> Outages mailing >> [email protected]https://puck.nether.net/mailman/listinfo/outages >> >> >> >> _______________________________________________ >> Outages mailing list >> [email protected] >> https://puck.nether.net/mailman/listinfo/outages >> >> > > _______________________________________________ > Outages mailing list > [email protected] > https://puck.nether.net/mailman/listinfo/outages > >
_______________________________________________ Outages mailing list [email protected] https://puck.nether.net/mailman/listinfo/outages
