Re: Level(3) ex-twtelecom midwest packet loss (4323)
Blake, There's no call to be blatantly offensive like that. -mel On Aug 28, 2015, at 8:35 AM, Blake Dunlap iki...@gmail.com wrote: I'll just leave this here https://honestnetworker.wordpress.com/2013/11/04/the-true-meaning-behind-most-rfos/ On Fri, Aug 28, 2015 at 8:26 AM, Jason Canady ja...@unlimitednet.us wrote: Mike, I would take it to mean someone screwed something up and they don't want to admit to it. :-) That's just a guess. -- Jason Canady Unlimited Net, LLC Responsive, Reliable, Secure www.unlimitednet.us ja...@unlimitednet.us twitter: @unlimitednet On 8/28/15 12:00 AM, Mike Hammett wrote: 08/28/2015 3:08 AM GMT Event Conclusion Summary Start: August 27, 2015 13:20 GMT Stop: August 28, 2015 00:00 GMT Root Cause: A protocol issue impacted IP services in multiple markets. Fix Action: Adjustments were made to clear the errors. Summary: The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any additional impact be experienced, please contact the Level 3 Technical Service Center. What the hell is a protocol issue? I'm not an idiot, you can tell me specifically what happened... - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com - Original Message - From: Ryan Gelobter rya...@atwgpc.net To: Mel Beckman m...@beckman.org Cc: nanog@nanog.org nanog@nanog.org Sent: Thursday, August 27, 2015 3:14:59 PM Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323) If you have access to the Level3 portal you should see ticket #9639047 under Network Events now. Event Summary:IP Network Event ~ Multiple Markets 08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time. 08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue. 08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. 08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist. 08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
Re: Level(3) ex-twtelecom midwest packet loss (4323)
Mike, I would take it to mean someone screwed something up and they don't want to admit to it. :-) That's just a guess. -- Jason Canady Unlimited Net, LLC Responsive, Reliable, Secure www.unlimitednet.us ja...@unlimitednet.us twitter: @unlimitednet On 8/28/15 12:00 AM, Mike Hammett wrote: 08/28/2015 3:08 AM GMT Event Conclusion Summary Start: August 27, 2015 13:20 GMT Stop: August 28, 2015 00:00 GMT Root Cause: A protocol issue impacted IP services in multiple markets. Fix Action: Adjustments were made to clear the errors. Summary: The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any additional impact be experienced, please contact the Level 3 Technical Service Center. What the hell is a protocol issue? I'm not an idiot, you can tell me specifically what happened... - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com - Original Message - From: Ryan Gelobter rya...@atwgpc.net To: Mel Beckman m...@beckman.org Cc: nanog@nanog.org nanog@nanog.org Sent: Thursday, August 27, 2015 3:14:59 PM Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323) If you have access to the Level3 portal you should see ticket #9639047 under Network Events now. Event Summary:IP Network Event ~ Multiple Markets 08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time. 08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue. 08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. 08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist. 08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
Re: Level(3) ex-twtelecom midwest packet loss (4323)
I'll just leave this here https://honestnetworker.wordpress.com/2013/11/04/the-true-meaning-behind-most-rfos/ On Fri, Aug 28, 2015 at 8:26 AM, Jason Canady ja...@unlimitednet.us wrote: Mike, I would take it to mean someone screwed something up and they don't want to admit to it. :-) That's just a guess. -- Jason Canady Unlimited Net, LLC Responsive, Reliable, Secure www.unlimitednet.us ja...@unlimitednet.us twitter: @unlimitednet On 8/28/15 12:00 AM, Mike Hammett wrote: 08/28/2015 3:08 AM GMT Event Conclusion Summary Start: August 27, 2015 13:20 GMT Stop: August 28, 2015 00:00 GMT Root Cause: A protocol issue impacted IP services in multiple markets. Fix Action: Adjustments were made to clear the errors. Summary: The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any additional impact be experienced, please contact the Level 3 Technical Service Center. What the hell is a protocol issue? I'm not an idiot, you can tell me specifically what happened... - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com - Original Message - From: Ryan Gelobter rya...@atwgpc.net To: Mel Beckman m...@beckman.org Cc: nanog@nanog.org nanog@nanog.org Sent: Thursday, August 27, 2015 3:14:59 PM Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323) If you have access to the Level3 portal you should see ticket #9639047 under Network Events now. Event Summary:IP Network Event ~ Multiple Markets 08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time. 08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue. 08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. 08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist. 08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
Re: Level(3) ex-twtelecom midwest packet loss (4323)
Seemed reasonably accurate to me. [image: Phone.com] http://www.phone.com/James SinkSenior Network Engineer js...@phone.com(800) 997-9179 x506Try Phone.com free! http://www.phone.com/?_tracking_id=10132CONFIDENTIALITY NOTICE: This e-mail and any attachments are for the exclusive and confidential use of the intended recipient. If you received this in error, please do not read, distribute, or take action in reliance upon this message. Instead, please notify us immediately by return e-mail and promptly delete this message and its attachments from your computer system. On Fri, Aug 28, 2015 at 8:45 AM, Mel Beckman m...@beckman.org wrote: Blake, There's no call to be blatantly offensive like that. -mel On Aug 28, 2015, at 8:35 AM, Blake Dunlap iki...@gmail.com wrote: I'll just leave this here https://honestnetworker.wordpress.com/2013/11/04/the-true-meaning-behind-most-rfos/ On Fri, Aug 28, 2015 at 8:26 AM, Jason Canady ja...@unlimitednet.us wrote: Mike, I would take it to mean someone screwed something up and they don't want to admit to it. :-) That's just a guess. -- Jason Canady Unlimited Net, LLC Responsive, Reliable, Secure www.unlimitednet.us ja...@unlimitednet.us twitter: @unlimitednet On 8/28/15 12:00 AM, Mike Hammett wrote: 08/28/2015 3:08 AM GMT Event Conclusion Summary Start: August 27, 2015 13:20 GMT Stop: August 28, 2015 00:00 GMT Root Cause: A protocol issue impacted IP services in multiple markets. Fix Action: Adjustments were made to clear the errors. Summary: The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any additional impact be experienced, please contact the Level 3 Technical Service Center. What the hell is a protocol issue? I'm not an idiot, you can tell me specifically what happened... - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com - Original Message - From: Ryan Gelobter rya...@atwgpc.net To: Mel Beckman m...@beckman.org Cc: nanog@nanog.org nanog@nanog.org Sent: Thursday, August 27, 2015 3:14:59 PM Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323) If you have access to the Level3 portal you should see ticket #9639047 under Network Events now. Event Summary:IP Network Event ~ Multiple Markets 08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time. 08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue. 08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. 08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist. 08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
Re: Level(3) ex-twtelecom midwest packet loss (4323)
ill just leave this here :) https://www.youtube.com/watch?v=0ilMx7k7mso On Fri, Aug 28, 2015 at 11:33 AM, Blake Dunlap iki...@gmail.com wrote: I'll just leave this here https://honestnetworker.wordpress.com/2013/11/04/the-true-meaning-behind-most-rfos/ On Fri, Aug 28, 2015 at 8:26 AM, Jason Canady ja...@unlimitednet.us wrote: Mike, I would take it to mean someone screwed something up and they don't want to admit to it. :-) That's just a guess. -- Jason Canady Unlimited Net, LLC Responsive, Reliable, Secure www.unlimitednet.us ja...@unlimitednet.us twitter: @unlimitednet On 8/28/15 12:00 AM, Mike Hammett wrote: 08/28/2015 3:08 AM GMT Event Conclusion Summary Start: August 27, 2015 13:20 GMT Stop: August 28, 2015 00:00 GMT Root Cause: A protocol issue impacted IP services in multiple markets. Fix Action: Adjustments were made to clear the errors. Summary: The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any additional impact be experienced, please contact the Level 3 Technical Service Center. What the hell is a protocol issue? I'm not an idiot, you can tell me specifically what happened... - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com - Original Message - From: Ryan Gelobter rya...@atwgpc.net To: Mel Beckman m...@beckman.org Cc: nanog@nanog.org nanog@nanog.org Sent: Thursday, August 27, 2015 3:14:59 PM Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323) If you have access to the Level3 portal you should see ticket #9639047 under Network Events now. Event Summary:IP Network Event ~ Multiple Markets 08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time. 08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue. 08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. 08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist. 08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
Re: Level(3) ex-twtelecom midwest packet loss (4323)
This reminded me of this great clip https://www.youtube.com/watch?v=ckIMuvumYrg On Fri, Aug 28, 2015 at 12:00 AM, Mike Hammett na...@ics-il.net wrote: 08/28/2015 3:08 AM GMT Event Conclusion Summary Start: August 27, 2015 13:20 GMT Stop: August 28, 2015 00:00 GMT Root Cause: A protocol issue impacted IP services in multiple markets. Fix Action: Adjustments were made to clear the errors. Summary: The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any additional impact be experienced, please contact the Level 3 Technical Service Center. What the hell is a protocol issue? I'm not an idiot, you can tell me specifically what happened... - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com - Original Message - From: Ryan Gelobter rya...@atwgpc.net To: Mel Beckman m...@beckman.org Cc: nanog@nanog.org nanog@nanog.org Sent: Thursday, August 27, 2015 3:14:59 PM Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323) If you have access to the Level3 portal you should see ticket #9639047 under Network Events now. Event Summary:IP Network Event ~ Multiple Markets 08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time. 08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue. 08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. 08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist. 08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
Re: Experience on Wanguard for 'anti' DDOS solutions
We've tried their products off an on for the past 3-4 years. Here are my impressions: * UI stuck in 1999. Can't click zoom, drill down, etc. * Inflexible UI. Want a bandwidth graph with only egress or ingress? Too bad. * Inexpensive. I don't like that it's licensed yearly, but it's not too much money. * Inaccurate flow processing. Do you have iBGP peering sessions between border routers? WANGuard will struggle mightily to correctly classify the traffic as internal or external. * Yes, it runs out of memory quickly during a spoofed SYN flood with many sources. This is due to setting the Top generator to Full. If you just want to mitigate and not have any insight into network data, set this to Extended and you'll be fine. But if you want to use WANGuard/WANSight as a network intelligence tool as well, you need to set the generator to Full and it will fall over. * Doesn't process IPFIX flow data properly. There's an old thread on the j-nsp list about this. Basically their support claims Juniper is broken (which I don't doubt) but then refuses to work around the issue. None of our other flow processing tools have these problems. * Support is responsive at times and is always cranky. I brought them two bonafide bugs in their product that they refused to admit. It got to the point where I asked for my money back and I think someone in sales lit up their support team. I get the feeling that the support team is staffed with employees who really don't like their job or working with customers. A bad combination. * The TAP generators with Myricom cards work well. The docs say you can use SolarFlare for TAPs but they don't work at all. Again, they blame SolarFlare and say that the cards are too complicatedbut fail to update their documentation saying this. * Doesn't support any kind of layer 7 detection or filtering. It's all very rudimentary layer 3-4 stuff. Considering how easy it is to block layer 3/4 attacks on your own, their filtering clusters don't offer much value. * No real scale out solution on the detection side. It's basically scale up your server or use clunky tech like NFS to share out directories across managers. * Works well enough to get you a rough idea of what's going on. It's also decently cheap. We use it as one part of our attack detection toolset. We don't use it for on-site attack mitigation. I'd recommend it if you don't want to use flow data and only want to use it for intelligence on TAP ports. -richard On Mon, Aug 10, 2015 at 6:58 AM, Marcel Duregards marcel.durega...@yahoo.fr wrote: Dear Nogers, We are currently evaluating some DDOS detection/mitigation solutions. Do you have any inputs/experiences on Wanguard from Andrisoft, please ?https://www.andrisoft.com/software/wanguard Currently we are just interested on the packets/flows sensors with the console for detection and RTBH trigger. Maybe the packet filtering (for scrubbing) will come later. Best Regards,-Marcel Duregards
Weekly Routing Table Report
This is an automated weekly mailing describing the state of the Internet Routing Table as seen from APNIC's router in Japan. The posting is sent to APOPS, NANOG, AfNOG, AusNOG, SANOG, PacNOG, CaribNOG and the RIPE Routing Working Group. Daily listings are sent to bgp-st...@lists.apnic.net For historical data, please see http://thyme.rand.apnic.net. If you have any comments please contact Philip Smith pfsi...@gmail.com. Routing Table Report 04:00 +10GMT Sat 29 Aug, 2015 Report Website: http://thyme.rand.apnic.net Detailed Analysis: http://thyme.rand.apnic.net/current/ Analysis Summary BGP routing table entries examined: 556877 Prefixes after maximum aggregation (per Origin AS): 210209 Deaggregation factor: 2.65 Unique aggregates announced (without unneeded subnets): 272891 Total ASes present in the Internet Routing Table: 51281 Prefixes per ASN: 10.86 Origin-only ASes present in the Internet Routing Table: 36680 Origin ASes announcing only one prefix: 16140 Transit ASes present in the Internet Routing Table:6362 Transit-only ASes present in the Internet Routing Table:169 Average AS path length visible in the Internet Routing Table: 4.4 Max AS path length visible: 45 Max AS path prepend of ASN ( 55644) 41 Prefixes from unregistered ASNs in the Routing Table: 1164 Unregistered ASNs in the Routing Table: 426 Number of 32-bit ASNs allocated by the RIRs: 10780 Number of 32-bit ASNs visible in the Routing Table:8239 Prefixes from 32-bit ASNs in the Routing Table: 30881 Number of bogon 32-bit ASNs visible in the Routing Table:18 Special use prefixes present in the Routing Table:1 Prefixes being announced from unallocated address space:417 Number of addresses announced to Internet: 2792002752 Equivalent to 166 /8s, 106 /16s and 148 /24s Percentage of available address space announced: 75.4 Percentage of allocated address space announced: 75.4 Percentage of available address space allocated: 100.0 Percentage of address space in use by end-sites: 97.6 Total number of prefixes smaller than registry allocations: 185757 APNIC Region Analysis Summary - Prefixes being announced by APNIC Region ASes: 137357 Total APNIC prefixes after maximum aggregation: 39703 APNIC Deaggregation factor:3.46 Prefixes being announced from the APNIC address blocks: 144673 Unique aggregates announced from the APNIC address blocks:59491 APNIC Region origin ASes present in the Internet Routing Table:5079 APNIC Prefixes per ASN: 28.48 APNIC Region origin ASes announcing only one prefix: 1207 APNIC Region transit ASes present in the Internet Routing Table:893 Average APNIC Region AS path length visible:4.5 Max APNIC Region AS path length visible: 45 Number of APNIC region 32-bit ASNs visible in the Routing Table: 1600 Number of APNIC addresses announced to Internet: 751999104 Equivalent to 44 /8s, 210 /16s and 152 /24s Percentage of available APNIC address space announced: 87.9 APNIC AS Blocks4608-4864, 7467-7722, 9216-10239, 17408-18431 (pre-ERX allocations) 23552-24575, 37888-38911, 45056-46079, 55296-56319, 58368-59391, 63488-64098, 131072-135580 APNIC Address Blocks 1/8, 14/8, 27/8, 36/8, 39/8, 42/8, 43/8, 49/8, 58/8, 59/8, 60/8, 61/8, 101/8, 103/8, 106/8, 110/8, 111/8, 112/8, 113/8, 114/8, 115/8, 116/8, 117/8, 118/8, 119/8, 120/8, 121/8, 122/8, 123/8, 124/8, 125/8, 126/8, 133/8, 150/8, 153/8, 163/8, 171/8, 175/8, 180/8, 182/8, 183/8, 202/8, 203/8, 210/8, 211/8, 218/8, 219/8, 220/8, 221/8, 222/8, 223/8, ARIN Region Analysis Summary Prefixes being announced by ARIN Region ASes:180309 Total ARIN prefixes after maximum aggregation:88398 ARIN Deaggregation factor: 2.04 Prefixes being announced from the ARIN address blocks: 183221 Unique aggregates announced from the ARIN address blocks: 86088 ARIN Region origin ASes present in the Internet Routing Table:16593 ARIN Prefixes per
Re: Level(3) ex-twtelecom midwest packet loss (4323)
On 8/27/2015 11:00 PM, Mike Hammett wrote: 08/28/2015 3:08 AM GMT Event Conclusion Summary Start: August 27, 2015 13:20 GMT Stop: August 28, 2015 00:00 GMT Root Cause: A protocol issue impacted IP services in multiple markets. Fix Action: Adjustments were made to clear the errors. Summary: The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any additional impact be experienced, please contact the Level 3 Technical Service Center. What the hell is a protocol issue? I'm not an idiot, you can tell me specifically what happened... I was having major issues yesterday. It wasn't with all traffic, and its seemed to have cleared up now. I did do packet inspections, and I kept getting ethernet frame check errors from certain sources. vast majority of packets did not have this extra data appended to the tcp portion of packet, and I think that is likely the reason for the issues, since those packets are supposed to be dropped if there is a checksum error. I'm by no means an expert, just voicing what I was seeing. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com - Original Message - From: Ryan Gelobter rya...@atwgpc.net To: Mel Beckman m...@beckman.org Cc: nanog@nanog.org nanog@nanog.org Sent: Thursday, August 27, 2015 3:14:59 PM Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323) If you have access to the Level3 portal you should see ticket #9639047 under Network Events now. Event Summary:IP Network Event ~ Multiple Markets 08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time. 08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue. 08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. 08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist. 08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
The Cidr Report
This report has been generated at Fri Aug 28 21:14:41 2015 AEST.
The report analyses the BGP Routing Table of AS2.0 router
and generates a report on aggregation potential within the table.
Check http://www.cidr-report.org/2.0 for a current version of this report.
Recent Table History
Date PrefixesCIDR Agg
21-08-15565311 307557
22-08-15565397 307372
23-08-15564695 307460
24-08-15564815 308049
25-08-15565247 307192
26-08-15565168 306930
27-08-15565019 306047
28-08-15565765 306135
AS Summary
51555 Number of ASes in routing system
20455 Number of ASes announcing only one prefix
3344 Largest number of prefixes announced by an AS
AS10620: Telmex Colombia S.A.,CO
120888576 Largest address span announced by an AS (/32s)
AS4134 : CHINANET-BACKBONE No.31,Jin-rong Street,CN
Aggregation Summary
The algorithm used in this report proposes aggregation only
when there is a precise match using the AS path, so as
to preserve traffic transit policies. Aggregation is also
proposed across non-advertised address space ('holes').
--- 28Aug15 ---
ASnumNetsNow NetsAggr NetGain % Gain Description
Table 565908 306049 25985945.9% All ASes
AS22773 3168 169 299994.7% ASN-CXA-ALL-CCI-22773-RDC -
Cox Communications Inc.,US
AS6389 2695 69 262697.4% BELLSOUTH-NET-BLK -
BellSouth.net Inc.,US
AS17974 2703 80 262397.0% TELKOMNET-AS2-AP PT
Telekomunikasi Indonesia,ID
AS39891 2473 33 244098.7% ALJAWWALSTC-AS Saudi Telecom
Company JSC,SA
AS7545 2908 624 228478.5% TPG-INTERNET-AP TPG Telecom
Limited,AU
AS28573 2297 148 214993.6% NET Serviços de Comunicação
S.A.,BR
AS9394 2104 201 190390.4% CTTNET China TieTong
Telecommunications
Corporation,CN
AS4766 3010 1280 173057.5% KIXS-AS-KR Korea Telecom,KR
AS10620 3344 1619 172551.6% Telmex Colombia S.A.,CO
AS9808 1585 75 151095.3% CMNET-GD Guangdong Mobile
Communication Co.Ltd.,CN
AS6983 1736 245 149185.9% ITCDELTA - Earthlink, Inc.,US
AS4755 2038 556 148272.7% TATACOMM-AS TATA
Communications formerly VSNL
is Leading ISP,IN
AS20115 1877 406 147178.4% CHARTER-NET-HKY-NC - Charter
Communications,US
AS3356 2537 1217 132052.0% LEVEL3 - Level 3
Communications, Inc.,US
AS9498 1386 121 126591.3% BBIL-AP BHARTI Airtel Ltd.,IN
AS18566 2148 956 119255.5% MEGAPATH5-US - MegaPath
Corporation,US
AS4323 1594 411 118374.2% TWTC - tw telecom holdings,
inc.,US
AS6147 1439 278 116180.7% Telefonica del Peru S.A.A.,PE
AS6849 1208 74 113493.9% UKRTELNET JSC UKRTELECOM,UA
AS4788 1191 68 112394.3% TMNET-AS-AP TM Net, Internet
Service Provider,MY
AS7303 1556 522 103466.5% Telecom Argentina S.A.,AR
AS7552 1400 366 103473.9% VIETEL-AS-AP Viettel
Corporation,VN
AS22561 1377 343 103475.1% CENTURYLINK-LEGACY-LIGHTCORE -
CenturyTel Internet Holdings,
Inc.,US
AS4808 1545 512 103366.9% CHINA169-BJ CNCGROUP IP
network China169 Beijing
Province Network,CN
AS38197 1312 366 94672.1% SUNHK-DATA-AS-AP Sun Network
(Hong Kong) Limited,HK
AS26615 1098 153 94586.1% Tim Celular S.A.,BR
AS8151 1693 763 93054.9% Uninet S.A. de C.V.,MX
AS7738 996 77 91992.3% Telemar Norte Leste S.A.,BR
AS8402 885 21 86497.6% CORBINA-AS OJSC Vimpelcom,RU
AS38285 982 133 84986.5% M2TELECOMMUNICATIONS-AU M2
BGP Update Report
BGP Update Report Interval: 20-Aug-15 -to- 27-Aug-15 (7 days) Observation Point: BGP Peering with AS131072 TOP 20 Unstable Origin AS Rank ASNUpds % Upds/PfxAS-Name 1 - AS9829 285802 7.0% 221.9 -- BSNL-NIB National Internet Backbone,IN 2 - AS38197 166147 4.0% 131.5 -- SUNHK-DATA-AS-AP Sun Network (Hong Kong) Limited,HK 3 - AS22059 116439 2.8% 58219.5 -- -Reserved AS-,ZZ 4 - AS26210 80874 2.0% 569.5 -- AXS Bolivia S. A.,BO 5 - AS370957253 1.4%2120.5 -- NET-CITY-SA - City of San Antonio,US 6 - AS22773 52705 1.3% 566.7 -- ASN-CXA-ALL-CCI-22773-RDC - Cox Communications Inc.,US 7 - AS28024 46358 1.1% 29.5 -- Nuevatel PCS de Bolivia S.A.,BO 8 - AS131090 37063 0.9% 134.3 -- CAT-IDC-4BYTENET-AS-AP CAT TELECOM Public Company Ltd,CAT ,TH 9 - AS13188 36755 0.9% 47.4 -- BANKINFORM-AS CONTENT DELIVERY NETWORK LTD,UA 10 - AS800134076 0.8% 173.9 -- NET-ACCESS-CORP - Net Access Corporation,US 11 - AS840231499 0.8% 106.4 -- CORBINA-AS OJSC Vimpelcom,RU 12 - AS56636 29589 0.7% 29589.0 -- ASVEDARU VEDA Ltd.,RU 13 - AS30295 29540 0.7%9846.7 -- 2ICSYSTEMSINC - 2iC Systems Inc.,CA 14 - AS25563 29032 0.7%9677.3 -- WEBLAND-AS Webland AG,CH 15 - AS199367 23140 0.6% 23140.0 -- AS_GBP Globe Business Publishing Limited,GB 16 - AS13999 22212 0.5% 52.6 -- Mega Cable, S.A. de C.V.,MX 17 - AS150520201 0.5%2525.1 -- DNIC-AS-01505 - Headquarters, USAISC,US 18 - AS22047 19138 0.5% 63.0 -- VTR BANDA ANCHA S.A.,CL 19 - AS684919087 0.5% 15.8 -- UKRTELNET JSC UKRTELECOM,UA 20 - AS200671 17998 0.4% 17998.0 -- SKOK-JAWORZNO SKOK Jaworzno,PL TOP 20 Unstable Origin AS (Updates per announced prefix) Rank ASNUpds % Upds/PfxAS-Name 1 - AS22059 116439 2.8% 58219.5 -- -Reserved AS-,ZZ 2 - AS56636 29589 0.7% 29589.0 -- ASVEDARU VEDA Ltd.,RU 3 - AS199367 23140 0.6% 23140.0 -- AS_GBP Globe Business Publishing Limited,GB 4 - AS200671 17998 0.4% 17998.0 -- SKOK-JAWORZNO SKOK Jaworzno,PL 5 - AS28 14778 0.4% 14778.0 -- PIRUM-AS Pirum Systems Limited,GB 6 - AS30295 29540 0.7%9846.7 -- 2ICSYSTEMSINC - 2iC Systems Inc.,CA 7 - AS25563 29032 0.7%9677.3 -- WEBLAND-AS Webland AG,CH 8 - AS375909254 0.2%9254.0 -- BCA-ASN,AO 9 - AS404937840 0.2%7840.0 -- FACILITYSOURCEINC - FacilitySource,US 10 - AS3935887061 0.2%7061.0 -- MUBEA-FLO - Mubea,US 11 - AS31357 14599 0.4%4866.3 -- TOMICA-AS Tomsk Information and Consulting Agency,RU 12 - AS219734473 0.1%4473.0 -- XPONET - Xponet,US 13 - AS327786 17399 0.4%4349.8 -- TELECOM-4G,SS 14 - AS197068 16094 0.4%4023.5 -- QRATOR HLL LLC,RU 15 - AS557413614 0.1%3614.0 -- WBSDC-NET-IN West Bengal Electronics Industry Development,IN 16 - AS287736580 0.2%3290.0 -- MASTER-SERVICE-AS Global Technology Ukraine Ltd,UA 17 - AS514287804 0.2%2601.3 -- ASIRONNET IRONNET Ltd.,CZ 18 - AS380005147 0.1%2573.5 -- CRISIL-AS [CRISIL Limited.Autonomous System],IN 19 - AS150520201 0.5%2525.1 -- DNIC-AS-01505 - Headquarters, USAISC,US 20 - AS45606 12600 0.3%2520.0 -- TOP 20 Unstable Prefixes Rank Prefix Upds % Origin AS -- AS Name 1 - 64.34.125.0/2458586 1.4% AS22059 -- -Reserved AS-,ZZ 2 - 76.191.107.0/24 57853 1.4% AS22059 -- -Reserved AS-,ZZ 3 - 192.40.244.0/24 51955 1.2% AS22773 -- ASN-CXA-ALL-CCI-22773-RDC - Cox Communications Inc.,US 4 - 61.7.155.0/24 35196 0.8% AS131090 -- CAT-IDC-4BYTENET-AS-AP CAT TELECOM Public Company Ltd,CAT ,TH 5 - 192.135.223.0/24 33877 0.8% AS8001 -- NET-ACCESS-CORP - Net Access Corporation,US 6 - 195.128.159.0/24 29589 0.7% AS56636 -- ASVEDARU VEDA Ltd.,RU 7 - 185.19.144.0/22 23140 0.5% AS199367 -- AS_GBP Globe Business Publishing Limited,GB 8 - 155.133.79.0/24 17998 0.4% AS200671 -- SKOK-JAWORZNO SKOK Jaworzno,PL 9 - 185.65.148.0/24 16091 0.4% AS197068 -- QRATOR HLL LLC,RU 10 - 185.38.132.0/24 14778 0.3% AS28 -- PIRUM-AS Pirum Systems Limited,GB 11 - 78.140.0.0/18 14590 0.3% AS31357 -- TOMICA-AS Tomsk Information and Consulting Agency,RU 12 - 187.244.68.0/22 13302 0.3% AS13999 -- Mega Cable, S.A. de C.V.,MX 13 - 199.60.234.0/23 10150 0.2% AS30295 -- 2ICSYSTEMSINC - 2iC Systems Inc.,CA 14 - 199.60.236.0/249942 0.2% AS30295 -- 2ICSYSTEMSINC - 2iC Systems Inc.,CA 15 - 92.43.216.0/21 9916 0.2% AS25563 -- WEBLAND-AS Webland AG,CH 16 - 185.84.192.0/229588 0.2% AS25563 -- WEBLAND-AS Webland AG,CH 17 -
Re: Level(3) ex-twtelecom midwest packet loss (4323)
Personally I thought it was funny I don't think it was meant to come across as offensive On 8/28/2015 11:45 AM, Mel Beckman wrote: Blake, There's no call to be blatantly offensive like that. -mel On Aug 28, 2015, at 8:35 AM, Blake Dunlap iki...@gmail.com wrote: I'll just leave this here https://honestnetworker.wordpress.com/2013/11/04/the-true-meaning-behind-most-rfos/ On Fri, Aug 28, 2015 at 8:26 AM, Jason Canady ja...@unlimitednet.us wrote: Mike, I would take it to mean someone screwed something up and they don't want to admit to it. :-) That's just a guess. -- Jason Canady Unlimited Net, LLC Responsive, Reliable, Secure www.unlimitednet.us ja...@unlimitednet.us twitter: @unlimitednet On 8/28/15 12:00 AM, Mike Hammett wrote: 08/28/2015 3:08 AM GMT Event Conclusion Summary Start: August 27, 2015 13:20 GMT Stop: August 28, 2015 00:00 GMT Root Cause: A protocol issue impacted IP services in multiple markets. Fix Action: Adjustments were made to clear the errors. Summary: The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any additional impact be experienced, please contact the Level 3 Technical Service Center. What the hell is a protocol issue? I'm not an idiot, you can tell me specifically what happened... - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com - Original Message - From: Ryan Gelobter rya...@atwgpc.net To: Mel Beckman m...@beckman.org Cc: nanog@nanog.org nanog@nanog.org Sent: Thursday, August 27, 2015 3:14:59 PM Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323) If you have access to the Level3 portal you should see ticket #9639047 under Network Events now. Event Summary:IP Network Event ~ Multiple Markets 08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time. 08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue. 08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. 08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist. 08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
Re: Level(3) ex-twtelecom midwest packet loss (4323)
Randy, Frame Check Sequence errors are not a problem if you are capturing your packets from an endpoint system such as a PC participating in communications. Modern network cards do checksum offloading, so the FCS is calculated and applied by the NIC instead of by the OS. If you're capturing on one of the endpoints, Wireshark will see frames transmitted by that endpoint machine before the NIC inserts a correct FCS. If communication succeeds (e.g., you see replies to pings), then there is not really an FCS error. If the FCS really was incorrect, the NIC would have discarded them, and you wouldn't see replies. It's best to capture directly from the wire (for example, from a switch span port or a network tap) rather than an endpoint involved in the communication. -mel On Aug 28, 2015, at 11:49 AM, Randy na...@afxr.netmailto:na...@afxr.net wrote: On 8/27/2015 11:00 PM, Mike Hammett wrote: 08/28/2015 3:08 AM GMT Event Conclusion Summary Start: August 27, 2015 13:20 GMT Stop: August 28, 2015 00:00 GMT Root Cause: A protocol issue impacted IP services in multiple markets. Fix Action: Adjustments were made to clear the errors. Summary: The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any additional impact be experienced, please contact the Level 3 Technical Service Center. What the hell is a protocol issue? I'm not an idiot, you can tell me specifically what happened... I was having major issues yesterday. It wasn't with all traffic, and its seemed to have cleared up now. I did do packet inspections, and I kept getting ethernet frame check errors from certain sources. vast majority of packets did not have this extra data appended to the tcp portion of packet, and I think that is likely the reason for the issues, since those packets are supposed to be dropped if there is a checksum error. I'm by no means an expert, just voicing what I was seeing. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com - Original Message - From: Ryan Gelobter rya...@atwgpc.netmailto:rya...@atwgpc.net To: Mel Beckman m...@beckman.orgmailto:m...@beckman.org Cc: nanog@nanog.orgmailto:nanog@nanog.org nanog@nanog.orgmailto:nanog@nanog.org Sent: Thursday, August 27, 2015 3:14:59 PM Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323) If you have access to the Level3 portal you should see ticket #9639047 under Network Events now. Event Summary:IP Network Event ~ Multiple Markets 08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time. 08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue. 08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. 08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist. 08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
