Re: Level(3) ex-twtelecom midwest packet loss (4323)
Blake, There's no call to be blatantly offensive like that. -mel On Aug 28, 2015, at 8:35 AM, Blake Dunlap iki...@gmail.com wrote: I'll just leave this here https://honestnetworker.wordpress.com/2013/11/04/the-true-meaning-behind-most-rfos/ On Fri, Aug 28, 2015 at 8:26 AM, Jason Canady ja...@unlimitednet.us wrote: Mike, I would take it to mean someone screwed something up and they don't want to admit to it. :-) That's just a guess. -- Jason Canady Unlimited Net, LLC Responsive, Reliable, Secure www.unlimitednet.us ja...@unlimitednet.us twitter: @unlimitednet On 8/28/15 12:00 AM, Mike Hammett wrote: 08/28/2015 3:08 AM GMT Event Conclusion Summary Start: August 27, 2015 13:20 GMT Stop: August 28, 2015 00:00 GMT Root Cause: A protocol issue impacted IP services in multiple markets. Fix Action: Adjustments were made to clear the errors. Summary: The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any additional impact be experienced, please contact the Level 3 Technical Service Center. What the hell is a protocol issue? I'm not an idiot, you can tell me specifically what happened... - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com - Original Message - From: Ryan Gelobter rya...@atwgpc.net To: Mel Beckman m...@beckman.org Cc: nanog@nanog.org nanog@nanog.org Sent: Thursday, August 27, 2015 3:14:59 PM Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323) If you have access to the Level3 portal you should see ticket #9639047 under Network Events now. Event Summary:IP Network Event ~ Multiple Markets 08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time. 08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue. 08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. 08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist. 08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
Re: Level(3) ex-twtelecom midwest packet loss (4323)
Mike, I would take it to mean someone screwed something up and they don't want to admit to it. :-) That's just a guess. -- Jason Canady Unlimited Net, LLC Responsive, Reliable, Secure www.unlimitednet.us ja...@unlimitednet.us twitter: @unlimitednet On 8/28/15 12:00 AM, Mike Hammett wrote: 08/28/2015 3:08 AM GMT Event Conclusion Summary Start: August 27, 2015 13:20 GMT Stop: August 28, 2015 00:00 GMT Root Cause: A protocol issue impacted IP services in multiple markets. Fix Action: Adjustments were made to clear the errors. Summary: The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any additional impact be experienced, please contact the Level 3 Technical Service Center. What the hell is a protocol issue? I'm not an idiot, you can tell me specifically what happened... - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com - Original Message - From: Ryan Gelobter rya...@atwgpc.net To: Mel Beckman m...@beckman.org Cc: nanog@nanog.org nanog@nanog.org Sent: Thursday, August 27, 2015 3:14:59 PM Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323) If you have access to the Level3 portal you should see ticket #9639047 under Network Events now. Event Summary:IP Network Event ~ Multiple Markets 08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time. 08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue. 08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. 08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist. 08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
Re: Level(3) ex-twtelecom midwest packet loss (4323)
I'll just leave this here https://honestnetworker.wordpress.com/2013/11/04/the-true-meaning-behind-most-rfos/ On Fri, Aug 28, 2015 at 8:26 AM, Jason Canady ja...@unlimitednet.us wrote: Mike, I would take it to mean someone screwed something up and they don't want to admit to it. :-) That's just a guess. -- Jason Canady Unlimited Net, LLC Responsive, Reliable, Secure www.unlimitednet.us ja...@unlimitednet.us twitter: @unlimitednet On 8/28/15 12:00 AM, Mike Hammett wrote: 08/28/2015 3:08 AM GMT Event Conclusion Summary Start: August 27, 2015 13:20 GMT Stop: August 28, 2015 00:00 GMT Root Cause: A protocol issue impacted IP services in multiple markets. Fix Action: Adjustments were made to clear the errors. Summary: The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any additional impact be experienced, please contact the Level 3 Technical Service Center. What the hell is a protocol issue? I'm not an idiot, you can tell me specifically what happened... - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com - Original Message - From: Ryan Gelobter rya...@atwgpc.net To: Mel Beckman m...@beckman.org Cc: nanog@nanog.org nanog@nanog.org Sent: Thursday, August 27, 2015 3:14:59 PM Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323) If you have access to the Level3 portal you should see ticket #9639047 under Network Events now. Event Summary:IP Network Event ~ Multiple Markets 08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time. 08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue. 08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. 08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist. 08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
Re: Level(3) ex-twtelecom midwest packet loss (4323)
Seemed reasonably accurate to me. [image: Phone.com] http://www.phone.com/James SinkSenior Network Engineer js...@phone.com(800) 997-9179 x506Try Phone.com free! http://www.phone.com/?_tracking_id=10132CONFIDENTIALITY NOTICE: This e-mail and any attachments are for the exclusive and confidential use of the intended recipient. If you received this in error, please do not read, distribute, or take action in reliance upon this message. Instead, please notify us immediately by return e-mail and promptly delete this message and its attachments from your computer system. On Fri, Aug 28, 2015 at 8:45 AM, Mel Beckman m...@beckman.org wrote: Blake, There's no call to be blatantly offensive like that. -mel On Aug 28, 2015, at 8:35 AM, Blake Dunlap iki...@gmail.com wrote: I'll just leave this here https://honestnetworker.wordpress.com/2013/11/04/the-true-meaning-behind-most-rfos/ On Fri, Aug 28, 2015 at 8:26 AM, Jason Canady ja...@unlimitednet.us wrote: Mike, I would take it to mean someone screwed something up and they don't want to admit to it. :-) That's just a guess. -- Jason Canady Unlimited Net, LLC Responsive, Reliable, Secure www.unlimitednet.us ja...@unlimitednet.us twitter: @unlimitednet On 8/28/15 12:00 AM, Mike Hammett wrote: 08/28/2015 3:08 AM GMT Event Conclusion Summary Start: August 27, 2015 13:20 GMT Stop: August 28, 2015 00:00 GMT Root Cause: A protocol issue impacted IP services in multiple markets. Fix Action: Adjustments were made to clear the errors. Summary: The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any additional impact be experienced, please contact the Level 3 Technical Service Center. What the hell is a protocol issue? I'm not an idiot, you can tell me specifically what happened... - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com - Original Message - From: Ryan Gelobter rya...@atwgpc.net To: Mel Beckman m...@beckman.org Cc: nanog@nanog.org nanog@nanog.org Sent: Thursday, August 27, 2015 3:14:59 PM Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323) If you have access to the Level3 portal you should see ticket #9639047 under Network Events now. Event Summary:IP Network Event ~ Multiple Markets 08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time. 08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue. 08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. 08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist. 08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
Re: Level(3) ex-twtelecom midwest packet loss (4323)
ill just leave this here :) https://www.youtube.com/watch?v=0ilMx7k7mso On Fri, Aug 28, 2015 at 11:33 AM, Blake Dunlap iki...@gmail.com wrote: I'll just leave this here https://honestnetworker.wordpress.com/2013/11/04/the-true-meaning-behind-most-rfos/ On Fri, Aug 28, 2015 at 8:26 AM, Jason Canady ja...@unlimitednet.us wrote: Mike, I would take it to mean someone screwed something up and they don't want to admit to it. :-) That's just a guess. -- Jason Canady Unlimited Net, LLC Responsive, Reliable, Secure www.unlimitednet.us ja...@unlimitednet.us twitter: @unlimitednet On 8/28/15 12:00 AM, Mike Hammett wrote: 08/28/2015 3:08 AM GMT Event Conclusion Summary Start: August 27, 2015 13:20 GMT Stop: August 28, 2015 00:00 GMT Root Cause: A protocol issue impacted IP services in multiple markets. Fix Action: Adjustments were made to clear the errors. Summary: The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any additional impact be experienced, please contact the Level 3 Technical Service Center. What the hell is a protocol issue? I'm not an idiot, you can tell me specifically what happened... - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com - Original Message - From: Ryan Gelobter rya...@atwgpc.net To: Mel Beckman m...@beckman.org Cc: nanog@nanog.org nanog@nanog.org Sent: Thursday, August 27, 2015 3:14:59 PM Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323) If you have access to the Level3 portal you should see ticket #9639047 under Network Events now. Event Summary:IP Network Event ~ Multiple Markets 08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time. 08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue. 08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. 08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist. 08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
Re: Level(3) ex-twtelecom midwest packet loss (4323)
This reminded me of this great clip https://www.youtube.com/watch?v=ckIMuvumYrg On Fri, Aug 28, 2015 at 12:00 AM, Mike Hammett na...@ics-il.net wrote: 08/28/2015 3:08 AM GMT Event Conclusion Summary Start: August 27, 2015 13:20 GMT Stop: August 28, 2015 00:00 GMT Root Cause: A protocol issue impacted IP services in multiple markets. Fix Action: Adjustments were made to clear the errors. Summary: The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any additional impact be experienced, please contact the Level 3 Technical Service Center. What the hell is a protocol issue? I'm not an idiot, you can tell me specifically what happened... - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com - Original Message - From: Ryan Gelobter rya...@atwgpc.net To: Mel Beckman m...@beckman.org Cc: nanog@nanog.org nanog@nanog.org Sent: Thursday, August 27, 2015 3:14:59 PM Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323) If you have access to the Level3 portal you should see ticket #9639047 under Network Events now. Event Summary:IP Network Event ~ Multiple Markets 08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time. 08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue. 08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. 08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist. 08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
Re: Level(3) ex-twtelecom midwest packet loss (4323)
On 8/27/2015 11:00 PM, Mike Hammett wrote: 08/28/2015 3:08 AM GMT Event Conclusion Summary Start: August 27, 2015 13:20 GMT Stop: August 28, 2015 00:00 GMT Root Cause: A protocol issue impacted IP services in multiple markets. Fix Action: Adjustments were made to clear the errors. Summary: The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any additional impact be experienced, please contact the Level 3 Technical Service Center. What the hell is a protocol issue? I'm not an idiot, you can tell me specifically what happened... I was having major issues yesterday. It wasn't with all traffic, and its seemed to have cleared up now. I did do packet inspections, and I kept getting ethernet frame check errors from certain sources. vast majority of packets did not have this extra data appended to the tcp portion of packet, and I think that is likely the reason for the issues, since those packets are supposed to be dropped if there is a checksum error. I'm by no means an expert, just voicing what I was seeing. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com - Original Message - From: Ryan Gelobter rya...@atwgpc.net To: Mel Beckman m...@beckman.org Cc: nanog@nanog.org nanog@nanog.org Sent: Thursday, August 27, 2015 3:14:59 PM Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323) If you have access to the Level3 portal you should see ticket #9639047 under Network Events now. Event Summary:IP Network Event ~ Multiple Markets 08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time. 08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue. 08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. 08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist. 08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
Re: Level(3) ex-twtelecom midwest packet loss (4323)
Personally I thought it was funny I don't think it was meant to come across as offensive On 8/28/2015 11:45 AM, Mel Beckman wrote: Blake, There's no call to be blatantly offensive like that. -mel On Aug 28, 2015, at 8:35 AM, Blake Dunlap iki...@gmail.com wrote: I'll just leave this here https://honestnetworker.wordpress.com/2013/11/04/the-true-meaning-behind-most-rfos/ On Fri, Aug 28, 2015 at 8:26 AM, Jason Canady ja...@unlimitednet.us wrote: Mike, I would take it to mean someone screwed something up and they don't want to admit to it. :-) That's just a guess. -- Jason Canady Unlimited Net, LLC Responsive, Reliable, Secure www.unlimitednet.us ja...@unlimitednet.us twitter: @unlimitednet On 8/28/15 12:00 AM, Mike Hammett wrote: 08/28/2015 3:08 AM GMT Event Conclusion Summary Start: August 27, 2015 13:20 GMT Stop: August 28, 2015 00:00 GMT Root Cause: A protocol issue impacted IP services in multiple markets. Fix Action: Adjustments were made to clear the errors. Summary: The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any additional impact be experienced, please contact the Level 3 Technical Service Center. What the hell is a protocol issue? I'm not an idiot, you can tell me specifically what happened... - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com - Original Message - From: Ryan Gelobter rya...@atwgpc.net To: Mel Beckman m...@beckman.org Cc: nanog@nanog.org nanog@nanog.org Sent: Thursday, August 27, 2015 3:14:59 PM Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323) If you have access to the Level3 portal you should see ticket #9639047 under Network Events now. Event Summary:IP Network Event ~ Multiple Markets 08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time. 08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue. 08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. 08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist. 08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
Re: Level(3) ex-twtelecom midwest packet loss (4323)
Randy, Frame Check Sequence errors are not a problem if you are capturing your packets from an endpoint system such as a PC participating in communications. Modern network cards do checksum offloading, so the FCS is calculated and applied by the NIC instead of by the OS. If you're capturing on one of the endpoints, Wireshark will see frames transmitted by that endpoint machine before the NIC inserts a correct FCS. If communication succeeds (e.g., you see replies to pings), then there is not really an FCS error. If the FCS really was incorrect, the NIC would have discarded them, and you wouldn't see replies. It's best to capture directly from the wire (for example, from a switch span port or a network tap) rather than an endpoint involved in the communication. -mel On Aug 28, 2015, at 11:49 AM, Randy na...@afxr.netmailto:na...@afxr.net wrote: On 8/27/2015 11:00 PM, Mike Hammett wrote: 08/28/2015 3:08 AM GMT Event Conclusion Summary Start: August 27, 2015 13:20 GMT Stop: August 28, 2015 00:00 GMT Root Cause: A protocol issue impacted IP services in multiple markets. Fix Action: Adjustments were made to clear the errors. Summary: The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any additional impact be experienced, please contact the Level 3 Technical Service Center. What the hell is a protocol issue? I'm not an idiot, you can tell me specifically what happened... I was having major issues yesterday. It wasn't with all traffic, and its seemed to have cleared up now. I did do packet inspections, and I kept getting ethernet frame check errors from certain sources. vast majority of packets did not have this extra data appended to the tcp portion of packet, and I think that is likely the reason for the issues, since those packets are supposed to be dropped if there is a checksum error. I'm by no means an expert, just voicing what I was seeing. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com - Original Message - From: Ryan Gelobter rya...@atwgpc.netmailto:rya...@atwgpc.net To: Mel Beckman m...@beckman.orgmailto:m...@beckman.org Cc: nanog@nanog.orgmailto:nanog@nanog.org nanog@nanog.orgmailto:nanog@nanog.org Sent: Thursday, August 27, 2015 3:14:59 PM Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323) If you have access to the Level3 portal you should see ticket #9639047 under Network Events now. Event Summary:IP Network Event ~ Multiple Markets 08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time. 08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue. 08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. 08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist. 08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
Re: Level(3) ex-twtelecom midwest packet loss (4323)
Finally got a call back from the Level3 NOC. There is a global ticket for this, #ST1878748. This is a wide-ranging outage effecting the entire Level3 network. I am supposed to get emailed updates now automatically, and will reflect those here. -mel On Aug 26, 2015, at 6:01 PM, Mel Beckman m...@beckman.org wrote: We continue to see 10 to 20 percent packet loss crossing TW border and even between clients in the same region (e.g. LA and Santa Barbara). No news from the NOC yet. -mel From: NANOG nanog-boun...@nanog.org on behalf of Jason Hellenthal jhellent...@dataix.net Sent: Wednesday, August 26, 2015 5:33 PM To: nanog@nanog.org Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323) Cleared up here in WI TW/Level3 COLO between 19:00 - 19:20 CST - 3235 Intertech Dr. Brookfield On Aug 26, 2015, at 16:44, Ryan K. Brooks r...@hack.net wrote: Seems to be impacting their entire network now. On 8/26/15 4:41 PM, Rafael Possamai wrote: I have been seeing the same issues, but haven't heard anything back yet. It has improved in the last 30 minutes or so, see below. http://imgur.com/KVAzetA * * On Wed, Aug 26, 2015 at 4:34 PM, Ryan K. Brooks r...@hack.net mailto:r...@hack.net wrote: Seeing packet loss on AS4323 since 2:30 Central time. NOC is unresponsive to phone and email. Anyone have an idea what's going on over there? -- Jason Hellenthal JJH48-ARIN
Re: Level(3) ex-twtelecom midwest packet loss (4323)
If you have access to the Level3 portal you should see ticket #9639047 under Network Events now. Event Summary:IP Network Event ~ Multiple Markets 08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time. 08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue. 08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. 08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist. 08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
Re: Level(3) ex-twtelecom midwest packet loss (4323)
08/28/2015 3:08 AM GMT Event Conclusion Summary Start: August 27, 2015 13:20 GMT Stop: August 28, 2015 00:00 GMT Root Cause: A protocol issue impacted IP services in multiple markets. Fix Action: Adjustments were made to clear the errors. Summary: The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any additional impact be experienced, please contact the Level 3 Technical Service Center. What the hell is a protocol issue? I'm not an idiot, you can tell me specifically what happened... - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com - Original Message - From: Ryan Gelobter rya...@atwgpc.net To: Mel Beckman m...@beckman.org Cc: nanog@nanog.org nanog@nanog.org Sent: Thursday, August 27, 2015 3:14:59 PM Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323) If you have access to the Level3 portal you should see ticket #9639047 under Network Events now. Event Summary:IP Network Event ~ Multiple Markets 08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time. 08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue. 08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. 08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist. 08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
Re: Level(3) ex-twtelecom midwest packet loss (4323)
I have been seeing the same issues, but haven't heard anything back yet. It has improved in the last 30 minutes or so, see below. http://imgur.com/KVAzetA On Wed, Aug 26, 2015 at 4:34 PM, Ryan K. Brooks r...@hack.net wrote: Seeing packet loss on AS4323 since 2:30 Central time. NOC is unresponsive to phone and email. Anyone have an idea what's going on over there?
Re: Level(3) ex-twtelecom midwest packet loss (4323)
Seems to be impacting their entire network now. On 8/26/15 4:41 PM, Rafael Possamai wrote: I have been seeing the same issues, but haven't heard anything back yet. It has improved in the last 30 minutes or so, see below. http://imgur.com/KVAzetA * * On Wed, Aug 26, 2015 at 4:34 PM, Ryan K. Brooks r...@hack.net mailto:r...@hack.net wrote: Seeing packet loss on AS4323 since 2:30 Central time. NOC is unresponsive to phone and email. Anyone have an idea what's going on over there?
Re: Level(3) ex-twtelecom midwest packet loss (4323)
Cleared up here in WI TW/Level3 COLO between 19:00 - 19:20 CST - 3235 Intertech Dr. Brookfield On Aug 26, 2015, at 16:44, Ryan K. Brooks r...@hack.net wrote: Seems to be impacting their entire network now. On 8/26/15 4:41 PM, Rafael Possamai wrote: I have been seeing the same issues, but haven't heard anything back yet. It has improved in the last 30 minutes or so, see below. http://imgur.com/KVAzetA * * On Wed, Aug 26, 2015 at 4:34 PM, Ryan K. Brooks r...@hack.net mailto:r...@hack.net wrote: Seeing packet loss on AS4323 since 2:30 Central time. NOC is unresponsive to phone and email. Anyone have an idea what's going on over there? -- Jason Hellenthal JJH48-ARIN signature.asc Description: Message signed with OpenPGP using GPGMail
Re: Level(3) ex-twtelecom midwest packet loss (4323)
We continue to see 10 to 20 percent packet loss crossing TW border and even between clients in the same region (e.g. LA and Santa Barbara). No news from the NOC yet. -mel From: NANOG nanog-boun...@nanog.org on behalf of Jason Hellenthal jhellent...@dataix.net Sent: Wednesday, August 26, 2015 5:33 PM To: nanog@nanog.org Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323) Cleared up here in WI TW/Level3 COLO between 19:00 - 19:20 CST - 3235 Intertech Dr. Brookfield On Aug 26, 2015, at 16:44, Ryan K. Brooks r...@hack.net wrote: Seems to be impacting their entire network now. On 8/26/15 4:41 PM, Rafael Possamai wrote: I have been seeing the same issues, but haven't heard anything back yet. It has improved in the last 30 minutes or so, see below. http://imgur.com/KVAzetA * * On Wed, Aug 26, 2015 at 4:34 PM, Ryan K. Brooks r...@hack.net mailto:r...@hack.net wrote: Seeing packet loss on AS4323 since 2:30 Central time. NOC is unresponsive to phone and email. Anyone have an idea what's going on over there? -- Jason Hellenthal JJH48-ARIN
