Hi Jason, I think that this was it. The first octet of our router’s Level3 facing interface MAC is a 6. Changed it to 0 and it does look much better now. Thanks!
-- Mark Wicker | Senior Network Engineer Esri | 380 New York St | Redlands, CA 92354 | USA T 909 793 2853 x2741 | [email protected]<mailto:[email protected]> | esri.com<http://www.esri.com/> From: Jason Rokeach [mailto:[email protected]] Sent: Tuesday, January 17, 2017 6:57 PM To: Mark Wicker <[email protected]> Cc: [email protected] Subject: Re: Level3 Internet service, out of order packets causing issues Hi Mark, I'm going to throw out a guess here. By any chance, is the first octet of your router's MAC address a 4 or a 6? In general, modern routers do not load balance per-packet, which is what caused out-of-order issues in days gone. Load balancing is usually done based on a hash of the source and/or destination IP of the packet, MPLS label, or Ethernet (on a switched interface). The most common cause for actual unordering of packets/frames in a modern service provider network, in my experience, is actually this hashing mechanism. Many vendor's hashing implementations assume, based on position in the frame, that a frame with a MAC address beginning with 4 or 6 is an IPv4 or IPv6 frame, not an MPLS frame. This can result in out of order packets. The most common fix is control word being applied on a pseudowire (assuming you are being carried across a pseudowire in the SP network), but if this is what is occurring, you could also resolve the issue by changing your MAC address. On Tue, Jan 17, 2017 at 2:12 PM, Mark Wicker <[email protected]<mailto:[email protected]>> wrote: Hi, I have 1G Level3 ethernet dedicated internet service as one of my ISP's at my company based in the Los Angeles (Inland Empire) area. After seeing strange application behavior while using this circuit, I failed it out of service and have been troubleshooting it with a directly connected machine (publically addressed, no firewall, nothing between this machine and our Level3 router). I have taken several packet captures while accessing various sites and have noticed large numbers of out of order packets which are wreaking havoc with TCP connections and other traffic. In my experience, per-packet load balancing across various different links can cause this issue. I do not see this behavior with my other ISP's. I have had several tickets opened with Level3 but have had no success. Any help here? Anyone out there seen this and have any contacts that may be able to help? FYI - we own our own public IP space and advertise via BGP to Level3. Currently I am using a dedicated /24 of our space advertised to Level3 only to ensure that the return path is through Level3 and not another ISP. Also, everything is single linked from a layer 2 and 3 perspective from the router to the test machine to ensure that the cause of any out of order packets is not on our end. Thanks, -- Mark Wicker | Senior Network Engineer Esri | 380 New York St | Redlands, CA 92354 | USA T 909 793 2853 x2741 | [email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> | esri.com<http://esri.com><http://www.esri.com/>
