Re: [c-nsp] GRE over IPSEC loss in IOS 15.x / ISR x9xx Routers
On Saturday, October 08, 2011 04:09:58 AM Dustin Schuemann wrote: I believe we have solved the issue. We tag our telnet and sip packets as AF 41. Removing the dscp AF 41 from these packets fixes the issue. A case of GBLX not remarking ingress Internet traffic from customers to 'DSCP default'? Mark. signature.asc Description: This is a digitally signed message part. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] GRE over IPSEC loss in IOS 15.x / ISR x9xx Routers
I believe we have solved the issue. We tag our telnet and sip packets as AF 41. Removing the dscp AF 41 from these packets fixes the issue. On Thu, Oct 6, 2011 at 2:05 PM, vinny_abe...@dell.com wrote: We saw something similar with Global Crossing on and off where any IPSec tunnels we had that transited their network would have loss over the tunnel with the encrypted traffic, but no loss from peer to peer. Removing Global Crossing from the equation solved the issue. I couldn't imagine how they were accomplishing that other than perhaps QoS or rate-limiting involving ESP or UDP 4500 traffic which was very hard to prove. I don't know of an esptraceroute tool. :) -Vinny -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto: cisco-nsp-boun...@puck.nether.net] On Behalf Of Dustin Schuemann Sent: Wednesday, October 05, 2011 9:22 PM To: Phil Mayers Cc: cisco-nsp@puck.nether.net Subject: Re: [c-nsp] GRE over IPSEC loss in IOS 15.x / ISR x9xx Routers Today I also noticed that all these connections are going over comcast business. Anyone seen anything like this? On Tue, Sep 27, 2011 at 5:43 PM, Dustin Schuemann dschuem...@gmail.com wrote: Do you have any other suggestions. TAC is kinda going around in circles. On Sep 27, 2011, at 3:43 AM, Phil Mayers wrote: On 09/27/2011 12:38 AM, Dustin Schuemann wrote: Disabling CEF didn't correct the issue. I'm not surprised. I'm amazed TAC would even suggest it. Disabling CEF on modern IOS isn't sensible. The slower code paths don't get properly tested any more, and whole (large) chunks of functionality only exist as CEF code. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] GRE over IPSEC loss in IOS 15.x / ISR x9xx Routers
We saw something similar with Global Crossing on and off where any IPSec tunnels we had that transited their network would have loss over the tunnel with the encrypted traffic, but no loss from peer to peer. Removing Global Crossing from the equation solved the issue. I couldn't imagine how they were accomplishing that other than perhaps QoS or rate-limiting involving ESP or UDP 4500 traffic which was very hard to prove. I don't know of an esptraceroute tool. :) -Vinny -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Dustin Schuemann Sent: Wednesday, October 05, 2011 9:22 PM To: Phil Mayers Cc: cisco-nsp@puck.nether.net Subject: Re: [c-nsp] GRE over IPSEC loss in IOS 15.x / ISR x9xx Routers Today I also noticed that all these connections are going over comcast business. Anyone seen anything like this? On Tue, Sep 27, 2011 at 5:43 PM, Dustin Schuemann dschuem...@gmail.comwrote: Do you have any other suggestions. TAC is kinda going around in circles. On Sep 27, 2011, at 3:43 AM, Phil Mayers wrote: On 09/27/2011 12:38 AM, Dustin Schuemann wrote: Disabling CEF didn't correct the issue. I'm not surprised. I'm amazed TAC would even suggest it. Disabling CEF on modern IOS isn't sensible. The slower code paths don't get properly tested any more, and whole (large) chunks of functionality only exist as CEF code. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] GRE over IPSEC loss in IOS 15.x / ISR x9xx Routers
Today I also noticed that all these connections are going over comcast business. Anyone seen anything like this? On Tue, Sep 27, 2011 at 5:43 PM, Dustin Schuemann dschuem...@gmail.comwrote: Do you have any other suggestions. TAC is kinda going around in circles. On Sep 27, 2011, at 3:43 AM, Phil Mayers wrote: On 09/27/2011 12:38 AM, Dustin Schuemann wrote: Disabling CEF didn't correct the issue. I'm not surprised. I'm amazed TAC would even suggest it. Disabling CEF on modern IOS isn't sensible. The slower code paths don't get properly tested any more, and whole (large) chunks of functionality only exist as CEF code. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] GRE over IPSEC loss in IOS 15.x / ISR x9xx Routers
On 09/27/2011 12:38 AM, Dustin Schuemann wrote: Disabling CEF didn't correct the issue. I'm not surprised. I'm amazed TAC would even suggest it. Disabling CEF on modern IOS isn't sensible. The slower code paths don't get properly tested any more, and whole (large) chunks of functionality only exist as CEF code. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] GRE over IPSEC loss in IOS 15.x / ISR x9xx Routers
Hey Dustin, We seen similar issue but with NAT enabled and that was on 12.4(15)T14, where first TCP SYN drops. Check bug CSCti13229. On 26/09/11 02:01, Dustin Schuemann wrote: We have about 200 sites connected to us via GRE tunnels over IPSEC over MPLS for primary connectivity, and GRE over IPSEC over the Internet for backup, and EIGRP routing handling the failover. Most of them are 2811HSEC/K9's, and they're working great. We've recently discovered issues with a couple of clients. They run fine over their primary GRE over IPSEC connection, but when they failover to backup we're losing certain packets (details will follow). What we found is that they're all on either 1941's or 2911's, and are running 15.0Mx IOS with advanced IP services. The rest of our clients are on 12.4T train, and none of them have any problems. We suspect it is an issue with the 15.x IOS. Specifically, we're seeing two packets consistently lost. The first is a TCP 'SYN-ACK' from a telnet server, and the second is a UDP SIP REGISTER OK message. Both packets are quite small (well under 500 bytes), so I don't suspect an MTU issue. Packet captures both show that they're being encrypted and sent by the head-end, but are lost before they reach the decrypted tunnel interface. So either they're being lost in the path across the Internet, or the decryption is failing. We see larger packets get through just fine, and other connections work great. We've opened a ticket with TAC but so far they have no clue. Since these routers can't be downgraded to 12.4, our current plans are to ship a 2811HSEC bundle with an identical configuration to these clients to see if we can verify that it's a 15.0 issue, but I'm curious if anybody's seen anything similar, or if somebody who's more familiar than I am with bug tracker can find anything. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] GRE over IPSEC loss in IOS 15.x / ISR x9xx Routers
I agree, except that it fixed the issues we were seeing. I hadn't heard of disabling CEF as an actual fix for a problem for many many years... then we saw it here with the 1921's and various IPSec scenarios. -Vinny -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Phil Mayers Sent: Tuesday, September 27, 2011 3:44 AM To: cisco-nsp@puck.nether.net Subject: Re: [c-nsp] GRE over IPSEC loss in IOS 15.x / ISR x9xx Routers On 09/27/2011 12:38 AM, Dustin Schuemann wrote: Disabling CEF didn't correct the issue. I'm not surprised. I'm amazed TAC would even suggest it. Disabling CEF on modern IOS isn't sensible. The slower code paths don't get properly tested any more, and whole (large) chunks of functionality only exist as CEF code. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] GRE over IPSEC loss in IOS 15.x / ISR x9xx Routers
Do you have any other suggestions. TAC is kinda going around in circles. On Sep 27, 2011, at 3:43 AM, Phil Mayers wrote: On 09/27/2011 12:38 AM, Dustin Schuemann wrote: Disabling CEF didn't correct the issue. I'm not surprised. I'm amazed TAC would even suggest it. Disabling CEF on modern IOS isn't sensible. The slower code paths don't get properly tested any more, and whole (large) chunks of functionality only exist as CEF code. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] GRE over IPSEC loss in IOS 15.x / ISR x9xx Routers
We've seen a couple of weird problems with 1921's running 15.0M(x)... We've observed certain things like IPSec client functionality breaking when failing over to backup circuits which worked perfectly fine under older code and older routers that could run this code with the same configuration. The only workaround TAC could offer was disable CEF... of course definitely not ideal, but even more odd I cannot find the performance impact on the 1900 series ISR's with CEF disabled. The routing performance document from Cisco doesn't list anything in the column for the 1941 in the process switching columns... only the Fast/CEF switching. We haven't seen any performance issues in our customer environments where we have to do this to fix functionality, but I'd much appreciate it if CEF actually worked with the feature sets in the router. Another thing that doesn't work with CEF enabled in this code train is terminating an IPSec tunnel on a loopback interface. Works ok in other version and works fine if I disable CEF. -Vinny -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Dustin Schuemann Sent: Sunday, September 25, 2011 6:01 PM To: cisco-nsp@puck.nether.net Subject: [c-nsp] GRE over IPSEC loss in IOS 15.x / ISR x9xx Routers We have about 200 sites connected to us via GRE tunnels over IPSEC over MPLS for primary connectivity, and GRE over IPSEC over the Internet for backup, and EIGRP routing handling the failover. Most of them are 2811HSEC/K9's, and they're working great. We've recently discovered issues with a couple of clients. They run fine over their primary GRE over IPSEC connection, but when they failover to backup we're losing certain packets (details will follow). What we found is that they're all on either 1941's or 2911's, and are running 15.0Mx IOS with advanced IP services. The rest of our clients are on 12.4T train, and none of them have any problems. We suspect it is an issue with the 15.x IOS. Specifically, we're seeing two packets consistently lost. The first is a TCP 'SYN-ACK' from a telnet server, and the second is a UDP SIP REGISTER OK message. Both packets are quite small (well under 500 bytes), so I don't suspect an MTU issue. Packet captures both show that they're being encrypted and sent by the head-end, but are lost before they reach the decrypted tunnel interface. So either they're being lost in the path across the Internet, or the decryption is failing. We see larger packets get through just fine, and other connections work great. We've opened a ticket with TAC but so far they have no clue. Since these routers can't be downgraded to 12.4, our current plans are to ship a 2811HSEC bundle with an identical configuration to these clients to see if we can verify that it's a 15.0 issue, but I'm curious if anybody's seen anything similar, or if somebody who's more familiar than I am with bug tracker can find anything. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] GRE over IPSEC loss in IOS 15.x / ISR x9xx Routers
Disabling CEF didn't correct the issue. Any more suggestions? On Sep 26, 2011, at 11:35 AM, vinny_abe...@dell.com vinny_abe...@dell.com wrote: We've seen a couple of weird problems with 1921's running 15.0M(x)... We've observed certain things like IPSec client functionality breaking when failing over to backup circuits which worked perfectly fine under older code and older routers that could run this code with the same configuration. The only workaround TAC could offer was disable CEF... of course definitely not ideal, but even more odd I cannot find the performance impact on the 1900 series ISR's with CEF disabled. The routing performance document from Cisco doesn't list anything in the column for the 1941 in the process switching columns... only the Fast/CEF switching. We haven't seen any performance issues in our customer environments where we have to do this to fix functionality, but I'd much appreciate it if CEF actually worked with the feature sets in the router. Another thing that doesn't work with CEF enabled in this code train is terminating an IPSec tunnel on a loopback interface. Works ok in other version and works fine if I disable CEF. -Vinny -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Dustin Schuemann Sent: Sunday, September 25, 2011 6:01 PM To: cisco-nsp@puck.nether.net Subject: [c-nsp] GRE over IPSEC loss in IOS 15.x / ISR x9xx Routers We have about 200 sites connected to us via GRE tunnels over IPSEC over MPLS for primary connectivity, and GRE over IPSEC over the Internet for backup, and EIGRP routing handling the failover. Most of them are 2811HSEC/K9's, and they're working great. We've recently discovered issues with a couple of clients. They run fine over their primary GRE over IPSEC connection, but when they failover to backup we're losing certain packets (details will follow). What we found is that they're all on either 1941's or 2911's, and are running 15.0Mx IOS with advanced IP services. The rest of our clients are on 12.4T train, and none of them have any problems. We suspect it is an issue with the 15.x IOS. Specifically, we're seeing two packets consistently lost. The first is a TCP 'SYN-ACK' from a telnet server, and the second is a UDP SIP REGISTER OK message. Both packets are quite small (well under 500 bytes), so I don't suspect an MTU issue. Packet captures both show that they're being encrypted and sent by the head-end, but are lost before they reach the decrypted tunnel interface. So either they're being lost in the path across the Internet, or the decryption is failing. We see larger packets get through just fine, and other connections work great. We've opened a ticket with TAC but so far they have no clue. Since these routers can't be downgraded to 12.4, our current plans are to ship a 2811HSEC bundle with an identical configuration to these clients to see if we can verify that it's a 15.0 issue, but I'm curious if anybody's seen anything similar, or if somebody who's more familiar than I am with bug tracker can find anything. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
[c-nsp] GRE over IPSEC loss in IOS 15.x / ISR x9xx Routers
We have about 200 sites connected to us via GRE tunnels over IPSEC over MPLS for primary connectivity, and GRE over IPSEC over the Internet for backup, and EIGRP routing handling the failover. Most of them are 2811HSEC/K9's, and they're working great. We've recently discovered issues with a couple of clients. They run fine over their primary GRE over IPSEC connection, but when they failover to backup we're losing certain packets (details will follow). What we found is that they're all on either 1941's or 2911's, and are running 15.0Mx IOS with advanced IP services. The rest of our clients are on 12.4T train, and none of them have any problems. We suspect it is an issue with the 15.x IOS. Specifically, we're seeing two packets consistently lost. The first is a TCP 'SYN-ACK' from a telnet server, and the second is a UDP SIP REGISTER OK message. Both packets are quite small (well under 500 bytes), so I don't suspect an MTU issue. Packet captures both show that they're being encrypted and sent by the head-end, but are lost before they reach the decrypted tunnel interface. So either they're being lost in the path across the Internet, or the decryption is failing. We see larger packets get through just fine, and other connections work great. We've opened a ticket with TAC but so far they have no clue. Since these routers can't be downgraded to 12.4, our current plans are to ship a 2811HSEC bundle with an identical configuration to these clients to see if we can verify that it's a 15.0 issue, but I'm curious if anybody's seen anything similar, or if somebody who's more familiar than I am with bug tracker can find anything. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
