Yea, I'll give the upgrade ago. I gotta schedule it out. In the meantime, I'm parshing through "debug ip packet" data to see what is being process switched. I set the debug condition to interface fa0/1 before I started. It looks like tons of stuff is being process switched for no apparent reason. Also, I have the router making calls to the defined NTP server but sourcing the addresses with the old isp interface's ip address of fa0/0) but going out the new ISP connection (fa0/1) for ntp updates when clearly the default 0.0.0.0 route is out the new isp connection (fa0/1). I don't understand why he's sourcing it like that. IP: s=orig_isp (local), d=129.7.1.66 (FastEthernet0/1 It also likes like main culprit on the new_ISP interface is "Post routing NAT". I may not be looking at the data correctly but it seems like my NAT traffic is not being switched in hardware and I'm not using any route-maps. Just the standard overload statement.
sample debug ip packet outputt 4.249 (FastEthernet0/1), len 78, sending full packet 036360: Dec 24 16:30:29.120: IP: s=172.18.1.23 (Vlan1), d=172.18.1.1, len 52, stop process pak for forus packet 036361: Dec 24 16:30:29.120: IP: s=172.18.1.23 (Vlan1), d=172.18.1.1, len 52, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 036362: Dec 24 16:30:29.124: IP: s=172.18.1.1 (local), d=172.18.1.23, len 52, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 036363: Dec 24 16:30:29.304: IP: s=172.18.1.23 (Vlan1), d=172.18.1.1, len 40, stop process pak for forus packet 036364: Dec 24 16:30:29.304: IP: s=172.18.1.23 (Vlan1), d=172.18.1.1, len 40, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 036365: Dec 24 16:30:30.324: IP: s=192.168.1.30 (local), d=192.168.3.1, len 76, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 036366: Dec 24 16:30:30.324: IP: s=orig_isp (local), d=129.7.1.66, len 76, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 036367: Dec 24 16:30:30.324: IP: s=orig_isp (local), d=129.7.1.66 (FastEthernet0/1), len 76, sending 036368: Dec 24 16:30:30.324: IP: s=orig_isp (local), d=129.7.1.66 (FastEthernet0/1), len 76, output feature, CCE Output Classification(5), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE IP: s=192.168.2.3 (Vlan10), d=192.168.1.30, len 40, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 036704: Dec 24 16:30:50.904: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30, len 40, stop process pak for forus packet 036705: Dec 24 16:30:50.904: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30, len 40, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 036706: Dec 24 16:30:51.032: IP: s=192.168.1.80 (Vlan10), d=192.168.1.30, len 28, stop process pak for forus packetpe 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 036372: Dec 24 16:30:30.324: IP: s=orig_isp (local), d=129.7.1.66 (FastEthernet0/1), len 76, output feature, Firewall (inspect) (38), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 036373: Dec 24 16:30:30.324: IP: s=orig_isp (local), d=129.7.1.66 (FastEthernet0/1), len 76, output feature, Post-Ingress-NetFlow (52), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 036374: Dec 24 16:30:30.328: IP: s=orig_isp (local), d=129.7.1.66 (FastEthernet0/1), len 76, sending full packet 036375: Dec 24 16:30:30.404: IP: s=192.168.1.79 (Vlan10), d=192.168.1.30, len 28, stop process pak for forus packet 036376: Dec 24 16:30:30.404: IP: s=192.168.1.79 (Vlan10), d=192.168.1.30, len 28, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE --More-- rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 036713: Dec 24 16:30:51.220: IP: s=192.168.1.30 (local), d=192.168.2.3, len 576, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE More.. 041793: Dec 24 16:46:59.141: IP: s=172.18.1.1 (local), d=172.18.1.23, len 52, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 041794: Dec 24 16:46:59.233: IP: s=192.168.1.80 (Vlan10), d=192.168.1.30, len 28, stop process pak for forus packet 041795: Dec 24 16:46:59.233: IP: s=192.168.1.80 (Vlan10), d=192.168.1.30, len 28, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 041796: Dec 24 16:46:59.233: IP: s=192.168.1.30 (local), d=192.168.1.80, len 28, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 041797: Dec 24 16:46:59.321: IP: s=172.18.1.23 (Vlan1), d=172.18.1.1, len 40, stop process pak for forus packet 041798: Dec 24 16:46:59.321: IP: s=172.18.1.23 (Vlan1), d=172.18.1.1, len 40, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 041799: Dec 24 16:46:59.893: IP: s=192.168.1.30 (local), d=192.168.1.13, len 56, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 041800: Dec 24 16:47:00.005: IP: s=192.168.1.146 (Vlan10), d=x-email-svc-x (FastEthernet0/1), len 68, output feature, CCE Output Classification(5), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 041801: Dec 24 16:47:00.005: IP: s=192.168.1.30 (local), d=192.168.1.146, len 56, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 042126: Dec 24 16:47:17.013: IP: s=192.168.1.30 (local), d=192.168.1.79, len 28, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 042127: Dec 24 16:47:17.017: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30, len 40, stop process pak for forus packet 042128: Dec 24 16:47:17.017: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30, len 40, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 042129: Dec 24 16:47:17.021: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30, len 40, stop process pak for forus packete 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 041806: Dec 24 16:47:00.033: IP: s=172.18.1.147 (Vlan1), d=172.18.1.1, len 40, stop process pak for forus packet 041807: Dec 24 16:47:00.033: IP: s=172.18.1.147 (Vlan1), d=172.18.1.1, len 40, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 041808: Dec 24 16:47:00.845: IP: s=192.168.1.79 (Vlan10), d=192.168.1.30, len 28, stop process pak for forus packet 041809: Dec 24 16:47:00.845: IP: s=192.168.1.79 (Vlan10), d=192.168.1.30, len 28, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 041810: Dec 24 16:47:00.845: IP: s=192.168.1.30 (local), d=192.168.1.79, len 28, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 92.168.1.30 (local), d=192.168.2.3, len 576, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 042136: Dec 24 16:47:17.217: IP: s=192.168.1.30 (local), d=192.168.2.3, len 576, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 042137: Dec 24 16:47:17.221: IP: s=192.168.1.30 (local), d=192.168.2.3, len 44, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 042138: Dec 24 16:47:17.221: IP: s=192.168.1.30 (local), d=192.168.2.3, len 444, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 042139: Dec 24 16:47:17.225: IP: s=192.168.1.31 (Vlan10), d=192.168.1.30, len 84, stop process pak for forus packet 042140: Dec 24 16:47:17.225: IP: s=192.168.1.31 (Vlan10), d=192.168.1.30, len 84, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 042141: Dec 24 16:47:17.225: IP: s=192.168.1.30 (local), d=192.168.1.31, len 84, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 042142: Dec 24 16:47:17.285: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30, len 40, stop process pak for forus packet 042143: Dec 24 16:47:17.285: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30, len 40, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE 042144: Dec 24 16:47:17.285: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30, len 40, stop process pak for forus packet 042145: Dec 24 16:47:17.285: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30, len 40, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE On Sat, Dec 24, 2011 at 9:25 AM, Chuck Church <[email protected]> wrote: > Silly question maybe, but do you have any logging in your ACLs? If not, > that first bug sounds possible. I’ve got a 2821 running 12.4(25f), doing > NAT overload with heavy QOS and policy routing, get about 99% route-cache > in both directions. Which is similar to your config when inspection is > off. IOS issue seems plausible.**** > > ** ** > > Chuck**** > > ** ** > > *From:* Jmail Clist [mailto:[email protected]] > *Sent:* Friday, December 23, 2011 4:41 PM > *To:* Reuben Farrelly > *Cc:* Chuck Church; [email protected] > > *Subject:* Re: [c-nsp] Cisco 2811 performance issue - dual(new) isp**** > > ** ** > > After running for most of the days, things are back to getting mainly > process switched. ?? Strange.**** > > **** > > rtr2811#sh int fa0/1 stats > FastEthernet0/1 > Switching path Pkts In Chars In Pkts Out Chars Out > Processor 3366529 213364344 66121 21868973 > Route cache 57045 40344237 50866 11970836 > Total 3423574 253708581 116987 33839809 > > **** > > On Fri, Dec 23, 2011 at 9:45 AM, Jmail Clist <[email protected]> wrote:** > ** > > > > That cef command was pretty useful. Before you scroll down to the > output/stats, here are the only two **** > > bugs that look like they might be related to my issue. With test #1, > (everything disabled), it was ALL **** > > process switched. Test #2 looks slightly better with only IP > virtual-reassembly enabled. Something is **** > > going on here and I'm more puzzled than ever. Test #3 caused lots of > process switching when doing the speed tests(???). Test #4 is even more > surprising because things seem better under "normal" traffic loads. > Thoughts?**** > > **** > > I'd like to find a FTP server to test against instead of using speedguide, > speakeasy, etc.**** > > > CSCsa67785 Bug Details > crypto-map/NAT/IPS wont work properly in CEF path > Symptoms: Packets may be dropped on the interface when NAT/IPSEC/IPS is > configured on the same interface. > Conditions: If IPSec/NAT and CBAC or IPS/IDS is configured on the same > interface and the packet gets punted by any of the features, then the > packet > may be dropped. > Workaround: Remove from the configuration the feature which punts the > packet > to process path.**** > > CSCtd25213 Bug Details > NAT not working for locally generated packets > Symptoms: NAT is not working for locally-generated packets. > Conditions: This symptom is observed when NAT is configured for inside and > outside addresses, and when a self-generated packet is sent to OL. > Workaround: Instead of using dynamic NAT, use static NAT for > self-generated > packets. **** > > > 1) disabled cbac/acl and ip virtual-reassembly**** > > interface FastEthernet0/1 > ip address x.x.x.x 255.255.255.0**** > > no ip redirects > ip nat outside**** > > no ip virtual-reassembly > duplex auto > speed auto > end **** > > rtr2811#sh int fa0/1 stats > FastEthernet0/1 > Switching path Pkts In Chars In Pkts Out Chars Out**** > > Processor 12212 757602 133 16723 > Route cache 173 20535 270 35125 > Total 12385 778137 403 51848 > rtr2811#sh ip cef switching statistics feature > IPv4 CEF input features: > Feature Drop Consume Punt Punt2Host Gave > route > NAT Outside 0 0 0 > 25 0 > Total 0 0 0 > 25 0 **** > > IPv4 CEF output features: > Feature Drop Consume Punt Punt2Host New > i/f > Post-routing NAT 0 0 0 > 68 0 > Total 0 0 0 > 68 0**** > > IPv4 CEF post-encap features: > Feature Drop Consume Punt Punt2Host New > i/f > Total 0 0 0 > 0 0**** > > IPv4 CEF for us features: > Feature Drop Consume Punt Punt2Host New > i/f > Total 0 0 0 > 0 0**** > > IPv4 CEF punt features: > Feature Drop Consume Punt Punt2Host New > i/f > Total 0 0 0 > 0 0**** > > IPv4 CEF local features: > Feature Drop Consume Punt Punt2Host Gave > route > Total 0 0 0 > 0 0 > rtr2811#**** > > > 2) enabled ip virtual-reassembly ONLY **** > > > interface FastEthernet0/1 > ip address x.x.x.x 255.255.255.0**** > > no ip redirects > ip nat outside**** > > ip virtual-reassembly > duplex auto > speed auto**** > > end **** > > rtr2811#sh int fa0/1 stats > FastEthernet0/1 > Switching path Pkts In Chars In Pkts Out Chars Out**** > > Processor 1277 78657 16 1589 > Route cache 14 3851 32 4087 > Total 1291 82508 48 5676 > rtr2811#sh ip cef switching statistics feature > IPv4 CEF input features: > Feature Drop Consume Punt Punt2Host Gave > route > NAT Outside 0 0 0 > 1 0 > Total 0 0 0 > 1 0 **** > > IPv4 CEF output features: > Feature Drop Consume Punt Punt2Host New > i/f > Post-routing NAT 0 0 0 > 12 0 > Total 0 0 0 > 12 0**** > > IPv4 CEF post-encap features: > Feature Drop Consume Punt Punt2Host New > i/f > Total 0 0 0 > 0 0**** > > IPv4 CEF for us features: > Feature Drop Consume Punt Punt2Host New > i/f > Total 0 0 0 > 0 0**** > > IPv4 CEF punt features: > Feature Drop Consume Punt Punt2Host New > i/f > Total 0 0 0 > 0 0**** > > IPv4 CEF local features: > Feature Drop Consume Punt Punt2Host Gave > route > Total 0 0 0 > 0 0 > rtr2811#**** > > > NOTE: After this I enabled CBAC-int & Ext_ACL-inbound again. Performance > was almost good as #2 still. I **** > > also cleared counters once more and waited 10 minutes. Here are the > results again. Any ideas????**** > > > 3) I ran a speedtest on www.speakeasy.net and process switching went > through the roo**** > > rtr2811#sh int fa0/1 stats > FastEthernet0/1 > Switching path Pkts In Chars In Pkts Out Chars Out**** > > Processor 17858 1157573 467 143934 > Route cache 1072 964530 837 98966 > Total 18930 2122103 1304 242900 > rtr2811# > rtr2811#running speedtest now > ^ > % Invalid input detected at '^' marker. **** > > rtr2811#sh int fa0/1 stats > FastEthernet0/1 > Switching path Pkts In Chars In Pkts Out Chars Out**** > > Processor 21414 1379133 507 159277 > Route cache 10317 10944391 8426 7415536 > Total 31731 12323524 8933 7574813 **** > > > rtr2811#sh int fa0/1 stats > FastEthernet0/1 > Switching path Pkts In Chars In Pkts Out Chars Out**** > > Processor 21490 1384753 513 162841 > Route cache 10322 10946281 8426 7415536 > Total 31812 12331034 8939 7578377 > rtr2811# **** > > 4) cleared counters one last time and let it from midnight to 9:39am**** > > rtr2811#sh int fa0/1 stats > FastEthernet0/1 > Switching path Pkts In Chars In Pkts Out Chars Out**** > > Processor 2091010 132620733 42136 13987400 > Route cache 42156 32749186 36559 10473996 > Total 2133166 165369919 78695 24461396 > rtr2811#sh ip cef switching statistics feature > IPv4 CEF input features: > Feature Drop Consume Punt Punt2Host Gave > route > Access List 11840 0 0 > 13286 0 > NAT Outside 0 0 0 > 3389 0 > Total 11840 0 0 > 16675 0 **** > > IPv4 CEF output features: > Feature Drop Consume Punt Punt2Host New > i/f > Post-routing NAT 0 0 0 > 28310 0 > Firewall (inspec 57 0 0 > 13 0 > Total 57 0 0 > 28323 0**** > > IPv4 CEF post-encap features: > Feature Drop Consume Punt Punt2Host New > i/f > Total 0 0 0 > 0 0**** > > IPv4 CEF for us features: > Feature Drop Consume Punt Punt2Host New > i/f > Total 0 0 0 > 0 0**** > > IPv4 CEF punt features: > Feature Drop Consume Punt Punt2Host New > i/f > Total 0 0 0 > 0 0**** > > IPv4 CEF local features: > Feature Drop Consume Punt Punt2Host Gave > route > Total 0 0 0 > 0 0 > rtr2811#**** > > On Thu, Dec 22, 2011 at 4:24 PM, Reuben Farrelly < > [email protected]> wrote:**** > > The command: > > router#show ip cef switching statistics feature > > Will show you which feature is causing traffic to be punted to CPU. > > Reuben **** > > > > > On 23/12/2011 7:42 AM, Chuck Church wrote:**** > > You're on the right path. The more important number is the packets in/out, > as opposed to the characters. Look at the ratio of packets in/out for > processor vs. Route-cache for the two interfaces. Fa0/1 is process > switching about 80% of them inbound. That's pretty bad. The output > looks > better. Compare that to VLAN 10, where in both directions, only about 10% > are process switched. The stats for the switchports are meaningless, so > you > can ignore those as the switch ASICs deal with those, until they hit the > VLAN int. Figure out what feature (or IOS bug??) is causing so much > process > switching, and I think it'll get better.**** > > ** ** > > ** ** > _______________________________________________ cisco-nsp mailing list [email protected] https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
