The router is process switching (and dropping) a lot of packets. I'd try it without CBAC and with ip virtual-reassembly disabled. See if you can get it to work at close to wirerate before re-enabling features. Might be worth trying a recent mainline 15.0 release, I seem to remember lots of weirdness on an 870 ISR running 12.4(24)T.
Thanks, Chuck From: Jmail Clist [mailto:[email protected]] Sent: Thursday, December 22, 2011 12:09 PM To: Chuck Church Cc: [email protected]; [email protected] Subject: Re: [c-nsp] Cisco 2811 performance issue - dual(new) isp The fa0/1 interface literally plugs into the cable modem for the ISP. Do you think I should increase the buffer size a bit for that interface? I'm sure there is a command for that. I added another "sh int" and the full "sh buffer" output near the bottom of this message. #sh ver Cisco IOS Software, 2800 Software (C2800NM-ADVENTERPRISEK9-M), Version 12.4(24)T FastEthernet0/1 buffers, 1536 bytes (total 384, permanent 384): 0 in free list (0 min, 384 max allowed) 384 hits, 0 fallbacks 384 max cache size, 256 in cache 29489950 hits in cache, 0 misses in cache interface FastEthernet0/1 ip address x.x.x.x 255.255.255.0 ip access-group BLOCK in no ip redirects ip nat outside ip inspect ISP2-cbac out ip virtual-reassembly duplex auto speed auto sh int fa0/1 FastEthernet0/1 is up, line protocol is up Hardware is MV96340 Ethernet, address is 0015.f956.d549 (bia 0015.f956.d549) Internet address is x.x.x.x/24 MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex, 100Mb/s, 100BaseTX/FX ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:00:00, output 00:00:00, output hang never Last clearing of "show interface" counters 1d12h Input queue: 0/75/167/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 267000 bits/sec, 84 packets/sec 5 minute output rate 49000 bits/sec, 24 packets/sec 9017021 packets input, 2522442236 bytes Received 7072336 broadcasts, 0 runts, 0 giants, 18 throttles 1934 input errors, 0 CRC, 0 frame, 0 overrun, 1934 ignored 0 watchdog 0 input packets with dribble condition detected 1425142 packets output, 377562825 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 unknown protocol drops 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier 0 output buffer failures, 0 output buffers swapped out rtr2811#sh buffer Buffer elements: 687 in free list (1119 max allowed) 80677703 hits, 0 misses, 619 created Public buffer pools: Small buffers, 104 bytes (total 106, permanent 50, peak 248 @ 1d02h): 86 in free list (20 min, 150 max allowed) 670704026 hits, 5941 misses, 4870 trims, 4926 created 1577 failures (0 no memory) Middle buffers, 600 bytes (total 41, permanent 25, peak 262 @ 7w0d): 39 in free list (10 min, 150 max allowed) 547716007 hits, 1867 misses, 3051 trims, 3067 created 421 failures (0 no memory) Big buffers, 1536 bytes (total 52, permanent 50, peak 122 @ 2d01h): 35 in free list (5 min, 150 max allowed) 6473777 hits, 98 misses, 133 trims, 135 created 36 failures (0 no memory) VeryBig buffers, 4520 bytes (total 10, permanent 10, peak 12 @ 19:59:39): 10 in free list (0 min, 100 max allowed) 26150 hits, 7 misses, 2 trims, 2 created 6 failures (0 no memory) Large buffers, 5024 bytes (total 1, permanent 0, peak 1 @ 1d00h): 1 in free list (0 min, 10 max allowed) 0 hits, 6 misses, 9 trims, 10 created 6 failures (0 no memory) Huge buffers, 18024 bytes (total 1, permanent 0, peak 1 @ 1d00h): 1 in free list (0 min, 4 max allowed) 0 hits, 6 misses, 9 trims, 10 created 6 failures (0 no memory) Interface buffer pools: Onboard DSPRM Pool buffers, 264 bytes (total 300, permanent 300): 0 in free list (0 min, 300 max allowed) 300 hits, 0 fallbacks 300 max cache size, 300 in cache 991346 hits in cache, 0 misses in cache Syslog ED Pool buffers, 600 bytes (total 132, permanent 132): 100 in free list (132 min, 132 max allowed) 532 hits, 0 misses CD2430 I/O buffers, 1536 bytes (total 0, permanent 0): 0 in free list (0 min, 0 max allowed) 0 hits, 0 fallbacks IPC buffers, 4096 bytes (total 140, permanent 140): 140 in free list (70 min, 280 max allowed) 0 hits, 0 fallbacks, 0 trims, 0 created 0 failures (0 no memory) Header pools: Header buffers, 0 bytes (total 768, permanent 768): 256 in free list (128 min, 1024 max allowed) 512 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) 512 max cache size, 512 in cache 103 hits in cache, 0 misses in cache C5510 Header Pool buffers, 0 bytes (total 768, permanent 768): 256 in free list (128 min, 1024 max allowed) 512 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) 512 max cache size, 512 in cache 986899 hits in cache, 0 misses in cache Particle Clones: 1024 clones, 101871 hits, 0 misses Public particle pools: F/S buffers, 256 bytes (total 768, permanent 768): 256 in free list (128 min, 1024 max allowed) 512 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) 512 max cache size, 512 in cache 0 hits in cache, 0 misses in cache Normal buffers, 1548 bytes (total 768, permanent 768): 768 in free list (128 min, 1024 max allowed) 0 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) On Wed, Dec 21, 2011 at 3:04 PM, Chuck Church <[email protected]> wrote: Hmmm. Well, there are a few variables. If one site does give you good results, then the router might not be totally at fault. You are getting 'ignore' errors on the interface with CBAC enabled, that's definitely slowing things down, as you're getting re-transmits and TCP window starting small again. Just curious, what does 'sh buffer' output look like? Thanks, Chuck From: Jmail Clist [mailto:[email protected]] Sent: Tuesday, December 20, 2011 11:43 PM To: Chuck Church Cc: [email protected] Subject: Re: [c-nsp] Cisco 2811 performance issue - dual(new) isp Chuck, Interesting. Not sure why it was so low. I switched back to this new ISP conn on fa0/1 tonight and ran some more tests. Below is various output immediately after testing. The second set of show outputs is after I applied CBAC inbound and a generic deny extended access list outbound on fa0/1. The CBAC is definitely raising my cpu, as expected. Performance is still low in my opinion, at least with testing on most "test" sites. The only site that gave me great results was speakeasy.net. rtr2811#sh int switching | begin FastEthernet0/1 FastEthernet0/1 Throttle count 11 Drops RP 11 SP 0 SPD Flushes Fast 0 SSE 0 SPD Aggress Fast 0 SPD Priority Inputs 20030942 Drops 0 Protocol IP Switching path Pkts In Chars In Pkts Out Chars Out Process 66120 22268396 37079 4417563 Cache misses 0 - - - Fast 410053 477119555 351638 183218275 Auton/SSE 0 0 0 0 Protocol DEC MOP Switching path Pkts In Chars In Pkts Out Chars Out Process 0 0 8697 669669 Cache misses 0 - - - Fast 0 0 0 0 Auton/SSE 0 0 0 0 Protocol ARP Switching path Pkts In Chars In Pkts Out Chars Out rtr2811#sh int fa0/1 FastEthernet0/1 is up, line protocol is up Hardware is MV96340 Ethernet, address is 0015.f956.d549 (bia 0015.f956.d549) Internet address is 200.200.200.200/24 MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex, 100Mb/s, 100BaseTX/FX ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:00:00, output 00:00:02, output hang never Last clearing of "show interface" counters 00:01:49 Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 731000 bits/sec, 108 packets/sec 5 minute output rate 357000 bits/sec, 45 packets/sec 17949 packets input, 14940931 bytes Received 5515 broadcasts, 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog 0 input packets with dribble condition detected 11012 packets output, 9300361 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets rtr2811#sh proc cpu sorted 1min CPU utilization for five seconds: 9%/1%; one minute: 14%; five minutes: 13% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 80 <tel:80%C2%A0%C2%A0%C2%A0%2097347040> 97347040 262361459 371 1.75% 1.77% 1.76% 0 IGMP Snooping Re 118 84936308 283025140 300 1.67% 1.54% 1.52% 0 IP Input 19 9391432 30838598 304 0.31% 1.20% 1.28% 0 ARP Input 182 392060 1300284614 0 1.03% 1.12% 1.12% 0 HQF Shaper Backg 92 19052984 60835327 313 0.39% 0.49% 0.50% 0 ILPM 3 2197644 20210291 108 0.23% 0.31% 0.31% 0 Skinny Msg Serve 314 169248 163513044 1 0.31% 0.30% 0.31% 0 PPP manager 125 1464 1486 985 0.47% 0.16% 0.12% 514 SSH Process 5 11185972 797585 14024 0.00% 0.15% 0.17% 0 Check heaps 315 88332 163513044 0 0.15% 0.14% 0.15% 0 PPP Events 91 6798308 5230434 1299 0.07% 0.12% 0.13% 0 tCOUNTER --More-- /////////////////////////////////////////////////////// After CBAC applied outbound/extended deny all access-list inbound FastEthernet0/1 is up, line protocol is up Hardware is MV96340 Ethernet, address is 0015.f956.d549 (bia 0015.f956.d549) Internet address is 200.200.200.200/24 MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec, reliability 255/255, txload 1/255, rxload 3/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex, 100Mb/s, 100BaseTX/FX ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:00:00, output 00:00:00, output hang never Last clearing of "show interface" counters 00:16:44 Input queue: 1/75/75/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 1445000 bits/sec, 183 packets/sec 5 minute output rate 527000 bits/sec, 94 packets/sec 162975 packets input, 141570515 bytes Received 51776 broadcasts, 0 runts, 0 giants, 6 throttles 379 input errors, 0 CRC, 0 frame, 0 overrun, 379 ignored 0 watchdog 0 input packets with dribble condition detected 80662 packets output, 52976137 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets ------ sh proc cpu sorted 1min CPU utilization for five seconds: 11%/2%; one minute: 20%; five minutes: 19% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 118 84950740 283074993 300 1.75% 1.81% 1.78% 0 IP Input 80 <tel:80%C2%A0%C2%A0%C2%A0%2097371132> 97371132 262405939 371 1.83% 1.79% 1.77% 0 IGMP Snooping Re 19 9408864 30883468 304 1.35% 1.29% 1.46% 0 ARP Input 182 392968 1300500059 0 1.11% 1.12% 1.12% 0 HQF Shaper Backg 321 3856440 2264838 1702 0.00% 0.86% 0.78% 0 SNMP ENGINE 92 19056692 60845592 313 0.39% 0.48% 0.50% 0 ILPM 3 2198364 20213772 108 0.31% 0.32% 0.31% 0 Skinny Msg Serve 314 169656 163540744 1 0.31% 0.30% 0.31% 0 PPP manager 319 2108780 4252988 495 0.07% 0.24% 0.16% 0 IP SNMP 5 11188032 797728 14024 0.00% 0.22% 0.19% 0 Check heaps sh ip traffic FastEthernet0/1 Throttle count 17 Drops RP 86 SP 0 SPD Flushes Fast 0 SSE 0 SPD Aggress Fast 0 SPD Priority Inputs 20080830 Drops 0 Protocol IP Switching path Pkts In Chars In Pkts Out Chars Out Process 66747 22533286 37856 4562880 Cache misses 0 - - - Fast 509124 600798823 421307 227932429 Auton/SSE 0 0 0 0 Protocol DEC MOP Switching path Pkts In Chars In Pkts Out Chars Out Process 0 0 8698 669746 Cache misses 0 - - - Fast 0 0 0 0 Auton/SSE 0 0 0 0 Protocol ARP Switching path Pkts In Chars In Pkts Out Chars Out On Tue, Dec 20, 2011 at 12:51 PM, Chuck Church <[email protected]> wrote: Are you sure your NAT/PAT statements are right? I think you said in an earlier email this output below was done right after the speedtest. However, the 5 minute load interval for fa0/1 shows only 1 pps output on this interface. The input packets look like all ARP, based on the int switching info. Is your traffic really going out the original circuit? Chuck -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Jmail Clist Sent: Tuesday, December 20, 2011 9:58 AM To: Daniel Hooper Cc: [email protected] Subject: Re: [c-nsp] Cisco 2811 performance issue - dual(new) isp Here is a show proc, show ip traffic and show int switching. Any feedback is very much appreciated. rtr2811#show proc cpu sorted 1min CPU utilization for five seconds: 26%/17%; one minute: 13%; five minutes: 13% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 80 95570780 <tel:80%20%20%20%2095570780> 257773304 370 1.83% 1.75% 1.76% 0 IGMP Snooping Re 118 83345888 278039819 299 1.51% 1.59% 1.67% 0 IP Input 19 8380332 26051685 321 2.31% 1.33% 1.34% 0 ARP Input 182 382324 1277585313 0 1.35% 1.14% 1.12% 0 HQF Shaper Backg 92 18735164 59773255 313 0.63% 0.51% 0.50% 0 ILPM 125 10440 8415 1240 0.00% 0.45% 0.73% 514 SSH Process 3 2156048 19855473 108 0.31% 0.31% 0.31% 0 Skinny Msg Serve 314 166160 160658064 1 0.31% 0.31% 0.31% 0 PPP manager 5 10986872 783696 14019 0.00% 0.21% 0.18% 0 Check heaps 315 86412 160658064 0 0.15% 0.14% 0.15% 0 PPP Events 91 6677076 5139100 1299 0.15% 0.12% 0.13% 0 tCOUNTER -------------------- rtr2811#sh ip traffic IP statistics: Rcvd: 5677 total, 231 local destination 0 format errors, 0 checksum errors, 7 bad hop count 0 unknown protocol, 5358 not a gateway 0 security failures, 0 bad options, 0 with options Opts: 0 end, 0 nop, 0 basic security, 0 loose source route 0 timestamp, 0 extended security, 0 record route 0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump 0 other Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble 0 fragmented, 0 fragments, 0 couldn't fragment Bcast: 33 received, 0 sent Mcast: 0 received, 21 sent Sent: 225 generated, 11351 forwarded Drop: 3 encapsulation failed, 0 unresolved, 0 no adjacency 0 no route, 0 unicast RPF, 0 forced drop 0 options denied Drop: 0 packets with source IP address zero Drop: 0 packets with internal loop back IP address 0 physical broadcast ICMP statistics: Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 0 unreachable 60 echo, 0 echo reply, 0 mask requests, 0 mask replies, 0 quench 0 parameter, 0 timestamp, 0 timestamp replies, 0 info request, 0 other 0 irdp solicitations, 0 irdp advertisements 0 time exceeded, 0 info replies Sent: 32 redirects, 0 unreachable, 0 echo, 61 echo reply 0 mask requests, 0 mask replies, 0 quench, 0 timestamp, 0 timestamp replies 0 info reply, 0 time exceeded, 0 parameter problem 0 irdp solicitations, 0 irdp advertisements UDP statistics: Rcvd: 40 total, 0 checksum errors, 34 no port Sent: 23 total, 0 forwarded broadcasts TCP statistics: Rcvd: 135 total, 0 checksum errors, 0 no port Sent: 116 total BGP statistics: Rcvd: 0 total, 0 opens, 0 notifications, 0 updates 0 keepalives, 0 route-refresh, 0 unrecognized Sent: 0 total, 0 opens, 0 notifications, 0 updates 0 keepalives, 0 route-refresh IP-EIGRP statistics: Rcvd: 0 total Sent: 0 total PIMv2 statistics: Sent/Received Total: 0/0, 0 checksum errors, 0 format errors Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0, Hellos: 0/0 Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0 Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0 Queue drops: 0 State-Refresh: 0/0 IGMP statistics: Sent/Received Total: 0/0, Format errors: 0/0, Checksum errors: 0/0 Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0 DVMRP: 0/0, PIM: 0/0 Queue drops: 0 OSPF statistics: Rcvd: 0 total, 0 checksum errors 0 hello, 0 database desc, 0 link state req 0 link state updates, 0 link state acks Sent: 0 total 0 hello, 0 database desc, 0 link state req 0 link state updates, 0 link state acks ARP statistics: Rcvd: 2916 requests, 0 replies, 0 reverse, 0 other Sent: 2 requests, 5 replies (0 proxy), 0 reverse Drop due to input queue full: 0 rtr2811# ------------------ FastEthernet0/1 is up, line protocol is up Hardware is MV96340 Ethernet, address is 0015.f956.d549 (bia 0015.f956.d549) Internet address is 200.200.200.200/24 MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex, 100Mb/s, 100BaseTX/FX ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:00:00, output 00:00:02, output hang never Last clearing of "show interface" counters 00:04:13 Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 115000 bits/sec, 64 packets/sec 5 minute output rate 18000 bits/sec, 1 packets/sec 24064 packets input, 9171019 bytes Received 17645 broadcasts, 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog 0 input packets with dribble condition detected 4516 packets output, 2399483 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets ------------------------ sh int switching FastEthernet0/1 Throttle count 10 Drops RP 10 SP 0 SPD Flushes Fast 0 SSE 0 SPD Aggress Fast 0 SPD Priority Inputs 15397736 Drops 0 Protocol IP Switching path Pkts In Chars In Pkts Out Chars Out Process 49490 16769897 30321 3863531 Cache misses 0 - - - Fast 249838 281487351 229388 101233546 Auton/SSE 0 0 0 0 Protocol DEC MOP Switching path Pkts In Chars In Pkts Out Chars Out Process 0 0 8548 658196 Cache misses 0 - - - Fast 0 0 0 0 Auton/SSE 0 0 0 0 Protocol ARP Switching path Pkts In Chars In Pkts Out Chars Out Process 15397767 923866020 919 55140 Cache misses 0 - - - Fast 0 0 0 0 Auton/SSE 0 0 0 0 Protocol CDP Switching path Pkts In Chars In Pkts Out Chars Out Process 0 0 4675 1907390 Cache misses 0 - - - Fast 0 0 0 0 Auton/SSE 0 0 0 0 Protocol Other Switching path Pkts In Chars In Pkts Out Chars Out Process 0 0 514555 30873300 Cache misses 0 - - - Fast 0 0 0 0 Auton/SSE 0 0 0 0 NOTE: all counts are cumulative and reset only after a reload. On Tue, Dec 20, 2011 at 2:03 AM, Daniel Hooper <[email protected]> wrote: > I have no faith in sites like speedtest.net actually reporting the true > speed of your link. > > Use your ISP's local FTP site & mirror sites or find one close to them > upstream and perform testing. > > -Dan > > -----Original Message----- > From: [email protected] [mailto: > [email protected]] On Behalf Of Jmail Clist > Sent: Tuesday, 20 December 2011 11:42 AM > To: [email protected] > Subject: Re: [c-nsp] Cisco 2811 performance issue - dual(new) isp > > I have attached another log called "latest_2811_logs". it contains the "sh > ip traffic, sh int switching and a "show proc cpu sorted 1min" output > immediately after doing a test on speedguide.net through the new isp2 > connection. I see some throttle drops but not much more. I'm still puzzled > at the poor performance. I put the interface back to auto as well and teh > results are the same. Any ideas? > > On Mon, Dec 19, 2011 at 9:03 PM, Christopher J. Wargaski > <[email protected]>wrote: > > > According to > > http://www.cisco.com/web/partners/downloads/765/tools/quickreference/r > > outerperformance.pdf, the router can handle up to 16.44 Mbps of fast / > > CEF switching. I was not able to see your attachments in the digest > > nor in the archive. Do you have CEF enabled? > > > > cjw > > > > > > Date: Mon, 19 Dec 2011 10:57:11 -0600 > >> From: Jmail Clist <[email protected]> > >> > >> To: [email protected] > >> Subject: [c-nsp] Cisco 2811 performance issue - dual(new) isp > >> Message-ID: > >> <CAO8NJwLyKmQj8jUJTYg2_82fyxyyQLRuCOraaeH3BG9ONPr8= > >> [email protected]> > >> Content-Type: text/plain; charset="iso-8859-1" > >> > >> > >> Hello, > >> > >> I have an issue that is really causing me grief. I recentely > >> inherited a small network. There is an existing 1.5mbps Internet > >> connection (fa0/0) (includes MPLS as well/same provider). We added a > >> new ISP that allows for 50mb down/5mb up. I added the new ISP to > >> fa0/1 and modified the NAT overload statements accordingly. I alo > >> changed the default route to ONLY use the new, faster ISP connection. > >> Using speedguide.net, I am only able to get 6 to 10mb down, most of > >> the time. if I plug a laptop into the cable modem then I get 37 to > >> 50mb down. Any idea why the 2811 is so slow? How much download speed > >> can I expect to get? Any assisstance would be very much appreciated. > >> > >> I have attached the config and various show outputs (nat, sh ver, > >> memory, etc.). > >> > >> Thank you, > >> > > > _______________________________________________ cisco-nsp mailing list [email protected] https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ _______________________________________________ cisco-nsp mailing list [email protected] https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
