RE: Rate Limiting on Cisco Router
Definitely worth the try. Your biggest enemy may be 12.4 IOS. It's bloated and buggy in my experience, but that has mostly been edge services. If 12.4 pegs your processor, you may want to check the software/hardware matrix and see if one of the older 12.0/2 service provider trains that they continued to add support for (probably some large customer's special requests). I don't know if it will support the G1, but if so, you might have better performance out of it. Jack We've implemented 400Mbps shaping with over twenty nested child policies (individual customer shaping and queuing within the 400M) on an NPE-G1 running 12.4(12c). CPU does start to become an issue at that point, and by removing the policy we can reach nearly 600Mbps on the same kit. We run standard ACL, OSPF, EIGRP, VRF Selection, MPLS, MP-BGP, etc. but do not run a full Internet BGP feed on these boxes, so you'll need to subtract that process usage if it applies. I would note that upgrading the box to 12.2(33)SRC caused a 20%+ increase in CPU attributed to the HQF (Hierarchical QOS Framework) process. We decided to stay with the 12.4 train. Cory Ayers CCIE #16874 (RS), CCIP Director of Network Strategy Education Networks of America
RE: Rate Limiting on Cisco Router
Pretty funny and good stuffsince no one really acheives true 100MB speeds anyways, then a 100MB port might actually traffic shape itself naturally!!! I forget what the actual speeds truly are... is it 80% advertised speeds? I'm not sure which is cheaper but I think Juniper has some low end Netscreens you can try also that have traffic shaping features. Subject: RE: Rate Limiting on Cisco Router From: gordsla...@ieee.org To: brandon@brandontek.com CC: nanog@nanog.org Date: Fri, 9 Jul 2010 06:33:04 +0100 On Thu, 2010-07-08 at 20:01 -0400, Brandon Kim wrote: What about purchasing a low-end packetshaper to be used in between? If - 1/ budget is a problem and 2/ you have no BSD knowledge inhouse and 3/ the LAN side is all ethernet you could have a stab at using a PFsense box with two (and strictly ONLY two, for this use) physical NICs. It has a GUI to set up traffic shaping (see the sticky on the pfsense forums) PFsense 1.2.3 is current, don't go for the experimental 2.0 for production. There's a book and commercial support if you need it, free support via forums if you can't. Only two physical NICs is necessary due to shaper problems with more than two, whereas in a firewalling role the slots are the only limit (but VLANS are the norm for bucketloads of ports on a firewall PFsense box) An ITX (Littlefalls etc) mobo with 512MB RAM with an extra PCI Intel NIC added will do you fine .. PFsense has nice traffic graphs, which helps you with shaping speeds in a big way. It also has a TFTP server available for it so it's handy for unmanned sites with only a few blue boxes ;) PS - a crazy afterthough - surely just about anything with a 10/100 ethernet link running at 100 and placed inline, cannot exceed 100Mbps - and probably less if it's plastic-cased? Try a few 8-port junkers and see what happens if you fancy a walk on the dangerous side. Watch out for errors and smoke :) Gord -- The drinker you are the smoker you get
Re: Rate Limiting on Cisco Router
On Thu, 8 Jul 2010, Alan Bryant wrote: We have tried the rate-limit command with various parameters and we are unable to keep it at 80. I have read that this is not the correct way to do it, but I'm not sure what is. What burst parameters are you using? Try something along the lines of: rate-limit input 8000 1500 1500 conform-action transmit exceed-action drop rate-limit output 8000 1500 1500 conform-action transmit exceed-action drop on your OC3 interface. Antonio Querubin 808-545-5282 x3003 e-mail/xmpp: t...@lava.net
Re: Rate Limiting on Cisco Router
On Thu, 8 Jul 2010, Alan Bryant wrote: The problem we have now is that we are only paying for 80 MB/s of the OC-3, and the ISP is leaving the capping of it up to us. I have BTW, rate-limiting of traffic that the ISP router sends to your router is best done at the ISP router. Antonio Querubin 808-545-5282 x3003 e-mail/xmpp: t...@lava.net
re: Rate Limiting on Cisco Router
That's strange, Are you paying for a CIR of 80Mb/s? Normally they only leave the limiting up to you if its more of a burstable connection, Like you pay for 80Mb/s but its a full line rate interface and its billed per Mb/s over 80 on a 95th percentile scheme. If that is the case you can safely go over 80Mb/s for a certian amount of time per month, Assuming its billed monthly. Nick Olsen Network Operations (321) 205-1100 x106 From: Alan Bryant a...@gtekcommunications.com Sent: Thursday, July 08, 2010 6:07 PM To: nanog@nanog.org Subject: Rate Limiting on Cisco Router Thanks again for all the responses to my previous post. We have a Cisco 7206VXR router with IOS of 12.4(12) and a PA-POS-1OC3 card ofr our OC3. The problem we have now is that we are only paying for 80 MB/s of the OC-3, and the ISP is leaving the capping of it up to us. I have googled and the only things I can find is that you can not do a real cap on this type of interface. We have tried the rate-limit command with various parameters and we are unable to keep it at 80. I have read that this is not the correct way to do it, but I'm not sure what is. Any advice? Pointers appreciated. -- Alan Bryant | Systems Administrator Gtek Computers Wireless, LLC. a...@gtekcommunications.com | www.gtek.biz O 361-777-1400 | F 361-777-1405
RE: Rate Limiting on Cisco Router
traffic-shape rate 7500 9000 9000 1000 for example. Your rate limit will police your traffic and drop it all. Traffic shaping produces a queue, and does not completely junk a packet. It becomes q'd, and produces a smoother output. ~Jay Murphy IP Network Specialist NM State Government IT Services Division PSB – IP Network Management Center Santa Fé, New México 87505 We move the information that moves your world. “Good engineering demands that we understand what we’re doing and why, keep an open mind, and learn from experience.” “Engineering is about finding the sweet spot between what's solvable and what isn't. Radia Perlman Please consider the environment before printing e-mail -Original Message- From: Antonio Querubin [mailto:t...@lava.net] Sent: Thursday, July 08, 2010 4:26 PM To: Alan Bryant Cc: nanog@nanog.org Subject: Re: Rate Limiting on Cisco Router On Thu, 8 Jul 2010, Alan Bryant wrote: The problem we have now is that we are only paying for 80 MB/s of the OC-3, and the ISP is leaving the capping of it up to us. I have BTW, rate-limiting of traffic that the ISP router sends to your router is best done at the ISP router. Antonio Querubin 808-545-5282 x3003 e-mail/xmpp: t...@lava.net Confidentiality Notice: This e-mail, including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. -- This email has been scanned by the Sybari - Antigen Email System.
Re: Rate Limiting on Cisco Router
I think if you try to traffic-shape 80Mbps on that platform you'll have problems. We have a 7200 with NPE-G1 (rate limited at 80Mbps) and it killed the CPU when the threshold was hit. I imagine that traffic-shaping would do the same to CPU and memory. I'd lab it first. Kenny On Thu, Jul 8, 2010 at 3:43 PM, Murphy, Jay, DOH jay.mur...@state.nm.uswrote: traffic-shape rate 7500 9000 9000 1000 for example. Your rate limit will police your traffic and drop it all. Traffic shaping produces a queue, and does not completely junk a packet. It becomes q'd, and produces a smoother output. ~Jay Murphy IP Network Specialist NM State Government IT Services Division PSB – IP Network Management Center Santa Fé, New México 87505 We move the information that moves your world. “Good engineering demands that we understand what we’re doing and why, keep an open mind, and learn from experience.” “Engineering is about finding the sweet spot between what's solvable and what isn't. Radia Perlman Please consider the environment before printing e-mail -Original Message- From: Antonio Querubin [mailto:t...@lava.net] Sent: Thursday, July 08, 2010 4:26 PM To: Alan Bryant Cc: nanog@nanog.org Subject: Re: Rate Limiting on Cisco Router On Thu, 8 Jul 2010, Alan Bryant wrote: The problem we have now is that we are only paying for 80 MB/s of the OC-3, and the ISP is leaving the capping of it up to us. I have BTW, rate-limiting of traffic that the ISP router sends to your router is best done at the ISP router. Antonio Querubin 808-545-5282 x3003 e-mail/xmpp: t...@lava.net Confidentiality Notice: This e-mail, including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. -- This email has been scanned by the Sybari - Antigen Email System.
Re: Rate Limiting on Cisco Router
Agree...when you rate limit verse shaping you can actually cause more traffic because the packets need to be retransmitted to deal with those that got dropped. On 07/08/2010 06:43 PM, Murphy, Jay, DOH wrote: traffic-shape rate 7500 9000 9000 1000 for example. Your rate limit will police your traffic and drop it all. Traffic shaping produces a queue, and does not completely junk a packet. It becomes q'd, and produces a smoother output. ~Jay Murphy IP Network Specialist NM State Government
RE: Rate Limiting on Cisco Router
On Thu, 8 Jul 2010, Murphy, Jay, DOH wrote: Traffic shaping produces a queue, and does not completely junk a packet. It becomes q'd, and produces a smoother output. Traffic-shaping 80Mb/s of traffic is probably not a good idea for your router cpu :) Antonio Querubin 808-545-5282 x3003 e-mail/xmpp: t...@lava.net
Re: Rate Limiting on Cisco Router
Antonio Querubin wrote: Traffic-shaping 80Mb/s of traffic is probably not a good idea for your router cpu :) Honestly, cpu overhead shouldn't be an issue with a traffic shape queue. If it is, probably a seriously underpowered router or poor code. Now if you applied extensive rules for various traffic at different rates and queue priorities, I could see lots of extra ticks. Don't get me wrong. I have 400mbps+ traffic shapes on my junipers and probably glad for the extensive hardware support (like I have a choice, no hardware support, the router won't do it) Jack
RE: Rate Limiting on Cisco Router
What about purchasing a low-end packetshaper to be used in between? I know this doesn't answer the question but could it be an option? Date: Thu, 8 Jul 2010 13:43:17 -1000 From: t...@lava.net To: jay.mur...@state.nm.us Subject: RE: Rate Limiting on Cisco Router CC: nanog@nanog.org On Thu, 8 Jul 2010, Murphy, Jay, DOH wrote: Traffic shaping produces a queue, and does not completely junk a packet. It becomes q'd, and produces a smoother output. Traffic-shaping 80Mb/s of traffic is probably not a good idea for your router cpu :) Antonio Querubin 808-545-5282 x3003 e-mail/xmpp: t...@lava.net
Re: Rate Limiting on Cisco Router
On Thu, 2010-07-08 at 16:35 -0700, Kenny Sallee wrote: I think if you try to traffic-shape 80Mbps on that platform you'll have problems. We have a 7200 with NPE-G1 (rate limited at 80Mbps) and it killed the CPU when the threshold was hit. I imagine that traffic-shaping would do the same to CPU and memory. I'd lab it first. I've seen that model preceded by a BSD machine with 2 physical ethernet NICs. When I asked - limiting for the 7206's outgoing, so I'm assuming that was a CPU thing. In that case the 7206 was just an edge box for the fibre, so doing nothing complex. Capped at 48Mbps (IIRC) in that case - YMMV. Also bear in mind that this is borderline black art - it needs a bit of testing to be sure it's working as you expect :) My usual technique is to replay some flows then set several iperf streams going simultaneously to see how it reacts. Sometimes limiting just seems to temporarily break down under stress in bizarre ways. Whether it fails open, restricted or closed seems to be very unpredictable and not very reproducible on some kit- keep your eye on it at first, or use BSD to do it if you're more familiar with that. Gord -- Awake! for morning in the bowl of light has flung the stone that puts the stars to flight
Re: Rate Limiting on Cisco Router
On Thu, 2010-07-08 at 18:54 -0500, Jack Bates wrote: underpowered router or poor code Agreed. So which is it? :) To be fair, some IOS versions were better than others at it in my limited experience of that chassis. Gord -- I hold you XAP
Re: Rate Limiting on Cisco Router
So you guys would not recommend the traffic shaping route on a 7206 with a NPE-G1? Is it the processor or memory that would not be able to handle it? I don't necessarily plan on doing anything other than limiting it at 80Mbps or whatever it is that we are capping ourselves at at the time. On Thu, Jul 8, 2010 at 7:13 PM, gordon b slater gordsla...@ieee.org wrote: On Thu, 2010-07-08 at 18:54 -0500, Jack Bates wrote: underpowered router or poor code Agreed. So which is it? :) To be fair, some IOS versions were better than others at it in my limited experience of that chassis. Gord -- I hold you XAP -- Alan Bryant | Systems Administrator Gtek Computers Wireless, LLC. a...@gtekcommunications.com | www.gtek.biz O 361-777-1400 | F 361-777-1405
Re: Rate Limiting on Cisco Router
Also, are there any upgrades that can be done to this router to increase it's processing power? Is there something better for the 7206VXR than the NPE-G1? I see the NPE-G2, but even on ebay it is very costly. On Thu, Jul 8, 2010 at 8:32 PM, Alan Bryant a...@gtekcommunications.com wrote: So you guys would not recommend the traffic shaping route on a 7206 with a NPE-G1? Is it the processor or memory that would not be able to handle it? I don't necessarily plan on doing anything other than limiting it at 80Mbps or whatever it is that we are capping ourselves at at the time. On Thu, Jul 8, 2010 at 7:13 PM, gordon b slater gordsla...@ieee.org wrote: On Thu, 2010-07-08 at 18:54 -0500, Jack Bates wrote: underpowered router or poor code Agreed. So which is it? :) To be fair, some IOS versions were better than others at it in my limited experience of that chassis. Gord -- I hold you XAP -- Alan Bryant | Systems Administrator Gtek Computers Wireless, LLC. a...@gtekcommunications.com | www.gtek.biz O 361-777-1400 | F 361-777-1405 -- Alan Bryant | Systems Administrator Gtek Computers Wireless, LLC. a...@gtekcommunications.com | www.gtek.biz O 361-777-1400 | F 361-777-1405
Re: Rate Limiting on Cisco Router
On 7/8/2010 18:40, Alan Bryant wrote: Also, are there any upgrades that can be done to this router to increase it's processing power? Is there something better for the 7206VXR than the NPE-G1? I see the NPE-G2, but even on ebay it is very costly. The NPE-G2 is the next step after the NPE-G1. ~Seth
Re: Rate Limiting on Cisco Router
On Thu, Jul 08, 2010 at 01:43:17PM -1000, Antonio Querubin wrote: Traffic-shaping 80Mb/s of traffic is probably not a good idea for your router cpu :) I concur, we shape a 100Mb/s ethernet down to 50Mb/s on a 3845, so that QoS is doable. The router gets brought to its knees around 40Mb/s. Turn off shaping and the router is usable all the way up to the 50Mb/s and then some. Is there a more reasonable way to do this on Cisco? max-reserved-bandwidth? -cjp
Re: Rate Limiting on Cisco Router
On Jul 8, 2010, at 4:05 PM, Alan Bryant wrote: Thanks again for all the responses to my previous post. We have a Cisco 7206VXR router with IOS of 12.4(12) and a PA-POS-1OC3 card ofr our OC3. The problem we have now is that we are only paying for 80 MB/s of the OC-3, and the ISP is leaving the capping of it up to us. I have googled and the only things I can find is that you can not do a real cap on this type of interface. We have tried the rate-limit command with various parameters and we are unable to keep it at 80. I have read that this is not the correct way to do it, but I'm not sure what is. Any advice? If your issue is cost for rates larger than 80 Mbps then you probably want to find out what applications are using the bulk of the bandwidth and either adjust your budget, or their performance expectations and allocate network resources expressly. Flow data (even local cache analysis v. exporting) would help you glean some of this, but external longer term analysis would surely be more useful - and there are lots of way you can do that - and use the data to either justify more budget or cull offending applications. As others have noted, rate *limiting* is usually indiscriminate and often results in non-determinism and far less 'goodput' than rate-shaping. If hardware constraints with those WAN-side PHY devices are gating, you could always do it on the LAN side, and perhaps much more selectively based on which application and associated network transaction traffic is the most valuable to your business and in your operating environment. Given, you didn't talk about asymmetries and egress traffic policy tweaking at the CPE to induce desired ingress levels is usually a science in and of it's self - but alas, if one must turn the steam valves ;-) I can't see application of any rate-limiting policies indiscriminately be desirable in any business environment, and suggest that if budget constrained worst case you align network resource allocation with critical business applications first via LAN-side rate-shaping functions - or AUPs, or -danny
Re: Rate Limiting on Cisco Router
On Thu, 8 Jul 2010, Alan Bryant wrote: So you guys would not recommend the traffic shaping route on a 7206 with a NPE-G1? Is it the processor or memory that would not be able to handle it? With a G1 you'll be able to shape just fine, even do fancy stuff like fair-queue within those 80 megs. I've done this on a NPE-300, but only egress, and as long as packet sizes were fairly large (normal TCP sessions with mostly 1500 byte packets + ACKs) it coped with 90 megs of traffic. So with the added power of G1 you should definitely try before ruling it out. Shaping is so much better than the packet dropping that a rate limiter does. -- Mikael Abrahamssonemail: swm...@swm.pp.se
RE: Rate Limiting on Cisco Router
On Thu, 2010-07-08 at 20:01 -0400, Brandon Kim wrote: What about purchasing a low-end packetshaper to be used in between? If - 1/ budget is a problem and 2/ you have no BSD knowledge inhouse and 3/ the LAN side is all ethernet you could have a stab at using a PFsense box with two (and strictly ONLY two, for this use) physical NICs. It has a GUI to set up traffic shaping (see the sticky on the pfsense forums) PFsense 1.2.3 is current, don't go for the experimental 2.0 for production. There's a book and commercial support if you need it, free support via forums if you can't. Only two physical NICs is necessary due to shaper problems with more than two, whereas in a firewalling role the slots are the only limit (but VLANS are the norm for bucketloads of ports on a firewall PFsense box) An ITX (Littlefalls etc) mobo with 512MB RAM with an extra PCI Intel NIC added will do you fine . PFsense has nice traffic graphs, which helps you with shaping speeds in a big way. It also has a TFTP server available for it so it's handy for unmanned sites with only a few blue boxes ;) PS - a crazy afterthough - surely just about anything with a 10/100 ethernet link running at 100 and placed inline, cannot exceed 100Mbps - and probably less if it's plastic-cased? Try a few 8-port junkers and see what happens if you fancy a walk on the dangerous side. Watch out for errors and smoke :) Gord -- The drinker you are the smoker you get
Re: Rate Limiting on Cisco Router
Mikael Abrahamsson wrote: With a G1 you'll be able to shape just fine, even do fancy stuff like fair-queue within those 80 megs. I've done this on a NPE-300, but only egress, and as long as packet sizes were fairly large (normal TCP sessions with mostly 1500 byte packets + ACKs) it coped with 90 megs of traffic. So with the added power of G1 you should definitely try before ruling it out. Definitely worth the try. Your biggest enemy may be 12.4 IOS. It's bloated and buggy in my experience, but that has mostly been edge services. If 12.4 pegs your processor, you may want to check the software/hardware matrix and see if one of the older 12.0/2 service provider trains that they continued to add support for (probably some large customer's special requests). I don't know if it will support the G1, but if so, you might have better performance out of it. Shaping is so much better than the packet dropping that a rate limiter does. Definitely. My favorite off the wall use of shaping was to smooth traffic flow on a Gig-e to reduce microbursts from overrunning the hardware rx on a 7513. :) Jack
