Re: Router performance on OpenBSD and OpenBGPD
Yeah that's what I was thinking... you not only eliminate a single point of failure, but you also split your pps throughput requirements in half. Danno Danno.appliedi.net/drupal/ -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Martin Toft Sent: Saturday, February 24, 2007 10:52 AM To: misc@openbsd.org Subject: Re: Router performance on OpenBSD and OpenBGPD On Thu, Feb 22, 2007 at 01:03:30PM -0800, Karsten McMinn wrote: On 2/21/07, Alex Thurlow [EMAIL PROTECTED] wrote: Oops, forgot that part. At 325Mbps, we do about 60,000pps, so that puts us at about 360,000pps needed for 2Gbps. You'll have a hard time finding benches for that. To date, the best reported is 150k pps which was on the intel E7520 chipset. That was using em drivers. You're safest best for the most performance possible would likely be using the intel 5000 chipset (i.e. SuperMicro X7DB* motherboards) coupled with SysKonnect SK-9S* line of network cards. Its probably a safe bet that you'll be capable of 200K pps, but beyond that is anyones guess. Assuming correct choice of hardware can get you half way to the goal, wouldn't it be an idea to buy two or more machines and use CARP loadbalancing? Or isn't this possible when we are talking BGP? Regards, Martin
Re: Router performance on OpenBSD and OpenBGPD
On 2007/02/25 20:05, Daniel Ouellet wrote: But I was wondering however if it wouldn't be possible to use the 72xx routers as dumb media converter? I don't think you can do this exactly, but you can run OSPF on them, let OpenBSD handle the main BGP sessions, and feed back a small BGP table to the cisco containing just the prefixes that it needs to know how to route. Something like this... physical: peer - cisco - openbgp e-bgp:peer - openbgp (n.b. multihop for ebgp sessions) ibgp: cisco - openbgp basically, cisco must know routes for any packets that will be fed to it. in some cases (e.g. one transit feed going into cisco) you may be able to get away with just a static default route to the transit on the cisco and OSPF or static routes back to your network. Same with layer3 switches if you need more PPS than you can handle on a PC and can live with limitations of the switches (e.g. restricted table sizes and buffers). I have ports for dynamips and dynagen if you need to play with cisco configs and don't have spare ciscos: http://spacehopper.org/openbsd/
Re: Router performance on OpenBSD and OpenBGPD
Stuart Henderson wrote: On 2007/02/21 18:38, Daniel Ouellet wrote: problem is really I can't replace Cisco DS3 and multi channel DS3 with OpenBSD yet for the lack of decent hardware for that! (; eotdm may be worth a look where you have both ends of the line. some vendors mentioned here: http://marc.10east.com/?l=cisco-nspm=117207521113785w=2 Thanks, not really doing how I would like it. But I was wondering however if it wouldn't be possible to use the 72xx routers as dumb media converter? Meaning, I have a few of them replaced by bgpd and using OpenBSD as a more effective router. I wonder how or if possible to actually configure the router to have all traffic from/to the DS3 port to go directly to/from a Fast Ethernet on that same router without the routing engine of that router to do anything what so ever. Some other interfaces on that router could stay the same and do as usual, etc. But pick for example two of them, one DS3 and one Fast Ethernet and configure them as a simple media converter if you like. In on one interface out on the other and reverse regardless of what it is. That would work well and allow to reuse old stuff put on the self now. (: Any idea if anyone have done something like this, or if that would even be possible? Using Cisco gear as dumb media converter for an OpenBSD driven network! That would be pretty cool! Then a logo on it as: OpenBSD power network! That would be sweet. Best, Daniel
Re: Router performance on OpenBSD and OpenBGPD
On Thu, Feb 22, 2007 at 01:03:30PM -0800, Karsten McMinn wrote: On 2/21/07, Alex Thurlow [EMAIL PROTECTED] wrote: Oops, forgot that part. At 325Mbps, we do about 60,000pps, so that puts us at about 360,000pps needed for 2Gbps. You'll have a hard time finding benches for that. To date, the best reported is 150k pps which was on the intel E7520 chipset. That was using em drivers. You're safest best for the most performance possible would likely be using the intel 5000 chipset (i.e. SuperMicro X7DB* motherboards) coupled with SysKonnect SK-9S* line of network cards. Its probably a safe bet that you'll be capable of 200K pps, but beyond that is anyones guess. Assuming correct choice of hardware can get you half way to the goal, wouldn't it be an idea to buy two or more machines and use CARP loadbalancing? Or isn't this possible when we are talking BGP? Regards, Martin
Re: Router performance on OpenBSD and OpenBGPD
On 2/21/07, Alex Thurlow [EMAIL PROTECTED] wrote: Oops, forgot that part. At 325Mbps, we do about 60,000pps, so that puts us at about 360,000pps needed for 2Gbps. You'll have a hard time finding benches for that. To date, the best reported is 150k pps which was on the intel E7520 chipset. That was using em drivers. You're safest best for the most performance possible would likely be using the intel 5000 chipset (i.e. SuperMicro X7DB* motherboards) coupled with SysKonnect SK-9S* line of network cards. Its probably a safe bet that you'll be capable of 200K pps, but beyond that is anyones guess.
Re: Router performance on OpenBSD and OpenBGPD
On Thu, Feb 22, 2007 at 08:52:37AM +0500, Shohrukh Shoyokubov wrote: I just wanted to ask this question to [EMAIL PROTECTED] My situation is 100Mbps/100Mbps that is needed to be managed. I need bandwidth management and I want to ask if someone has such experience. I plan to implement it on OpenBSD. Any recommendations? Yes, please don't piggyback on unrelated threads. Joachim
Re: Router performance on OpenBSD and OpenBGPD
On Wed, Feb 21, 2007 at 05:17:22PM -0600, Alex Thurlow wrote: So anywhere I look for router performance on OpenBSD, all the benchmarks are on small lines or old machines. I also see mentions of people using it in large scale installations, which is what I'm looking to do. I thought I'd ask here and see what people have done. I have 2 GigE lines from different providers balanced via BGP with full routes from both providers. Currently, these are running through a Linux/Quagga/Iptables router/firewall with a P4 3.2 GHz. The distro is Gentoo, and we've stripped it down quite a bit. We're pushing streaming video, so it's almost all outbound traffic by about a 30:1 factor, and our average packet size is quite large - around 1200 bytes. At the moment, when we hit about 350Mbps, the router gets to ~30% CPU usage, and it appears that we stop being able to pass all the traffic at full speed. I don't see packet loss, but our traffic graph flattens a good bit. At those rates, we also start to see crashing, but we haven't been able to figure out the exact cause of those either. So, long story short, I need a new router. We've looked at Cisco, etc. and for what we're doing, it looks like we need a carrier class router. I can get a decked out 12008 for about $8k, but I'd rather not spend that much, or use the 2 feet of rack space. I've used OpenBSD/PF for firewalls in the past, and loved them, so I'd like to use it for a router if it can handle what we need. Basically, I need to be able to saturate both of those GigE lines. I'm willing to buy the brand-newest hardware - the PCI express bus should be able to do 2.5 Gbps, but I can't find anything that says I can push that much through software. I was also looking at the Intel I/O Accelerator, but I didn't see if there was OpenBSD support for it. I'm sure if there is, that would help get me to be able to push the traffic I want to. A long explanation, but I'm just hoping someone could give me some insight here. I don't have the faintest clue about that kind of speed, and the old box next to me would probably faint if showed these numbers. Still, some of the stuff below, while tangential, might be useful. OpenBGP, by any right, should not be a problem if you are not doing anything grossly stupid (like trying to run this in 8 MB of memory). The intel accelerator you mention is not supported, so that wouldn't help any. The one point I miss is failover capability; both the Cisco and OpenBSD should be able to do this, but it's worth noting - and having. Joachim
Re: Router performance on OpenBSD and OpenBGPD
* Alex Thurlow [EMAIL PROTECTED] [2007-02-22 00:25]: I have 2 GigE lines from different providers balanced via BGP with full routes from both providers. Currently, these are running through a Linux/Quagga/Iptables router/firewall with a P4 3.2 GHz. The distro is Gentoo, and we've stripped it down quite a bit. We're pushing streaming video, so it's almost all outbound traffic by about a 30:1 factor, and our average packet size is quite large - around 1200 bytes. At the moment, when we hit about 350Mbps, the router gets to ~30% CPU usage, and it appears that we stop being able to pass all the traffic at full speed. I don't see packet loss, but our traffic graph flattens a good bit. At those rates, we also start to see crashing, but we haven't been able to figure out the exact cause of those either. So, long story short, I need a new router. We've looked at Cisco, etc. and for what we're doing, it looks like we need a carrier class router. I can get a decked out 12008 for about $8k, but I'd rather not spend that much, or use the 2 feet of rack space. I've used OpenBSD/PF for firewalls in the past, and loved them, so I'd like to use it for a router if it can handle what we need. Basically, I need to be able to saturate both of those GigE lines. I'm willing to buy the brand-newest hardware - the PCI express bus should be able to do 2.5 Gbps, but I can't find anything that says I can push that much through software. well... it depends. we have a router at a customer that I have seen peaking above 750 MBit/s, and that was with relatively mean traffic (i. e. not all nice big packets). so I'd say there is a realistic chance to get reasonably close (and if everything else fails, you can still split outgoing over two or so). naturally, that requires somewhat carefully selected hardware, and these are ones of the very few machines I run where we do not go for GENERIC.* for a reason. -- Henning Brauer, [EMAIL PROTECTED], [EMAIL PROTECTED] BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting - Hamburg Amsterdam
Re: Router performance on OpenBSD and OpenBGPD
Alex Thurlow wrote: We're pushing streaming video, so it's almost all outbound traffic by about a 30:1 factor, and our average packet size is quite large - around 1200 bytes. At the moment, when we hit about 350Mbps, the router gets to ~30% CPU usage, and it appears that we stop being able to pass all the traffic at full speed. I don't see packet loss, but our traffic graph flattens a good bit. At those rates, we also start to see crashing, but we haven't been able to figure out the exact cause of those either. The issue as explain in the archive many times is not the level of traffic, but the number of packets per seconds you pass and it's based also on good network cards. Many can do in the 500mbps with their OpenBSD router and more without to much issues. But again, what is the limit is the pps, not the bps. S, if all your packets are in the 1200 bytes as you put here, you sure can test it with one OpenBSD and you sure should have no issue with good decent hardware, but more importantly, good network cards. That's really the key here. I use it in public peering places no issues and I keep rolling out more and more and my next one, as I go slow to be safe will be in Equinix where I have close to 100 sessions and many full bgp feeds as well. Test and adjust for your own needs, but you sure should be able to do that better then your current setup. Funny that some replace their setup with Cisco and I replace Cisco with OpenBSD as much as I can! My only problem is really I can't replace Cisco DS3 and multi channel DS3 with OpenBSD yet for the lack of decent hardware for that! (; But every Ethernet type are going away from Cisco one after the others and hopefully before the end of the year, all will be gone! Best, Daniel
Re: Router performance on OpenBSD and OpenBGPD
Oops, forgot that part. At 325Mbps, we do about 60,000pps, so that puts us at about 360,000pps needed for 2Gbps. Daniel Ouellet wrote: Alex Thurlow wrote: We're pushing streaming video, so it's almost all outbound traffic by about a 30:1 factor, and our average packet size is quite large - around 1200 bytes. At the moment, when we hit about 350Mbps, the router gets to ~30% CPU usage, and it appears that we stop being able to pass all the traffic at full speed. I don't see packet loss, but our traffic graph flattens a good bit. At those rates, we also start to see crashing, but we haven't been able to figure out the exact cause of those either. The issue as explain in the archive many times is not the level of traffic, but the number of packets per seconds you pass and it's based also on good network cards. Many can do in the 500mbps with their OpenBSD router and more without to much issues. But again, what is the limit is the pps, not the bps. S, if all your packets are in the 1200 bytes as you put here, you sure can test it with one OpenBSD and you sure should have no issue with good decent hardware, but more importantly, good network cards. That's really the key here. I use it in public peering places no issues and I keep rolling out more and more and my next one, as I go slow to be safe will be in Equinix where I have close to 100 sessions and many full bgp feeds as well. Test and adjust for your own needs, but you sure should be able to do that better then your current setup. Funny that some replace their setup with Cisco and I replace Cisco with OpenBSD as much as I can! My only problem is really I can't replace Cisco DS3 and multi channel DS3 with OpenBSD yet for the lack of decent hardware for that! (; But every Ethernet type are going away from Cisco one after the others and hopefully before the end of the year, all will be gone! Best, Daniel
Re: Router performance on OpenBSD and OpenBGPD
On 2007/02/21 18:38, Daniel Ouellet wrote: problem is really I can't replace Cisco DS3 and multi channel DS3 with OpenBSD yet for the lack of decent hardware for that! (; eotdm may be worth a look where you have both ends of the line. some vendors mentioned here: http://marc.10east.com/?l=cisco-nspm=117207521113785w=2
Re: Router performance on OpenBSD and OpenBGPD
On 21 Feb 2007, at 23:41, Henning Brauer wrote: * Alex Thurlow [EMAIL PROTECTED] [2007-02-22 00:25]: I have 2 GigE lines from different providers balanced via BGP with full routes from both providers. Currently, these are running through a Linux/Quagga/Iptables router/firewall with a P4 3.2 GHz. The distro is Gentoo, and we've stripped it down quite a bit. We're pushing streaming video, so it's almost all outbound traffic by about a 30:1 factor, and our average packet size is quite large - around 1200 bytes. At the moment, when we hit about 350Mbps, the router gets to ~30% CPU usage, and it appears that we stop being able to pass all the traffic at full speed. I don't see packet loss, but our traffic graph flattens a good bit. At those rates, we also start to see crashing, but we haven't been able to figure out the exact cause of those either. So, long story short, I need a new router. We've looked at Cisco, etc. and for what we're doing, it looks like we need a carrier class router. I can get a decked out 12008 for about $8k, but I'd rather not spend that much, or use the 2 feet of rack space. I've used OpenBSD/PF for firewalls in the past, and loved them, so I'd like to use it for a router if it can handle what we need. Basically, I need to be able to saturate both of those GigE lines. I'm willing to buy the brand-newest hardware - the PCI express bus should be able to do 2.5 Gbps, but I can't find anything that says I can push that much through software. well... it depends. we have a router at a customer that I have seen peaking above 750 MBit/s, and that was with relatively mean traffic (i. e. not all nice big packets). so I'd say there is a realistic chance to get reasonably close (and if everything else fails, you can still split outgoing over two or so). naturally, that requires somewhat carefully selected hardware, and these are ones of the very few machines I run where we do not go for GENERIC.* for a reason. -- Henning Brauer, [EMAIL PROTECTED], [EMAIL PROTECTED] BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting - Hamburg Amsterdam What are the main changes you make to GENERIC Henning? --- Liam J. Foy [EMAIL PROTECTED]
Re: Router performance on OpenBSD and OpenBGPD
I just wanted to ask this question to [EMAIL PROTECTED] My situation is 100Mbps/100Mbps that is needed to be managed. I need bandwidth management and I want to ask if someone has such experience. I plan to implement it on OpenBSD. Any recommendations? Shohrukh Alex Thurlow wrote: So anywhere I look for router performance on OpenBSD, all the benchmarks are on small lines or old machines. I also see mentions of people using it in large scale installations, which is what I'm looking to do. I thought I'd ask here and see what people have done. I have 2 GigE lines from different providers balanced via BGP with full routes from both providers. Currently, these are running through a Linux/Quagga/Iptables router/firewall with a P4 3.2 GHz. The distro is Gentoo, and we've stripped it down quite a bit. We're pushing streaming video, so it's almost all outbound traffic by about a 30:1 factor, and our average packet size is quite large - around 1200 bytes. At the moment, when we hit about 350Mbps, the router gets to ~30% CPU usage, and it appears that we stop being able to pass all the traffic at full speed. I don't see packet loss, but our traffic graph flattens a good bit. At those rates, we also start to see crashing, but we haven't been able to figure out the exact cause of those either. So, long story short, I need a new router. We've looked at Cisco, etc. and for what we're doing, it looks like we need a carrier class router. I can get a decked out 12008 for about $8k, but I'd rather not spend that much, or use the 2 feet of rack space. I've used OpenBSD/PF for firewalls in the past, and loved them, so I'd like to use it for a router if it can handle what we need. Basically, I need to be able to saturate both of those GigE lines. I'm willing to buy the brand-newest hardware - the PCI express bus should be able to do 2.5 Gbps, but I can't find anything that says I can push that much through software. I was also looking at the Intel I/O Accelerator, but I didn't see if there was OpenBSD support for it. I'm sure if there is, that would help get me to be able to push the traffic I want to. A long explanation, but I'm just hoping someone could give me some insight here. Alex Thurlow Technical Director Blastro, Inc.
