Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-19 Thread Kip Macy
On Fri, Jul 11, 2008 at 11:44 PM, Brian McGinty [EMAIL PROTECTED] wrote: Hi Brian I very much doubt that this is ceteris paribus. This is 384 random IPs - 384 random IP addresses with a flow lookup for each packet. Also, I've read through igb on Linux - it has a lot of optimizations that the

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-19 Thread Brian McGinty
G'day Kip, I'm hoping to get to it some time in August. I'm a bit behind in my contracts at the moment. A few weeks ago, I did a quick comparison of the driver between FreeBSD and Linux, and found quite a few differences that's worth pulling over. The guy from Intel working on FreeBSD, Jack?,

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-19 Thread Kip Macy
On Sat, Jul 19, 2008 at 7:17 PM, Brian McGinty [EMAIL PROTECTED] wrote: G'day Kip, I'm hoping to get to it some time in August. I'm a bit behind in my contracts at the moment. A few weeks ago, I did a quick comparison of the driver between FreeBSD and Linux, and found quite a few

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-14 Thread Bruce Evans
On Mon, 7 Jul 2008, Robert Watson wrote: On Mon, 7 Jul 2008, Bruce Evans wrote: (1) sendto() to a specific address and port on a socket that has been bound to INADDR_ANY and a specific port. (2) sendto() on a specific address and port on a socket that has been bound to a specific IP

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-12 Thread Brian McGinty
Hi Brian I very much doubt that this is ceteris paribus. This is 384 random IPs - 384 random IP addresses with a flow lookup for each packet. Also, I've read through igb on Linux - it has a lot of optimizations that the FreeBSD driver lacks and I have yet to implement. Hey Kip, when will

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-11 Thread Stefan Lambrev
Hi Paul, Paul wrote: I tested Linux in bridge configuration with the same machine and it CPUed out at about 600kpps through the bridge.. 600kpps incoming or 600kpps incoming+ outgoing ? That's a bit low :/ Soft interrupt using all the cpu. Same opteron , 82571EB Pci express NIC. Tried

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-11 Thread Bart Van Kerckhove
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Good news is using iptables only adds a few percentage onto the CPU usage. But still, what's with that.. So far FreeBSD got the highest pps rating for forwarding. I haven't tried bridge mode. Ipfw probably takes a big hit in that too though.

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-10 Thread Paul
I tested Linux in bridge configuration with the same machine and it CPUed out at about 600kpps through the bridge.. That's a bit low :/ Soft interrupt using all the cpu. Same opteron , 82571EB Pci express NIC. Tried SMP/ non-smp , load balanced irqs, etc.. Good news is using iptables

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-08 Thread Kip Macy
On Mon, Jul 7, 2008 at 6:07 PM, Mike Tancsa [EMAIL PROTECTED] wrote: At 02:44 PM 7/7/2008, Paul wrote: Also my 82571 NIC supports multiple received queues and multiple transmit queues so why hasn't anyone written the driver to support this? It's not a 10gb card and it still supports it and

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-08 Thread Kip Macy
On Mon, Jul 7, 2008 at 6:22 PM, Paul [EMAIL PROTECTED] wrote: I read through the IGB driver, and it says 82575/6 only... which is the new chip Intel is releasing on the cards this month 2 port and october 4 port, but the chips are on some of the motherboards right now. Why can't it also use

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-08 Thread Robert Watson
On Mon, 7 Jul 2008, Artem Belevich wrote: As was already mentioned, we can't avoid all cache misses as there's data that's recently been updated in memory via DMA and therefor kicked out of cache. However, we may hide some of the latency penalty by prefetching 'interesting' data early.

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-08 Thread Stefan Lambrev
Hi, Kip Macy wrote: On Mon, Jul 7, 2008 at 6:07 PM, Mike Tancsa [EMAIL PROTECTED] wrote: At 02:44 PM 7/7/2008, Paul wrote: Also my 82571 NIC supports multiple received queues and multiple transmit queues so why hasn't anyone written the driver to support this? It's not a 10gb card

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-08 Thread Kip Macy
I have a pre-production card. With some bug fixes and some tuning of interrupt handling (custom stack - I've been asked to push the changes back in to CVS, I just don't have time right now) an otherwise unoptimized igb can forward 1.04Mpps from one port to another (1.04 Mpps in on igb0 and

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-08 Thread Paul
Will someone confirm if it will support the 82571EB ? I don't see a reason why not as it's very similar hardware and it's available now in large quantities so making 82571 part of igb I think would be a good idea. Kip Macy wrote: I have a pre-production card. With some bug fixes and some

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-08 Thread Bruce Evans
On Mon, 7 Jul 2008, Erik Trulsson wrote: On Mon, Jul 07, 2008 at 10:30:53PM +1000, Bruce Evans wrote: On Mon, 7 Jul 2008, Andre Oppermann wrote: The theoretical maximum at 64byte frames is 1,488,100. I've looked up my notes the 1.244Mpps number can be ajusted to 1.488Mpps. Where is the

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-08 Thread Stefan Lambrev
Hi, Kip Macy wrote: On Mon, Jul 7, 2008 at 6:22 PM, Paul [EMAIL PROTECTED] wrote: I read through the IGB driver, and it says 82575/6 only... which is the new chip Intel is releasing on the cards this month 2 port and october 4 port, but the chips are on some of the motherboards right now.

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-08 Thread Artem Belevich
On 7/8/08, Robert Watson [EMAIL PROTECTED] wrote: There were some patches floating around for if_em to do a prefetch of the first bit of packet data on packets before handing them up the stack. My I found Andre Oppermann's optimization patch mentioned in july 2005 status report:

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-08 Thread Paul
But this is probably no routing table, and single source and dst ips or very limited number of ips and ports. the entire problem with Linux is the route cache, try and generate random source ips and random source/dst ports and it won't even do 100kpps without problems. I would like to log into

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-08 Thread Kip Macy
On Tue, Jul 8, 2008 at 1:46 PM, Brian McGinty [EMAIL PROTECTED] wrote: I have a pre-production card. With some bug fixes and some tuning of interrupt handling (custom stack - I've been asked to push the changes back in to CVS, I just don't have time right now) an otherwise unoptimized igb can

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-08 Thread Brian McGinty
I have a pre-production card. With some bug fixes and some tuning of interrupt handling (custom stack - I've been asked to push the changes back in to CVS, I just don't have time right now) an otherwise unoptimized igb can forward 1.04Mpps from one port to another (1.04 Mpps in on igb0 and

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Andre Oppermann
Robert Watson wrote: Experience suggests that forwarding workloads see significant lock contention in the routing and transmit queue code. The former needs some kernel hacking to address in order to improve parallelism for routing lookups. The latter is harder to address given the hardware

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Robert Watson
On Mon, 7 Jul 2008, Andre Oppermann wrote: Robert Watson wrote: Experience suggests that forwarding workloads see significant lock contention in the routing and transmit queue code. The former needs some kernel hacking to address in order to improve parallelism for routing lookups. The

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Andre Oppermann
Ingo Flaschberger wrote: Dear Paul, I tried all of this :/ still, 256/512 descriptors seem to work the best. Happy to let you log into the machine and fiddle around if you want :) yes, but I'm shure I will also not be able to achieve much more pps. As it seems that you hit

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Andre Oppermann
Robert Watson wrote: On Mon, 7 Jul 2008, Andre Oppermann wrote: Robert Watson wrote: Experience suggests that forwarding workloads see significant lock contention in the routing and transmit queue code. The former needs some kernel hacking to address in order to improve parallelism for

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Andre Oppermann
Paul, to get a systematic analysis of the performance please do the following tests and put them into a table for easy comparison: 1. inbound pps w/o loss with interface in monitor mode (ifconfig em0 monitor) 2. inbound pps w/ fastforward into a single blackhole route 3. inbound pps /w

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Andre Oppermann
Paul wrote: SMP DISABLED on my Opteron 2212 (ULE, Preemption on) Yields ~750kpps in em0 and out em1 (one direction) I am miffed why this yields more pps than a) with all 4 cpus running and b) 4 cpus with lagg load balanced over 3 incoming connections so 3 taskq threads SMP adds quite some

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Robert Watson
On Mon, 7 Jul 2008, Andre Oppermann wrote: Distributing the interrupts and taskqueues among the available CPUs gives concurrent forwarding with bi- or multi-directional traffic. All incoming traffic from any particular interface is still serialized though. ... although not on multiple input

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Andre Oppermann
Robert Watson wrote: On Mon, 7 Jul 2008, Andre Oppermann wrote: Distributing the interrupts and taskqueues among the available CPUs gives concurrent forwarding with bi- or multi-directional traffic. All incoming traffic from any particular interface is still serialized though. ...

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Andre Oppermann
Bruce Evans wrote: On Mon, 7 Jul 2008, Andre Oppermann wrote: Ingo Flaschberger wrote: I don't think you will be able to route 64byte packets at 1gbit wirespeed (2Mpps) with a current x86 platform. You have to take inter-frame gap and other overheads too. That gives about 1.244Mpps max on

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Kris Kennaway
Andre Oppermann wrote: Robert Watson wrote: Experience suggests that forwarding workloads see significant lock contention in the routing and transmit queue code. The former needs some kernel hacking to address in order to improve parallelism for routing lookups. The latter is harder to

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Bruce Evans
On Mon, 7 Jul 2008, Andre Oppermann wrote: Bruce Evans wrote: What are the other overheads? I calculate 1.644Mpps counting the inter-frame gap, with 64-byte packets and 64-header_size payloads. If the 64 bytes is for the payload, then the max is much lower. The theoretical maximum at

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Robert Watson
On Mon, 7 Jul 2008, Bruce Evans wrote: I use low-end memory, but on the machine that does 640 kpps it somehow has latency almost 4 times as low as on new FreeBSD cluster machines (~42 nsec instead of ~150). perfmon (fixed for AXP and A64) and hwpmc report an average of 11 k8-dc-misses per

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Robert Watson
On Mon, 7 Jul 2008, Robert Watson wrote: The last of these should really be quite a bit faster than the first of these, but I'd be interested in seeing specific measurements for each if that's possible! And, if you're feeling particualrly subject to suggestion, you might consider comparing

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Bruce Evans
On Mon, 7 Jul 2008, Robert Watson wrote: Since you're doing fine-grained performance measurements of a code path that interests me a lot, could you compare the cost per-send on UDP for the following four cases: (1) sendto() to a specific address and port on a socket that has been bound to

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Erik Trulsson
On Mon, Jul 07, 2008 at 10:30:53PM +1000, Bruce Evans wrote: On Mon, 7 Jul 2008, Andre Oppermann wrote: Bruce Evans wrote: What are the other overheads? I calculate 1.644Mpps counting the inter-frame gap, with 64-byte packets and 64-header_size payloads. If the 64 bytes is for the

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Andre Oppermann
Bruce Evans wrote: On Mon, 7 Jul 2008, Andre Oppermann wrote: Bruce Evans wrote: What are the other overheads? I calculate 1.644Mpps counting the inter-frame gap, with 64-byte packets and 64-header_size payloads. If the 64 bytes is for the payload, then the max is much lower. The

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Robert Watson
On Mon, 7 Jul 2008, Bruce Evans wrote: (1) sendto() to a specific address and port on a socket that has been bound to INADDR_ANY and a specific port. (2) sendto() on a specific address and port on a socket that has been bound to a specific IP address (not INADDR_ANY) and a specific

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Bruce Evans
On Mon, 7 Jul 2008, Andre Oppermann wrote: Paul, to get a systematic analysis of the performance please do the following tests and put them into a table for easy comparison: 1. inbound pps w/o loss with interface in monitor mode (ifconfig em0 monitor) ... I won't be running many of these

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Andre Oppermann
Bruce Evans wrote: On Mon, 7 Jul 2008, Andre Oppermann wrote: Paul, to get a systematic analysis of the performance please do the following tests and put them into a table for easy comparison: 1. inbound pps w/o loss with interface in monitor mode (ifconfig em0 monitor) ... I won't be

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Bruce Evans
On Mon, 7 Jul 2008, Andre Oppermann wrote: Bruce Evans wrote: So it seems that the major overheads are not near the driver (as I already knew), and upper layers are responsible for most of the cache misses. The packet header is accessed even in monitor mode, so I think most of the cache misses

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Paul
one that will later on handle the taskqueue to process the packets. That adds overhead. Ideally the interrupt for each network interface is bound to exactly one pre-determined CPU and the taskqueue is bound to the same CPU. That way the overhead for interrupt and taskqueue scheduling can be

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Paul
I use low-end memory, but on the machine that does 640 kpps it somehow has latency almost 4 times as low as on new FreeBSD cluster machines (~42 nsec instead of ~150). perfmon (fixed for AXP and A64) and hwpmc report an average of 11 k8-dc-misses per sendto() while sending via bge at 640 kpps.

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Bruce Evans
On Tue, 8 Jul 2008, Bruce Evans wrote: On Mon, 7 Jul 2008, Andre Oppermann wrote: Bruce Evans wrote: So it seems that the major overheads are not near the driver (as I already knew), and upper layers are responsible for most of the cache misses. The packet header is accessed even in monitor

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Artem Belevich
Hi, As was already mentioned, we can't avoid all cache misses as there's data that's recently been updated in memory via DMA and therefor kicked out of cache. However, we may hide some of the latency penalty by prefetching 'interesting' data early. I.e. we know that we want to access some

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Mike Tancsa
At 02:44 PM 7/7/2008, Paul wrote: Also my 82571 NIC supports multiple received queues and multiple transmit queues so why hasn't anyone written the driver to support this? It's not a 10gb card and it still supports it and it's widely Intel actually maintains the driver. Not sure if there

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Julian Elischer
Artem Belevich wrote: Hi, As was already mentioned, we can't avoid all cache misses as there's data that's recently been updated in memory via DMA and therefor kicked out of cache. However, we may hide some of the latency penalty by prefetching 'interesting' data early. I.e. we know that we

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Peter Jeremy
On 2008-Jul-07 13:25:13 -0700, Julian Elischer [EMAIL PROTECTED] wrote: what you need is a speculative prefetch where you an tell teh processor We will probably need the following address so start getting it while we go do other stuff. This looks like the PREFETCH instructions that exist in at

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Julian Elischer
Peter Jeremy wrote: On 2008-Jul-07 13:25:13 -0700, Julian Elischer [EMAIL PROTECTED] wrote: what you need is a speculative prefetch where you an tell teh processor We will probably need the following address so start getting it while we go do other stuff. This looks like the PREFETCH

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Artem Belevich
Prefetching when you are waiting for the data isn't a help. Agreed. Got to start prefetch around put your memory latency herens before you actually need the data and move on doing other things that do not depend on the data you've just started prefetching. what you need is a speculative

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Mike Tancsa
At 02:44 PM 7/7/2008, Paul wrote: Also my 82571 NIC supports multiple received queues and multiple transmit queues so why hasn't anyone written the driver to support this? It's not a 10gb card and it still supports it and it's widely available and not too expensive either. The new 82575/6

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-07 Thread Paul
I read through the IGB driver, and it says 82575/6 only... which is the new chip Intel is releasing on the cards this month 2 port and october 4 port, but the chips are on some of the motherboards right now. Why can't it also use the 82571 ? doesn't make any sense.. I haven't tried it but just

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-06 Thread Andrew Snow
I'm no expert, but I imagine the problem is because the net processing of FreeBSD is not pipelined enough. We are now able to affordably throw many gigabytes of RAM into a machine, as well 2 to 8 CPUs. So why not allow for big buffers and multiple processing steps? I be happy to give up a

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-05 Thread Paul
ULE + PREEMPTION for non SMP no major differences with SMP with ULE/4BSD and preemption ON/OFF 32 bit UP test coming up with new cpu and I'm installing dragonfly sometime this weekend :] UP: 1mpps in one direction with no firewall/no routing table is not too bad, but 1mpps both directions is

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-05 Thread Paul
UP 32 bit test vs 64 bit: negligible difference in forwarding performance without polling slightly better polling performance but still errors at lower packet rates same massive hit with ipfw loaded Installing dragonfly in a bit.. If anyone has a really fast PPC type system or SUN or something

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-05 Thread Bart Van Kerckhove
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Paul / Ingo, I tried all of this :/ still, 256/512 descriptors seem to work the best. Happy to let you log into the machine and fiddle around if you want :) I've been watching this thread closely, since I'm in a very similair situation. A few

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-05 Thread Chris Marlatt
Bart Van Kerckhove wrote: The netfpga site seems more or less dead - is this project still alive? It does look like a very interesting idea, even though it's currently quite linux-centric (and according to docs doesn't have VLAN nor ip6 support, the former being quite a dealbreaker) Just last

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-05 Thread Julian Elischer
Bart Van Kerckhove wrote: Perhaps you have some better luck at some different hardware systems (ppc, mips, ..?) or use freebsd only for routing-table-updates and special network-cards (netfpga) for real routing. The netfpga site seems more or less dead - is this project still alive? It does

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-04 Thread Bruce Evans
On Fri, 4 Jul 2008, Paul wrote: Numbers are maximum with near 100% cpu usage and some errors occuring, just for testing. FreeBSD 7.0-STABLE FreeBSD 7.0-STABLE #6: Thu Jul 3 19:32:38 CDT 2008 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/ROUTER amd64 CPU: Dual-Core AMD Opteron(tm) Processor

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-04 Thread Ingo Flaschberger
Dear Paul, Opteron UP mode, no polling input (em0) output packets errs bytespackets errs bytes colls 1071020 0 66403248 2 0404 0 that looks good. (but seems to be near the limit). Polling turned on provided

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-04 Thread Ingo Flaschberger
Dear Paul, what could cause this? *) kern.polling.idle_poll enabled? *) kern.polling.user_frac ? *) kern.polling.reg_frac ? *) kern.polling.burst_max ? *) kern.polling.each_burst ? I tried tons of different values for these and nothing made any significant difference. Idle polling makes a

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-04 Thread Paul
I tried all of this :/ still, 256/512 descriptors seem to work the best. Happy to let you log into the machine and fiddle around if you want :) Paul Ingo Flaschberger wrote: Dear Paul, what could cause this? *) kern.polling.idle_poll enabled? *) kern.polling.user_frac ? *)

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-03 Thread Paul
Preliminary 32 bit results... When I started out it looked like 32 bit was worse than 64 bit, but it's just the timers are different. For instance, 4000 hz in 64 bit gives better results than 4000hz in 32 bit. Low HZ gives better result with polling on in 32 bit Bottom line, so far I'm not

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-03 Thread Bruce Evans
On Wed, 2 Jul 2008, Paul wrote: ... ---Reboot with 4096/4096(my guess is that it will be a lot worse, more errors..) Without polling, 4096 is horrible, about 200kpps less ... :/ Turning on polling.. polling on, 4096 is bad, input (em0)

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-03 Thread Paul
Bruce Evans wrote: On Wed, 2 Jul 2008, Paul wrote: ... ---Reboot with 4096/4096(my guess is that it will be a lot worse, more errors..) Without polling, 4096 is horrible, about 200kpps less ... :/ Turning on polling.. polling on, 4096 is bad, input

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-03 Thread Bruce Evans
On Thu, 3 Jul 2008, Paul wrote: Bruce Evans wrote: No polling: 843762 25337 52313248 1 0178 0 763555 0 47340414 1 0178 0 830189 0 51471722 1 0178 0 838724 0 52000892 1 0178

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-03 Thread Ingo Flaschberger
Dear Stefan, So my maximum without polling is close to 800kpps but if I push that it starts locking me from doing things, or how many kpps do you want to achieve? Do not know for Paul but, I want to be able to route (and/or bridge to handle) 600-700mbps syn flood, which is something like

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-03 Thread Ingo Flaschberger
Dear Paul, Tomorrow comes opteron so it's 1ghz faster than this one, and I can see if it scales directly with cpu speed or what happens. can you send me a lspci -v? I did another SMP test with an interesting results. I took one of the cpus out of the machine, so it was just left with a

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-03 Thread Ingo Flaschberger
Dear Stefan, So my maximum without polling is close to 800kpps but if I push that it starts locking me from doing things, or how many kpps do you want to achieve? Do not know for Paul but, I want to be able to route (and/or bridge to handle) 600-700mbps syn flood, which is something like

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-03 Thread Paul
Bruce Evans wrote: On Thu, 3 Jul 2008, Paul wrote: Bruce Evans wrote: No polling: 843762 25337 52313248 1 0178 0 763555 0 47340414 1 0178 0 830189 0 51471722 1 0178 0 838724 0 52000892

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-03 Thread Paul
Err.. pciconf -lv ? [EMAIL PROTECTED]:0:0:0: class=0x05 card=0x151115d9 chip=0x036910de rev=0xa2 hdr=0x00 vendor = 'Nvidia Corp' device = 'MCP55 Memory Controller' class = memory subclass = RAM [EMAIL PROTECTED]:0:1:0: class=0x060100 card=0x151115d9

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-03 Thread Steve Bertrand
Ingo Flaschberger wrote: My next router appliance will be: http://www.axiomtek.com.tw/products/ViewProduct.asp?view=429 This is exactly the device that I have been testing with (just rebranded). Steve ___ freebsd-net@freebsd.org mailing list

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-02 Thread Paul
SMP DISABLED on my Opteron 2212 (ULE, Preemption on) Yields ~750kpps in em0 and out em1 (one direction) I am miffed why this yields more pps than a) with all 4 cpus running and b) 4 cpus with lagg load balanced over 3 incoming connections so 3 taskq threads I would be willing to set up test

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-02 Thread Ingo Flaschberger
Dear Paul, SMP DISABLED on my Opteron 2212 (ULE, Preemption on) Yields ~750kpps in em0 and out em1 (one direction) I am miffed why this yields more pps than a) with all 4 cpus running and b) 4 cpus with lagg load balanced over 3 incoming connections so 3 taskq threads because less locking,

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-02 Thread Paul
Ipfw rule was simply allow ip from any to any :) This is 64bit i'm testing now.. I have a 32 bit install I tested on another machine but it only has bge NIC and wasn't performing as well so I'll reinstall 32 bit on this 2212 and test then drop in the (3ghz) and test. I still don't like

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-02 Thread Ingo Flaschberger
Dear Paul, I still don't like the huge hit ipfw and lagg take :/ I think, you can't use fastforward with with lagg. ** I tried polling in UP mode and I got some VERY interesting results.. CPU is 44% idle (idle polling isn't on) but I'm getting errors! It's doing 530kpps with ipfw loaded,

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-02 Thread Stefan Lambrev
Hi Ingo Flaschberger wrote: Dear Paul, I still don't like the huge hit ipfw and lagg take :/ You have to try PF, then you will respect IPFW again ;) -cut- So my maximum without polling is close to 800kpps but if I push that it starts locking me from doing things, or how many kpps do you

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-02 Thread Adrian Chadd
2008/7/2 Stefan Lambrev [EMAIL PROTECTED]: Do not know for Paul but, I want to be able to route (and/or bridge to handle) 600-700mbps syn flood, which is something like 1500kpps in every direction. Is it unrealistic? If the code is optimized to fully utilize MP I do not see a reason why quad

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-02 Thread Freddie Cash
On Mon, Jun 30, 2008 at 6:39 PM, Ingo Flaschberger [EMAIL PROTECTED] wrote: I'm curious now... how do you change individual device polling via sysctl? not via sysctl, via ifconfig: # enable interface polling /sbin/ifconfig em0 polling /sbin/ifconfig em1 polling /sbin/ifconfig em2 polling

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-01 Thread Stefan Lambrev
Hi, Ingo Flaschberger wrote: Dear Rudy, I used polling in FreeBSD 5.x and it helped a bunch. I set up a new router with 7.0 and MSI was recommended to me. (I noticed no difference when moving from polling - MSI, however, on 5.4 polling seemed to help a lot. What are people using in 7.0?

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-01 Thread Ingo Flaschberger
Dear Paul, I have been unable to even come close to livelocking the machine with the em driver interrupt moderation. So that to me throws polling out the window. I tried 8000hz with polling modified to allow 1 burst and it makes no difference higher hz-values gives you better latenca

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-01 Thread Ingo Flaschberger
Dear Paul, Dual Opteron 2212, Recompiled kernel with 7-STABLE and removed a lot of junk in the config, added options NO_ADAPTIVE_MUTEXES not sure if that makes any difference or not, will test without. Used ULE scheduler, used preemption, CPUTYPE=opteron in /etc/make.conf

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-01 Thread Paul
Thanks.. I was hoping I wasn't seeing things : I do not like inconsistencies.. :/ Stefan Lambrev wrote: Greetings Paul, --OK I'm stumped now.. Rebuilt with preemption and ULE and preemption again and it's not doing what it did before.. I saw this in my configuration too :) Just leave

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-01 Thread Paul
I am going to.. I have an opteron 270 dual set up on 32 bit and the 2212 is set up on 64 bit :) Today should bring some 32 bit results as well as etherchannel results. Ingo Flaschberger wrote: Dear Paul, Dual Opteron 2212, Recompiled kernel with 7-STABLE and removed a lot of junk in the

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-01 Thread Paul
I can't reproduce the 580kpps maximum that I saw when I first compiled for some reason, I don't understand, the max I get even with ULE and preemption is now about 530 and it dips to 480 a lot.. The first time I tried it it was at 580 and dipped to 520...what the?.. (kernel config attached at

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-01 Thread Paul
ULE without PREEMPTION is now yeilding better results. input (em0) output packets errs bytespackets errs bytes colls 571595 40639 34564108 1 0226 0 577892 48865 34941908 1 0178 0 545240

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-01 Thread Paul
Ok, now THIS is absoultely a whole bunch of ridiculousness.. I set up etherchannel, and I'm evenly distributing packets over em0 em1 and em2 to lagg0 and i get WORSE performance than with a single interface.. Can anyone explain this one? This is horrible. I got em0-em2 taskq's using 80% cpu

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-07-01 Thread Paul
Apparently lagg hasn't been giant fixed :/ Can we do something about this quickly? with adaptive giant i get more performance on lagg but the cpu usage is smashed 100% I get about 50k more pps per interface (so 150kpps total which STILL is less than a single gigabit port) Check it out 68

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-06-30 Thread Stefan Lambrev
Paul wrote: The higher I set the buffer the worse it is.. 256 and 512 I get about 50-60k more pps than i do with 2048 or 4096.. You would think it would be the other way around but obviously there is some contention going on. :/ Looks like in bridge mode hw.em.rxd=512 and hw.em.txd=512 yields

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-06-30 Thread Mike Tancsa
At 04:04 AM 6/29/2008, Paul wrote: This is just a question but who can get more than 400k pps forwarding performance ? OK, I setup 2 boxes on either end of a RELENG_7 box from about May 7th just now, to see with 2 boxes blasting across it how it would work. *However*, this is with no

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-06-30 Thread Paul
With hours and days of tweaking i can't even get 500k pps :/ no firewall no anything else.. What is your kernel config? Sysctl configs? My machine i'm testing on is dual opteron 2212 , with intel 2 port 82571 nic.. Using 7-STABLE and I tried 6-stable and -current I get the RTM_MISS with 7 and

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-06-30 Thread Steve Bertrand
Mike Tancsa wrote: At 04:04 AM 6/29/2008, Paul wrote: This is just a question but who can get more than 400k pps forwarding performance ? OK, I setup 2 boxes on either end of a RELENG_7 box from about May 7th just now, to see with 2 boxes blasting across it how it would work. *However*,

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-06-30 Thread Paul
I am getting this message with normal routing. say... em0 10.1.1.1/24 em1 10.2.2.1/24 using a box 10.1.1.2 on em0 and having another box on 10.2.2.2 on em1 I send packet from 10.1.1.2 which goes through em0 and has a route to 10.2.2.2 out em1 of course and I get MASSIVE RTM_MISS messages

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-06-30 Thread Ingo Flaschberger
Dear Paul, I am getting this message with normal routing. say... em0 10.1.1.1/24 em1 10.2.2.1/24 using a box 10.1.1.2 on em0 and having another box on 10.2.2.2 on em1 I send packet from 10.1.1.2 which goes through em0 and has a route to 10.2.2.2 out em1 of course and I get MASSIVE

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-06-30 Thread Wilkinson, Alex
0n Mon, Jun 30, 2008 at 03:44:48PM -0400, Mike Tancsa wrote: OK, I setup 2 boxes on either end of a RELENG_7 box from about May 7th just now, to see with 2 boxes blasting across it how it would work. *However*, this is with no firewall loaded and, I must enable ip fast

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-06-30 Thread Ingo Flaschberger
Dear Alex, OK, I setup 2 boxes on either end of a RELENG_7 box from about May 7th just now, to see with 2 boxes blasting across it how it would work. *However*, this is with no firewall loaded and, I must enable ip fast forwarding. Without that enabled, the box just falls over.

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-06-30 Thread Wilkinson, Alex
0n Tue, Jul 01, 2008 at 03:00:31AM +0200, Ingo Flaschberger wrote: Dear Alex, OK, I setup 2 boxes on either end of a RELENG_7 box from about May 7th just now, to see with 2 boxes blasting across it how it would work. *However*, this is with no firewall

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-06-30 Thread Ingo Flaschberger
Dear Alex, if possible, a ned header is created at the other network-cards-buffer and the ip-data is copied from network-card-buffer to network-card-buffer directly. So how does one enable ip fast forwarding on FreeBSD ? sysctl -w net.inet.ip.fastforwarding=1 usually interface

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-06-30 Thread Support (Rudy)
Ingo Flaschberger wrote: usually interface polling is also chosen to prevent lock-ups. man polling I used polling in FreeBSD 5.x and it helped a bunch. I set up a new router with 7.0 and MSI was recommended to me. (I noticed no difference when moving from polling - MSI, however, on 5.4

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-06-30 Thread Steve Bertrand
Wilkinson, Alex wrote: So how does one enable ip fast forwarding on FreeBSD ? Not to take anything away from Ingo's response, but to inform how to add the functionality to span across reboots, add the following line to /etc/sysctl.conf net.inet.ip.fastforwarding=1 Steve

Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

2008-06-30 Thread Steve Bertrand
Support (Rudy) wrote: Ingo Flaschberger wrote: usually interface polling is also chosen to prevent lock-ups. man polling I used polling in FreeBSD 5.x and it helped a bunch. I set up a new router with 7.0 and MSI was recommended to me. (I noticed no difference when moving from polling -

  1   2   >