On Tuesday 24 October 2006 00:00, Sylwester S. Biernacki wrote: > Hello, > > about a month ago I wrote I'm glad about em(4) driver which works > pretty well on few of my boxes. However I need to change my > opinion... after what I saw today in the lab: > > We have connected pretty well testing box - Navtel InterWatch > (www.navtelcom.com). > It has one 6 slots and in one of them it has 2 GigE TX ports. > We've configured the following scenario: > > 1) > Navtel port A --- em0 --- em1 --- Navtel port B > > 2) > Navtel port A --- em2 --- em3 --- Navtel port B > > em0 and em1 are built-in mainboard: > em0 at pci4 dev 0 function 0 "Intel PRO/1000 PT (82571EB)" rev 0x06: apic 2 > int 16 (irq 10), em1 at pci4 dev 0 function 1 "Intel PRO/1000 PT (82571EB)" > rev 0x06: apic 2 int 17 (irq 11), > > and em2 and em3 are Intel Dual Port Server Adapter on PCI-e (4x): > em2 at pci5 dev 0 function 0 "Intel PRO/1000MT (82573E)" rev 0x03: apic 2 > int 19 (irq 10), em3 at pci6 dev 0 function 0 "Intel PRO/1000MT (82573L)" > rev 0x00: apic 2 int 16 (irq 11), > > pf is disabled, between em0 and em1 whole traffic goes through > kernel routing process (Navtel port A and em0 in one /24 network, > and em1 and Navtel port B are in different /24 network) > > sysctl tcp.send & receive space is turned to 65535 > > When we generate 500Mbps of traffic (1434 bytes in ethernet, which > gives 1400 bytes of payload in TCP) from port A to B and from B to A > (two streams, each of 500Mbps) everything works pretty well on > PT chips and MT chips. > > When we change payload to 64bytes (called IP killer :P) and put it > in scenario 1) machine gets freeze and no packet is comming to port > B of Navtel device. It's rather normal, there are no ASIC-based > boxes which can work with such traffic :) > > What was strange, when we connected Navtel to em2 and em3 which is one > PCI-e (4x) dual-port card and started to generate traffic from port > A to B and from B to A machine has restarted about a second after test > started. > > We've change sysctl values to move machine to debugger if anything > goes bad, but it didn't change anything. I think that it's somehow > connected to chipsets of that cards and mainboard bridges which are > responsible for transferring packets through the mainboard. > > Anyway, I started to feel badly about Intel... > > Second test was to generate 900Mbps of pure IP traffice (payload > 1400 bytes) from port A to B and second stream from B to A. > In scenario 1 machine got freeze and hasn't forward any packet from > em0 to em1. When we changed em0/1 to em2/3 all traffic is comming > from port A to B without any loss and machine gets 75% interrupt on > uniprocessor kernel. > > So, we've changed to MP kernel and... scenario 1) hasn't changed at all > (freeze all the time), and scenario 2 got 100% idle and 0% > interrupts on both CPUs (strange, I thought that it'll be 1/2 of > previous 75% :P). Anyway, when we connected anything to em0/1 ports > during that test and generate more than 100Mbps (bittwist software > packet generator run at the second box) our test machine got freeze > again... > > What else? Kernel is taken from CVS -current tree. > > After all these tests I'm changing my opinion about Intel cards, > especially when I read that PT chipsets are Intel's newest "baby". > > Does anyone got simmilar problems ? > Maybe there are other ways to tune NICs to work under such traffic > (buffers on NIC?). I'm not an expert in Intel network cards so any > idea will be appreciated :) > > Maybe you can tell about other chipsets that works fine under such > "heavy" traffic ?
I have read that people have tested with *very* high load with success... I am not the best expert....but you don't say anything about the OpenBSD config. At high load you probably have to change net.inet.ip.ifq.maxlen, kern.maxclusters, net.inet.tcp.recvspace, net.inet.tcp.sendspace, net.inet.tcp.rfc1323, kern.somaxconn among some other things... If you for example run out of "maxclusters" the server will freeze (as you mentioned)... Try OpenBSD FAQ ;-) After 3.8, major performance gains regarding cpu/interrupt load has been done in the em driver among many other fixes in the driver. Can't see you even mention the OpenBSD version you use.... And another thing... SMP kernel wont help if (but you don't) you run PF as it can't make use of SMP. Maybe it could be worse... But I don't know for sure. When it comes to the NIC;s, many on the list will probably tell you the marvel chip is a good one. But you probably know that if you read the list. But it will probably wont help you if you don't tune the server right anyway... /Per-Olov
