Just some ideas to help. I have an application that is trying to capture, modify, resequence, and route very small packets. The configuration I found the best is the following (for kernel parameters):
sysctl -w net.core.wmem_max=1310710 sysctl -w net.core.rmem_max=1310710 sysctl -w net.core.wmem_default=1249280 sysctl -w net.core.rmem_default=1249280 sysctl -w net.core.netdev_budget=1024 sysctl -w net.core.dev_weight=128 sysctl -w net.core.netdev_max_backlog=10000 sysctl -w net.ipv4.udp_mem="11416320 15221760 22832640" sysctl -w net.core.optmem_max=40960 I also used ethtool with the -G option to increase the nic's hardware buffers to their max. The machine it is running on is pretty beefy, but cpu cycles are not limiting the throughput. I found the factor that effected performance the most is cache coherency and limiting inter-process interrupts. The best way I found to do this was to have the application that is receiving the packets running on the same cores that the kernel is pulling them off the nic. Since my machine had two chips each with 4-6 cores (8-12 logical cores), I limited my application to run on the same chip as the nic was receiving its interrupts. To control which logical core the nic handles the interrupts: cat /proc/interrupts | grep eth[number] That should show you the irq(s) that handles the interrupts. Then do something like: echo "1" > /proc/irq/[irq#_from_above]/smp_affinity to set what core handles the interrupt (1 is a bitmask saying to use the first logical core). Finally do something like: taskset -p 0x5 [your application i.e. tcpdump] (0x5 is a bitmask saying which cpu cores to use) to control what cores the process gets scheduled on. I found this the best way to get the most throughput out of the application, and the most likely way to reproduce results. Without doing this, I would get some runs that performed well, and others that were junk, and most that wavered. Oh, stop the irqbalancer service as that may overwrite what you did to the /proc/irq/# Also use top to monitor your top processes. If you see a ksoftirqd service getting lots of cpu cycles, you will be dropping packets and may be a good indicator that you should try a different irq#. Good luck. If you find any other tips to squeeze more performance please write back as I am also trying to squeeze the most performance possible out of my application. - Gabe Black -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of M. V. Sent: Sunday, January 23, 2011 9:42 AM To: tcpdump-workers Subject: [tcpdump-workers] HUGE packet-drop hi guys, i'm stucked with a confusing problem. i'm trying to dump gigabit network traffic to file. i'm using Debian5.0.3 AMD64 on an HP Proliant DL360 G5 (2*Quad-Core 2.33 hz - 4GiB RAM - 2*76GiB HardDisk). but whatever i try, even at 600Mbps rate, i got a huge packet-drop. *) i tried both tcpdump (1.1.1) && dumpcap (1.4.3) : (got similar results) - dumpcap -i eth0 -w /dump_folder/dump.pcap [-b filesize:150000] -s 0 - tcpdump -i eth0 -w /dump_folder/dump.pcap -s 0 *) i tried recompiling those two, with different libpcap versions (0.9.8 has best result, and i tried 1.0 and .1.1.1) *) i tried playing with kernel parameters: net.core.rmem_default = 40971510 net.core.rmem_max = 40971510 net.core.wmem_default = 40971510 net.core.wmem_max = 40971510 net.core.netdev_max_backlog = 300000 net.ipv4.tcp_timestamps = 0 net.ipv4.tcp_sack = 1 net.ipv4.tcp_window_scaling = 1 net.core.optmem_max = 2048000 .... *) i even tried writing on different disks (SDD, RamDisk): no change in results. ----------------------------------------------------------------------------------------- the result is not good at all. these are my tests and results: * iperf server: iperf -s -u 5 * iperf client : iperf -c ... -u -b 600m -t 10000 for dumping traffic at 600Mbps for 150 seconds, this is the best result i had so far: total received packets: ~8,000,000 dropped packets: ~3,500,000 so my best result is more than 40% packet-drop, which is a disaster. can anyone help me with this? any suggestions? am i doing something terribly wrong? thank you. - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe. - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
