>-----Original Message----- >From: Zoop [mailto:[email protected]] >Sent: Tuesday, December 20, 2011 7:07 AM >To: [email protected] >Subject: [E1000-devel] Problem with 82575GB Card and packets/s >capabilities. > >I have a Dell 710 server using two 82575GB Intel Quad cards. I am using >two >interfaces on each card and using the bonding driver within linux. I >have run >into a problem where it seems when one of the interfaces in the bonding >pair >hits about 116,000 packets/s that it is unable to handle more traffic. >Odd >thing is I don't really seem to see any interface errors via Ethtool >etc. > >What I do notice is that when it does do this the system load avg goes >above 1. > I have seen one of the ksoftirqd process sit at about 70%. Sometimes a >restart >of irqbalance will cause this to move or clear the issue. I an not >familiar >enough with the irq affinity settings to play with it myself yet. > >I would expect to be able to get a lot more out of the cards I am >assuming >something is wrong with my setup. I am using the default settings for >the igb >driver and have made some adjustments to the kernel itself. Generally >most of >the packets through the system are being routed. > >settings via the proc that I changed >net.core.optmem_max = 20480 ># Increase number of incoming connections backlog >net.core.netdev_max_backlog = 4000 >net.core.dev_weight = 64 ># # Bump up default r/wmem to max >net.core.rmem_default = 262141 >net.core.wmem_default = 262141 ># # Bump up max r/wmem >net.core.rmem_max = 262141 >net.core.wmem_max = 262141 > >This is a Dual Quad core 2.4Ghz Intel system > > >Kernel messages for one of the interfaces: >[ 5.404672] igb 0000:0d:00.0: Intel(R) Gigabit Ethernet Network >Connection >[ 5.404677] igb 0000:0d:00.0: eth6: (PCIe:2.5Gb/s:Width x4) >00:1b:21:14:34:54 >[ 5.404762] igb 0000:0d:00.0: eth6: PBA No: E34573-001 >[ 5.404765] igb 0000:0d:00.0: Using MSI-X interrupts. 4 rx queue(s), >4 tx >queue(s) >[ 5.404796] igb 0000:0d:00.1: PCI INT B -> GSI 50 (level, low) -> IRQ >50 >[ 5.404807] igb 0000:0d:00.1: setting latency timer to 64 >[ 5.405636] igb 0000:0d:00.1: irq 155 for MSI/MSI-X >[ 5.405645] igb 0000:0d:00.1: irq 156 for MSI/MSI-X >[ 5.405655] igb 0000:0d:00.1: irq 157 for MSI/MSI-X >[ 5.405665] igb 0000:0d:00.1: irq 158 for MSI/MSI-X >[ 5.405675] igb 0000:0d:00.1: irq 159 for MSI/MSI-X >[ 5.405684] igb 0000:0d:00.1: irq 160 for MSI/MSI-X >[ 5.405694] igb 0000:0d:00.1: irq 161 for MSI/MSI-X >[ 5.405703] igb 0000:0d:00.1: irq 162 for MSI/MSI-X >[ 5.405720] igb 0000:0d:00.1: irq 163 for MSI/MSI-X > > >Some vmstat output > >procs -----------memory---------- ---swap-- -----io---- -system-- ---- >cpu---- > 1 0 264 2018016 474692 4344560 0 0 0 0 180402 1209 0 >4 95 0 > 1 0 264 2018296 474692 4344616 0 0 0 0 189968 1186 0 >3 97 0 > 0 0 264 2017896 474692 4344668 0 0 0 0 183079 1345 0 >3 97 0 > 0 0 264 2017892 474692 4344728 0 0 0 16 189246 1161 0 >7 93 0 > 1 0 264 2018076 474692 4344772 0 0 0 0 177470 1013 0 >6 94 0 > 0 0 264 2017972 474692 4344820 0 0 0 0 185231 1145 0 >3 97 0 > 1 0 264 2018040 474692 4344864 0 0 0 16 165925 1577 0 >7 93 0 > 1 0 264 2017820 474692 4344932 0 0 0 0 173961 1245 0 >7 93 0 > 1 0 264 2017812 474692 4344988 0 0 0 20 184350 1282 0 >3 97 0 > 0 0 264 2018040 474692 4345048 0 0 0 0 177809 1185 0 >3 97 0 > 0 0 264 2017864 474692 4345108 0 0 0 0 181536 1217 0 >3 97 0 > 0 0 264 2017860 474692 4345164 0 0 0 0 181901 1143 0 >4 96 0 > 0 0 264 2017704 474692 4345216 0 0 0 0 170947 1233 0 >9 90 0 > 1 0 264 2017660 474692 4345272 0 0 0 20 173678 1087 0 >6 94 0 > 1 0 264 2016904 474692 4345928 0 0 0 1036 184045 1338 0 >5 95 0 > 1 0 264 2010440 474692 4351944 0 0 0 0 179177 1356 0 >7 92 0 > 1 0 264 2010300 474692 4352624 0 0 0 904 167765 1321 0 >9 91 0 > 1 0 264 2010460 474692 4352068 0 0 0 148 195913 1262 0 >4 96 0 > 0 0 264 2010464 474692 4352136 0 0 0 8 192341 1943 0 >4 96 0 > 0 0 264 2010412 474692 4352220 0 0 0 0 179244 1673 0 >4 95 0 > 0 0 264 2010516 474692 4352292 0 0 0 0 168384 1335 0 >5 95 0 > 0 0 264 2010020 474692 4352364 0 0 0 16 188664 1350 0 >3 97 0 > > >If you think the graphs and data from the interface and load on the >server would >be helpful let me know I will figure out how to get them available to >you. > >Let me know what other information I can provide here. > >I did see this post and it looks like it is related maybe to what I am >doing, >but it seems to not have completed. > >http://comments.gmane.org/gmane.linux.drivers.e1000.devel/7221 > >Thank you. > > > >------------------------------------------------------------------------ >------ >Write once. Port to many. >Get the SDK and tools to simplify cross-platform app development. Create >new or port existing apps to sell to consumers worldwide. Explore the >Intel AppUpSM program developer opportunity. appdeveloper.intel.com/join >http://p.sf.net/sfu/intel-appdev >_______________________________________________ >E1000-devel mailing list >[email protected] >https://lists.sourceforge.net/lists/listinfo/e1000-devel >To learn more about Intel® Ethernet, visit >http://communities.intel.com/community/wired
Hello, What bonding mode are you using in this setup? Can you check, when the system gets into this state, whether you can ping the device or the bond directly? Can you provide a dmesg output during the problem state and also lspci -vv for the system. Thanks, Carolyn Carolyn Wyborny Linux Development LAN Access Division Intel Corporation ------------------------------------------------------------------------------ Ridiculously easy VDI. With Citrix VDI-in-a-Box, you don't need a complex infrastructure or vast IT resources to deliver seamless, secure access to virtual desktops. With this all-in-one solution, easily deploy virtual desktops for less than the cost of PCs and save 60% on VDI infrastructure costs. Try it free! http://p.sf.net/sfu/Citrix-VDIinabox _______________________________________________ E1000-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
