On Thu, 2013-04-18 at 02:42 +0000, Xavier Trilla wrote: > Hi Eric, > > We have been doing some more research, and finally we managed to get a > fast kernel with a new minimal configuration using the last release of > the 2.6.32 (2.6.32.60). We compiled what we think is a kernel with > options that could affect network performance removed. We used our > general networking/computing knowledge and reviewed every option, so > I'm sure it's not perfect but should be a better start point than the > one we had. Then we did a "make oldconfig" to adapt this ".config" to > a 3.4.41 kernel reviewing every new option. > > But again we have the same situation, 2.6.32 kernel seems to be > extremely fast (we have successfully routed more than 2Mpps) and 3.4 > is not able to route more than 500Kpps, and again when we do a "perf > top" we see _raw_spin_lock_irqsave consuming a lot of CPU. > > BTW: Remember that we use for this test just one Xeon 5620 in a dual > Xeon environment, so 50% CPU usage means we are using the 100% of the > cores assigned to RSS queues. > > Kernel 3.4.41 CLEAN > > Packet generator: Two bonesi instances each one sending about 500Kpps > from 50K sources. > Machine reaches 100% CPU usage on cores with RSS queues assigned, and > doesn't route more than 500kpps. > > perf top output: http://pastebin.com/xByZnxL1 > perf record output: http://pastebin.com/2idhM3V1 > > Kernel 2.6.32.60 CLEAN > > Packet Generator: Two bonesi instances each one sending about 500Kpps > 50K sources. > Machine routes about 1Mpps (Bonesi doesn't generate exactly 500Kpps > per instance, so a small difference is normal). > > perf record output: http://pastebin.com/gfvcQNZv > perf top output: http://pastebin.com/8Qpp604p > > Note: We use a CISCO 3750G to measure the pps in each port, so we can > see number of packets coming in and out (Flow control is disabled on > the switch ports). > > As you may see, the main difference is that on kernel 3.4.41 > _raw_spin_lock_irqsave is using the 46.4% of the CPU (So, 92.8% of the > Cores assigned to RSS queues). > > Also, I'm starting to have doubts about this being related with IGB > driver... but anyway if you have any idea that could lead us in the > right direction it would be extremely useful :) > > P.S.: I'm not familiar at all with lists, so I just did a "reply all" > from my e-mail client but I see you'll get a direct copy of this > mail... Should it work like that or I shall just reply to the > [email protected] mail address? If I did wrong please > just let me know. > > And... Thanks for your help :) > > Saludos cordiales, > Xavier Trilla P. > Silicon Hosting >
Check on your .config files on both kernels : grep IOMMU .config Then try to boot your 3.4.41 machine adding "iommu=off" Or read the documentation about IOMMU in general... ------------------------------------------------------------------------------ Precog is a next-generation analytics platform capable of advanced analytics on semi-structured data. The platform includes APIs for building apps and a phenomenal toolset for data science. Developers can use our toolset for easy data analysis & visualization. Get a free account! http://www2.precog.com/precogplatform/slashdotnewsletter _______________________________________________ E1000-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
