> >From: Scott Silverman [mailto:[email protected]] >Sent: Tuesday, December 24, 2013 10:03 AM >To: Skidmore, Donald C >Cc: [email protected] >Subject: Re: [E1000-devel] RSS Configuration in ixgbe (Queue limit?) > >Don, > >I'm not sure I understand your answer. If the driver can only filter to 16 >queues, what are the extra 24 queues in the default 40-queue count doing? They >definitely receive, and >service, interrupts according to /proc/interrupts. > >What is the difference between the (up to) 16 queues I can configure with RSS >and the default 40 that the driver configures? > >In my particular situation I seem to find that I have a lot more >rx_no_dma_resources with the default 40, and quite a bit fewer with RSS=16,16. >I was hoping 24 might be a >goldilocks style "just right" but I can only get >that many when my hardware has 24 cores (hardware that I don't have any >rx_no_dma_resources problems with). > > > > > >Thanks, > >Scott Silverman > >On Tue, Dec 24, 2013 at 11:25 AM, Skidmore, Donald C ><[email protected]> wrote: >> -----Original Message----- >> From: Scott Silverman [mailto:[email protected]] >> Sent: Tuesday, December 24, 2013 6:28 AM >> To: [email protected] >> Subject: [E1000-devel] RSS Configuration in ixgbe (Queue limit?) >> >> In attempting to troubleshoot the issue with ring buffer drops I wrote about >> in another thread, I've noticed some behavior in ixgbe that does not match >> documentation. >> >> Configuring RSS (with RSS=#,#) to a value between 1-16 works as expected. >> Setting no value (or setting 0, default) configures as many queues as there >> are cores (on a 20 core box with hyperthreading, that is 40 cores per >> interface). The documentation says that it will only configure the lesser of >> cores or 16. >> >> Attempting to manually set a value higher than 16 fails, and acts as if a >> value >> of 0 was entered. (i.e. RSS=24,24 has no effect, and the full 40 queues are >> created). >> >> Should it be stopping at 16? If not, why can't I manually choose queue counts >> of 17-(cores-1) queues? >> >> >> Thanks, >> >> Scott Silverman >Hey Scott, > >While the driver can only use RSS to filter to 16 queue it can support more >queues in general. And since ATR is on by default and can support your 40 >queues so they are all >allocated. > >Thanks, >-Don Skidmore <[email protected]> > >Hey Scott,
The RSS has on this hardware only produces a 4bit mask and thus we can only direct to 16 queues. ATR however can use all the queues. So what happens is incoming patches are first checked to see if they fall in an ATR bucket. If the packet does it is put on that flows queue (in your case any of the 40) if there are no matches the HW sends it to whatever RSS hashes it to (one of the 16 queues). For ATR to work you need to have one queue per CPU, the whole idea is to place ingress flows on the same CPU as the egress packets from that flow. Thus on a system with 40 logical CPU's you have to have 40 queues. So for example you could test your "goldilocks" theory by limiting the CPU or turning off hyper-threading. As for the rx_no_dma_resources counts, if for the 40 queue case, your traffic may only contains a limited number of flows so you might be only scheduling to a limited number of queues/CPU's and over whelming them? When your test is running do you see the interrupt counts going up on all the devices queues? (cat /proc/interrupts | grep ethX) Thanks, -Don Skidmore <[email protected]> ------------------------------------------------------------------------------ Rapidly troubleshoot problems before they affect your business. Most IT organizations don't have a clear picture of how application performance affects their revenue. With AppDynamics, you get 100% visibility into your Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro! http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk _______________________________________________ E1000-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
