Hey Hank, Well looks like all your traffic is just hashing to 2 queues. You have ATR enabled but it isn’t being used, do to this being UDP traffic. That isn’t a problem since RSS hash will occur on anything that doesn’t match ATR (in your case everything). All this means that you only have 2 flows and thus all the work is being done with only two queues. To get a better hash spread you could modify the RSS hash key, but I would first look at your traffic to see if you even have more than 2 flows operating. Maybe something can be done in the application to allow for more parallelism, run for threads for instances (assuming each thread opens its own socket)?
As for the rx_no_dma_resource counter it is tried in directly to one of our HW counters. It gets bumped if the target queue is disabled (unlikely in your case) or there are no free descriptors in the target queue. The later makes sense since all of your traffic is going to just two queue that appear to not be getting drained fast enough. Thanks, -Don <donald.c.skidm...@intel.com> From: Hank Liu [mailto:hank.tz...@gmail.com] Sent: Wednesday, September 07, 2016 4:51 PM To: Skidmore, Donald C <donald.c.skidm...@intel.com> Cc: Rustad, Mark D <mark.d.rus...@intel.com>; e1000-devel@lists.sourceforge.net Subject: Re: [E1000-devel] Intel 82599 AXX10GBNIAIOM cards for 10G SFPs UDP performance issue Hi Don, I got log for you to look at. See attached... Thanks and let me know. BTW, can anyone tell me what could cause rx_no_dma_resource? Hank On Wed, Sep 7, 2016 at 4:04 PM, Skidmore, Donald C <donald.c.skidm...@intel.com<mailto:donald.c.skidm...@intel.com>> wrote: ATR is application targeted receive. It may be useful for you but the flow isn’t directed to a CPU until you transmit and since you mentioned you don’t do much transmission it would have to be via the ACK’s. Likewise the flows will need to stick around for a while to gain any advantage from it. Still it wouldn’t hurt to test using the ethtool command Alex mentioned in another email. In general I would like to see you just go with the default of 16 RSS queues and not attempt to mess with the affinization of the interrupt vectors. If the performance is still bad I would be interested how the flows were being distributed between the queues. You can see this via packet counts per queue you get out of ethtool stats. What I want to eliminate is the possibility that RSS is seeing all your traffic as one flow. Thanks, -Don <donald.c.skidm...@intel.com<mailto:donald.c.skidm...@intel.com>> From: Hank Liu [mailto:hank.tz...@gmail.com<mailto:hank.tz...@gmail.com>] Sent: Wednesday, September 07, 2016 3:40 PM To: Rustad, Mark D <mark.d.rus...@intel.com<mailto:mark.d.rus...@intel.com>> Cc: Skidmore, Donald C <donald.c.skidm...@intel.com<mailto:donald.c.skidm...@intel.com>>; e1000-devel@lists.sourceforge.net<mailto:e1000-devel@lists.sourceforge.net> Subject: Re: [E1000-devel] Intel 82599 AXX10GBNIAIOM cards for 10G SFPs UDP performance issue Mark, Thanks! Test app can specify how many pthread to handle connections. I have tried 4, 8, 16, etc, but none of them make significant difference. CPU usage on receive end is moderate (50-60%). If I want to poll aggressively to prevent any drop in UDP layer, then it might go up a bit. On the CPU set that handle network interrupts, I did pin those CPUs, I can see interrupt rate is pretty even on all CPUs involved. Since seeing a lot of rx_no_dma_resource and this counter is read out through 82599 controller, I like to know why it happened. Note: I already bumped rx ring size to maximum (4096) I can set in ethtool. BTW, what is ATR? I didn't set up any filter... Hank On Wed, Sep 7, 2016 at 2:19 PM, Rustad, Mark D <mark.d.rus...@intel.com<mailto:mark.d.rus...@intel.com>> wrote: Hank Liu <hank.tz...@gmail.com<mailto:hank.tz...@gmail.com>> wrote: *From:* Hank Liu [mailto:hank.tz...@gmail.com<mailto:hank.tz...@gmail.com>] *Sent:* Wednesday, September 07, 2016 10:20 AM *To:* Skidmore, Donald C <donald.c.skidm...@intel.com<mailto:donald.c.skidm...@intel.com>> *Cc:* e1000-devel@lists.sourceforge.net<mailto:e1000-devel@lists.sourceforge.net> *Subject:* Re: [E1000-devel] Intel 82599 AXX10GBNIAIOM cards for 10G SFPs UDP performance issue Thanks for quick response and helping. I guess I didn't make it clear is that the application (receiver, sender) open 240 connections each connection has 34 Mbps traffic. You say that there are 240 connections, but how many threads is your app using? One per connection? What does the cpu utilization look like on the receiving end? Also, the current ATR implementation does not support UDP, so you are probably better off not pinning the app threads at all and trusting that the scheduler will migrate them to the cpu that is getting their packets via RSS. You should still set the affinity of the interrupts in that case. The default number of queues should be fine. -- Mark Rustad, Networking Division, Intel Corporation
------------------------------------------------------------------------------
_______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
