Hi Stefan, Ubuntu is a tricky one. I keep reminding our management that I'm noticing more people in the community are using it especially LTS. They (Ubuntu) make it hard though without versioning like SLES and RHEL, doing kcompat can be a challenge. Not that I think that is at all related to your issue. And you are correct we do validate and actively add compat support for RHEL and SLES. :)
I think ATR may work for you with your current system setup if you map the interrupts in the 1-to-1 fashion I mentioned below. Since you have your user space processes pinned to a given CPU all flows that transmit a SYN from that CPU will be redirected back to that queue/CPU pair. Not sure you would be effected much by the sampling rate as it rate is our mechanism for identify non-SYN packet flows, it is the rate at which we check a given packet to see if a ATR hash bucket exist for it and if not create one. That said I also believe your mechanism will work, possible even better since you don't have to wait for the ATR hash bucket to be created by the transmission of the SYN (meaning connections remotely initiated will most likely not have their initial SYN direct to the CPU you want) and won't have to deal with the limit on ATR hash buckets (if you have LOTS of ports open from one IP address). Still I would turn off ATR and thus default to RSS. This however will limit you to 16 queues, which depending on the size of your system may or may not be an issue. In truth you might want to look at perfect filter. With it you could direct let the adapter to send all traffic to a given queue. Which, from my limited understanding of what you're doing, looks to be just what you're looking for. Basically map an IP address to a give queue. Thanks, -Don Skidmore <donald.c.skidm...@intel.com> > -----Original Message----- > From: Stefan Puiu [mailto:stefan.p...@gmail.com] > Sent: Tuesday, December 09, 2014 2:28 AM > To: Skidmore, Donald C > Cc: e1000-devel@lists.sourceforge.net > Subject: Re: [E1000-devel] rx_no_dma_resources woes > > Hi Don, > > Thanks for your reply. > > On Mon, Dec 8, 2014 at 10:04 PM, Skidmore, Donald C > <donald.c.skidm...@intel.com> wrote: > > Hey Stefan, > > > > I'm not sure why you are seeing more rx_no_dma_resource on one > version of Ubuntu than another. We sadly don't validate against Ubuntu. > > Actually, we haven't found a version of Ubuntu that works :). > > I guess you're validating against RHEL and SLES? > > [...] > > Likewise ATR stands for Application Targeted Receive. It is on by default > > an > attempts to direct patch flows to the CPU which the SYN was sent out on. It > assumes a one to one mapping between queue number and CPU number > (what the set_irq_affinity script does). If your script is not doing the same > it > would defiantly undermined ATF and could lead to flows being directed and > in correct queue/CPU. Could differences in your script be causing an issue? > > I was talking about the "Ageing Timer Rollover" that was being discussed in > the other thread. Apparently that can't be set from the HP BIOS. I guess > you're talking about the NIC setting that identifies identical flows. I'm not > sure we need ATR, since we expect traffic coming from 8 src IPs and we > assign each src IP to its own core. I noticed there's an AtrSampleRate > setting, > but that's not available in the driver that ships with our kernel. > > We don't use set_irq_affinity because we want to associate queues from NIC > 1 to cores from CPU0 (node0) and queues from NIC to cores from CPU1; we > have two processes, each running on one CPU core and each handling one > NIC. The system we run on is a 2-node NUMA with 16 cores each, so I'm not > handling cpus >= 32. As far as I can tell, we're not assigning the same > interrupt / queue to the same CPU or the other way around. > > Snipping from our script ($DEV and $NODE are received as arguments): > > for i in `seq 0 7`; do > ip_digit=`expr $i + 10` > ethtool -U $DEV flow-type ip4 src-ip 10.10.10.$ip_digit action $i done > [...] > startCpu=$(cat /sys/devices/system/node/node${NODE}/cpulist|cut -d- -f1) > > IRQS=`grep $DEV-TxRx /proc/interrupts | cut -d: -f1 | tr -d " "` if [ -z > "$IRQS" ]; > then > echo "No interrupts assigned to $DEV, maybe device is down?" > exit 1 > fi > > cpu=$startCpu > for irq in $IRQS; do > mask=$((1<<$cpu)) > mask_hex=`printf "%X" $mask` > echo "mask=$mask_hex irq=$irq" > printf " %s" $mask_hex > /proc/irq/$irq/smp_affinity > cpu=`expr $cpu + 1` > done > > Thanks, > Stefan. ------------------------------------------------------------------------------ Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration & more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
