Hi Mihai, On Thu, Aug 25, 2011 at 10:16 AM, Mihai Dobrescu <[email protected]> wrote: > For the moment I don't have a proper setup to test the described > behavior, but the problem seems to be related to the batch settings in > the NIC driver. > > b1) An easy fix would be to try to change the the batch value from 16 > to 8 or 4 to see if the latency improves. Check the driver's source > folder (ixgbe_main.c) and look for something like rxdctl |= (16<<16); > txdctl |= (16<<16).
I tried adjusting this parameter to a few different settings. It worked with batching levels of 16 and 8. For some reason when I tried 4 and 2 packets we're delivered out of the driver regardless of how many I sent. Looking at the latency, it seems that for small bursts of packets (e.g., 8, 16, 32 and thereabouts) the latency through RouteBricks is very low--about 14us or so. When I try large packet bursts (e.g., 16,000 packets back to back) there is an interesting queueing-type behavior in which the latency for the earlier packets is 14us, and then it linearly climbs up to ~560us or so, and then jumps back down to 14 and repeats in a sawtooth pattern. This may be due to an RSS hash issue, given that the receive queue stats show that most packets end up in a single queue (vs being spread over the 4 queues that are instantiated in the nic). I'm going to have my packet generator randomize the source IP address to hopefully spread those packets over those queues and retry. Thanks again for your help, George _______________________________________________ click mailing list [email protected] https://amsterdam.lcs.mit.edu/mailman/listinfo/click
