Thanks Mihai, >> Looking at the latency, it seems that for small bursts of packets >> (e.g., 8, 16, 32 and thereabouts) the latency through RouteBricks is >> very low--about 14us or so. > > That's good news :)
Yep. I believe that this single packet or small groups of packet latency results show the benefits of a polling 10G driver (since none of the other RouteBricks-specific functionality is being engaged, such as the multi-queue support and/or the distributed overlay forwarding network). Adding NAPI support to the ixgbe driver would be really great to prevent packets from getting "stuck" in the ixgbe driver (which prevents things like ssh from working since the SYN packet isn't forwarded to the destination. >> When I try large packet bursts (e.g., 16,000 packets back to back) >> there is an interesting queueing-type behavior in which the latency >> for the earlier packets is 14us, and then it linearly climbs up to >> ~560us or so, and then jumps back down to 14 and repeats in a sawtooth >> pattern. > > There are 2 forms of batching: NIC batching and Click batching. The > described behavior might be related to Click batching. You could check > the "BURST" parameters in PollDevice and ToDevice to adjust the Click > batching. I will try that today. However the sawtooth is over 10,000+ packets, and so I don't think it is related to a small constant amount of batching. Otherwise I would expect latency of packets to go up for 8-16 packets then go back down. What I see now is that the first 10-20 packets have latencies in the 12-14us range, then it linearly goes up to ~500-1000us over the next 10,000 packets, then there is a discontinuous jump back to 14us and the process repeats itself. I think maybe the issue is that only one kernel thread is handling the packets at this time. > The RSS delivery to a particular queue is done based on the > (SrcIP,DstIP,SrcPort,DstPort) tuple. If you send packets using the > same headers, they will all end up in the same queue. I'm going to vary the 5-tuple and try again to see if I can spread the packets across the nic queues, which should uniformly spread the packets across those nic queues. Thanks, George _______________________________________________ click mailing list [email protected] https://amsterdam.lcs.mit.edu/mailman/listinfo/click
