On Fri, 2016-09-16 at 10:57 -0700, Michael Ma wrote: > This is actually the problem - if flows from different RX queues are > switched to the same RX queue in IFB, they'll use different processor > context with the same tasklet, and the processor context of different > tasklets might be the same. So multiple tasklets in IFB competes for > the same core when queue is switched. > > The following simple fix proved this - with this change even switching > the queue won't affect small packet bandwidth/latency anymore: > > in ifb.c: > > - struct ifb_q_private *txp = dp->tx_private + > skb_get_queue_mapping(skb); > + struct ifb_q_private *txp = dp->tx_private + > (smp_processor_id() % dev->num_tx_queues); > > This should be more efficient since we're not sending the task to a > different processor, instead we try to queue the packet to an > appropriate tasklet based on the processor ID. Will this cause any > packet out-of-order problem? If packets from the same flow are queued > to the same RX queue due to RSS, and processor affinity is set for RX > queues, I assume packets from the same flow will end up in the same > core when tasklet is scheduled. But I might have missed some uncommon > cases here... Would appreciate if anyone can provide more insights.
Wait, don't you have proper smp affinity for the RX queues on your NIC ? ( Documentation/networking/scaling.txt RSS IRQ Configuration ) A driver ndo_start_xmit() MUST use skb_get_queue_mapping(skb), because the driver queue is locked before ndo_start_xmit()) (for non NETIF_F_LLTX drivers at least) In case of ifb, __skb_queue_tail(&txp->rq, skb); could corrupt the skb list. In any case, you could have an action to do this before reaching IFB.
