On Fri, 2016-09-16 at 10:57 -0700, Michael Ma wrote:
> This is actually the problem - if flows from different RX queues are
> switched to the same RX queue in IFB, they'll use different processor
> context with the same tasklet, and the processor context of different
> tasklets might be the same. So multiple tasklets in IFB competes for
> the same core when queue is switched.
> The following simple fix proved this - with this change even switching
> the queue won't affect small packet bandwidth/latency anymore:
> in ifb.c:
> - struct ifb_q_private *txp = dp->tx_private +
> + struct ifb_q_private *txp = dp->tx_private +
> (smp_processor_id() % dev->num_tx_queues);
> This should be more efficient since we're not sending the task to a
> different processor, instead we try to queue the packet to an
> appropriate tasklet based on the processor ID. Will this cause any
> packet out-of-order problem? If packets from the same flow are queued
> to the same RX queue due to RSS, and processor affinity is set for RX
> queues, I assume packets from the same flow will end up in the same
> core when tasklet is scheduled. But I might have missed some uncommon
> cases here... Would appreciate if anyone can provide more insights.
Wait, don't you have proper smp affinity for the RX queues on your NIC ?
( Documentation/networking/scaling.txt RSS IRQ Configuration )
A driver ndo_start_xmit() MUST use skb_get_queue_mapping(skb), because
the driver queue is locked before ndo_start_xmit()) (for non
NETIF_F_LLTX drivers at least)
In case of ifb, __skb_queue_tail(&txp->rq, skb); could corrupt the skb
In any case, you could have an action to do this before reaching IFB.