Hi Dmitry, On Fri, Sep 11, 2015 at 01:58:42PM +0300, Dmitry Sivachenko wrote: > For reference: I tracked this down to be FreeBSD-specific problem: > https://lists.freebsd.org/pipermail/freebsd-net/2015-September/043314.html > > Thanks all for your help.
Thanks for the update. What I'm seeing in your description looks very much like equivalent issues we used to face with softirq on Linux, so it's possible that you're in the worst case where work cannot be aggregated but comes with a huge overhead. Also maybe you have pf or something like this eating some extra CPU. I can't be specific, I don't use FreeBSD myself, but like Linux it's a modern and performant OS so I think you'll come to a solution. I don't know if you can pin processes to CPUs but there could be interesting tests to run regarding how processes and interrupts are pinned. Also if your NIC supports multiple queues, you'll need to check how interrupts are delivered. It would be possible that you're facing a scalability issue in the network driver or stack, maybe just in case where too few sockets are used or when packets get highly reordered. Cheers, Willy
