TL;DR: To test the new, more efficient iflib transmit routine, please add this tunable to loader.conf net.iflib.prefer_mpring=0 and reboot.
Iflib is a network driver framework that is used by the most popular wired ethernet drivers in the tree. To avoid lock contention when multiple threads are competing for the same network interface queue, it uses a queuing mechanism called "mp_ring". I have written a simple replacement called "simple_tx" based on a bounded buf_ring. In most testing, including both uncontended and highly contended scenarios, its far faster than mp_ring and uses less CPU. I've done testing on large AMD servers (up to 96c/192t) and smaller Intel servers (8c/16t). The test was a custom in-kernel packet generator sending 2-mbuf chains with 42b UDP headers and 1300b UDP payloads in separate mbufs. Each packet generator thread was bound to a core, and traffic targeted a single NIC queue (Intel used Intel 10GbE ixl(4), and AMD used Broadcom 400GbE bnxt(4)). Test results here: https://people.freebsd.org/~gallatin/mpring_vs_simple_tx/ Olivier Cochard did testing for me on smaller home-gateway appliances, with these results for his packet forwarding benchmark: ┌──────────────────┬───────────┬────────────┬─────────┬─────────┐ │ image │ simple_tx │ median pps │ min │ max │ ├──────────────────┼───────────┼────────────┼─────────┼─────────┤ │ n312956D58901 │ off │ 468,280 │ 467,781 │ 469,373 │ ├──────────────────┼───────────┼────────────┼─────────┼─────────┤ │ n312956D58901 │ on │ 734,893 │ 721,068 │ 741,998 │ └──────────────────┴───────────┴────────────┴─────────┴─────────┘ This has been running stably on select canaries at work (using bnxt, ixgbe, ixl + an unreleased iflib driver) and on my personal machines (em, igb, igc) for months as I've worked on it. I've also done testing in a vm with ALTQ to make sure ALTQ still works (nothing can help ALTQ performance). I realize this code is run by a *LOT* of us, and I want to have a slow transition period. I plan to switch our fleet at work to simple_tx (it has been running on select canaries up until now). If there are no unfixed regressions found either at work, or reported here, I plan to make simple_tx the default after the next Stabilization week . After another Stabilization week, I plan to remove mp_ring from iflib in -current. I do not plan to MFC this, Thank you for testing, Drew
