Hello, The IRQs are managed by IO APIC, and ends on the LAPIC (integrated to the CPU). Behavior depends on its hardware implementation, but the most common is "dispatch the interrupt to the first available handler, starting at 0".
The indirection table is a 4 tuple of src/dst ip/port. It's pretty unpleasant to get/set the correct table. Instead, your NIC should provide '--config-ntuple' / '-N' feature, to set readable & efficient rules like : # Send all http traffic to queue 3: ethtool -N eth0 rx-flow-hash tcp4 dst-port action 3 Once the CPU receives an interrupt, the kernel schedules the local ksoftirqd process and masks the IRQs (to avoid being notified again for something he already knows: he has network packets to process). ksoftirqd will call the NAPI poll function, and poll packets from the NIC, being limited by 2 sysctls: net.core.netdev_budget (max number of packets to fetch, on all ifaces) and net.core.netdev_budget_usecs (max usecs to spend polling packets). This set of packets is then processed by the higher networking layers. I've started (and never finished..) a map of the kernel network path [1] [2] and related metrics/sysctls... would that be of any use for some of you ? I should probably finish it.. To go with that, here's a small perl script [3] that helps get/set the IRQs & process affinities in a more readable way. [1] : https://www.draw.io/#Uhttps%3A%2F%2Fraw.githubusercontent.com%2FSaruspete%2FLinuxNetworking%2Fmaster%2Fdiagrams%2FNetwork-Receive.xml [2] : https://github.com/Saruspete/LinuxNetworking [3] : https://github.com/Saruspete/smp_affinity Cheers, Adrien. On Friday, March 15, 2019 at 6:10:45 PM UTC+1, Wojciech Kudla wrote: > > You can affinitze (almost all) irqs to specific cpu(s) by manipulating the > mask in /proc/interrupts/$irq/smp_affinity. > You need to make sure irq balancing is disabled as well. > These two measures combined should give you ultimate control over which > cups process which interrupts. > > > On Fri, 15 Mar 2019, 17:04 Peter Veentjer, <[email protected] > <javascript:>> wrote: > >> Hi Wojciech, >> >> thank you for reply. >> >> It should remain static and I see it is quite static. But what prevents >> that an arbitrary CPU is picking up an interrupt? >> >> Perhaps it is my interpretation of the following sentence: >> >> "By default,an IRQ may be handled on any CPU" from >> https://www.kernel.org/doc/Documentation/networking/scaling.txt >> >> I guess it means 'in most cases the same CPU will process it, but no hard >> guarantees are given. >> >> Instead of 'each interrupt for the same IRQ will be processed by a random >> CPU' >> >> >> >> On Friday, March 15, 2019 at 5:08:53 PM UTC+2, Wojciech Kudla wrote: >>> >>> The rx-queue to CPU affinity you're referring to should remain fairly >>> static if the packets are getting processed fast enough. >>> If you are interested in controlling this behavior you can manipulate >>> receive flow hash indirection tables (ethtool - x) , irq affinities >>> (/proc/interrupts/$irq/smp_affinity) >>> or configure relevant stack if you're using solarflare (through onload >>> profile). >>> >>> Hope this helps. >>> >>> >>> On Fri, 15 Mar 2019, 14:40 Peter Veentjer, <[email protected]> wrote: >>> >>>> I have a question about RSS and how a CPU is selected to process the >>>> interrupt. >>>> >>>> So with RSS you have multiple rx-queue and each rx-queue has an IRQ >>>> associated to it. Each CPU can pick up the IRQ as clearly explained here: >>>> >>>> https://www.kernel.org/doc/Documentation/networking/scaling.txt >>>> >>>> It is possible to using IRQ/CPU affinity to prevent that just any CPU >>>> picking up the IRQ. >>>> >>>> My question is about the default behavior when no explicit affinity has >>>> been set. >>>> >>>> Is there any advantage of letting the same rx-queue being processed by >>>> different CPU's? In case of a random CPU picking up same IRQ, you could >>>> end >>>> a packet being processed at a different CPU for every packet being >>>> received. This will cause a lot of cache coherence traffic which will >>>> certainly not improve performance. >>>> >>>> When I look at /proc/interrupts of my local system, it seems the same >>>> CPU is processing a specific rx-queue. So it looks that the OS is already >>>> applying some form of affinity to map an IRQ to a CPU. >>>> >>>> Can someone shed a light on this? >>>> >>>> Thanks. >>>> >>> -- You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
