I hope the following makes sense. The following comes from the developers. questions: 1 .did you run any traffic or NIC was in idle state? 2. did you create VFs?
please check the following repro steps to see if they match yours
1. on diskless the boot script was changed to: intel_iommu=on
irqaffinity=0-1,24-25
/proc/irq/default_smp_affinity is "000...000,03000003"
2. compiled, installed, and reloaded both the i40e and iavf drivers
3. /proc/interrupts shows:
CPU0 CPU0 CPU1 CPU2 CPU3 CPU4
CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11
CPU12 CPU13 CPU14 CPU15 CPU16 CPU17 CPU18
CPU19 CPU20 CPU21 CPU22 CPU23 CPU24 CPU25
CPU26 CPU27 CPU28 CPU29 CPU30 CPU31 CPU32
CPU33 CPU34 CPU35 CPU36 CPU37 CPU38 CPU39
CPU40 CPU41 CPU42 CPU43 CPU44 CPU45 CPU46
CPU47 CPU48 CPU49 CPU50 CPU51 CPU52 CPU53
CPU54 CPU55 CPU56 CPU57 CPU58 CPU59 CPU60
CPU61 CPU62 CPU63
0: 82 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
IR-IO-APIC-edge timer
8: 58 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
IR-IO-APIC-edge rtc0
9: 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
IR-IO-APIC-fasteoi acpi
10: 232 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
IR-IO-APIC-edge ipmi_si
16: 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
IR-IO-APIC-fasteoi i801_smbus
24: 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
IR-PCI-MSI-edge PCIe PME, pciehp
@
4. created 2 VFs on first interface of NIC.
5. /proc/interrupts shows:
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5
CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12
CPU13 CPU14 CPU15 CPU16 CPU17 CPU18 CPU19
CPU20 CPU21 CPU22 CPU23 CPU24 CPU25 CPU26
CPU27 CPU28 CPU29 CPU30 CPU31 CPU32 CPU33
CPU34 CPU35 CPU36 CPU37 CPU38 CPU39 CPU40
CPU41 CPU42 CPU43 CPU44 CPU45 CPU46 CPU47
CPU48 CPU49 CPU50 CPU51 CPU52 CPU53 CPU54
CPU55 CPU56 CPU57 CPU58 CPU59 CPU60 CPU61
CPU62 CPU63
0: 82 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
IR-IO-APIC-edge timer
8: 58 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
IR-IO-APIC-edge rtc0
9: 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
IR-IO-APIC-fasteoi acpi
10: 232 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
IR-IO-APIC-edge ipmi_si
16: 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
IR-IO-APIC-fasteoi i801_smbus
24: 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
IR-PCI-MSI-edge PCIe PME, pciehp
25: 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
IR-PCI-MSI-edge PCIe PME
@
Todd Fujinaka
Software Application Engineer
Data Center Group
Intel Corporation
todd.fujin...@intel.com
-----Original Message-----
From: Chris Friesen <chris.frie...@windriver.com>
Sent: Friday, January 15, 2021 12:20 PM
To: Fujinaka, Todd <todd.fujin...@intel.com>; e1000-devel@lists.sourceforge.net
Subject: Re: [E1000-devel] IRQ affinity not working for iavf devices?
The irqbalance daemon is not running. At the time of the analysis, "cat
/proc/irq/<NUM>/smp_affinity_list" showed "0-1,24-25" but cat /proc/interrupts
showed the interrupt counts increasing on CPU 4.
If the interrupt affinity was modified by a userspace tool like irqbalance, I
believe that /proc/irq/<NUM>/smp_affinity_list would show the updated affinity.
This is the thing that I find really weird, the mismatch between the configured
affinity and the observed interrupt counts.
Chris
On 1/15/2021 11:03 AM, Fujinaka, Todd wrote:
> I've been reminded that there are other daemons that could be moving around
> the CPU affinities, such as irqbalance. Make sure that daemon isn't started.
>
> Todd Fujinaka
> Software Application Engineer
> Data Center Group
> Intel Corporation
> todd.fujin...@intel.com
>
> -----Original Message-----
> From: Fujinaka, Todd <todd.fujin...@intel.com>
> Sent: Friday, January 15, 2021 8:44 AM
> To: Chris Friesen <chris.frie...@windriver.com>;
> e1000-devel@lists.sourceforge.net
> Subject: Re: [E1000-devel] IRQ affinity not working for iavf devices?
>
> Sorry for the slow response. I'm asking around internally as I'm not that
> familiar with iavf. I'll let you know when I hear back.
>
> Todd Fujinaka
> Software Application Engineer
> Data Center Group
> Intel Corporation
> todd.fujin...@intel.com
>
> -----Original Message-----
> From: Chris Friesen <chris.frie...@windriver.com>
> Sent: Tuesday, January 12, 2021 3:53 PM
> To: e1000-devel@lists.sourceforge.net
> Subject: [E1000-devel] IRQ affinity not working for iavf devices?
>
> Hi,
>
> I have a CentOS 7 linux system with 48 logical CPUs and a number of
> Intel NICs running the i40e driver. It was booted with
> irqaffinity=0-1,24-25 in the kernel boot args, resulting in
> /proc/irq/default_smp_affinity showing "0000,03000003". CPUs 2-11 are set
> as "isolated" in the kernel boot args.
>
> The iavf driver is 3.7.61.20 and the i40e driver is 2.10.19.82
>
> The problem I'm seeing is that /proc/interrupts shows iavf interrupts on
> other CPUs. For example, here are some on CPU 4 where I would not expect to
> see any interrupts given that "cat /proc/irq/<NUM>/smp_affinity_list" reports
> "0-1,24-25".
>
> cat /proc/interrupts | grep -e CPU -e 941: -e 942: -e 943: -e 944: -e
> 945: -e 961: -e 962: -e 963: -e 964: -e 965:
>
> CPU0 CPU1 CPU2 CPU3 CPU4 CPU5
> 941: 0 0 0 0 28490 0
> IR-PCI-MSI-edge iavf-0000:b5:03.6:mbx
> 942: 0 0 0 0 333832
> 0 IR-PCI-MSI-edge iavf-net1-TxRx-0
> 943: 0 0 0 0 300842
> 0 IR-PCI-MSI-edge iavf-net1-TxRx-1
> 944: 0 0 0 0 333845
> 0 IR-PCI-MSI-edge iavf-net1-TxRx-2
> 945: 0 0 0 0 333822
> 0 IR-PCI-MSI-edge iavf-net1-TxRx-3
> 961: 0 0 0 0 28492
> 0 IR-PCI-MSI-edge iavf-0000:b5:02.7:mbx
> 962: 0 0 0 0 435608
> 0 IR-PCI-MSI-edge iavf-net1-TxRx-0
> 963: 0 0 0 0 394832
> 0 IR-PCI-MSI-edge iavf-net1-TxRx-1
> 964: 0 0 0 0 398414
> 0 IR-PCI-MSI-edge iavf-net1-TxRx-2
> 965: 0 0 0 0 192847
> 0 IR-PCI-MSI-edge iavf-net1-TxRx-3
>
> Is this expected? It seems like the iavf and/or the i40e aren't respecting
> the configured SMP affinity for the interrupt in question.
>
> Thanks,
>
> Chris
>
>
>
>
> _______________________________________________
> E1000-devel mailing list
> E1000-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/e1000-devel
> To learn more about Intel Ethernet, visit
> https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet
>
> _______________________________________________
> E1000-devel mailing list
> E1000-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/e1000-devel
> To learn more about Intel Ethernet, visit
> https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel Ethernet, visit
https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet
