On 06.11.2023 15:27, Roger Pau Monne wrote: > The current implementation of x2APIC requires to either use Cluster Logical or > Physical mode for all interrupts. However the selection of Physical vs > Logical > is not done at APIC setup, an APIC can be addressed both in Physical or > Logical > destination modes concurrently. > > Introduce a new x2APIC mode called Mixed, which uses Logical Cluster mode for > IPIs, and Physical mode for external interrupts, thus attempting to use the > best method for each interrupt type. > > Using Physical mode for external interrupts allows more vectors to be used, > and > interrupt balancing to be more accurate. > > Using Logical Cluster mode for IPIs allows fewer accesses to the ICR register > when sending those, as multiple CPUs can be targeted with a single ICR > register > write. > > A simple test calling flush_tlb_all() 10000 times on a tight loop on AMD EPYC > 9754 with 512 CPUs gives the following figures in nano seconds: > > x mixed > + phys > * cluster > N Min Max Median Avg Stddev > x 25 3.5131328e+08 3.5716441e+08 3.5410987e+08 3.5432659e+08 1566737.4 > + 12 1.231082e+09 1.238824e+09 1.2370528e+09 1.2357981e+09 2853892.9 > Difference at 95.0% confidence > 8.81472e+08 +/- 1.46849e+06 > 248.774% +/- 0.96566% > (Student's t, pooled s = 2.05985e+06) > * 11 3.5099276e+08 3.5561459e+08 3.5461234e+08 3.5415668e+08 1415071.9 > No difference proven at 95.0% confidence > > So Mixed has no difference when compared to Cluster mode, and Physical mode is > 248% slower when compared to either Mixed or Cluster modes with a 95% > confidence. > > Note that Xen uses Cluster mode by default, and hence is already using the > fastest way for IPI delivery at the cost of reducing the amount of vectors > available system-wide. > > Make the newly introduced mode the default one. > > Note the printing of the APIC addressing mode done in connect_bsp_APIC() has > been removed, as with the newly introduced mixed mode this would require more > fine grained printing, or else would be incorrect. The addressing mode can > already be derived from the APIC driver in use, which is printed by different > helpers. > > Suggested-by: Andrew Cooper <[email protected]> > Signed-off-by: Roger Pau Monné <[email protected]>
Reviewed-by: Jan Beulich <[email protected]>
