From: Naman Jain <[email protected]> Sent: Thursday, October 29, 2026 9:01 PM > > On 10/30/2026 8:09 AM, Michael Kelley wrote: > > From: Naman Jain <[email protected]> Sent: Wednesday, October 28, > > 2026 11:43 PM > >> > >> CPUs no longer running an mm may remain in mm_cpumask(). Hyper-V treats > >> these CPUs as flush targets and bypasses the native callbacks that trim > >> the mask. A single-threaded process can therefore keep issuing > >> all-processor flushes after visiting all CPUs and returning to one CPU. > >> > >> Share the native target predicate and filter the mask before building > >> either hypercall format or selecting HV_FLUSH_ALL_PROCESSORS. Preserve > >> lazy-TLB exclusion and unfiltered page-table-freeing flushes, and use > >> native callbacks for periodic trimming and ASID transitions. > >> > >> When no remote targets remain, request immediate trimming through native > >> callbacks so stale memberships do not unnecessarily prevent later > >> invalidations from using the local fast path. Leave lazy CPU-mask > >> maintenance during context switches unchanged. > >> > >> Fixes: 209954cbc7d0 ("x86/mm/tlb: Update mm_cpumask lazily") > >> Signed-off-by: Naman Jain <[email protected]> > > > > I've read the commit message but haven't yet delved into the code for > > this patch. But the commit message feels like it is missing some context, > > particularly with the "Fixes:" tag. > > > > I recall the last significant change to hyperv_flush_tlb_multi(), which > > takes > > CPUs in lazy tlb mode into account, and thereby avoids unnecessary flushing > > (commit 493cc07385cf, made in March 2023). It was a case where the > > native x86 flush code had gotten smarter, and the Hyper-V PV flush > > functions had not been updated to be smarter in the same way. Against > > that context, what is the situation here? Commit 209954cbc7d0 came a > > year and a half later in November 2024. Did it make things worse for the > > Hyper-V PV flush functions, and hence is deemed to have "broken" > > them? Or is it another native flush improvement that the Hyper-V PV > > flush functions just don't take advantage of? > > > > I'm also curious how the issue was noticed. Was there a real-world > > performance problem, or is it a case of looking at the code and > > noticing a native improvement that hadn't been applied to the > > Hyper-V PV flush functions? The scenario of a single-threaded process > > visiting all CPUs and then returning to one CPU seems contrived, > > particularly as CPU counts get large. It may illustrate the improvement > > that is possible, but to me seems unlikely to be how the problem was > > originally noticed. > > > > I intended to review the code carefully, but the additional context > > would be helpful in doing that review. > > > > Michael > > > > Thanks Michael for looking into this. > > We first observed a performance regression between Linux 6.8 and 7.0 > on Azure MI300X VMs running SuperBench's eight-GPU PyTorch LSTM training > benchmark, using distributed data parallelism in FP32 and FP16. > Recreating DataLoader workers each epoch triggered long host-to-GPU > copy stalls and timeouts. We bisected the regression to commit being > fixed and then developed the reduced reproducer to isolate the excessive > Hyper-V TLB flushes. The CPU-visiting test came out of that > investigation, it was not how the problem was originally noticed. > > 209954cbc7d0 ("x86/mm/tlb: Update mm_cpumask lazily") changed where > stale mm_cpumask entries are removed. Previously, switching a CPU from > mm A to mm B cleared its bit in A's mask. After that change, the bit is > left set for removal by the native TLB flush callback when it finds a > different mm loaded. > > The Hyper-V hypercall path bypasses that callback, so historical > memberships can persist. The lazy-TLB check from 493cc07385cf does not > cover this case: a CPU actively running B can have is_lazy == false > while still appearing in A's mask. Hyper-V then unnecessarily targets > it when flushing A. If the historical memberships cover all present > CPUs, the all-processors shortcut is selected before the per-CPU > lazy-TLB checks. > > So the later change did make the Hyper-V path worse for affected > workloads, rather than merely adding a native optimization that > Hyper-V could also benefit from. That is the reason for the Fixes tag. > This is a performance regression, not a stale-translation correctness > failure.
Thanks. That's exactly what I wanted to know as background information. It will help a lot when reviewing your patch. > > The reporter has also tested the patch on a 96-vCPU MI300X VM. The > unpatched SuperBench workload hit the 600-second timeout; two patched > runs completed in 424 and 436 seconds, with all eight ranks validated. > The kernel configurations matched apart from kernel identity, and > the GPU driver source and executable text were unchanged. Good info. That's a pretty significant difference in a macro-level benchmarking tool. Michael > > For the reduced reproducer, the 96-CPU-history case dropped from > 109.196 us per iteration to 1.594 and 1.616 us in the two patched runs, > about a 98.5% reduction. Each iteration contains two mprotect() calls > and a write, these are not individual hypercall timings. > > The patch preserves your lazy-TLB exclusion and lazy CPU-mask > maintenance during context switches. It filters the targets before > selecting the hypercall format or the all-processors shortcut, and > uses native callbacks for mask trimming. > > I restricted the commit msg to the mentioned 32 vCPU VM because that was > the one I got my hands on. I'll try to update the commit message or the > comments to explain the real workload, bisection, and distinction > between lazy TLB mode and lazy CPU-mask maintenance. But I'll wait for > any review comments to address them together. > > Thanks, > Naman > > >> --- > >> On a 32-vCPU Hyper-V guest, after a single-threaded process visited all > >> CPUs and returned to one CPU, the reproducer's median time per iteration > >> (two mprotect() calls and a write) dropped from 17.235 us to 1.622 us, > >> a 90.6% reduction. This could be more on higher vCPU VMs. > >> > <snip> >
