I've been pursuing a bug in a virtual machine (KVM) that I would like to share
in here. The VM gets stuck when running perf in a VM and getting soft lockups.
The bug happens upstream (Linux 6.6-rc4 - 8a749fd1a8720d461). The same kernel
is being used in the host and in the guest.
The problem only happens in a very specific circumstances:
1) PMU needs to be enabled in the guest
2) Libvirt/QEMU needs to use a custom CPU:
* Here is the qemu line:
-cpu Skylake-Server,kvm-pv-eoi=on,pmu=on
* Any other CPU seems to hit the problem
* Even using Skylake-Server on a Skylake server
* Using CPU passthrough workaround the problem
3) You need to use 6 or more events in perf.
* This is a line that reproduces the problem:
# perf stat -e cpu-clock -e context-switches -e cpu-migrations -e
page-faults -e cycles -e instructions -e branches ls
* Removing any of these events (totaling 5 events) makes `perf` work
again
4) This problem happens on upstream, 6.4 and 5.19
* This problem doesn't seem to happen on 5.12
Problem
========
When running perf in the circumstances above, the VM is stuck, with a lot of
stack traces. This is some messages:
kernel:[ 400.314381] watchdog: BUG: soft lockup - CPU#3 stuck for
26s! [kworker/u68:11:6853]
kernel:[ 400.324380] watchdog: BUG: soft lockup - CPU#8 stuck for
26s! [dynoKernelMon:9781]
kernel:[ 404.368380] watchdog: BUG: soft lockup - CPU#30 stuck for
22s! [kworker/30:2:1326]
Here is part of the stack. The full stack is in the pastebin below:
nmi_cpu_backtrace (lib/nmi_backtrace.c:115)
nmi_cpu_backtrace_handler (arch/x86/kernel/apic/hw_nmi.c:47)
nmi_handle (arch/x86/kernel/nmi.c:149)
__intel_pmu_enable_all (arch/x86/include/asm/jump_label.h:27
include/linux/jump_label.h:207 arch/x86/include/asm/msr.h:147
arch/x86/include/asm/msr.h:262 arch/x86/events/intel/core.c:2239)
__intel_pmu_enable_all (arch/x86/include/asm/jump_label.h:27
include/linux/jump_label.h:207 arch/x86/include/asm/msr.h:147
arch/x86/include/asm/msr.h:262 arch/x86/events/intel/core.c:2239)
default_do_nmi (arch/x86/kernel/nmi.c:347)
exc_nmi (arch/x86/kernel/nmi.c:543)
end_repeat_nmi (arch/x86/entry/entry_64.S:1471)
__intel_pmu_enable_all (arch/x86/include/asm/jump_label.h:27
include/linux/jump_label.h:207 arch/x86/include/asm/msr.h:147
arch/x86/include/asm/msr.h:262 arch/x86/events/intel/core.c:2239)
__intel_pmu_enable_all (arch/x86/include/asm/jump_label.h:27
include/linux/jump_label.h:207 arch/x86/include/asm/msr.h:147
arch/x86/include/asm/msr.h:262 arch/x86/events/intel/core.c:2239)
__intel_pmu_enable_all (arch/x86/include/asm/jump_label.h:27
include/linux/jump_label.h:207 arch/x86/include/asm/msr.h:147
arch/x86/include/asm/msr.h:262 arch/x86/events/intel/core.c:2239)
More info
=========
Soft lockup messages in the guest:
https://paste.debian.net/1293888/
Full log from the guest:
https://paste.debian.net/1293891/
vCPU stacks dumped from the host (cat /proc/<vcpu>/stack):
https://paste.debian.net/1293887/
Qemu (version 7.1.0) command line
https://paste.debian.net/1293894/
