In modern fleet environments, capturing a full multi-gigabyte vmcore via
kdump can occasionally fail or be intentionally skipped due to storage,
network, or encryption constraints. In these scenarios, the extracted
dmesg buffer (e.g. vmcore-dmesg.txt) is often the sole artefact
available for post-mortem triage.
Currently, appending supplemental system state (such as memory usage, held
locks, active timers, task states, or ftrace buffers) via panic_print prior
to kdump requires enabling crash_kexec_post_notifiers. Doing so exposes the
crash path to registered panic notifiers (vastly increasing the risk of
hanging or deadlocking) and forces panic_print output to be flushed
synchronously to physical consoles. On a slow serial console or IPMI
Serial-over-LAN (e.g. 115200 baud), flushing comprehensive sys_info()
output (e.g. tasks or stacktraces) can exceed 10 seconds, allowing the
hardware timer to expire and reset the system before kdump can initialise.
Furthermore, on large systems, high-volume telemetry can wrap the printk
ring buffer and overwrite the initial Oops, BUG, or panic stack trace.
This series introduces the "crash_kexec_in_memory_sys_info" kernel
command-line parameter to decouple in-memory telemetry collection from
both panic notifiers and physical console flushing when kdump is active:
1. Fixes duplicate and dropped all-CPU backtraces when SYS_INFO_ALL_BT
is configured in panic_print across early SMP shutdown and
__sys_info().
2. Avoids spinning indefinitely on zone->lock in show_free_areas()
during NMI, Oops, or panic contexts by falling back to non-blocking
trylock acquisitions and skipping contended zones.
3. Avoids blocking on timekeeping seqcounts and hrtimer_cpu_base locks
in sysrq_timer_list_show() and print_active_timers() during NMI,
Oops, or panic contexts.
4. Adds printk console flush suppression across nbcon, legacy
consoles, and klogd, alongside ring buffer tail pinning so
pre-crash records can still be recycled while protecting the
initial crash output from being overwritten by subsequent telemetry
or ftrace dumps.
5. Introduces crash_kexec_in_memory_sys_info and invokes
panic_in_memory_sys_info() from __crash_kexec() prior to
machine_crash_shutdown() when crash_kexec_post_notifiers is not
enabled. It also marks the start of crash diagnostics across
bust_spinlocks(), __report_bug(), oops_enter(), and vpanic(), and
reorders __sys_info() so compact, high-signal dumps (SYS_INFO_MEM,
SYS_INFO_LOCKS, SYS_INFO_TIMERS, SYS_INFO_BLOCKED_TASKS) execute
before high-volume dumps (SYS_INFO_ALL_BT, SYS_INFO_TASKS,
SYS_INFO_FTRACE).
Changes since v1:
- Split the single patch into a 5-patch series separating prerequisite
panic all-CPU backtrace fixes, crash-time lock safety in show_mem()
and timer_list, printk/tracing infrastructure, and the crash_kexec
integration
- Moved in-memory sys_info() collection from vpanic() into
__crash_kexec() via panic_in_memory_sys_info() with local interrupts
disabled so all crash_kexec() entry points (including direct Oops/die
paths prior to vpanic()) are covered, and kept console flush
suppression and tail pinning active through machine_crash_shutdown()
and machine_kexec() unless machine_kexec() returns
- Replaced unconditional ring buffer tail freezing with crash-start
sequence pinning (printk_mark_panic_start(),
printk_clear_panic_start(), and printk_is_tail_full()), allowing
pre-crash records to be recycled while preserving the initial
Oops/panic trace, finalising the last record when full, and stopping
ftrace_dump_one() before consuming unprinted trace entries
- Extended printk_suppress_console_flush() to cover nbcon atomic
flushes and kthreads, legacy console_flush_one_record() and kthreads,
and wake_up_klogd(), and dropped unnecessary EXPORT_SYMBOL_GPL
exports
- Added crash-context trylock and NMI-safe timekeeping handling in
show_free_areas(), sysrq_timer_list_show(), and
print_active_timers(), and fixed duplicate/dropped all-CPU backtraces
when SYS_INFO_ALL_BT is used during panic
- Link to v1:
https://lore.kernel.org/lkml/[email protected]/
Aaron Tomlin (5):
panic, sys_info: Fix duplicate and dropped all-CPU backtraces in panic
mm/show_mem: Avoid blocking on zone->lock during Oops or panic
timers: Avoid blocking on hrtimer_cpu_base lock during Oops or panic
printk, tracing: Support console flush suppression and tail pinning
panic, crash_core, sys_info: Introduce crash_kexec_in_memory_sys_info
.../admin-guide/kernel-parameters.txt | 11 ++
include/linux/panic.h | 2 +
include/linux/printk.h | 27 +++
include/linux/sched.h | 3 +
include/linux/sys_info.h | 4 +
kernel/crash_core.c | 6 +
kernel/exit.c | 10 ++
kernel/fork.c | 3 +
kernel/panic.c | 87 ++++++++-
kernel/printk/internal.h | 4 +
kernel/printk/nbcon.c | 11 +-
kernel/printk/printk.c | 29 ++-
kernel/printk/printk_ringbuffer.c | 168 +++++++++++++++++-
kernel/time/timer_list.c | 13 +-
kernel/trace/trace.c | 20 ++-
lib/bug.c | 18 +-
lib/bust_spinlocks.c | 22 ++-
lib/sys_info.c | 33 ++--
mm/show_mem.c | 15 +-
19 files changed, 447 insertions(+), 39 deletions(-)
--
2.55.0
