Hi, Breno,
在 2025/7/23 00:56, Breno Leitao 写道:
Introduce a generic infrastructure for tracking recoverable hardware
errors (HW errors that did not cause a panic) and record them for vmcore
consumption. This aids post-mortem crash analysis tools by preserving
a count and timestamp for the last occurrence of such errors.
Add centralized logging for three common sources of recoverable hardware
errors:
The term "recoverable" is highly ambiguous. Even within the x86
architecture, different vendors define errors differently. I'm not
trying to be pedantic about classification. As far as I know, for 2-bit
memory errors detected by scrub, AMD defines them as deferred errors
(DE) and handles them with log_error_deferred, while Intel uses
machine_check_poll. For 2-bit memory errors consumed by processes, both
Intel and AMD use MCE handling viado_machine_check(). Does your
HWERR_RECOV_MCE only focus on synchronous UE errors handled in
do_machine_check? What makes it special?
- PCIe AER Correctable errors
- x86 Machine Check Exceptions (MCE)
- APEI/CPER GHES corrected or recoverable errors
hwerror_data is write-only at kernel runtime, and it is meant to be
read from vmcore using tools like crash/drgn. For example, this is how
it looks like when opening the crashdump from drgn.
>>> prog['hwerror_data']
(struct hwerror_info[3]){
{
.count = (int)844,
.timestamp = (time64_t)1752852018,
},
...
This helps fleet operators quickly triage whether a crash may be
influenced by hardware recoverable errors (which executes a uncommon
code path in the kernel), especially when recoverable errors occurred
shortly before a panic, such as the bug fixed by
commit ee62ce7a1d90 ("page_pool: Track DMA-mapped pages and unmap them
when destroying the pool")
This is not intended to replace full hardware diagnostics but provides
a fast way to correlate hardware events with kernel panics quickly.
Suggested-by: Tony Luck <tony.l...@intel.com>
Signed-off-by: Breno Leitao <lei...@debian.org>
---
Changes in v3:
- Add more information about this feature in the commit message
(Borislav Petkov)
- Renamed the function to hwerr_log_error_type() and use hwerr as
suffix (Borislav Petkov)
- Make the empty function static inline (kernel test robot)
- Link to v2:
https://lore.kernel.org/r/20250721-vmcore_hw_error-v2-1-ab65a6b43...@debian.org
Changes in v2:
- Split the counter by recoverable error (Tony Luck)
- Link to v1:
https://lore.kernel.org/r/20250714-vmcore_hw_error-v1-1-8cf45edb6...@debian.org
---
arch/x86/kernel/cpu/mce/core.c | 3 +++
drivers/acpi/apei/ghes.c | 8 ++++++--
drivers/pci/pcie/aer.c | 2 ++
include/linux/vmcore_info.h | 14 ++++++++++++++
kernel/vmcore_info.c | 18 ++++++++++++++++++
5 files changed, 43 insertions(+), 2 deletions(-)
diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/core.c
index 4da4eab56c81d..cb225a42eebbb 100644
--- a/arch/x86/kernel/cpu/mce/core.c
+++ b/arch/x86/kernel/cpu/mce/core.c
@@ -45,6 +45,7 @@
#include <linux/task_work.h>
#include <linux/hardirq.h>
#include <linux/kexec.h>
+#include <linux/vmcore_info.h>
#include <asm/fred.h>
#include <asm/cpu_device_id.h>
@@ -1692,6 +1693,8 @@ noinstr void do_machine_check(struct pt_regs *regs)
out:
instrumentation_end();
+ /* Given it didn't panic, mark it as recoverable */
+ hwerr_log_error_type(HWERR_RECOV_MCE);
clear:
mce_wrmsrq(MSR_IA32_MCG_STATUS, 0);
}
diff --git a/drivers/acpi/apei/ghes.c b/drivers/acpi/apei/ghes.c
index a0d54993edb3b..ebda2aa3d68f2 100644
--- a/drivers/acpi/apei/ghes.c
+++ b/drivers/acpi/apei/ghes.c
@@ -43,6 +43,7 @@
#include <linux/uuid.h>
#include <linux/ras.h>
#include <linux/task_work.h>
+#include <linux/vmcore_info.h>
#include <acpi/actbl1.h>
#include <acpi/ghes.h>
@@ -1136,13 +1137,16 @@ static int ghes_proc(struct ghes *ghes)
{
struct acpi_hest_generic_status *estatus = ghes->estatus;
u64 buf_paddr;
- int rc;
+ int rc, sev;
rc = ghes_read_estatus(ghes, estatus, &buf_paddr, FIX_APEI_GHES_IRQ);
if (rc)
goto out;
- if (ghes_severity(estatus->error_severity) >= GHES_SEV_PANIC)
+ sev = ghes_severity(estatus->error_severity);
+ if (sev == GHES_SEV_RECOVERABLE || sev == GHES_SEV_CORRECTED)
+ hwerr_log_error_type(HWERR_RECOV_GHES);
APEI does not define an error type named GHES. GHES is just a kernel
driver name. Many hardware error types can be handled in GHES (see
ghes_do_proc), for example, AER is routed by GHES when firmware-first
mode is used. As far as I know, firmware-first mode is commonly used in
production. Should GHES errors be categorized into AER, memory, and CXL
memory instead?
Thanks.
Shuai
