Re: [lustre-discuss] Kernel panic when reading Lustre osc stats on GPU nodes

[Anna Fuchs via lustre-discuss]

Hi,
thanks for the suggestions. The GPUs are used frequently and already 
have heavier system-call-intensive CPU load. However, since the issue 
occurs across different machines, we don't suspect a structural hardware 
defect. It's more likely a bug in the NVIDIA stack, possibly triggered 
by concurrent sys/debug usage.

At times, we even had the query interval down to one second, but the 
issue doesn't reproduce reliably, not over several hours or a few days.

It seems like no one else has encountered this so far.
We'll keep investigating — thanks again!

Best
Anna


On 5/7/25 18:15, Oleg Drokin wrote:
Hello!

"An uncorrectable ECC error detected" does sound like there's some
hardware problem, while it is strange you only get this on GPU nodes
(Extra power load leading to higher chances of memory corruption + more
frequent kernel memory scannong increasing the chance to hit such
curruption?) I'd expect you'd be seeing other crashes on such GPU nodes
.

Can you just generate some other cpu load (that involves system calls)
on those nodes perhaps and see if suddenly crashes go up as well, just
in some other area?

On Wed, 2025-05-07 at 17:23 +0200, Anna Fuchs via lustre-discuss wrote:
  
Dear all,
  
We're facing an issue that is hopefully not directly related to
Lustre itself (we're not using community Lustre), but maybe someone
here has seen something similar or knows someone who has.
  
On our GPU partition with A100-SXM4-80GB GPUs (VBIOS version:
92.00.36.00.02), we’re trying to read IOPS statistics (osc_stats) via
the files under /sys/kernel/debug/lustre/osc/ (we’re running 160
OSTs, Lustre version 2.14.0_ddn184). Our goal is to sample the data
at 5-second intervals, then aggregate and postprocess it into
readable metrics.
  
We have a collectd daemon running, which had been stable for a long
time. After integrating the IOPS metric, however, we occasionally hit
a kernel panic (see crash dump excerpts below). The issue appears to
originate somewhere in the GPU firmware stack, but we're unsure why
this happens and how it's related to reading Lustre metrics.
  
The problem occurs often, but is hard to reproduce and happens at
random. We’re hesitant to run the scripts frequently since a crash
could interrupt critical GPU workloads. That said, limited test runs
over several hours often work fine, especially after a fresh reboot.
The CPU-only nodes run the same scripts without issues all the time.
  
  
Could this be a sign that /sys/kernel/debug is being overwhelmed
somehow? Although that shouldn’t normally cause a kernel panic.
  
We’d appreciate any insights, experiences, or pointers, even indirect
ones.
  
Thanks in advance!
  
Anna
  

  
  
2024-12-17 17:11:28 [2453606.802826] NVRM: Xid (PCI:0000:03:00): 120,
pid='<unknown>', name=<unknown>, GSP task timeout @ pc:0x4bd36c4,
task:
1
2024-12-17 17:11:28 [2453606.802835] NVRM:     Reported by libos
task:0 v2.0 [0] @ ts:1734451888
2024-12-17 17:11:28 [2453606.802837] NVRM:     RISC-V CSR State:
2024-12-17 17:11:28 [2453606.802840] NVRM:
mstatus:0x000000001e000000  mscratch:0x0000000000000000
mie:0x0000000000000880  mip:0x
0000000000000000
2024-12-17 17:11:28 [2453606.802842] NVRM:
mepc:0x0000000004bd36c4  mbadaddr:0x00000100badca700
mcause:0x8000000000000007
2024-12-17 17:11:28 [2453606.802844] NVRM:     RISC-V GPR State:
[...]
2024-12-17 17:11:29 [2453606.803121] NVRM: Xid (PCI:0000:03:00): 140,
pid='<unknown>', name=<unknown>, An uncorrectable ECC error detected
(p
ossible firmware handling failure) DRAM:-1840691462, LTC:0, MMU:0,
PCIE:0
[...]
2024-12-17 17:30:03 [2454721.362906] Kernel panic - not syncing:
Fatal exception
2024-12-17 17:30:03 [2454721.611822] Kernel Offset: 0x5200000 from
0xffffffff81000000 (relocation range: 0xffffffff80000000-
0xffffffffbfffffff)
2024-12-17 17:30:03 [2454721.770927] ---[ end Kernel panic - not
syncing: Fatal exception ]---



_______________________________________________
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
_______________________________________________
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org