Hi,
On a Mali-G52 (Rockchip RK3566), a run of job timeouts and soft-stop
failures ends with the kernel taking paging requests at corrupt
addresses, and an oops on the next fork(). The machine does not panic:
it keeps running and keeps routing packets, but every process that
clones a VMA dies, so it is unusable and needs a power cycle.
I have ruled out the obvious candidate before writing -- details below
-- so I am fairly sure this is not the 2019 per-slot reset race.
Happy to be told otherwise.
The sequence
============
Captured over netconsole, because everything on the box stops before it
can write to disk:
panfrost fde60000.gpu: Soft-stop failed
panfrost fde60000.gpu: Soft-stop failed
panfrost fde60000.gpu: gpu sched timeout, js=1, config=0x7300,
status=0x8, head=0xd73cf480, tail=0xd73cecc0,
sched_job=00000000317fe955
panfrost fde60000.gpu: Unhandled Page fault in AS0 at VA 0x000000001E0C2900
raw fault status: 0x664003C2
decoded fault status: SLAVE FAULT
exception type 0xC2: TRANSLATION_FAULT_2
access type 0x3: WRITE
source id 0x6640
panfrost fde60000.gpu: js fault, js=0, status=DATA_INVALID_FAULT,
head=0xd496dfc0, tail=0xd496dfc0
Unable to handle kernel paging request at virtual address 3f87ae4e3f41d124
Unable to handle kernel paging request at virtual address ffff3a2c0129b997
Internal error: Oops: 0000000096000004 [#1] SMP
EC = 0x25: DABT (current EL), FSC = 0x04: level 0 translation fault
CPU: 0 PID: 1 Comm: systemd Tainted: G W O
6.1.84-10-rk2410-nocsf #10
Hardware name: Radxa ZERO 3 (DT)
pc : anon_vma_interval_tree_insert+0x90/0xb4
Call trace:
anon_vma_interval_tree_insert+0x90/0xb4
anon_vma_clone+0x110/0x144
anon_vma_fork+0x30/0x134
dup_mmap+0x2c4/0x4bc
copy_process+0xa84/0x1314
kernel_clone+0xb0/0x288
__do_sys_clone+0x68/0x9c
__arm64_sys_clone+0x20/0x28
Four separate paging requests at unrelated garbage addresses precede
the oops, which is why this reads as corruption rather than one bad
dereference.
Worth noting the GPU's own faulting access is a write the MMU blocked
(TRANSLATION_FAULT_2, access type 0x3: WRITE), so it should not itself
have reached kernel memory. That is what makes me look at the recovery
path rather than at the GPU.
Why it is expensive: networking never clones a VMA, so the box still
answers pings, while sshd -- which forks per connection -- dies at
kex_exchange_identification, before authentication. No credential can
get in, and journald is gone for the same reason, so nothing about this
survives on the machine.
What I ruled out
================
The per-slot reset race (two timeout threads racing in the reset path,
the T820 problem from 2019) does not apply here. The serialization is
present in its modern form: panfrost_job_init() does
pfdev->reset.wq = alloc_ordered_workqueue("panfrost-reset", 0);
and passes that workqueue to drm_sched_init() for every slot, so an
ordered workqueue means panfrost_job_timedout() cannot run
concurrently for two slots. I had reached for that explanation first,
because the log shows js=0 and js=1 faulting in the same window.
This is also not a vendor divergence. The board runs a Rockchip BSP, so
the next question was whether it had dropped something. It has not:
against upstream linux-6.1.y, this kernel's panfrost_job.c differs by
nine lines, all DRM scheduler API signature adaptations
(drm_sched_start, drm_sched_init, hw_rq_count -> credit_count), with no
logic changes. So the above happens on a stock upstream 6.1 panfrost
path with the serialized reset in place.
A question about HW_ISSUE_8408
==============================
"Repeatedly Soft-stopping a job chain consisting of (Vertex Shader,
Cache Flush, Tiler) jobs causes DATA_INVALID_FAULT"
That is close to word-for-word what the log shows: repeated soft-stop
failures on js=1 (vertex/tiler), DATA_INVALID_FAULT on js=0. But it is
declared only in hw_issues_t600, not for G52. Is that because G52 does
not have it, or because nobody has had cause to flag it there?
Against that, HW_ISSUE_9435 is in hw_issues_all -- the compute
endpoint's 4-deep task queue means a soft stop cannot complete until
all four tasks do -- so "Soft-stop failed" on its own may be
unremarkable, and I would rather say so than lean on it.
Recoverable vs fatal
====================
Two cascades on one boot, with identical DATA_INVALID_FAULT counts:
recoverable fatal
span 3 s 14 s
gpu sched timeout 6 10
DATA_INVALID_FAULT 3 3
Soft-stop failed 0 2
TRANSLATION_FAULT_2 0 1
outcome GPU recovers, oops
display restores
at +4.4 s, ~70 s
more rendering on
the same boot
So escalation tracks the driver failing to stop the hung job rather
than the number of faulting jobs. Given HW_ISSUE_9435 that may be
correlation rather than cause.
What I do not have
==================
- A mainline reproduction. It is a vendor kernel, though as above its
panfrost is upstream's to nine API-signature lines, so "vendor
kernel" is not by itself the explanation. If reproducing on mainline
is the precondition for looking at this, say so and I will see what
I can arrange.
- A minimal reproducer. The trigger is a full application rendering a
large scene.
- A devcoredump: the driver logs "failed to allocate devcoredump file"
at exactly the moment one would be most useful.
panfrost_core_dump(job) is called from panfrost_job_timedout()
immediately before the reset, so that may deserve its own look.
- Determinism. Five runs in an identical environment gave one cascade
and four clean runs.
Environment
===========
Mali-G52 r1 MC1 on Rockchip RK3566
mali-g52 id 0x7402 major 0x1 minor 0x0
[drm] Initialized panfrost 1.2.0 20180908 for fde60000.gpu on minor 1
features: 00000000,00000cf7 issues: 00000000,00000400
kernel 6.1.84-10-rk2410-nocsf #10 (Rockchip BSP), aarch64
taint G W O -- out-of-tree panel_mipi_dbi, gpio_backlight
Userspace is panvk (Mesa 26.3.0-devel), which needs
PAN_I_WANT_A_BROKEN_VULKAN_DRIVER=1. A separate userspace problem on
the same hardware is filed as mesa/mesa#16595; this report is only
about the kernel turning a GPU fault into memory corruption, which no
userspace submission should be able to do.
I have the full 301-line netconsole capture -- the above trimmed from
it -- and can send it, or anything else that would help. If you want
instrumentation run on this board, I can do that; each fatal
occurrence costs a physical power cycle, so a targeted request beats a
broad one.
One note for anyone reproducing: raise /proc/sys/kernel/printk to 7
first. At the default console loglevel of 4 none of these panfrost
messages reach any console, netconsole included, and a run that
faulted looks clean. Four earlier occurrences here produced no log at
all for that reason.
Thanks,
Joel
