Hi Tomeu,
A ping on this one. It has been on the list since 31 July and 2/2
carries Jiaxing's Reviewed-by.
I am pinging now rather than just waiting because the failure it
describes stopped being a code reading yesterday. I hit it on hardware
while testing Jiaxing's v7 RK3576 series on RK3588, on a tree that did
not have this series applied.
The sequence is the one 2/2 predicts: unbind core 0 while cores 1 and 2
stay bound, then bind it back. What happens without these patches, on
an Orange Pi 5 Plus with all three NPU cores:
- the returning core takes the index of a core that is still live.
The driver prints
rocket fdab0000.npu: Rockchip NPU core 2 version: 1179210309
for the device that is physically core 0;
- inference stops being correct. The same MobileNet V1 input that
classified as "military uniform" before the rebind classifies as
"toilet tissue" after it, and the oracle hashes change with it;
- throughput falls from 88.6 to 1.9 inferences per second, and core 0
stops taking interrupts entirely - 0.00 per inference where it had
been taking 41.74 - while core 1 absorbs everything;
- the next unbind then dies. Ten "NPU job timed out" in a row,
followed by
Unable to handle kernel paging request at virtual address
dead000000000122
pc : destroy_workqueue+0x1b8/0x3e0
Call trace:
destroy_workqueue+0x1b8/0x3e0
drm_sched_fini+0x178/0x1a8 [gpu_sched]
rocket_job_fini+0x28/0x60 [rocket]
rocket_core_fini+0x4c/0x78 [rocket]
rocket_remove+0x6c/0x150 [rocket]
... unbind_store
That is LIST_POISON2 being dereferenced. The sysfs write never
returns, the task is in uninterruptible sleep and cannot be killed,
and the machine needs a reboot.
That is the second bullet of 2/2's commit message happening: the bind
reuses the index of a still-live core and overwrites its slot while its
IRQ handler and its DRM scheduler are still active.
With the two patches applied and nothing else changed, the same
sequence - core 2 out and back, core 0 out and back, all three out and
all three back - runs clean. Twelve inference runs across two modules,
one oracle hash for all of them, correct classification every time, and
nothing in dmesg beyond the probe messages.
I am happy to resend with the tag collected if that is easier. There
is also a practical reason to have it in: the three-core RK3588 test
that Jiaxing asked for on his v7 cannot run to completion on a tree
without this, because the core-0 rebind step is what trips it.
Thanks,
Igor
