Hi Jiaxing, On Sat, Sep 19, 2026 at 09:17:25PM +1200, Jiaxing Hu wrote: > I read the tally as 8 of the 126 scored inferences missing on all 48 > channels across the three runs, and 5 of those, in the two runs you > traced, sitting on the 5 -ECANCELED completions.
Nested that way, yes. One correction: six runs, three per arm. Five of the eight zeros are on 2+3 and three on 2+3+4. In the two traced runs the -ECANCELED completions sit on exactly the inferences that scored 0/48 there: rounds 1, 4, 8 and the one after suspend/resume on 2+3, round 7 on 2+3+4. The other three zeros are in runs without the kprobe. After that mail, on the evening of 16 September, I reached the other path. A 1x1 convolution gets there when the input, not the weights, overflows the CBUF: 80x80x64 in, 48 out, so rkt_split_tasks() splits on input rows and rkt_ml_subgraph_invoke() submits one job with task_count 2 (rows 0-63 and 64-79). A graph whose weights do not fit goes the other way: reuse_weights_cbuf is false and each task becomes its own one-task job, back on the scheduler thread. Same board and config (JOB_TIMEOUT_MS=2, PROVE_LOCKING, DEBUG_ATOMIC_SLEEP), the induced-reset protocol at 60 inferences a run. On 2+3: five runs, 310 inferences, 180 timeouts, one of the runs with a 32-output variant that splits the same way. On 2+3+4: three runs, 186 inferences, 181 timeouts. In the three traced runs on 2+3, 24 of 110 timeouts cut into a job after the IRQ thread had submitted its next task (9, 9 and 6); in the one traced run on 2+3+4, 11 of 61. No lockdep report, warning or MMU fault in either boot, and debug_locks 1 at the end of every run. So the lock scope ran with the IRQ thread submitting tasks, but this does not show the race is closed: the window in hw_submit() is microseconds, one reset began while the IRQ thread was in it (in the 32-output run), and waiting on job_lock itself was not instrumented. One result bears on 4/14. Without it the core stayed active after every cancelled job (113 of 113) and no cancelled job was followed by another (0 of 109). With it the core suspended after each (122 of 122) and 84 of 120 cancelled jobs were followed by another, the next job waiting for resume against the same 2 ms. That is the timeout feeding itself, not a defect, and it is why 181 of 186 inferences timed out on that arm against 180 of 310. Core 0, one client. If 4/14 changes shape in v14 I will re-run that arm, with the two-task job in it. Scripts, counting and this draft were prepared with an LLM assistant; I ran the tests, and every number came from the raw files of each run. Regards, Igor
