JCNTH opened a new pull request, #20429: URL: https://github.com/apache/tvm/pull/20429
**Stacked on the bind-group cache PR (#20428); only the last commit is this PR.** Most kernels in a decode step receive the same POD arguments every token, and they land in the same pool buffer because a pool buffer is tied to the launch's position in the batch. Remember the words last written to each pool buffer and skip `queue.writeBuffer` when the buffer already starts with the launch's words. Pool buffers are written only by `submitShader` and keep their contents across flushes. Comparison is on int32 bit patterns (exact for floats: `-0.0 != 0.0`, NaN equals itself). The pool keeps its own copy of the words, taken before the write and committed after it, so this composes with #18961 (one reused array per shader) and a throwing write leaves the contents unknown. Uploads per token 199 → 54 on Llama-3.2-1B; the remainder are the arguments that really change (sequence positions, KV lengths). Tests: unchanged vs changed arguments per batch position, prefix reuse between kernels with different POD counts incl. a replaced pool buffer, float bit patterns, and the pool-owned record surviving a failed write or a caller that overwrites its array. lint/tsc clean; jest 34 passed. ## Measurement (shared by the four runtime PRs) Real WebLLM (0.2.85, built from source) against tvmjs from each commit; Llama-3.2-1B-Instruct-q4f16_1, Apple M4 Max, Chrome Canary; same model lib, weights and prompt, temperature 0; one engine per arm, arms interleaved within each round with rotating order; 12 rounds × 256 decode tokens; "paired" is the median per-round ratio to `main`. Generated text identical in all arms. Logits through the WebLLM API are bit-exact against `main` for the whole stack. | tvmjs | passes / token | bind groups / token | uploads / token | decode tok/s | paired | |---|---|---|---|---|---| | main | 184 | 184 | 199 | 210.0 | 1.000 | | + shared compute pass | 5 | 184 | 199 | 218.1 | 1.026 | | + bind group cache | 5 | 4 | 199 | 222.5 | 1.062 | | + skip unchanged uploads | 5 | 4 | 54 | 237.2 | 1.119 | | + periodic submission (separate run, vs its own main at 206.8) | | | | 257.1 | 1.277 | Repeat runs of the first three: 1.118×, 1.137×, 1.136×; 1.04–1.08× when decoding after a 2750-token prompt; prefill neutral (0.97–1.04×). The saving is a fixed cost per launch, so larger models see less: Llama-3.1-8B 1.07× (54.5 → 58.9 tok/s) for the first three. One adapter tested. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
