JCNTH opened a new pull request, #20427: URL: https://github.com/apache/tvm/pull/20427
The runtime batches dispatches into one command encoder (#18871) but still begins and ends a compute pass for every kernel launch. An LLM decode step is a few hundred launches. This keeps the compute pass open on the pending encoder and reuses it for consecutive dispatches. The pass is ended before a GPU-to-GPU copy is recorded on the encoder and before the encoder is finished; every flush point already goes through `flushCommands()`. Order and data dependencies are unchanged: WebGPU treats each dispatch inside a compute pass as its own usage scope, so a buffer written by one dispatch can be read by the next one in the same pass, exactly as with separate passes. Tests: the mock device in `test_webgpu.js` now enforces the encoder lock (a copy, a second pass or `finish()` while a pass is open throws); new tests cover pass sharing across dispatches, every flush point ending the pass before the encoder is finished, and a launch that throws with the pass open. `npm run lint`, `tsc --noEmit` clean; jest `tests/node/test_webgpu.js` 22 passed. First of four stacked PRs on the per-launch overhead of the WebGPU runtime (this → bind group cache → unchanged-upload skip → periodic submission). Compatible with #19418 and #18961. ## Measurement (shared by the four runtime PRs) Real WebLLM (0.2.85, built from source) against tvmjs from each commit; Llama-3.2-1B-Instruct-q4f16_1, Apple M4 Max, Chrome Canary; same model lib, weights and prompt, temperature 0; one engine per arm, arms interleaved within each round with rotating order; 12 rounds × 256 decode tokens; "paired" is the median per-round ratio to `main`. Generated text identical in all arms. Logits through the WebLLM API are bit-exact against `main` for the whole stack. | tvmjs | passes / token | bind groups / token | uploads / token | decode tok/s | paired | |---|---|---|---|---|---| | main | 184 | 184 | 199 | 210.0 | 1.000 | | + shared compute pass | 5 | 184 | 199 | 218.1 | 1.026 | | + bind group cache | 5 | 4 | 199 | 222.5 | 1.062 | | + skip unchanged uploads | 5 | 4 | 54 | 237.2 | 1.119 | | + periodic submission (separate run, vs its own main at 206.8) | | | | 257.1 | 1.277 | Repeat runs of the first three: 1.118×, 1.137×, 1.136×; 1.04–1.08× when decoding after a 2750-token prompt; prefill neutral (0.97–1.04×). The saving is a fixed cost per launch, so larger models see less: Llama-3.1-8B 1.07× (54.5 → 58.9 tok/s) for the first three. One adapter tested. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
