JCNTH opened a new pull request, #20430: URL: https://github.com/apache/tvm/pull/20430
**Stacked on the unchanged-upload PR (#20429); only the last commit is this PR. Overlaps #19418, see below.** Since #18871 commands are submitted only at flush points (sync, readback, host write). For a run of launches such as one LLM decode step the host therefore encodes everything first and the GPU starts afterwards, so encode time and GPU time add up instead of overlapping. On the GPU clock a Llama-3.2-1B decode step showed 2.0 ms of idle time between kernels per 5.4 ms. Submit the pending commands once `maxDispatchesPerFlush` (default 32) launches have been recorded since the last submission; 0 turns periodic submission off. The field name and default follow #19418, which proposes the same knob for UI responsiveness. Two differences from #19418: 1. A periodic submission must not restart the uniform pool. Every launch between two flush points keeps its own pool buffer at a stable position, which the bind group cache and the unchanged-upload check rely on; calling `flushCommands()` there took uploads from 13.9k to 50.7k per 256-token generation. So this adds `submitPendingCommands()` (finish + submit, keep the pool position) and has `flushCommands()` restart the pool even when nothing is pending, since a periodic submission may already have sent everything and any later `writeBuffer` is ordered after it. 2. `maxDispatchesPerFlush <= 0` there means "flush after every dispatch"; here `0` turns periodic submission off (the conventional meaning of 0 for an interval) and "after every dispatch" is spelled `1`. Happy to align either way. Interval sweep on top of the three PRs below (1B, paired): never 1.000, 8: 1.101, 16: 1.107, 32: 1.110, 64: 1.091; uploads unchanged at every interval; text identical. Alone on `main` the mechanism gives 1.18×; the whole stack 1.28× (206.8 → 257.1 tok/s). One invariant found by an independent review and now pinned by a test: `sync()` must still await the queue after a periodic submission that followed a readback, because the readback is no longer the queue tail (`pendingGPUToCPUCopyIsQueueTail` is cleared by the submission). Tests: submission every N launches and a flush point restarting the count, `0` disables and the field is read at every launch, pool positions survive periodic submissions so caches still hit, the pool restarts at a flush point even with nothing pending, sync after readback + periodic submission, destroy-after-submit ordering. lint/tsc clean; jest 41 passed. ## Measurement (shared by the four runtime PRs) Real WebLLM (0.2.85, built from source) against tvmjs from each commit; Llama-3.2-1B-Instruct-q4f16_1, Apple M4 Max, Chrome Canary; same model lib, weights and prompt, temperature 0; one engine per arm, arms interleaved within each round with rotating order; 12 rounds × 256 decode tokens; "paired" is the median per-round ratio to `main`. Generated text identical in all arms. Logits through the WebLLM API are bit-exact against `main` for the whole stack. | tvmjs | passes / token | bind groups / token | uploads / token | decode tok/s | paired | |---|---|---|---|---|---| | main | 184 | 184 | 199 | 210.0 | 1.000 | | + shared compute pass | 5 | 184 | 199 | 218.1 | 1.026 | | + bind group cache | 5 | 4 | 199 | 222.5 | 1.062 | | + skip unchanged uploads | 5 | 4 | 54 | 237.2 | 1.119 | | + periodic submission (separate run, vs its own main at 206.8) | | | | 257.1 | 1.277 | Repeat runs of the first three: 1.118×, 1.137×, 1.136×; 1.04–1.08× when decoding after a 2750-token prompt; prefill neutral (0.97–1.04×). The saving is a fixed cost per launch, so larger models see less: Llama-3.1-8B 1.07× (54.5 → 58.9 tok/s) for the first three. One adapter tested. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
