JCNTH commented on PR #19418: URL: https://github.com/apache/tvm/pull/19418#issuecomment-5817217416
Related: #20430 adds the same `maxDispatchesPerFlush` knob (same name and default of 32), measured for throughput rather than UI responsiveness. On Llama-3.2-1B decode in WebLLM the periodic submission alone is 1.18× (paired, 12 rounds), because the GPU starts on the first dispatches while the host is still encoding the rest. Two things from that work that may be useful here: 1. The periodic submission should not go through `flushCommands()`, because that restarts the uniform buffer pool. Every launch between two flush points then no longer has a stable pool position, which is what lets a bind-group cache (#20428) and an unchanged-upload check (#20429) hit; with `flushCommands()` uploads went from 13.9k to 50.7k per 256-token generation. #20430 splits out a `submitPendingCommands()` that finishes and submits without restarting the pool, and has `flushCommands()` restart the pool even when nothing is pending. 2. `maxDispatchesPerFlush <= 0` here means "submit after every dispatch"; #20430 spells that as `1` and uses `0` for "off". Happy to align with whichever this PR settles on. If this one lands first I will rebase #20430 onto it and reduce it to the pool-position fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
