CharlieFRuan opened a new pull request, #16554:
URL: https://github.com/apache/tvm/pull/16554
This PR introduces various WebGPU changes to accommodate the new
`PagedKVCache` interface. All changes below are essential for making models
that use PagedKVCache runnable under WebGPU:
- Require exactly same-dtype matching for WebGPU smem reuse in
`storage_rewrite.cc`
- Rename `AttentionKVCache` to `AttentionKVCacheLegacy` for the old KVcache
interface in `lm_support.cc`; include `paged_kv_cache.cc` when making
`wasm_runtime` subsequently
- In WebGPU codegen:
- Declare local variables within the function scope rather than the module
scope
- Generate `while (true)` rather than `while (1)`
- Require 10 `maxStorageBuffersPerShaderStage` rather than the default 8
from the WebGPU device when initializing runtime; this is required for new
kernels introduced in PagedKVCache
- In `deviceCopyToCPU()`, when raw bytes to write are not multiples of 4, we
pad them, as required by WebGPU's `writeBuffer()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
