MasterJH5574 opened a new pull request, #19904: URL: https://github.com/apache/tvm/pull/19904
FlashInfer 0.6.3 changes the paged-attention plan/run ABI: the prefill/decode plans take new arguments (e.g. window_left, fixed_split_size, disable_split_kv; the decode plan now dispatches dtype through empty q/kv tensors), the runs add enable_pdl and drop the explicit stream, and the kernels consume separate key/value paged caches read through tensor strides rather than one combined tensor. This updates the runtime attention backend (paged MHA, ragged, decode and MLA) to the new signatures and to the Array<int64_t> plan-info representation. FlashInfer 0.6.3 reads tensors from `data` directly and does not honor the DLPack `byte_offset` field. mlc's auxiliary index tensors (qo_indptr, kv_indptr, page_indptr, page_indices, length_info) are views packed into a shared workspace and so carry a non-zero byte_offset; passed as-is the kernels read the wrong addresses (e.g. a ragged prefill processed only the first query row). Three zero-copy DLPack view helpers address this: `ZeroByteOffsetView` folds byte_offset into the data pointer, `PagedKVCacheView` exposes the combined (num_pages, 2, ...) page tensor as separate strided key/value caches, and `SliceLastDimView` slices the last dimension for MLA. This also completes the MLA FlashInfer path, which previously shipped only the test and module generator. The MLA run splits the query into nope/pe parts and the paged cache into ckv/kpe parts, and the ragged self-attention is given its own uncompressed head dims and per-query kv head count via a 5-element backend spec, since they differ from the compressed MLA cache. The MHA and MLA FlashInfer KV-cache tests are re-enabled as regression coverage, guarded on FlashInfer availability (inline-RoPE is skipped as unsupported by FlashInfer). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
