vinx13 opened a new pull request, #17326: URL: https://github.com/apache/tvm/pull/17326
This adds support for tree attention with tokens in the kv cache. KVCache is now stateful w.r.t the tree state, multiple decoding rounds can be performed with tree mask and the accepted tokens can be committed at once at the end. cc @MasterJH5574 @tqchen -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
