Lunderberg commented on PR #16313: URL: https://github.com/apache/tvm/pull/16313#issuecomment-1883408980
Whether or not there's an advantage depends on the specific shapes of the arguments. The benefit occurs if the sizes of `x*A` and `x*B` are small compared to the size of `A*B`. In principle, this condition can't be proven in the general case, because it depends on a dynamic `batch_size` in `x`. In practice, the batch size tends to be small compared to the dimensions of the weight matrices, so it is generally beneficial. This transform is not intended to be enabled in a default pass, but as a tool to be used in building model-specific optimization pipelines. Specifically, for batched LoRA evaluation, a computation similar to [S-LoRA](https://arxiv.org/abs/2311.03285) can be implemented with the following sequence: 1. Initial state: `x*(W + A*B)` 2. `ExpandMatmulOfSum`: `x*W + x*(A*B)` 3. `AdjustMatmulOrder`: `x*W + (x*A)*B` 4. `CombineParallelMatmul`: Compuite `x*W` and `x*A` simultaneously as `x*concat(W,A)` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
