Lunderberg opened a new pull request, #16312:
URL: https://github.com/apache/tvm/pull/16312
Whether an optimizations should be performed may depend on when the
variables in an expression are known.
For example, consider a LoRA-adjusted model, with base weights `W` of shape
`[m,n]`, LoRA components `A` and `B` with shapes `[r,n]` and `[m,r]`
respectively, and activations `x` with shape `[n,1]`. The LoRA-adjusted matmul
could be computed either as `(W + B*A)*x` or as `(W*x + B*(A*x))`.
If `A` and `B` are provided at run-time, then computing `(W + B*(A*x))`
requires significantly fewer computations.
* `(W + B*A)*x`: `m*n*(2*r + 3)` operations
1. `B*A`: `2*m*n*r` operations using a naive matmul
2. Adding `W` to (1): `m*n` operations
3. Multiplying `x` by (2): `2*m*n` operations
* `(W*x + B*(A*x))`: (2*m*n + r*(2*n + 2*m + 1))
1. `W*x`: `2*m*n` operations
2. `A*x`: `2*r*n` operations
3. Multiplying `B` by (2): `2*m*r` operations
4. Adding (1) and (3)`: `m` operations
However, if `A` and `B` are known at compile-time, then computing `(W
+ B*A)*x` groups all compile-time values together, allowing them to be
computed earlier (i.e. using `LiftTransformParams`)
* `(W + B*A)*x`: `2*m*n` operations
1. `B*A`: 0 operations, computed at compile-time
2. Adding `W` to (1): 0 operations, computed at compile-time
3. Multiplying `x` by (2): `2*m*n` operations
Since the choice of optimized expression depends on which parameters can be
computed at compile-time, it is useful to have a utility that identifies values
that can be computed at compile-time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
