gemini-code-assist[bot] commented on code in PR #18401:
URL: https://github.com/apache/tvm/pull/18401#discussion_r2466471139
##########
tests/python/relax/test_frontend_from_exported_program.py:
##########
@@ -657,6 +675,122 @@ def main(
verify_model(ReLU6_2(), example_args, {}, expected_relu6_2,
run_ep_decomposition=True)
verify_model(ReLU6_3(), example_args, {}, expected_relu6_3,
run_ep_decomposition=True)
+ # selu
+ class SELU(Module):
+ def forward(self, input):
+ return torch.nn.functional.selu(input)
+
+ @tvm.script.ir_module
+ class expected_selu:
+ @R.function
+ def main(
+ input: R.Tensor((1, 3, 10, 10), dtype="float32")
+ ) -> R.Tuple(R.Tensor((1, 3, 10, 10), dtype="float32")):
+ with R.dataflow():
+ lv: R.Tensor((1, 3, 10, 10), dtype="float32") = R.exp(input)
+ lv1: R.Tensor((1, 3, 10, 10), dtype="float32") = R.subtract(
+ R.const(1.0, "float32"), lv
+ )
+ lv2: R.Tensor((1, 3, 10, 10), dtype="float32") = R.nn.relu(lv1)
+ lv3: R.Tensor((1, 3, 10, 10), dtype="float32") = R.multiply(
+ R.const(-1.6732631921768188, "float32"), lv2
+ )
+ lv4: R.Tensor((1, 3, 10, 10), dtype="float32") =
R.nn.relu(input)
+ lv5: R.Tensor((1, 3, 10, 10), dtype="float32") = R.add(lv3,
lv4)
+ gv: R.Tuple(R.Tensor((1, 3, 10, 10), dtype="float32")) = (lv5,)
+ R.output(gv)
+ return gv
Review Comment:
The decomposition of `selu` seems to be missing the `scale` factor.
According to the PyTorch documentation, `selu(x) = scale * (max(0,x) + min(0,
alpha * (exp(x) - 1)))`, which is equivalent to `scale * elu(x, alpha)`. The
current implementation only computes `elu(x, alpha)` but omits the final
multiplication by `scale` (`1.0507009873554805`).
Also, the value of `alpha` used here (`1.67326319...`) is slightly different
from the one specified in the documentation (`1.67326324...`), which might be
due to floating-point precision differences, but the missing scale factor is a
more significant issue.
```suggestion
with R.dataflow():
lv: R.Tensor((1, 3, 10, 10), dtype="float32") = R.exp(input)
lv1: R.Tensor((1, 3, 10, 10), dtype="float32") = R.subtract(
R.const(1.0, "float32"), lv
)
lv2: R.Tensor((1, 3, 10, 10), dtype="float32") =
R.nn.relu(lv1)
lv3: R.Tensor((1, 3, 10, 10), dtype="float32") = R.multiply(
R.const(-1.6732632423543772, "float32"), lv2
)
lv4: R.Tensor((1, 3, 10, 10), dtype="float32") =
R.nn.relu(input)
lv5: R.Tensor((1, 3, 10, 10), dtype="float32") = R.add(lv3,
lv4)
lv6: R.Tensor((1, 3, 10, 10), dtype="float32") =
R.multiply(lv5, R.const(1.0507009873554805, "float32"))
gv: R.Tuple(R.Tensor((1, 3, 10, 10), dtype="float32")) =
(lv6,)
R.output(gv)
return gv
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
