ehsanmok commented on code in PR #13074:
URL: https://github.com/apache/tvm/pull/13074#discussion_r996180529
##########
python/tvm/relay/frontend/onnx.py:
##########
@@ -944,6 +946,36 @@ def _impl_v1(cls, inputs, attr, params):
return Gelu._impl_v1([inp], attr, params)
+class LayerNormalization(OnnxOpConverter):
+ """Operator converter for LayerNormalization from Microsoft onnxruntime
contrib opset."""
+
+ @classmethod
+ def _impl_v17(cls, inputs, attr, params):
+ x = inputs[0]
+ gamma = inputs[1]
+ beta = inputs[2]
+ axis = attr.get("axis", -1)
+ eps = attr.get("epsilon", 1e-5)
+ # according to the onnx doc, given the int axis (default -1)
+ # to compute the mean and inv_stdev which are of dim [d[0], ...,
d[axis-1], 1, ..., 1]
+ # the actual computation is over (axis, ..., rank(x) - 1) axes
+ # see
https://github.com/onnx/onnx/blob/main/docs/Changelog.md#layernormalization-17
+ rank = len(infer_shape(x))
+ axis = tuple(range(axis, rank)) if axis >= 0 else tuple(range(rank +
axis, rank))
+ dtype = infer_type(x).checked_type.dtype
+ mean = _op.mean(x, axis, keepdims=True)
Review Comment:
As far as I can tell, the `_op.variance` takes care of not doing reduction
twice correctly! note that the test cases test for actual match between all
outputs (with dims included) and the "wide" axes takes care of over what axes
the mean and variance are computed which conform to what onnx spec is referring
[here](https://github.com/onnx/onnx/blob/main/docs/Changelog.md#layernormalization-17)
where
> the shape of Mean and InvStdDev is `[d[0], ..., d[axis-1], 1, ..., 1]`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
