ehsanmok commented on code in PR #13074:
URL: https://github.com/apache/tvm/pull/13074#discussion_r996180529
##########
python/tvm/relay/frontend/onnx.py:
##########
@@ -944,6 +946,36 @@ def _impl_v1(cls, inputs, attr, params):
return Gelu._impl_v1([inp], attr, params)
+class LayerNormalization(OnnxOpConverter):
+ """Operator converter for LayerNormalization from Microsoft onnxruntime
contrib opset."""
+
+ @classmethod
+ def _impl_v17(cls, inputs, attr, params):
+ x = inputs[0]
+ gamma = inputs[1]
+ beta = inputs[2]
+ axis = attr.get("axis", -1)
+ eps = attr.get("epsilon", 1e-5)
+ # according to the onnx doc, given the int axis (default -1)
+ # to compute the mean and inv_stdev which are of dim [d[0], ...,
d[axis-1], 1, ..., 1]
+ # the actual computation is over (axis, ..., rank(x) - 1) axes
+ # see
https://github.com/onnx/onnx/blob/main/docs/Changelog.md#layernormalization-17
+ rank = len(infer_shape(x))
+ axis = tuple(range(axis, rank)) if axis >= 0 else tuple(range(rank +
axis, rank))
+ dtype = infer_type(x).checked_type.dtype
+ mean = _op.mean(x, axis, keepdims=True)
Review Comment:
Not sure if I understand completely! the test cases test for actual match
between all outputs (with dims included) and the "wide" axes takes care of over
what axes the mean and variance are computed which conform to what onnx spec is
referring
[here](https://github.com/onnx/onnx/blob/main/docs/Changelog.md#layernormalization-17)
where
> the shape of Mean and InvStdDev is `[d[0], ..., d[axis-1], 1, ..., 1]`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
