AndrewZhaoLuo commented on code in PR #13074:
URL: https://github.com/apache/tvm/pull/13074#discussion_r996157034
##########
python/tvm/relay/frontend/onnx.py:
##########
@@ -944,6 +946,36 @@ def _impl_v1(cls, inputs, attr, params):
return Gelu._impl_v1([inp], attr, params)
+class LayerNormalization(OnnxOpConverter):
Review Comment:
Do the tests themselves test for these outputs? In the spec they are listed
as optional
LayerNorm, much like BatchNorm has different behavior between training and
inference time. As ONNX frontend is pretty much for inference, I would say it's
ok to drop the other two outputs if it just simplifies the code a lot.
##########
python/tvm/relay/frontend/onnx.py:
##########
@@ -944,6 +946,36 @@ def _impl_v1(cls, inputs, attr, params):
return Gelu._impl_v1([inp], attr, params)
+class LayerNormalization(OnnxOpConverter):
+ """Operator converter for LayerNormalization from Microsoft onnxruntime
contrib opset."""
+
+ @classmethod
+ def _impl_v17(cls, inputs, attr, params):
+ x = inputs[0]
+ gamma = inputs[1]
+ beta = inputs[2]
+ axis = attr.get("axis", -1)
+ eps = attr.get("epsilon", 1e-5)
+ # according to the onnx doc, given the int axis (default -1)
+ # to compute the mean and inv_stdev which are of dim [d[0], ...,
d[axis-1], 1, ..., 1]
+ # the actual computation is over (axis, ..., rank(x) - 1) axes
+ # see
https://github.com/onnx/onnx/blob/main/docs/Changelog.md#layernormalization-17
+ rank = len(infer_shape(x))
+ axis = tuple(range(axis, rank)) if axis >= 0 else tuple(range(rank +
axis, rank))
+ dtype = infer_type(x).checked_type.dtype
+ mean = _op.mean(x, axis, keepdims=True)
Review Comment:
I believe this is inefficient as mean will do a reduction along the axis and
variance will do two reductions (one to find mean, one to find the mean of (X -
E[X])^2. Giving three total reductions across the axis.
Instead just manually do E[(X - E[X])^2] to get two total reductions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
