kexinyu edited a comment on issue #17164: net.Cast("float16") doesn't work:
Check failed: (*in_type)[i] == dtype_param (2 vs. 0) : This layer requires
uniform type. Expected 'float32' v.s. given 'float16' at 'gamma'
URL:
https://github.com/apache/incubator-mxnet/issues/17164#issuecomment-569778847
> This is a known issue with BatchNorm operator not supporting float16. It
needs some work to support, not currently a bug but not-implemented feature
> @mxnet-label-bot update [Feature ]
Then how does the current BatchNorm operator handle **float16** inputs in
mixed precision training?
https://github.com/apache/incubator-mxnet/blob/b6972bb055fc44481b072db3abb90e26ee27c787/src/operator/nn/batch_norm-inl.h#L292
Here, if the inputs use float16, `DType` becomes float16 but `AccReal` is
float32.
https://github.com/apache/incubator-mxnet/blob/5fb29167a1a66480864486bf59c6b4e980ce7daa/src/operator/nn/batch_norm.cu#L241
Is `gamma * (inp - mean) * invstd + beta` okay with mixed precision
operands?
https://github.com/apache/incubator-mxnet/blob/692f49f2b1b9df1bb226c586405226291c6095cf/src/operator/nn/convolution.cc#L297
Or are they the reason why `UNIFORM_TYPE_CHECK` fails with error
`Check failed: (*in_type)[i] == dtype_param (2 vs. 0) : This layer requires
uniform type. Expected 'float32' v.s. given 'float16' at 'gamma'`
?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
