[GitHub] sxjscience commented on issue #10029: [MXNET-58]Layer Normalization in C++

[GitBox]

sxjscience commented on issue #10029: [MXNET-58]Layer Normalization in C++
URL: https://github.com/apache/incubator-mxnet/pull/10029#issuecomment-418783877
 
 
   @marvis, Would you submit an issue describing the problem with some 
examples. I’ve rechecked the code and it should support in_channels=0.
   
   
   From: Xingjian SHI
   Sent: Wednesday, September 5, 2018 9:53:59 PM
   To: apache/incubator-mxnet; apache/incubator-mxnet
   Cc: Mention
   Subject: Re: [apache/incubator-mxnet] [MXNET-58]Layer Normalization in C++ 
(#10029)
   
   Currently no. I’ll try to support it soon.
   
   Get Outlook for iOS
   
   From: marvis 
   Sent: Wednesday, September 5, 2018 9:02:55 PM
   To: apache/incubator-mxnet
   Cc: Xingjian SHI; Mention
   Subject: Re: [apache/incubator-mxnet] [MXNET-58]Layer Normalization in C++ 
(#10029)
   
   
   Is there a way to infer the in_channels? I am implementing Scale layer, 
which has the same problem.
   
   assert in_channels != 0, "in_channels == 0 is currently not supported"
   
   —
   You are receiving this because you were mentioned.
   Reply to this email directly, view it on 
GitHub,
 or mute the 
thread.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sxjscience commented on issue #10029: [MXNET-58]Layer Normalization in C++

[GitBox]

sxjscience commented on issue #10029: [MXNET-58]Layer Normalization in C++
URL: https://github.com/apache/incubator-mxnet/pull/10029#issuecomment-418739384
 
 
   Currently no. I’ll try to support it soon.
   
   Get Outlook for iOS
   
   From: marvis 
   Sent: Wednesday, September 5, 2018 9:02:55 PM
   To: apache/incubator-mxnet
   Cc: Xingjian SHI; Mention
   Subject: Re: [apache/incubator-mxnet] [MXNET-58]Layer Normalization in C++ 
(#10029)
   
   
   Is there a way to infer the in_channels? I am implementing Scale layer, 
which has the same problem.
   
   assert in_channels != 0, "in_channels == 0 is currently not supported"
   
   —
   You are receiving this because you were mentioned.
   Reply to this email directly, view it on 
GitHub,
 or mute the 
thread.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sxjscience commented on issue #10029: [MXNET-58]Layer Normalization in C++

[GitBox]

sxjscience commented on issue #10029: [MXNET-58]Layer Normalization in C++
URL: https://github.com/apache/incubator-mxnet/pull/10029#issuecomment-371998870
 
 
   @marcoabreu Yes, here the benchmark result. My reference implementation is 
the following LayerNorm that is implemented by stacking broadcasting/reducing 
operators:
   
   ```python
   class LayerNormStackSmallOp(HybridBlock):
   """Applies layer normalization to the n-dimensional input array.
   Stack bcast/reduce
   """
   def __init__(self, axis=1, epsilon=1e-5, center=True, scale=True,
beta_initializer='zeros', gamma_initializer='ones',
in_channels=0, prefix=None, params=None):
   super(LayerNormStackSmallOp, self).__init__(prefix=prefix, 
params=params)
   self._kwargs = {'eps': epsilon, 'axis': axis}
   self._axis = axis
   self._epsilon = epsilon
   self._center = center
   self._scale = scale
   assert in_channels != 0, "in_channels == 0 is currently not 
supported"
   if self._center:
   self.gamma = self.params.get('gamma', grad_req='write' if scale 
else 'null',
shape=(in_channels,), 
init=gamma_initializer,
allow_deferred_init=True)
   if self._scale:
   self.beta = self.params.get('beta', grad_req='write' if center 
else 'null',
   shape=(in_channels,), 
init=beta_initializer,
   allow_deferred_init=True)
   
   def moments(self, F, data):
   mean = F.mean(data=data, axis=self._axis, keepdims=True)
   var = F.mean(F.square(F.broadcast_minus(data, mean)),
axis=self._axis, keepdims=True)
   return mean, var
   
   def hybrid_forward(self, F, data, gamma, beta):
   if not self._center and not self._scale:
   return data
   mean, var = self.moments(F, data)
   norm_data = F.broadcast_minus(data, mean)
   norm_data = F.broadcast_mul(norm_data, mx.sym.rsqrt(var + 
self._epsilon))
   norm_data = F.broadcast_mul(norm_data, gamma)
   norm_data = F.broadcast_add(norm_data, beta)
   return norm_data
   ```
   I run the layer normalization on data with shape=(128, 1024, 100), axis=-1
   
   |  Forward-only  | Time  | Peak GPU Memory |
   | - | -- | - |
   | Layer Norm (Stack Small) | 6.859ms  | 105MB  |
   | Layer Norm (Implemented) | 4.784ms  | 53MB  |
   
   |  Forward + Backward  | Time  | Peak GPU Memory |
   | - | -- | - |
   | Layer Norm (Stack Small) | 7.741 + 17.682 = 25.423ms  | 367MB  |
   | Layer Norm (Implemented) | 5.137 + 10.943 = 16.08ms  | 53MB  |
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sxjscience commented on issue #10029: [MXNET-58]Layer Normalization in C++

[GitBox]

sxjscience commented on issue #10029: [MXNET-58]Layer Normalization in C++
URL: https://github.com/apache/incubator-mxnet/pull/10029#issuecomment-371914848
 
 
   Does anyone has time to review it? The doc page of the latest build is in 
http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-10029/7/index.html


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sxjscience commented on issue #10029: [MXNET-58]Layer Normalization in C++

[GitBox]

sxjscience commented on issue #10029: [MXNET-58]Layer Normalization in C++
URL: https://github.com/apache/incubator-mxnet/pull/10029#issuecomment-371374211
 
 
   Here's the new doc of InstanceNorm 
http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-10029/4/api/python/gluon/nn.html#mxnet.gluon.nn.InstanceNorm
 @zhanghang1989 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sxjscience commented on issue #10029: [MXNET-58]Layer Normalization in C++

[GitBox]

sxjscience commented on issue #10029: [MXNET-58]Layer Normalization in C++
URL: https://github.com/apache/incubator-mxnet/pull/10029#issuecomment-371374211
 
 
   Here's the new doc of InstanceNorm 
http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-10029/4/api/python/gluon/nn.html#mxnet.gluon.nn.LayerNorm
 @zhanghang1989 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services