pengzhao-intel commented on issue #10089: enable all activations in MKLDNN.
   Look at the code, two implementations are the difference for the soft_relu. 
So, we get the different results.
   /*! \brief SoftReLU, also known as softplus activation */
   struct softrelu : public mxnet_op::tunable {
     template<typename DType>
     MSHADOW_XINLINE static DType Map(DType a) {
       // Avoid overflow of exp for large inputs.
       // Thresholds 20.0 is chosen such that softrelu(a) = a
       // for a > 20 using floating precision
       if (a > DType(20.0f)) {
         return a;
       } else {
         return DType(math::log1p(math::exp(a)));
   MXNET_UNARY_MATH_OP(softrelu_grad, -math::expm1(-a));
   template <typename T>
   T soft_relu_fwd(T s) {
       return logf(1 + ::expf(s));
   template <typename T>
   T soft_relu_bwd(T dd, T s) {
       return dd / (1 + ::expf(-s));

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

With regards,
Apache Git Services

Reply via email to