[GitHub] [incubator-mxnet] ptrendx commented on a change in pull request #15545: Softmax optimization for GPU

Wed, 21 Aug 2019 09:11:38 -0700 

ptrendx commented on a change in pull request #15545: Softmax optimization for 
GPU
URL: https://github.com/apache/incubator-mxnet/pull/15545#discussion_r316273650
 
 

 ##########
 File path: src/operator/nn/softmax-inl.h
 ##########
 @@ -301,7 +282,7 @@ __global__ void softmax_compute_kernel(DType *in, OType 
*out, index_t M, int axi
 
   red::sum::SetInitValue(smem[x]);
 
 Review comment:
   Not sure I understand.
   
![softmax](https://user-images.githubusercontent.com/8398980/63448504-e0a62e80-c3f2-11e9-8b57-1382384f07e5.gif)
   
   That is not very numerically stable, so there is another step introduced 
that finds the maximum x_i first and compute
   
![softmax](https://user-images.githubusercontent.com/8398980/63448793-92455f80-c3f3-11e9-8016-ed54e2289e69.gif)
   It does not matter which maximum you take.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[email protected]


With regards,
Apache Git Services

Reply via email to