TaoLv commented on issue #16735: Use single-bit for mask in dropout operator URL: https://github.com/apache/incubator-mxnet/pull/16735#issuecomment-586608057 > What algorithm is used in TF and pytorch? @pengzhao-intel I don't think TF has a fused dropout operator. It's implement with several small operators. See https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/nn_ops.py#L4456. So the backward path should go through the backward of these small operators. Hence no bit-mask there. For PyTorch, I see there is a fused one: https://github.com/pytorch/pytorch/blob/master/tools/autograd/templates/Functions.cpp#L634. The mask tensor should either be Boolean or has compatible type as grad. So no bit-mask either. > In the ideal case, it would be to enhance MKL dropout to store mask using bits. But it requires modification of the VSL APIs. @apeforest , so far there is no dropout functionality in MKL or MKL-DNN. Here we just use VSL to generate random values. So even we can generate bit-mask, it will increase additional computation for `mask_val * 1.0 / pkeep` which can be reused from forward path.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
