TaoLv commented on a change in pull request #16735: Use single-bit for mask in
dropout operator
URL: https://github.com/apache/incubator-mxnet/pull/16735#discussion_r361028665
##########
File path: src/operator/nn/dropout-inl.h
##########
@@ -187,28 +188,98 @@ class DropoutOp {
const index_t N,
const index_t step,
DType *dropout_out,
- DType *mask_out,
+ uint8_t *mask_out,
const DType *input_data,
const real_t pkeep) {
+ CHECK_EQ(step & 7, 0);
RNG_KERNEL_LOOP(xpu, DType, id, gen, N, step, {
const real_t rand_num = static_cast<real_t>(genImpl.uniform());
- mask_out[i] = mshadow_op::threshold_eq::Map<real_t>(rand_num, pkeep) *
(1.0f / pkeep);
- dropout_out[i] = input_data[i] * mask_out[i];
- });
+ // mask_out is set per bit position
+ // therefore bitwise shift need to be performed here
+ auto mask_idx = i / 8;
+ auto mask_offset = i % 8;
+ bool mask_val = mshadow_op::threshold_eq::Map<real_t>(rand_num, pkeep);
+ if (mask_val) {
+ // set bit
+ mask_out[mask_idx] |= 1U << mask_offset;
+ } else {
+ // clear bit
+ mask_out[mask_idx] &= ~(1U << mask_offset);
+ }
+ dropout_out[i] = mask_val * input_data[i] * (1.0f / pkeep);
+ })
+ }
+ };
+
+ struct DropoutBackwardKernel {
+ MSHADOW_XINLINE static void Map(index_t i,
+ OpReqType req,
+ DType *igrad,
+ DType *ograd,
+ const uint8_t *mask,
+ const real_t pkeep) {
+ auto mask_idx = i / 8;
+ uint8_t mask_offset = i % 8;
+ bool mask_val = (mask[mask_idx] >> mask_offset) & 1U;
+ KERNEL_ASSIGN(igrad[i], req, mask_val * ograd[i] * (1 / pkeep));
Review comment:
Hah, no strong opinion. Just see doing shift on immediate number more often.
Feel free to pick either one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
