[GitHub] [incubator-mxnet] yuxihu commented on issue #13825: Dropout is Slow

[GitBox]

yuxihu commented on issue #13825: Dropout is Slow
URL: 
https://github.com/apache/incubator-mxnet/issues/13825#issuecomment-488796151
 
 
   Tested with mxnet-cu90mkl==1.5.0b20190501 on g3.8xlarge instance.
   
   Code example (same as above):
   ```
   import mxnet as mx
   import time
   
   tic = time.time()
   
   data = mx.nd.random.uniform(shape=(100, 3, 224, 224), ctx=mx.gpu())
   for i in range(100):
       # using mode='always' to force dropout behaviour
       data = mx.nd.Dropout(data, 0.5, mode='always')
   mx.nd.waitall()
   
   print("mx.nd.Dropout: ", time.time() - tic)
   
   tic = time.time()
   data = mx.nd.random.uniform(shape=(100, 3, 224, 224), ctx=mx.gpu())
   for i in range(100):
       dropout_mask = mx.nd.random.uniform(shape=(100, 3, 224, 224), 
ctx=mx.gpu()) > 0.5
       data = data * dropout_mask
   mx.nd.waitall()
   
   print("Custom Dropout: ", time.time() - tic)
   
   net = mx.gluon.nn.HybridSequential()
   net.add(mx.gluon.nn.Conv2D(channels=32, kernel_size=3, strides=2))
   net.add(mx.gluon.nn.BatchNorm())
   net.add(mx.gluon.nn.Activation('relu'))
   net.add(mx.gluon.nn.Dropout(0.5))
   net.add(mx.gluon.nn.GlobalMaxPool2D())
   net.add(mx.gluon.nn.Dense(units=1000))
   net.initialize(ctx=mx.gpu())
   
   tic = time.time()
   for i in range(10):
       data = mx.nd.random.uniform(shape=(100, 3, 224, 224), ctx=mx.gpu())
       with mx.autograd.record():
           output = net(data)
       output.backward()
       mx.nd.waitall()
   
   print("mx.gluon.nn.Dropout: ", time.time() - tic)
   
   net = mx.gluon.nn.HybridSequential()
   net.add(mx.gluon.nn.Conv2D(channels=32, kernel_size=3, strides=2))
   net.add(mx.gluon.nn.BatchNorm())
   net.add(mx.gluon.nn.Activation('relu'))
   net.add(mx.gluon.nn.GlobalMaxPool2D())
   net.add(mx.gluon.nn.Dense(units=1000))
   net.initialize(ctx=mx.gpu())
   
   tic = time.time()
   for i in range(10):
       data = mx.nd.random.uniform(shape=(100, 3, 224, 224), ctx=mx.gpu())
       with mx.autograd.record():
           output = net(data)
       output.backward()
       mx.nd.waitall()
   
   print("Without mx.gluon.nn.Dropout: ", time.time() - tic)
   ```
   
   Performance results (in seconds):
   ```
   mx.nd.Dropout:  2.713014841079712
   Custom Dropout:  3.227015733718872
   mx.gluon.nn.Dropout:  0.9665834903717041
   Without mx.gluon.nn.Dropout:  0.7113254070281982
   ```
   Looks like the dropout performance has been significantly improved with 
CUDNN support.
   
   @sandeep-krishnamurthy Please help close this issue. 
   
   @thomelane Please reopen the issue if you have any further questions. Thanks.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services