PistonY commented on issue #13709: Why FP16 training speed is too slow on Tesla T4 in Gluon? URL: https://github.com/apache/incubator-mxnet/issues/13709#issuecomment-450801208 @eric-haibin-lin Hi~I test it with mxnet profiler [here](https://gist.github.com/PistonY/b9dfc38051f3fe6322f154ec4664d0eb) are my script and result.It looks good. But when I actually use it,it's still slow. Here is [script](https://gist.github.com/PistonY/1b869bcd92189b1c84d43d69989a401d). I print time between ```python st_t = time() with autograd.record(): output = train_net(trans.astype(dtype, copy=False)) loss = Loss(output, labels.astype(dtype, copy=False)) loss.backward() trainer.step(batch_size) end_t = time() print(end_t - st_t) ``` when fp16: ```shell float16 Start training with mixup. [15:23:12] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:97: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable) 2.0626039505004883 0.26385951042175293 0.2520616054534912 0.2604227066040039 0.25570082664489746 0.26578330993652344 0.25952720642089844 0.2606792449951172 0.2637202739715576 0.3433563709259033 0.2613410949707031 ``` fp32: ```shell float32 Start training with mixup. [15:36:23] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:97: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable) 0.8311986923217773 0.20481181144714355 0.278350830078125 0.18038034439086914 0.21913409233093262 0.2587764263153076 0.17470550537109375 0.21522021293640137 0.2749063968658447 0.2962362766265869 0.2280411720275879 0.37300872802734375 0.18066024780273438 0.28769636154174805 0.2858397960662842 0.28676462173461914 0.24347591400146484 0.23549628257751465 0.29531288146972656 ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
