stu1130 opened a new pull request #14884: [Dependency Update] Upgrade cuDNN & NCCL URL: https://github.com/apache/incubator-mxnet/pull/14884 ## Description ## Upgrade the CUDA 9.0/9.2/10.0 with latest cuDNN **7.5.1** & NCCL **2.4.2** ## Checklist ## Run three models ResNet50 with ImageNet & LSTM with PTB & MLP with MNIST Performance shown below Environment: P3.16xlarge Deep Learning Base AMI The unit above is **samples/per second** ### ResNet ### **model**: Resnet50 **dataset**: Imagenet **number of gpu**: 8 **epochs**: 3 (only to test throughput) **preprocess command**: sudo pip install gluoncv==0.2.0b20180625 **command**: python mxnet_benchmark/train_imagenet.py --use-rec --batch-size 128 --dtype float32 —num-data-workers 40 —num-epochs 3 —gpus 0,1,2,3,4,5,6,7 --lr 0.05 --last-gamma —mode symbolic —model resnet50_v1b —rec-train /home/ubuntu/data/train-passthrough.rec —rec-train-idx /home/ubuntu/data/train-passthrough.idx —rec-val /home/ubuntu/data/val-passthrough.rec —rec-val-idx /home/ubuntu/data/val-passthrough.idx **github repo**: https://github.com/rahul003/deep-learning-benchmark-mirror.git* | Throughput Tables | cuDNN 7.5.1/NCCL 2.4.2 | cuDNN 7.3.1/NCCL 2.3.4 | Perforamnce Difference| |:----------|:------------------------:|:--------------------:|:---------------------:| | CUDA 10 | 2831.54405 | 2821.9832 | 0.339% | | CUDA 9.2 | 2832.36803 | 2843.28968 | -0.384% | | CUDA 9.0| 2815.83939 | 2851.92915 | -1.265% | **There is another performance regression with --batch-size 256 --dtype float16 --mode hybrid, please find more details on #14838 ### LSTM ### **model**: LSTM **dataset**: PTB(Penn Treebank) **number of gpu**: 1 **epochs**: 10 **command**: python2 benchmark_driver.py --framework mxnet --task-name mkl_lstm_ptb_symbolic --num-gpus 1 --epochs 10 --metrics-suffix test --kvstore local python word_language_model/lstm_bucketing.py —num-hidden 650 —num-embed 650 —gpus 0 --epochs 10 --kv-store local | Throughput Tables | cuDNN 7.5.1/NCCL 2.4.2 | cuDNN 7.3.1/NCCL 2.3.4 | Perforamnce Difference| |:----------|:------------------------:|:--------------------:|:---------------------:| | CUDA 10 | 847.98222 | 868.28966 | -2.339% | | CUDA 9.2 | 1005.25185 | 1051.06692 | -4.359% | | CUDA 9.0| 1002.59081 | 1028.46962 | -1.265% | **The CUDA 10 have a performance regression issue, please see #14725 to find more details.** ### MLP ### **model**: 3 dense layers with num_hidden=64 and relu as activation **dataset**: MNIST **number of gpu**: 1 **epochs**: 10 **command**: python2 benchmark_runner.py —framework mxnet —metrics-policy mlp —task-name mlp —metrics-suffix test —num-gpus 1 —command-to-execute 'python3 mlp.py' —data-set mnist | Throughput Tables | cuDNN 7.5.1/NCCL 2.4.2 | cuDNN 7.3.1/NCCL 2.3.4 | Perforamnce Difference| |:----------|:------------------------:|:--------------------:|:---------------------:| | CUDA 10 | 4192.20685 | 4094.76838 | 2.38% | | CUDA 9.2 | 4212.68214 | 4280.69164 | -1.589% | | CUDA 9.0| 4232.10159 | 4273.43268 | -0.967%| ## Comments ## @szha @lanking520 @eric-haibin-lin
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
