Caenorst opened a new issue #9874: ResNet-50 is slower on Volta since #8302 URL: https://github.com/apache/incubator-mxnet/issues/9874 ## Description I ran the Minimum reproducible example with the setup below at two different version (before and after #8302): Here are the results: d03182f4fc084741ea185aac2eb02d0e7b4151f9 (before #8302): - real data: 5644 samples / s - synthetic data: 5971 samples / s c3e3a832bfeceff070ba263aa8a4489ca27f452e (after #8302): - real data: 5461 samples / s - synthetic data: 5740 samples / s Latest: - real data: 5425 samples / s - synthetic data: 5817 samples / s @ptrendx @DickJC123 @mkolod ## Environment info (Required) CPUs: Intel Xeon E5-2698 v4 (x2) GPUs: Nvidia V100 (x8) ## Build info (Required if built from source) From the default config.mk (in make/config.mk) added: ```USE_CUDA=1 USE_CUDNN=1 CUDA_ARCH := -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_70,code=compute_70 USE_CUDA_PATH=/usr/local/cuda USE_LIBJPEG_TURBO=1 USE_LIBJPEG_TURBO_PATH=/usr USE_NCCL=1``` ## Minimum reproducible example python /mxnet/example/image-classification/train_imagenet.py --benchmark 0 --gpu 0,1,2,3,4,5,6,7 --batch-size 1024 --num-epochs 1 --data-train /data/imagenet/train-480-val-256-recordio/train.rec --data-train-idx /data/imagenet/train-480-val-256-recordio/train.idx --data-val /data/imagenet/train-480-val-256-recordio/val.rec --disp-batches 100 --network resnet-v1 --num-layers 50 --data-nthreads 40 --min-random-scale 0.533 --max-random-shear-ratio 0 --max-random-rotate-angle 0 --max-random-h 0 --max-random-l 0 --max-random-s 0 --dtype float16 --kv-store device
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
