jonatan1626 commented on issue #17159: Performance regression from 1.4.1 to 1.5.1 URL: https://github.com/apache/incubator-mxnet/issues/17159#issuecomment-568850668 Thanks @TaoLv for that information! @wuxun-zhang I may be missing something, but I these are the numbers from the rerun. These are my steps: ``` - I start a new c5.18xlarge instance - Choose: Deep Learning AMI (Ubuntu 16.04) Version 26.0 - ami-07728e9e2742b0662 - Launch the instance and copy the git repo to it. - sudo apt-get install numactl - pip install opencv-python - pip install <mxnet_version> - export OMP_NUM_THREADS=18 - export KMP_AFFINITY=granularity=fine,noduplicates,compact,1,0 - numactl --physcpubind=0-17 --membind=0 python gluon_resnext50.py --iteration 10000 ``` Does that seem right or am I missing something? I have also installed mxnet-mkl==1.5.1 onto the other machines and rerun the benchmark and they give around the same numbers I've posted below. ``` 1.4.1 Install: pip install mxnet-mkl==1.4.1 Output: Model: resnext-50 ----Latency---- Model Partition Time: 0 Model Load Time: 48.482418060302734 First Inference: 71.05255126953125 p99: 14.76740837097168 p90: 14.683246612548828 p50: 14.641046524047852 Avg: 14.643481583276303 StdDev: 0.41109802796022626 ``` ``` Install: pip install mxnet-mkl==1.5.1 Output: Model: resnext-50 ----Latency---- Model Partition Time: 0 Model Load Time: 59.892892837524414 First Inference: 90.64745903015137 p99: 23.79322052001953 p90: 23.267030715942383 p50: 22.50361442565918 Avg: 22.492807133890796 StdDev: 0.5872117870215227 ``` ``` Install: pip install mxnet-mkl==1.6.0b20191025 Output: Model: resnext-50 ----Latency---- Model Partition Time: 0 Model Load Time: 240.96202850341797 First Inference: 266.71481132507324 p99: 16.834497451782227 p90: 16.765832901000977 p50: 16.708850860595703 Avg: 16.70833106470542 StdDev: 0.10597406036163494 ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
