wkcn opened a new issue #13928: MXNet 1.5.0 is slower than 1.3.0 when intputs are variant URL: https://github.com/apache/incubator-mxnet/issues/13928 ## Description Hi! I have an experiment about Object Counting, which needs variant inputs. I write the code with Gluon, and hybridize the model with `static_alloc=True` I found there is obvious difference between MXNet 1.5.0 and MXNet 1.3.0, and I checked it on two servers. I think the method of memory allocation for Gluon may be changed after MXNet 1.3.0. Thanks! ## Environment info (Required) OS: ubuntu 14.04 GPU: Tesla M40 x 4 ## Minimum reproducible example I write a minimum reproducible example without dataset. [Code](https://gist.githubusercontent.com/wkcn/69f0f6d2ca467816dc481a00c225104f/raw/2899896f42a920ff0fde5ff93b9a16d16aec507f/test_fcn_for_mxnet.py) - Performance for test code [a fully convolutional model(vgg16 without FC layer), variant inputs]: MXNet 1.5.0: 10 images / sec MXNet 1.3.0: 40+ images / sec The performances are the same when input shape is fixed. Input shape: (9, 3, 300\~512, 300\~512) in NCHW order Package used (Python/R/Scala/Julia): Python 2.7.12, 3.7.1 MXNet is installed by pip: ``` # MXNet 1.5.0 pip install mxnet-cu80 --pre # MXNet 1.3.0 pip install mxnet-cu80==1.3.0 ``` ## Steps to reproduce Download [the test code](https://gist.githubusercontent.com/wkcn/69f0f6d2ca467816dc481a00c225104f/raw/2899896f42a920ff0fde5ff93b9a16d16aec507f/test_fcn_for_mxnet.py). Run the test code in different version (1.3.0 and 1.5.0) of MXNet. ## Performance I test several versions of MXNet. version|performance ----------|---------------- 1.4.0b20181207|slow 1.3.1b20181101|slow 1.3.1b20181010|slow 1.3.1b20181004|fast 1.3.1b20181001|fast Some pre-build versions don't support CUDA9.0, so I cound't test it. The performance drops during 20181004 to 20181010. If changing the dilation of dilated conv to 1, the performance will be normal. It seems the problem occurs in dilated conv.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
