jonatan1626 commented on issue #17159: Performance regression from 1.4.1 to 1.5.1 URL: https://github.com/apache/incubator-mxnet/issues/17159#issuecomment-568630243 Quick update on this, so commit id c5265fbf635965b5124df2d7dcea1b57c6508b73 (I'm labeling this commit number 182, meaning this is 182 commits behind 1.5.1) gives us: "p50": 18.60, "p90": 22.57, "p99": 27.02 The following commits give this error (181, 180): ---------------------------------------------------------------- File "gluon_resnext50.py", line 24, in <module> run_resnext50_gluon(*parse_inputs()) File "gluon_resnext50.py", line 20, in run_resnext50_gluon log_name='resnext50' File "/home/ubuntu/recreate/recreate/perf_script/utils.py", line 67, in run_model infer_time1,infer_results = infer(model, inputs, bsize) File "/home/ubuntu/recreate/recreate/perf_script/infer.py", line 115, in infer_imagenet_gluon return gluon_forward(model,mx_img,bsize) File "/home/ubuntu/recreate/recreate/perf_script/infer.py", line 151, in gluon_forward results.wait_to_read() File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 1819, in wait_to_read check_call(_LIB.MXNDArrayWaitToRead(self.handle)) File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet/base.py", line 253, in check_call raise MXNetError(py_str(_LIB.MXGetLastError())) mxnet.base.MXNetError: [02:10:27] src/operator/nn/mkldnn/mkldnn_base.cc:398: Unknown MKLDNN format for 5 dimensions: 108 Stack trace: [bt] (0) /home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2445fb) [0x7f33d0e745fb] [bt] (1) /home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x250fe5) [0x7f33d0e80fe5] [bt] (2) /home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::NDArray::GetMKLDNNData(mkldnn::memory::primitive_desc const&) const+0x1f4) [0x7f33d320d784] [bt] (3) /home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::GetWeights(mxnet::NDArray const&, mkldnn::memory::primitive_desc const&, int)+0x21) [0x7f33d0e87d11] [bt] (4) /home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::op::MKLDNNConvolutionForwardFullFeature(mxnet::op::MKLDNNConvFullParam const&, mxnet::OpContext const&, mxnet::op::MKLDNNConvForward*, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&)+0x4da) [0x7f33d0ea35ba] [bt] (5) /home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::op::MKLDNNConvolutionForward(nnvm::NodeAttrs const&, mxnet::OpContext const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&)+0x434) [0x7f33d0ea4d04] [bt] (6) /home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x72497a) [0x7f33d135497a] [bt] (7) /home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x241b547) [0x7f33d304b547] [bt] (8) /home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2468cef) [0x7f33d3098cef] -------------------------------------------------------- Commit id e5902ec626400b8b12b2ac1da082ed86bafa67ed (22) is also not runnable. This is the next commit that works commit id d6fd15131f6ab915fc3085b9fdd505e521cc66a1 (21) which gives: "p50": 27.37 "p90": 30.61, "p99": 34.53 There are 160ish commits in between these two commit ids, I haven't tried running them all. I guess the next step would be to try running to see if any of them work at all?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
