[GitHub] [incubator-mxnet] josephevans opened a new issue #18511: Possible Performance Regressions

[GitBox]

josephevans opened a new issue #18511:
URL: https://github.com/apache/incubator-mxnet/issues/18511


   Hi MXNet team,
   
   We recently switched our benchmarking system to use v1.x builds (previously 
using builds from v2.0) and noticed some possible performance regressions. We 
also changed the benchmarks to only run weekly instead of daily, so 
unfortunately we only have 2 data points for comparison. Here are the benchmark 
tests with possible performance regressions:
   
   * gnmt_result_1
   * mobile_pose_mobilenetv3_small_result_8 
   * resnet50_v1_fp32_imperative_result_8
   * resnext101_64x4d_result_8
   * inceptionv3_result_1
       * Regressions in both 1.x and 2.0
   * inference_drmm
   * simple_pose_resnet50_v1b_result_8
   * inference_caffenet
   * inference_resnet50v2_fp16
   * inference_resnet50v2
   * inference_resnet152v2
   * squeezenet1.0_result_1
   
   Please see the attached files for graphs and supporting evidence.
   <img width="1322" alt="gnmt_result_1" 
src="https://user-images.githubusercontent.com/4230183/84064767-10b9c080-a978-11ea-9ec6-64f54b989e8c.png";>
   <img width="1322" alt="mobile_pose_mobilenetv3_small_result_8" 
src="https://user-images.githubusercontent.com/4230183/84064773-13b4b100-a978-11ea-9036-43aad507075e.png";>
   <img width="1322" alt="resnet50_v1_fp32_imperative_result_8" 
src="https://user-images.githubusercontent.com/4230183/84064775-144d4780-a978-11ea-9c9a-420d8d57cf1f.png";>
   <img width="1322" alt="resnext101_64x4d_result_8" 
src="https://user-images.githubusercontent.com/4230183/84064776-14e5de00-a978-11ea-9b18-1b1f9c2029b8.png";>
   <img width="1322" alt="inceptionv3_result_1" 
src="https://user-images.githubusercontent.com/4230183/84064777-157e7480-a978-11ea-8003-335550ce3920.png";>
   <img width="1322" alt="inference_drmm" 
src="https://user-images.githubusercontent.com/4230183/84064782-16170b00-a978-11ea-94fb-70e9bae0cde3.png";>
   <img width="1322" alt="inference_caffenet" 
src="https://user-images.githubusercontent.com/4230183/84064784-16170b00-a978-11ea-950b-7ee70f7b7ce0.png";>
   <img width="1322" alt="inference_resnet50v2_fp16" 
src="https://user-images.githubusercontent.com/4230183/84064787-16afa180-a978-11ea-98d3-afec47812c7b.png";>
   <img width="1322" alt="inference_resnet50v2" 
src="https://user-images.githubusercontent.com/4230183/84064789-17483800-a978-11ea-8aa0-c0669e0a8f50.png";>
   <img width="1322" alt="inference_resnet152v2" 
src="https://user-images.githubusercontent.com/4230183/84064793-17e0ce80-a978-11ea-9cfa-4029d83ee4ba.png";>
   <img width="1322" alt="squeezenet1 0_result_1" 
src="https://user-images.githubusercontent.com/4230183/84064795-18796500-a978-11ea-88f3-7bc2d301e9da.png";>
   <img width="1322" alt="simple_pose_resnet50_v1b_result_8" 
src="https://user-images.githubusercontent.com/4230183/84064798-18796500-a978-11ea-8bb7-aac172d07c69.png";>
   
   
   
   
   
   
   
   
   
   
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org