bgawrych commented on issue #18244: URL: https://github.com/apache/incubator-mxnet/issues/18244#issuecomment-635911844
I discovered that slowdown is caused by ENABLE_TESTCOVERAGE=ON flag not by OpenMP. I think overhead comes from locking some resources by injected code, because OMP_NUM_THREADS=1 improves speed of tests. With ENABLE_TESTCOVERAGE=1 & unset OMP_NUM_THREADS: - test_operator.py::test_lstm_bidirectional – test time: 305s - tests/python/unittest/test_gluon.py::test_slice_pooling2d_slice_pooling2d - test time: 479.93s With ENABLE_TESTCOVERAGE=1 & OMP_NUM_THREADS=1: - test_operator.py::test_lstm_bidirectional – test time: 236.45s - tests/python/unittest/test_gluon.py::test_slice_pooling2d_slice_pooling2d - test time: 336.66s Without ENABLE_TESTCOVERAGE & unset OMP_NUM_THREADS: - test_operator.py::test_lstm_bidirectional – test time: **40.93s** - tests/python/unittest/test_gluon.py::test_slice_pooling2d_slice_pooling2d - test time: **58.14s** Summary: - setting OMP_NUM_THREADS=1 helps - ENABLE_TESTCOVERAGE flag is root cause of slowdown - setting OMP_NUM_THREADS/4 - for parallel tests helps (or first bullet) CC: @TaoLv , @leezu , @szha ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
