stsukrov edited a comment on issue #14087: Poor performance of the libmxnet if OMP_PLACES environment variable is present URL: https://github.com/apache/incubator-mxnet/issues/14087#issuecomment-461733006 @pengzhao-intel Thanks, Patric. Appreciate your help during the vacation. NNVM_EXEC_MATCH_RANGE and MXNET_SUBGRAPH_BACKEND are additional optimizations. Great to know. Let's put them aside for a while to investigate just the issue. My results: 1. Nothing was set: ``` INFO:root:network: resnetv1-50 INFO:root:device: cpu(0) /home/stsukrov/workspace/mxnet/python/mxnet/module/base_module.py:67: UserWarning: Data provided by label_shapes don't match names specified by label_names ([] vs. ['softmax_label']) warnings.warn(msg) INFO:root:batch size 1, dtype float32, images/sec: 85.332118 ``` 2. OMP_NUM_THREAD was set to 36 (my cores) ``` INFO:root:network: resnetv1-50 INFO:root:device: cpu(0) /home/stsukrov/workspace/mxnet/python/mxnet/module/base_module.py:67: UserWarning: Data provided by label_shapes don't match names specified by label_names ([] vs. ['softmax_label']) warnings.warn(msg) INFO:root:batch size 1, dtype float32, images/sec: 89.931452 ``` 3. OMP_NUM_THREAD **AND** OMP_PLACES=cores are set ``` INFO:root:network: resnetv1-50 INFO:root:device: cpu(0) /home/stsukrov/workspace/mxnet/python/mxnet/module/base_module.py:67: UserWarning: Data provided by label_shapes don't match names specified by label_names ([] vs. ['softmax_label']) warnings.warn(msg) INFO:root:batch size 1, dtype float32, images/sec: 4.841513 ``` 4. set OMP_NUM_THREAD and KMP_AFFILITY. Keep OMP_PLACES ``` OMP: Warning #181: OMP_PLACES: ignored because KMP_AFFINITY has been defined INFO:root:network: resnetv1-50 INFO:root:device: cpu(0) /home/stsukrov/workspace/mxnet/python/mxnet/module/base_module.py:67: UserWarning: Data provided by label_shapes don't match names specified by label_names ([] vs. ['softmax_label']) warnings.warn(msg) INFO:root:batch size 1, dtype float32, images/sec: 4.848529 ``` Note the message about ignoring OMP_PLACES. 5. set OMP_NUM_THREAD and KMP_AFFILITY. Remove OMP_PLACES ``` INFO:root:network: resnetv1-50 INFO:root:device: cpu(0) /home/stsukrov/workspace/mxnet/python/mxnet/module/base_module.py:67: UserWarning: Data provided by label_shapes don't match names specified by label_names ([] vs. ['softmax_label']) warnings.warn(msg) INFO:root:batch size 1, dtype float32, images/sec: 99.267594 ``` Looks good to me. So, yes. just OMP_NUM_THREAD and KMP_AFFILITY do. OMP_PLACES kills the performance.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
