[GitHub] pengzhao-intel commented on issue #9744: Performance regression when OMP_NUM_THREADS environment variable is not set
pengzhao-intel commented on issue #9744: Performance regression when OMP_NUM_THREADS environment variable is not set URL: https://github.com/apache/incubator-mxnet/issues/9744#issuecomment-405774278 @billy19murahmi please ignore this parameter. It's not used anymore. Sorry for the confusion. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] pengzhao-intel commented on issue #9744: Performance regression when OMP_NUM_THREADS environment variable is not set
pengzhao-intel commented on issue #9744: Performance regression when OMP_NUM_THREADS environment variable is not set URL: https://github.com/apache/incubator-mxnet/issues/9744#issuecomment-380327239 @cjolivier01 how about the progress of KMP setting? We prefer this can be merged into release 1.2 so that the user can get the best performance by default. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] pengzhao-intel commented on issue #9744: Performance regression when OMP_NUM_THREADS environment variable is not set
pengzhao-intel commented on issue #9744: Performance regression when OMP_NUM_THREADS environment variable is not set URL: https://github.com/apache/incubator-mxnet/issues/9744#issuecomment-367911898 The OMP works now from the C code and I think KMP should be. There's C level API too, _int kmp_set_affinity_ in OpenMP, also on this page: https://software.intel.com/en-us/node/522691 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] pengzhao-intel commented on issue #9744: Performance regression when OMP_NUM_THREADS environment variable is not set
pengzhao-intel commented on issue #9744: Performance regression when OMP_NUM_THREADS environment variable is not set URL: https://github.com/apache/incubator-mxnet/issues/9744#issuecomment-367911898 There's C level API too, _int kmp_set_affinity_ in OpenMP. The OMP works now and I think KMP should be. Also on this page: https://software.intel.com/en-us/node/522691 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] pengzhao-intel commented on issue #9744: Performance regression when OMP_NUM_THREADS environment variable is not set
pengzhao-intel commented on issue #9744: Performance regression when OMP_NUM_THREADS environment variable is not set URL: https://github.com/apache/incubator-mxnet/issues/9744#issuecomment-367911898 There're C level API too, _int kmp_set_affinity_ in openmp. The OMP works now and I think KMP should be. Also in this page: https://software.intel.com/en-us/node/522691 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] pengzhao-intel commented on issue #9744: Performance regression when OMP_NUM_THREADS environment variable is not set
pengzhao-intel commented on issue #9744: Performance regression when OMP_NUM_THREADS environment variable is not set URL: https://github.com/apache/incubator-mxnet/issues/9744#issuecomment-367911898 There're C level API too, _int kmp_set_affinity_ in openmp. Also in this page: https://software.intel.com/en-us/node/522691 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] pengzhao-intel commented on issue #9744: Performance regression when OMP_NUM_THREADS environment variable is not set
pengzhao-intel commented on issue #9744: Performance regression when OMP_NUM_THREADS environment variable is not set URL: https://github.com/apache/incubator-mxnet/issues/9744#issuecomment-367904801 Normally, we set KMP_AFFILITY with OMP_NUM_THREADS together. And when the OMP threads are invoked, it will be used. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] pengzhao-intel commented on issue #9744: Performance regression when OMP_NUM_THREADS environment variable is not set
pengzhao-intel commented on issue #9744: Performance regression when OMP_NUM_THREADS environment variable is not set URL: https://github.com/apache/incubator-mxnet/issues/9744#issuecomment-367873227 Thanks, @cjolivier01, I mean we set this environment in the code as OMP_NUM_THREADS. Actually, this setting is useful when hyperthreading is ON and we only want to run each thread in a physical core as below picture shown (KMP_AFFINITY=granularity=fine,compact,1,0). So, this environment will guide how to map thread into the physical core. https://software.intel.com/en-us/node/522691  **KMP_AFFINITY=granularity=fine,compact,1,0** This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] pengzhao-intel commented on issue #9744: Performance regression when OMP_NUM_THREADS environment variable is not set
pengzhao-intel commented on issue #9744: Performance regression when OMP_NUM_THREADS environment variable is not set URL: https://github.com/apache/incubator-mxnet/issues/9744#issuecomment-364357479 We found the similar issue that the performance will be worse if we don't set OMP_NUM_THREADS. @cbalioglu good catch! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] pengzhao-intel commented on issue #9744: Performance regression when OMP_NUM_THREADS environment variable is not set
pengzhao-intel commented on issue #9744: Performance regression when OMP_NUM_THREADS environment variable is not set URL: https://github.com/apache/incubator-mxnet/issues/9744#issuecomment-367711172 @cjolivier01 Thanks for the fix. I have tested in my local with your code. The OMP_NUM_THREADS issue is fixed, but 10% performance degrade from KMP setting. Could you help fix this issue too? Please see my log: [patric@mlt-skx075 mxnet-reg]$ export KMP_AFFINITY=granularity=fine,noduplicates,compact,1,0 [patric@mlt-skx075 mxnet-reg]$ export OMP_NUM_THREADS=56 [patric@mlt-skx075 mxnet-reg]$ python example/image-classification/benchmark_score.py INFO:root:network: resnet-50 INFO:root:device: cpu(0) /home/patric/develop/omp_fix/python/mxnet/module/base_module.py:65: UserWarning: Data provided by label_shapes don't match names specified by label_names ([] vs. ['softmax_label']) warnings.warn(msg) [23:03:45] src/operator/nn/mkldnn/mkldnn_base.cc:60: Allocate 411041792 bytes with malloc directly **INFO:root:batch size 128, image/sec: 189.037230** [patric@mlt-skx075 mxnet-reg]$ unset KMP_AFFINITY [patric@mlt-skx075 mxnet-reg]$ unset OMP_NUM_THREADS [patric@mlt-skx075 mxnet-reg]$ python example/image-classification/benchmark_score.py INFO:root:network: resnet-50 INFO:root:device: cpu(0) /home/patric/develop/omp_fix/python/mxnet/module/base_module.py:65: UserWarning: Data provided by label_shapes don't match names specified by label_names ([] vs. ['softmax_label']) warnings.warn(msg) [23:04:20] src/operator/nn/mkldnn/mkldnn_base.cc:60: Allocate 411041792 bytes with malloc directly **INFO:root:batch size 128, image/sec: 169.450103** [patric@mlt-skx075 mxnet-reg]$ export OMP_NUM_THREADS=56 [patric@mlt-skx075 mxnet-reg]$ python example/image-classification/benchmark_score.py INFO:root:network: resnet-50 INFO:root:device: cpu(0) /home/patric/develop/omp_fix/python/mxnet/module/base_module.py:65: UserWarning: Data provided by label_shapes don't match names specified by label_names ([] vs. ['softmax_label']) warnings.warn(msg) [23:04:44] src/operator/nn/mkldnn/mkldnn_base.cc:60: Allocate 411041792 bytes with malloc directly **INFO:root:batch size 128, image/sec: 168.383920** [patric@mlt-skx075 mxnet-reg]$ export KMP_AFFINITY=granularity=fine,noduplicates,compact,1,0 [patric@mlt-skx075 mxnet-reg]$ python example/image-classification/benchmark_score.py INFO:root:network: resnet-50 INFO:root:device: cpu(0) /home/patric/develop/omp_fix/python/mxnet/module/base_module.py:65: UserWarning: Data provided by label_shapes don't match names specified by label_names ([] vs. ['softmax_label']) warnings.warn(msg) [23:05:10] src/operator/nn/mkldnn/mkldnn_base.cc:60: Allocate 411041792 bytes with malloc directly **INFO:root:batch size 128, image/sec: 188.962690** This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] pengzhao-intel commented on issue #9744: Performance regression when OMP_NUM_THREADS environment variable is not set
pengzhao-intel commented on issue #9744: Performance regression when OMP_NUM_THREADS environment variable is not set URL: https://github.com/apache/incubator-mxnet/issues/9744#issuecomment-364357479 We found the similar issue that the performance will be better when we set OMP_NUM_THREADS. @cbalioglu good catch! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
