samskalicky commented on issue #12255: Pretty high cpu load when import mxnet URL: https://github.com/apache/incubator-mxnet/issues/12255#issuecomment-443385116 I added a 2 second delay between launching each process in the script and found that most processes complete the import between 1.7-1.9 seconds. This 2 second delay prevents processes from competing for resources at the same time. There appears to be some bad contention going on thats causing the large delay. Adding the 2 second delay prevented large exponential increase in import delay. Heres the data for 1-36 processes with this 2 second delay using the pip wheel: 1: 2.002468 2: 4.005214 3: 6.007431 4: 8.617790 5: 10.012702 6: 12.015214 7: 14.018058 8: 16.020579 9: 18.022354 10: 20.025505 11: 22.027436 12: 24.030987 13: 26.033214 14: 28.035729 15: 30.037884 16: 32.041215 17: 34.043598 18: 36.044175 19: 38.048719 20: 40.051221 21: 42.053396 22: 44.055702 23: 46.058514 24: 48.060241 25: 50.063950 26: 52.727346 27: 54.068652 28: 56.071813 29: 58.073548 30: 60.076997 31: 62.079071 32: 64.081306 33: 66.716197 34: 68.085373 35: 70.088400 36: 72.092169 So while the 12 seconds for 6 processes isnt ideal, it much better than the 785 that was found earlier without the 2 second delay. So short-term workaround is to add a ~2second delay between launching each process to avoid contention. We'll continue debugging and trace the OpenMP problem and try to resolve it.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services