HyukjinKwon edited a comment on issue #25545: [SPARK-28843][PYTHON] Set OMP_NUM_THREADS to executor cores for python URL: https://github.com/apache/spark/pull/25545#issuecomment-524595422 I doubt if it is always better. What if, for instance, `spark.executor.cores` is set to 16 because logical cores are 16 by hyper-threading. Physical cores is, let's say 8. I don't know which one `OMP_NUM_THREADS` respects by default but from my rough research it depends on case-by-case. Are we sure if `OMP_NUM_THREADS` = 16 is better or `OMP_NUM_THREADS` = 8 is better, or are we sure which number `OMP_NUM_THREADS` will pick? It is already configurable by users and pretty clear about how to use it. I think we should better investigate to confirm this case is always better with describing the investigation made in PR description, rather than targeting one specific case. Otherwise I wouldn't fix it.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
