HyukjinKwon edited a comment on issue #25545: [SPARK-28843][PYTHON] Set OMP_NUM_THREADS to executor cores for python URL: https://github.com/apache/spark/pull/25545#issuecomment-524595422 I doubt if it is always better. What if, for instance, `spark.executor.cores` is set to 16 because logical cores are 16 by hyper-threading. Physical cores is, let's say 8. I don't know which one `OMP_NUM_THREADS` respects by default but from my rough research it depends on case-by-case. Are we sure if `OMP_NUM_THREADS` = 16 is better or `OMP_NUM_THREADS` = 8 is better, or are we sure which number `OMP_NUM_THREADS` will pick? It is already configurable by users and pretty clear about how to use it. I think we should better investigate to confirm this case is always better with describing the investigation made in PR description, rather than targeting one specific case. Otherwise I wouldn't fix it. If you guys still prefer to fix it because it's better to match this number in general (or most cases), let's update migration guide as well that `OMP_NUM_THREADS` will be now matched to `spark.executor.cores`.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
