HyukjinKwon edited a comment on issue #25545: [SPARK-28843][PYTHON] Set OMP_NUM_THREADS to executor cores for python URL: https://github.com/apache/spark/pull/25545#issuecomment-525975790 Yup, release note sounds fine to me. I left a 'release-note' tag in the JIRA. @rdblue, now, what's left is to be clear, in PR description, about what this change affect in PySpark APIs. - Can you clarify that the actual core usage limit is up to `OMP_NUM_THREADS` * actual python processes in PySpark APIs? Since we fork each worker for each execution, is the actual core limit `OMP_NUM_THREADS` * forked processes? I would like to have a clear idea about what happens to core usages in PySpark after this changes. - Also, please describe why `OMP_NUM_THREADS` could affect Python and PySpark - Please update PR description to reflect the current status. Now `OMP_NUM_THREADS` will be only set if it's unset.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
