HyukjinKwon commented on issue #25545: [SPARK-28843][PYTHON] Set OMP_NUM_THREADS to executor cores for python URL: https://github.com/apache/spark/pull/25545#issuecomment-525958352 I think we can say like `OMP_NUM_THREADS` was set to mahcine's default cores before. So UDF and RDD APIs were executed with those number of (system) threads individually before. Now this is matched to `spark.executor.cores`. Set `spark.executorEnv.OMP_NUM_THREADS` to restore previous behaviour .
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
