rdblue commented on issue #25545: [SPARK-28843][PYTHON] Set OMP_NUM_THREADS to executor cores for python URL: https://github.com/apache/spark/pull/25545#issuecomment-524071718 I don't know about any documentation on this, but I'm not very familiar with using numpy. This also seems like something that would be easily overlooked. We see a significant increase because the number of cores in the hardware we're testing tripled. That's the only reason why we caught it, combined with the fact that we limit python memory. That caused jobs to fail, which made is go looking for this. Also, that issue wasn't SO, it was from numpy's issue tracker.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
