mattoh91 commented on PR #39267: URL: https://github.com/apache/spark/pull/39267#issuecomment-1503329907
@rithwik-db Can I clarify that the `num_processors` attribute of the TorchDistributor class refers to the number of spark.executor.cores used, and not the number of spark.executor.instances? Trying to use >1 `num_processors` seems to take up more cores / slots on a single executor during training (using spark operator on k8s). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
