zhengruifeng commented on PR #40695: URL: https://github.com/apache/spark/pull/40695#issuecomment-1501300769
> The existing code it executes pytorch code in client side, but in spark connect case, we should execute pytorch code in server side yes, I feel it is non-trivial to execute pytorch code in server side, since we need to launch a new Python process in the server side and then communicate with it. > I propose to make TorchDistributor._run_local_training only supports spark legacy mode agree > running on spark local mode cluster, but user set TorchDistributor.local_mode=False why not just failing it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
