TomeHirata opened a new pull request, #58375: URL: https://github.com/apache/spark/pull/58375
### What changes were proposed in this pull request? In `TorchDistributor._execute_command`, the loop that streams subprocess output compares `log_streaming_client.sock.getsockname()[0]` with `log_streaming_client.sock.getpeername()[0]` to decide whether to echo each log line to stdout (avoiding duplication when client and server run on the same node). This comparison is now wrapped in a `try/except OSError` block. If the socket call raises, `same_node` falls back to `False`, and the line is written to stdout. ### Why are the changes needed? The log-streaming socket is best effort and can be dropped by a cloud-provider idle-timeout while torch/NCCL initializes (roughly 24 seconds of silence). The next log line then calls `getpeername()` on the dead socket, raising `OSError: [Errno 107] Transport endpoint is not connected`. Because `_execute_command` runs inside a barrier task, this uncaught error aborts the entire multi-node training job approximately 30 seconds in. `LogStreamingClient.send()` in `log_communication.py` already guards against this case (try/except → mark client failed); the getsockname/getpeername comparison was the only remaining unguarded call site. Reproduced on Databricks Runtime 17.3.19 ML GPU against Apache Spark 4.0. ### Does this PR introduce _any_ user-facing change? No API change. Multi-node `TorchDistributor` training that previously crashed with `OSError: [Errno 107] Transport endpoint is not connected` ~30 s into a run will now degrade the best-effort log socket gracefully and continue training. ### How was this patch tested? Added `TorchDistributorBaselineUnitTests.test_execute_command_survives_log_socket_drop` in `python/pyspark/ml/torch/tests/test_distributor.py`. The test configures a mock `LogStreamingClient` whose `getpeername()` raises `OSError(107, ...)`, then asserts that `_execute_command` still completes and the output line reaches stdout. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude (claude-sonnet-4-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
