mathewjacob1002 commented on code in PR #41973:
URL: https://github.com/apache/spark/pull/41973#discussion_r1264335633
##########
python/pyspark/ml/torch/distributor.py:
##########
@@ -514,11 +514,54 @@ def _execute_command(
f"Command {cmd} failed with return code {task.returncode}. "
f"The {last_n_msg} included below: {task_output}"
)
+ @staticmethod
+ def _get_output_from_framework_wrapper(framework_wrapper:
Optional[Callable], input_params: Dict, train_object: Union[Callable, str],
run_pytorch_file_fn: Optional[Callable], *args, **kwargs) -> Optional[Any]:
+ """
+ This function is meant to get the output from framework wrapper
function by passing in the correct arguments,
+ depending on the type of train_object.
+
+ Parameters
+ ----------
+ framework_wrapper: Optional[Callable]
+ Function pointer that will be invoked. Can either be the function
that runs distributed training on
+ files if train_object is a string. Otherwise, it will be the
function that runs distributed training
+ for functions if the train_object is a Callable
+ input_params: Dict
+ A dictionary that maps parameter to arguments for the command to
be created.
+ train_object: Union[Callable, str]
Review Comment:
Tried again to make it more obvious which is which. But in a nutshell,
train_object is passed in from the user, and the framework_wrapper is something
that DeepspeedTorchDistributor decides based on the type of train_object.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
