[GitHub] [spark] rithwik-db commented on a diff in pull request #39267: [WIP][SPARK-41592][PYTHON][ML] Pytorch file Distributed Training

2023-01-17 Thread GitBox
rithwik-db commented on code in PR #39267: URL: https://github.com/apache/spark/pull/39267#discussion_r1072943054 ## python/pyspark/ml/torch/distributor.py: ## @@ -428,6 +432,84 @@ def _run_local_training( return output +def _get_spark_task_program( +

[GitHub] [spark] rithwik-db commented on a diff in pull request #39267: [WIP][SPARK-41592][PYTHON][ML] Pytorch file Distributed Training

2023-01-13 Thread GitBox
rithwik-db commented on code in PR #39267: URL: https://github.com/apache/spark/pull/39267#discussion_r1069068934 ## python/pyspark/ml/torch/distributor.py: ## @@ -428,6 +432,84 @@ def _run_local_training( return output +def _get_spark_task_program( +

[GitHub] [spark] rithwik-db commented on a diff in pull request #39267: [WIP][SPARK-41592][PYTHON][ML] Pytorch file Distributed Training

2023-01-13 Thread GitBox
rithwik-db commented on code in PR #39267: URL: https://github.com/apache/spark/pull/39267#discussion_r1069064636 ## python/pyspark/ml/torch/distributor.py: ## @@ -428,6 +432,84 @@ def _run_local_training( return output +def _get_spark_task_program( +

[GitHub] [spark] rithwik-db commented on a diff in pull request #39267: [WIP][SPARK-41592][PYTHON][ML] Pytorch file Distributed Training

2023-01-13 Thread GitBox
rithwik-db commented on code in PR #39267: URL: https://github.com/apache/spark/pull/39267#discussion_r1069064636 ## python/pyspark/ml/torch/distributor.py: ## @@ -428,6 +432,84 @@ def _run_local_training( return output +def _get_spark_task_program( +

[GitHub] [spark] rithwik-db commented on a diff in pull request #39267: [WIP][SPARK-41592][PYTHON][ML] Pytorch file Distributed Training

2023-01-13 Thread GitBox
rithwik-db commented on code in PR #39267: URL: https://github.com/apache/spark/pull/39267#discussion_r1069064636 ## python/pyspark/ml/torch/distributor.py: ## @@ -428,6 +432,84 @@ def _run_local_training( return output +def _get_spark_task_program( +

[GitHub] [spark] rithwik-db commented on a diff in pull request #39267: [WIP][SPARK-41592][PYTHON][ML] Pytorch file Distributed Training

2023-01-13 Thread GitBox
rithwik-db commented on code in PR #39267: URL: https://github.com/apache/spark/pull/39267#discussion_r1069064636 ## python/pyspark/ml/torch/distributor.py: ## @@ -428,6 +432,84 @@ def _run_local_training( return output +def _get_spark_task_program( +

[GitHub] [spark] rithwik-db commented on a diff in pull request #39267: [WIP][SPARK-41592][PYTHON][ML] Pytorch file Distributed Training

2023-01-12 Thread GitBox
rithwik-db commented on code in PR #39267: URL: https://github.com/apache/spark/pull/39267#discussion_r1068757070 ## python/pyspark/ml/torch/distributor.py: ## @@ -428,6 +432,84 @@ def _run_local_training( return output +def _get_spark_task_program( +

[GitHub] [spark] rithwik-db commented on a diff in pull request #39267: [WIP][SPARK-41592][PYTHON][ML] Pytorch file Distributed Training

2023-01-12 Thread GitBox
rithwik-db commented on code in PR #39267: URL: https://github.com/apache/spark/pull/39267#discussion_r1068755416 ## python/pyspark/ml/torch/distributor.py: ## @@ -428,6 +432,84 @@ def _run_local_training( return output +def _get_spark_task_program( +

[GitHub] [spark] rithwik-db commented on a diff in pull request #39267: [WIP][SPARK-41592][PYTHON][ML] Pytorch file Distributed Training

2023-01-12 Thread GitBox
rithwik-db commented on code in PR #39267: URL: https://github.com/apache/spark/pull/39267#discussion_r1068755416 ## python/pyspark/ml/torch/distributor.py: ## @@ -428,6 +432,84 @@ def _run_local_training( return output +def _get_spark_task_program( +

[GitHub] [spark] rithwik-db commented on a diff in pull request #39267: [WIP][SPARK-41592][PYTHON][ML] Pytorch file Distributed Training

2023-01-12 Thread GitBox
rithwik-db commented on code in PR #39267: URL: https://github.com/apache/spark/pull/39267#discussion_r1068751120 ## python/pyspark/ml/torch/distributor.py: ## @@ -428,6 +432,84 @@ def _run_local_training( return output +def _get_spark_task_program( +

[GitHub] [spark] rithwik-db commented on a diff in pull request #39267: [WIP][SPARK-41592][PYTHON][ML] Pytorch file Distributed Training

2023-01-12 Thread GitBox
rithwik-db commented on code in PR #39267: URL: https://github.com/apache/spark/pull/39267#discussion_r1068751565 ## python/pyspark/ml/torch/distributor.py: ## @@ -428,6 +432,84 @@ def _run_local_training( return output +def _get_spark_task_program( +

[GitHub] [spark] rithwik-db commented on a diff in pull request #39267: [WIP][SPARK-41592][PYTHON][ML] Pytorch file Distributed Training

2023-01-12 Thread GitBox
rithwik-db commented on code in PR #39267: URL: https://github.com/apache/spark/pull/39267#discussion_r1068750849 ## python/pyspark/ml/torch/distributor.py: ## @@ -428,6 +432,84 @@ def _run_local_training( return output +def _get_spark_task_program( +

[GitHub] [spark] rithwik-db commented on a diff in pull request #39267: [WIP][SPARK-41592][PYTHON][ML] Pytorch file Distributed Training

2023-01-11 Thread GitBox
rithwik-db commented on code in PR #39267: URL: https://github.com/apache/spark/pull/39267#discussion_r1067576354 ## python/pyspark/ml/torch/distributor.py: ## @@ -407,13 +418,6 @@ def _run_local_training( try: if self.use_gpu: gpus_owned