This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 51b8014c3b8 [SPARK-45986][ML][PYTHON] Fix
`pyspark.ml.torch.tests.test_distributor` in Python 3.11
51b8014c3b8 is described below
commit 51b8014c3b8e719b42d67ff1875b69780a059701
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 19 17:33:08 2023 -0800
[SPARK-45986][ML][PYTHON] Fix `pyspark.ml.torch.tests.test_distributor` in
Python 3.11
### What changes were proposed in this pull request?
This PR fixes `pyspark.ml.torch.tests.test_distributor` test to compare the
sorted results. The output fro PyTouch for the same input returns a different
order of output which I presume is from different Python version (e.g., the
order in `set`).
### Why are the changes needed?
To make the tests deterministic.
### Does this PR introduce _any_ user-facing change?
No, test-only.
### How was this patch tested?
Manually tested via:
```bash
./python/run-tests --python-executables=python3 --testnames
'pyspark.ml.torch.tests.test_distributor
TorchDistributorLocalUnitTests.test_local_training_succeeds'
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #43892 from HyukjinKwon/SPARK-45986.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
python/pyspark/ml/torch/tests/test_distributor.py | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/python/pyspark/ml/torch/tests/test_distributor.py
b/python/pyspark/ml/torch/tests/test_distributor.py
index 29ee20d1dfb..8afc141f056 100644
--- a/python/pyspark/ml/torch/tests/test_distributor.py
+++ b/python/pyspark/ml/torch/tests/test_distributor.py
@@ -361,8 +361,7 @@ class TorchDistributorLocalUnitTestsMixin:
def _get_inputs_for_test_local_training_succeeds(self):
return [
- ("0,1,2", 1, True, "1"),
- ("0,1,2", 3, True, "1,2,0"),
+ ("0,1,2", 3, True, "0,1,2"),
("0,1,2", 2, False, "0,1,2"),
(None, 3, False, "NONE"),
]
@@ -381,9 +380,12 @@ class TorchDistributorLocalUnitTestsMixin:
dist._run_training_on_pytorch_file = lambda *args:
os.environ.get(
CUDA_VISIBLE_DEVICES, "NONE"
)
+ output = dist._run_local_training(
+ dist._run_training_on_pytorch_file, "train.py", None
+ )
self.assertEqual(
- expected,
-
dist._run_local_training(dist._run_training_on_pytorch_file, "train.py", None),
+ sorted(expected.split(",")),
+ sorted(output.split(",")),
)
# cleanup
if cuda_env_var:
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
