This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 9b9f433122c [SPARK-41775][PYTHON][FOLLOWUP] Use `pyspark.cloudpickle`
instead of `cloudpickle`
9b9f433122c is described below
commit 9b9f433122cf72ca5265d10d4b15f6cd2a77c00b
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Jan 24 02:26:11 2023 -0800
[SPARK-41775][PYTHON][FOLLOWUP] Use `pyspark.cloudpickle` instead of
`cloudpickle`
### What changes were proposed in this pull request?
This is a follow-up of #39369 which aims to use `pyspark.cloudpickle`
instead of outside `cloudpickle` dependency.
### Why are the changes needed?
Apache PySpark should not use two versions of `cloudpickle`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
Closes #39715 from dongjoon-hyun/SPARK-41775.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
python/pyspark/ml/torch/distributor.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/pyspark/ml/torch/distributor.py
b/python/pyspark/ml/torch/distributor.py
index fabed806fe4..5f0e930515c 100644
--- a/python/pyspark/ml/torch/distributor.py
+++ b/python/pyspark/ml/torch/distributor.py
@@ -15,7 +15,6 @@
# limitations under the License.
#
-import cloudpickle # type: ignore
from contextlib import contextmanager
import collections
import logging
@@ -31,6 +30,7 @@ import textwrap
import time
from typing import Union, Callable, List, Dict, Optional, Any, Tuple, Generator
+from pyspark import cloudpickle
from pyspark.sql import SparkSession
from pyspark.ml.torch.log_communication import ( # type: ignore
get_driver_host,
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
