This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 86f15e4a779e [SPARK-47054][PYTHON][TESTS] Remove pinned version of
torch for Python 3.12 support
86f15e4a779e is described below
commit 86f15e4a779ec746373c78c189830cb339b07492
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Thu Feb 15 22:28:37 2024 -0800
[SPARK-47054][PYTHON][TESTS] Remove pinned version of torch for Python 3.12
support
### What changes were proposed in this pull request?
This PR unpins the version for torch in our CI.
This PR is dependent on https://github.com/apache/spark/pull/45115
### Why are the changes needed?
Testing latest version. This also blocks SPARK-46078.
### Does this PR introduce _any_ user-facing change?
No, dev-only.
### How was this patch tested?
Manually tested via `./dev/lint-python`.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #45113 from HyukjinKwon/SPARK-47054.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
dev/infra/Dockerfile | 6 +++---
dev/requirements.txt | 2 +-
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/dev/infra/Dockerfile b/dev/infra/Dockerfile
index 53810756d30c..fa663bc6e419 100644
--- a/dev/infra/Dockerfile
+++ b/dev/infra/Dockerfile
@@ -100,7 +100,7 @@ ARG CONNECT_PIP_PKGS="grpcio==1.59.3 grpcio-status==1.59.3
protobuf==4.25.1 goog
# Add torch as a testing dependency for TorchDistributor and
DeepspeedTorchDistributor
RUN python3.9 -m pip install $BASIC_PIP_PKGS unittest-xml-reporting
$CONNECT_PIP_PKGS && \
- python3.9 -m pip install 'torch<=2.0.1' torchvision --index-url
https://download.pytorch.org/whl/cpu && \
+ python3.9 -m pip install torch torchvision --index-url
https://download.pytorch.org/whl/cpu && \
python3.9 -m pip install deepspeed torcheval && \
python3.9 -m pip cache purge
@@ -111,7 +111,7 @@ RUN apt-get update && apt-get install -y \
&& rm -rf /var/lib/apt/lists/*
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10
RUN python3.10 -m pip install $BASIC_PIP_PKGS unittest-xml-reporting
$CONNECT_PIP_PKGS && \
- python3.10 -m pip install 'torch<=2.0.1' torchvision --index-url
https://download.pytorch.org/whl/cpu && \
+ python3.10 -m pip install torch torchvision --index-url
https://download.pytorch.org/whl/cpu && \
python3.10 -m pip install deepspeed torcheval && \
python3.10 -m pip cache purge
@@ -122,7 +122,7 @@ RUN apt-get update && apt-get install -y \
&& rm -rf /var/lib/apt/lists/*
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11
RUN python3.11 -m pip install $BASIC_PIP_PKGS unittest-xml-reporting
$CONNECT_PIP_PKGS && \
- python3.11 -m pip install 'torch<=2.0.1' torchvision --index-url
https://download.pytorch.org/whl/cpu && \
+ python3.11 -m pip install torch torchvision --index-url
https://download.pytorch.org/whl/cpu && \
python3.11 -m pip install deepspeed torcheval && \
python3.11 -m pip cache purge
diff --git a/dev/requirements.txt b/dev/requirements.txt
index 46a02450d375..6fcd04b6d44a 100644
--- a/dev/requirements.txt
+++ b/dev/requirements.txt
@@ -61,7 +61,7 @@ googleapis-common-protos-stubs==2.2.0
grpc-stubs==1.24.11
# TorchDistributor dependencies
-torch<=2.0.1
+torch
torchvision
torcheval
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
