This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 37f8221926b [SPARK-46079][INFRA] Install `torch` nightly only at
Python 3.12 in Infra docker image
37f8221926b is described below
commit 37f8221926b8e9f5fbec565db1a33bfd4cd20345
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Nov 23 16:47:21 2023 -0800
[SPARK-46079][INFRA] Install `torch` nightly only at Python 3.12 in Infra
docker image
### What changes were proposed in this pull request?
This PR aims to install `torch` nightly distribution and `torcheval` for
Apache Spark 4.0.0 preparation.
### Why are the changes needed?
Currently, `pytorch` is not available in Python 3.12 environment.
- https://github.com/pytorch/pytorch/issues/110436
We can start to use `nightly` build and switch to the official one via
SPARK-46078 when it's released.
### Does this PR introduce _any_ user-facing change?
No. This is an infra only update.
### How was this patch tested?
1. Passed the image building.
2. Verified manually
```
$ docker run -it --rm
ghcr.io/dongjoon-hyun/apache-spark-ci-image:master-6975332773 python3.12 -m pip
freeze | grep torch
torch==2.2.0.dev20231123+cpu
torcheval==0.0.7
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #43988 from dongjoon-hyun/SPARK-46079.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
dev/infra/Dockerfile | 3 +++
1 file changed, 3 insertions(+)
diff --git a/dev/infra/Dockerfile b/dev/infra/Dockerfile
index 225de5f9ed5..10ae49b7166 100644
--- a/dev/infra/Dockerfile
+++ b/dev/infra/Dockerfile
@@ -136,3 +136,6 @@ RUN apt-get update && apt-get install -y \
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.12
RUN python3.12 -m pip install numpy 'pyarrow>=14.0.0' 'six==1.16.0'
'pandas<=2.1.3' scipy unittest-xml-reporting plotly>=4.8 'mlflow>=2.8.1'
coverage matplotlib openpyxl 'scikit-learn>=1.3.2'
RUN python3.12 -m pip install 'grpcio==1.59.3' 'grpcio-status==1.59.3'
'protobuf==4.25.1' 'googleapis-common-protos==1.56.4'
+# TODO(SPARK-46078) Use official one instead of nightly build when it's ready
+RUN python3.12 -m pip install --pre torch --index-url
https://download.pytorch.org/whl/nightly/cpu
+RUN python3.12 -m pip install torcheval
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
