This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 8dcdac234ba [SPARK-46005][INFRA] Upgrade `scikit-learn` to 1.3.2 and
`mlflow` to 2.8.1
8dcdac234ba is described below
commit 8dcdac234ba9c9c9cb80b8a8cc1beab17412fbb8
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Mon Nov 20 08:36:00 2023 -0800
[SPARK-46005][INFRA] Upgrade `scikit-learn` to 1.3.2 and `mlflow` to 2.8.1
### What changes were proposed in this pull request?
1, upgrade `scikit-learn` to the latest 1.3.2
2, sepcify the lower bound of `mlflow` (`mlflow==2.8.1` was already used in
CI)
### Why are the changes needed?
`scikit-learn` was pinned in https://github.com/apache/spark/pull/39467 due
to a [mlflow
issue](https://github.com/apache/spark/pull/39467#issuecomment-1376753692)
seems it has been already fixed
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
ci
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #43905 from zhengruifeng/infra_upgrade_sklearn.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
dev/infra/Dockerfile | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/dev/infra/Dockerfile b/dev/infra/Dockerfile
index 3aac09aa5c1..141c079f393 100644
--- a/dev/infra/Dockerfile
+++ b/dev/infra/Dockerfile
@@ -93,7 +93,7 @@ RUN Rscript -e "devtools::install_version('preferably',
version='0.4', repos='ht
ENV R_LIBS_SITE
"/usr/local/lib/R/site-library:${R_LIBS_SITE}:/usr/lib/R/library"
RUN pypy3 -m pip install numpy 'pandas<=2.1.3' scipy coverage matplotlib
-RUN python3.9 -m pip install numpy 'pyarrow>=14.0.0' 'pandas<=2.1.3' scipy
unittest-xml-reporting plotly>=4.8 'mlflow>=2.3.1' coverage matplotlib openpyxl
'memory-profiler==0.60.0' 'scikit-learn==1.1.*'
+RUN python3.9 -m pip install numpy 'pyarrow>=14.0.0' 'pandas<=2.1.3' scipy
unittest-xml-reporting plotly>=4.8 'mlflow>=2.8.1' coverage matplotlib openpyxl
'memory-profiler==0.60.0' 'scikit-learn>=1.3.2'
# Add Python deps for Spark Connect.
RUN python3.9 -m pip install 'grpcio>=1.48,<1.57' 'grpcio-status>=1.48,<1.57'
'protobuf==4.25.1' 'googleapis-common-protos==1.56.4'
@@ -110,7 +110,7 @@ RUN apt-get update && apt-get install -y \
python3.10 python3.10-distutils \
&& rm -rf /var/lib/apt/lists/*
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10
-RUN python3.10 -m pip install numpy 'pyarrow>=14.0.0' 'pandas<=2.1.3' scipy
unittest-xml-reporting plotly>=4.8 'mlflow>=2.3.1' coverage matplotlib openpyxl
'memory-profiler==0.60.0' 'scikit-learn==1.1.*'
+RUN python3.10 -m pip install numpy 'pyarrow>=14.0.0' 'pandas<=2.1.3' scipy
unittest-xml-reporting plotly>=4.8 'mlflow>=2.8.1' coverage matplotlib openpyxl
'memory-profiler==0.60.0' 'scikit-learn>=1.3.2'
RUN python3.10 -m pip install 'grpcio>=1.48,<1.57' 'grpcio-status>=1.48,<1.57'
'protobuf==4.25.1' 'googleapis-common-protos==1.56.4'
RUN python3.10 -m pip install 'torch<=2.0.1' torchvision --index-url
https://download.pytorch.org/whl/cpu
RUN python3.10 -m pip install torcheval
@@ -122,7 +122,7 @@ RUN apt-get update && apt-get install -y \
python3.11 python3.11-distutils \
&& rm -rf /var/lib/apt/lists/*
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11
-RUN python3.11 -m pip install numpy 'pyarrow>=14.0.0' 'pandas<=2.1.3' scipy
unittest-xml-reporting plotly>=4.8 'mlflow>=2.3.1' coverage matplotlib openpyxl
'memory-profiler==0.60.0' 'scikit-learn==1.1.*'
+RUN python3.11 -m pip install numpy 'pyarrow>=14.0.0' 'pandas<=2.1.3' scipy
unittest-xml-reporting plotly>=4.8 'mlflow>=2.8.1' coverage matplotlib openpyxl
'memory-profiler==0.60.0' 'scikit-learn>=1.3.2'
RUN python3.11 -m pip install 'grpcio>=1.48,<1.57' 'grpcio-status>=1.48,<1.57'
'protobuf==4.25.1' 'googleapis-common-protos==1.56.4'
RUN python3.11 -m pip install 'torch<=2.0.1' torchvision --index-url
https://download.pytorch.org/whl/cpu
RUN python3.11 -m pip install torcheval
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
