This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new f737799e15fe [SPARK-54159][INFRA] Fix `pip install` commands to use
proper quotation marks
f737799e15fe is described below
commit f737799e15fe668bb0dfc3203a4bca1e7485466e
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Nov 5 18:21:46 2025 -0800
[SPARK-54159][INFRA] Fix `pip install` commands to use proper quotation
marks
### What changes were proposed in this pull request?
This PR aims to a few `pip install` command to use proper quotation marks
like the following.
```
-RUN python3.13 -m pip install --ignore-installed blinker>=1.6.2 # mlflow
needs this
+RUN python3.13 -m pip install --ignore-installed 'blinker>=1.6.2' # mlflow
needs this
```
```
-RUN python3.13 -m pip install numpy>=2.1 pyarrow>=18.0.0 six==1.16.0
pandas==2.3.3 scipy coverage matplotlib openpyxl grpcio==1.67.0
grpcio-status==1.67.0 lxml jinja2 && \
+RUN python3.13 -m pip install 'numpy>=2.1' 'pyarrow>=18.0.0' 'six==1.16.0'
'pandas==2.3.3' scipy coverage matplotlib openpyxl 'grpcio==1.67.0'
'grpcio-status==1.67.0' lxml jinja2 && \
```
### Why are the changes needed?
SHELL handles `>=` before `pip install` command receives it.
### Does this PR introduce _any_ user-facing change?
No behavior change because this is only changing infra.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
No.
This patch had conflicts when merged, resolved by
Committer: Dongjoon Hyun <[email protected]>
Closes #52857 from dongjoon-hyun/SPARK-54159.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 37689bfea45f3dd3259d3a60b33469cc9bf248a6)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
dev/create-release/spark-rm/Dockerfile | 2 +-
dev/infra/Dockerfile | 12 ++++++------
dev/spark-test-image/numpy-213/Dockerfile | 2 +-
dev/spark-test-image/python-310/Dockerfile | 2 +-
dev/spark-test-image/python-311-classic-only/Dockerfile | 2 +-
dev/spark-test-image/python-311/Dockerfile | 2 +-
dev/spark-test-image/python-312/Dockerfile | 2 +-
dev/spark-test-image/python-313-nogil/Dockerfile | 4 ++--
dev/spark-test-image/python-313/Dockerfile | 2 +-
dev/spark-test-image/python-314/Dockerfile | 2 +-
10 files changed, 16 insertions(+), 16 deletions(-)
diff --git a/dev/create-release/spark-rm/Dockerfile
b/dev/create-release/spark-rm/Dockerfile
index 679998b89392..af8c34852ca0 100644
--- a/dev/create-release/spark-rm/Dockerfile
+++ b/dev/create-release/spark-rm/Dockerfile
@@ -98,7 +98,7 @@ ARG CONNECT_PIP_PKGS="grpcio==1.76.0 grpcio-status==1.76.0
protobuf==6.33.0 goog
# Install Python 3.10 packages
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10
-RUN python3.10 -m pip install --ignore-installed blinker>=1.6.2 # mlflow needs
this
+RUN python3.10 -m pip install --ignore-installed 'blinker>=1.6.2' # mlflow
needs this
RUN python3.10 -m pip install --ignore-installed 'six==1.16.0' # Avoid
`python3-six` installation
RUN python3.10 -m pip install $BASIC_PIP_PKGS unittest-xml-reporting
$CONNECT_PIP_PKGS && \
python3.10 -m pip install 'torch<2.6.0' torchvision --index-url
https://download.pytorch.org/whl/cpu && \
diff --git a/dev/infra/Dockerfile b/dev/infra/Dockerfile
index 423b6ba820d0..fdfd89bdf06d 100644
--- a/dev/infra/Dockerfile
+++ b/dev/infra/Dockerfile
@@ -101,7 +101,7 @@ ARG CONNECT_PIP_PKGS="grpcio==1.76.0 grpcio-status==1.76.0
protobuf==6.33.0 goog
# Install Python 3.10 packages
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10
-RUN python3.10 -m pip install --ignore-installed blinker>=1.6.2 # mlflow needs
this
+RUN python3.10 -m pip install --ignore-installed 'blinker>=1.6.2' # mlflow
needs this
RUN python3.10 -m pip install --ignore-installed 'six==1.16.0' # Avoid
`python3-six` installation
RUN python3.10 -m pip install $BASIC_PIP_PKGS unittest-xml-reporting
$CONNECT_PIP_PKGS && \
python3.10 -m pip install torch torchvision --index-url
https://download.pytorch.org/whl/cpu && \
@@ -114,7 +114,7 @@ RUN apt-get update && apt-get install -y \
python3.9 python3.9-distutils \
&& rm -rf /var/lib/apt/lists/*
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.9
-RUN python3.9 -m pip install --ignore-installed blinker>=1.6.2 # mlflow needs
this
+RUN python3.9 -m pip install --ignore-installed 'blinker>=1.6.2' # mlflow
needs this
RUN python3.9 -m pip install --force $BASIC_PIP_PKGS unittest-xml-reporting
$CONNECT_PIP_PKGS && \
python3.9 -m pip install torch torchvision --index-url
https://download.pytorch.org/whl/cpu && \
python3.9 -m pip install torcheval && \
@@ -125,7 +125,7 @@ RUN apt-get update && apt-get install -y \
python3.11 \
&& rm -rf /var/lib/apt/lists/*
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11
-RUN python3.11 -m pip install --ignore-installed blinker>=1.6.2 # mlflow needs
this
+RUN python3.11 -m pip install --ignore-installed 'blinker>=1.6.2' # mlflow
needs this
RUN python3.11 -m pip install $BASIC_PIP_PKGS unittest-xml-reporting
$CONNECT_PIP_PKGS && \
python3.11 -m pip install torch torchvision --index-url
https://download.pytorch.org/whl/cpu && \
python3.11 -m pip install deepspeed torcheval && \
@@ -136,7 +136,7 @@ RUN apt-get update && apt-get install -y \
python3.12 \
&& rm -rf /var/lib/apt/lists/*
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.12
-RUN python3.12 -m pip install --ignore-installed blinker>=1.6.2 # mlflow needs
this
+RUN python3.12 -m pip install --ignore-installed 'blinker>=1.6.2' # mlflow
needs this
RUN python3.12 -m pip install $BASIC_PIP_PKGS unittest-xml-reporting
$CONNECT_PIP_PKGS lxml && \
python3.12 -m pip install torch torchvision --index-url
https://download.pytorch.org/whl/cpu && \
python3.12 -m pip install torcheval && \
@@ -148,8 +148,8 @@ RUN apt-get update && apt-get install -y \
&& rm -rf /var/lib/apt/lists/*
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.13
# TODO(SPARK-49862) Add BASIC_PIP_PKGS and CONNECT_PIP_PKGS to Python 3.13
image when it supports Python 3.13
-RUN python3.13 -m pip install --ignore-installed blinker>=1.6.2 # mlflow needs
this
-RUN python3.13 -m pip install numpy>=2.1 pyarrow>=18.0.0 six==1.16.0
pandas==2.3.3 scipy coverage matplotlib openpyxl grpcio==1.76.0
grpcio-status==1.76.0 lxml jinja2 && \
+RUN python3.13 -m pip install --ignore-installed 'blinker>=1.6.2' # mlflow
needs this
+RUN python3.13 -m pip install 'numpy>=2.1' 'pyarrow>=18.0.0' 'six==1.16.0'
'pandas==2.3.3' scipy coverage matplotlib openpyxl 'grpcio==1.76.0'
'grpcio-status==1.76.0' lxml jinja2 && \
python3.13 -m pip cache purge
# Remove unused installation packages to free up disk space
diff --git a/dev/spark-test-image/numpy-213/Dockerfile
b/dev/spark-test-image/numpy-213/Dockerfile
index d33fb5f5d30e..06b21bb555bb 100644
--- a/dev/spark-test-image/numpy-213/Dockerfile
+++ b/dev/spark-test-image/numpy-213/Dockerfile
@@ -75,6 +75,6 @@ ARG CONNECT_PIP_PKGS="grpcio==1.76.0 grpcio-status==1.76.0
protobuf==6.33.0 goog
# Install Python 3.11 packages
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11
-RUN python3.11 -m pip install --ignore-installed blinker>=1.6.2 # mlflow needs
this
+RUN python3.11 -m pip install --ignore-installed 'blinker>=1.6.2' # mlflow
needs this
RUN python3.11 -m pip install $BASIC_PIP_PKGS unittest-xml-reporting
$CONNECT_PIP_PKGS && \
python3.11 -m pip cache purge
diff --git a/dev/spark-test-image/python-310/Dockerfile
b/dev/spark-test-image/python-310/Dockerfile
index 46cfce36832b..99d231f47bd7 100644
--- a/dev/spark-test-image/python-310/Dockerfile
+++ b/dev/spark-test-image/python-310/Dockerfile
@@ -70,7 +70,7 @@ ARG CONNECT_PIP_PKGS="grpcio==1.76.0 grpcio-status==1.76.0
protobuf==6.33.0 goog
# Install Python 3.10 packages
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10
-RUN python3.10 -m pip install --ignore-installed blinker>=1.6.2 # mlflow needs
this
+RUN python3.10 -m pip install --ignore-installed 'blinker>=1.6.2' # mlflow
needs this
RUN python3.10 -m pip install --ignore-installed 'six==1.16.0' # Avoid
`python3-six` installation
RUN python3.10 -m pip install $BASIC_PIP_PKGS unittest-xml-reporting
$CONNECT_PIP_PKGS && \
python3.10 -m pip install torch torchvision --index-url
https://download.pytorch.org/whl/cpu && \
diff --git a/dev/spark-test-image/python-311-classic-only/Dockerfile
b/dev/spark-test-image/python-311-classic-only/Dockerfile
index c2d48c5ce877..484c94098844 100644
--- a/dev/spark-test-image/python-311-classic-only/Dockerfile
+++ b/dev/spark-test-image/python-311-classic-only/Dockerfile
@@ -73,7 +73,7 @@ ARG TEST_PIP_PKGS="coverage unittest-xml-reporting"
# Install Python 3.11 packages
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11
-RUN python3.11 -m pip install --ignore-installed blinker>=1.6.2 # mlflow needs
this
+RUN python3.11 -m pip install --ignore-installed 'blinker>=1.6.2' # mlflow
needs this
RUN python3.11 -m pip install $BASIC_PIP_PKGS $TEST_PIP_PKGS && \
python3.11 -m pip install torch torchvision --index-url
https://download.pytorch.org/whl/cpu && \
python3.11 -m pip install deepspeed torcheval && \
diff --git a/dev/spark-test-image/python-311/Dockerfile
b/dev/spark-test-image/python-311/Dockerfile
index 920f21bd47ee..a45ab4554848 100644
--- a/dev/spark-test-image/python-311/Dockerfile
+++ b/dev/spark-test-image/python-311/Dockerfile
@@ -74,7 +74,7 @@ ARG CONNECT_PIP_PKGS="grpcio==1.76.0 grpcio-status==1.76.0
protobuf==6.33.0 goog
# Install Python 3.11 packages
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11
-RUN python3.11 -m pip install --ignore-installed blinker>=1.6.2 # mlflow needs
this
+RUN python3.11 -m pip install --ignore-installed 'blinker>=1.6.2' # mlflow
needs this
RUN python3.11 -m pip install $BASIC_PIP_PKGS unittest-xml-reporting
$CONNECT_PIP_PKGS && \
python3.11 -m pip install torch torchvision --index-url
https://download.pytorch.org/whl/cpu && \
python3.11 -m pip install deepspeed torcheval && \
diff --git a/dev/spark-test-image/python-312/Dockerfile
b/dev/spark-test-image/python-312/Dockerfile
index db1039f5cb26..e05fe327a030 100644
--- a/dev/spark-test-image/python-312/Dockerfile
+++ b/dev/spark-test-image/python-312/Dockerfile
@@ -74,7 +74,7 @@ ARG CONNECT_PIP_PKGS="grpcio==1.76.0 grpcio-status==1.76.0
protobuf==6.33.0 goog
# Install Python 3.12 packages
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.12
-RUN python3.12 -m pip install --ignore-installed blinker>=1.6.2 # mlflow needs
this
+RUN python3.12 -m pip install --ignore-installed 'blinker>=1.6.2' # mlflow
needs this
RUN python3.12 -m pip install $BASIC_PIP_PKGS unittest-xml-reporting
$CONNECT_PIP_PKGS lxml && \
python3.12 -m pip install torch torchvision --index-url
https://download.pytorch.org/whl/cpu && \
python3.12 -m pip install torcheval && \
diff --git a/dev/spark-test-image/python-313-nogil/Dockerfile
b/dev/spark-test-image/python-313-nogil/Dockerfile
index a50bf670b3f3..d4469a9cd494 100644
--- a/dev/spark-test-image/python-313-nogil/Dockerfile
+++ b/dev/spark-test-image/python-313-nogil/Dockerfile
@@ -76,6 +76,6 @@ ARG CONNECT_PIP_PKGS="grpcio==1.76.0 grpcio-status==1.76.0
protobuf==6.33.0 goog
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.13t
# TODO: Add BASIC_PIP_PKGS and CONNECT_PIP_PKGS when it supports Python 3.13
free threaded
# TODO: Add lxml, grpcio, grpcio-status back when they support Python 3.13
free threaded
-RUN python3.13t -m pip install --ignore-installed blinker>=1.6.2 # mlflow
needs this
-RUN python3.13t -m pip install numpy>=2.1 pyarrow>=19.0.0 six==1.16.0
pandas==2.3.3 scipy coverage matplotlib openpyxl jinja2 && \
+RUN python3.13t -m pip install --ignore-installed 'blinker>=1.6.2' # mlflow
needs this
+RUN python3.13t -m pip install 'numpy>=2.1' 'pyarrow>=19.0.0' 'six==1.16.0'
'pandas==2.3.3' scipy coverage matplotlib openpyxl jinja2 && \
python3.13t -m pip cache purge
diff --git a/dev/spark-test-image/python-313/Dockerfile
b/dev/spark-test-image/python-313/Dockerfile
index f74c48bf346f..1d3d73c9feb9 100644
--- a/dev/spark-test-image/python-313/Dockerfile
+++ b/dev/spark-test-image/python-313/Dockerfile
@@ -74,7 +74,7 @@ ARG CONNECT_PIP_PKGS="grpcio==1.76.0 grpcio-status==1.76.0
protobuf==6.33.0 goog
# Install Python 3.13 packages
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.13
-RUN python3.13 -m pip install --ignore-installed blinker>=1.6.2 # mlflow needs
this
+RUN python3.13 -m pip install --ignore-installed 'blinker>=1.6.2' # mlflow
needs this
RUN python3.13 -m pip install $BASIC_PIP_PKGS unittest-xml-reporting
$CONNECT_PIP_PKGS lxml && \
python3.13 -m pip install torch torchvision --index-url
https://download.pytorch.org/whl/cpu && \
python3.13 -m pip install torcheval && \
diff --git a/dev/spark-test-image/python-314/Dockerfile
b/dev/spark-test-image/python-314/Dockerfile
index 2a36da897e2d..236e95e3f52c 100644
--- a/dev/spark-test-image/python-314/Dockerfile
+++ b/dev/spark-test-image/python-314/Dockerfile
@@ -74,7 +74,7 @@ ARG CONNECT_PIP_PKGS="grpcio==1.76.0 grpcio-status==1.76.0
protobuf==6.33.0 goog
# Install Python 3.14 packages
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.14
-RUN python3.14 -m pip install --ignore-installed blinker>=1.6.2 # mlflow needs
this
+RUN python3.14 -m pip install --ignore-installed 'blinker>=1.6.2' # mlflow
needs this
RUN python3.14 -m pip install $BASIC_PIP_PKGS unittest-xml-reporting
$CONNECT_PIP_PKGS lxml && \
python3.14 -m pip install torch torchvision --index-url
https://download.pytorch.org/whl/cpu && \
python3.14 -m pip install torcheval && \
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
