This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new caec93ae70aa [SPARK-45990][SPARK-45987][PYTHON][CONNECT] Upgrade
`protobuf` to 4.25.1 to support `Python 3.11`
caec93ae70aa is described below
commit caec93ae70aa0a0f83da64f134e1d4824256be14
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Nov 18 22:53:47 2023 -0800
[SPARK-45990][SPARK-45987][PYTHON][CONNECT] Upgrade `protobuf` to 4.25.1 to
support `Python 3.11`
### What changes were proposed in this pull request?
This PR aims to upgrade `protobuf` from `3.20.3` to `4.25.1` to fix PySpark
failures on Python 3.11 environment for Apache Spark 4.0.0.
### Why are the changes needed?
Currently, Python 3.11 Daily CI is failing.
- https://github.com/apache/spark/actions/workflows/build_python.yml
`v4.25.0` is the first release where `protobuf` starts to support Python
3.11 officially.
- https://pypi.org/project/protobuf/4.25.0/
- https://pypi.org/project/protobuf/4.25.1/ (the latest)
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
I also verified this manually.
```
$ python/run-tests --modules=pyspark-connect --parallelism=1
--python-executables=python3.11
...
Finished test(python3.11): pyspark.sql.connect.window (6s)
Tests passed in 780 seconds
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #43885 from dongjoon-hyun/SPARK-45990.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.github/workflows/build_and_test.yml | 4 ++--
dev/infra/Dockerfile | 6 +++---
dev/requirements.txt | 2 +-
3 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index 96d2b5fcc231..135f43d07b13 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -256,7 +256,7 @@ jobs:
- name: Install Python packages (Python 3.8)
if: (contains(matrix.modules, 'sql') && !contains(matrix.modules,
'sql-')) || contains(matrix.modules, 'connect')
run: |
- python3.8 -m pip install 'numpy>=1.20.0' pyarrow pandas scipy
unittest-xml-reporting 'grpcio>=1.48,<1.57' 'grpcio-status>=1.48,<1.57'
'protobuf==3.20.3'
+ python3.8 -m pip install 'numpy>=1.20.0' pyarrow pandas scipy
unittest-xml-reporting 'grpcio>=1.48,<1.57' 'grpcio-status>=1.48,<1.57'
'protobuf==4.25.1'
python3.8 -m pip list
# Run the tests.
- name: Run tests
@@ -708,7 +708,7 @@ jobs:
mkdir -p $HOME/buf
tar -xvzf buf-Linux-x86_64.tar.gz -C $HOME/buf --strip-components 1
rm buf-Linux-x86_64.tar.gz
- python3.9 -m pip install 'protobuf==3.20.3' 'mypy-protobuf==3.3.0'
+ python3.9 -m pip install 'protobuf==4.25.1' 'mypy-protobuf==3.3.0'
- name: Python code generation check
if: inputs.branch != 'branch-3.3' && inputs.branch != 'branch-3.4'
run: if test -f ./dev/connect-check-protos.py; then
PATH=$PATH:$HOME/buf/bin PYTHON_EXECUTABLE=python3.9
./dev/connect-check-protos.py; fi
diff --git a/dev/infra/Dockerfile b/dev/infra/Dockerfile
index a6e55200a55b..cc00289b963b 100644
--- a/dev/infra/Dockerfile
+++ b/dev/infra/Dockerfile
@@ -92,7 +92,7 @@ RUN pypy3 -m pip install numpy 'pandas<=2.1.3' scipy coverage
matplotlib
RUN python3.9 -m pip install numpy 'pyarrow>=14.0.0' 'pandas<=2.1.3' scipy
unittest-xml-reporting plotly>=4.8 'mlflow>=2.3.1' coverage matplotlib openpyxl
'memory-profiler==0.60.0' 'scikit-learn==1.1.*'
# Add Python deps for Spark Connect.
-RUN python3.9 -m pip install 'grpcio>=1.48,<1.57' 'grpcio-status>=1.48,<1.57'
'protobuf==3.20.3' 'googleapis-common-protos==1.56.4'
+RUN python3.9 -m pip install 'grpcio>=1.48,<1.57' 'grpcio-status>=1.48,<1.57'
'protobuf==4.25.1' 'googleapis-common-protos==1.56.4'
# Add torch as a testing dependency for TorchDistributor
RUN python3.9 -m pip install 'torch<=2.0.1' torchvision --index-url
https://download.pytorch.org/whl/cpu
@@ -107,7 +107,7 @@ RUN apt-get update && apt-get install -y \
&& rm -rf /var/lib/apt/lists/*
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10
RUN python3.10 -m pip install numpy 'pyarrow>=14.0.0' 'pandas<=2.1.3' scipy
unittest-xml-reporting plotly>=4.8 'mlflow>=2.3.1' coverage matplotlib openpyxl
'memory-profiler==0.60.0' 'scikit-learn==1.1.*'
-RUN python3.10 -m pip install 'grpcio>=1.48,<1.57' 'grpcio-status>=1.48,<1.57'
'protobuf==3.20.3' 'googleapis-common-protos==1.56.4'
+RUN python3.10 -m pip install 'grpcio>=1.48,<1.57' 'grpcio-status>=1.48,<1.57'
'protobuf==4.25.1' 'googleapis-common-protos==1.56.4'
RUN python3.10 -m pip install 'torch<=2.0.1' torchvision --index-url
https://download.pytorch.org/whl/cpu
RUN python3.10 -m pip install torcheval
RUN python3.10 -m pip install deepspeed
@@ -119,7 +119,7 @@ RUN apt-get update && apt-get install -y \
&& rm -rf /var/lib/apt/lists/*
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11
RUN python3.11 -m pip install numpy 'pyarrow>=14.0.0' 'pandas<=2.1.3' scipy
unittest-xml-reporting plotly>=4.8 'mlflow>=2.3.1' coverage matplotlib openpyxl
'memory-profiler==0.60.0' 'scikit-learn==1.1.*'
-RUN python3.11 -m pip install 'grpcio>=1.48,<1.57' 'grpcio-status>=1.48,<1.57'
'protobuf==3.20.3' 'googleapis-common-protos==1.56.4'
+RUN python3.11 -m pip install 'grpcio>=1.48,<1.57' 'grpcio-status>=1.48,<1.57'
'protobuf==4.25.1' 'googleapis-common-protos==1.56.4'
RUN python3.11 -m pip install 'torch<=2.0.1' torchvision --index-url
https://download.pytorch.org/whl/cpu
RUN python3.11 -m pip install torcheval
RUN python3.11 -m pip install deepspeed
diff --git a/dev/requirements.txt b/dev/requirements.txt
index f222f99d5831..2658f8eec82d 100644
--- a/dev/requirements.txt
+++ b/dev/requirements.txt
@@ -53,7 +53,7 @@ py
# Spark Connect (required)
grpcio>=1.48,<1.57
grpcio-status>=1.48,<1.57
-protobuf==3.20.3
+protobuf==4.25.1
googleapis-common-protos==1.56.4
# Spark Connect python proto generation plugin (optional)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
