This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 78246e106e9e [SPARK-60719][PYTHON][CONNECT] Upgrade the minimum
version of grpcio to 1.84.1
78246e106e9e is described below
commit 78246e106e9e429bb6aec98caefc89796088d738
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Wed Nov 4 12:57:37 2026 +0900
[SPARK-60719][PYTHON][CONNECT] Upgrade the minimum version of grpcio to
1.84.1
### What changes were proposed in this pull request?
This PR raises the minimum versions of `grpcio` and `grpcio-status`, which
Spark Connect in PySpark requires, from 1.76.0 to 1.84.1:
- `python/packaging/classic/setup.py` (the `connect` and `pipelines` extras
of `pyspark`), `python/packaging/client/setup.py` (`pyspark-client`) and
`python/packaging/connect/setup.py` (`pyspark-connect`).
- `dev/requirements.txt` and the installation guide.
- `pyproject.toml`: the `connect` dependency group, and the
`internal_ci_connect` and `ci_connect_pandas_3` groups that the Python test
images install (through `ci_connect_standard`, `ci_connect_minimum`, `ci_lint`
and `ci_docs`), so the image of the minimum versions also tests 1.84.1.
- The CI jobs and the release image that pin `grpcio==1.76.0` and
`grpcio-status==1.76.0`. They already pin `protobuf==6.33.5`, the minimum that
`grpcio-status` 1.84.1 requires.
The job of the Spark 4.0 client (`build_python_connect40.yml`) keeps
1.76.0, and the code generation of the protos is unchanged.
### Why are the changes needed?
grpcio 1.84.1 updates the root certificates that it bundles to the Chromium
root store (grpc/grpc#43699). The Python client of Spark Connect creates its
TLS channels with `grpc.ssl_channel_credentials()`
(`DefaultChannelBuilder.toChannel`), without root certificates, so it trusts
the root certificates bundled in grpcio. With this change, installing or
upgrading PySpark with Spark Connect also upgrades grpcio to a version with the
updated roots.
### Does this PR introduce _any_ user-facing change?
Yes. `pyspark[connect]`, `pyspark[pipelines]`, `pyspark-client` and
`pyspark-connect` now require `grpcio>=1.84.1` and `grpcio-status>=1.84.1`, and
so `protobuf>=6.33.5`, which `grpcio-status` 1.84.1 requires.
### How was this patch tested?
Locally, on Python 3.12:
- With `grpcio==1.84.1`, `grpcio-status==1.84.1` and `protobuf==6.33.5`,
`pyspark.sql.connect.utils.check_dependencies()` passes, and
`DefaultChannelBuilder("sc://spark.example.com:443/;use_ssl=true").toChannel()`
creates a TLS channel for `SparkConnectServiceStub`.
- `grpcio-status==1.84.1` resolves with `protobuf==6.33.5`, and not with
`protobuf==6.33.0`.
- The pins of `internal_ci_connect` (`grpcio==1.84.1`,
`grpcio-status==1.84.1`, `googleapis-common-protos==1.71.0`,
`zstandard==0.25.0` and `protobuf==6.33.5`) resolve together.
- The bundled root certificates: 130 in grpcio 1.76.0 and 99 in 1.84.1,
which no longer includes `AAA Certificate Services` or the emSign roots other
than `emSign Root CA - G1`.
CI runs the Python and Connect tests with the new pins.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
This pull request and its description were written by Isaac.
Closes #59925 from HyukjinKwon/SPARK-60719.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.github/workflows/build_and_test.yml | 2 +-
.github/workflows/build_python_connect.yml | 2 +-
.github/workflows/maven_test.yml | 2 +-
.github/workflows/python_hosted_runner_test.yml | 2 +-
dev/create-release/spark-rm/Dockerfile | 4 ++--
dev/requirements.txt | 4 ++--
pyproject.toml | 12 ++++++------
python/docs/source/getting_started/install.rst | 8 ++++----
python/packaging/classic/setup.py | 2 +-
python/packaging/client/setup.py | 2 +-
python/packaging/connect/setup.py | 2 +-
11 files changed, 21 insertions(+), 21 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index cff932a8fc30..a636de174cc4 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -377,7 +377,7 @@ jobs:
- name: Install Python packages (Python 3.12)
if: (contains(matrix.modules, 'sql') && !contains(matrix.modules,
'sql-')) || contains(matrix.modules, 'connect') || contains(matrix.modules,
'yarn')
run: |
- python3.12 -m pip install 'numpy>=1.23.2' pyarrow 'pandas==2.3.3'
pyyaml scipy unittest-xml-reporting 'lxml==4.9.4' 'grpcio==1.76.0'
'grpcio-status==1.76.0' 'protobuf==6.33.5' 'zstandard==0.25.0'
+ python3.12 -m pip install 'numpy>=1.23.2' pyarrow 'pandas==2.3.3'
pyyaml scipy unittest-xml-reporting 'lxml==4.9.4' 'grpcio==1.84.1'
'grpcio-status==1.84.1' 'protobuf==6.33.5' 'zstandard==0.25.0'
python3.12 -m pip list
- name: Download precompiled artifact
uses: actions/download-artifact@v8
diff --git a/.github/workflows/build_python_connect.yml
b/.github/workflows/build_python_connect.yml
index 867b2d017a3d..90fbdd346d4f 100644
--- a/.github/workflows/build_python_connect.yml
+++ b/.github/workflows/build_python_connect.yml
@@ -54,7 +54,7 @@ jobs:
python packaging/client/setup.py sdist
cd dist
pip install pyspark*client-*.tar.gz
- pip install 'grpcio==1.76.0' 'grpcio-status==1.76.0'
'protobuf==6.33.5' 'googleapis-common-protos==1.71.0' 'graphviz==0.20.3'
'six==1.16.0' 'pandas==2.3.3' scipy 'plotly<6.0.0' 'mlflow>=2.8.1' coverage
matplotlib openpyxl 'memory-profiler>=0.61.0' 'scikit-learn>=1.3.2'
'graphviz==0.20.3' 'torch<2.6.0' torchvision torcheval deepspeed
unittest-xml-reporting 'zstandard==0.25.0'
+ pip install 'grpcio==1.84.1' 'grpcio-status==1.84.1'
'protobuf==6.33.5' 'googleapis-common-protos==1.71.0' 'graphviz==0.20.3'
'six==1.16.0' 'pandas==2.3.3' scipy 'plotly<6.0.0' 'mlflow>=2.8.1' coverage
matplotlib openpyxl 'memory-profiler>=0.61.0' 'scikit-learn>=1.3.2'
'graphviz==0.20.3' 'torch<2.6.0' torchvision torcheval deepspeed
unittest-xml-reporting 'zstandard==0.25.0'
- name: List Python packages
run: python -m pip list
- name: Run tests (local)
diff --git a/.github/workflows/maven_test.yml b/.github/workflows/maven_test.yml
index b83be98f947f..80b9102a8243 100644
--- a/.github/workflows/maven_test.yml
+++ b/.github/workflows/maven_test.yml
@@ -216,7 +216,7 @@ jobs:
- name: Install Python packages (Python 3.12)
if: contains(matrix.modules, 'resource-managers#yarn') ||
(contains(matrix.modules, 'sql#core')) || contains(matrix.modules, 'connect')
run: |
- python3.12 -m pip install 'numpy>=1.23.2' pyarrow 'pandas==2.3.3'
pyyaml scipy unittest-xml-reporting 'grpcio==1.76.0' 'grpcio-status==1.76.0'
'protobuf==6.33.5' 'zstandard==0.25.0'
+ python3.12 -m pip install 'numpy>=1.23.2' pyarrow 'pandas==2.3.3'
pyyaml scipy unittest-xml-reporting 'grpcio==1.84.1' 'grpcio-status==1.84.1'
'protobuf==6.33.5' 'zstandard==0.25.0'
python3.12 -m pip list
- name: Download precompiled artifact
id: download-precompiled
diff --git a/.github/workflows/python_hosted_runner_test.yml
b/.github/workflows/python_hosted_runner_test.yml
index 4159cb651c66..5ff58fdf88b7 100644
--- a/.github/workflows/python_hosted_runner_test.yml
+++ b/.github/workflows/python_hosted_runner_test.yml
@@ -174,7 +174,7 @@ jobs:
python${{matrix.python}} -m pip install --ignore-installed
'blinker>=1.6.2'
python${{matrix.python}} -m pip install --ignore-installed
'six==1.16.0'
python${{matrix.python}} -m pip install numpy 'pyarrow>=23.0.0'
'six==1.16.0' 'pandas==2.3.3' scipy 'plotly<6.0.0' 'mlflow>=2.8.1' coverage
matplotlib openpyxl 'memory-profiler>=0.61.0' 'scikit-learn>=1.3.2'
unittest-xml-reporting && \
- python${{matrix.python}} -m pip install 'grpcio==1.76.0'
'grpcio-status==1.76.0' 'protobuf==6.33.5' 'googleapis-common-protos==1.71.0'
'zstandard==0.25.0' 'graphviz==0.20.3' && \
+ python${{matrix.python}} -m pip install 'grpcio==1.84.1'
'grpcio-status==1.84.1' 'protobuf==6.33.5' 'googleapis-common-protos==1.71.0'
'zstandard==0.25.0' 'graphviz==0.20.3' && \
python${{matrix.python}} -m pip cache purge
- name: List Python packages
run: python${{matrix.python}} -m pip list
diff --git a/dev/create-release/spark-rm/Dockerfile
b/dev/create-release/spark-rm/Dockerfile
index e0a3a0c63a2a..bf4dd2afbfc1 100644
--- a/dev/create-release/spark-rm/Dockerfile
+++ b/dev/create-release/spark-rm/Dockerfile
@@ -43,7 +43,7 @@ ARG BASIC_PIP_PKGS="numpy pyarrow>=18.0.0 six==1.16.0
pandas==2.3.3 scipy plotly
mlflow>=2.8.1 coverage matplotlib openpyxl memory-profiler>=0.61.0
scikit-learn>=1.3.2 twine==3.4.1"
# Python deps for Spark Connect
-ARG CONNECT_PIP_PKGS="grpcio==1.76.0 grpcio-status==1.76.0 protobuf==6.33.5 \
+ARG CONNECT_PIP_PKGS="grpcio==1.84.1 grpcio-status==1.84.1 protobuf==6.33.5 \
googleapis-common-protos==1.71.0 graphviz==0.20.3"
# Install Python 3.11 packages
@@ -61,7 +61,7 @@ RUN python3.11 -m pip install 'sphinx==8.2.3' mkdocs
'pydata_sphinx_theme>=0.13'
ipython sphinx_plotly_directive 'numpy>=1.23.2' pyarrow pandas \
'plotly>=4.8' 'docutils' 'mypy==1.19.1' 'pytest==7.1.3' \
'pytest-mypy-plugins==1.9.3' 'ruff==0.14.8' 'pandas-stubs==1.2.0.53' \
- 'grpcio==1.76.0' 'grpc-stubs==1.24.11'
'googleapis-common-protos-stubs==2.2.0'
+ 'grpcio==1.84.1' 'grpc-stubs==1.24.11'
'googleapis-common-protos-stubs==2.2.0'
# Set Python 3.11 as the default
RUN ln -sf "$(which python3.11)" "/usr/local/bin/python" && \
diff --git a/dev/requirements.txt b/dev/requirements.txt
index edcaf053a449..59bcbeb8a178 100644
--- a/dev/requirements.txt
+++ b/dev/requirements.txt
@@ -59,8 +59,8 @@ jira>=3.5.2
PyGithub
# Spark Connect (required)
-grpcio>=1.76.0
-grpcio-status>=1.76.0
+grpcio>=1.84.1
+grpcio-status>=1.84.1
googleapis-common-protos>=1.71.0
protobuf==6.33.5
zstandard>=0.25.0
diff --git a/pyproject.toml b/pyproject.toml
index 457a80234ea4..a3508ca99f37 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -85,8 +85,8 @@ connect = [
{include-group = "internal_pandas"},
{include-group = "internal_protobuf"},
{include-group = "internal_pyarrow"},
- "grpcio>=1.76.0",
- "grpcio-status>=1.76.0",
+ "grpcio>=1.84.1",
+ "grpcio-status>=1.84.1",
"googleapis-common-protos>=1.71.0",
"zstandard>=0.25.0",
"graphviz>=0.20",
@@ -218,8 +218,8 @@ dev = [
internal_ci_connect = [
{include-group = "connect"},
- "grpcio==1.76.0",
- "grpcio-status==1.76.0",
+ "grpcio==1.84.1",
+ "grpcio-status==1.84.1",
"googleapis-common-protos==1.71.0",
"zstandard==0.25.0",
"protobuf==6.33.5",
@@ -276,8 +276,8 @@ ci_classic_pandas_3 = [
]
ci_connect_pandas_3 = [
- "grpcio==1.76.0",
- "grpcio-status==1.76.0",
+ "grpcio==1.84.1",
+ "grpcio-status==1.84.1",
"googleapis-common-protos==1.71.0",
"zstandard==0.25.0",
"protobuf==6.33.5",
diff --git a/python/docs/source/getting_started/install.rst
b/python/docs/source/getting_started/install.rst
index 27309d6120e7..ef20d49254dc 100644
--- a/python/docs/source/getting_started/install.rst
+++ b/python/docs/source/getting_started/install.rst
@@ -239,8 +239,8 @@ Package Supported version Note
========================== ================= ==========================
`pandas` >=2.2.0,<3.0.0 Required for Spark Connect
`pyarrow` >=18.0.0 Required for Spark Connect
-`grpcio` >=1.76.0 Required for Spark Connect
-`grpcio-status` >=1.76.0 Required for Spark Connect
+`grpcio` >=1.84.1 Required for Spark Connect
+`grpcio-status` >=1.84.1 Required for Spark Connect
`googleapis-common-protos` >=1.71.0 Required for Spark Connect
`zstandard` >=0.25.0 Required for Spark Connect
`graphviz` >=0.20 Optional for Spark Connect
@@ -324,8 +324,8 @@ Package Supported version Note
========================== =================
===================================================
`pandas` >=2.2.0,<3.0.0 Required for Spark Connect and
Spark SQL
`pyarrow` >=18.0.0 Required for Spark Connect and
Spark SQL
-`grpcio` >=1.76.0 Required for Spark Connect
-`grpcio-status` >=1.76.0 Required for Spark Connect
+`grpcio` >=1.84.1 Required for Spark Connect
+`grpcio-status` >=1.84.1 Required for Spark Connect
`googleapis-common-protos` >=1.71.0 Required for Spark Connect
`zstandard` >=0.25.0 Required for Spark Connect
`pyyaml` >=3.11 Required for spark-pipelines
command line interface
diff --git a/python/packaging/classic/setup.py
b/python/packaging/classic/setup.py
index fdc97807329a..a7e46f9db849 100755
--- a/python/packaging/classic/setup.py
+++ b/python/packaging/classic/setup.py
@@ -173,7 +173,7 @@ if in_spark:
_minimum_pandas_version = "2.2.0"
_minimum_numpy_version = "1.23.2"
_minimum_pyarrow_version = "18.0.0"
-_minimum_grpc_version = "1.76.0"
+_minimum_grpc_version = "1.84.1"
_minimum_googleapis_common_protos_version = "1.71.0"
_minimum_pyyaml_version = "3.11"
_minimum_zstandard_version = "0.25.0"
diff --git a/python/packaging/client/setup.py b/python/packaging/client/setup.py
index ea3de164bf5a..74bb75cb7738 100755
--- a/python/packaging/client/setup.py
+++ b/python/packaging/client/setup.py
@@ -135,7 +135,7 @@ try:
_minimum_pandas_version = "2.2.0"
_minimum_numpy_version = "1.23.2"
_minimum_pyarrow_version = "18.0.0"
- _minimum_grpc_version = "1.76.0"
+ _minimum_grpc_version = "1.84.1"
_minimum_googleapis_common_protos_version = "1.71.0"
_minimum_pyyaml_version = "3.11"
_minimum_zstandard_version = "0.25.0"
diff --git a/python/packaging/connect/setup.py
b/python/packaging/connect/setup.py
index d24d0436891b..24845643215d 100755
--- a/python/packaging/connect/setup.py
+++ b/python/packaging/connect/setup.py
@@ -87,7 +87,7 @@ try:
_minimum_pandas_version = "2.2.0"
_minimum_numpy_version = "1.23.2"
_minimum_pyarrow_version = "18.0.0"
- _minimum_grpc_version = "1.76.0"
+ _minimum_grpc_version = "1.84.1"
_minimum_googleapis_common_protos_version = "1.71.0"
_minimum_pyyaml_version = "3.11"
_minimum_zstandard_version = "0.25.0"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]