This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 78246e106e9e [SPARK-60719][PYTHON][CONNECT] Upgrade the minimum 
version of grpcio to 1.84.1
78246e106e9e is described below

commit 78246e106e9e429bb6aec98caefc89796088d738
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Wed Nov 4 12:57:37 2026 +0900

    [SPARK-60719][PYTHON][CONNECT] Upgrade the minimum version of grpcio to 
1.84.1
    
    ### What changes were proposed in this pull request?
    
    This PR raises the minimum versions of `grpcio` and `grpcio-status`, which 
Spark Connect in PySpark requires, from 1.76.0 to 1.84.1:
    
    - `python/packaging/classic/setup.py` (the `connect` and `pipelines` extras 
of `pyspark`), `python/packaging/client/setup.py` (`pyspark-client`) and 
`python/packaging/connect/setup.py` (`pyspark-connect`).
    - `dev/requirements.txt` and the installation guide.
    - `pyproject.toml`: the `connect` dependency group, and the 
`internal_ci_connect` and `ci_connect_pandas_3` groups that the Python test 
images install (through `ci_connect_standard`, `ci_connect_minimum`, `ci_lint` 
and `ci_docs`), so the image of the minimum versions also tests 1.84.1.
    - The CI jobs and the release image that pin `grpcio==1.76.0` and 
`grpcio-status==1.76.0`. They already pin `protobuf==6.33.5`, the minimum that 
`grpcio-status` 1.84.1 requires.
    
    The job of the Spark 4.0 client (`build_python_connect40.yml`) keeps 
1.76.0, and the code generation of the protos is unchanged.
    
    ### Why are the changes needed?
    
    grpcio 1.84.1 updates the root certificates that it bundles to the Chromium 
root store (grpc/grpc#43699). The Python client of Spark Connect creates its 
TLS channels with `grpc.ssl_channel_credentials()` 
(`DefaultChannelBuilder.toChannel`), without root certificates, so it trusts 
the root certificates bundled in grpcio. With this change, installing or 
upgrading PySpark with Spark Connect also upgrades grpcio to a version with the 
updated roots.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `pyspark[connect]`, `pyspark[pipelines]`, `pyspark-client` and 
`pyspark-connect` now require `grpcio>=1.84.1` and `grpcio-status>=1.84.1`, and 
so `protobuf>=6.33.5`, which `grpcio-status` 1.84.1 requires.
    
    ### How was this patch tested?
    
    Locally, on Python 3.12:
    
    - With `grpcio==1.84.1`, `grpcio-status==1.84.1` and `protobuf==6.33.5`, 
`pyspark.sql.connect.utils.check_dependencies()` passes, and 
`DefaultChannelBuilder("sc://spark.example.com:443/;use_ssl=true").toChannel()` 
creates a TLS channel for `SparkConnectServiceStub`.
    - `grpcio-status==1.84.1` resolves with `protobuf==6.33.5`, and not with 
`protobuf==6.33.0`.
    - The pins of `internal_ci_connect` (`grpcio==1.84.1`, 
`grpcio-status==1.84.1`, `googleapis-common-protos==1.71.0`, 
`zstandard==0.25.0` and `protobuf==6.33.5`) resolve together.
    - The bundled root certificates: 130 in grpcio 1.76.0 and 99 in 1.84.1, 
which no longer includes `AAA Certificate Services` or the emSign roots other 
than `emSign Root CA - G1`.
    
    CI runs the Python and Connect tests with the new pins.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Claude Opus 5.5)
    
    This pull request and its description were written by Isaac.
    
    Closes #59925 from HyukjinKwon/SPARK-60719.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 .github/workflows/build_and_test.yml            |  2 +-
 .github/workflows/build_python_connect.yml      |  2 +-
 .github/workflows/maven_test.yml                |  2 +-
 .github/workflows/python_hosted_runner_test.yml |  2 +-
 dev/create-release/spark-rm/Dockerfile          |  4 ++--
 dev/requirements.txt                            |  4 ++--
 pyproject.toml                                  | 12 ++++++------
 python/docs/source/getting_started/install.rst  |  8 ++++----
 python/packaging/classic/setup.py               |  2 +-
 python/packaging/client/setup.py                |  2 +-
 python/packaging/connect/setup.py               |  2 +-
 11 files changed, 21 insertions(+), 21 deletions(-)

diff --git a/.github/workflows/build_and_test.yml 
b/.github/workflows/build_and_test.yml
index cff932a8fc30..a636de174cc4 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -377,7 +377,7 @@ jobs:
     - name: Install Python packages (Python 3.12)
       if: (contains(matrix.modules, 'sql') && !contains(matrix.modules, 
'sql-')) || contains(matrix.modules, 'connect') || contains(matrix.modules, 
'yarn')
       run: |
-        python3.12 -m pip install 'numpy>=1.23.2' pyarrow 'pandas==2.3.3' 
pyyaml scipy unittest-xml-reporting 'lxml==4.9.4' 'grpcio==1.76.0' 
'grpcio-status==1.76.0' 'protobuf==6.33.5' 'zstandard==0.25.0'
+        python3.12 -m pip install 'numpy>=1.23.2' pyarrow 'pandas==2.3.3' 
pyyaml scipy unittest-xml-reporting 'lxml==4.9.4' 'grpcio==1.84.1' 
'grpcio-status==1.84.1' 'protobuf==6.33.5' 'zstandard==0.25.0'
         python3.12 -m pip list
     - name: Download precompiled artifact
       uses: actions/download-artifact@v8
diff --git a/.github/workflows/build_python_connect.yml 
b/.github/workflows/build_python_connect.yml
index 867b2d017a3d..90fbdd346d4f 100644
--- a/.github/workflows/build_python_connect.yml
+++ b/.github/workflows/build_python_connect.yml
@@ -54,7 +54,7 @@ jobs:
           python packaging/client/setup.py sdist
           cd dist
           pip install pyspark*client-*.tar.gz
-          pip install 'grpcio==1.76.0' 'grpcio-status==1.76.0' 
'protobuf==6.33.5' 'googleapis-common-protos==1.71.0' 'graphviz==0.20.3' 
'six==1.16.0' 'pandas==2.3.3' scipy 'plotly<6.0.0' 'mlflow>=2.8.1' coverage 
matplotlib openpyxl 'memory-profiler>=0.61.0' 'scikit-learn>=1.3.2' 
'graphviz==0.20.3' 'torch<2.6.0' torchvision torcheval deepspeed 
unittest-xml-reporting 'zstandard==0.25.0'
+          pip install 'grpcio==1.84.1' 'grpcio-status==1.84.1' 
'protobuf==6.33.5' 'googleapis-common-protos==1.71.0' 'graphviz==0.20.3' 
'six==1.16.0' 'pandas==2.3.3' scipy 'plotly<6.0.0' 'mlflow>=2.8.1' coverage 
matplotlib openpyxl 'memory-profiler>=0.61.0' 'scikit-learn>=1.3.2' 
'graphviz==0.20.3' 'torch<2.6.0' torchvision torcheval deepspeed 
unittest-xml-reporting 'zstandard==0.25.0'
       - name: List Python packages
         run: python -m pip list
       - name: Run tests (local)
diff --git a/.github/workflows/maven_test.yml b/.github/workflows/maven_test.yml
index b83be98f947f..80b9102a8243 100644
--- a/.github/workflows/maven_test.yml
+++ b/.github/workflows/maven_test.yml
@@ -216,7 +216,7 @@ jobs:
       - name: Install Python packages (Python 3.12)
         if: contains(matrix.modules, 'resource-managers#yarn') || 
(contains(matrix.modules, 'sql#core')) || contains(matrix.modules, 'connect')
         run: |
-          python3.12 -m pip install 'numpy>=1.23.2' pyarrow 'pandas==2.3.3' 
pyyaml scipy unittest-xml-reporting 'grpcio==1.76.0' 'grpcio-status==1.76.0' 
'protobuf==6.33.5' 'zstandard==0.25.0'
+          python3.12 -m pip install 'numpy>=1.23.2' pyarrow 'pandas==2.3.3' 
pyyaml scipy unittest-xml-reporting 'grpcio==1.84.1' 'grpcio-status==1.84.1' 
'protobuf==6.33.5' 'zstandard==0.25.0'
           python3.12 -m pip list
       - name: Download precompiled artifact
         id: download-precompiled
diff --git a/.github/workflows/python_hosted_runner_test.yml 
b/.github/workflows/python_hosted_runner_test.yml
index 4159cb651c66..5ff58fdf88b7 100644
--- a/.github/workflows/python_hosted_runner_test.yml
+++ b/.github/workflows/python_hosted_runner_test.yml
@@ -174,7 +174,7 @@ jobs:
           python${{matrix.python}} -m pip install --ignore-installed 
'blinker>=1.6.2'
           python${{matrix.python}} -m pip install --ignore-installed 
'six==1.16.0'
           python${{matrix.python}} -m pip install numpy 'pyarrow>=23.0.0' 
'six==1.16.0' 'pandas==2.3.3' scipy 'plotly<6.0.0' 'mlflow>=2.8.1' coverage 
matplotlib openpyxl 'memory-profiler>=0.61.0' 'scikit-learn>=1.3.2' 
unittest-xml-reporting && \
-          python${{matrix.python}} -m pip install 'grpcio==1.76.0' 
'grpcio-status==1.76.0' 'protobuf==6.33.5' 'googleapis-common-protos==1.71.0' 
'zstandard==0.25.0' 'graphviz==0.20.3' && \
+          python${{matrix.python}} -m pip install 'grpcio==1.84.1' 
'grpcio-status==1.84.1' 'protobuf==6.33.5' 'googleapis-common-protos==1.71.0' 
'zstandard==0.25.0' 'graphviz==0.20.3' && \
           python${{matrix.python}} -m pip cache purge
       - name: List Python packages
         run: python${{matrix.python}} -m pip list
diff --git a/dev/create-release/spark-rm/Dockerfile 
b/dev/create-release/spark-rm/Dockerfile
index e0a3a0c63a2a..bf4dd2afbfc1 100644
--- a/dev/create-release/spark-rm/Dockerfile
+++ b/dev/create-release/spark-rm/Dockerfile
@@ -43,7 +43,7 @@ ARG BASIC_PIP_PKGS="numpy pyarrow>=18.0.0 six==1.16.0 
pandas==2.3.3 scipy plotly
     mlflow>=2.8.1 coverage matplotlib openpyxl memory-profiler>=0.61.0 
scikit-learn>=1.3.2 twine==3.4.1"
 
 # Python deps for Spark Connect
-ARG CONNECT_PIP_PKGS="grpcio==1.76.0 grpcio-status==1.76.0 protobuf==6.33.5 \
+ARG CONNECT_PIP_PKGS="grpcio==1.84.1 grpcio-status==1.84.1 protobuf==6.33.5 \
     googleapis-common-protos==1.71.0 graphviz==0.20.3"
 
 # Install Python 3.11 packages
@@ -61,7 +61,7 @@ RUN python3.11 -m pip install 'sphinx==8.2.3' mkdocs 
'pydata_sphinx_theme>=0.13'
     ipython sphinx_plotly_directive 'numpy>=1.23.2' pyarrow pandas \
     'plotly>=4.8' 'docutils' 'mypy==1.19.1' 'pytest==7.1.3' \
     'pytest-mypy-plugins==1.9.3' 'ruff==0.14.8' 'pandas-stubs==1.2.0.53' \
-    'grpcio==1.76.0' 'grpc-stubs==1.24.11' 
'googleapis-common-protos-stubs==2.2.0'
+    'grpcio==1.84.1' 'grpc-stubs==1.24.11' 
'googleapis-common-protos-stubs==2.2.0'
 
 # Set Python 3.11 as the default
 RUN ln -sf "$(which python3.11)" "/usr/local/bin/python" && \
diff --git a/dev/requirements.txt b/dev/requirements.txt
index edcaf053a449..59bcbeb8a178 100644
--- a/dev/requirements.txt
+++ b/dev/requirements.txt
@@ -59,8 +59,8 @@ jira>=3.5.2
 PyGithub
 
 # Spark Connect (required)
-grpcio>=1.76.0
-grpcio-status>=1.76.0
+grpcio>=1.84.1
+grpcio-status>=1.84.1
 googleapis-common-protos>=1.71.0
 protobuf==6.33.5
 zstandard>=0.25.0
diff --git a/pyproject.toml b/pyproject.toml
index 457a80234ea4..a3508ca99f37 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -85,8 +85,8 @@ connect = [
     {include-group = "internal_pandas"},
     {include-group = "internal_protobuf"},
     {include-group = "internal_pyarrow"},
-    "grpcio>=1.76.0",
-    "grpcio-status>=1.76.0",
+    "grpcio>=1.84.1",
+    "grpcio-status>=1.84.1",
     "googleapis-common-protos>=1.71.0",
     "zstandard>=0.25.0",
     "graphviz>=0.20",
@@ -218,8 +218,8 @@ dev = [
 
 internal_ci_connect = [
     {include-group = "connect"},
-    "grpcio==1.76.0",
-    "grpcio-status==1.76.0",
+    "grpcio==1.84.1",
+    "grpcio-status==1.84.1",
     "googleapis-common-protos==1.71.0",
     "zstandard==0.25.0",
     "protobuf==6.33.5",
@@ -276,8 +276,8 @@ ci_classic_pandas_3 = [
 ]
 
 ci_connect_pandas_3 = [
-    "grpcio==1.76.0",
-    "grpcio-status==1.76.0",
+    "grpcio==1.84.1",
+    "grpcio-status==1.84.1",
     "googleapis-common-protos==1.71.0",
     "zstandard==0.25.0",
     "protobuf==6.33.5",
diff --git a/python/docs/source/getting_started/install.rst 
b/python/docs/source/getting_started/install.rst
index 27309d6120e7..ef20d49254dc 100644
--- a/python/docs/source/getting_started/install.rst
+++ b/python/docs/source/getting_started/install.rst
@@ -239,8 +239,8 @@ Package                    Supported version Note
 ========================== ================= ==========================
 `pandas`                   >=2.2.0,<3.0.0    Required for Spark Connect
 `pyarrow`                  >=18.0.0          Required for Spark Connect
-`grpcio`                   >=1.76.0          Required for Spark Connect
-`grpcio-status`            >=1.76.0          Required for Spark Connect
+`grpcio`                   >=1.84.1          Required for Spark Connect
+`grpcio-status`            >=1.84.1          Required for Spark Connect
 `googleapis-common-protos` >=1.71.0          Required for Spark Connect
 `zstandard`                >=0.25.0          Required for Spark Connect
 `graphviz`                 >=0.20            Optional for Spark Connect
@@ -324,8 +324,8 @@ Package                    Supported version Note
 ========================== ================= 
===================================================
 `pandas`                   >=2.2.0,<3.0.0    Required for Spark Connect and 
Spark SQL
 `pyarrow`                  >=18.0.0          Required for Spark Connect and 
Spark SQL
-`grpcio`                   >=1.76.0          Required for Spark Connect
-`grpcio-status`            >=1.76.0          Required for Spark Connect
+`grpcio`                   >=1.84.1          Required for Spark Connect
+`grpcio-status`            >=1.84.1          Required for Spark Connect
 `googleapis-common-protos` >=1.71.0          Required for Spark Connect
 `zstandard`                >=0.25.0          Required for Spark Connect
 `pyyaml`                   >=3.11            Required for spark-pipelines 
command line interface
diff --git a/python/packaging/classic/setup.py 
b/python/packaging/classic/setup.py
index fdc97807329a..a7e46f9db849 100755
--- a/python/packaging/classic/setup.py
+++ b/python/packaging/classic/setup.py
@@ -173,7 +173,7 @@ if in_spark:
 _minimum_pandas_version = "2.2.0"
 _minimum_numpy_version = "1.23.2"
 _minimum_pyarrow_version = "18.0.0"
-_minimum_grpc_version = "1.76.0"
+_minimum_grpc_version = "1.84.1"
 _minimum_googleapis_common_protos_version = "1.71.0"
 _minimum_pyyaml_version = "3.11"
 _minimum_zstandard_version = "0.25.0"
diff --git a/python/packaging/client/setup.py b/python/packaging/client/setup.py
index ea3de164bf5a..74bb75cb7738 100755
--- a/python/packaging/client/setup.py
+++ b/python/packaging/client/setup.py
@@ -135,7 +135,7 @@ try:
     _minimum_pandas_version = "2.2.0"
     _minimum_numpy_version = "1.23.2"
     _minimum_pyarrow_version = "18.0.0"
-    _minimum_grpc_version = "1.76.0"
+    _minimum_grpc_version = "1.84.1"
     _minimum_googleapis_common_protos_version = "1.71.0"
     _minimum_pyyaml_version = "3.11"
     _minimum_zstandard_version = "0.25.0"
diff --git a/python/packaging/connect/setup.py 
b/python/packaging/connect/setup.py
index d24d0436891b..24845643215d 100755
--- a/python/packaging/connect/setup.py
+++ b/python/packaging/connect/setup.py
@@ -87,7 +87,7 @@ try:
     _minimum_pandas_version = "2.2.0"
     _minimum_numpy_version = "1.23.2"
     _minimum_pyarrow_version = "18.0.0"
-    _minimum_grpc_version = "1.76.0"
+    _minimum_grpc_version = "1.84.1"
     _minimum_googleapis_common_protos_version = "1.71.0"
     _minimum_pyyaml_version = "3.11"
     _minimum_zstandard_version = "0.25.0"


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to