HyukjinKwon opened a new pull request, #59925:
URL: https://github.com/apache/spark/pull/59925
### What changes were proposed in this pull request?
This PR raises the minimum versions of `grpcio` and `grpcio-status`, which
Spark Connect in PySpark requires, from 1.76.0 to 1.84.1:
- `python/packaging/classic/setup.py` (the `connect` and `pipelines` extras
of `pyspark`), `python/packaging/client/setup.py` (`pyspark-client`) and
`python/packaging/connect/setup.py` (`pyspark-connect`).
- `dev/requirements.txt` and the installation guide.
- The CI jobs and the release image that pin `grpcio==1.76.0` and
`grpcio-status==1.76.0`. They already pin `protobuf==6.33.5`, the minimum that
`grpcio-status` 1.84.1 requires.
The job of the Spark 4.0 client (`build_python_connect40.yml`) keeps 1.76.0,
and the code generation of the protos is unchanged.
### Why are the changes needed?
grpcio 1.84.1 updates the root certificates that it bundles to the Chromium
root store (grpc/grpc#43699). The Python client of Spark Connect creates its
TLS channels with `grpc.ssl_channel_credentials()`
(`DefaultChannelBuilder.toChannel`), without root certificates, so it trusts
the root certificates bundled in grpcio. With this change, installing or
upgrading PySpark with Spark Connect also upgrades grpcio to a version with the
updated roots.
### Does this PR introduce _any_ user-facing change?
Yes. `pyspark[connect]`, `pyspark[pipelines]`, `pyspark-client` and
`pyspark-connect` now require `grpcio>=1.84.1` and `grpcio-status>=1.84.1`, and
so `protobuf>=6.33.5`, which `grpcio-status` 1.84.1 requires.
### How was this patch tested?
Locally, on Python 3.12:
- With `grpcio==1.84.1`, `grpcio-status==1.84.1` and `protobuf==6.33.5`,
`pyspark.sql.connect.utils.check_dependencies()` passes, and
`DefaultChannelBuilder("sc://spark.example.com:443/;use_ssl=true").toChannel()`
creates a TLS channel for `SparkConnectServiceStub`.
- `grpcio-status==1.84.1` resolves with `protobuf==6.33.5`, and not with
`protobuf==6.33.0`.
- The bundled root certificates: 130 in grpcio 1.76.0 and 99 in 1.84.1,
which no longer includes `AAA Certificate Services` or the emSign roots other
than `emSign Root CA - G1`.
CI runs the Python and Connect tests with the new pins.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
