Please vote on releasing the following candidate as Apache Spark Connect Rust Client 4.2.0.
This is the *first* release of the ground-up Rust rewrite of the Spark Connect Python client (published to PyPI as pyspark-client-rust): a drop-in replacement for pyspark-client <https://pypi.org/project/pyspark-client/>, backed by a native Rust core instead of py4j/grpcio. The vote is open until Thursday, September 4, 2026, 5:00 PM PDT and passes if a majority of +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark Connect Rust Client 4.2.0 [ ] -1 Do not release this package because … TAG: https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc1 (commit: f77df70) RELEASE FILES (wheels + sdist, built and attached by GitHub Actions): https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc1 DOCUMENTATION: https://apache.github.io/spark-connect-rust/ LIST OF ISSUES: https://issues.apache.org/jira/projects/SPARK/versions/12357484 HOW TO TRY IT — PYTHON (pip) The release attaches abi3 wheels (Linux x86_64/aarch64, macOS universal2, Windows x86_64; CPython 3.9+) plus a source distribution. Grab the wheel for your platform from the release page and install it directly, e.g.: pip install https://github.com/apache/spark-connect-rust/releases/download/v4.2.0-rc1/pyspark_client_rust-4.2.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl Then use it exactly like PySpark. See if your existing workloads work with this. HOW TO TEST IT — RUST (cargo) The Rust crates are intentionally NOT on crates.io for a release candidate (crates.io is immutable; the crates are published there only when the release is finalized). Test against the tagged source instead: # A) Pin a git dependency to the RC tag in your project's Cargo.toml: [dependencies] apache-spark-connect = { git = "https://github.com/apache/spark-connect-rust";, tag = "v4.2.0-rc1" } # B) Or build and run the crates' own tests from the tag: git clone --branch v4.2.0-rc1 https://github.com/apache/spark-connect-rust cd spark-connect-rust cargo build --release cargo test HIGHLIGHTS - *Drop-in PySpark Connect Python API*, verified against real PySpark 4.2.0 two ways: the official PySpark test suite runs against this client, and a behavioral API-parity sweep compares the Python API surface (functions and DataFrame / Column / SparkSession / GroupedData / Window methods) call-by-call to PySpark. - Spark Connect Rust API *full* coverage, including *Rust UDF*s (docs: https://apache.github.io/spark-connect-rust/). - *Native Rust transport* via tonic; cloudpickle vendored to match upstream, so no grpcio/py4j at runtime. and *native type conversion.* WHY DOES THE VERSION START AT 4.2.0? Even though this is the first release of the rewrite, the version starts at 4.2.0 to track the Apache Spark / PySpark version it targets. This (1) removes any ambiguity about which Spark version a given client is compatible with, and (2) lets PySpark users pip install pyspark-client-rust== exactly as they would pyspark-client, keeping the same versioning across both. Thanks, Hyukjin Kwon
