Hi, Did you build this with other languages other than python in mind?
Should we be using the rust core from other languages as well or just python? Thanks, Ed Sent from Outlook for iOS<https://aka.ms/o0ukef> ________________________________ From: Hyukjin Kwon <[email protected]> Sent: Saturday, 29 August 2026 23:35:58 To: dev <[email protected]> Subject: [VOTE] Release Apache Spark Connect Rust Client (+ pyspark-client in Rust) 4.2.0 (RC1) Please vote on releasing the following candidate as Apache Spark Connect Rust Client 4.2.0. This is the first release of the ground-up Rust rewrite of the Spark Connect Python client (published to PyPI as pyspark-client-rust): a drop-in replacement for pyspark-client<https://pypi.org/project/pyspark-client/>, backed by a native Rust core instead of py4j/grpcio. The vote is open until Thursday, September 4, 2026, 5:00 PM PDT and passes if a majority of +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark Connect Rust Client 4.2.0 [ ] -1 Do not release this package because … TAG: https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc1 (commit: f77df70) RELEASE FILES (wheels + sdist, built and attached by GitHub Actions): https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc1 DOCUMENTATION: https://apache.github.io/spark-connect-rust/ LIST OF ISSUES: https://issues.apache.org/jira/projects/SPARK/versions/12357484 HOW TO TRY IT — PYTHON (pip) The release attaches abi3 wheels (Linux x86_64/aarch64, macOS universal2, Windows x86_64; CPython 3.9+) plus a source distribution. Grab the wheel for your platform from the release page and install it directly, e.g.: pip install https://github.com/apache/spark-connect-rust/releases/download/v4.2.0-rc1/pyspark_client_rust-4.2.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl Then use it exactly like PySpark. See if your existing workloads work with this. HOW TO TEST IT — RUST (cargo) The Rust crates are intentionally NOT on crates.io<http://crates.io/> for a release candidate (crates.io<http://crates.io/> is immutable; the crates are published there only when the release is finalized). Test against the tagged source instead: # A) Pin a git dependency to the RC tag in your project's Cargo.toml: [dependencies] apache-spark-connect = { git = "https://github.com/apache/spark-connect-rust";, tag = "v4.2.0-rc1" } # B) Or build and run the crates' own tests from the tag: git clone --branch v4.2.0-rc1 https://github.com/apache/spark-connect-rust cd spark-connect-rust cargo build --release cargo test HIGHLIGHTS * Drop-in PySpark Connect Python API, verified against real PySpark 4.2.0 two ways: the official PySpark test suite runs against this client, and a behavioral API-parity sweep compares the Python API surface (functions and DataFrame / Column / SparkSession / GroupedData / Window methods) call-by-call to PySpark. * Spark Connect Rust API full coverage, including Rust UDFs (docs: https://apache.github.io/spark-connect-rust/). * Native Rust transport via tonic; cloudpickle vendored to match upstream, so no grpcio/py4j at runtime. and native type conversion. WHY DOES THE VERSION START AT 4.2.0? Even though this is the first release of the rewrite, the version starts at 4.2.0 to track the Apache Spark / PySpark version it targets. This (1) removes any ambiguity about which Spark version a given client is compatible with, and (2) lets PySpark users pip install pyspark-client-rust== exactly as they would pyspark-client, keeping the same versioning across both. Thanks, Hyukjin Kwon
