+1 (non-binding) Thanks, that makes sense. I don’t expect there to be much objection. I was mainly surprised that there hadn’t been a separate discussion before the release vote, given the addition of the Python wrapper and new package.
El dom, 30 ago 2026 a las 3:23, Hyukjin Kwon (<[email protected]>) escribió: > It is the Rust Client, and the new one is a Python wrapper of the Rust > client, which conceptually thought did not require an SPIP but rather a > separate feature. If you feel like it has to be discussed separately, I can > bring it up and pause the release vote for now. > > On Sun, 30 Aug 2026 at 10:20, Ángel Álvarez Pascua < > [email protected]> wrote: > >> Could you please point me to the previous dev@ discussion or SPIP for >> this proposal? >> >> I found the earlier discussion about creating the Spark Connect Rust >> client repository, but not one covering the ground-up Rust rewrite and the >> new pyspark-client-rust package. Given the scope and user-facing nature >> of these changes, I would have expected them to be discussed before >> reaching the release vote, possibly through the SPIP process. >> >> El dom, 30 ago 2026 a las 0:37, Hyukjin Kwon (<[email protected]>) >> escribió: >> >>> Please vote on releasing the following candidate as Apache Spark Connect >>> Rust >>> Client 4.2.0. >>> >>> This is the *first* release of the ground-up Rust rewrite of the Spark >>> Connect >>> Python client (published to PyPI as pyspark-client-rust): a drop-in >>> replacement >>> for pyspark-client <https://pypi.org/project/pyspark-client/>, backed >>> by a native Rust core instead of py4j/grpcio. >>> >>> The vote is open until Thursday, September 4, 2026, 5:00 PM PDT and >>> passes if a majority of +1 PMC votes are cast, with a minimum of >>> 3 +1 votes. >>> >>> [ ] +1 Release this package as Apache Spark Connect Rust Client 4.2.0 >>> [ ] -1 Do not release this package because … >>> >>> TAG: >>> https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc1 >>> (commit: f77df70) >>> >>> RELEASE FILES (wheels + sdist, built and attached by GitHub Actions): >>> https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc1 >>> >>> DOCUMENTATION: >>> https://apache.github.io/spark-connect-rust/ >>> >>> LIST OF ISSUES: >>> https://issues.apache.org/jira/projects/SPARK/versions/12357484 >>> HOW TO TRY IT — PYTHON (pip) >>> >>> The release attaches abi3 wheels (Linux x86_64/aarch64, macOS universal2, >>> Windows x86_64; CPython 3.9+) plus a source distribution. Grab the wheel >>> for >>> your platform from the release page and install it directly, e.g.: >>> >>> pip install >>> https://github.com/apache/spark-connect-rust/releases/download/v4.2.0-rc1/pyspark_client_rust-4.2.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl >>> >>> Then use it exactly like PySpark. See if your existing workloads work >>> with this. >>> HOW TO TEST IT — RUST (cargo) >>> >>> The Rust crates are intentionally NOT on crates.io for a release >>> candidate >>> (crates.io is immutable; the crates are published there only when the >>> release is >>> finalized). Test against the tagged source instead: >>> >>> # A) Pin a git dependency to the RC tag in your project's Cargo.toml: >>> [dependencies] >>> apache-spark-connect = { git = >>> "https://github.com/apache/spark-connect-rust";, tag = "v4.2.0-rc1" } >>> >>> # B) Or build and run the crates' own tests from the tag: >>> git clone --branch v4.2.0-rc1 https://github.com/apache/spark-connect-rust >>> cd spark-connect-rust >>> cargo build --release >>> cargo test >>> >>> HIGHLIGHTS >>> >>> - *Drop-in PySpark Connect Python API*, verified against real >>> PySpark 4.2.0 two >>> ways: the official PySpark test suite runs against this client, and a >>> behavioral API-parity sweep compares the Python API surface >>> (functions and >>> DataFrame / Column / SparkSession / GroupedData / Window methods) >>> call-by-call >>> to PySpark. >>> - Spark Connect Rust API *full* coverage, including *Rust UDF*s >>> (docs: https://apache.github.io/spark-connect-rust/). >>> - *Native Rust transport* via tonic; cloudpickle vendored to match >>> upstream, so >>> no grpcio/py4j at runtime. and *native type conversion.* >>> >>> WHY DOES THE VERSION START AT 4.2.0? >>> >>> Even though this is the first release of the rewrite, the version starts >>> at >>> 4.2.0 to track the Apache Spark / PySpark version it targets. This (1) >>> removes >>> any ambiguity about which Spark version a given client is compatible >>> with, and >>> (2) lets PySpark users pip install pyspark-client-rust== exactly >>> as they would pyspark-client, keeping the same versioning across both. >>> >>> Thanks, >>> Hyukjin Kwon >>> >>>
