Ah that's fair. Let me raise a thread separately to see if there are more thoughts too but let me not pause this vote at the same time for now š
On Sun, 30 Aug 2026 at 10:28, Ćngel Ćlvarez Pascua < [email protected]> wrote: > +1 (non-binding) > > Thanks, that makes sense. I donāt expect there to be much objection. I > was mainly surprised that there hadnāt been a separate discussion before > the release vote, given the addition of the Python wrapper and new package. > > > > El dom, 30 ago 2026 a las 3:23, Hyukjin Kwon (<[email protected]>) > escribiĆ³: > >> It is the Rust Client, and the new one is a Python wrapper of the Rust >> client, which conceptually thought did not require an SPIP but rather a >> separate feature. If you feel like it has to be discussed separately, I can >> bring it up and pause the release vote for now. >> >> On Sun, 30 Aug 2026 at 10:20, Ćngel Ćlvarez Pascua < >> [email protected]> wrote: >> >>> Could you please point me to the previous dev@ discussion or SPIP for >>> this proposal? >>> >>> I found the earlier discussion about creating the Spark Connect Rust >>> client repository, but not one covering the ground-up Rust rewrite and the >>> new pyspark-client-rust package. Given the scope and user-facing nature >>> of these changes, I would have expected them to be discussed before >>> reaching the release vote, possibly through the SPIP process. >>> >>> El dom, 30 ago 2026 a las 0:37, Hyukjin Kwon (<[email protected]>) >>> escribiĆ³: >>> >>>> Please vote on releasing the following candidate as Apache Spark >>>> Connect Rust >>>> Client 4.2.0. >>>> >>>> This is the *first* release of the ground-up Rust rewrite of the Spark >>>> Connect >>>> Python client (published to PyPI as pyspark-client-rust): a drop-in >>>> replacement >>>> for pyspark-client <https://pypi.org/project/pyspark-client/>, backed >>>> by a native Rust core instead of py4j/grpcio. >>>> >>>> The vote is open until Thursday, September 4, 2026, 5:00 PM PDT and >>>> passes if a majority of +1 PMC votes are cast, with a minimum of >>>> 3 +1 votes. >>>> >>>> [ ] +1 Release this package as Apache Spark Connect Rust Client 4.2.0 >>>> [ ] -1 Do not release this package because ā¦ >>>> >>>> TAG: >>>> https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc1 >>>> (commit: f77df70) >>>> >>>> RELEASE FILES (wheels + sdist, built and attached by GitHub Actions): >>>> https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc1 >>>> >>>> DOCUMENTATION: >>>> https://apache.github.io/spark-connect-rust/ >>>> >>>> LIST OF ISSUES: >>>> https://issues.apache.org/jira/projects/SPARK/versions/12357484 >>>> HOW TO TRY IT ā PYTHON (pip) >>>> >>>> The release attaches abi3 wheels (Linux x86_64/aarch64, macOS >>>> universal2, >>>> Windows x86_64; CPython 3.9+) plus a source distribution. Grab the >>>> wheel for >>>> your platform from the release page and install it directly, e.g.: >>>> >>>> pip install >>>> https://github.com/apache/spark-connect-rust/releases/download/v4.2.0-rc1/pyspark_client_rust-4.2.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl >>>> >>>> Then use it exactly like PySpark. See if your existing workloads work >>>> with this. >>>> HOW TO TEST IT ā RUST (cargo) >>>> >>>> The Rust crates are intentionally NOT on crates.io for a release >>>> candidate >>>> (crates.io is immutable; the crates are published there only when the >>>> release is >>>> finalized). Test against the tagged source instead: >>>> >>>> # A) Pin a git dependency to the RC tag in your project's Cargo.toml: >>>> [dependencies] >>>> apache-spark-connect = { git = >>>> "https://github.com/apache/spark-connect-rust";, tag = "v4.2.0-rc1" } >>>> >>>> # B) Or build and run the crates' own tests from the tag: >>>> git clone --branch v4.2.0-rc1 >>>> https://github.com/apache/spark-connect-rust >>>> cd spark-connect-rust >>>> cargo build --release >>>> cargo test >>>> >>>> HIGHLIGHTS >>>> >>>> - *Drop-in PySpark Connect Python API*, verified against real >>>> PySpark 4.2.0 two >>>> ways: the official PySpark test suite runs against this client, and >>>> a >>>> behavioral API-parity sweep compares the Python API surface >>>> (functions and >>>> DataFrame / Column / SparkSession / GroupedData / Window methods) >>>> call-by-call >>>> to PySpark. >>>> - Spark Connect Rust API *full* coverage, including *Rust UDF*s >>>> (docs: https://apache.github.io/spark-connect-rust/). >>>> - *Native Rust transport* via tonic; cloudpickle vendored to match >>>> upstream, so >>>> no grpcio/py4j at runtime. and *native type conversion.* >>>> >>>> WHY DOES THE VERSION START AT 4.2.0? >>>> >>>> Even though this is the first release of the rewrite, the version >>>> starts at >>>> 4.2.0 to track the Apache Spark / PySpark version it targets. This (1) >>>> removes >>>> any ambiguity about which Spark version a given client is compatible >>>> with, and >>>> (2) lets PySpark users pip install pyspark-client-rust== exactly >>>> as they would pyspark-client, keeping the same versioning across both. >>>> >>>> Thanks, >>>> Hyukjin Kwon >>>> >>>>
