So I guess my question is are we planning to distribute it as a Python package? If so let's dig in some more first and try and clarify in docs the difference between the two different official connect clients first, if not then I think it doesn't matter (until we do).
On Sat, Aug 29, 2026 at 6:38 PM Hyukjin Kwon <[email protected]> wrote: > Hi all, > > Spinning this out of the RC1 vote thread. Alongside the Rust Spark Connect > client, > we now have pyspark-client-rust: a Python wrapper that installs the same > pyspark > surface as pyspark-client (existing PySpark Connect code runs unchanged), > but with > the native Rust client (tonic) as the engine instead of grpcio/py4j. It is > checked > against PySpark 4.2.0 (official test suite + call-by-call API parity), and > the Rust > crates work standalone too. > > I personally thought that it does not need a SPIP - it is a separate > feature/package, not a core > change - but it is user-facing, so I wanted a dedicated thread for any > thoughts. > Not blocking the RC1 vote. > > Architecture: https://apache.github.io/spark-connect-rust/architecture/ > Repo: https://github.com/apache/spark-connect-rust > > So the main question is, how do you guys like it? > > Thanks, > Hyukjin Kwon > -- Twitter: https://twitter.com/holdenkarau Fight Health Insurance: https://www.fighthealthinsurance.com/ <https://www.fighthealthinsurance.com/?q=hk_email> Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau Pronouns: she/her
