I agree with the idea that we should call out and make it clear. Yeah we also have Spark Classic and Connect (also I personally don't thing that it was a bad idea.. it had good and bad) but this is a bit different because it supports exactly the same API surface although we should still be some bugs and things to fix.
I think I need to make this crystal clear before cutting another RC. > Plus if you try to support multiple versions of a Spark server, including versions older than 4.2, you would need to support both the native client and rust client simultaneously. This is a great call, and I do plan to do this by setting up the CI just exactly like how we do in the original Spark Connect clients. On Mon, Aug 31, 2026 at 12:44 AM Adam Binford <[email protected]> wrote: > Love the idea and can't wait to try it out. My biggest question/concern is > if the "pyspark" package name should be reused or it it should be scoped > under something like "pyspark_rust". I get the idea of a drop in > replacement, but having had to deal with enough conflicts for people > between pyspark and pyspark-client, this seems like it would just cause > even more confusion. Users may not know which library they are using, and > hit issues with the rust client and accidentally report them as native > spark client bugs. Plus if you try to support multiple versions of a Spark > server, including versions older than 4.2, you would need to support both > the native client and rust client simultaneously. > > Adam > > > > > > On Sat, Aug 29, 2026, 9:53 PM Hyukjin Kwon <[email protected]> wrote: > >> Yeah I was thinking about pyspark-client-rust. It is just same API >> background but quite fast for small quries. Let me kill the release vote on >> my own first and clarify it and start next RC >> >> On Sun, Aug 30, 2026 at 10:50 AM Holden Karau <[email protected]> >> wrote: >> >>> So I guess my question is are we planning to distribute it as a Python >>> package? If so let's dig in some more first and try and clarify in docs the >>> difference between the two different official connect clients first, if not >>> then I think it doesn't matter (until we do). >>> >>> On Sat, Aug 29, 2026 at 6:38 PM Hyukjin Kwon <[email protected]> >>> wrote: >>> >>>> Hi all, >>>> >>>> Spinning this out of the RC1 vote thread. Alongside the Rust Spark >>>> Connect client, >>>> we now have pyspark-client-rust: a Python wrapper that installs the >>>> same pyspark >>>> surface as pyspark-client (existing PySpark Connect code runs >>>> unchanged), but with >>>> the native Rust client (tonic) as the engine instead of grpcio/py4j. It >>>> is checked >>>> against PySpark 4.2.0 (official test suite + call-by-call API parity), >>>> and the Rust >>>> crates work standalone too. >>>> >>>> I personally thought that it does not need a SPIP - it is a separate >>>> feature/package, not a core >>>> change - but it is user-facing, so I wanted a dedicated thread for any >>>> thoughts. >>>> Not blocking the RC1 vote. >>>> >>>> Architecture: https://apache.github.io/spark-connect-rust/architecture/ >>>> Repo: https://github.com/apache/spark-connect-rust >>>> >>>> So the main question is, how do you guys like it? >>>> >>>> Thanks, >>>> Hyukjin Kwon >>>> >>> >>> >>> -- >>> Twitter: https://twitter.com/holdenkarau >>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>> <https://www.fighthealthinsurance.com/?q=hk_email> >>> Books (Learning Spark, High Performance Spark, etc.): >>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>> Pronouns: she/her >>> >>
