I can't say that the proposal to use Rust or any other non-JVM language for something that feels like a new core component of Spark sits easy with me, That is not to say that I am firmly opposed or can't be persuaded, but just that I am now uneasy. The arguments in the SPIP for using Rust don't strike me as particularly strong, being closer to "I/we like Rust" rather than a more persuasive compare-and-contrast of the relative merits for this component written and maintained in Rust, Scala or Java. The idea that the basic design of this gateway should be independent enough that it could conceivably be written in any of several languages without the end users caring much or at all is, I think, a good one, but that on its own doesn't persuade me that it shouldn't be written in the same version of Scala as the rest of core Spark, or in Scala 3, which has considerable advantages over Scala 2 for those components that are independent enough from the Scala 2 Spark core.
Also, this new gateway does not strike me as analogous to spark-connect-swift or spark-connect-go or spark-connect-rust or some of the nascent talk about something like spark-connect-scala3 since those are all efforts to provide functionality for particular language-using subsets of the community, not for more or less everyone. No conclusions from me yet, just wanting to see more discussion before deciding. On Fri, Jul 31, 2026 at 11:54 PM vaquar khan <[email protected]> wrote: > > +1 > > A community-maintained implementation already exists for this functionality. > I have added a few comments to the SIP document outlining how this prior art > relates to the current proposal. > > Regards, > Viquar Khan > > On Fri, 31 Jul 2026 at 23:37, Dongjoon Hyun <[email protected]> wrote: >> >> For the record, Apache Spark community currently maintains seven GitHub >> repositories like the following: >> >> - https://github.com/apache/spark (Last >> Commit: Today) >> - https://github.com/apache/spark-connect-swift (Last Commit: >> Today) >> - https://github.com/apache/spark-kubernetes-operator (Last Commit: 3 days >> ago) >> - https://github.com/apache/spark-website (Last Commit >> 5 days ago) >> - https://github.com/apache/spark-docker (Last >> Commit: 1 week ago) >> - https://github.com/apache/spark-connect-rust (Last Commit: 2 >> months ago) >> - https://github.com/apache/spark-connect-go (Last Commit: 11 >> months ago) >> >> Since there is no bug-free software, I hope the new proposed repository >> builds a new sub-community by being one of active Apache Spark repositories >> instead of one-time snapshot drop-off. >> >> - https://github.com/apache/spark-connect-gateway >> >> Sincerely, >> Dongjoon. >> >> On 2026/08/01 04:30:05 John Zhuge wrote: >> > +1 Thanks. >> > >> > On Fri, Jul 31, 2026 at 8:41 PM Dongjoon Hyun <[email protected]> wrote: >> > >> > > Thank you, Liang-Chi. >> > > >> > > Since it's written in Rust like "apache/spark-connect-rust", >> > > "apache/spark-connect-gateway" proposal sounds good to me. >> > > >> > > Where is the reference implementation? I didn't find any GitHub link in >> > > SPARK-58455 and SPIP. >> > > >> > > Dongjoon. >> > > >> > > On 2026/07/31 22:49:44 Chao Sun wrote: >> > > > +1 >> > > > >> > > > On Fri, Jul 31, 2026 at 1:26 PM DB Tsai <[email protected]> wrote: >> > > > >> > > > > +1 >> > > > > >> > > > > An open-source reference gateway that makes it easier to run Spark >> > > Connect >> > > > > at scale with session affinity, authentication, high availability, >> > > > > multi-tenancy, rate limiting, and audit logging built in is great for >> > > > > production adoption of Spark Connect. >> > > > > >> > > > > >> > > > > DB Tsai | https://www.dbtsai.com/ | PGP 0x9FB9FAA3 >> > > > > >> > > > > On Thursday, July 30th, 2026 at 3:43 PM, L. C. Hsieh >> > > > > <[email protected] >> > > > >> > > > > wrote: >> > > > > >> > > > > Hi all, >> > > > > >> > > > > I'd like to start a discussion on a SPIP proposing an official >> > > > > gateway >> > > > > component for Spark Connect deployments. >> > > > > >> > > > > Problem: A Spark Connect client has to know the address of one >> > > > > specific >> > > > > server and stay with it for the lifetime of its session, because >> > > session >> > > > > state lives in that server's driver. As soon as a deployment runs >> > > > > more >> > > than >> > > > > one Spark Connect server — per-team servers, capacity, redundancy — >> > > there >> > > > > is no built-in way to put a single address in front of the fleet. A >> > > generic >> > > > > load balancer can't do this either: the routing key (session_id) >> > > > > lives >> > > > > inside the request protobuf, not in headers. So today every team >> > > writes its >> > > > > own proxy glue, and everyone solves the same problem from scratch. >> > > > > >> > > > > Proposal: The Spark Connect Gateway is a standalone, wire-compatible >> > > gRPC >> > > > > proxy that sits in front of a pool of Spark Connect servers. >> > > > > >> > > > > Clients are untouched: the connection string points at the gateway >> > > > > and >> > > > > everything else is unchanged. The proposal adds nothing to and >> > > > > changes >> > > > > nothing in the Spark Connect protocol. >> > > > > >> > > > > SPIP doc: >> > > > > >> > > https://docs.google.com/document/d/16RNnRJjSDBSlcgeBjVz_Mv-SkqvH33F9tIsR1Ii_P3c/edit?tab=t.0#heading=h.221xxblzqzu >> > > > > JIRA: https://issues.apache.org/jira/browse/SPARK-58455 >> > > > > >> > > > > Feedback is very welcome. >> > > > > >> > > > > Thanks, >> > > > > Liang-Chi >> > > > > >> > > > > >> > > > > >> > > > >> > > >> > > --------------------------------------------------------------------- >> > > To unsubscribe e-mail: [email protected] >> > > >> > > >> > >> > -- >> > John Zhuge >> > >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: [email protected] >> --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
