*+1* A community-maintained implementation already exists for this functionality. I have added a few comments to the SIP document outlining how this prior art relates to the current proposal. Regards, Viquar Khan
On Fri, 31 Jul 2026 at 23:37, Dongjoon Hyun <[email protected]> wrote: > For the record, Apache Spark community currently maintains seven GitHub > repositories like the following: > > - https://github.com/apache/spark > (Last Commit: Today) > - https://github.com/apache/spark-connect-swift (Last Commit: > Today) > - https://github.com/apache/spark-kubernetes-operator (Last Commit: 3 > days ago) > - https://github.com/apache/spark-website (Last > Commit 5 days ago) > - https://github.com/apache/spark-docker (Last > Commit: 1 week ago) > - https://github.com/apache/spark-connect-rust (Last Commit: > 2 months ago) > - https://github.com/apache/spark-connect-go (Last Commit: > 11 months ago) > > Since there is no bug-free software, I hope the new proposed repository > builds a new sub-community by being one of active Apache Spark repositories > instead of one-time snapshot drop-off. > > - https://github.com/apache/spark-connect-gateway > > Sincerely, > Dongjoon. > > On 2026/08/01 04:30:05 John Zhuge wrote: > > +1 Thanks. > > > > On Fri, Jul 31, 2026 at 8:41 PM Dongjoon Hyun <[email protected]> > wrote: > > > > > Thank you, Liang-Chi. > > > > > > Since it's written in Rust like "apache/spark-connect-rust", > > > "apache/spark-connect-gateway" proposal sounds good to me. > > > > > > Where is the reference implementation? I didn't find any GitHub link in > > > SPARK-58455 and SPIP. > > > > > > Dongjoon. > > > > > > On 2026/07/31 22:49:44 Chao Sun wrote: > > > > +1 > > > > > > > > On Fri, Jul 31, 2026 at 1:26 PM DB Tsai <[email protected]> wrote: > > > > > > > > > +1 > > > > > > > > > > An open-source reference gateway that makes it easier to run Spark > > > Connect > > > > > at scale with session affinity, authentication, high availability, > > > > > multi-tenancy, rate limiting, and audit logging built in is great > for > > > > > production adoption of Spark Connect. > > > > > > > > > > > > > > > DB Tsai | https://www.dbtsai.com/ | PGP 0x9FB9FAA3 > > > > > > > > > > On Thursday, July 30th, 2026 at 3:43 PM, L. C. Hsieh < > [email protected] > > > > > > > > > wrote: > > > > > > > > > > Hi all, > > > > > > > > > > I'd like to start a discussion on a SPIP proposing an official > gateway > > > > > component for Spark Connect deployments. > > > > > > > > > > Problem: A Spark Connect client has to know the address of one > specific > > > > > server and stay with it for the lifetime of its session, because > > > session > > > > > state lives in that server's driver. As soon as a deployment runs > more > > > than > > > > > one Spark Connect server — per-team servers, capacity, redundancy — > > > there > > > > > is no built-in way to put a single address in front of the fleet. A > > > generic > > > > > load balancer can't do this either: the routing key (session_id) > lives > > > > > inside the request protobuf, not in headers. So today every team > > > writes its > > > > > own proxy glue, and everyone solves the same problem from scratch. > > > > > > > > > > Proposal: The Spark Connect Gateway is a standalone, > wire-compatible > > > gRPC > > > > > proxy that sits in front of a pool of Spark Connect servers. > > > > > > > > > > Clients are untouched: the connection string points at the gateway > and > > > > > everything else is unchanged. The proposal adds nothing to and > changes > > > > > nothing in the Spark Connect protocol. > > > > > > > > > > SPIP doc: > > > > > > > > > https://docs.google.com/document/d/16RNnRJjSDBSlcgeBjVz_Mv-SkqvH33F9tIsR1Ii_P3c/edit?tab=t.0#heading=h.221xxblzqzu > > > > > JIRA: https://issues.apache.org/jira/browse/SPARK-58455 > > > > > > > > > > Feedback is very welcome. > > > > > > > > > > Thanks, > > > > > Liang-Chi > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe e-mail: [email protected] > > > > > > > > > > -- > > John Zhuge > > > > --------------------------------------------------------------------- > To unsubscribe e-mail: [email protected] > >
