Thank Dongjoon for putting this on the record — the variance in your list shows the risk is real, and it's the right bar to hold a new repository to.
On the commitment side: this isn't a finished artifact being handed over. I intend to keep maintaining and developing it actively, as the commit history over the past months reflects, and I'll treat issue and review responsiveness on the new repository as part of that commitment — "no bug-free software" cuts both ways, and being there when the bugs arrive is the point. I'd also hope the deployment-facing nature of the component draws operators into filing issues and patches over time, as has happened for spark-kubernetes-operator. And if any committers or contributors in this thread are interested in being involved from the start, I'd genuinely welcome that — a sub-community is easier to build from the first release than retrofitted later. On Fri, Jul 31, 2026 at 9:37 PM Dongjoon Hyun <[email protected]> wrote: > For the record, Apache Spark community currently maintains seven GitHub > repositories like the following: > > - https://github.com/apache/spark > (Last Commit: Today) > - https://github.com/apache/spark-connect-swift (Last Commit: > Today) > - https://github.com/apache/spark-kubernetes-operator (Last Commit: 3 > days ago) > - https://github.com/apache/spark-website (Last > Commit 5 days ago) > - https://github.com/apache/spark-docker (Last > Commit: 1 week ago) > - https://github.com/apache/spark-connect-rust (Last Commit: > 2 months ago) > - https://github.com/apache/spark-connect-go (Last Commit: > 11 months ago) > > Since there is no bug-free software, I hope the new proposed repository > builds a new sub-community by being one of active Apache Spark repositories > instead of one-time snapshot drop-off. > > - https://github.com/apache/spark-connect-gateway > > Sincerely, > Dongjoon. > > On 2026/08/01 04:30:05 John Zhuge wrote: > > +1 Thanks. > > > > On Fri, Jul 31, 2026 at 8:41 PM Dongjoon Hyun <[email protected]> > wrote: > > > > > Thank you, Liang-Chi. > > > > > > Since it's written in Rust like "apache/spark-connect-rust", > > > "apache/spark-connect-gateway" proposal sounds good to me. > > > > > > Where is the reference implementation? I didn't find any GitHub link in > > > SPARK-58455 and SPIP. > > > > > > Dongjoon. > > > > > > On 2026/07/31 22:49:44 Chao Sun wrote: > > > > +1 > > > > > > > > On Fri, Jul 31, 2026 at 1:26 PM DB Tsai <[email protected]> wrote: > > > > > > > > > +1 > > > > > > > > > > An open-source reference gateway that makes it easier to run Spark > > > Connect > > > > > at scale with session affinity, authentication, high availability, > > > > > multi-tenancy, rate limiting, and audit logging built in is great > for > > > > > production adoption of Spark Connect. > > > > > > > > > > > > > > > DB Tsai | https://www.dbtsai.com/ | PGP 0x9FB9FAA3 > > > > > > > > > > On Thursday, July 30th, 2026 at 3:43 PM, L. C. Hsieh < > [email protected] > > > > > > > > > wrote: > > > > > > > > > > Hi all, > > > > > > > > > > I'd like to start a discussion on a SPIP proposing an official > gateway > > > > > component for Spark Connect deployments. > > > > > > > > > > Problem: A Spark Connect client has to know the address of one > specific > > > > > server and stay with it for the lifetime of its session, because > > > session > > > > > state lives in that server's driver. As soon as a deployment runs > more > > > than > > > > > one Spark Connect server — per-team servers, capacity, redundancy — > > > there > > > > > is no built-in way to put a single address in front of the fleet. A > > > generic > > > > > load balancer can't do this either: the routing key (session_id) > lives > > > > > inside the request protobuf, not in headers. So today every team > > > writes its > > > > > own proxy glue, and everyone solves the same problem from scratch. > > > > > > > > > > Proposal: The Spark Connect Gateway is a standalone, > wire-compatible > > > gRPC > > > > > proxy that sits in front of a pool of Spark Connect servers. > > > > > > > > > > Clients are untouched: the connection string points at the gateway > and > > > > > everything else is unchanged. The proposal adds nothing to and > changes > > > > > nothing in the Spark Connect protocol. > > > > > > > > > > SPIP doc: > > > > > > > > > https://docs.google.com/document/d/16RNnRJjSDBSlcgeBjVz_Mv-SkqvH33F9tIsR1Ii_P3c/edit?tab=t.0#heading=h.221xxblzqzu > > > > > JIRA: https://issues.apache.org/jira/browse/SPARK-58455 > > > > > > > > > > Feedback is very welcome. > > > > > > > > > > Thanks, > > > > > Liang-Chi > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe e-mail: [email protected] > > > > > > > > > > -- > > John Zhuge > > > > --------------------------------------------------------------------- > To unsubscribe e-mail: [email protected] > >
