+1
An open-source reference gateway that makes it easier to run Spark Connect at scale with session affinity, authentication, high availability, multi-tenancy, rate limiting, and audit logging built in is great for production adoption of Spark Connect. DB Tsai | https://www.dbtsai.com/ | PGP 0x9FB9FAA3 On Thursday, July 30th, 2026 at 3:43 PM, L. C. Hsieh <[email protected]> wrote: > Hi all, > > I'd like to start a discussion on a SPIP proposing an official gateway > component for Spark Connect deployments. > > Problem: A Spark Connect client has to know the address of one specific > server and stay with it for the lifetime of its session, because session > state lives in that server's driver. As soon as a deployment runs more than > one Spark Connect server — per-team servers, capacity, redundancy — there is > no built-in way to put a single address in front of the fleet. A generic load > balancer can't do this either: the routing key (session_id) lives inside the > request protobuf, not in headers. So today every team writes its own proxy > glue, and everyone solves the same problem from scratch. > > Proposal: The Spark Connect Gateway is a standalone, wire-compatible gRPC > proxy that sits in front of a pool of Spark Connect servers. > > Clients are untouched: the connection string points at the gateway and > everything else is unchanged. The proposal adds nothing to and changes > nothing in the Spark Connect protocol. > > SPIP doc: > https://docs.google.com/document/d/16RNnRJjSDBSlcgeBjVz_Mv-SkqvH33F9tIsR1Ii_P3c/edit?tab=t.0#heading=h.221xxblzqzu > JIRA: https://issues.apache.org/jira/browse/SPARK-58455 > > Feedback is very welcome. > > Thanks, > Liang-Chi >
publickey - [email protected] - 0x9FB9FAA3.asc
Description: application/pgp-keys
signature.asc
Description: OpenPGP digital signature
