L. C. Hsieh created SPARK-58455:
-----------------------------------
Summary: SPIP: Spark Connect Gateway — Session Affinity, Auth, and
Multi-Tenancy for Spark Connect Servers
Key: SPARK-58455
URL: https://issues.apache.org/jira/browse/SPARK-58455
Project: Spark
Issue Type: New Feature
Components: Connect
Affects Versions: 4.3.0
Reporter: L. C. Hsieh
Assignee: L. C. Hsieh
This SPIP proposes an official gateway component for Spark Connect deployments.
Problem
A Spark Connect client has to know the address of one specific server and stay
with it for the lifetime of its session, because session state lives in that
server's driver. As soon as a deployment runs more than one Spark Connect
server — per-team servers, capacity, redundancy — there is no built-in way to
put a single address in front of the fleet. A generic L7 load balancer cannot
do this: the routing key (session_id) lives inside the request protobuf, not
in headers. Today every team writes its own proxy glue.
Proposal
The Spark Connect Gateway is a standalone, wire-compatible gRPC proxy in front
of a pool of Spark Connect servers, providing:
* Session-affinity routing keyed on (tenant, user, session_id), round-robin
placement for new sessions
* Backend discovery via static lists or Kubernetes Endpoints watching
(composes with plain Deployments, apache/spark-kubernetes-operator, and the
Kubeflow Spark Operator)
* Multi-replica HA via a shared affinity store (Redis reference backend behind
a pluggable interface; single-replica deployments need no external store)
* Authentication (none / static tokens / JWT / OIDC), identity-based tenant
resolution, and per-tenant backend pools
* Per-tenant rate limiting, structured audit logging, Prometheus metrics,
OpenTelemetry tracing
* Helm chart for deployment
Clients are untouched: the connection string points at the gateway and nothing
else changes. The proposal adds nothing to and changes nothing in the Spark
Connect protocol.
*Non-goals*: driver lifecycle management (stays with the operator /
provisioner), distributing one query across drivers, session-state replication
across drivers, TLS termination (delegated to ingress / service mesh).
SPIP document:
https://docs.google.com/document/d/16RNnRJjSDBSlcgeBjVz_Mv-SkqvH33F9tIsR1Ii_P3c/edit?usp=sharing
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
