[
https://issues.apache.org/jira/browse/SPARK-58455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh updated SPARK-58455:
--------------------------------
Labels: SPIP (was: )
> SPIP: Spark Connect Gateway — Session Affinity, Auth, and Multi-Tenancy for
> Spark Connect Servers
> -------------------------------------------------------------------------------------------------
>
> Key: SPARK-58455
> URL: https://issues.apache.org/jira/browse/SPARK-58455
> Project: Spark
> Issue Type: New Feature
> Components: Connect
> Affects Versions: 4.3.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
> Labels: SPIP
>
> This SPIP proposes an official gateway component for Spark Connect
> deployments.
> Problem
> A Spark Connect client has to know the address of one specific server and
> stay
> with it for the lifetime of its session, because session state lives in that
> server's driver. As soon as a deployment runs more than one Spark Connect
> server — per-team servers, capacity, redundancy — there is no built-in way
> to
> put a single address in front of the fleet. A generic L7 load balancer
> cannot
> do this: the routing key (session_id) lives inside the request protobuf, not
> in headers. Today every team writes its own proxy glue.
> Proposal
> The Spark Connect Gateway is a standalone, wire-compatible gRPC proxy in
> front
> of a pool of Spark Connect servers, providing:
> * Session-affinity routing keyed on (tenant, user, session_id), round-robin
> placement for new sessions
> * Backend discovery via static lists or Kubernetes Endpoints watching
> (composes with plain Deployments, apache/spark-kubernetes-operator, and
> the
> Kubeflow Spark Operator)
> * Multi-replica HA via a shared affinity store (Redis reference backend
> behind
> a pluggable interface; single-replica deployments need no external store)
> * Authentication (none / static tokens / JWT / OIDC), identity-based tenant
> resolution, and per-tenant backend pools
> * Per-tenant rate limiting, structured audit logging, Prometheus metrics,
> OpenTelemetry tracing
> * Helm chart for deployment
> Clients are untouched: the connection string points at the gateway and
> nothing
> else changes. The proposal adds nothing to and changes nothing in the Spark
> Connect protocol.
> *Non-goals*: driver lifecycle management (stays with the operator /
> provisioner), distributing one query across drivers, session-state
> replication
> across drivers, TLS termination (delegated to ingress / service mesh).
> SPIP document:
> https://docs.google.com/document/d/16RNnRJjSDBSlcgeBjVz_Mv-SkqvH33F9tIsR1Ii_P3c/edit?usp=sharing
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
