[
https://issues.apache.org/jira/browse/SPARK-58455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh updated SPARK-58455:
--------------------------------
Description:
This SPIP proposes an official gateway component for Spark Connect deployments.
Problem:
A Spark Connect client has to know the address of one specific server and stay
with it for the lifetime of its session, because session state lives in that
server's driver. As soon as a deployment runs more than one Spark Connect
server — per-team servers, capacity, redundancy — there is no built-in way to
put a single address in front of the fleet. A generic load balancer cannot
do this: the routing key (session_id) lives inside the request protobuf, not
in headers. Today every team writes its own proxy glue.
Proposal:
The Spark Connect Gateway is a standalone, wire-compatible gRPC proxy in front
of a pool of Spark Connect servers.
Clients are untouched: the connection string points at the gateway and nothing
else changes. The proposal adds nothing to and changes nothing in the Spark
Connect protocol.
SPIP document:
https://docs.google.com/document/d/16RNnRJjSDBSlcgeBjVz_Mv-SkqvH33F9tIsR1Ii_P3c/edit?usp=sharing
was:
This SPIP proposes an official gateway component for Spark Connect deployments.
Problem
A Spark Connect client has to know the address of one specific server and stay
with it for the lifetime of its session, because session state lives in that
server's driver. As soon as a deployment runs more than one Spark Connect
server — per-team servers, capacity, redundancy — there is no built-in way to
put a single address in front of the fleet. A generic load balancer cannot
do this: the routing key (session_id) lives inside the request protobuf, not
in headers. Today every team writes its own proxy glue.
Proposal
The Spark Connect Gateway is a standalone, wire-compatible gRPC proxy in front
of a pool of Spark Connect servers, providing:
* Session-affinity routing keyed on (tenant, user, session_id), round-robin
placement for new sessions
* Backend discovery via static lists or Kubernetes Endpoints watching
(composes with plain Deployments, apache/spark-kubernetes-operator, and the
Kubeflow Spark Operator)
* Multi-replica HA via a shared affinity store (Redis reference backend behind
a pluggable interface; single-replica deployments need no external store)
* Authentication (none / static tokens / JWT / OIDC), identity-based tenant
resolution, and per-tenant backend pools
* Per-tenant rate limiting, structured audit logging, Prometheus metrics,
OpenTelemetry tracing
* Helm chart for deployment
Clients are untouched: the connection string points at the gateway and nothing
else changes. The proposal adds nothing to and changes nothing in the Spark
Connect protocol.
*Non-goals*: driver lifecycle management (stays with the operator /
provisioner), distributing one query across drivers, session-state replication
across drivers, TLS termination (delegated to ingress / service mesh).
SPIP document:
https://docs.google.com/document/d/16RNnRJjSDBSlcgeBjVz_Mv-SkqvH33F9tIsR1Ii_P3c/edit?usp=sharing
> SPIP: Spark Connect Gateway — Session Affinity, Auth, and Multi-Tenancy for
> Spark Connect Servers
> -------------------------------------------------------------------------------------------------
>
> Key: SPARK-58455
> URL: https://issues.apache.org/jira/browse/SPARK-58455
> Project: Spark
> Issue Type: New Feature
> Components: Connect
> Affects Versions: 4.3.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
> Labels: SPIP
>
> This SPIP proposes an official gateway component for Spark Connect
> deployments.
> Problem:
> A Spark Connect client has to know the address of one specific server and
> stay
> with it for the lifetime of its session, because session state lives in that
> server's driver. As soon as a deployment runs more than one Spark Connect
> server — per-team servers, capacity, redundancy — there is no built-in way
> to
> put a single address in front of the fleet. A generic load balancer cannot
> do this: the routing key (session_id) lives inside the request protobuf, not
> in headers. Today every team writes its own proxy glue.
> Proposal:
> The Spark Connect Gateway is a standalone, wire-compatible gRPC proxy in
> front
> of a pool of Spark Connect servers.
> Clients are untouched: the connection string points at the gateway and
> nothing
> else changes. The proposal adds nothing to and changes nothing in the Spark
> Connect protocol.
> SPIP document:
> https://docs.google.com/document/d/16RNnRJjSDBSlcgeBjVz_Mv-SkqvH33F9tIsR1Ii_P3c/edit?usp=sharing
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
