[
https://issues.apache.org/jira/browse/SPARK-58455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18100930#comment-18100930
]
Dongjoon Hyun commented on SPARK-58455:
---------------------------------------
Where is the current implementation repository, [~viirya]?
> SPIP: Spark Connect Gateway — Session Affinity, Auth, and Multi-Tenancy for
> Spark Connect Servers
> -------------------------------------------------------------------------------------------------
>
> Key: SPARK-58455
> URL: https://issues.apache.org/jira/browse/SPARK-58455
> Project: Spark
> Issue Type: New Feature
> Components: Connect
> Affects Versions: 4.3.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
> Labels: SPIP
>
> This SPIP proposes an official gateway component for Spark Connect
> deployments.
> Problem:
> A Spark Connect client has to know the address of one specific server and
> stay
> with it for the lifetime of its session, because session state lives in that
> server's driver. As soon as a deployment runs more than one Spark Connect
> server — per-team servers, capacity, redundancy — there is no built-in way
> to
> put a single address in front of the fleet. A generic load balancer cannot
> do this: the routing key (session_id) lives inside the request protobuf, not
> in headers. Today every team writes its own proxy glue.
> Proposal:
> The Spark Connect Gateway is a standalone, wire-compatible gRPC proxy in
> front
> of a pool of Spark Connect servers.
> Clients are untouched: the connection string points at the gateway and
> nothing
> else changes. The proposal adds nothing to and changes nothing in the Spark
> Connect protocol.
> SPIP document:
> https://docs.google.com/document/d/16RNnRJjSDBSlcgeBjVz_Mv-SkqvH33F9tIsR1Ii_P3c/edit?usp=sharing
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
