turboFei opened a new pull request, #7510:
URL: https://github.com/apache/kyuubi/pull/7510
## Why are the changes needed?
Apache Spark 3.4 introduced Spark Connect — a gRPC-based client-server
protocol using Protocol Buffers that decouples clients from the Spark driver.
PySpark users can now connect via `spark.remote("sc://host:port/")` instead of
embedding a local SparkSession. Currently Kyuubi has no gRPC listener and
cannot accept Spark Connect clients, meaning PySpark thin clients and any
non-JVM Spark Connect clients (Go, Rust) cannot benefit from Kyuubi's
multi-tenancy, session pooling, and access control.
This PR adds planning artifacts (proposal, design, specs, tasks) for
implementing Spark Connect support in Kyuubi as a multi-tenant gRPC proxy
gateway.
## What Changes
This is a **planning/proposal PR** containing `openspec/` artifacts only —
no production code is changed. The design proposes:
- A new profile-gated Maven module `kyuubi-spark-connect` exposing a
`SparkConnectService` gRPC endpoint (default port 15002).
- **Proxy architecture**: Kyuubi authenticates clients and routes Spark
Connect protobuf streams to a Spark engine running in `--spark-connect` mode.
Kyuubi does NOT embed a SparkSession (preserving the server/engine module
boundary).
- Session lifecycle mapped onto Kyuubi's existing `SessionManager`; per-user
engine limits and idle timeouts apply unchanged.
- Engine gRPC port discovered via ZooKeeper node attribute or Kubernetes pod
label.
- Bearer-token auth via existing `AuthenticationProvider` chain; user
identity overrides client-supplied `UserContext.user_name`.
- Feature off by default (`kyuubi.frontend.spark.connect.enabled=false`);
additive and non-breaking.
### Artifacts included
| File | Description |
|---|---|
| `openspec/changes/kyuubi-spark-connect-grpc-server/proposal.md` |
Motivation, capability list, impact |
| `openspec/changes/kyuubi-spark-connect-grpc-server/design.md` |
Architecture, key decisions, risks, open questions |
|
`openspec/changes/kyuubi-spark-connect-grpc-server/specs/spark-connect-frontend/spec.md`
| gRPC server lifecycle, RPC surface, TLS, config keys |
|
`openspec/changes/kyuubi-spark-connect-grpc-server/specs/spark-connect-session/spec.md`
| Session lifecycle, user isolation, event logging |
|
`openspec/changes/kyuubi-spark-connect-grpc-server/specs/spark-connect-engine-proxy/spec.md`
| Port discovery, transparent forwarding, failure handling |
|
`openspec/changes/kyuubi-spark-connect-grpc-server/specs/spark-connect-auth/spec.md`
| Auth interceptor, identity propagation, authz checks |
| `openspec/changes/kyuubi-spark-connect-grpc-server/tasks.md` | 37
implementation tasks across 10 groups |
## How was this patch tested?
This PR contains planning artifacts only; no code to test. Implementation
PRs will include unit and integration tests per `tasks.md` section 9.
## Was this patch authored or co-authored using generative AI tooling?
Assisted-by: Claude:claude-sonnet-4-6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
