nsivabalan opened a new issue, #20605:
URL: https://github.com/apache/hudi/issues/20605
### Feature Description
**What the feature achieves:**
The high-level intent is to add authentication support to the Spark Connect
endpoint. The exact design is open: this issue asks for research and a proposal
first (see below), not an implementation straight away.
Concretely: add client authentication and TLS to the Helm-deployed Spark
Connect endpoint (`hudi-lakehouse/charts/hudi-spark-connect`), so it can run
outside a fully trusted network.
Today the chart README says: "The service has no authentication or TLS. Keep
it on a trusted network: any client that can connect can read and write using
the server's storage credentials." In Kubernetes executor mode, that trust
extends to the driver's ServiceAccount, which can create pods.
**Why this feature is needed:**
Every client path — PySpark Connect, Foundry ADBC (Superset, Zeppelin), and
`hudi-agent-gateway` — connects as an anonymous, fully privileged user. This is
the main blocker to running the endpoint anywhere but a locked-down namespace,
and was explicitly deferred out of the first patches in #20234.
### User Experience
**How users will use this feature:**
- Configuration changes needed:
- Chart values to enable authentication (e.g. a shared bearer token from
an existing Secret) and TLS (certificate from an existing Secret), off by
default for local-dev.
- Matching client settings: PySpark Connect connection string (`token`,
`use_ssl`), Foundry ADBC URI (`auth_type`, `tls`), the Superset/Zeppelin
examples, and the gateway's `spark-connect` backend.
- API changes: none to Hudi; chart values and client configuration only.
- Usage examples: `quickstart.sh --spark-connect` should keep working
unchanged; a documented values snippet turns on token + TLS.
**Questions the proposal should answer (starting points, not a fixed plan):**
1. Pick the mechanism: Spark Connect's server-side gRPC interceptor hook vs.
terminating TLS and checking a token in a proxy/sidecar in front of the server.
Record the choice and why.
2. Confirm each client can authenticate: PySpark Connect, Foundry ADBC 0.2.1
(which `auth_type` values it supports), the Superset dialect (note it currently
requires `auth_type=none` and rejects credentials), Zeppelin, and the gateway.
3. Chart support, secrets handling, and offline chart tests; extend the live
workflow to run with auth on.
4. Update the README trust-model section and the 2.0 known-limitations notes.
**Out of scope:** per-user identity, authorization or multi-tenant isolation
(the server still uses one set of storage credentials), and Spark UI / history
server access control. These stay follow-ups.
### Hudi RFC Requirements
**RFC PR link:** (if applicable)
**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? No
- Does this change storage format? No
- Justification: deployment and client-configuration change in
`hudi-lakehouse` only.
---
Parent: #20234
@cshuo could you take this up? Before implementing, please research the
options and come back with a short proposal on this issue: the mechanism you
recommend, which clients it works with, what changes in the chart and client
configuration, and what stays out of scope. We'd like the work to land within
2.0 so the Spark Connect endpoint doesn't ship without any authentication.
Please flag early if it doesn't fit the 2.0 timeline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]