nsivabalan opened a new issue, #20605:
URL: https://github.com/apache/hudi/issues/20605

   ### Feature Description
   
   **What the feature achieves:**
   The high-level intent is to add authentication support to the Spark Connect 
endpoint. The exact design is open: this issue asks for research and a proposal 
first (see below), not an implementation straight away.
   
   Concretely: add client authentication and TLS to the Helm-deployed Spark 
Connect endpoint (`hudi-lakehouse/charts/hudi-spark-connect`), so it can run 
outside a fully trusted network.
   
   Today the chart README says: "The service has no authentication or TLS. Keep 
it on a trusted network: any client that can connect can read and write using 
the server's storage credentials." In Kubernetes executor mode, that trust 
extends to the driver's ServiceAccount, which can create pods.
   
   **Why this feature is needed:**
   Every client path — PySpark Connect, Foundry ADBC (Superset, Zeppelin), and 
`hudi-agent-gateway` — connects as an anonymous, fully privileged user. This is 
the main blocker to running the endpoint anywhere but a locked-down namespace, 
and was explicitly deferred out of the first patches in #20234.
   
   ### User Experience
   
   **How users will use this feature:**
   - Configuration changes needed:
     - Chart values to enable authentication (e.g. a shared bearer token from 
an existing Secret) and TLS (certificate from an existing Secret), off by 
default for local-dev.
     - Matching client settings: PySpark Connect connection string (`token`, 
`use_ssl`), Foundry ADBC URI (`auth_type`, `tls`), the Superset/Zeppelin 
examples, and the gateway's `spark-connect` backend.
   - API changes: none to Hudi; chart values and client configuration only.
   - Usage examples: `quickstart.sh --spark-connect` should keep working 
unchanged; a documented values snippet turns on token + TLS.
   
   **Questions the proposal should answer (starting points, not a fixed plan):**
   1. Pick the mechanism: Spark Connect's server-side gRPC interceptor hook vs. 
terminating TLS and checking a token in a proxy/sidecar in front of the server. 
Record the choice and why.
   2. Confirm each client can authenticate: PySpark Connect, Foundry ADBC 0.2.1 
(which `auth_type` values it supports), the Superset dialect (note it currently 
requires `auth_type=none` and rejects credentials), Zeppelin, and the gateway.
   3. Chart support, secrets handling, and offline chart tests; extend the live 
workflow to run with auth on.
   4. Update the README trust-model section and the 2.0 known-limitations notes.
   
   **Out of scope:** per-user identity, authorization or multi-tenant isolation 
(the server still uses one set of storage credentials), and Spark UI / history 
server access control. These stay follow-ups.
   
   ### Hudi RFC Requirements
   
   **RFC PR link:** (if applicable)
   
   **Why RFC is/isn't needed:**
   - Does this change public interfaces/APIs? No
   - Does this change storage format? No
   - Justification: deployment and client-configuration change in 
`hudi-lakehouse` only.
   
   ---
   
   Parent: #20234
   
   @cshuo could you take this up? Before implementing, please research the 
options and come back with a short proposal on this issue: the mechanism you 
recommend, which clients it works with, what changes in the chart and client 
configuration, and what stays out of scope. We'd like the work to land within 
2.0 so the Spark Connect endpoint doesn't ship without any authentication. 
Please flag early if it doesn't fit the 2.0 timeline.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to