cshuo commented on issue #20605: URL: https://github.com/apache/hudi/issues/20605#issuecomment-7002357990
I propose using **Spark Connect’s built-in bearer-token authentication**, with TLS terminated by a gRPC sidecar or an existing gRPC gateway. Spark 4.1.3 already checks `Authorization: Bearer <token>` and returns `UNAUTHENTICATED` for a missing or invalid token. This gives every Spark client path the same authentication check and avoids maintaining a custom interceptor or token validation rules in the proxy. [Spark interceptor](https://github.com/apache/spark/blob/v4.1.3/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/PreSharedKeyAuthenticationInterceptor.scala#L22-L43) **Chart and secrets.** The chart would populate `SPARK_CONNECT_AUTHENTICATE_TOKEN` from an existing Kubernetes Secret. With the sidecar enabled, the chart would mount an existing TLS Secret in the sidecar, bind Spark to loopback, and expose only the sidecar’s TLS port through the Service. Operators with an existing gRPC gateway could terminate TLS there and forward the `authorization` header to a private Spark Service over a trusted or encrypted backend connection. Authentication and TLS would be opt-in so `quickstart.sh --spark-connect` continues to work with its defaults. [Spark token configuration](https://github.com/apache/spark/blob/v4.1.3/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala#L352-L357) **Client compatibility.** I’d verify the following before calling the feature ready for 2.0. - **Foundry ADBC 0.2.1** is the first check because both Superset and Zeppelin depend on it. The pinned driver’s tests cover `auth_type=none` and `auth_type=token` for `api=connect` and show how to pass credentials through ADBC `db_kwargs`. The successful upstream TLS test disables certificate verification. I’d verify that `SELECT 1` succeeds against our Helm deployment with `auth_type=token`, `tls=true`, and certificate verification enabled against a trusted CA. I’d flag any interoperability issue early so we can assess its impact on 2.0. [Foundry 0.2.1 authentication tests](https://github.com/adbc-drivers/spark/blob/go/v0.2.1/go/validation/tests/test_auth.py) - **Superset and Zeppelin** each need a live query with credentials supplied outside the saved URI or notebook. One detail from the current code: the Superset dialect defaults to `auth_type=none`, forwards an explicit `auth_type`, and rejects username/password in the URL. The Zeppelin live test explicitly requires `auth_type=none` because it stores the URI in the notebook. I’d verify passing Superset’s stored credential through `db_kwargs` and loading Zeppelin’s token from the interpreter environment at runtime. - **PySpark Connect and `hudi-agent-gateway`** each need an end-to-end query with `token`, `use_ssl=true`, and a trusted certificate. The chart’s readiness probe also opens a PySpark session, so I’d verify that it authenticates when the feature is enabled. **Tests and docs.** I’d add offline Helm tests for the default, sidecar, and external-gateway configurations, covering Secret references, bind addresses, and exposed ports. I’d extend the existing live workflow with an auth-enabled run that verifies a successful query, `UNAUTHENTICATED` for missing and incorrect tokens, and TLS failure for an untrusted certificate. The chart README would include a working token + TLS values example and explain the trust model: every token holder still uses the server’s storage credentials. The 2.0 known-limitations notes would list per-user identity, authorization, multi-tenant isolation, and access control for the Spark UI and History Server as follow-up work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
