alexandrefimov opened a new pull request, #7670: URL: https://github.com/apache/kyuubi/pull/7670
### Why are the changes needed? Part of #7511, the first slice agreed there: Spark Connect running inside the Spark SQL engine, next to the Thrift frontend, so that Spark Connect clients reach a Kyuubi-managed engine — the same `SparkContext`, the same session extensions, and so the same authorization rules a JDBC client gets. This PR starts the service and stops there. The endpoint is not advertised in discovery yet, the engine's idle and shutdown checks do not count Connect sessions yet, and the token still travels on the command line; those are the next PRs on this issue, and the new deployment page lists them as limitations so the feature is not mistaken for finished. Connect is started by configuration alone: the driver plugin goes into `spark.plugins` and the bound endpoint is read back reflectively. A compile-time reference does not build in the engine module — `io.grpc` is relocated to `org.apache.kyuubi.shade.io.grpc` there and the Scala signatures of `SparkConnectService` carry `io.grpc` types — which also keeps the default `spark-3.5` build free of Connect classes. Which engines serve Connect is decided in `SparkProcessBuilder`, where the values are the server's own. `kyuubi.engine.spark.connect.enabled` is immutable, but the engine rebuilds its `KyuubiConf` from every `spark.kyuubi.*` entry it is handed, so the prefixed copy of that key is dropped from the engine's command line. An engine that would not run as the session user — any share level but `USER`, or `kyuubi.engine.doAs.enabled=false` — starts without Connect rather than failing, because the share level is the session's own choice. That matters because a plan submitted over Connect runs as the user the engine runs as and not as the client's `user_id`: on 4.0.3 with the Ranger extension, a client claiming `user_id=bob` against an engine launched for `testalice` is denied as `testalice`. With Connect on, the engine does refuse to start when the runtime cannot authenticate its clients — Spark 3.5 serves Connect with no authentication at all, `spark.connect.authenticate.token` arrived in 4.0 — or when no token is configured. An endpoint that authenticates nobody is worse than no endpoint. The Spark 4.0 requirement is a capability probe (`Connect.getAuthenticateToken` is present) rather than a version comparison, following the "feature detection, not version strings" rule in `AGENTS.md`. ### How was this patch tested? `SparkConnectLauncherSuite` covers the configuration the engine writes (the plugin appended once and only once, an ephemeral port unless the deployment picked one, deployment values kept), each of the three refusals, and the reflection itself against stand-in classes shaped like Spark's. `SparkProcessBuilderSuite` covers the server-side decision, including a session trying to turn Connect on through `spark.kyuubi.engine.spark.connect.enabled`. No suite starts the real service: the engine module does not depend on `spark-connect`, and adding the client stubs is a problem of its own, since Spark 4.x builds them against protobuf 4.x while Kyuubi carries 3.25.8. So this commit was also run end to end against Spark 4.0.3 — `spark-4.0.3-bin-hadoop3`, server built with `-Pspark-4.0 -Pscala-2.13`, `spark.master=local[*]`, `USER` share level, beeline as the client: - feature on with a token configured: the engine logs `Spark Connect service is listening on sc://192.168.1.85:57214` — the port it reports is the one the service bound, read back through the reflection this PR adds — and JDBC queries keep working; - feature off at the server, client connecting with `?spark.kyuubi.engine.spark.connect.enabled=true`: the value does reach the session, `SET` returns `true`, while the engine's launch command carries no such config and Connect never starts; - feature on, session asking for `kyuubi.engine.share.level=CONNECTION`: the server logs `Not starting Spark Connect for this engine: it serves the session user only with kyuubi.engine.share.level=USER and kyuubi.engine.doAs.enabled=true, which are CONNECTION and true here`, and that engine starts without Connect; - feature on with no token: the engine refuses to start with `no Spark Connect token is configured. Set spark.connect.authenticate.token or the SPARK_CONNECT_AUTHENTICATE_TOKEN environment variable so that clients are authenticated.` The Spark 3.5 case — Connect without authentication — is covered by the suite only; I did not build a 3.5 runtime for it. ### Was this patch authored or co-authored using generative AI tooling? Assisted-by: Claude Opus 5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
