turboFei commented on issue #7511: URL: https://github.com/apache/kyuubi/issues/7511#issuecomment-5207381059
Yes, this looks like the right direction and a reasonable first slice. The user-aware placement gap you identified is important for Kyuubi’s USER share level. Raising it in SPARK-58455 makes sense. Backend ownership should be matched against the identity verified by the gateway, rather than a client-supplied user field. For the initial Kyuubi slice, I suggest keeping the scope narrow: - target one Spark 4.x profile initially and keep the feature disabled by default; - start Spark Connect inside the Spark SQL engine, alongside the existing Thrift frontend, without introducing Spark runtime dependencies into `kyuubi-server`; - advertise the bound Connect endpoint and the minimum required backend metadata through the existing engine discovery mechanism; - start with the USER share level and document other share levels as follow-up work; - configure backend authentication when launching the engine, but do not publish authentication secrets in discovery metadata. One lifecycle detail should be included in the first slice: existing engine idle and graceful-shutdown logic tracks Thrift sessions and operations. A Connect session must keep the engine alive, and draining an engine must remove its Connect endpoint before waiting for active Connect work to finish. Otherwise an engine could be considered idle and terminated while it still has an active Connect session. YARN gateway-pool integration can be a separate follow-up after endpoint advertisement is established, since it likely needs a Kyuubi-aware discovery integration on the upstream gateway side. Validating the execution identity and Ranger behavior end to end is also the right approach. We should not assume that identity propagation through the gateway automatically preserves the existing authorization semantics. Please go ahead with a design for this first slice before starting the implementation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
