alexandrefimov commented on issue #7511: URL: https://github.com/apache/kyuubi/issues/7511#issuecomment-5206946246
Thanks @turboFei. I went through SPARK-58455 and the reference implementation. I agree that Kyuubi should not build another Connect-aware proxy. I found one integration gap that matters for Kyuubi's default USER share level. The gateway includes user_id in the affinity key, but a new session is currently placed round-robin within a tenant pool ([crates/routing](https://github.com/viirya/spark-connect-gateway/blob/main/crates/routing/src/lib.rs)). If Kyuubi publishes several per-user engines into the same pool, there is nothing yet that prevents Alice's first session from being sent to Bob's engine. The routing interface already receives the full session key, so backend owner metadata plus a user-aware selection policy may be enough. I'll raise that upstream. On the Kyuubi side, the remaining scope looks like: optionally start Spark Connect in a Kyuubi-launched Spark engine and advertise its endpoint through the existing discovery metadata; expose the live engine pool to the gateway, including YARN deployments; coordinate idle shutdown and drain with Connect activity; provide the backend authentication token when the engine is launched. I would keep routing, affinity and Connect authentication in the upstream gateway, and execution and authorization in Spark. Per-user engines should preserve Kyuubi's existing execution identity, but I would validate that with Ranger end to end rather than assume it works automatically. For the first Kyuubi slice, I'd target one Spark 4.x line, disabled by default: start Connect alongside Thrift and advertise the endpoint. Does that look like the right first slice, especially if user-aware backend placement is handled in the upstream gateway? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
