pan3793 commented on PR #5062: URL: https://github.com/apache/kyuubi/pull/5062#issuecomment-1643276584
> I'm not sure which service it better for in-cluster client connect. For THTIFT_BINARY (TCP) it does not matter, because it's a long connection, the whole lifecycle of a session while sticky in one Kyuubi Server Pod. For REST (HTTP), I think headless should be better. In the batch submission (aka Spark Jar application), suppose the `spark-submit` was executed in the Kyuubi-Server-1, the following request to fetch local logs of `spark-submit` process must be handled by the same Pod, the current implementation is: 1. Client => [Kyuubi Server Service] => random Kyuubi Server Pod 2.1 If the Kyuubi Server Pod holds the batch session => handle and return response 2.2 If the Kyuubi Server Pod does not hold the batch session, check the state store (and external storage like MySQL) to find which Kyuubi Server holds it and forward the request to it, wait for the response, then pass through it to the Client The 2.2 could be enhanced by - the response contains the Kyuubi-Server-1 Pod address - the client always(until network exception) tries to access the Kyuubi-Server-1 directly to avoid Kyuubi Server internal forwarding overhead. In such a case, I would say Headless Service seems more suitable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
