Hi Shengkai. Thanks for driving the proposal, it's been silent too long.
I have a few questions: about the Architecture > The architecture of the Gateway is in the following graph. Is the TableEnvironment shared for all sessions ? about the REST Endpoint > /v1/sessions Are both local file and remote file supported for `libs` and `jars`? Does sql gateway support upload file? >/v1/sessions/:session_handle/configure_session Can this api be replaced with `/v1/sessions/:session_handle/statements` ? >/v1/sessions/:session_id/operations/:operation_handle/status `:session_id` is a typo, it should be `:session_handdle` >/v1/sessions/:session_handle/statements >The statement must be a single command Does this api support `begin statement set ... end` or `statement set begin ... end` DO `ADD JAR`, `REMOVE JAR` support ? If yes, how to manage the jars? >/v1/sessions/:session_handle/operations/:operation_handle/result/:token >"type": # string value of LogicalType Some LogicalTypes can not be serialized, such as: CharType(0) about Options > endpoint.protocol I think REST is not a kind of protocol[1], but is an architectural style. The value should be `HTTP`. about SQLGatewayService API > Catalog API > ... I think we should avoid providing such api, because once catalog api is changed or added, This class should also be changed. SQL statement is a more general interface. > Options > sql-gateway.session.idle.timeout >sql-gateway.session.check.interval >sql-gateway.worker.keepalive.time It's better we can keep the option style as Flink, the level should not be too deep. sql-gateway.session.idle.timeout -> sql-gateway.session.idle-timeout sql-gateway.session.check.interval -> sql-gateway.session.check-interval sql-gateway.worker.keepalive.time -> sql-gateway.worker.keepalive->time [1] https://restfulapi.net/ Best, Godfrey Nicholas Jiang <[email protected]> 于2022年5月5日周四 14:58写道： > > Hi Shengkai, > > I have another concern about the submission of batch job. Does the Flink SQL > gateway support to submit batch job? In Kyuubi, BatchProcessBuilder is used > to submit batch job. What about the Flink SQL gateway? > > Best regards, > Nicholas Jiang > > On 2022/04/24 03:28:36 Shengkai Fang wrote: > > Hi. Jiang. > > > > Thanks for your feedback！ > > > > > Do the public interfaces of GatewayService refer to any service? > > > > We will only expose one GatewayService implementation. We will put the > > interface into the common package and the developer who wants to implement > > a new endpoint can just rely on the interface package rather than the > > implementation. > > > > > What's the behavior of SQL Client Gateway working on Yarn or K8S? Does > > the SQL Client Gateway support application or session mode on Yarn? > > > > I think we can support SQL Client Gateway to submit the jobs in > > application/sesison mode. > > > > > Is there any event trigger in the operation state machine? > > > > Yes. I have already updated the content and add more details about the > > state machine. During the revise, I found that I mix up the two concepts: > > job submission and job execution. In fact, we only control the submission > > mode at the gateway layer. Therefore, we don't need to mapping the > > JobStatus here. If the user expects that the synchronization behavior is to > > wait for the completion of the job execution before allowing the next > > statement to be executed, then the Operation lifecycle should also contains > > the job's execution, which means users should set `table.dml-sync`. > > > > > What's the return schema for the public interfaces of GatewayService? > > Like getTable interface, what's the return value schema? > > > > The API of the GatewayService return the java objects and the endpoint can > > organize the objects with expected schema. The return results is also list > > the section ComponetAPI#GatewayService#API. The return type of the > > GatewayService#getTable is `ContextResolvedTable`. > > > > > How does the user get the operation log? > > > > The OperationManager will register the LogAppender before the Operation > > execution. The Log Appender will hijack the logger and also write the log > > that related to the Operation to another files. When users wants to fetch > > the Operation log, the GatewayService will read the content in the file and > > return. > > > > Best, > > Shengkai > > > > > > > > > > Nicholas Jiang <[email protected]> 于2022年4月22日周五 16:21写道： > > > > > Hi Shengkai. > > > > > > Thanks for driving the proposal of SQL Client Gateway. I have some > > > knowledge of Kyuubi and have some questions about the design: > > > > > > 1.Do the public interfaces of GatewayService refer to any service? If > > > referring to HiveService, does GatewayService need interfaces like > > > getQueryId etc. > > > > > > 2.What's the behavior of SQL Client Gateway working on Yarn or K8S? Does > > > the SQL Client Gateway support application or session mode on Yarn? > > > > > > 3.Is there any event trigger in the operation state machine? > > > > > > 4.What's the return schema for the public interfaces of GatewayService? > > > Like getTable interface, what's the return value schema? > > > > > > 5.How does the user get the operation log? > > > > > > Thanks, > > > Nicholas Jiang > > > > > > On 2022/04/21 06:42:30 Shengkai Fang wrote: > > > > Hi, Flink developers. > > > > > > > > I want to start a discussion about the FLIP-91: Support Flink SQL > > > > Gateway[1]. Flink SQL Gateway is a service that allows users to submit > > > and > > > > manage their jobs in the online environment with the pluggable > > > > endpoints. > > > > The reason why we introduce the Gateway with pluggable endpoints is that > > > > many users have their preferences. For example, the HiveServer2 users > > > > prefer to use the gateway with HiveServer2-style API, which has numerous > > > > tools. However, some filnk-native users may prefer to use the REST API. > > > > Therefore, we propose the SQL Gateway with pluggable endpoint. > > > > > > > > In the FLIP, we also propose the REST endpoint, which has the similar > > > > APIs compared to the gateway in the ververica/flink-sql-gateway[2]. At > > > the > > > > last, we discuss how to use the SQL Client to submit the statement to > > > > the > > > > Gateway with the REST API. > > > > > > > > I am glad that you can give some feedback about FLIP-91. > > > > > > > > Best, > > > > Shengkai > > > > > > > > [1] > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway > > > > [2] https://github.com/ververica/flink-sql-gateway > > > > > > > > >
