Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Thank Shengkai for the new round of updating. I don't have comments on the updates. +1 for starting the vote. Best, Jark On Thu, 19 May 2022 at 20:18, Shengkai Fang wrote: > Hi, Timo. > > Thanks for your feedback! > > > SQLGatewayService.getFunction / UserDefinedFunctionInfo > > Yes. I miss

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi, Timo. Thanks for your feedback! > SQLGatewayService.getFunction / UserDefinedFunctionInfo Yes. I miss some parts in the FLIP. I have fix the errors now. > configure_session Thanks for your inputs. Considering the difference, I am still prone to use the `configure_session`. >

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi Shengkai, thanks again for the update. I don't have much to add: > I still think we should use a new state machine Thanks for the explanation about async/sync behavior. I thought we also want to use the Gateway for job status updates. But if FINISHED only refers to the job submission, the

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi, all. After discussing with Becket Qin offline, I modified the FLIP a little. The change is as follow: 1. Add /api_versions in the REST API. The api is used by the client to know the current REST Endpoint supports which version. The client is able to choose the version for later

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi, Jark, Timo. Nice to have an agreement! Thanks for Jark's inputs about the multiple version Flink. I have already updated the FLIP in the rejected alternatives about details. 1. We should definitely just use LogicalTypeJsonSerializer and not a second JSON representation. Our concern is

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi Shengkai, Hi Jark, thanks for the additional explanation and the update of the FLIP. This will help us in the future for documenting our decisions. The arguments why to include the Gateway into the main repo make a lot of sense to me. Esp. also because both CLI and gateway need some

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi Shengkai, One comment on the configure_session VS initialize_session, I think configure_session is better than initialize_session, because "initialize" sounds like this method can only be invoked once. But I can see this method is useful to be invoked several times to configure the session

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi Timo, I agree with Shengkai that we should keep Gateway in the Flink main repo. Here are my thoughts: 1) It is not feasible and maintainable to invoke different versions of Flink dependencies in one JVM. I'm afraid it would be a nightmare when the code is messed up with all the reflections.

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi Timo. Thanks for your feedback! > It would be great if you can copy/paste some of the tricky design decisions into the FLIP. Yes. I have already added a section about the `Rejected Alternatives`. It includes the discuss about the - TableInfo and FunctionInfo VS CatalogTable and

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi Shengkai, sorry for jumping in so late into the discussion. There was a lot going on in the last 3 weeks. This FLIP is very important and it is great that you tackle this topic which is planned since we started with the SQL Client. The ML discussion is already quite long. It would be

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Thank Shengkai for driving this work! +1 to start a VOTE. Best, Jark On Tue, 10 May 2022 at 12:13, Shengkai Fang wrote: > Hi, everyone. > > Thanks for all the inputs. Hope my feedback can address most of questions. > > After the long discussion, I think we have reached the consensus about the

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi, everyone. Thanks for all the inputs. Hope my feedback can address most of questions. After the long discussion, I think we have reached the consensus about the design. If the discussion doesn't get more response, I think we can start the vote tomorrow. Best, Shengkai Shengkai Fang

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi JingSong. Thanks for your feedback. > reorganize the FLIP, what Pluggable Endpoint Discovery is, and how users to add new Endpoint, before introducing SQLGatewayService. I update the FLIP: reorganize the order and add more details. > Then I have some doubts about the name

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi Shengkai, thanks for your reply. > REST API is the user interface. The REST API transforms the request to the invocation of the SQLGatewayService that is the one doing the work. We split the Gateway into the SQLGatewayService and Endpoint(REST API) and its benefit is that all the Endpoints

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi Martijn. It seems we have reached consensus to support the Gateway inside the Flink code base. Hi, Echo. Thanks for your interest. > whether flink-sql-gateway should be retained in the Flink project. I think the discussion above is clear. It is the essential tool to provide out-of-box

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi Shengkai, Thanks for driving the proposal, I've been paying attention to the ververica/flink-sql-gateway[1] project recently. Because the Flink version the company is currently using is 1.14, so try to upgrade the gateway to Flink-1.14, because of the changes in flink-table api makes the

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi everyone, Happy to see that this discussion is very much active. Couple of comments: > It's not about internal interfaces. Flink itself doesn't provide backward compatibility for public APIs. Is that so? In FLIP-196 [1] is explicitly stated "What we guarantee in terms of stability is that a

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi Martijn, Shengkai >I don't think that the Gateway is a 'core' >function of Flink which should be included with Flink. I have a different viewpoint. For SQL users, it provides out-of-box experience. Just like SQL-client, users can enjoy variable sql experiences after downloading/building a

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Thanks, Shengkai for driving. And all for your discussion. > intergate the Gateway into the Flink code base After I talk with Shengkai offline and read the topic `Powering HTAP at ByteDance with Apache Flink` of Flink Forward Asia. I think it is better to integrate Gateway code into the Flink

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Thanks Martijn, Nicholas, Godfrey, Jark and Jingsong feedback > I would like to understand why it's complicated to make the upgrades > problematic I aggree with Jark's point. The API is not very stable in the Flink actually. For example, the Gateway relies on the planner. But in release-1.14

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Thanks Shengkai for driving. And all for your discussion. > The reason why we introduce the gateway with pluggable endpoints is that many > users has their preferences. For example, the HiveServer2 users prefer to use > the gateway with HiveServer2-style API, which has numerous tools. However,

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi Martijn, Regarding maintaining Gateway inside or outside Flink code base, I would like to share my thoughts: > I would like to understand why it's complicated to make the upgrades problematic. Is it because of relying on internal interfaces? If so, should we not consider making them public?

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi Shengkai. Thanks for driving the proposal, it's been silent too long. I have a few questions: about the Architecture > The architecture of the Gateway is in the following graph. Is the TableEnvironment shared for all sessions ? about the REST Endpoint > /v1/sessions Are both local file and

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi Shengkai, I have another concern about the submission of batch job. Does the Flink SQL gateway support to submit batch job? In Kyuubi, BatchProcessBuilder is used to submit batch job. What about the Flink SQL gateway? Best regards, Nicholas Jiang On 2022/04/24 03:28:36 Shengkai Fang wrote:

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi Shengkai, > Agreed. The FLIP mainly focus on the Gateway. I think it's better to rename the name to the "Support SQL Gateway". WDYT? +1 > I think it's better to intergate the Gateway into the Flink code base. The reason behind is > 1. The Gateway relies on the Flink implementation, I think

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi Marijn and LuNing. Thanks for your feedback! > The FLIP is called "SQL Client Gateway", but isn't this a REST Gateway which would be used by Flink's SQL Client (or other applications)? Agreed. The FLIP mainly focus on the Gateway. I think it's better to rename the name to the "Support SQL

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

> * Should we introduce the REST Gateway as part of Flink's main repository? Wouldn't we be better off to maintain this in a separate repository under ASF? Ideally you would like to be able to support multiple Flink versions with one version of the REST Gateway I think? We would be better off

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi ShengKai, After I read FLIP-91[1], I want to add an init-file option. Its functionality is the same as option '-i' of Flink SQL Client. When I use Catalog(HiveCatalog), I need to execute `CREATE CATALOG` by this option after SQL Gateway starts every time. Shall we name this option

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi Shengkai, Thanks for opening this discussion. I did a first brief pass over the FLIP and I have a couple of questions/remarks: * The FLIP is called "SQL Client Gateway", but isn't this a REST Gateway which would be used by Flink's SQL Client (or other applications)? * From a user

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi. Jiang. Thanks for your feedback！ > Do the public interfaces of GatewayService refer to any service? We will only expose one GatewayService implementation. We will put the interface into the common package and the developer who wants to implement a new endpoint can just rely on the interface

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi Shengkai. Thanks for driving the proposal of SQL Client Gateway. I have some knowledge of Kyuubi and have some questions about the design: 1.Do the public interfaces of GatewayService refer to any service? If referring to HiveService, does GatewayService need interfaces like getQueryId etc.

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi, Paul. Thanks for you feedback! > Is operation status exposed to users or just for internal usage to represent the job status Only in the synchronized job submission, the operation status contains the job status. When user chooses the synchronized job submission, it means - user expects to

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi Shengkai, Good to see FLIP-91 is revisited after such a long time. Big +1 for the proposal. I’ve been using the SQL gateway for a while, just put in my 2 cents: 1. Is operation status exposed to users or just for internal usage to represent the job status? I’m assuming the latter, or

[DISCUSS] FLIP-91: Support SQL Client Gateway

Hi, Flink developers. I want to start a discussion about the FLIP-91: Support Flink SQL Gateway[1]. Flink SQL Gateway is a service that allows users to submit and manage their jobs in the online environment with the pluggable endpoints. The reason why we introduce the Gateway with pluggable

Re: [DISCUSS] FLIP-91 - Support SQL Client Gateway

Hi everyone, I was curious about the progress on this FLIP-91. Is this actively being developed? I believe the code is in development at https://github.com/ververica/flink-sql-gateway, is this the right REPO? I haven't seen much activity on this since sometime last year. I wanted to

Re: [DISCUSS] FLIP-91 - Support SQL Client Gateway

Hi all, I also agree with Stephan and Timo that the SQL Client should be a simple "shell around the table environment". About "making this a standalone project", I agree with Timo, and I think keeping SQL client in Flink codebase can ensure SQL client integrity (has both embedded mode and gateway

Re: [DISCUSS] FLIP-91 - Support SQL Client Gateway

Hi everyone, I agree with Stephan that the SQL Client should be a simple "shell around the table environment". However, I see a contradiction in the mentioned advantages "not limited by Flink committer reviews" and "quicker independent releases". If most functionality must be contained in

Re: [DISCUSS] FLIP-91 - Support SQL Client Gateway

Hi all! I think this is a useful feature. Two questions about this proposal: (1) The SQL client tried to be a hybrid between a SQL client and a gateway server (which blew up in complexity and never finished). Would having a dedicated gateway component mean that we can simplify the client and

Re: [DISCUSS] FLIP-91 - Support SQL Client Gateway

Hi devs, I've updated the FLIP-91 [0] according to feedbacks. Please take another look. Best, godfrey [0] https://docs.google.com/document/d/1DKpFdov1o_ObvrCmU-5xi-VrT6nR2gxq-BbswSSI9j8/

Re: [DISCUSS] FLIP-91 - Support SQL Client Gateway

Hi, +1 to the general idea. Supporting sql client gateway mode will bridge the connection between Flink SQL and production environment. Also the JDBC driver is a quite good supplement for usability of Flink SQL, users will have more choices to try out Flink SQL such as Tableau. I went through

Re: [DISCUSS] FLIP-91 - Support SQL Client Gateway

The general idea sounds great. I'm going to keep up with the progress soon. Best, tison. Bowen Li 于2020年1月5日周日 下午12:59写道： > +1. It will improve user experience quite a bit. > > > On Thu, Jan 2, 2020 at 22:07 Yangze Guo wrote: > > > Thanks for driving this, Xiaoling! > > > > +1 for supporting

Re: [DISCUSS] FLIP-91 - Support SQL Client Gateway

+1. It will improve user experience quite a bit. On Thu, Jan 2, 2020 at 22:07 Yangze Guo wrote: > Thanks for driving this, Xiaoling! > > +1 for supporting SQL client gateway. > > Best, > Yangze Guo > > > On Thu, Jan 2, 2020 at 9:58 AM 贺小令 wrote: > > > > Hey everyone, > > FLIP-24 > >

Re: [DISCUSS] FLIP-91 - Support SQL Client Gateway

Thanks for driving this, Xiaoling! +1 for supporting SQL client gateway. Best, Yangze Guo On Thu, Jan 2, 2020 at 9:58 AM 贺小令 wrote: > > Hey everyone, > FLIP-24 > > proposes the whole conception and architecture of SQL

[DISCUSS] FLIP-91 - Support SQL Client Gateway

Hey everyone, FLIP-24 proposes the whole conception and architecture of SQL Client. The embedded mode is already supported since release-1.5, which is helpful for debugging/demo purposes. Many users ask that how to submit a