Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Hi all, If there is no question related to this FLIP, we will start the voting thread on 3/6 (Monday). thank you all. Best Regards, Ran Tao https://github.com/chucheng92 Ran Tao 于2023年3月1日周三 17:40写道： > Hi, Timo. thanks. > > I have copied the content of docs and updated the FLIP. > > Thanks

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Hi, Timo. thanks. I have copied the content of docs and updated the FLIP. Thanks for your advice about consistency(ILIKE). Because calcite already support parse ILIKE and exist SqlLibraryOperators.ILIKE(which means it's not std). It's will not difficult to support it, and we just need to support

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Hi Ran Tao, Thanks for working on this FLIP. The FLIP is in a pretty good shape already and I don't have much to add. Will we also support ILIKE in queries? Or is this a pure DDL expressions? For consistency, we should support it in SELECT and Table API as well. I hope this is not too much

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

thanks Sergey, sounds good. You can add in FLIP ticket[1]. [1] https://issues.apache.org/jira/browse/FLINK-31256 Best Regards, Ran Tao https://github.com/chucheng92 Sergey Nuyanzin 于2023年2月28日周二 19:44写道： > >>Currently I think we can load from the jar and check the services file to > >> get

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

>>Currently I think we can load from the jar and check the services file to >> get the connector type. but is it necessary we may continue to discuss. >>Hi, Sergey, WDYT？ Another idea is FactoryUtil#discoverFactories and check if it implements DynamicTableSourceFactory or DynamicTableSinkFactory

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Hi, Jark. thanks. > About ILIKE I have updated the FLIP for ILIKE support (Including existing showTables & showColumns how to change). > About show connectors @Sergey, Currently I think we can load from the jar and check the services file to get the connector type. but is it necessary we may

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Besides, if we introduce the ILIKE, we should also add this feature for the previous SHOW with LIKE statements. They should be included in this FLIP. Best, Jark > 2023年2月28日 17:40，Jark Wu 写道： > > Hi Ran, > > Could you add descriptions about what’s the behavior and differences between > the

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Hi Ran, Could you add descriptions about what’s the behavior and differences between the LIKE and ILIKE? Besides, I don’t see the SHOW CONNECTOR syntax and description and how it works in the FLIP. Is it intended to be included in this FLIP? Best, Jark > 2023年2月28日 10:58，Ran Tao 写道： > >

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Hi, guys. thanks for advices. allow me to make a small summary: 1.Support ILIKE 2.Using catalog api to support show operations 3.Need a dedicated FLIP try to support INFORMATION_SCHEMA 4.Support SHOW CONNECTORS If there are no other questions, i will try to start a VOTE for this FLIP. WDYT?

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Hi Jing, I think we can support it, it's not a very difficult job. thank you. Best Regards, Ran Tao Jing Ge 于2023年2月28日周二 06:04写道： > Hi Ran, > > It would be great if we could support ILIKE [1]. If you figure out that the > effort is bigger than you expected, you might want to create a ticket

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Hi Ran, It would be great if we could support ILIKE [1]. If you figure out that the effort is bigger than you expected, you might want to create a ticket for beginners(?) who are interested in Flink development to make a further contribution. WDYT? Best regards, Jing [1]

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Hi Jark, thanks for your comment. >Considering they > are orthogonal and information schema requires more complex design and > discussion, it deserves a separate FLIP I'm ok with a separate FLIP for INFORMATION_SCHEMA. >Sergey, are you willing to contribute this FLIP? Seems I need to have more

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

HI, Jing. thanks. @about ILIKE, from my collections of some popular engines founds that just snowflake has this syntax in show with filtering. do we need to support it? if yes, then current some existed show operations need to be addressed either. @about ShowOperation with like. it's a good idea.

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Hi, @Aitozi This is exactly why LoD has been introduced: to avoid exposing internal structure(2nd and lower level API). @Jark IMHO, there is no conflict between LoD and "High power-to-weight ratio" with the given example, List.subList() returns List interface itself, no internal or further

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Hi Ran, NIT: ILIKE is supported by Clickhouse. Since Flink is focusing on the unified batch and stream data processing, it might make sense to support it. It is more convenient for users. Thanks for clarifying the definition of "useLike" and "notLike". It seems that "notLike" depends on

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Hi, Jark, thanks for explantations. if there are no other questions, i will try to start a VOTE to address current things in this FLIP. We may open another dedicated FLIP to discuss INFORMATION_SCHEMA. hi, guys, WDYT? Best Regards, Ran Tao Jark Wu 于2023年2月25日周六 22:57写道： > Hi Sergey, > > I

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Hi Sergey, I think INFORMATION_SCHEMA is a very interesting idea, and I hope we can support it. However, it doesn't conflict with the idea of auxiliary statements. I can see different benefits of them. The information schema provides powerful and flexible capabilities but needs to learn the

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Thanks John. It seems that most people prefer the information_schema implementation. information_schema does have more benefits (however, the show operation is also an option and supplement). Otherwise, the sql syntax and keywords may be changed frequently. Of course, it will be more complicated

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Hello Ran, Thanks for the FLIP! Do you mind if we revisit the topic of doing this by adding an Information schema? The SHOW approach requires modifying the parser/language for every gap we identify. On the flip side, an Information schema is much easier to discover and remember how to use,

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

thanks for explanation >But it's not clear to me what exactly > you want to display? Is it the name of the plugin? I was thinking about name, type (source/sink) and may be version (not sure if it's possible right now) On Fri, Feb 24, 2023 at 12:46 PM Ran Tao wrote: > Hi, Sergey. thanks. first

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Hi, Sergey. thanks. first step we can support filtering for show operations in this FLIP try to align other engines. If we want to support describe other objects, of course we need to design how to get these metadatas or informations and printAsStyle. (So it maybe need another FLIP for more

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Thanks for driving the FLIP I have a couple of questions Am I right that INFORMATION_SCHEMA mentioned by Timo[1] is out of scope of this FLIP? I noticed there are some support of it in Spark[2]/Hive[3]/Snowflake[4] and others Does it make sense to add support for connectors e.g. show

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Hi Jing, > we'd better reduce the dependency chain and follow the Law of Demeter(LoD, clean code). > Adding a new method in TableEnvironment is therefore better than calling an API chain I think I don't fully agree that LoD is a good practice. Actually, I would prefer to keep the API clean and

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Hi, Thanks for the nice proposal, Ran. Regarding this api usage, I have some discussion with @twalthr before as here Personally, I think leaking the Catalog to the user side is not suitable, since there are some

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Hi, JingSong, Jing. thank for sharing your opinions. What you say makes sense, both approaches have pros and cons. If it is a modification of `TableEnvrionment`, such as listDatabases(catalog). It is actually consistent with the other overloaded methods before, and defining this method means

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Hi Jingson, Thanks for sharing your thoughts. Please see my reply below. On Thu, Feb 23, 2023 at 10:16 AM Jingsong Li wrote: > Hi Jing Ge, > > First, flink-table-common contains all common classes of Flink Table, > I think it is hard to bypass its dependence. > If any time when we use

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Hi Jing Ge, First, flink-table-common contains all common classes of Flink Table, I think it is hard to bypass its dependence. Secondly, almost all methods in Catalog looks useful to me, so if we are following LoD, we should add all methods again to TableEnvironment. I think it is redundant.

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Hi Jingson, Thanks for the knowledge sharing. IMHO, it looks more like a design guideline question than just avoiding public API change. Please correct me if I'm wrong. Catalog is in flink-table-common module and TableEnvironment is in flink-table-api-java. Depending on how and where those

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Hi Jingsong. thanks. i got it. In this way, there is no need to introduce new API changes. Best Regards, Ran Tao Jingsong Li 于2023年2月23日周四 12:26写道： > Hi Ran, > > I mean we can just use > TableEnvironment.getCatalog(getCurrentCatalog).get().listDatabases(). > > We don't need to provide new

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Hi Ran, I mean we can just use TableEnvironment.getCatalog(getCurrentCatalog).get().listDatabases(). We don't need to provide new apis just for utils. Best, Jingsong On Thu, Feb 23, 2023 at 12:11 PM Ran Tao wrote: > > Hi Jingsong, thanks. > > The implementation of these statements in

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Hi Jingsong, thanks. The implementation of these statements in TableEnvironmentImpl is called through the catalog api. but it does support some new override methods on the catalog api side, and I will update it later. Thank you. e.g. TableEnvironmentImpl @Override public String[]

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Thanks for the proposal. +1 for the proposal. I am confused about "Proposed TableEnvironment SQL API Changes", can we just use catalog api for this requirement? Best, Jingsong On Thu, Feb 23, 2023 at 10:48 AM Jacky Lau wrote: > > Hi Ran: > Thanks for driving the FLIP. the google doc looks

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

thanks Jing. there is a small mistake.useLike it means whether to enable like. e.g. SHOW TABLES from cat1.db1 not like 't%' // useLike is true, notLike is true SHOW TABLES from cat1.db1 like 't%' // useLike is true, notLike is false useLike both are true. I have updated the flip for this.

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Hi Ran: Thanks for driving the FLIP. the google doc looks really good. it is important to improve user interactive experience. +1 to support this feature. Jing Ge 于2023年2月23日周四 00:51写道： > Hi Ran, > > Thanks for driving the FLIP. It looks overall good. Would you like to add > a description of

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Hi Ran, Thanks for driving the FLIP. It looks overall good. Would you like to add a description of useLike and notLike? I guess useLike true is for "LIKE" and notLike true is for "NOT LIKE" but I am not sure if I understood it correctly. Furthermore, does it make sense to support "ILIKE" too?

[DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

Currently flink sql auxiliary statements has supported some good features such as catalog/databases/table support. But these features are not very complete compared with other popular engines such as spark, presto, hive and commercial engines such as snowflake. For example, many engines support