Last additional comment on Option 2. The reason why I prefer option 3 is that in option 3 all objects internally are identified with 3 parts. This makes it easier to handle at different locations e.g. while persisting views, as all objects have uniform representation.
On Thu, 19 Sep 2019, 07:31 Dawid Wysakowicz, <wysakowicz.da...@gmail.com> wrote: > Hi, > I think it makes sense to start voting at this point. > > Option 1: Only 1-part identifiers > PROS: > - allows shadowing built-in functions > CONS: > - incosistent with all the other objects, both permanent & temporary > - does not allow shadowing catalog functions > > Option 2: Special keyword for built-in function > I think this is quite similar to the special catalog/db. The thing I am > strongly against in this proposal is the GLOBAL keyword. This keyword has a > meaning in rdbms systems and means a function that is present for a > lifetime of a session in which it was created, but available in all other > sessions. Therefore I really don't want to use this keyword in a different > context. > > Option 3: Special catalog/db > > PROS: > - allows shadowing built-in functions > - allows shadowing catalog functions > - consistent with other objects > CONS: > - we introduce a special namespace for built-in functions > > I don't see a problem with introducing the special namespace. In the end > it is very similar to the keyword approach. In this case the catalog/db > combination would be the "keyword" > > Therefore my votes: > Option 1: -0 > Option 2: -1 (I might change to +0 if we can come up with a better keyword) > Option 3: +1 > > Best, > Dawid > > > On Thu, 19 Sep 2019, 05:12 Xuefu Z, <usxu...@gmail.com> wrote: > >> Hi Aljoscha, >> >> Thanks for the summary and these are great questions to be answered. The >> answer to your first question is clear: there is a general agreement to >> override built-in functions with temp functions. >> >> However, your second and third questions are sort of related, as a >> function >> reference can be either just function name (like "func") or in the form or >> "cat.db.func". When a reference is just function name, it can mean either >> a >> built-in function or a function defined in the current cat/db. If we >> support overriding a built-in function with a temp function, such >> overriding can also cover a function in the current cat/db. >> >> I think what Timo referred as "overriding a catalog function" means a temp >> function defined as "cat.db.func" overrides a catalog function "func" in >> cat/db even if cat/db is not current. To support this, temp function has >> to >> be tied to a cat/db. What's why I said above that the 2nd and 3rd >> questions >> are related. The problem with such support is the ambiguity when user >> defines a function w/o namespace, "CREATE TEMPORARY FUNCTION func ...". >> Here "func" can means a global temp function, or a temp function in >> current >> cat/db. If we can assume the former, this creates an inconsistency because >> "CREATE FUNCTION func" actually means a function in current cat/db. If we >> assume the latter, then there is no way for user to create a global temp >> function. >> >> Giving a special namespace for built-in functions may solve the ambiguity >> problem above, but it also introduces artificial catalog/database that >> needs special treatment and pollutes the cleanness of the code. I would >> rather introduce a syntax in DDL to solve the problem, like "CREATE >> [GLOBAL] TEMPORARY FUNCTION func". >> >> Thus, I'd like to summarize a few candidate proposals for voting purposes: >> >> 1. Support only global, temporary functions without namespace. Such temp >> functions overrides built-in functions and catalog functions in current >> cat/db. The resolution order is: temp functions -> built-in functions -> >> catalog functions. (Partially or fully qualified functions has no >> ambiguity!) >> >> 2. In addition to #1, support creating and referencing temporary functions >> associated with a cat/db with "GLOBAL" qualifier in DDL for global temp >> functions. The resolution order is: global temp functions -> built-in >> functions -> temp functions in current cat/db -> catalog function. >> (Resolution for partially or fully qualified function reference is: temp >> functions -> persistent functions.) >> >> 3. In addition to #1, support creating and referencing temporary functions >> associated with a cat/db with a special namespace for built-in functions >> and global temp functions. The resolution is the same as #2, except that >> the special namespace might be prefixed to a reference to a built-in >> function or global temp function. (In absence of the special namespace, >> the >> resolution order is the same as in #2.) >> >> My personal preference is #1, given the unknown use case and introduced >> complexity for #2 and #3. However, #2 is an acceptable alternative. Thus, >> my votes are: >> >> +1 for #1 >> +0 for #2 >> -1 for #3 >> >> Everyone, please cast your vote (in above format please!), or let me know >> if you have more questions or other candidates. >> >> Thanks, >> Xuefu >> >> >> >> >> >> >> >> On Wed, Sep 18, 2019 at 6:42 AM Aljoscha Krettek <aljos...@apache.org> >> wrote: >> >> > Hi, >> > >> > I think this discussion and the one for FLIP-64 are very connected. To >> > resolve the differences, think we have to think about the basic >> principles >> > and find consensus there. The basic questions I see are: >> > >> > - Do we want to support overriding builtin functions? >> > - Do we want to support overriding catalog functions? >> > - And then later: should temporary functions be tied to a >> > catalog/database? >> > >> > I don’t have much to say about these, except that we should somewhat >> stick >> > to what the industry does. But I also understand that the industry is >> > already very divided on this. >> > >> > Best, >> > Aljoscha >> > >> > > On 18. Sep 2019, at 11:41, Jark Wu <imj...@gmail.com> wrote: >> > > >> > > Hi, >> > > >> > > +1 to strive for reaching consensus on the remaining topics. We are >> > close to the truth. It will waste a lot of time if we resume the topic >> some >> > time later. >> > > >> > > +1 to “1-part/override” and I’m also fine with Timo’s “cat.db.fun” way >> > to override a catalog function. >> > > >> > > I’m not sure about “system.system.fun”, it introduces a nonexistent >> cat >> > & db? And we still need to do special treatment for the dedicated >> > system.system cat & db? >> > > >> > > Best, >> > > Jark >> > > >> > > >> > >> 在 2019年9月18日，06:54，Timo Walther <twal...@apache.org> 写道： >> > >> >> > >> Hi everyone, >> > >> >> > >> @Xuefu: I would like to avoid adding too many things incrementally. >> > Users should be able to override all catalog objects consistently >> according >> > to FLIP-64 (Support for Temporary Objects in Table module). If functions >> > are treated completely different, we need more code and special cases. >> From >> > an implementation perspective, this topic only affects the lookup logic >> > which is rather low implementation effort which is why I would like to >> > clarify the remaining items. As you said, we have a slight consenus on >> > overriding built-in functions; we should also strive for reaching >> consensus >> > on the remaining topics. >> > >> >> > >> @Dawid: I like your idea as it ensures registering catalog objects >> > consistent and the overriding of built-in functions more explicit. >> > >> >> > >> Thanks, >> > >> Timo >> > >> >> > >> >> > >> On 17.09.19 11:59, kai wang wrote: >> > >>> hi, everyone >> > >>> I think this flip is very meaningful. it supports functions that >> can be >> > >>> shared by different catalogs and dbs, reducing the duplication of >> > functions. >> > >>> >> > >>> Our group based on flink's sql parser module implements create >> function >> > >>> feature, stores the parsed function metadata and schema into mysql, >> and >> > >>> also customizes the catalog, customizes sql-client to support custom >> > >>> schemas and functions. Loaded, but the function is currently global, >> > and is >> > >>> not subdivided according to catalog and db. >> > >>> >> > >>> In addition, I very much hope to participate in the development of >> this >> > >>> flip, I have been paying attention to the community, but found it is >> > more >> > >>> difficult to join. >> > >>> thank you. >> > >>> >> > >>> Xuefu Z <usxu...@gmail.com> 于2019年9月17日周二 上午11:19写道： >> > >>> >> > >>>> Thanks to Tmo and Dawid for sharing thoughts. >> > >>>> >> > >>>> It seems to me that there is a general consensus on having temp >> > functions >> > >>>> that have no namespaces and overwrite built-in functions. (As a >> side >> > note >> > >>>> for comparability, the current user defined functions are all >> > temporary and >> > >>>> having no namespaces.) >> > >>>> >> > >>>> Nevertheless, I can also see the merit of having namespaced temp >> > functions >> > >>>> that can overwrite functions defined in a specific cat/db. However, >> > this >> > >>>> idea appears orthogonal to the former and can be added >> incrementally. >> > >>>> >> > >>>> How about we first implement non-namespaced temp functions now and >> > leave >> > >>>> the door open for namespaced ones for later releases as the >> > requirement >> > >>>> might become more crystal? This also helps shorten the debate and >> > allow us >> > >>>> to make some progress along this direction. >> > >>>> >> > >>>> As to Dawid's idea of having a dedicated cat/db to host the >> temporary >> > temp >> > >>>> functions that don't have namespaces, my only concern is the >> special >> > >>>> treatment for a cat/db, which makes code less clean, as evident in >> > treating >> > >>>> the built-in catalog currently. >> > >>>> >> > >>>> Thanks, >> > >>>> Xuefiu >> > >>>> >> > >>>> On Mon, Sep 16, 2019 at 5:07 PM Dawid Wysakowicz < >> > >>>> wysakowicz.da...@gmail.com> >> > >>>> wrote: >> > >>>> >> > >>>>> Hi, >> > >>>>> Another idea to consider on top of Timo's suggestion. How about we >> > have a >> > >>>>> special namespace (catalog + database) for built-in objects? This >> > catalog >> > >>>>> would be invisible for users as Xuefu was suggesting. >> > >>>>> >> > >>>>> Then users could still override built-in functions, if they fully >> > qualify >> > >>>>> object with the built-in namespace, but by default the common >> logic >> > of >> > >>>>> current dB & cat would be used. >> > >>>>> >> > >>>>> CREATE TEMPORARY FUNCTION func ... >> > >>>>> registers temporary function in current cat & dB >> > >>>>> >> > >>>>> CREATE TEMPORARY FUNCTION cat.db.func ... >> > >>>>> registers temporary function in cat db >> > >>>>> >> > >>>>> CREATE TEMPORARY FUNCTION system.system.func ... >> > >>>>> Overrides built-in function with temporary function >> > >>>>> >> > >>>>> The built-in/system namespace would not be writable for permanent >> > >>>> objects. >> > >>>>> WDYT? >> > >>>>> >> > >>>>> This way I think we can have benefits of both solutions. >> > >>>>> >> > >>>>> Best, >> > >>>>> Dawid >> > >>>>> >> > >>>>> >> > >>>>> On Tue, 17 Sep 2019, 07:24 Timo Walther, <twal...@apache.org> >> wrote: >> > >>>>> >> > >>>>>> Hi Bowen, >> > >>>>>> >> > >>>>>> I understand the potential benefit of overriding certain built-in >> > >>>>>> functions. I'm open to such a feature if many people agree. >> > However, it >> > >>>>>> would be great to still support overriding catalog functions with >> > >>>>>> temporary functions in order to prototype a query even though a >> > >>>>>> catalog/database might not be available currently or should not >> be >> > >>>>>> modified yet. How about we support both cases? >> > >>>>>> >> > >>>>>> CREATE TEMPORARY FUNCTION abs >> > >>>>>> -> creates/overrides a built-in function and never consideres >> > current >> > >>>>>> catalog and database; inconsistent with other DDL but acceptable >> for >> > >>>>>> functions I guess. >> > >>>>>> CREATE TEMPORARY FUNCTION cat.db.fun >> > >>>>>> -> creates/overrides a catalog function >> > >>>>>> >> > >>>>>> Regarding "Flink don't have any other built-in objects (tables, >> > views) >> > >>>>>> except functions", this might change in the near future. Take >> > >>>>>> https://issues.apache.org/jira/browse/FLINK-13900 as an example. >> > >>>>>> >> > >>>>>> Thanks, >> > >>>>>> Timo >> > >>>>>> >> > >>>>>> On 14.09.19 01:40, Bowen Li wrote: >> > >>>>>>> Hi Fabian, >> > >>>>>>> >> > >>>>>>> Yes, I agree 1-part/no-override is the least favorable thus I >> > didn't >> > >>>>>>> include that as a voting option, and the discussion is mainly >> > between >> > >>>>>>> 1-part/override builtin and 3-part/not override builtin. >> > >>>>>>> >> > >>>>>>> Re > However, it means that temp functions are differently >> treated >> > >>>> than >> > >>>>>>> other db objects. >> > >>>>>>> IMO, the treatment difference results from the fact that >> functions >> > >>>> are >> > >>>>> a >> > >>>>>>> bit different from other objects - Flink don't have any other >> > >>>> built-in >> > >>>>>>> objects (tables, views) except functions. >> > >>>>>>> >> > >>>>>>> Cheers, >> > >>>>>>> Bowen >> > >>>>>>> >> > >>>>>> >> > >>>> >> > >>>> -- >> > >>>> Xuefu Zhang >> > >>>> >> > >>>> "In Honey We Trust!" >> > >>>> >> > >> >> > > >> > >> > >> >> -- >> Xuefu Zhang >> >> "In Honey We Trust!" >> >
