Hi Timo, I am OK if you think they are not bug and they should not be included in 1.10.
I think they have been accepted in FLIP-63. And there is no objection. It has been more than three months since the discussion of FLIP-63. It's been six months since Flink added these two syntaxs. But I can also start discussion and vote thread for FLIP-63 again, to make sure once again that everyone is happy. Best, Jingsong Lee On Thu, Dec 12, 2019 at 11:35 PM Timo Walther <twal...@apache.org> wrote: > Hi Jingsong, > > I will also add my opinion here for future discussions: > > We had long discussions around SQL syntax in the past (e.g. for > WATERMARK or the concept of SYSTEM/TEMPORARY catalog objects) but in the > end all parties were happy and we came up with a good long-term solution > that is unlikely to be changed in the near future. IMHO we can differ > from the standard esp. for DDL statements (every SQL vendors has custom > syntax there) but still we should have time to hear many opinions. > However, for DQL and DML statements we need to be even more cautious > because here we enter SQL standard areas. > > Happy to discuss a partion syntax for 1.11 and go with the solution that > Jark proposed using a config option. > > Thanks, > Timo > > > On 12.12.19 09:40, Jark Wu wrote: > > Hi Jingsong, > > > > Thanks for the explanation, I think I misunderstood your point at the > > begining. > > As FLIP-63 proposed, INSERT OVERWRITE and INSERT PARTITION syntax are > added > > to Flink's > > SQL syntax, but CREATE PARTITION TABLE should be limited under Hive > > dialect. > > However, the current implementation is opposite, INSERT OVERWRITE > > and INSERT PARTITION are > > under the dialect limitation, but CREATE PARTITION TABLE is not. > > > > So it is indeed a bug which should be fixed in 1.10. > > > > Best, > > Jark > > > > On Thu, 12 Dec 2019 at 16:35, Jingsong Li <jingsongl...@gmail.com> > wrote: > > > >> Hi Jark, > >> > >> Let's recall FLIP-63, > >> We supported these syntax in hive dialect at 1.9. All of my reasons for > >> launching FLIP-63 are to bring partition support to Flink itself. > >> Not only batch, but also we have the need to stream jobs to write > partition > >> files today, which is also one of our very important application > scenarios. > >> > >> The original intention of FLIP-63 is to bring all partition syntax to > >> Flink, but in the end you and I have some different opinion in creating > >> partition table, so our consensus is to leave it in hive dialect, only > it. > >> > >> [1] > >> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support > >> > >> Best, > >> Jingsong Lee > >> > >> On Thu, Dec 12, 2019 at 4:05 PM Jark Wu <imj...@gmail.com> wrote: > >> > >>> Hi jingsong, > >>> > >>> Watermark is not a standard syntax, that's why we had a FLIP and long > >>> discussion to add it to > >>> Flink's SQL syntax. I think if we want to add INSERT OVERWRITE and > >>> PARTITION syntax to > >>> Flink's own syntax, we also need a FLIP or a VOTE, and this may > can't > >>> happen soon (we should > >>> hear more people's opinions on this). > >>> > >>> Regarding to the sql-dialect configuration, I was not saying to involve > >> the > >>> whole FLIP-89. I mean we can just > >>> start a VOTE to expose it as `table.planner.sql-dialect` and include it > >> in > >>> 1.10. The change can be very small, by > >>> adding a ConfigOption and changing the implementation of > >>> TableConfig#getSqlDialect/setSqlDialect. I believe > >>> it is smaller and safer than changing the parser. > >>> > >>> Btw, I cc'ed Yu Li and Gary into the discussion, because release > managers > >>> should be aware of this. > >>> > >>> Best, > >>> Jark > >>> > >>> > >>> > >>> On Thu, 12 Dec 2019 at 11:47, Danny Chan <yuzhao....@gmail.com> wrote: > >>> > >>>> Thanks Jingsong for bring up this discussion ~ > >>>> > >>>> After reviewing FLIP-63, it seems that we have made a conclusion for > >> the > >>>> syntax > >>>> > >>>> - INSERT OVERWRITE ... > >>>> - INSERT INTO … PARTITION > >>>> > >>>> Which means that they should not have the Hive dialect limitation, so > >> I’m > >>>> inclined that the behaviors for SQL-CLI is unexpected, or a “bug” that > >>> need > >>>> to fix. > >>>> > >>>> We did not make a conclusion for the syntax: > >>>> > >>>> - CREATE TABLE … PARTITIONED BY ... > >>>> > >>>> Which means that the behavior of it is under-discussion, so it is okey > >> to > >>>> be without the HIVE dialect limitation, we do not actually have any > >> table > >>>> sources/sinks that support such a DDL so for current code base, users > >>>> should not be influenced by the behaviors change. > >>>> > >>>> So I’m > >>>> > >>>> +1 to remove the hive dialect limitations for INSERT OVERWRITE and > >> INSERT > >>>> PARTITION > >>>> +0 to add yaml dialect conf to SQL-CLI because FLIP-89 is not finished > >>>> yet, we better do this until FLIP-89 is resolved. > >>>> > >>>> Best, > >>>> Danny Chan > >>>> 在 2019年12月11日 +0800 PM5:29，Jingsong Li <jingsongl...@gmail.com>，写道： > >>>>> Hi Dev, > >>>>> > >>>>> After cutting out the branch of 1.10, I tried the following functions > >>> of > >>>>> SQL-CLI and found that it does not support: > >>>>> - insert overwrite > >>>>> - PARTITION (partcol1=val1, partcol2=val2 ...) > >>>>> The SQL pattern is: > >>>>> INSERT { INTO | OVERWRITE } TABLE tablename1 [PARTITION > >> (partcol1=val1, > >>>>> partcol2=val2 ...) select_statement1 FROM from_statement; > >>>>> It is a surprise to me. > >>>>> The reason is that we only allow these two grammars in hive dialect. > >>> And > >>>>> SQL-CLI does not have an interface to switch dialects. > >>>>> > >>>>> Because it directly hinders the SQL-CLI's insert syntax in hive > >>>> integration > >>>>> and seriously hinders the practicability of SQL-CLI. > >>>>> And we have introduced these two grammars in FLIP-63 [1] to Flink. > >>>>> Here are my question: > >>>>> 1.Should we remove hive dialect limitation for these two grammars? > >>>>> 2.Should we fix this in 1.10? > >>>>> > >>>>> [1] > >>>>> > >>>> > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support > >>>>> > >>>>> Best, > >>>>> Jingsong Lee > >>>> > >>> > >> > >> > >> -- > >> Best, Jingsong Lee > >> > > > > -- Best, Jingsong Lee
