Hi Jark, Let's recall FLIP-63, We supported these syntax in hive dialect at 1.9. All of my reasons for launching FLIP-63 are to bring partition support to Flink itself. Not only batch, but also we have the need to stream jobs to write partition files today, which is also one of our very important application scenarios.
The original intention of FLIP-63 is to bring all partition syntax to Flink, but in the end you and I have some different opinion in creating partition table, so our consensus is to leave it in hive dialect, only it. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support Best, Jingsong Lee On Thu, Dec 12, 2019 at 4:05 PM Jark Wu <imj...@gmail.com> wrote: > Hi jingsong, > > Watermark is not a standard syntax, that's why we had a FLIP and long > discussion to add it to > Flink's SQL syntax. I think if we want to add INSERT OVERWRITE and > PARTITION syntax to > Flink's own syntax, we also need a FLIP or a VOTE, and this may can't > happen soon (we should > hear more people's opinions on this). > > Regarding to the sql-dialect configuration, I was not saying to involve the > whole FLIP-89. I mean we can just > start a VOTE to expose it as `table.planner.sql-dialect` and include it in > 1.10. The change can be very small, by > adding a ConfigOption and changing the implementation of > TableConfig#getSqlDialect/setSqlDialect. I believe > it is smaller and safer than changing the parser. > > Btw, I cc'ed Yu Li and Gary into the discussion, because release managers > should be aware of this. > > Best, > Jark > > > > On Thu, 12 Dec 2019 at 11:47, Danny Chan <yuzhao....@gmail.com> wrote: > > > Thanks Jingsong for bring up this discussion ~ > > > > After reviewing FLIP-63, it seems that we have made a conclusion for the > > syntax > > > > - INSERT OVERWRITE ... > > - INSERT INTO … PARTITION > > > > Which means that they should not have the Hive dialect limitation, so I’m > > inclined that the behaviors for SQL-CLI is unexpected, or a “bug” that > need > > to fix. > > > > We did not make a conclusion for the syntax: > > > > - CREATE TABLE … PARTITIONED BY ... > > > > Which means that the behavior of it is under-discussion, so it is okey to > > be without the HIVE dialect limitation, we do not actually have any table > > sources/sinks that support such a DDL so for current code base, users > > should not be influenced by the behaviors change. > > > > So I’m > > > > +1 to remove the hive dialect limitations for INSERT OVERWRITE and INSERT > > PARTITION > > +0 to add yaml dialect conf to SQL-CLI because FLIP-89 is not finished > > yet, we better do this until FLIP-89 is resolved. > > > > Best, > > Danny Chan > > 在 2019年12月11日 +0800 PM5:29，Jingsong Li <jingsongl...@gmail.com>，写道： > > > Hi Dev, > > > > > > After cutting out the branch of 1.10, I tried the following functions > of > > > SQL-CLI and found that it does not support: > > > - insert overwrite > > > - PARTITION (partcol1=val1, partcol2=val2 ...) > > > The SQL pattern is: > > > INSERT { INTO | OVERWRITE } TABLE tablename1 [PARTITION (partcol1=val1, > > > partcol2=val2 ...) select_statement1 FROM from_statement; > > > It is a surprise to me. > > > The reason is that we only allow these two grammars in hive dialect. > And > > > SQL-CLI does not have an interface to switch dialects. > > > > > > Because it directly hinders the SQL-CLI's insert syntax in hive > > integration > > > and seriously hinders the practicability of SQL-CLI. > > > And we have introduced these two grammars in FLIP-63 [1] to Flink. > > > Here are my question: > > > 1.Should we remove hive dialect limitation for these two grammars? > > > 2.Should we fix this in 1.10? > > > > > > [1] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support > > > > > > Best, > > > Jingsong Lee > > > -- Best, Jingsong Lee
