Hi Yuan Kui, Thanks for creating the FLIP. A couple of questions / remarks
1. While the FLIP talks about watermark options for Table API & SQL, I only see proposed syntax for SQL, not for the Table API. What is your proposal for the Table API? 2. A rejected alternative is adding watermark related options in the SQL DDL because it would make the DDL complex and lengthy. But looking at the current hints for dynamic table options, we already provide the option to specify or override table options dynamically [1]. So I would think that these watermark options actually should be part of the SQL DDL and that a user can use the dynamic table hints to specify/override these options if needed. The advantage of putting these options in the SQL DDL is that the user has one location where these types of options need to be specified. If we would only do this via hints, the user would need to specify these options in two places. What do you think? Best regards, Martijn [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/hints/#dynamic-table-options On Wed, Feb 22, 2023 at 7:58 AM Shammon FY <zjur...@gmail.com> wrote: > Hi kui > > Thanks for initializing this discussion. I have two questions > > 1. What will happen if we add watermark related options in `the connector > options`? Will the connector ignore these options or throw an exception? > How can we support this? > > 2. If one table is used by two operators with different watermark params, > what will happen? For example, there is a source table T and user submit > the following job > > SELECT T1.id, T1.cnt_val, T2.sum_val FROM (SELECT id, count(val) as cnt_val > FROM T /*+ WATERMARK PARAMS(‘rowtime’ = ’time_column’, ‘rowtime_expression’ > = ’time_column - INTERVAL 5 SECOND') */ GROUP BY id) T1 JOIN (SELECT id, > sum(val) as sum_val FROM T /*+ WATERMARK PARAMS(‘rowtime’ = ’time_column’, > ‘rowtime_expression’ = ’time_column - INTERVAL 10 SECOND') */ GROUP BY id) > T2 ON T1.id=T2.id > > Best, > Shammon > > > On Wed, Feb 22, 2023 at 11:28 AM 郑舒力 <nezhazh...@gmail.com> wrote: > > > Hi kui, > > > > There is a scenario using watermark that cannot be implemented through > SQL > > now： > > We only support specify rowtime column in table DDL，if the rowtime field > > is generated by join dimension table , it cannot be implemented > > > > Can we consider implement through HINTS like : > > Select * from t1 join t2 /*+ WATERMARK PARAMS(‘rowtime’ = ’time_column’, > > ‘rowtime_expression’ = ’time_column - INTERVAL 5 SECOND') */ > > > > > > > > > 2023年2月22日 10:22，kui yuan <catye...@gmail.com> 写道： > > > > > > Hi devs, > > > > > > > > > I'd like to start a discussion thread for FLIP-296[1]. This comes from > an > > > offline discussion with @Yun Tang, and we hope to enrich table API & > SQL > > to > > > support many watermark-related features which were only implemented at > > the > > > datastream API level. > > > > > > > > > Basically, we want to introduce watermark options in table API & SQL > via > > > SQL hint named 'WATERMARK_PARAMS' to support features: > > > > > > 1、Configurable watermark emit strategy > > > > > > 2、Dealing with idle sources > > > > > > 3、Watermark alignment > > > > > > > > > Last but not least, thanks to Qingsheng and Jing Zhang for the initial > > > reviews. > > > > > > > > > Looking forward to your thoughts and any feedback is appreciated! > > > > > > > > > [1] > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240884405 > > > > > > > > > Best > > > > > > Yuan Kui > > > > >
