Hi Kui, Thanks for bringing this to the discussion. Toward the FLIP, I have several questions.
1. What's the behavior if there are multiple table sources, among which some do not support `SupportsWatermarkPushDown`? > So the features that this flip intends to support are only for those > sources that implement the `SupportsWatermarkPushDown` interface. > 2. Is there any reason for having a different parameter name for the global and hint configuration? Personally, I feel this is a little hard to remember. Can we unify the parameter naming? > For the 'ON_EVENT' strategy, option ‘emit-gap-on-event’ can configure how > many events to emit a watermark, the default value is 1. We will also add > a global parameter 'table.exec.watermark-emit.gap' to achieve the same > goal, which will be valid for each source and will ease the user's > configuration to some extent. > 3. As Martijn mentioned, the API changes are absent in the FLIP. Can you add more details? Best, Jane On Wed, Feb 22, 2023 at 4:29 PM Martijn Visser <martijnvis...@apache.org> wrote: > Hi Yuan Kui, > > Thanks for creating the FLIP. A couple of questions / remarks > > 1. While the FLIP talks about watermark options for Table API & SQL, I only > see proposed syntax for SQL, not for the Table API. What is your proposal > for the Table API? > > 2. A rejected alternative is adding watermark related options in the SQL > DDL because it would make the DDL complex and lengthy. But looking at the > current hints for dynamic table options, we already provide the option to > specify or override table options dynamically [1]. So I would think that > these watermark options actually should be part of the SQL DDL and that a > user can use the dynamic table hints to specify/override these options if > needed. The advantage of putting these options in the SQL DDL is that the > user has one location where these types of options need to be specified. If > we would only do this via hints, the user would need to specify these > options in two places. What do you think? > > Best regards, > > Martijn > > [1] > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/hints/#dynamic-table-options > > On Wed, Feb 22, 2023 at 7:58 AM Shammon FY <zjur...@gmail.com> wrote: > > > Hi kui > > > > Thanks for initializing this discussion. I have two questions > > > > 1. What will happen if we add watermark related options in `the connector > > options`? Will the connector ignore these options or throw an exception? > > How can we support this? > > > > 2. If one table is used by two operators with different watermark params, > > what will happen? For example, there is a source table T and user submit > > the following job > > > > SELECT T1.id, T1.cnt_val, T2.sum_val FROM (SELECT id, count(val) as > cnt_val > > FROM T /*+ WATERMARK PARAMS(‘rowtime’ = ’time_column’, > ‘rowtime_expression’ > > = ’time_column - INTERVAL 5 SECOND') */ GROUP BY id) T1 JOIN (SELECT id, > > sum(val) as sum_val FROM T /*+ WATERMARK PARAMS(‘rowtime’ = > ’time_column’, > > ‘rowtime_expression’ = ’time_column - INTERVAL 10 SECOND') */ GROUP BY > id) > > T2 ON T1.id=T2.id > > > > Best, > > Shammon > > > > > > On Wed, Feb 22, 2023 at 11:28 AM 郑舒力 <nezhazh...@gmail.com> wrote: > > > > > Hi kui, > > > > > > There is a scenario using watermark that cannot be implemented through > > SQL > > > now： > > > We only support specify rowtime column in table DDL，if the rowtime > field > > > is generated by join dimension table , it cannot be implemented > > > > > > Can we consider implement through HINTS like : > > > Select * from t1 join t2 /*+ WATERMARK PARAMS(‘rowtime’ = > ’time_column’, > > > ‘rowtime_expression’ = ’time_column - INTERVAL 5 SECOND') */ > > > > > > > > > > > > > 2023年2月22日 10:22，kui yuan <catye...@gmail.com> 写道： > > > > > > > > Hi devs, > > > > > > > > > > > > I'd like to start a discussion thread for FLIP-296[1]. This comes > from > > an > > > > offline discussion with @Yun Tang, and we hope to enrich table API & > > SQL > > > to > > > > support many watermark-related features which were only implemented > at > > > the > > > > datastream API level. > > > > > > > > > > > > Basically, we want to introduce watermark options in table API & SQL > > via > > > > SQL hint named 'WATERMARK_PARAMS' to support features: > > > > > > > > 1、Configurable watermark emit strategy > > > > > > > > 2、Dealing with idle sources > > > > > > > > 3、Watermark alignment > > > > > > > > > > > > Last but not least, thanks to Qingsheng and Jing Zhang for the > initial > > > > reviews. > > > > > > > > > > > > Looking forward to your thoughts and any feedback is appreciated! > > > > > > > > > > > > [1] > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240884405 > > > > > > > > > > > > Best > > > > > > > > Yuan Kui > > > > > > > > >
