Thanks for your proposal. +1 to yuxia, consider watermark-related hints as option hints.
Personally, I am cautious about adding SQL syntax, WATERMARK_PARAMS is also SQL syntax to some extent. We can use OPTIONS to meet this requirement if possible. Best, Jingsong On Thu, Feb 23, 2023 at 10:41 AM yuxia <luoyu...@alumni.sjtu.edu.cn> wrote: > > Hi, Yuan Kui. > Thanks for driving it. > IMO, the 'OPTIONS' hint may be not only specific to the connector options. > Just as a reference, we also have `sink.parallelism`[1] as a connector > options. It enables > user to specific the writer's parallelism dynamically per-query. > > Personally, I perfer to consider watermark-related hints as option hints. So, > user can define a default watermark strategy for the table, and if user > dosen't needed to changes it, they need to do nothing in their query instead > of specific it ervery time. > > [1] > https://nightlies.apache.org/flink/flink-docs-master/zh/docs/connectors/table/filesystem/#sink-parallelism > > Best regards, > Yuxia > > Best regards, > Yuxia > > ----- 原始邮件 ----- > 发件人: "kui yuan" <catye...@gmail.com> > 收件人: "dev" <dev@flink.apache.org> > 抄送: "Jark Wu" <imj...@gmail.com> > 发送时间: 星期三, 2023年 2 月 22日 下午 10:08:11 > 主题: Re: [DISCUSS] FLIP-296: Watermark options for table API & SQL > > Hi all, > > Thanks for the lively discussion and I will respond to these questions one > by one. However, there are also some common questions and I will answer > together. > > @郑 Thanks for your reply. The features mentioned in this flip are only for > those source connectors that implement the SupportsWatermarkPushDown > interface, generating watermarks in other graph locations is not in the > scope of this discussion. Perhaps another flip can be proposed later to > implement this feature. > > @Shammon Thanks for your reply. In Flip-296, a rejected alternative is > adding watermark related options in the connector options，we believe that > we should not bind the watermark-related options to a connector to ensure > semantic clarity. > > > What will happen if we add watermark related options in `the connector > > options`? Will the connector ignore these options or throw an exception? > > How can we support this? > > If user defines different watermark configurations for one table in two > places, I tend to prefer the first place would prevail, but we can also > throw exception or just print logs to prompt the user, which are > implementation details. > > > If one table is used by two operators with different watermark params, > > what will happen? > > @Martijn Thanks for your reply. I'm sorry that we are not particularly > accurate, this hint is mainly for SQL, not table API. > > > While the FLIP talks about watermark options for Table API & SQL, I only > > see proposed syntax for SQL, not for the Table API. What is your proposal > > for the Table API > > @Jane Thanks for your reply. For the first question, If the user uses this > hint on those sourse that does not implement the SupportsWatermarkPushDown > interface, it will be completely invalid. The task will run as normal as if > the hint had not been used. > > > What's the behavior if there are multiple table sources, among which > > some do not support `SupportsWatermarkPushDown`? > > @Jane feedback that 'WATERMARK_PARAMS' is difficult to remember, perhaps > the naming issue can be put to the end of the discussion, because more > people like @Martijn @Shuo are considering whether these configurations > should be put into the DDL or the 'OPTIONS' hint. Here's what I > think, Putting these configs into DDL or putting them into 'OPTIONS' hint > is actually the same thing, because the 'OPTIONS' hint is mainly used to > configure the properties of conenctor. The reason why I want to use a new > hint is to make sure the semantics clear, in my opinion the configuration > of watermark should not be mixed up with connector. However, a new hint > does make it more difficult to use to some extent, for example, when a user > uses both 'OPTIONS' hint and 'WATERMARK_PARAMS' hint. For this point, maby > it is more appropriate to use uniform 'OPTIONS' hint. > On the other hand, if we enrich more watermark option keys in 'OPTIONS' > hints, The question will be what we treat the definatrions of 'OPTIONS' > hint, is this only specific to the connector options or could be more? > Maybe @Jark could share more insights here. In my opion, 'OPTIONS' is only > related to the connector options, which is not like the gernal watermark > options. > > > > Shuo Cheng <njucs...@gmail.com> 于2023年2月22日周三 19:17写道： > > > Hi Kui, > > > > Thanks for driving the discussion. It's quite useful to introduce Watermark > > options. I have some questions: > > > > What kind of hints is "WATERMARK_PARAMS"? > > Currently, we have two kinds of hints in Flink: Dynamic Table Options & > > Query Hints. As described in the Flip, "WATERMARK_PARAMS" is more like > > Dynamic Table Options. So two questions arise here: > > > > 1) Are these watermark options to be exposed as connector WITH options? Aa > > described in SQL Hints doc[1], "Dynamic Table Options allow to specify or > > override table options dynamically", which implies that these options can > > also be configured in WITH options. > > > > 2) Do we really need a new hint name like 'WATERMARK_PARAMS', table > > options use "OPTIONS" as hint name, like '/*+ > > OPTIONS('csv.ignore-parse-errors'='true') */', maybe we can enrich more > > table option keys for watermark, e.g., /*+ > > OPTIONS('watermark.emit-strategy'='ON_PERIODIC') */. > > > > > > [1] > > > > https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/table/sql/queries/hints/ > > > > On Wed, Feb 22, 2023 at 10:22 AM kui yuan <catye...@gmail.com> wrote: > > > > > Hi devs, > > > > > > > > > I'd like to start a discussion thread for FLIP-296[1]. This comes from an > > > offline discussion with @Yun Tang, and we hope to enrich table API & SQL > > to > > > support many watermark-related features which were only implemented at > > the > > > datastream API level. > > > > > > > > > Basically, we want to introduce watermark options in table API & SQL via > > > SQL hint named 'WATERMARK_PARAMS' to support features: > > > > > > 1、Configurable watermark emit strategy > > > > > > 2、Dealing with idle sources > > > > > > 3、Watermark alignment > > > > > > > > > Last but not least, thanks to Qingsheng and Jing Zhang for the initial > > > reviews. > > > > > > > > > Looking forward to your thoughts and any feedback is appreciated! > > > > > > > > > [1] > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240884405 > > > > > > > > > Best > > > > > > Yuan Kui > > > > >
