Hi all,
Thanks for the lively discussion and I will respond to these questions one
by one. However, there are also some common questions and I will answer
together.
@郑 Thanks for your reply. The features mentioned in this flip are only for
those source connectors that implement the SupportsWatermarkPushDown
interface, generating watermarks in other graph locations is not in the
scope of this discussion. Perhaps another flip can be proposed later to
implement this feature.
@Shammon Thanks for your reply. In Flip-296, a rejected alternative is
adding watermark related options in the connector options，we believe that
we should not bind the watermark-related options to a connector to ensure
semantic clarity.
> What will happen if we add watermark related options in `the connector
> options`? Will the connector ignore these options or throw an exception?
> How can we support this?
If user defines different watermark configurations for one table in two
places, I tend to prefer the first place would prevail, but we can also
throw exception or just print logs to prompt the user, which are
implementation details.
> If one table is used by two operators with different watermark params,
> what will happen?
@Martijn Thanks for your reply. I'm sorry that we are not particularly
accurate, this hint is mainly for SQL, not table API.
> While the FLIP talks about watermark options for Table API & SQL, I only
> see proposed syntax for SQL, not for the Table API. What is your proposal
> for the Table API
@Jane Thanks for your reply. For the first question, If the user uses this
hint on those sourse that does not implement the SupportsWatermarkPushDown
interface, it will be completely invalid. The task will run as normal as if
the hint had not been used.
> What's the behavior if there are multiple table sources, among which
> some do not support `SupportsWatermarkPushDown`?
@Jane feedback that 'WATERMARK_PARAMS' is difficult to remember, perhaps
the naming issue can be put to the end of the discussion, because more
people like @Martijn @Shuo are considering whether these configurations
should be put into the DDL or the 'OPTIONS' hint. Here's what I
think, Putting these configs into DDL or putting them into 'OPTIONS' hint
is actually the same thing, because the 'OPTIONS' hint is mainly used to
configure the properties of conenctor. The reason why I want to use a new
hint is to make sure the semantics clear, in my opinion the configuration
of watermark should not be mixed up with connector. However, a new hint
does make it more difficult to use to some extent, for example, when a user
uses both 'OPTIONS' hint and 'WATERMARK_PARAMS' hint. For this point, maby
it is more appropriate to use uniform 'OPTIONS' hint.
On the other hand, if we enrich more watermark option keys in 'OPTIONS'
hints, The question will be what we treat the definatrions of 'OPTIONS'
hint, is this only specific to the connector options or could be more?
Maybe @Jark could share more insights here. In my opion, 'OPTIONS' is only
related to the connector options, which is not like the gernal watermark
options.
Shuo Cheng <njucs...@gmail.com> 于2023年2月22日周三 19:17写道：
> Hi Kui,
>
> Thanks for driving the discussion. It's quite useful to introduce Watermark
> options. I have some questions:
>
> What kind of hints is "WATERMARK_PARAMS"?
> Currently, we have two kinds of hints in Flink: Dynamic Table Options &
> Query Hints. As described in the Flip, "WATERMARK_PARAMS" is more like
> Dynamic Table Options. So two questions arise here:
>
> 1) Are these watermark options to be exposed as connector WITH options? Aa
> described in SQL Hints doc[1], "Dynamic Table Options allow to specify or
> override table options dynamically", which implies that these options can
> also be configured in WITH options.
>
> 2) Do we really need a new hint name like 'WATERMARK_PARAMS', table
> options use "OPTIONS" as hint name, like '/*+
> OPTIONS('csv.ignore-parse-errors'='true') */', maybe we can enrich more
> table option keys for watermark, e.g., /*+
> OPTIONS('watermark.emit-strategy'='ON_PERIODIC') */.
>
>
> [1]
>
> https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/table/sql/queries/hints/
>
> On Wed, Feb 22, 2023 at 10:22 AM kui yuan <catye...@gmail.com> wrote:
>
> > Hi devs,
> >
> >
> > I'd like to start a discussion thread for FLIP-296[1]. This comes from an
> > offline discussion with @Yun Tang, and we hope to enrich table API & SQL
> to
> > support many watermark-related features which were only implemented at
> the
> > datastream API level.
> >
> >
> > Basically, we want to introduce watermark options in table API & SQL via
> > SQL hint named 'WATERMARK_PARAMS' to support features:
> >
> > 1、Configurable watermark emit strategy
> >
> > 2、Dealing with idle sources
> >
> > 3、Watermark alignment
> >
> >
> > Last but not least, thanks to Qingsheng and Jing Zhang for the initial
> > reviews.
> >
> >
> > Looking forward to your thoughts and any feedback is appreciated!
> >
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240884405
> >
> >
> > Best
> >
> > Yuan Kui
> >
>
