Hi Xu Huang, I noticed your discussions mentioned that the identifier of watermarks needs to be global across the whole job. Therefore, would it be better to add to the FLIP a proposal to the naming convention of identifiers? For example, the identifiers are encouraged to be named with its holding module/connector as a prefix, like "runtime-backlog”. It is not a compulsory requirement, and is more like a suggestion.
Best, Yufneng > 2024年12月17日 11:13，Xu Huang <huangxu.wal...@gmail.com> 写道： > > Thank you for participating in the discussion. > > @jrlee....@gmail.com <jrlee....@gmail.com> >> If I have a large number of generalized watermarks that need to be > created, where should they be declared? > > In the current design, the generalized watermark should be declared only > once in the first user-defined function or source reader that needs to > create and send it downstream. > Furthermore, all watermark identifiers must remain unique within a single > application. Once the upstream operator declares a generalized watermark, > the downstream operator will be aware of it and will be able to process it > accordingly. > > Best, Xu Huang > > Junrui Lee <jrlee....@gmail.com> 于2024年12月16日周一 19:40写道： > >> Hi Xu Huang, >> >> Thanks for the proposal! >> >> I have a question: If I have a large number of generalized watermarks that >> need to be created, where should they be declared? Should they be declared >> only once in a single Source, or in all operators that need to send, >> receive, and process them? >> >> Best regards, >> >> Junrui >> >> Xu Huang <huangxu.wal...@gmail.com> 于2024年12月10日周二 15:21写道： >> >>> Hi Devs, >>> >>> Jeyhun Karimov, Weijie Guo and I would like to initiate a discussion >> about >>> FLIP-467: Introduce Generalized Watermarks [1]. >>> >>> Based on our findings, we recognize the need for specific events that >>> require propagation and alignment across streams, functioning similarly >> to >>> watermarks. An example of this is the IsProcessingBacklog event proposed >> in >>> FLIP-309 [2]. >>> >>> >>> This has inspired us to create a more generalized watermark framework >> that >>> transcends traditional event time semantics. The generalized watermark >>> framework allows users to define a variety of events that can be emitted >>> from the source or other operators, propagate through the streams, and be >>> received by downstream operators with aligned properties. With this >>> abstraction, users and developers can design specialized events according >>> to their needs, such as EventTime watermark or idleness watermark status. >>> >>> >>> Note that this feature only worked for DataStream V2. >>> >>> For more details, please refer to FLIP-467 [1]. We look forward to your >>> feedback. >>> >>> >>> Best, >>> >>> Jeyhun Karimov, Weijie Guo and Xu Huang >>> >>> [1] >>> >>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-467%3A+Introduce+Generalized+Watermarks >>> >>> [2] >>> >>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-309%3A+Support+using+larger+checkpointing+interval+when+source+is+processing+backlog >>> >>
