Hi Xu Huang, Thanks for the proposal!
I have a question: If I have a large number of generalized watermarks that need to be created, where should they be declared? Should they be declared only once in a single Source, or in all operators that need to send, receive, and process them? Best regards, Junrui Xu Huang <huangxu.wal...@gmail.com> 于2024年12月10日周二 15:21写道： > Hi Devs, > > Jeyhun Karimov, Weijie Guo and I would like to initiate a discussion about > FLIP-467: Introduce Generalized Watermarks [1]. > > Based on our findings, we recognize the need for specific events that > require propagation and alignment across streams, functioning similarly to > watermarks. An example of this is the IsProcessingBacklog event proposed in > FLIP-309 [2]. > > > This has inspired us to create a more generalized watermark framework that > transcends traditional event time semantics. The generalized watermark > framework allows users to define a variety of events that can be emitted > from the source or other operators, propagate through the streams, and be > received by downstream operators with aligned properties. With this > abstraction, users and developers can design specialized events according > to their needs, such as EventTime watermark or idleness watermark status. > > > Note that this feature only worked for DataStream V2. > > For more details, please refer to FLIP-467 [1]. We look forward to your > feedback. > > > Best, > > Jeyhun Karimov, Weijie Guo and Xu Huang > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-467%3A+Introduce+Generalized+Watermarks > > [2] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-309%3A+Support+using+larger+checkpointing+interval+when+source+is+processing+backlog >
