Hi Congxian, Thank's for your feedback. You raised a point I also already thought about. As "assignTimestampsAndWatermarks" creates an operator extending the standard AbstractUdfStreamOperator, I can also implement a RichFunction watermark assigner with full state access. In my case, I was also wondering whether it's a good idea to have a stateful watermark assigner or whether its more practical to have no state on start and build my histogram over time, new with each job restart... That's also why I asked on the mailing list so I can feedback of other people customizing the watermark assigners.
Best regards Theo Von: "Congxian Qiu" <[email protected]> An: "Theo Diefenthal" <[email protected]> CC: "user" <[email protected]> Gesendet: Samstag, 30. Mai 2020 05:06:12 Betreff: Re: Auto adjusting watermarks? Hi Could it be store a histogram data in custom ` BoundedOutOfOrdernessTimestampExtractor` and adjust the `maxOutOfOrderness` according to the histogram data ok for you case? (be careful, such histogram data would not snapshot out when checkpointing) Best, Congxian Theo Diefenthal < [ mailto:[email protected] | [email protected] ] > 于2020年5月30日周六 上午4:35写道: Hi there, Currently I have a job pipeline reading data from > 10 different kind of sources with each having different out-of-orderness characteristics. I am currently working on adjusting the watermarks for each source "properly". I work with BoundedOutOfOrdernessTimestampExtractor and, as usual, I want the maxOutOfOrderness as low as possible while still keeping as much elements as possible in time as late arrivals trigger rather expensive computations. Now I thought, what I probably want is something like "I want to have about 99.9% of my elements within the allowed lateness". Of course, I don't know the future events out-of-orderness, but I can predict it from the past, e.g. via a histogram with a 99.9% percentile, and adjust the maxOutOfOrdernesss dynamically. As Flink provides rather simplified Timestamp Assigner only but allows me to create my own ones with arbitrary complexity, I was wondering if somebody of you already did something like that, if that's a viable approach and I'm on a good track here? Best regards Theo
