Thanks for the insightful sharing! Best, Zhanghao Chen ________________________________ From: Lasse Nedergaard <lassenedergaardfl...@gmail.com> Sent: Thursday, May 15, 2025 13:10 To: Zhanghao Chen <zhanghao.c...@outlook.com> Cc: mosin...@yandex.ru <mosin...@yandex.ru>; user@flink.apache.org <user@flink.apache.org> Subject: Re: Keyed watermarks: A fine-grained watermark generation for Apache Flink
Hi I have talked with the community about this for many years last time at Flink forward 2024 in Berlin. The use case are simple. If you receive data from IoT devices over the gsm network. The clock on all the devices aren’t synchronised the IoT devices can buffer data to reduce the cost for transmitting data and they can be offline for many reasons and therefore can the data be delayed for hours and days. If you want to use window functions for each device you can’t do that without having a watermark for each device. Make sense? Med venlig hilsen / Best regards Lasse Nedergaard Den 15. maj 2025 kl. 03.23 skrev Zhanghao Chen <zhanghao.c...@outlook.com>: Thanks for sharing! It is an interesting idea. The generalized watermark [1] introduced in DataStreamV2 might be sufficient to implement it. It'll be great if you could share more contexts on why this is useful in your pipelines. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-467%3A+Introduce+Generalized+Watermarks Best, Zhanghao Chen ________________________________ From: Мосин Николай <mosin...@yandex.ru> Sent: Thursday, May 15, 2025 3:58 To: user@flink.apache.org <user@flink.apache.org> Subject: Keyed watermarks: A fine-grained watermark generation for Apache Flink I found paper https://scholar.google.com/scholar?q=10.1016/j.future.2025.107796 where described Keyed Watermarks that is what I need in my pipelines. Does anyone know is it planned to implement Keyed Watermarks in Flink and when?
