Yes, metrics from IoT is my case now.
In additional to unsync clocks I also have devices that could buffer data when offline and resend it later when become online and that data also must be processed in common pipeline. But now it will be marked as 'late' and will be drop.
I do some workarounds, but all my attempts almost failed due to the lack of timers that would be relay on WM and which I do not use now.
----------------
Кому: Zhanghao Chen (zhanghao.c...@outlook.com);
Копия: user@flink.apache.org;
Тема: Keyed watermarks: A fine-grained watermark generation for Apache Flink;
15.05.2025, 08:10, "Lasse Nedergaard" <lassenedergaardfl...@gmail.com>:
HiI have talked with the community about this for many years last time at Flink forward 2024 in Berlin.The use case are simple. If you receive data from IoT devices over the gsm network. The clock on all the devices aren’t synchronised the IoT devices can buffer data to reduce the cost for transmitting data and they can be offline for many reasons and therefore can the data be delayed for hours and days.If you want to use window functions for each device you can’t do that without having a watermark for each device.Make sense?Med venlig hilsen / Best regardsLasse NedergaardDen 15. maj 2025 kl. 03.23 skrev Zhanghao Chen <zhanghao.c...@outlook.com>:
Thanks for sharing! It is an interesting idea. The generalized watermark [1] introduced in DataStreamV2 might be sufficient to implement it. It'll be great if you could share more contexts on why this is useful in your pipelines.Best,Zhanghao ChenFrom: Мосин Николай <mosin...@yandex.ru>
Sent: Thursday, May 15, 2025 3:58
To: user@flink.apache.org <user@flink.apache.org>
Subject: Keyed watermarks: A fine-grained watermark generation for Apache FlinkI found paper https://scholar.google.com/scholar?q=10.1016/j.future.2025.107796 where described Keyed Watermarks that is what I need in my pipelines. Does anyone know is it planned to implement Keyed Watermarks in Flink and when?
