Re: Identifying missing events in keyed streams

2018-10-13 Thread Averell
Thank you Fabian. Tried (2), and it's working well. I found one more benefit of (2) over (3) is that it allow me to easily raise multiple levels of alarms for each keyed stream (i.e: minor: missed 2 cycles, major: missed 5 cycles,...) Thanks for your help. Regards, Averell -- Sent from:

Re: Identifying missing events in keyed streams

2018-10-11 Thread Fabian Hueske
I'd go with 2) because the logic is simple and it is (IMO) much easier to understand what is going on and what state is kept. Am Do., 11. Okt. 2018 um 12:42 Uhr schrieb Averell : > Hi Fabian, > > Thanks for the suggestion. > I will try with that support of removing timers. > > I have also tried

Re: Identifying missing events in keyed streams

2018-10-11 Thread Averell
Hi Fabian, Thanks for the suggestion. I will try with that support of removing timers. I have also tried approach (3) - using session windows, and it works: I set session gap to 2 minutes, and use an aggregation window function to keep the amount of in-memory data for each keyed stream to the

Re: Identifying missing events in keyed streams

2018-10-10 Thread Fabian Hueske
Hi Averell, I'd go with approach 2). As of Flink 1.6.0 you can delete timers. But even if you are on a pre-1.6 version, a ProcessFunction would be the way to go, IMO. You don't need to register a timer for each event. Instead, you can register the first timer with the first event and have a

Identifying missing events in keyed streams

2018-10-04 Thread Averell
Hi everyone, I have a keyed stream which is expecting events every fixed interval (let's say 1 minute). I want to raise alarms for any key which has received no events in n-periods. What should be the cheapest way (in term of performance ) to do this? I thought of some solutions, but don't know