Re: Efficiently processing sparse events in a time windows

2020-09-29 Thread Steven Murdoch
Thanks David, this is very helpful. I'm glad that it's not just that I had missed something obvious from the (generally very clear) documentation. I found various features that felt almost right (e.g. the priority queue behind Timers) but nothing that did the job. The temporal state idea does

Re: Efficiently processing sparse events in a time windows

2020-09-24 Thread David Anderson
Steven, I'm pretty sure this is a scenario that doesn't have an obvious good solution. As you have discovered, the window API isn't much help; using a process function does make sense. The challenge is finding a data structure to use in keyed state that can be efficiently accessed and updated.

Efficiently processing sparse events in a time windows

2020-09-23 Thread Steven Murdoch
Hello, I am trying to do something that seems like it should be quite simple but I haven’t found an efficient way to do this with Flink and I expect I’m missing something obvious here. The task is that I would like to process a sequence of events when a certain number appear within a keyed