Streaming time window

2015-12-10 Thread Martin Neumann
Hej, Is it possible to extract the start and end window time stamps from within a window operator? I have an event time based window that does a simple fold function. I want to put the output into elasticsearch and want to preserve the start and end timestamp of the data so I can directly

Re: Streaming time window

2015-12-10 Thread Fabian Hueske
Hi Martin, you can get the start and end time of a window from the TimeWindow object. The following Scala code snippet shows how to access the window end time (start time is equivalent): .timeWindow(Time.minutes(5)) .trigger(new EarlyCountTrigger(earlyCountThreshold)) .apply { ( key: Int,

Re: Streaming time window

2015-12-10 Thread Fabian Hueske
Sure. You don't need a trigger, but a WindowFunction instead of the FoldFunction. Only the WindowFunction has access to the Window object. Something like this: poissHostStreams .timeWindow(Time.of(WINDOW_SIZE, TimeUnit.MILLISECONDS)) .apply(new WindowFunction

Re: Streaming time window

2015-12-10 Thread Martin Neumann
I will give this a try. Though I'm not sure I can switch over to WindowFunction. I work with potentially huge Windows, the Fold gives me a minimal and constant memory footprint. Switching to WindowFunction will require to keep the Window in Memory before it can be processed (at least to my