Re: Deterministic processing with out-of-order streams

2016-11-07 Thread Vinay Patil
Hi Jaromir, You can make use of Custom Trigger and set the allowed lateness to max value. I have kept the custom trigger code (EventTimeTrigger) same as Flink 1.0.3, doing this the late elements will not be discarded and they will be assigned single windows , now you can decide what you want to

Re: Deterministic processing with out-of-order streams

2016-11-07 Thread Till Rohrmann
You're right if you want to guarantee a deterministic computation for an arbitrary allowed lateness. In the general case, you would never be able to calculate the final result of a window in a finite time, because there might always be another element which arrives later. However, for most

Re: Deterministic processing with out-of-order streams

2016-11-07 Thread Jaromir Vanek
Hi Till, thank you for your answer. I am afraid defining an allowed lateness won't help. It will just change the problem by constant time. If we agree an element can come in arbitrary time after watermark (depends on the network latency), it may be assigned to the window or may be not if comes

Re: Deterministic processing with out-of-order streams

2016-11-07 Thread Till Rohrmann
Hi Jaromir, deterministic processing with late elements is indeed more difficult than without them. What you have to do is to send updates to your downstream operators in case that you see late elements. This can either be an incremental update or a retraction with the corrected value. It