Re: Watermarks in Event Time windowing

2018-09-14 Thread David Anderson
To clarify one thing: keep in mind that Flink does not support per-key watermarks. Watermarks are typically generated per-source, or in the case of kafka, can be per-partition. An idle source (or in the case of kafka, an idle partition) can prevent an event-time window from being triggered, but you

Re: Watermarks in Event Time windowing

2018-09-13 Thread Taher Koitawala
Yes in many cases what we have faced that let's say in a keyed stream an element of a specific key comes in which triggers a new window. If a corresponding elements of the same key does not arrive a new watermark is not generated for the window to purge. Then we faced issues with flink keeping reco

Re: Watermarks in Event Time windowing

2018-09-13 Thread vino yang
Hi Taher, For some questions, I suggest you read the documentation related to Flink EventTime first, for example [1] About this question: What happens if the watermark is same as the timestamp? Here "timestamp", do you mean the current timestamp of Processing time? If that's the best, it's an id

Watermarks in Event Time windowing

2018-09-13 Thread Taher Koitawala
Hi All, Can someone show a good example of how watermarks need to be generated when using EventTime windows? What happens if the watermark is same as the timestamp? How does the watermark help in the window to be triggered and what if watermarks are kept behind the currentTimestamps in