Hi Ray, in principle, watermarks can be injected anywhere in a stream by calling DataStream.assignTimestampsAndWatermarks(). However, timestamps are usually injected as soon as possible after a stream in ingested (before the first shuffle). The reason is that watermarks depend on the order of events (and their timestamps) in the stream. While Flink guarantees the order of events within a partition, a shuffle interleaves events of different partitions in an unpredictable way such that it is not possible to reason about the order of timestamps afterwards.
The most common way to inject watermarks is directly inside of a SourceFunction or with a TimestampAssigner before the first shuffle. Best, Fabian 2017-06-09 0:46 GMT+02:00 Ray Ruvinskiy <[email protected]>: > I’m trying to build a mental model of how watermarks get injected into the > stream. Suppose I have a stream with a parallel source, and I’m running a > cluster with multiple task managers. Does each parallel source reader > inject watermarks, which are then forwarded to downstream consumers and > shuffled between task managers? Or are watermarks created after the > shuffle, when the stream records reach their destined task manager and > right before they’re processed by the operator? > > > > Thanks, > > > > Ray >
