I would like to refrain from adding additional tasks as much as possible. I agree with Gyula that extending the reader to track watermarks and call a handler whenever the watermark advances would be a nice way to implement this.
On Tue, May 12, 2015 at 2:40 PM, Aljoscha Krettek <aljos...@apache.org> wrote: > Hi Folks, > as I said in the subject. How will this work? I'm in the process about > thinking how to implement low watermarks in Streaming. I'm thinking > that the implementation should be quite similar to how the > checkpointing barriers will be implemented since they also flush out > stuff. > > Now I'm wondering how this will work with merged Streams and the > output selectors (split streams). It seems to me that there are a lot > of paths that elements can take to arrive at operators. The problem I > have is that an operator can only emit a low watermark itself if it > knows that all input operators have sent him a low watermark with that > value (the low watermark is the minimum of the low watermarks of all > upstream operators). I imagine that the checkpoint barriers exhibit > the same behaviour. > > Do we maybe have to add an explicit union (merge) operator and change > how split streams are implemented? > > What are your thoughts? > > Cheers, > Aljoscha >
